Adversarial Environment Design via Regret-Guided Diffusion Models
Authors: Hojun Chung
Junseo Lee
Minsoo Kim
Dohyeong Kim
Songhwai Oh
Venue: NeurIPS 2024 spotlight (2024)
Keywords: deep reinforcement learning
curriculum learning
environment design
Primary area: reinforcement_learning
TL;DR: We propose a novel UED algorithm that uses regret-guided diffusion models to improve agent robustness.

Abstract: Training agents that are robust to environmental changes remains a significant challenge in deep reinforcement learning (RL). Unsupervised environment design (UED) has recently emerged to address this issue by generating a set of training environments tailored to the agent's capabilities. While prior works demonstrate that UED has the potential to learn a robust policy, their performance is constrained by the capabilities of the environment generation. To this end, we propose a novel UED algorithm, adversarial environment design via regret-guided diffusion models (ADD). The proposed method guides the diffusion-based environment generator with the regret of the agent to produce environments that the agent finds challenging but conducive to further improvement. By exploiting the representation power of diffusion models, ADD can directly generate adversarial environments while maintaining the diversity of training environments, enabling the agent to effectively learn a robust policy. Our experimental results demonstrate that the proposed method successfully generates an instructive curriculum of environments, outperforming UED baselines in zero-shot generalization across novel, out-of-distribution environments.

Adversarial Environment Design via Regret-Guided Diffusion Models
Hojun Chung', Junseo Lee', Minsoo Kim', Dohyeong Kim?, and Songhwai 12,+
'Interdisciplinaryy Program Artificial Intelligence and ASRI, Scoul National University DDpartment of Electrical and Computer Engineering and ASRI, Seoul National University [hojun. chung, junseo.lee, minsoo .kim, dohyeong. kim} @r snu, ac ac.kr, songhwai@amu. ac.kr
Abstract
Training agents that are robust to environmental changes remains significant challenge in deep reinforcement learning (RL). Unsupervised environment design (UED) has recently emerged address this issue by generating a set of training environments tailored to the agent's s capabilities. While prior works demonstrate that UED has the potential learn a robust policy, their performance constrained by the capabilities of the environment generation. To this end, we propose a novel UED algorithm, adversarial environment design via regret-guided diffusion models (ADD). The proposed method guides the diffusion-basee environment generator with the regret of the agent to produce environments that the agent finds challenging but conducive to further improvement. By exploiting the representation power of diffusion models, ADD can directly generate adversarial environments while maintaining the diversity of training environments, enabling the agent effectively learn a robust policy. Our experimental results demonstrate that the proposed method successfully generates instructive curriculum of environments, outperforming UED baselines in zero-shot generalization across novel, out-of- distribution environments. Project page: https///llbbsmaagiiuubiippppeees githubio/projjes//ADD
1 Introduction
Deep reinforcement learning (RL) has achieved great success in various challenging domains, such as Atari [1], GO [2], and real-world robotics tasks [3, 4]. Despite the progress, deep RI. agent struggles with the generalization problem; it often fails in unseen environments even with small difference from the training environment distribution [5, 6]. To train well-generalizing policies, various prior works have used domain randomization (DR) [7, 8, 9], which provides RL agents with randomly generated environments. While DR enhances the diversity of the training environments, it requires a large number of trials to generate meaningful structures in high-dimensional domains. Curriculum reinforcement learning [10, 11] [] has been demonstrated to address these issues by pro- viding instructive sequences of environments. Since manually designing effective curriculum for complicated tasks is challenging, prior works [12, 13] focus on generating curricula that consider the current agent's capabilities. Recently, unsupervised environment design (UED, [14]) has emerged as a scalable approach, notable for its advantage of requiring no prior knowledge. UED algorithms alternate between training the policy and designing training environments that maximize the regret of the agent. This closed-loop framework ensures the agent learns minimax regret policy [15], assuming that the two-player game between the agent and the environment generator reaches the Nash equilibrium.
"Corresponding author: Songhwai Oh
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

There are two main approaches for UED: 1) learning-basee methods, which employ an environment generator trained via reinforcement learning, and 2) replay-based methods, which selectively replay among previously generated environments. The learning-based methods [14, 16, 17] utilize an adaptive generator that controls the parameters that fully define the environment configuration. The generator receives regret of the agent as reward and is trained via reinforcement learning to produce environments that maximize the regret. While the learning-based methods can directly generate meaningful environments, training the generator with RL is unstable due to the moving manifold [16]. Additionally, we observe that the RL-based generator has limited environment coverage. which limits the generalization capability of the trained agent. In contrast, the replay- basey methods [18, 19, 20] employ a random generator and select environments to revisit among previously generated environments. Since the random generator can produce diverse environments without additional training, they outperform the learning-based methods in zero-shot generalization tasks [20]. However, the replay- methods sample inefficient as they require additional episodes to evaluate the regret on the randomly generated environments.
In this work, propose sample-efficient and robust UED algorithm by leveraging the strong representation power of diffusion models [21]. First, to make UED suitable for using a diffusion model as generator, we introduce soft UED, which augments the regret objective of UED with an entropy regularization term, as done in maximum entropy RL [22]. By incorporating the entropy term, can ensure the diversity of the generated environments. Then, we present adversarial environment design via regret- -uided diffusion models (ADD), which guides a diffusion-basee generator with the regret of the agent produce environments that are conducive to the performance improvement of the agent. Enabling this regret guidance requires the gradient of the regret with respect to the environment parameter. However, since the true value of the regret is intractable and the regret estimation methods used in prior works on UED are not differentiable, a new form of regret estimation method is needed. To this end, we propose a novel method that enables the estimation of the regret in differentiable form by utilizing an environment critic, which predicts a return distribution of the current policy on the given environment. This enables us effectively integrate diffusion models within the UED framework, significantly enhancing the environment generation capability.
environment
Since the regret- regret-guided diffusion does not require an additional training of the environment generator, can preserve the ability cover the high-dimensional environment domain as the random generator of the replay-based method. Moreover, ADD can directly generate meaningful environments via regret-guided sampling the learning-based methods. By doing so, ADD effectively combines the strengths of previous UED methods while addressing some of their limitations. Additionally, unlike other UED methods, ADD allows us control the difficulty levels of the environments generates by guiding the generator with the probability of achieving a specific return. It enables the reuse of the learned generator in various applications, such as generating benchmarks.
We conduct extensive experiments across challenging tasks commonly used in UED research: par- tially observable maze navigation and 2D bipedal locomotion over challenging terrain. Experimental results show that ADD achieves higher zero-shot generalization performance in unseen environments compared the baselines. Furthermore, our analysis on the generated environments demonstrates that ADD produces an instructive curriculum with varying complexity while covering a large en- vironment configuration space. As result, it is shown that the proposed method successfully generates adversarial environments and facilitates the agent to learn a policy with solid generalization capabilities.
2 Related Work
2.1 Unsupervsied Curriculum Reinforcement Learning
While curriculum reinforcement learning [13, 23, 24] has been shown to enhance the generalization performance of the RL agent, Dennis et al. [14] first introduce the concept of the unsupervised environment design (UED). UED encompasses various environment generation mehods, such as POET [12, 25) and GPN[26). In this work, we follow the original concept of UED, which aims to learn a minimax regret policy [15] by generating training environments that maximize the regret of the agent. Based on this concept, the learning-basee methods train an environment generator via reinforcement learning. PAIRED [14] estimates the regret with difference between returns

---

obtained by two distinct agents, and trains RL-based generator by utilizing the regret as reward. Recently, CLUTR [16] and SHED [17] utilize generative models to improve the performance of PAIRED. CLUTR trains the environment generator on the learned latent space, and SHED supplies the environment generator with augmented experiences created by diffusion models. Despite the progress, training the generator via RL is unstable due to the moving manifold [16, 27] and often struggles to generate diverse environments. On the other hand, replay-based methods based on PLR [18] utilize a random environment generator and decide which environments to replay. ACCEL [20] combines the evolutionary approaches [12, 25J and PLR by taking random mutation on replayed environments. While these replay- based methods show scalable performance on a large-scale domain [28] and outperform the learning-bast based methods, they do not have the ability to directly generate meaningful environments. Unlike prior UED methods, augment the regret objective of UED with an entropy regularization term and propose a method that employs a diffusion model as an environment generator to enhance the environment generation capability. Our work is also closely related to data augmentation for training robust policy. Particularly, DRAGEN [29] and ISAGrasp [30] augment existing data in learned latent spaces to train a policy that is robust to unseen scenarios. Our algorithm, on the other hand, focuses on generating curricula of environments without any prior knowledge and dataset.
2.2 Diffusion Models
Diffusion models [21, 31, 32] have achieved remarkable performance in various domains, such as image generation [33], video generation [34], and robotics [35, 36, 37]. Particularly, diffusion models effectively perform conditional generation using guidance to generate samples conditioned on class labels [38, 39] or text inputs [40, 41, 42]. Prior works also guide the diffusion models utilizing an additional network or loss functions, such as adversarial guidance to generate images to attack classifier [43], safety guidance using pre-defined functions to generate safety-critical driving scenarios [44], and guidance using reward functions trained by human preferences to generate censored samples. [45]. We note that our implementation of the regret-guided diffusion model is based on the work of Dhariwal et al. [38] and Yoon et al. [45].
3 Background
3.1 Unsupervised Environment Design
Unsupervised environment design (UED, [14]) aims to provide an adaptive curriculum to learn policy that successfully generalizes to various environments. The environments are represented using a Markov decision process (MDP), defined as (A, P, R. Pu. 7), where A is set of actions, S is a set of states, P S x A x S + [0, 1] is transition model, R : S x A + R is reward function, Pu : S - [0, 1] is an initial state distribution and 7 is a discount factor. UED employs an environment generator that designs environments by controlling free environment parameters of underspecified environments, which is represented using an underspecified Markov decision process (UMDP). UMDP is defined as .M- = (A, S, θ P.M R.M ..M..... ()), where θ is set of free environment parameters. Assigning a value to the free environment parameter E O results in a specific MDP Ma with the environment configuration (P" (0)( RD (0), o = Pb (0)). For example, when learning a mobile robot to navigate towards the goal point while avoiding obstacles, 0 could represent the positions of obstacles, the position of the goal, and the start position of the robot.
UED algorithms alternate between designing a set of environments and training the agent on the generated environments. The environment generator first produces an environment parameter 0 that maximizes the agent's regret. The regret of the policy ι on environment M is defined as,
REGRET T 0) == (T 0) + max V(Z''O0 ''Ell
where II is set of policies and ((T, 0) = Egg pp
>N o is an expected return where
In is reward obtained by π at timestep n on M Then, the agent is trained on the generated environment to maximize the expected return, resulting in minimizing the regret. This framework can be formulated with the following two-player minimax game:
min max REGRET 0).
Hlll BEB
3

---

It is ensured that the agent learns the minimax regret policy x" E argmin max REGRET(T,, 8) by
*ell AE
assuming the two-player game (2) reaches the Nash equilibrium [14, 19]. However, learning the minimax regret policy is challenging. Since the objective (2) does not guarantee the diversity of generated environments, the agent may not be trained on sufficiently various environments.
3.2 Diffusion Probabilistic Models
A diffusion probabilistic model [21] is a generative model that generates samples from noise via iterative denoising steps. Diffusion models start with perturbing data by progressively adding Gaussian noise, called the forward process. The forward process can be modeled with a value- preserving stochastic differential equation (VP SDE, [31]):
3e X,dt + B,dW,,
dX,
where t E [0, T] is continuous diffusion time variable, Be > is variance schedule, and W a standard Brownian motion. Since the forward process (3) has tractable conditional marginal distributions pr (Xe = =N(varxX (1 (1-a)) where = e J 8 dt pr(Xr) will be corrupted into I), when T oc.
Generating samples following the data distribution (.)ta()) requires a reverse process, a reverse-time SDE that has the same marginal distributions as the forward process (3). By Anderson's 's theorem [46], the reverse process can be formulated with a reverse-time SDE defined as,
1 Vx,
dx, -3e Xr
Vx, (x.) dt + B.dW,.
Hence, learning a diffusion model means learning a score network s (X,,)) that approximates a score function Vx. log pr (X,)) The score network is trained via score matching [47], then plugged into the reverse process (4):
[ 2 (x(,,, dt VidW, dx, -B
(5)
Indeed, we can generate samples by solving the approximated reverse process (5) with an initial condition Xy ~ (O,
To generate samples with label Y using the diffusion model, the score function of the conditional distribution Pr( ((2lY)) should be estimated. Since Pe (x(X, (X,lV) x Pe(X,, Y) = Pr(X, x)),,(Y (Y|X,) due Bayes' rule, conditional samples can be generated by solving the reverse process with classifier guidance [38]:
8x(X,, 1) = (X, r. wvx log (Y(p))
dx, -3e ト X, S (X,, () dt + v B,dW.
(6)
where (Y),), is a time-dependent classifier network and w > 0 is a guidance weight to scale classifier gradients.
4 Proposed Method
In this section, we describe our approach to employ a diffusion model as an environment generator to enhance the environment generation capability. We first introduce soft UED, which mutates UED to be more suitable for using a diffusion model as generator by augmenting the original objective with the entropy regularization term. Then, we propose a novel soft UED algorithm, adversarial environment design via regret-guided diffusion models (ADD). ADD consists of two key components: environment generation by using the regret as a guidance, and 2) method to estimate the regret in differentiable form. We present these key components in detail and conclude the section with an explanation of the overall system and its advantages compared to prior UED methods.
diffusion-bacee

---

4.1 Soft Unsupervised Environment Design
In this section, we introduce soft UED, designed guarantee the diversity of environments by adding an entropy regularization term to the original UED objective (2). Soft UED is defined as the following minimax game between the agent and the environment generator:
min max E [REGRET((T, 0) + w hel AEDa A~A
((),,
where A is a distribution over θ, DA is a set of distributions over O, H(A) == Σ A(0) log (0)
BEE
an entropy of A, and w a regularization coefficient. Based on the fact that H is concave, we can show that the strong duality holds:
Proposition 4.1. Let LIT, A) = Es~A [REGRET(T, 0)] (A) and assume that S, A, and O are finite. Then, min max (IT, A) = max min (TT, A). *ell AEDA
AEDA TEll
The proof is detailed Appendix A.I. Proposition 4.1 implies that there exists a valid optimal point (ū, A), and it is stationary for alternative optimization of π and A. Hence, the agent will learn soft minimax regret policy ग argmin max L(T, A) if it reaches the optimal point. One
=E AEDA
of the most significant difference from the original UED is the role of the environment generator. Instead of finding specific environment parameter that maximizes the regret, soft UED updates the environment generator to sample environment parameters from a distribution that maximizes the objective function of soft UED.
We note that the soft UED framework encompasses prior UED algorithms. In the learning- methods, the generator is trained with RL using an entropy bonus, which is known to enhance performance [48] and plays a similar role to (A). The replay-based methods also consider the diversity of environments by sampling enviromment parameters from probability distribution proportional to the regret, instead of selecting a parameter that maximizes the regret. Therefore, soft UED can be considered as a general framework that incorporates practical methods.
4.2 Regret-Guided Diffusion Models
Soft UED converts the problem of generating regret-maximizing environments into a problem of sampling the environment parameter from a desired distribution A := argmax (TT,A) is
AEDA
well-known fact that A has a closed-form solution as follows:
n(() exp([RREGEET(TT())
A Th (0)
(8)
C
where C is normalizing constant, and u--) denotes uniform distribution over θ. Inspired by the classifier guidance (6), we solve this sampling problem by guiding a pre-trained diffusion model with the regret. To this end, we decompose the score function of A follows:
Ve, log A7 (6.) Va, log (0,) + ~ V, REGRET, 0t),
where is a diffusion time variable, 0g is an environment parameter perturbed by the forward process (3), (-) denotes distribution of 0, when 0, un-). AT (-) denotes distribution of 9, when Oo ~ (-).., and REGRET,(T, (TT, 02) is time-dependent regret on the noised environment 0r, which equal REGRET(Z,, 0o). We approximate the first term V,, log ur (0.).) with score network 8, (O,, t) ~ Va, log u. (0,) that is learned by training a diffusion-based environment generator on the randomly generated environment dataset before the agent begins learn. Then, we can formulate regret- gegret-guided reverse process with reverse- time SDE as follows:
(0,, t) = s0(0,, t) + V., REGRET, 0z),
d0, -3 0 + s0(0,,)) 1) di VBdW,
(10)
Hence, a gradient of the regret tractable, we can sample an environment parameter Ag from the desired distribution A by solving the regret-guided reverse process (10) with an initial condition Or ~ (o, /). However, the regret (1) intractable since we cannot access the environment-specffic optimal policy. Prior works on UED propose various methods to estimate the regret using episodic returns or temporal difference errors, but none of them are differentiable w.r.t. 0, since agents cannot access the environment parameter and the reward function.
5

---

Episodic results
Environments
Regret guidance
ceitoommentcritee
Diffusion- based environment generator
Figure Overview of ADD. After the agent trained on environments produced by the environment generator, the environment critic is updated using the episodic results. Then, the environment critic guides the diffusion- based environment generator with the regret to produce adversarial environments. By repeating this process, the agent learns policy that is robust to environmental changes.
Differentiable Regret Estimation
In order to estimate the regret in a differentiable form, we present novel method based on flexible regret [14], which is known to enhance the performance of the learning- UEDs [16]. The main idea is to estimate the regret with difference between the maximum and average episodic returns that can be achieved by the agent. To make it differentiable w.r.t. Or, we utilize an environment critic that predicts the return of the agent in the given environment parameter, as done in DSAGE [49] and LPG [50]. The environment critic Ty learns to predict a distribution of returns, analogous to distributional RL [5]], better capture the stochasticity of the environment and policy. Based support defined as { Umin + * Umin MMM-- which is set of centers of M bins that evenly divide the return domain [Uminn Umaz]. we obtain an estimated categorical return distribution from an environment critic output 1(0,, (b)) E TRM as follows:
ex,(l((,,,t,))
(0,, w.p.
(11)
j= ex,(1,(0,,,;)))
To align 2, with a true return distribution, we train the environment critic by gradient descent on the cross entropy loss between (0,, t) and a target distribution, which is constructed by projecting episodic returns that the agent achieves on the environment M onto the support { -M-1
After the environment critic is updated, we estimate the regret (1) with difference between the maximum return that the current agent can achieve and average of the predicted return distribution. However, the process of finding maximum achievable return from the distribution is not differen- tiable. To address this issue, we further approximate the maximum with conditional value risk (CVaR), based on the fact that CVaR, (2) converges to the maximum as a risk measure a goes zero. As result, estimate the regret of the agent as follows:
R t) = CVaR, f)) ((,,
(12)
4.4 Adversarial Environment Design via Regret-Guided Diffusion Models
overview of ADD is provided in Figure First, a diffusion-basee environment generator, which is pre-trained on the randomly generated environment dataset, produces set of environments for agent. After the agent interacts with the generated environments and trained via reinforcement learning, the episodic results are utilized to update the environment critic. Then, the environment critic estimates the regret of the agent differentiable form (12) and guides the reverse process of the cesttusion-basee environment generator (10), resulting in environment parameters following the

---

distribution that maximizes the soft UED objective (7). By repeating this process, the agent learns the soft minimax regret policy, which is robust to the variations of environments. pseudocode of the algorithm is shown in Appendix A.4.
Since ADI does not require additional training of the pre-trained diffusion model, the ability to cover the high-dimensional environment domain can be preserved. Furthermore, ADD enables the generator directly produce meaningful environments via regret-guided reverse process. Therefore, ADD can be seen as having both the advantage of the replay-based methods and learning-based methods while resolving some of their limitations. Additionally, we can control the difficulty level of the generated environments after the training of the RL agent is over. In detail, we can generate environments of difficulty level & E {1, ........ M) } by replacing the regret in the regret-guided reverse process (10) with log probability of achieving a specific return ZM & as follows:
(0,,t =ss(0, t) + w log Pr( (0,, zM-k),
[ 0, (0,, dt V,,WW.
(13)
d0, - 2
It enables the reuse of the learned generator and environment critic in various applications, such as generating benchmarks with varying difficulties.
5
Experiments
Tasks We conduct extensive experiments with two challenging tasks. First, we evaluate the proposed method on a partially observable navigation task with a discrete action space and sparse rewards. Then, we assess the performance of our algorithm on 2D bipedal locomotion task, which has continuous action space while offering dense rewards.
Baselines We compare the proposed method against several UED baselines. For the learning- basseng-based method, we use PAIRED [14], which trains the environment generator via reinforcement learning. For the replay- method, use PLR- [19], which utilizes the random generator and updates the agent only with episodes from the replayed environments. To benchmark performance, we use ACCEL [20], current state-of-the-ar UED algorithm that applies random mutations to replayed environments, Among the two implementation methods of the ACCEL, we use the one that samples environment parameters from the full parameter range, rather than the one that restricts the sampling to range that ensures simple environments are generated, as the latter could be seen as incorporating prior knowledge. Domain randomization (DR) also included in baselines so that we can demonstrate the effectiveness of UED. Lastly, we use ADD w/o guidance to show whether the regret guidance induces the diffusion model generate adversarial environments and enhances the performance of the agent.
Outline We first train cerfusion-baeed environment generator on the randomly generated envi- ronment dataset. Then, we use proximal policy optimization (PPO, [52]) to train the agent on the environments generated by UED methods. To evaluate the generalization capability of the trained agent, we measure the zero-shot transfer performance in challenging, human-designed environments. Additionally, to understand where the differences in performance originate, we conduct quantitative and qualitative analyses on the curriculum of the generated environments by tracking complexity metrics and drawing t-SNE plots. For space consideration, we elaborate on detailed experimental settings including environment parameterization methods in Appendix B.
5.1 Partially Observable Navigation Task
We first evaluate the proposed method on a maze navigation task [14], which is based on the Minigrid [53]. In this task, an agent is trained to take discrete action using an observation from its surroundings to receive a sparse reward upon reaching a goal. For prior UED methods, we set the maximum number of blocks in grid environment to 60, aligning with Parker-Holder et al. [20]. For the proposed method, we train the diffusion-based environment generator on 10M random environments whose number of blocks uniformly varies from zero to 60. Then, we train the L.STM-based policy for 250M environmental steps and evaluate the zero-shot performance on 12 challenging test environments from prior works [14, 19].
7

---

(a) Zero-shot performance
(b) Training curves
(c) Complexity metrics
(d) t-SNE plots
Mean solved rate
Labyrinth
Number of blocks
ADD
1.00
0.3
0.75
40
20
30
20
10
Saked
0.50
0.4
-10
10
0.0
0.00
0.0
0.5
2.5
0.0
0.5
1.0
-40
000
1ed
plot solved
Maze
Shortest path length
PAIRED
1.0
1.00
40-
30-
20-
sehedd rete
0.6
10
Salved
2.50
-10
-20
3.00
Steps
1d0
Steps
DR
PAIRED
PLR
ACCEL
ADD w/o guidance
ADD (ours)
Figure 2: Partially observable navigation results. (a): Zero-shot performance on the 12 test environments. We report results across five random seeds, each evaluated over 100 independent episodes per environment. (b): Training curves on two challenging test environments. (c): Complexity metrics of the generated environments during training. (d): t-SNE embedding of the generated environments during training.
Performance. In Figure 2(a), we report the mean solved rate and box plot to compare the zero-shot performance of the learned policy the test environments. The result demonstrates that ADD outperforms the baseline methods while achieving 85% of the mean solved rate, which is 18% higher compared the ACCEL. Furthermore, ADD achieves the highest Q1, Q2, and Q3 values while its interquartile range, defined as Q3 Q1, is 51% smaller than ACCEL. Therefore, we can infer that the proposed method consistently outperforms the baselines in the challenging test environments. In Figure 2(b), we report training curves on two test environments consisting of Maze and Labyrinth. The results demonstrate that ADD shows the monotonic performance improvement and achieves a higher solved rate compared to other baselines. While ACCEL shows 13% higher mean solved rate than DR, PAIRED and PLR- do not show notable performance improvement by applying UED techniques. Particularly, PAIRED shows 8% lower mean solved rate compared to DR, demonstrating the challenge of the learning-batt rate similar to DR, demonstrating that the regret guidance is critical for training the robust policy. Please refer to Appendix C.I for detailed per-environment results.
UED methods. ADD w/o guidance shows 63% of mean solved
We note that the performance is measured after the fixed number of environmental steps, in line with some UED papers [19, 54] and traditional deep reinforcement learning research. In contrast, Parker-Holde et al. [20] recorded performance after the fixed number of policy updates. Since the replay-base based methods require additional episodes without policy updates, using the number of environmental steps may be seen as unfair to PLR- and ACCEL. To address this issue, we also measured the performance of our method trained with only half the environmental steps, aligning the number of policy updates with PLR- and ACCEL. When using half the environmental steps, ADD achieves a 72% mean solved rate, which ties with ACCEL. This demonstrates that the proposed method remains competitive, even when using the number of policy updates metric.
Generated curriculum. In Figure 2(c), we report complexity metrics consisting of the number of blocks and shortest path length. The results demonstrate that complexity metrics of ADD w/o guidance and DR are almost consistent over time since they do not consider the policy. While PLR eventually generates environments with larger number of blocks compared to DR, ADD and PAIRED generate curriculum with significantly increasing complexity. Specifically, ADD eventually generates environments with the second largest number of blocks and the longest shortest path. ACCEL also shows significantly increasing complexities despite being based on - This because only up to 60 blocks exist the 13X13 grid, random mutation increases the expectation
8

---

Bosic Env
Hardcore Env
Stairs Env
PitSep Env
Stump Enw
Roughness Eny
202
200
THE returr
200
300
200
XO0
-100
-100
-103
-100
(a) Zero-shot performance
Stump Height
Stair Helght
Gap
Stair Steps
Ground Roughness
Train Return
100
-50
2
1000
Steps
Staps
Staps
Staps
Staps
Stess
14
(b) Generated curriculum
DR
PAIRED
PLR
ACCEL
ADD w/o guidance
(ours)
Figure 3: 2D bipedal locomotion task results. (a): Zero-shot performance on the six test environ- ments. We report results across five random seeds, each evaluated over 100 independent episodes per environment. (b): Complexity metrics of the generated environments and episodic return achieved during training.
of the number of blocks. Therefore, it can be seen that ADD and PAIRED efficiently generate complex environments by adapting the current policy while PLR struggles to find environments with high complexity. To compare the distribution of generated environments, we report t-SNE [55] plots in Figure 2(d). While the environments generated by PAIRED eventually cover only a specific region, the environments generated by ADD cover a significantly larger region over time. The results demonstrate that ADD successfully generates adversarial environments while preserving diversity.
5.2 2D Bipedal Locomotion Task
We evaluate the proposed method on the 2D bipedal locomotion task, which is based on the Bipedal- Walker task in OpenAl Gym [56] and adopted by Parker Holder et al. [20]. In this task, an agent is trained to control its four motors using observation from its Lidar sensor and joints to walk over challenging terrain. UED methods, including the proposed algorithm, need provide environment parameters consisting of stump height, stair height, pit gap, stair steps, and ground roughness. We note that each parameter increases the difficulty the environment as value increases. We train the RL agent for two billion environmental steps and evaluate the zero-shot performance on six test environments.
Performance. Figure 3a)) shows the average return on each test environment. The proposed algorithm achieves the highest return across all environments, with an average of 149.6. Even with half the environmental steps, it achieves a score of 127.4, still surpassing PLR ACCEL shows lower performance than PLR which can be attributed the lower sample efficiency induced by the additional interaction between the environment and the agent to assess the modified environments. On the other hand, PAIRED achieves the lowest return in all test environments except the easiest Basic Environment. This shows that the learning-based methods struggle to train a robust policy in practice. We note that a recent work [48] stabilizes the training PAIRED this task by integrating the evolutionary concept of ACCEL. While applying the evolutionary approach to ADD is possible, we leave for future work. Lastly, ADD w/o guidance demonstrates superior generalization performance compared to DR. Although these two methods are theoretically identical, this difference is presumably caused by the limited size of the dataset used for training the diffusion-bar based environment generator.
Generated curriculum Figure 3(b) presents the complexity metrics of the generated training environ- ments and the episodic returns achieved by the RL agent. Unlike the partially observable navigation task, the complexity measure of the environments generated by ADD gradually decreases. This result arises since the randomly generated environments are excessively challenging for the current agent. As evidence, examining the returns achieved by the agent in the generated environments reveals that all methods, except for ADD, consistently yield returns of or below. From these results, we can infer that the proposed algorithm generates environments that are not merely more difficult but are

---

Easy
Difficult
Figure 4: Controllable generation results for the partially observable navigation task. The figure shows the results of guiding the generator generate progressively more difficult environments. We note that each row is generated from the same initial noise 0r.
conducive to the agent's learning process. For detailed experimental results including qualitative analysis on the generated environments, please refer to Appendix C.2.
5.3 Controllable Generation
To demonstrate the ability control the difficulty level of the generated environments, provide the example environment generation results in Figure 4. We control the difficulty level k by guiding the diffusion-based environment generator with log probability of achieving a specific return 2M k. as described in (13). We vary & from zero to M 1 so that the difficulty level of generated environments increases. Environments generated with k 0. which are shown in the leftmost images, include fewer blocks and close proximity between the agent's 's starting position and the goal. As k increases, environments are generated with greater number of blocks and larger distance between the starting position and the goal, resulting in the elimination of all possible paths when k M 1. The results demonstrate that we can effectively control the difficulty level the environment using the inffusion-based environment generator and learned environment critic, without domain knowledge. We also present the results controlling difficulty levels for the 2D bipedal locomotion task in Appendix C.2.
6 Limitation
While the proposed method is suitable for training a robust policy, there exist several limitations. First, despite the existence of the optimal point proven in Proposition 4.1, convergence to such optimal point not guaranteed. Furthermone, the difference between the true value of the regret and its estimate not tightly bounded. Lastly, updating the environment critic using episodic results cannot exactly capture the current agent's capability since the policy is updated after the episode. Hence, exploring methods to estimate the regret with rigorous theoretical background would be an interesting topic for future work.
7 Conclusion
In this work, we present novel UED algorithm that exploits the representation power of diffusion models. We first introduce soft UED, which augments the original UED objective with an entropy regularization term to make it suitable for using diffusion- based environment generator. Then, propose ADD, which guides the pre-trained diffusion model with a novel regret estimator to produce environments that are conducive to train robust policy. Our experimental results demonstrate that ADD is capable of training a policy that successfully generalizes to challenging environments. Moreover, has been verified that ADD generates an instructive curriculum with varying complexity while covering large environment configuration spaces.
10

---

Acknowledgments and Disclosure of Funding
This work was partly supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-01190 [SW Star Lab] Robot Learning: Efficient, Safe, and Socially-Acceppable Machine Learning, 40%), Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by Paneral-Purpose Deep Reinforcement Learning Using Metaverse for Real World Applications, 40%), and Institute of Information & com- munications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) Artificial Intelligence Graduate School Program [Scoul National Universityl,
the Ministry of Science and ICT ANRF-202RIA2222000
20%)..
References
[1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, (11877540)55295533 Feb. 2015.
[2] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, al. Mas- tering the game of go with deep neural networks and tree search. Nature, 529(7587):484 489, Jan. 2016.
[3] Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Proceedings of the Conference on Robot Learning. PMLR, Dec. 2022.
[4] Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620(7976):982-999) 987, Aug. 2023.
[5] Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In Proceedings of the International Conference on Machine Learning. PMLR, Jun. 2019.
[6] Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint urXiv:1806.07937 Jun. 2018
[7] Nick Jakobi. Evolutionary robotics and the radical envelope-of-nois hypothesis. Adaptive Behavior, 6(2):325-36) 368,, Sep. 1997.
[8] Fereshteh Sadeghi and Sergey Levine. CAD2RL: Real single-image flight without single real image. In Proceedings of Robotics: Sicence and Systems, Jul. 2017.
[9] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to real world. In Proceedings of the International Conference Intelligent Robots and Systems. IEEE, Sep. 2017.
[10] Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre Oudeyer. Auto- matic curriculum learning for deep rl: short survey. In Proceedings of the International Joint Conference Artificial Intelligence, Mar. 2020.
[11] Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, ((1):7332-7431, Jan. 2020.
[12] Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley, Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv:1901.01753 Feb. 2019.

---

[13] Seungjae Lee. Daesol Cho, Jonghae Park, and H Jin Kim. CQM: Curriculum reinforcement learning with a quantized world model. Advances in Neural Information Processing Systems, Dec. 2024.
[14] Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. In Advances Neural Information Processing Systems, Dec. 2020.
[15] Leonard J Savage. The theory of statistical decision. Journal of the American Statistical Association, 46(253):55 67, Mar. 1951.
[16] Abdus Salam Azad, Izzeddin Gur, Jasper Emhoff, Nathaniel Alexis, Aleksandra Faust, Pieter Abbeel, and Ion Stoica. CLUTR: Curriculum learning via unsupervised task representation learning. In Proceedings of the International Conference on Machine Learning. PMLR, Jul. 2023.
[17] Dexun Li and Pradeep Varakantham. Enhancing the hierarchical environment design via generative trajectory modeling. arXiv preprint arXiv:2310.00301 Feb. 2024.
[18] Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In Proceedings of the International Conference on Machine Learning. PMLR, Jul. 2021.
[19] Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay-guided adversarial environment design. In Advances in Neural Information Processing Systems, Dec. 2021.
[20] Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. In Proceedings of the International Conference on Machine Learning. PMLR, Jul. 2022.
[21] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, Dec. 2020.
[22] Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, Dec. 2010.
[23] Peide Huang, Mengdi Xu, Jiacheng Zhu, Laixi Shi, Fei Fang, and Ding Zhao. Curriculum reinforcement learning using optimal transport via gradual domain adaptation. In Advances in Neural Information Processing Systems, Dec. 2022.
[24] Daesol Cho, Seungjae Lee, and H Jin Kim. Outcome-directee reinforcement learning by uncer- tainty& & temporal distance-aware curriculum goal generation. In Proceedings of International Conference on Learning Representations, Feb. 2023.
[25] Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth Stanley, Enhanced POET: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. Proceedings of the International Conference on Machine Learning. PMLR, Jul. 2020.
[26] Philip Bontrager and Julian Togelius. Learning to generate levels from nothing. In Proceedings of the IEEE Conference on Games, Aug. 2021.
[27] Eric Mazumdar, Lillian J Ratifff and S Shankar Sastry. On gradient- based learning in continuous games. SIAM Journal on Mathematics of Data Science, 1331110, May 2020.
[28] Jakob Bauer, Kate Baumli, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Michael Chang. Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, et al. Human- timescale adaptation in an open-ended task space. In Proceedings of the International Confer- ence on Machine Learning. PMLR, Jul. 2023.
[29] Allen Ren and Anirudha Majumdar. Distributionally robust policy learning via adversarial environment generation. IEEE Robotics and Automation Letters, 7(2):1379-1333 Apr. 2022.
12

---

[30] Zoey Chen. Karl Van Wyk, Yu-Wei Chao, Wei Yang, Arsalan Mousavian, Abhishek Gupta, and Dieter Fox. Learning robust real-world dexterous grasping policies via implicit shape augmentation. In Proceedings of the Conference on Robot Learning, Dec. 2022.
[31] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Proceedings of the International Conference on Learning Representations, May 2021.
[32] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Proceedings of the International Conference on Learning Representations, May 2020.
[33] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in neural information processing systems, Dec. 2022.
[34] Jonathan Ho, Tim Salimans. Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models, In Advances in Neural Information Processing Systems, Dec. 2022.
[35] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In Proceedings of the International Conference on Machine Learning, Jul. 2022.
[36] Mineui Hong, Minjae Kang, and Songhwai Oh. Diffused task-agnostic milestone planner. In Advances in Neural Information Processing Systems, Dec. 2023.
[37] Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Policy-guided guided diffusion. arXiv preprint arXiv:2404.06356, Apr. 2024.
[38] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems, Dec. 2021.
[39] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2007.12598, Jul. 2022.
[40] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, Jun. 2022.
[41] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.061235 Apr. 2022.
[42] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, Jun. 2023.
[43] Xinquan Chen, Xitong Gao, Juanjuan Zhao, Kejiang Ye, and Cheng-Zhong Xu. Advdiffuser: Natural adversarial example synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Oct. 2023.
[44] Chejian Xu, Ding Zhao, Alberto Sungiovanni-Vincetellii safety-critical scenario generation for autonomous vehicles. In Proceedings of the International Conference on Machine Learning Workshop on New Frontiers in Adversarial Machine Learning,
Vincentelli, and Bo Li. Diffscene: Diffusion-
Jul. 2023.
[45] TaeHo Yoon, Kibeom Myoung, Keon Lee, Jaewoong Cho, Albert No, and Ernest Ryu. Censored sampling of diffusion models using 3 minutes of human feedback. In Advances in Neural Information Processing Systems, Dec. 2023.
[46] Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-33 336-26,,,,,,, May 1982.
13

---

[47] Yang Song and Stefano Ermon. Improved techniques for training score- score-based generative models. In Advances in Neural Information Processing Systems, Dec. 2020.
[48] Ishita Mediratta, Minqi Jiang, Jack Parker-Holder, Michael Dennis, Eugene Vinitsky, and Tim Rocktäschel. Stabilizing unsupervised environment design with a learned adversary. In Proceedings of the Conference on Lifelong Learning Agents. PMLR, Aug. 2023.
[49] Varun Bhatt, Bryon Tjanaka, Matthew Fontaine, and Stefanos Nikolaidis. Deep surrogate assisted generation of environments. In Advances in Neural Information Processing Systems, Dec. 2022.
[50] Matthew T Jackson, Minqi Jiang, Jack Parker-Holder Risto Vuorio, Chris Lu, Greg Farquhar, Shimon Whiteson, and Jakob Foerster. Discovering general reinforcement learning algorithms with adversarial environment design. Dec. 2023.
[51] Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforce- ment learning. Proceedings of the International Conference on Machine Learning, Aug. 2017.
[52] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 Aug. 2017.
[53] Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo Salem Lahlou, Suman Pal, Pablo Samuel Castro, and J Terry. Minigrid &amp; miniworld: Modular &amp; customizable reinforcement learning environments for goal-oriented tasks. In Advances in Neural Information Processing Systems, Dec. 2023.
Lucas Willems,
[54] Samuel Garein, James Doran, Shangmin Guo, Christopher G Lucas, and Stefano V Albrecht. Dred: Zero-shot transfer in reinforcement learning via data-regularised environment design. In Proceedings of the International Conference on Machine Learning. PMIR,, Jul. 2024.
[55] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86) 2605, Aug. 2008
[56] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv:1606.0/540 Jun. 2016.
[57] Umar Syed, Michael Bowling, and Robert E Schapire. programming. In Proceedings of the International Conference on Machine Learning, Jul. 2008.
Apprenticeship learning using linear
[58] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, Dec. 2016.
[59] Maurice Sion. On general minimax theorems. Pacific Journal of Mathematics, 8(1):171-176. Mar. 1958.
[60] Olaf Ronneberger, Philipp Fischer, and Thomas Brox U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the Medical Image Computing and Computer- Assisted Intervention, Nov. 2015.
14

---

A Algorithm Details
A.1 Proof of Proposition 4.1
In this section, show that the minimax problem of soft UED (7) has minimax duality gap. We assume that S, A, and è are finite to avoid the technical issues regarding compactness of a set of distributions. Following Section 3.1, denote a reward function, transition probability, and initial state distribution of an environment . with Rº, pe, and pl. respectively.
We first define an occupancy measure. for a policy ι (a]s) Em y" Pr(s,, = s| 0). Then, there is a one-to-one correspondence between II and a set of valid occupancy measures D == (p= Ea ((8, a) = =8(() + (s]s',, (olls,d')p(ee d')p('', [57, 58).
II and an environment MM, as p a)
where 0 is an ith element of O it is obvious that there is one-to-one correspondence between II and a set of valid global occupancy measures D C 'DD' ... Therefore, can replace in the objective function of soft UED (7) with Pr. as in the following lemma:
If we define global occupancy measure Par [
|@|
Lemma A.1. if L(pm, A) = (V)(()) - a,
a)) + 1 H(A), where V*(0) =
max Vm 0). then [p=, L(T,A). xell
Proof. Based on the definition of the regret (1), we have
1 H(A)
L( A)
E [R] 0)]
O~A
w
1 H(A)
(T, +
(OV ())()))(()) H(A)
(14)
Σ(V Σ (s(,,))"((ssa)))) - H(A)
= A()
Now, we can rewrite the objective function of soft UED with a global occupancy measure as follows:
min max A), DE AEDA
(15)
Then, we can prove Proposition 4.1 by showing (15) has zero duality gap. However, we cannot apply minimax theorem [59] directly since D is not convex set. To resolve this issue, we first augment the problem as follows:
min max L(p, A), PED AEDA
(16)
where : (E WEPEIK E K w/ 1, vk E (1, K} wy 2 0. D} is a convex -
- hull D. We show that the augmented problem (16) has zero minimax duality gap, and end the proof by showing the optimal values of the augmented problem can also be reached by the original problem (15).
Lemma A.2. min max L(p, A) = max min L(p, A) PED AEDA
AEDA PED
Proof. Since L(p, A) is a línear combination of / it is convex for all Furthermore, L.p, A) is concave for since the entropy H is concave. Therefore, based on the fact that D and DA are both convex and compact, the augmented problem has zero duality gap due to minimax theorem [59]. D
15

---

Lemma A.3. For every A E DA and corresponding p" (A) argmin L(p, A), there exists p' E D such that (A), = A()
PED
Proof. For every A DA and corresponding p" p*(A) argmin L(p, A), there exist K N, Wi:K 2
PEd
and PI:K E D such that Σ wk and p*(A) =E WkPk. Then, following inequality holds:
min(T[pp, A) 2 (A)
Σ we Tep A) min A))
(17)
where first inequality holds due to the definition of (A), equality holds since L is linear for p. and second inequality holds since p² (A) ) is convex combination of Pi:k. Then, p' E argmin L(p,))
pElpxtem
an element of D and satisfies L(p" (A), A) = A))
Lemma A.3 ensures the minimum value of L achieved over D is also achievable over D, implying the optimal value is the same for both the original and augmented problems. It confirms that strong duality holds for the original problem (15) as well.
Proposition A.4. min max A) max min L(p, A) ΛEDA PED
PED AEDA
Hence, using Lemma A.I and Proposition A.4, we can prove Proposition 4.1:
min max L(T A) min max A), pED AEDa
min L),, max min L(m, A)
(18)
Kell AEDA
AEDa PD
AEDa well
A.2 Diffusion Models
In this section, present implementation details diffusion models. To solve the forward and reverse processes (3, 4), we follow the implementation of DDPM [21], which can be viewed as a discretization of VP SDE (3). As result, the forward process (3) is implemented follows:
0. V 1111-30-111 v BiZ4,
(19)
where Z, ~ (N, In the appendix, we make a slight abuse notation by considering t as a discrete
variable to provide detailed implementation specifics. To solve the reverse process (4), utilize
error network (0,,, t) vI (68).(6.tt) instead of using Sø, where a -I 1- Br, and 3, follows the linear noise schedule. The error network trained to minimize the loss (oo. () = Be car00 vT 112. To futher accelerate the sampling, we apply DDIM
sampling [32] as follows:
V1 ar VI
0--1
0r
) (Or,
(20)
VI Be
B
1 Be
Since DDIM sampling (20) deterministic, can sample the environment parameter 0 with 7" denoising steps, which is less than the original diffusion timestep T'.
A.3 Environment Critic Update
In this section, present details updating the environment critic. After the interaction between the RL agent with policy π and an environment Mg, we construct a target return distribution 2 (urget (0) using episodic returns. Specifically, episodic returns that the RL agent achieves are
(u k= we give equal probabilities to each return and project them into the support {=))M² result, the target return distribution is constructed follows:
M-1 As
K
targot (0) = w.p.
Ck
Zz
(21)
K
k=
(Mmag Monen) is a width of each bin. Then, train the environment critic to produce the return distribution close to the target distribution
where 1:1g bounds argument in the range [0, 1], and
16

---

T' (21). To that end, we first sample t from ,, and obtain 6, by solving the forward process (3) with initial condition &p = 0, then construct the estimated return distribution 2, (0,, t) using an output of the enviromment critic Ty. (f,, t). Then, th is updated via gradient descent to minimize the cross entropy loss between E, () and - (0). To prevent overfitting, we store episodic results in buffer, then sample the environment parameters and their corresponding target return distribution
from the buffer for training the environment critic.
A.4 Pseudocode of ADD
Algorithm Adversarial Environment Design via Rlegret-Guider Diffusion Models Policy network æy. diffusion model $g, and environment critic network Tp- Initialize network parameters É. Φ, s.
Input:
Train the diffusion model 8 on a dataset of randomly generated environments. for each epochs do
Sample a set of environment parameters {0')
B via regret-guided reverse process (10), whose
regret is estimated using the environment critic (12).
Run episode on set of environments (M ) and update the policy Th via RL.
Update the environment critic Ty using episodic returns (Appendix A.3). end for
B Experiment Details and Hyperparameters
In this section, we provide comprehensive explanation of the experiments discussed in Section 5. We begin by detailing the two tasks: partially observable navigation and 2D bipedal locomotion. Then, conclude the section by reporting the hyperparameters employed in the experiments.
B.1 Partially Observable Navigation Task
Environment details. In the partially observable navigation task, which is based on the Minigrid [53] and adopted for UED in prior works [14], the agent trained to find and reach goal in the grid maze environment. Each maze environment is a 15 x 15 grid whose cells on the edge are all walls, and the cells inside can contain walls, agents, or goals. When the agent reaches the goal, it receives a reward of 1 - N/N where N is length of the episode and Nmaz is a maximum length of each episode, it does not reach the goal, it receives a reward O. The agent uses a x grid around itself and its direction as an observation and chooses one of the actions: turn left, turn right, or go forward.
Environment generation. Aligning with Parker- et al. [20], we limit the number of walls that can exist in the 13 x 13 grid, excluding the walls on the edges, to 60. The RL generator of PAIRED selects locations to place walls over 60 steps, ensuring no changes if a wall already exists. After placing all the walls, it selects the starting position of the agent and the goal location. If wall exists at those locations, it removes the wall and places the agent or the goal. On the other hand, the random generator used by PLR and ACCEL uniformly samples the number of walls between and 60 in advance and then choose the position to place the walls, agent and goal location randomly. This random generation is also used to create a dataset for training the diffusion-basse generator of the proposed algorithm. Specifically, each environment parameter data is represented with 13 x 13 x 3 image. The first channel represents the location of the walls, with a value of one if a wall is present in the cell, and zero if it is not. The second channel indicates the starting position of the agent, with a value of one for the starting cell, 0.5 for the cell after moving forward once, and zero for all other cells. Finally, the third channel has value of one for the cell corresponding to the goal, and zero otherwise. After training the diffusion based environment generator on the randomly generated dataset, we can produce the environment parameter by employing the trained generator to solve reverse process (5), as shown in Figure 5. Training time. All methods are trained utilizing RTX 3090Ti. To train DR and ADD w/o guidance, they require almost 48 hours to run 250 million environment steps. ADD requires almost 56 hours, and PLR- and ACCEL requires almost 100 hours for each random seed.
environment
17

---

OT
eo
Figure 5: Maze environment generation using diffusion models. We represent the maze environ- ment with a parameter 9 E R 13x 13x3 with each channel indicating the location of walls, the agent, and the goal. After training the diffusion-based environment generator dataset of randomly generated environment parameters, we can sample maze environments by solving the reverse process (5).
B.2 2D Bipedal Locomotion Task
Environment details. For the 2D bipedal locomotion task, we conduct the experiment using a modified version of the Bipedal environment from OpenAl Gym [56], as done in Parker- Holder et al. [20]. In this task, the agent trained to walk over challenging terrains by controlling four joints, and action is decided using 24-dimensional observation, which is consisting of Lidar measurements, linear and angular velocities of the robot, positions and speeds of joints, and contact information. The agent receives positive reward for moving forward, and receives -100 reward if it falls to the ground. If the agent reaches the opposite end of the terrain, total reward it receives is over 300.
Environment generation. In the 2D bipedal locomotion task, we generate the environment by deciding the eight-dimensional environment parameter TR which consisting of min max stump height, min max stair height, min max pit gap stair steps, and roughness of the terrain. The RL generator of PAIRED selects each parameter sequentially, and the random generator of PLR and ACCEL randomly decide each parameter by sampling real number from its domain, which is reported Table 1. We employ the random generator construct dataset of environment parameters, and train the diffusion-based environment generator, which will be used to produce the enviromment parameter while training the agent using the proposed method. After the environment parameter is decided, the entire environment is generated by the procedural content generation algorithm.
Training time. All methods are trained utilizing RTX 3090Ti. To train DR and ADD w/o guidance, they require almost 80 hours 2 billion environment steps. ADD requires almost 92 hours, and PLR and ACCEL requires almost 160 hours to run the experiment for each random seed.
Table I: Domain of the environment parameter for the 2D bipedal locomotion task.
Stump Height
Stair Height
Pit Gap
Stair Steps
Roughness
Domain
5.00
[0.0, 5.0]
[0.0, 10.0)
[.00, 10.0]
[1.0, 9.0
18

---

B.3 Hyperparameters
To train RL agents using UED baselines, follow the implementation and hyperparameters of Parker- Holder et al. [20], which is available at https:/ //gi thub. com/ www.baokreesearrreeeeerraaaaaaae /dcd. The same parameters and network architecture were used train the ADD agent, and the hyperparameters used are reported in the Table 2. To train the diffusion-based environment generator, we followed the implementation of Dhariwal et al. [38] and Yoon et al. [45], which available at https: / //github com/tet memmtttrzim/// hereiim/aa dif dataset is set to 10 million. For the partially observable navigation task, the architecture of the error network E follows the UNet [60]. which utilizes three residual blocks with channel multipliers [1, 2] for each resolution. For the 2D bipedal locomotion task, the architecture error network four-layer MLP with sinusoidal time embedding. We report detailed hyperparameters used to train diffusion models in Table Lastly. we report detailed hyperparameters for regret- grrred diffusion process and training the environment critic in Table 4. The network architecture of the environment critic is based on the UNet encoder for the partially observable navigation task and a four-layer MLP for the 2D bipedal locomotion task.
haa--human-feedeeee
and the size of the randomly generated
Table 2: Hyperparameters used for training the RI agent in each task
Parameter
Minigrid
BipedalWalker
Y
0.995
0.99
ĀGAE
0.95
0.9
PPO rollout length
256
2000
PPO epochs
5
PPO minibatches for epoch
PPO clip range
0.2
0.2
PPO number workers
32
Adam learning rate
le-4
2e-4
Adam E
le-5
1e-5
PPO max gradient norm
0.5
0.5
PPO value clipping
True
False
Return normalization
False
True
Value loss coefficient
0.5
0.5
Entropy coefficient
0.0
1e-3
LSTM-based policy
True
False
Table 3: Hyperparameters used for training the diffusion-based environment generator
Parameter
Minigrid
BipedalWalker
DDPM timestep T
1000
1000
Network architecture
UNet
MLP
hidden dimension
128
256
Batch size
128
512
Dropout
0.0
0.0
AdamW learning rate
le-4
1e-4
AdamW weight decay
0.05
0.0
AdamW B
0.9
0.9
AdamW B2
0.999
0,999
EMA rate
0.9999
0.9999
Number of training steps
3e5
1.5e5
19

---

Table 4: Hyperparameters used for the regret guidance and fraining the environment critic
Parameter
Minigrid
BipedaiWalker
DDIM timestep T
50
200
Number of bins M
100
100
Return domain Umin: Uman
[0. 1]
[0,300]
Guidance weight w
5.0
15.0
CVaR risk level a Environment critic minibatches
0.15
0.3
128
128
Environment ciritic epochs
5
Environment critic buffer size
1600
800
C Detailed Experimental Results
In this section, we present detailed experimental results in the partially observable navigation task and 2D bipedal locomotion task. For each task, we will provide specific zero-shot generalization performance, t-SNE plots of all baselines, and the training environments generated by the proposed algorithm. Additionally, we will show examples of environments generated with varying difficulty levels using the method described in Section 4.4.
C.1 Partially Observable Navigation Task
Zero-shot transfer test results. After training the RL agent in the partially observable navigation task, we evaluate the generalization capability of the learned policy by testing the agent twelve unseen environments, which are shown in Figure 6. In each environment, the agent is evaluated for 100 independent episodes, and the full result is reported in Table We note that the reported quartile values represent the average of the quartile values of the solved rates achieved in the test environments for each seed. The results demonstrate that the proposed method achieves the best performance in five out of 12 test environments, with particularly high mean and quartile values compared the baselines. Therefore, we can infer that ADD successfully trains an agent that is robust environmental changes and generalizes to various environments.
FourRooms
15Rooms
16Rooms2
Labyrinth
Labyrinth2
Maze
Trfy
Maze2
Maze3
SimpleCrossing
SmallCorridor
LargeCorridor
PerfectMaze (M)
Figure 6: Zero-shot test environments for the partially observable navigation task. SimpleCross-
ing and FourRooms environments adopted from Shevalier-Boisverr et al. [53], and other test environments are adopted from Dennis al. [14] and Jiang et al. [19, 20].
20

---

Table 5: Partially observable navigation task results. The table shows the average solved rate and standard deviation over five independent runs, each run evaluated by 100 independent episodes for each test environment. All methods train the agent using LSTM-base --sdd PPO and evaluated after 250M environmental steps.
Environment
DR
PAIRED
PLR
ACCEL
ADD w/o guidance
ADD
FourRnoms
0.62 + 0.01
0.51 + 0.05
0.64 0.10
0.51 0.05
0.61 + 0.07
0.61 + 0.09
16Rooms
0.78 = 0.21
0.96 = 0.41
0.72 + 0.09
0.97 = 0.05
0.93 # 0.15
0,91 = 0.13
16Rooms2
0,50 0.36
0.47 1 0.28
0.60 1 0.50
0.59 0.1
0.63 1 0.39
1.00 1 0.10
Lahyrinth
0.74 0.42
0.74 0.47
0.63 0.42
0.96 0.08
0,56 + 0.41
1.00 =
Labyrinth2
0.62 0.49
0.49
0.64 0.50
0.73 0.36
0.51 0.54
0.97 =
Maze
0.36 0.47
0.21
0.82 0.32
0,18 0.31
0.79 +
Maze2
0.46 0.48
0.60 0.17
0.18 0.54
0.97 0.03
0.62 0.44
0.76 = 0.42
Maze3
0.96 0.09
0.64 0.18
0,90 0.42
0.61 0.48
0.77 0.38
0.74 - 0.31
SimpleCrossing
0.87 0.06
0.82 0.04
0,88
0.09
0.75 0.10
0.87 + 0.02
0.80 + 0.13
SmallCorridor
0.63 = 0.31
0.79 = 0.02
0.97 - 0.18
0.55 = 0.46
0,84 0.23
0.94 = 0.09
LargeCoridor
0.74 1 0.35
0.43 1 0.23
0.79 1 0.36
0.56 1 0.51
0.65 1 0.33
0.95 1 0.05
PerfectMaze (M)
0.45 + 0.20
0.43 + 0.13
0.37 + 0.24
0.64 + 0.19
0.45 + 0.21
0.82 + 0.18
Mean
0.64 0.13
0.59 1 0.10
0.63 = 0.07
0.72 = 0.07
0.63 1 0.20
0.85 = 0.05
First quartile
0.39 = 0.25
0.26 + 0.25
0.32 = 0.17
0.55 = 0.15
0.41 = 0.28
0.79 = 0.07
Second quartile
0.76 1 0.19
0.62 1 0.16
0.71 1 0.13
0.85 1 0.05
0.66 1 0.30
0.93 1 0.05
Third quartile
0.89 + 0.09
0.92 + 0.08
0.95 + 0.01
0.98 = 0.01
0.86 + 0.17
0.99 = 0.01
Figure 7: Examples of environments generated at the beginning of the agent's training in the partially observable task. The figure shows 18 example environments that are generated right after the initiation of the agent's 's learning.
Generated training environments. To support the claim regarding the generated curriculum, provide example scenes of generated training environments Figure and Figure 8. Comparing two figures reveals that the environments generated after training the agent with 200 million environmental steps are more complex and contain a larger number of blocks, aligning with the quantitative results shown in Figure 2(c). The reason for this difference is that the agent is trained, it achieves near-optimal performance in simple environments. Consequently, the environment critic predicts that the agent's 's regret will larger in environments with a greater number of blocks and increased complexity.
21

---

Figure 8: Examples of environments generated after 200 million environmental steps in the partially observable task. The figure shows 18 example environments that are generated after 200 million environmental steps
DR
PAIRED
ACCEL
ADD w/o guidance
ADD
Figure 9: t-SNE plots of training environments the partially observable navigation task. The figure shows t-SNE plots of ADD and baselines. The results obtained by mapping the environment parameters to a latent space using the encoder of the learned environment critic, followed by training t-SNE.
t-SNE plots. To conduct a qualitative analysis on the generated environments, we visualize training environments generated from ADD and baselines. Since the parameter of the maze environment is high-dimensional and hard to define the meaningful distance, we first utilize the encoder of the environment critic to map environment parameters the learned latent space, then train t-SNE on the latent vectors. The t-SNE results are shown in Figure 9. The results demonstrate that the proposed method generate sufficiently diverse environments comparable to those produced by random generator.
C.2 2D Bipedal Locomotion Task
Zero-shot transfer test results. For the 2D bipedal locomotion task, we assess the zero-shot performance of the trained RL agent in six unseen test environments shown in Figure 10. In each test environment, we evaluate the performance for 100 independent episodes. this task, PAIRED struggles to train the RL-based environment generator since the value function often diverges. Therefore, we adopted the zero-shot performance result PAIRED reported in Parker Holder et al. [20]. We report the full results in Table 6, and the results demonstrate that the proposed method outperforms baselines across all test environments.
22

---

Basic
Hardcore
Stair
PitGap
Stump
Roughness
Figure 10: Test environments for the 2D bipedal locomotion task. Basic and Hardcore environ- ments are adopted from OpenAI Gym [56], and other four test environments are adopted from Parker Holder al. [20].
Table 6: 2D bipedal locomotion task results. The table shows the average return and standard deviation over five independent runs, and each run is evaluated by 100 independent trials on each test environment. All methods train the agent using PPO and evaluated after two billion environmental steps.
Environment
DR
PAIRED
PLR
ADD guidance
ADD
Basic
153.7
21.9
280.8
Hardeore
10.6
10.1
-47.2 1 10.6
116.6 : 27.8
53.1 1 26.6
35.5 + 16.5
140.1
28.8
Stairs
10.5 + 8.1
-27.4
12.1
58.4 23.6
33.6 + 18.5
24.3 +94
75.4
34.1
PitGup
-1.8
10.7
-68.2
9.7
54.2 +:
29.6
22.4
6.7 2.244
143.2
74.9
Stump
-18.5 1 20.9
-76.0
1: 10.3
26.3
-23.2
20.8
1.33
58.2
52.6
Roughness
22.5
12.5
-5.1
25.9
144.5 = 30,8
85.2 - 21.8
85.1 26.6
168.9
38.8
Mean
29.5
16.5
-2.9 1 14.5
114.8 - 17.3
71.7 1
15.9
69.3 1 12.0
149.6
33.0
Generated training environments. To demonstrate the generated environments vary the agent learns, we provide the generated environments the early and later stages of the training in Figure 11 and Figure 12. It can be observed that the environments shown in Figure 12 much simpler compared to those in Figure 11, which is consistent with the quantitative analysis provided in Figure 3(b). This difference occurs because randomly sampling environment parameters result in overly challenging environments that hinder the agent's learning, and the environment critic guides the generator produce simpler environments that are conducive to the agent's learning.
23

---

Figure 11: Examples environments generated at the beginning of the agent's training in the 2D bipedal locomotion task. The figure presents 15 example environments that are generated right after the agent starts learning.
Figure 12: Examples of environments generated after 1.5 billion environmental steps in the 2D bipedal locomotion task. The figure presents 15 example environments that are generated after 1,5 billion environmental steps.
24

---

t-SNE plots. As in the previous task, we trained t-SNE to visualize the generated environments for the 2D bipedal locomotion task. Since the environment parameters for this task have low dimensionality and the meaningful distance can be computed using 1.2 distance, we use the raw environment parameters without mapping them to a learned latent space. The results shown in Figure 13 demonstrate that the proposed method produces sufficiently diverse training environments. In contrast, training environments generated by PAIRED are concentrated in small area. Due to this difference, the agent trained with the proposed algorithm demonstrates better performance across all test environments.
Controllable generation. As done in the partially observable navigation task, we provide the results of manipulating the difficulty of the generated enviromments in Figure 14. The results show that when we guide the generator to produce low-difficulty environments, nearly flat terrains are produced. As the desired difficulty increases, the environments become progressively more complex and challenging to walk over. This additional capability of ADD allows the trained environment critic and diffusion-based generator to reused in applications such as benchmark generation.
PAIRED
PLR
1m:
Figure 13: t-SNE plots of training environments in the 2D bipedal locomotion task. The figure shows t-SNE plots of ADD and baselines, which are obtained by training a t-SNE to visualize environment parameters in the two-dimensional space.
Easy
Difficult
Figure 14: Controllable generation results in the 2D bipedal locomotion task. The figure shows how the environments change as the desired difficulty level increases. We note that each column is generated using the same initial noise Or and random seed.
25

---

C.3
Ablation Study
We conduct ablation studies analyze the role of entropy regularization term and the number of environments used in diffusion pre-training. Frist, to show whether adding an entropy term to the original UED objective plays a critical role, we measure the zero-shot generalization performance of the trained agent with varying which is defined in the soft UED objective EĄ^A [REGRET 0)] 1. (A). The experimental results are shown in Table 7. From the results, we observed that performance decreases as w becomes large. Since the influence of the entropy term diminishes as w increases, it can be seen that our experimental results highlight the importance of the entropy term.
Next, to analyze the influence of the number of samples used during the pre-training phase, we trained the diffusion model using one million samples, which is 100 times fewer than the original experiment, and measured the performance of the proposed algorithm. The result is mean success rate of 0.76 + 0.07 in the partially observable navigation task. This is about 1 % lower than the result reported in the original experiment. The result demonstrates that a larger number of samples used in pre-training would lead to better performance, which is quite trivial since the diffusion model can generate more diverse environments when trained with larger number of samples. Additionally, we note that since we are dealing with an unsupervised setting and the samples used in pre-training are generated through random sampling, there is no need to worry about data scarcity.
Table 7: Ablation study on the entropy regularization term. The table shows the zero-shot generalization performance in the partially observable navigation task in accordance to the entropy coefficient W. We measure the average success rate over five independent seeds.
W
10
20
40
80
Mean success rate
0.85 0.05
0.81
0.05
0.82 + 0.03 0.64 + 0.07 0.47 + 0.16
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification: The paper's contributions and scope accurately described in the abstract and instruction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work discussed in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

We provide the full set of assumptions in Proposition 4.1, and the complete proof is presented in Appendix A.1
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide experiment details and hyperparameters in Appendix B.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [Yes]
We submit the code with a detailed instruction, and all experiments are reproducible.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. ec/
Policy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips... public/guider "CooeSubmisss sssisssss wwsssssosiiyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the training and test details are provided in Appendix B and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: In Figure 2 and Figure 3, all the results are accompanied by error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
29

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the type of compute workers and computation time in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //tt://eee...eeeeeeeeeeee http///urris.ccpppllieeeeeeeeeee htce/public/eee cttphieisuiideeinccccccc ines?
Answer: [Yes]
Justification:
Our research fully conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is not expected to cause direct societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not have negative case, don't need special safe- guards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all the papers that produced the code package, and provide URLs in Appendix B.3.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
31

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide the details about training in Appendix B.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "agICpYKJGy", "reviewer": "Reviewer_rCfY", "summary": "This paper proposes a diffusion model with differentiable regret estimate for unsupervised environment design. The authors write a diffusion process to model environment parameters where the process is described in terms of a scoring function and derivative of the regret. The scoring function is pre-trained on a set of random environments. The diffusion process is further fine-tuned with the regret to generate environments for curriculum learning of an agent. A critic, which is trained with cross-entropy loss using binned environment returns, is used to approximate the regret. By using the derivative of the critic w.r.t. environment parameters, diffusion process is fine-tuned with entropy augmented regret. Experimental results on continuous and discrete control domains show that the model is competitive with previous best models. Using regret guidance significantly improves the performance.", "questions": "Please see above for specific questions.", "limitations": "The paper addresses the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a diffusion process which is well suited for modeling continuous parameter. It implements a differentiable regret approximation that better assigns credit for the diffusion process. The formulation through the optimal environment distribution with a trainable scoring function is also interesting.", "weaknesses": "There are a few things that need more clarification and ablations.\n\n1. While the paper shows that the diffusion process with a differentiable regret estimate gives good results, it is not clear which component is the most critical. Is the diffusion process critical for the success? Can you train PAIRED with a differentiable regret?\n\n2. Similar to above, is the entropy term critical? Can you train the diffusion process without it and get comparable results?\n\n3. Can you explain the reason why you trained the critic model with binned returns rather than using actual returns in a regression objective?\n\n4. How critical is the pre-training for the scoring function? How does the performance change with less or more number of environment samples?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate Reviewer rCfY for the valuable feedback and review. Below is our response to the reviewer's comments and questions.\n\n### Weak 1: Is the diffusion process critical for the success? Can you train PAIRED with a differentiable regret?\nThe proposed algorithm critically relies on both the diffusion process and differentiable regret. Without a differentiable regret, using only the pre-trained diffusion process would theoretically result in generating random environments, failing to create a meaningful curriculum. This outcome is evident in Figures 2 and 3, where the performance of ADD without guidance is shown. Additionally, since differentiable regret is an approximation of the previously used regret estimation method, it is possible to train PAIRED [1] using this approach, but significant performance improvement is not expected.\n\n### Weak 2: is the entropy term critical? Can you train the diffusion process without it and get comparable results?\nThe entropy term is also critical. Adding the entropy term ensures that the distribution from which environment parameters are sampled becomes the softmax distribution of regret, as shown in Equation 8. Without the entropy term, it would be challenging to implement the sampling of environment parameters that maximize regret using the diffusion process, to the best of our knowledge. However, by setting \\omega to a exteremely high value, we can effectively simulate the absence of the entropy term. We will include an ablation study to explore the performance changes with varying \\omega values in the appendix. We report partial results in the following table.\n\n**Table 1**: Performance in accordance to \\omega in partially observable navigation task. Blanks stand for experiments that are not finished yet.\n|\\omega|5|10|20|40|80|160|\n|:-----------:|:-------------:|:-----------:|:-------------:|:-------------:|:-------------:|:-------------:|\n|mean success rate|0.85 +/- 0.05|0.81 +/- 0.05|0.82 +/- 0.03|-|-|-|\n\nAdditionally, as reviewer TVkr pointed out, there is a prior work [2] that claims adding high entropy bonus when training neural generator of PAIRED yields better performance. This supports our claim that the entropy term plays a critical role.\n\n### Weak 3: Can you explain the reason why you trained the critic model with binned returns rather than using actual returns in a regression objective?\nThe reason for training the critic model with binned returns, rather than using actual returns in a regression objective, is to obtain a differentiable regret estimate. We need to estimate the maximum and average returns, and to accurately reflect the stochasticity of the environment and policy, we trained a network to predict the distribution of returns, similar to distributional RL. We used binned returns following one of the foundational distributional RL studies, C51 [3]. However, it is also possible to use methods like Implicit Quantile Networks (IQN, [4]), which learn the distribution with actual returns as the output, and are known to yield better performance.\n\n### Weak 4: How critical is the pre-training for the scoring function? How does the performance change with less or more number of environment samples?\nPre-training the scoring function is an essential step. Through pre-training, the diffusion process becomes capable of sampling a wide range of environment parameters. This diversity enables the diffusion process to generate meaningful curricula when guided by differentiable regret. As the number of environment samples used in pre-training increases, we expect the diffusion process's ability to generate diverse environments to improve, thereby enhancing overall performance. We are conducting experiments with a smaller number of environment samples , and we will add the results to the appendix.\n\n### References\n[1] Dennis et al. \"Emergent Complexity and Zero-Shot Transfer via Unsupervised Environment Design.\" Advances in Neural Information Processing Systems. 2020.\n\n[2] Mediratta et al. \"Stabilizing Unsupervised Environment Design with a Learned Adversary.\" Conference on Lifelong Learning Agents. 2023.\n\n[3] Bellemare et al. \"A Distributional Perspective on Reinforcement Learning.\" International Conference on Machine Learning. 2017.\n\n[4] Dabney et al. \"Implicit Quantile Networks for Distributional Reinforcement Learning.\" International Conference on Machine Learning. 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ecG98MpCsB", "reviewer": "Reviewer_TVkr", "summary": "This paper proposes an approach for gradient directed, regret-based UED based on guiding a pre-trained diffusion model.", "questions": "Have you tried replacing equation 12 with a PAIRED-style loss maximising the expectation between a protagonist and antagonist?", "limitations": "See weaknesses.", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 4, "strengths": "This paper addresses a major shortcoming of prior UED approaches. In the past gradient-based UED approaches have been out-performed by sample-based or evolutionary approaches for searching for environments. While there was a general intuition that gradient based approaches would ultimately be more scalable, it was difficult to realise this intuition due to the fact that environment-design is a high-dimensional optimisation problem for which RL algorithms have struggled. Using insights from generative modelling is a natural approach to bridging this gap, and this paper provides a few tricks which seem to have been missing from prior attempts.\n\nSpecifically, pre-training the diffusion model on random levels, and using learned guidance seems to be a particularly powerful combination.", "weaknesses": "My main concern is that It seems like the baseline results in the bipedal walker domain don't replicate the results from prior work, suggesting some sort of bug/lack of tuning in the implementation? It appears that if I compare the numbers of ADD to the results in the ACCEL paper it gets lower or equivalent performance to ACCEL itself? My main reason for not currently raising my score is questions about the accuracy of this evaluation.\n\nThere are some tricks that are rediscovered from prior work which should be attributed. Specifically, entropy regularisation for neural generators is studied in [1], and training a sort of critic to evaluate levels is studied in [2]. That being said, ADD puts a unique spin on both of these.\n\n[1] Mediratta, Ishita, et al. \"Stabilizing unsupervised environment design with a learned adversary.\" _Conference on Lifelong Learning Agents_. PMLR, 2023.\n\n[2] Bhatt, Varun, et al. \"Deep surrogate assisted generation of environments.\" _Advances in Neural Information Processing Systems_ 35 (2022): 37762-37777.\n\n\nFor the citation of environment design, it is correct to cite the UED formalism was from Dennis et al. but for the more general concept of designing environments it would be best to also cite contemporaries  POET[3] and GPN [4].\n\n[3] Wang, Rui, et al. \"Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions.\" _arXiv preprint arXiv:1901.01753_ (2019).\n\n[4] Bontrager, Philip, and Julian Togelius. \"Learning to generate levels from nothing.\" _2021 IEEE Conference on Games (CoG)_. IEEE, 2021.\n\n\nIt would also be best to be careful not to equate minimax regret based UED approaches with all UED approaches as is done on line 83, as there are many non minimax regret approaches to UED such as POET, SAMPLR, CLUTR, and DRED. \n\n\nPre-training on random levels seems like it leaves only a limited amount that the diffusion model could learn. It would be interesting to periodically fine-tune the diffusion model on newly generated levels to increase the power of the generator over time. This would generally be much more convincing as it could scale much further off of the distribution of random levels.\n\n#### Clarity\n\nIt would be helpful to include the per-transfer environment bar-plots as is traditional with UED papers, it seems like the same information is included in Table 5 and 6 but they are much harder to read in that format, and it is difficult to tell where error bars overlap.\n\nIt would also be helpful to include the bootstrapped CI's plot often in UED papers as recommended by [5]\n\nFigure 4 and Figure 9 would be amazing to include in the main body, and go a long way towards explaining and demonstrating the method.\n\n[5] Agarwal, Rishabh, et al. \"Deep reinforcement learning at the edge of the statistical precipice.\" _Advances in neural information processing systems_ 34 (2021): 29304-29320.|\n|APA||\n\nHow do you arrive at the ADD acronym? It is a bit difficult for me to remember it and what it stands for.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate Reviewer TVkr for the valuable feedback and review. Below is our response to the reviewer's comments and questions.\n\n### Weak1: ACCEL results in the bipedal walker domain\n\nThere are two main differences between the original ACCEL paper and our experiments. First, the domain of environment parameters is different. As shown in Table 6 of the original ACCEL paper [1], ACCEL was implemented in two ways, ACCEL+ and ACCEL++ (we replace dagger of the original paper with \"+\" for readability in the review format). ACCEL++, which shows better performance, sampled environment parameters from an easy parameter range where the difficulty level is inherently low (Table 1 below or Table 9 of the ACCEL paper). In contrast, ACCEL+ and other baselines generated environments from the full parameter range (Table 2 below). While ACCEL++ shows powerful performance, we decided to use ACCEL+ as a baseline since it can be seen that ACCEL++ uses prior knowledge of which parameters create simple environments. It is one of the reasons why the average score of ACCEL is lower than that of the original paper.\n\nThe second difference is that the ACCEL paper recorded performance after the fixed number of policy updates, whereas we recorded performance after the fixed number of environmental steps, following other UED papers [2, 3] and traditional deep reinforcement learning research. Since ACCEL requires episodes without policy updates after mutating replayed environments, the number of environmental steps per policy update is higher than other baselines. Therefore, when using environmental steps as the metric, the performance of ACCEL degrades than that reported in the original paper.\n\nWe will include an explanation of this difference in the paper to ensure that readers are aware of these differences. Addtionally, we will add the performance of our method trained for only half of environmental steps used in the original experiment to accommodate the perspective that using the number of policy updates as a metric is more reasonable. In bipedal walker domain, average score of our method after half of environmental steps is 127.4 +/- 16.0, and we believe it is still competitive performance compared to baselines. \n\n**Table 1**: Easy parameter range in the bipedal walker domain\n|stump height|stair height|stair steps|roughness|pit gap|\n|:-----------:|:-------------:|:-----------:|:-------------:|:-------------:|\n|[0, 0.4]|[0, 0.4]|[1, 1]|[0, 0.6]|[0, 0.8]|\n\n**Table 2**: Full parameter range in the bipedal walker domain\n|stump height|stair height|stair steps|roughness|pit gap|\n|:-----------:|:-------------:|:-----------:|:-------------:|:-------------:|\n|[0, 5]|[0, 5]|[1, 9]|[0, 10]|[0, 10]|\n\n### Weak 2: Prior works and citations\nWe appreciate the reviewer for highlighting prior works ([4], [5]) related to our approach. We will add these to related works and reference them to strengthen our claim regarding the addtion of the entropy term and the use of a learned critic. We will also cite POET [6] and GPN [7] to help readers better understand previous research on the general concept of environment design.\n\n### Weak 3: No minimax regret UED\nWe will add an explanation to the related works to clarify that there are UED methods with non-minimax regret objectives.\n\n### Weak 4: Periodically fine-tuning the diffusion model\nWe agree that periodically fine-tuning the diffusion model using newly generated levels could potentially result in a more powerful generator. However, we are not sure that fine-tuning the diffusion model using levels generated by the diffusion model itself would genuinely enhance the generator's capabilities. One possible approach to address this issue is to use another UED algorithm in parallel to generate levels that our diffusion model cannot create. We believe it could be an interesting future work.\n\n### Weak 5: Clarity\n**Additional plots**: We will include per-transfer bar plots and bootstrapped CI plots below Table 5 and 6 to improve the readability of the experimental results.\n\n**Figure 4 and Figure 9**: We will include Figure 4 and its explanation in the beginning of Section 5.1. We will also include Figure 9 after Section 5.2 and add its explanation, which is described in Appendix A.5,  after Section 4.4. \n\n**ADD acronym**: The ADD acronym stands for \"A\"dversarial environment \"D\"esign via regret-guided \"D\"iffusion models. We welcome any suggestions for a better acronym.\n\n### Question 1: Replacing equation 12 with a PAIRED-style loss\nWe have not yet attempted to replace Equation 12 with a PAIRED-style loss, which predicts the difference between the antagonist and protagonist. However, we believe it would be a worthwhile experiment and could potentially yield competitive performance.\n\n### References\n[1] Parker-Holder et al. \"Evolving Curricula with Regret-Based Environment Design.\" International Conference on Machine Learning. 2022.\n\n[2] Jiang et al. \"Replay-Guided Adversarial Environment Design.\" Advances in Neural Information Processing Systems. 2021.\n\n[3] Garcin et al. \"DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Design.\" International Conference on Machine Learning. 2024.\n\n[4] Mediratta et al. \"Stabilizing Unsupervised Environment Design with a Learned Adversary.\" Conference on Lifelong Learning Agents. 2023.\n\n[5] Bhatt et al. \"Deep Surrogate Assisted Generation of Environments.\" Advances in Neural Information Processing Systems. 2022.\n\n[6]  Wang et al. \"Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions.\" arXiv:1901.01753. 2019.\n\n[7] Bontrager et al. \"Learning to Generate Levels from Nothing.\" IEEE Conference on Games. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SzTpbevK28", "reviewer": "Reviewer_1NF7", "summary": "This work applies regret-guided diffusion models to the UED setting in order to generate adversarial environments that preserve diversity.", "questions": "* What are the assumptions on the structure of the environment parameterization? Can this be applied to all simulated environments? Or only a subset with certain properties?\n* Why were RGB channels used to describe the maze environment instead of e.g. a binary (continuized for the diffusion process); it seems odd to parameterize the environment based on the observation space, and then to extract these parameters themselves based on the observation values. Are there certain parameterizations for the diffusion process that work better/worse? What properties does this representation space need to have?\n* Are the t-SNE embeddings generated using all points, and then each individual plot displays just the relevant method’s points?", "limitations": "* The limitations are outlined accurately by the authors in Section 6. However, I would like the authors to address the first question above, which I believe is an additional limitation that would be worth discussing.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The contributions are well motivated and appear to be novel.\n* The writing is generally clear and concise.\n* The paper is contextualized well within prior literature.", "weaknesses": "* Minor typos/grammatical issues:\n    * Line 253: “challenging”—This word alone does not hold any descriptive power. Because the nature of the tasks are described next in impartial terms, this word seems redundant/rhetorical.\n* The limitations do not mention the assumed structure of the environments; I am not convinced that diffusion can be applied to all environment parameterizations. \n* Why is shortest path length used as a complexity metric? It seems to describe the tail of the distribution; Why not average path length? Also, number of blocks does not seem to be a well-motivated complexity metric either.\n* While ADD “successfully generates adversarial environments while preserving diversity”, the behavior improvements seem to be minimal based on the training curves / zero shot performance. \n* A more general point: it is hard to tell what exactly the authors are trying to show from the experiments; the conclusion nicely sums up the analysis, but it would have been nice to have been told at the top of “experiments” what each experiment is trying to test.\n* Given the prevalence of UED domains in the literature, a third evaluation domain would really strengthen this work. Otherwise, I am concerned about the applicability of this method to other environments, especially since the authors have not addressed the implicit assumptions about workable environment parameterizations for their diffusion approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate Reviewer 1NF7 for the valuable feedback and review. Below is our response to the reviewer's comments and questions.\n### Weak 1: Minor typo \"challenging\"\nWe agree that the word \"challenging\" in line 253 is redundant, so we will exclude it.\n### Weak 2: About complexity metrics: shortest path length and number of blocks\nWe used shortest path length and the number of blocks following prior works [1, 2, 3]. While the number of blocks may not always correlate with an environment's complexity, there is a tendency for environments to become more maze-like and challenging as the number of blocks increases when the goal and start positions are fixed. Thus, considering both the number of blocks and the shortest path length can give a rough estimate of the environment's complexity. Additionally, we agree that a metric considering the lengths of all paths can better explain the complexity of an environment. However, since there are numerous paths to the goal within the time limit, simply averaging these paths does not meaningfully explain complexity. Hence, a carefully designed metric considering the lengths of possible paths is needed, and we believe this is an area that requires further exploration by UED researchers.\n### Weak 3: About the expression “successfully generates adversarial environments while preserving diversity”\nIt summarizes the complexity and diversity of the environments generated during the training process, which are shown in Figures 2(c) and 2(d). We would like to point out that it is not about the performance.\n### Weak 4: What each experiment is trying to test\nTo help readers better understand the purpose of each experiment, we will improve the explanation in Line 266-272 as follows.\n\"The primary goal of our experiments is to evaluate the generalization capability of the trained agent. To this end, we measured the zero-shot transfer performance in challenging, human-designed environments. Additionally, to understand where the differences in performance originate, we conducted quantitative and qualitative analyses of the curriculum of the generated environments by tracking complexity metrics and generating t-SNE plots.\"\n### Weak 5: A third evaluation domain and applicability\nRegarding the reviewer's concern about applicability, we would like to point out that most tasks used in current UED research either have a form similar to Minigrid or utilize a continuous parameter space. Hence, sampling environment parameters using a diffusion model is feasible. Therefore, we agree that experiments in third domain will strengthen our work, but not necessary to demonstrate our method's applicability. For deeper discussion on the workable parameterization, please refer to the response to the question 1.\n### Question 1: About the assumptions on the structure of the environment parameterization\nThe assumed environment parameterization is one that can be learned by the diffusion model. If the parameter space is continuous, methods such as DDPM [4] can be applied. Even when the parameter space is discrete, methods such as D3PM [5] can be applied. Hence, we believe that our algorithm can cover most tasks that can be handled by existing UED methods, which model the environment generation process as an MDP [1] or randomly sample environment parameters [2, 3]. \n\nEven for environment parameters requiring constraints, such as ensuring that a path always exists from start to goal in a maze environment, we can utilize guided diffusion and rejection sampling to meet the constraints. Furthermore, when the constraint is complex, the diffusion model can be an effective solution, as discussed in [6]. Therefore, it can be seen that our method has a potential to handle both continuous and discrete parameters with constraints. However, if the environment itself is challenging to parameterize, it would be difficult to apply ours and other UED algorithms.\n\nWe hope this response sufficiently addresses the reviewer's question. If there is an example of a parameterization where challenges in applying our approach is expected, we would be happy to discuss it further.\n### Question 2: About RGB channels used to describe the maze environment\nThe representation that we used is a continuized binary, where open spaces are represented by 0 and other elements by 1. And there are two reasons why we use three channels. First, it allows us to represent obstacles, a start point, and a goal point in different channels using binary values, which we believe is a natural representation. Second, it is easy to visulize. It is technically feasible to use a single channel and represent each element with a different value. We expect this would make training even easier than with our representation, as the reduced dimension could lead to faster learning. We hope we have understood the reviewer's question correctly and our respones addresses it sufficiently.\n### Question 3: About t-SNE embeddings\nWe generated the t-SNE embeddings using all environment parameters, and each plot displays only the points corresponding to the respective methods. By doing so, we were able to compare the diversity of environments generated by each baseline.\n### References\n[1] Dennis et al. \"Emergent Complexity and Zero-Shot Transfer via Unsupervised Environment Design.\" Advances in Neural Information Processing Systems. 2020.\n\n[2] Jiang et al. \"Replay-Guided Adversarial Environment Design.\" Advances in Neural Information Processing Systems. 2021.\n\n[3] Parker-Holder et al. \"Evolving Curricula with Regret-Based Environment Design.\" International Conference on Machine Learning. 2022.\n\n[4] Ho et al. \"Denoising Diffusion Probabilistic Models.\" Advances in Neural Information Processing Systems. 2020.\n\n[5] Austin et al. \"Structured Denoising Diffusion Models in Discrete State-Spaces.\" Advances in Neural Information Processing Systems. 2021.\n\n[6] Yang et al. \"Compositional Diffusion-Based Continuous Constraint Solvers.\" Conference on Robot Learning. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c5uZmFAUhY", "reviewer": "Reviewer_a2dv", "summary": "This work focuses on Unsupervised Environment Design (UED), a problem setting whereby a teacher designs environments for a student, learning to solve the task. This area of research has been in focus recently due to its ability to train more general agents in an open-ended setting. The authors look to build on recent work using generative models to generate environments, and propose a diffusion based approach ADD. The method seems sound and leads to gains in two of the canonical UED environments. I recommend acceptance as it incrementally improves UED literature.", "questions": "* The meta data plots show ACCEL starting with the same number of stumps etc as the other baselines. However in the original implementation/paper it should be starting with very simple terrains. Was this the case in your work?", "limitations": "The limitation of requiring additional data to pre train the diffusion model is not mentioned, I think this is actually the largest weakness by far. It could be interesting to see if pre trained foundation models work well here too.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strengths are listed below. Since I recommend acceptance, the focus of the review is on weaknesses to improve the paper as much as possible for the camera ready version.\n* The method makes sense intuitively and seems to work as expected, especially with the curriculum results.\n* The paper is well written and clearly builds on previous works (e.g. in the related work discussion). \n* Experiments are well presented and show clear performance gains.", "weaknesses": "* The x axis for the plots is showing total steps and not student gradient updates. This puts replay methods at a huge disadvantage since they take ~2x the number of environment steps, but the policy isnt training so arguably it should not count when comparing the effectiveness of the curriculum. I think this should be changed to match the literature.\n* The baselines are fairly weak, aside from PLR and ACCEL (which have been weakened by the previous point). Can the authors compare against CLUTR and show why ADD produces a better curriculum? To me that is the obvious baseline. Further, note the improvements to PAIRED proposes in [1] which makes it a much stronger baseline. I would say the original PAIRED is more of a concept than a baseline at this point due to known deficiencies. \n* There is a clear limitation of your method that you require a pre-training phase with access to the environment, and do not count the number of steps as part of training because it is \"unsupervised\". This is fine, but then makes it wrong to show training steps for the replay methods. Please either 1) include the steps from this in the x axis for ADD (which will drastically weaken performance) or 2) switch the x axis to student gradient updates. Otherwise it is an apples to oranges comparison in your favour.\n* I would love to see examples of the generated environments in the main body, for me it is more useful than the theory part, but maybe that is subjective :)\n* More of a general comment, I find it strange that UED papers don't cite [2]. It is the largest scale demonstration of the value of UED and motivates research on these methods. It explicitly shows PLR is effective for curricula over a massive task space with a 500M parameter transformer based policy. This clearly shows that UED research can have impact on large scale AGI focused projects, so I would have expected the UED community to all be very excited about this.\n\n[1] Mediratta et al. \"Stabilizing Unsupervised Environment Design with a Learned Adversary\". CoLLAs 2023\n[2] Bauer et al. \"Human Timescale Adaptation in an Open-Ended Task Space\". ICML 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate Reviewer a2dv for the valuable feedback and review. Below is our response to the reviewer's comments and questions.\n### Weak 1: About the x axis for the plots\nWe agree that using total steps is disadvantageous for replay methods. However, we want to point out that if we use student updates, it becomes disadvantageous for methods that do not use a replay. We deliberated on this issue when organizing the experimental results, and decided to use total steps based on two reasons; (1) it is common in reinforcement learing (RL) research to evaluate the performance of the online RL algorithm after the fixed number of environmental steps. (2) replay methods prior to ACCEL also used total steps for the x axis [1, 2].\n\nDespite our decision, we realize that there could be a controversy about this issue. Hence, we measure the performance of our method trained for only half of environmental steps used in the original experiment to accommodate the perspective that using the number of policy updates as a metric is more reasonable. When using half of environmental steps, average score of our method in bipedal walker is 127.4 +/- 16.0, and average success rate in maze environment is 0.72 +/- 0.04. We believe the performance is still competitive compared to baselines, and will add this performace to the appendix C.1 and C.2.\n\n### Weak 2: About the baselines\nWe agree that it would be beneficial to compare our method against CLUTR [3] and PAIRED with high entropy and protagonist-antagonist behavioral cloning [4]. However, we want to point out that [3] was proposed to solve combinatorial explosion issue and that the bipedal walker domain is not a target domain. Furthermore, [4] eventually utilizes replay in the bipedal walker, so it is not adequate methods to represent learning-based methods. Based on these points, we did not include them in the baselines, but since experiments in the maze environment is possible, we will conduct additional experiments on these baselines and add the result to the Appendix C.1.\n\n### Weak 3: About including the pre-training steps into the x axis\nWe want to point out that there is no interaction with environments during the pre-training phase since we randomly sample parameters without access to environments and train the diffusion model using the sampled parameters. So we disagree about the reviewer's concern that we should take account the steps from pre-training phase in the x axis. However, similar to the reviewer's point, we agree that the time spent on pre-training should be considered for a fair comparison. To resolve this issue, we will include the performance measured after the number of environmental steps that is adjusted to reflect the time spent on pre-training. Since pre-training took much less time than agent training (9 hours vs 56 hours in the maze environment, and 7 hours vs 92 hours in the BipedalWalker environment), we expect there will be no drastic performance degradation.\n\n### Weak 4: Examples of generated environments in the main body\nWe will resize Figure 7 and Figure 12 and add each to Section 5.1 and Section 5.2.\n\n### Weak 5: An additional citation\nWe appreciate the reviewer for pointing out the remarkable research [5] related to UED. We will add it to our citations.\n\n### Question 1: About the meta data plots of ACCEL\nThe ACCEL paper [6] describes two implementation methods. The difference between the two methods lies in the domain from which the initial environment parameters are sampled. One method samples from the full parameter range, while the other restricts sampling to a range that ensures simple environments are generated. Starting with a simple environment and gradually evolving it is a simple yet powerful idea. However, when using ACCEL as a baseline, it might not be fair because it already incorporates prior knowledge of which parameters create simple environments. Thus, we chose to use the ACCEL implementation that samples parameters from the full parameter range as the baseline. As a result, complex environments are generated at the beginning of training, leading to differences from the original paper's plot.\n\n### References\n[1] Jiang et al. \"Prioritized Level Replay.\" International Conference on Machine Learning. 2021.\n\n[2] Jiang et al. \"Replay-Guided Adversarial Environment Design.\" Advances in Neural Information Processing Systems. 2021.\n\n[3] Azad et al. \"CLUTR: Curriculum learning via unsupervised task representation learning.\" International Conference on Machine Learning. 2023.\n\n[4] Mediratta et al. \"Stabilizing Unsupervised Environment Design with a Learned Adversary.\" Conference on Lifelong Learning Agents. 2023.\n\n[5] Bauer et al. \"Human Timescale Adaptation in an Open-Ended Task Space.\" International Conference on Machine Learning. 2023.\n\n[6] Parker-Holder et al. \"Evolving Curricula with Regret-Based Environment Design.\" International Conference on Machine Learning. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper "Adversarial Environment Design via Regret-Guided Diffusion Models" presents an approach to unsupervised environment design (UED) in reinforcement learning. The authors propose using a diffusion model guided by differentiable regret to generate adversarial environments, which leads to the creation of meaningful curricula for training agents. The reviewers agree that the paper makes a significant contribution to the field and recommend its acceptance.

The strengths of the paper lie in its use of diffusion models and differentiable regret to guide the generation of adversarial environments. The proposed method, ADD, demonstrates superior performance compared to state-of-the-art UED methods in terms of zero-shot transfer to challenging, human-designed environments. The authors provide a comprehensive analysis of the generated curricula, showing that ADD successfully creates diverse and increasingly complex environments that facilitate the learning of robust agents.

The reviewers raised several important questions and concerns, which the authors addressed thoroughly in their rebuttal. They clarified the critical role of the diffusion process, differentiable regret, and the entropy term in the success of their method. The authors also provided additional experimental results and ablation studies to support their claims and demonstrate the robustness of their approach.

While some reviewers suggested including additional baselines and evaluation domains, the authors convincingly argued that their chosen baselines and domains are representative of the current UED literature. They also discussed the applicability of their method to various environment parameterizations, as well as its potential to handle both continuous and discrete parameters with constraints. Overall, the reviewers' concerns have been adequately addressed in the rebuttal, and the authors have promised a commitment to improving the paper based on the feedback received. Given the contributions and the overall quality of the work, I recommend accepting this paper for publication at NeurIPS.
Decision: Accept (spotlight)