Improving Regret Approximation for Unsupervised Dynamic Environment Generation
Authors: Harry Mead
Bruno Lacerda
Jakob Nicolaus Foerster
Nick Hawes
Venue: NeurIPS 2025 poster ()
Keywords: UED
Reinforcement Learning
Primary area: reinforcement_learning
TL;DR: Improving both regret approximation and level generation for unsupervised environment design, allowing for scaling to larger environments.

Abstract: Unsupervised Environment Design (UED) seeks to automatically generate training curricula for reinforcement learning (RL) agents, with the goal of improving generalisation and zero-shot performance. However, designing effective curricula remains a difficult problem, particularly in settings where small subsets of environment parameterisations result in significant increases in the complexity of the required policy. Current methods struggle with a difficult credit assignment problem and rely on regret approximations that fail to identify challenging levels, both of which are compounded as the size of the environment grows. We propose Dynamic Environment Generation for UED (DEGen) to enable a denser level generator reward signal, reducing the difficulty of credit assignment and allowing for UED to scale to larger environment sizes. We also introduce a new regret approximation, Maximised Negative Advantage (MNA), as a significantly improved metric to optimise for, that better identifies more challenging levels. We show empirically that MNA outperforms current regret approximations and when combined with DEGen, consistently outperforms existing methods, especially as the size of the environment grows. We have made all our code available here: \url{https://github.com/HarryMJMead/Dynamic-Environment-Generation-for-UED}.

Improving Regret Approximation for Unsupervised Dynamic Environment Generation
Harry Mead University of Oxford
Bruno Lacerda University of Oxford
Jakob Foerster University of Oxford
Nick Hawes University of Oxford
Abstract
Unsupervised Environment Design (UED) seeks to automatically generate train- ing curricula for reinforcement learning (RL.) agents, with the goal of improving generalisation and zero-shot performance. However, designing effective curric- ula remains difficult problem, particularly in settings where small subsets of environment parameterisations result in significant increases in the complexity of the required policy. Current methods struggle with difficult credit assignment problem and rely regret approximations that fail to identify challenging levels, both of which are compounded as the size of the environment grows. We propose Dynamic Environment Generation for UED (DEGen) to enable a denser level gen- erator reward signal, reducing the difficulty of credit assignment and allowing for UED to scale to larger environment sizes. We also introduce a new regret approxi- mation, Maximised Negative Advantage (MNA), significantly improved metric to optimise for, that better identifies more challenging levels. We show empirically that MNA outperforms current regret approximations and when combined with DEGen, consistently outperforms existing methods, especially as the size of the environment grows. We have made all our code available here: /tttps://gith::ii com/HaarryyhJoead/Dyyamii Environment
- Generation foreaattooooooo--r -oo-
Introduction
Deep Reinforcement Learning (RL) has been effective in training highly-capable agents in number of different challenging settings, such as real-world robotics applications 2 48 28), or games such Go [52], Chess [53), Starcraft [58] and Dota [5). However, these deep-RL RL agents tend to exhibit poor generalisation when transferred to tasks or environments with only small changes to those used to train [62 9
In order to address this lack of robustness, randiin-andommsaation (DR), training over diversity of environment parameterisations, has proven successful in number of applications. However, DR relies on random parameterisations resulting in useful training examples, and in complex envi- ronments this may not be the case. Automated Curriculum Learning (ACL) []6 42 methods aim to produce adaptive curricula for training that ensure the generation of useful training examples whilst maintaining sufficiently diverse distribution these environment parameterisations. These
methods have shown success over naive domain- ranauminrandomisttiionn approaches 43 38).
However, manually designing suitable curriculum for learning may in itself be a challenge, whilst also limiting the capacity for open-ended learning [59 60). Recent work has focused on Unsupervised Environment Design (UED) [1], which has emerged as widely applicable curriculum design method as no prior environment knowledge is required. In the UED literature, each parameterisation of the environment is referred to a level, and UED frames the curriculum design problem as the interaction between a teacher agent designing levels and student agent training on these levels. The majority of existing work focuses on maximising student regret [1] 27 40 8), as prior work [II] has shown that if the student and teacher reach a Nash equilibrium of a minimax regret game, the
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) No Key Required
(b) Key Required
Figure I: Examples of two possible randomly generated levels. In the first, the agent (red triangle) simply navigate the goal (green square), whereas in the second, it required to first obtain the key (two blue triangles) in order to unlock the door (blue unfilled square) blocking the path to the goal
student must necessarily be able to solve all solvable environments. However, computing regret is intractable for many complex tasks, these methods require regret approximations.
Generally UED methods can be categorised as either relying on a learnt level generator or a curation process that selects and replays levels from randomly generated set [25]
I 35 27
40
Existing UED methods have focused on environments such as minigrid I1S 8 or bipedal walker
[40] where there is relatively smooth transition in difficulty between levels. However, there are many environments where small subset of paramaterisations may induce a step-change in difficulty for the student. For example, in the level shown in Figure [(a) the addition of the door and key the level generation have no effect on the difficulty of the level for the student agent. The key can simply be ignored and the door acts any other wall, and the agent is able navigate to the goal directly. However, in Figure [(b) the door blocks the agent's path to the goal, so it is necessary for the agent to first find the key before being able to unlock the door and reach the goal. In this example, these key-requiring mazes represent a very small subset of possible random levels, but they also represent the levels potentially these, it is necessary for methods to identify and train on this more challenging subset of levels.
difficult learn. Thus, for UED be effective in environments such as
Whilst replay-based based methods sufficient in small environments, the challenge of sampling and identifying more difficult subsets of levels is amplified as the size of the environment increases. Due to their reliance random level generation, we show that replay methods fail in larger environments, and thus it is necessary to use learnt level generators. However, training a generator that generates a full level prior to student rollouts presents challenging credit assignment task, given the long time horizon and sparse rewards. In order to address these challenges with learnt level generation, we propose Dynamic Environment Generation for Unsupervised Environment Design (DEGen). Our method involves dynamically generating the environment as the student agent explores the level, enabling much denser teacher reward signal, and reducing the difficulty of credit assignment.
However, we show in this work that current regret approximations are insufficient, both for identifying these most difficult subsets of levels and for use in training level generators. We propose Maximised Negative Advantage (MNA) as a more effective regret approximation and show substantial empirical performance improvements
existing regret approximation metries. Using MNA, show that DEGen performs substantially better than existing generators that rely on full level generation upfront. We show that DEGen is capable of matching exceeding the performance of existing replay methods in small environments, unlike previous learnt generators, but performs substantially better as the size of the environment increases. We also show that MNA consistently improves performance over current regret approximations for all UED methods.

---

Our contributions are:
We introduce Dyanamic Environment Generation, for UED (DBGen) new method of environment generation, showing performance improvements over existing learnt generators in small environments and significant performance improvements over all methods in larger environments.
We introduce Maximised Negative Advantage (MNA) as new regret approximation and show substantial improvements over existing metrics.
2 Background
2.1
Unsupervised Environment Design
Given a specific environment, we can model a level as Partially Observable Markov Decision Process (POMDP). POMDPs can be defined by a tuple (S. A, O, T,I, R. Po: 7), where S, and O are the set of states, actions and observations respectively. T S A + S is the transition function, mapping a state-action pair (&t, az) to the subsequent state 81+1,
- is the observation function that maps a given state to an observation, R S x A - IR is the reward function, Po is the distribution over initial states, and 7 is the discount factor.
In order to extend this formulation to the framework of UED, the Underspecified POMDP (UPOMDP) introduced M.. defined by the tuple M (S, A, O, TM IIMM, RMM. P6M 7). The UPOMDP formulation introduces θ the set of all possible free environment parameters 8, for which a specific 0 results in the environment configuration defined by the POMDP MM with the transition, state and reward functions T I", R² and the initial state distribution p
Generally, the UED objective is to identify training levels that maximise the student's regret, given the current student policy æ. The regret is defined as:
Regret( TT, 0) ( 0) + ((TT,
where za is the optimal policy given 6, and ( 0) Ex,Mo counted return of the policy As such, UED can be framed as two player minimax regret game:
or the expected dis-
min max Regret(m, 0). eE
By framing UED as this regret-based minimax game, if the environment satisfies the reward conditions outlined in .TΤ, we can guarantee that if the student and teacher policies reach a Nash equilibrium, then the student policy must necessarily be capable of solving all solvable levels.
2.2 Existing UED Methods
Whilst the optimal objective shown in Equation I has robustness guarantees, in practise, it is infeasible for UED given тे is required. UED methods such as PAIRED |II or CLUTR [3] introduce an additional antagonist agent, and regret is approximated as the difference between the performance of the antagonist and student policies. Both these methods rely on RL-trained teacher, where the teacher aims to maximise the performance difference between the antagonist and the student. However, these RL-trained teachers tend to struggle with maintaining diversity over training environments 25). Some techniques have shown improve performance, such as behaviour cloning between the antagonist and protagonist and the use of high entropy coefficients [35]. However these RI--based methods still tend be outperformed by replay-based methods.
Rather than relying on learnt generator, PLR (27). relies on maintaining a replay buffer of high- regret levels that have been sampled from random generator. PLR alternates between sampling new random levels and replaying previously sampled levels. PLR relies on score function to approximate regret, with the two most commonly used score functions being Positive Value Loss (PVL) and Maximum Monte Carlo (MaxMC) 22 40).
PVL is defined as
1
max 00,
Σ (Ay)
(3)
T
3

---

Algorithm DEGen
Initialise: student policy Thi generator policy A&z
while not converged do
// Sample N trajectories
for n E 1 : N do
Initialise empty level // Take T student steps
for 1g : do
// Generate partial level
Sample A actions to generate section of level that has been observed but not generated
// Take student action
Sample action
end for
compute score using student trajectory T assign reward to generator trajectory Tg end for
Update Φi according to sampled student trajectories
Update de according to sampled generator trajectories end while
where 7 is the discount factor, from the Generalised Advantage Estimator [50] and ò, is the 1-step TD-error at timestep f. We view PVL approximating regret as the average advantage, but with the advantage clipped at O. As such, maximising PVL can be seen as effectively maximising states where the student does better than expected, which intuitively appears to be mismatched with the regret objective. Despite this, empirically, PVL has been shown to be effective.
MaxMC approximates regret using the maximum achieved return (Rmar) on a given level, and is defined as
T
Σ T 1-0
Rmax - ((s))
where (s²) is the learnt value function approximation for the value of the current policy state 8,. MaxMC appears a more intuitive approximation for regret than PVL. However by relying on a Monte Carlo approximation for regret, MaxMC requires sufficient number of trials in a level such that Rmar a good approximation for the optimal return. Additionally, MaxMC can only be used in environments where reward is obtained at the final step, Rmar is dependent the full episode reward.
Whilst PLR has been shown be effective in number of domains, relying on random level generation necessarily means that no insight gained from past levels to influence future levels generation. ACCEL (4) addresses this by augmenting PLR such that new levels are generated by mutating existing levels previously in the replay buffer. This evolutionary approach enables some capacity for learning from previously identified high-regret levels, but still relies random mutations for new level generation.
Similarly to PLR, SFL relies on sampling set of randomly generated levels and selecting those with the highest scores for training. However, rather than this score metric approximating regret, SFL aims to train on levels with high learnability, defined as p(1 p), where p success rate of the current policy on the sampled level.
An alternate approach for UED is Sampling for Learnability (SFL) 49
3 Dynamic Environment Generation
Existing work 25 35 has shown that replay-based UED methods generally outperform methods relying a learnt generator. This can be attributed to number of factors, but primarily, level generation presents difficulties for learning. The level generation learning problem has long time horizon and sparse rewards, and so credit assignment is challenging. Current UED have focused on relatively small environments where it feasible to sample useful training levels from random level generation. However, the size of the environment grows, and especially with environments

---

with features that can add complexity, it becomes more difficult to sample useful levels. Therefore, it becomes necessary to use a learnt generator instead.
Figure 2: Illustration of how DEGen generates the level as the student agent explores the level, with purple colouring indicating areas that are yet to be observed.
In order to address this, whilst also reducing the challenges inherent to RL-based learnt generators, we propose Dynamic Environment Generation (DEGen), outlined Algorithm [ Rather than simply generating the entire environment level initially, we exploit the partial observability of the student agent and instead only generate those parts of the level the student observes, as illustrated Figure This allows for much denser reward signal, reducing the difficulty of credit assignment.
If we consider the PVL and MaxMC regret approximations shown in Equations and respectively, both can be written in the form
Regret T Σ G₁,
where approximate regret the of some value G2 across the student trajectory. If the entire level is generated initially, this approximation must simply be assigned as the reward for the last step in the level generation trajectory, resulting in very sparse reward signal for the generator. However, if the level is generated as the student trajectory unrolls, we are able to assign a much denser generator reward. If we have the function t, T(gg) that maps the generator timestep fy to the student timestep t, we can instead assign the generator reward (ry) at timestep t as
T( +1)-1
rte T
Σ
G₁-
(6)
This increased reward density reduces some of the difficulty of the credit assignment challenge for training the level generator. Additionally, DEGen reduces noise in the credit assignment by only generating parts of the level the student observers. If the entire level is generated initially, it is likely that some parts of the level will never be observed by the student, and so have no effect on the score of the level. As such, these actions only serve to add additional noise the credit assignment task, an issue which is negated by DEGen.
Another issue present in existing RL-based generators is lack of diversity in generated levels. Previous work (35) has shown that high entropy coefficient can reduce this issuc, however we still observed reduced level diversity with existing RL-based generators. As the student policy stochastic, by generating the level based on where the student explores, we introduce an greater degree of stochasticity in the level generation, which increases the diversity of generated levels. We also found that introducing some additional randomness in level generation specifically randomly initialising the starting location of the student agent improved training level diversity, and zero shot agent performance.
Maximised Negative Advantage
Whilst the PVL and MaxMC regret approximations have shown success number of domain, there are flaws with both metrics in relation to generating challenging levels. For PVL., high positive advantage will result in a high PVL. score, whereas generally, more challenging levels would tend to be more difficult than expected, and so more likely to result in high negative advantage. For MaxMC,
5

---

high scores require at least one high return rollout, and again, for the most challenging levels, this is unlikely to occur. Whilst PVL and MaxMC have shown success with use replay-based methods, we observed poor performance when these metrics were directly optimised for, shown in Appendix D.2
We instead propose Maximised Negative Advantage (MNA) as more suitable metric.
Consider the policy и with the value-function (ss). We are aiming for an approximation of Equa- tion requiring an approximation for both ((Tg. 0) and (=, 0). If we assume deterministic environment, given a trajectory of length T, if (ss) is the true value function, we can lower bound
U ( 8)
V(so),
(V1))) + 0o),
U 0) 2 max
(7)
(((T) + Σ rskk ax)
We label this maximum over value functions
V(t)):
ymax (st) max
(8)
1+n-1
7 ((++n)
Σ
x rrsk, ax)
Given this, the true value function ((s)) gives us the expected performance of the current policy, and the maximum over value functions Vipaa (sp) lower bounds the performance of the optimal policy, we can lower bound the regret as
Regret 2 V(s)) + V/naa (sp).
However, in practise, the exact value function V(s) will generally be unknown, and instead must be approximated with learnt value function (s). As this learnt value function may overestimate the value of the state, the inequality in Equation 7 does necessarily hold when V(s) replaced with (s). Additionally, the (5o) approximation for (TT, 0) will be biased, being learnt value approximation. We can reduce the likelihood of (s)a exceeding U 0) by instead using the approximation (ymax(so)) where < T, although this instead results in potentially overly conservative approximation. Similarly, we can reduce the bias of our approximation for (TT,, 0) by instead using the approximation q (s.) + Σk= >*r(s*, ag), however this then introduces greater variance. We therefore define the n-step regret approximation timestep t
t+n-1
+ G(m) - (
E (r axt) Frmax (8t)
(10)
and we note the similarity between this regret approximation and the negative n-step advantage estimation [50]. In order to balance both the bias and variance of the U( 0) approximation, and the conservativeness the UTTg. 6) approximation, line with the Generalised Advantage Estimator [50], we introduce the regret approximation
G = (1 (1- xm
(11)
Empirically, we find that rather than just approximating regret at the first state, using the mean regret approximation more effective metric
1
Σ
GA
(12)
T
t=0
We show empirically that this regret approximation is much suitable optimisation metric for learnt generators, whilst also showing improved performance when used in replay-basee methods.

---

4.1 Solvability
Whilst the metric in Equation 12 does allow for more challenging levels to be sampled than existing metrics, the reliance on the learnt value function ((s) to determine maximum possible performance does present issues with ensuring level solvability. If the environment satisfies the reward conditions [II that ensure the teacher-student regret Nash equilibrium results in an student policy capable of solving all possible solvable levels, then regret is maximised by generating solvable levels. Therefore, a good regret approximation metric should not result in high scores for unsolvable levels. Both PVL and MaxMC implicitly score low for unsolvable levels. For the Nash equilibrium result to hold, the reward conditions [1]] necessitate that the maximum achievable return for an unsolvable level F Fmez must not exceed the minimum return achievable in a solved level Smin- In the case of MaxMC, higher Rmar can be achieved if the level is solvable, and so solvable levels will generally score higher. For PVL., higher returns will tend to result higher advantage, and therefore a higher score, so solvable levels that can achieve higher returns will score high.
The issue with this implicit bias towards solvable levels is that in practice, this manifests bias towards levels with a high success rate [49]. i.e. the current policy solves the level with a high probability. Therefore, these metrics generally do not produce sufficiently challenging levels for training. SFL [49] has shown that training using levels with approximately 50% success rate results in strong training performance. However, determining the exact success rate for given level requires substantially more environment rollouts than necessary for metries such as PVL or MaxMC.
While MNA is capable of identifying challenging levels with significantly fewer rollouts than directly scoring based on success rate, over-approximations of state value (ss may result in high scores for unsolvable levels. In order to compensate for this, we introduce an explicit penalisation for unsolvable levels. As it is often intractable to determine exact solvability of levels in complex environments, we instead introduce approximate unsolvablilty, where we define level as approximately unsolvable if it has never been solved, e.g. has a success rate of 0%. Therefore, if level approximately unsolvable, we set the score for the level to zero. As such, our final proposed regret approximation is
T
1
MNA
G
C
(13)
1=
where E' is O if the level is approximately unsolvable and 1 otherwise.
5 Experimental Setup
For this work, we examine the standard minigrid environment used in previous UED work [1] 25 40) as well as evaluating UED performance on the modified minigrid with the addition of key and locked door. In line with exisiting UED work, our main evaluation metric is zero-shot performance on a set of hand- -and-nesi test levels. For the standard minigrid, we use the set of test levels used in previous work [T0 49). For the modified key minigrid, we modify this set of levels so as to require the agent to unlock the door to reach the goal. Previous work has only examined minigrid when students are trained using 13x13 levels, but in order to scale UED to larger environments, we need to ensure that the student is still capable of learning complex skills, such as unlocking door, even when trained in larger environments. To assess the student's 's ability to solve levels that require the door to be unlocked, we evaluate student performance on the existing key minigrid test levels but when trained on levels that are 17x17 and 21x21.
In order to assess the effectiveness of both MNA and DEGen, we compare a number of different existing level generation methods and number of regret approximation metrics. For regret approximation metrics, we compare MNA to the existing metrics, MaxMC and PVL. For assessing these metrics, we use the existing UED methods PLR [25] and ACCEL [40]. We include Domain Randomisation (DR) to show the relative performance of UED compared to a naive, random approach. Additionally, we include SFL [49 for non-regret based approach. For the standard minigrid environment, we also include the Initial Gen baseline, corresponding to an RL-trained teacher that generates the full environment prior to student rollouts, however we show that this performs substantially worse than all other methods so do not include the key minigrid domain. All student agents are trained using PPO [51), well as the teacher agents used in DEGen and Initial
Baselines:
7

---

Gen. Detailed training hyperparameters for all domains and UED methods are found in Appendix
B.
Results
6.1 Minigrid
illustrates the performance of MNA and DEGen compared existing baselines. From these plots, it i8 clear that Intial Gen performs substantially worse than all other methods, whereas DEGen performs comparably to existing replay-based methods. This substantial performance deficit can likely be attributed to lack of diversity in the generated levels, see Appendix D.5] We see a far greater diversity in the levels generated via DEGen, and this, along with the reduced credit assignment challenge, allows for DEGen to substantially outperform Initial Gen, despite the former also relying on an RL-trained teacher. We also see that for both PLR and ACCEL, MNA outperforms the existing MaxMC and PVL regret approximations. However, Figure 3 also shows that in this setting, early in training, DEGen is outperformed by replay-based methods using MNA. This suggests that, whilst the final DEGen performance is equal or greater than the performance of sapla-bbsse methods, the additional challenge of learning the generator does impact initial performance.
Figure
DEGen MNA
PLR MNA
PLR PVL
PLR MaxMC
DR
ACCEL
0.6
0.5
Mean Solve Rate
Return
0.2
0.1
0.0
1.0
2.0
3.0
0.0
1.0
2.0
2.5
3.0
PPO Update Step
1e4
PPO Update Step
1e4
(a) Mean Return
(h) Mean Solve Rate
Figure 3: Minigrid zero-shot performance on hand-designed test set, showing mean and standard error across 8 runs.
6.2 Key Minigrid
see that [igureeccompnn the performance of the various tested UED methods in the key minigrid domain. Due to the increased challenge of level generation with the addition of key and locked door, there is far more variance in the relative performance of each of the methods. In this domain, we DEGen outperform all existing baselines, as well as the MNA-based replay methods. PLR using either MaxMC or PVL performs extremely poorly, whereas PLR using MNA. which is much more capable identifying challenging levels, the best performing baseline. This highlights the improved regret approximation of MNA compared to existing metrics. We also note that SFL performs poorly in this minigiid setting. Learnability only considers the final outcome of an episode, rather than the full student trajectory, and this result suggests that this may be insufficient in domains where additional environment features result more complex subset levels.
6.3 Increased Environment Size for Key Minigrid
Up to this point, we have examined only the relatively small 13x13 13 environment setting. However, as outlined above, with the aim of scaling UED larger, more complex environments, examine the

---

DEGen
MaxMC
ACCEL MNA
ACCEL FVL
ACCEL MOXME
SRL
Initial Gen MNA
0.7
1.0
0.8
Mean Solve Rate
0.6
0.4
Moan
0.4
0.2
0.0
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Step
Update Step
1e4
(a) Mean Return
(b) Mean Solve
Figure 4: Minigrid with key and locked door zero-shot performance on hand-designed test set, trained on 13x13 training levels, showing mean and standard error across runs.
DEGen MNA
ACCEL MNA
PLR MNA
0.8
0.B
Mean Solve Rate
0.6
0.6
Salve
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Update Step
164
PPO Update Step
(a) 17x17
(b) 21x21
Figure Minigrid with key and locked door zero-shot performance on hand-designed set, trained on larger training levels, showing mean and standard error across 8 runs.
performance of current UED methods when simply scaling up a version of minigrid sizes larger than 13x13. Figure Shoows the performance of DEGen, PLR and ACCEL using MNA when training levels are 17 and 21x21. Whilst there performance drop compared the 13x13 levels, as the size of the environment grows, DEGen substantially outperforms existing methods. This result highlights that, with any domains with additional environment features such as the key and door that can potentially add additional complexity to policy learning, as the environment size grows, it becomes substantially harder to sample useful levels where these features are necessary. Therefore it is necessary to use trained generator and DEGen is able to overcome the credit assignment challenges present in previous methods for training learnt generators.
shows
7 Discussion
clear that from these results that DEGen does substantially outperform existing baselines both the minigrid, and key minigrid environments, and this performance improvement is increased the

---

environment size increases. Scaling environment size is likely necessity if future work aims to move UED from the current set toy small-scale environments used, more useful real-world domains, and this work shows that replay-based methods perform poorly with relatively small increases to environment size. We have demonstrated that DRGen is an effective method of addressing these issues, both compared to generating the full level prior to student rollouts, and replay-based methods.
8 Limitations
Whilst the results we have presented in this paper do show strong performance from DEGen compared existing baselines, the domains presented in this paper present relatively simple mappings between the level representation and the agent's 's current observation. In more complex domains, such as
3D Games [22 13 20). real-world robotics applications 39 63 21. it will be more complex
to determine how specific environment parameters affect what the agent is currently observing. In order for UED bridge the gap from the current of game domains to real-world applications, would be necessary for DEGen or DEGen-like methods to address this limitation. We believe that World Models [9 represent a promising direction for future research to address this. In its current form, DEGen relies on both the agent and the generator interacting with a fixed environment, where the environment has an explicit mapping between the agent's 's observation and the level parameters. and the generator is only able to generate the level where the agent has observed. However, world model guided by a regret approximation such as MNA would represent generator that could directly generate observations for the agent. Rather than relying on explicit mapping between level and observation, this mapping could be learnt with environment data when training the world model. Whilst the training of a world model would add additional computational cost to the training process, would enable DEGen methodology to be applied to substantially more complex environments. With the advent of highly general world models such as the Genie series of world models [6] this could represent a path to training highly general policies that are effective in a wide variety of applications.
Additionally, in line with previous work [25] 40 8 we have compared the relative performance of UED methods based on the number of student PPO update steps. However, training the DEGen teacher agent adds high computational cost to the training loop, and so training using DEGen takes approximately four times long as training using methods such as PLR and ACCEL. Full details on compute time and experiment specification can be found in Appendix B Therefore, replay- based methods may be preferable for small environment sizes where similar performance achieved. However again, it is clear than as environment size grows, the maximum performance achieved by DEGen exceeds replay-based methods, and so the additional time cost justified, given the performance gains.
9 Conclusion
In this paper, we introduce a new level generation method, Dynamic Environment Generation for UED, and new regret approximation metric, Maximised Negative Advantage. We outline how current UED methods fail as training environment size increases, and show that DEGen capable of mitigating the issues associated both with these larger environments, and with RL-based level generation. We show that the use of MNA enables DEGen to outperform existing baselines, whilst also showing that the of MNA consistently improves the performance of existing UED methods. These performance improvements are most evident in the more complex key minigird domain. We believe there is sigmificant potential for future UED research address larger and more complex environments, and that approaches based on MNA and DEGen provide promising foundation for this advancement.
Acknowledgements
This work was supported by the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems [EP/S024050/1], Lacerda and Hawes have received EPSRC funding via the "From Sensing Collaboration" programme grant [EP/V000748/1].
10

---

References
[1] lige Akkaya, Marcin Andrychowicz, Macick Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, al. Solving rubik's cube with robot hand. arXiv preprint 1910001100.0/13. 2019.
[2] OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert. Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Roboticx Research, 39)) 2020.
Abdus Salam Azad, Izzeddin Gur, Jasper Emhoff, Nathaniel Alexis, Aleksandra Faust, Picter Abheel, and Ion Stoica. Clutr: Curriculum learning via unsupervised task representation learning. In International Conference on Machine Learning, pages 1361-1395. PMLR, 2023.
[4] Dåmaris S Bento, André G Pereira, and Levi HS Lelis. Procedural generation of initial states of sokoban. arXiv preprint arXiv:1907. 1et.i:9977005,.. 2019.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota with large scale deep reinforcement learning. arXiv preprini 1912 200000000 2019.
[6] Jake Bruce, Michael D Dennis. Ashley Edwards, Jack Parker- Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first Interaatinaal Conference on Machine Learning, 2024.
[7] CRFG Campos, Walter Oliveira Sá, Joao Gusmao Teixeira, and Lelis. Mixed-initiative tool to speed up content creation in physics-based games. Proceedings of SBGames, 2017:590-593, 2017.
Hojun Chung, Junseo Lee, Minsoo Kim, Dohycong Kim, and Songhwai Oh. Adversarial environment design via regret-guided diffusion models. Advances Neural Information Processing Systems, 37:63715- 63746, 2024.
[9] Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generalization reinforcement learning. In International conference on muchine learning, pages 1282-1289. PMLR, 2019.
[10] Samuel Coward, Michael Beukman, and Jakob Foerster. Jaxued: simple and useable ued library in jax. arXiv preprint arXiv:2403.1309) 20244
[11] Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 33:1 3049-13061, 2020.
[12] Jonathon Doran and Ian Parberry. A prototype quest generator based on a structural amalysis of quests from four mmorpgs. In Proceedings of the 2nd international workshop on procedural content generation in games, pages 1-8, 2011.
[13] Sam Earle, Maria Edwards, Ahmed Khalifa, Philip Bontrager, and Julian Togelius. Learning controllable content generators. In 2021 IEEE Conference on Games (CoG), pages 1-9, IEFE, 2021.
[14] Kuan Fang, Yuke Zhu, Silvio Savarese, and Fei-Fei. Adaptive procedural task generation for hard- exploration problems. International Conference on Learning Representations, 2021.
[15] Raihana Ferdous, Fitsum Kifetew, Davide Prandi, and Angelo Susi. Towards agent-based testing of 3d games using reinforcement learning. Proceedings of the 37th IEEE/ACM international conference on audomated software engineering, pages 1-8, 2022.
[16] Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for rein- forcement learning agents. International conference on machine learning, pages 1515-1528, PMLR, 2018.
[17] Thomas Foster, Anya Sims, Johannes Forkel, Mattie Fellows, and Jakob Foerster. Learning to reason at the frontier of learnability, arXiv preprint arXiv:2502. 120..1222, 2025.
[18] Ahmet Güzel, Matthew Thomas Jackson, Jarek Luca Liesen, Tim Rocktäschel. Jakob Nicolaus Foerster, Iliaa Bogunovic, and Jack Parker-Holder. Imagined autocurricula. arXiv preprint arXiv:2509. 13341, 2025.
[19] David Ha and Jürgen Schmidhuber. World models. arXiv preprint 1rXiv:1803.10122 2(3). 2018.
[20] Danijar Hafner, Wilson Yan, and Timothy Lillicrap. Training agents inside of scalable world models. arXiv preprint 2009 24527, 2025.

---

[21] Dong Han, Beni Mulyana, Vladimir Stankovic, and Samuel Cheng. A survey on deep reinforcement learning algorithms for robotic manipulation. Sensors, 23(7):3762, 2023.
[22] Jack Harmer, Linus Gisslén, Jorge del Val, Henrik Holst, Joakim Bergdahl, Tom Olsson, Kristoffer Sjōö, and Magnus Nordin. Imitation learning with concurrent actions in 3d games. In 2018 IEEE Conference on Computational Intelligence and Games (CIG), pages 1-8. IEEE, 2018.
[23] Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309. 17080, 2023.
[24] Matthew T Jackson, Minqi Jiang, Jack Parker-Holder Risto Vuorio, Chris Lu, Greg Farquhar, Shimon Whiteson, and Jakob Foerster. Discovering general reinforcement learning algorithms with adversarial environment design. Advances in Neural Information Processing Systems, 36:79980-79998, 2023.
[25] Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob Foerster, Edward Grefenstette, and Tim Rock- täschel. Replay-guided adversarial environment design. Advances in Neural Information Processing Systems, 14418844-897) 2021.
[26] Minqi Jiang, Michael Dennis, Jack Parker-Holder, Andrei Lupu, Heinrich Küttler, Edward Grefenstette. Tim Rocktäschel, and Jakob Foerster. Grounding aleatoric uncertainty for unsupervised environment design. Advances in Neural Information Processing Systems, 35:32868-3253 2022.
[27] Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference on Machine Learning, pages 4940-494 4950. PMLR, 2021.
[28] Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 920-7976,,922-9877 2023.
[29] Ahmed Khalifa, Philip Bontrager, Sam Earle, and Julian Togelius. Pegrl:
Procedural content generation via reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 16, pages 95-101, 2020.
[30] Ahmed Khalifa and Magda Fayek. Automatic puzzle level generation: A general approach using a description language. Institute of Electrical and Electronics Engineers, 2015.
[31] Paul Hyunjin Kim, Jacob Grove, Skylar Wurster, and Roger Crawfis. Design-centric maze generation. In Proceedings of the 14th International Conference the Foundations of Digital Games, pages 1-9, 2019.
[32] Antonios Liapis. 10 years of the peg workshop: Past and future trends. Proceedings of the 15th International Conference on the Foundations of Digital Games, pages 1-10, 2020.
[33] Antonios Liapis, Gillian Smith, and Noor Shaker. Mixed-initiative content creation. In Procedural content generation in games, pages 195-214. Springer, 2016.
[34] Tambet Matüieen, Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum learning. IEEE transactions on neural networks and learning systems, 31(9):3732-3740, 2019.
[35] Ishita Mediratta, Minqi Jiang. Jack MarkerHolded Michael Dennis, Eugene Vinitsky, and Tim Rocktäschel. Stabilizing unsupervised environment design with a learned adversary. In Conference on Lifelong Learning Agents, pages 270-291. PMIRR, 2023.
[36] Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world models. In International Conference on Learning Representations, 2023.
[37] Nathan Monette, Alistair Letcher, Michael Beukman, Matthew Thomas Jackson, Alexander Rutherford, Alexander David Goldie, and Jakob Nicolaus Foerster. An optimisation framework for unsupervised environment design. In Reinforcement Learning Conference, 2025.
[38] Sanmit Narvekar, Bci Peng, Mattco Lconetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, (1181)11500 2020.
[39] Hai Nguyen and Hung La. Review of deep reinforcement learning for robot manipulation. In 2019 Third IEEE international conference robotic computing (IRC), pages 590-595. IERR, 2019.
[40] Jack Holder, Minqi Jiang. Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefen- stette, and Tim Rocktäschel. Evolving curricula with regret- regret-based environment design. In International Conference on Machine Learning, pages 17473 17498. PMLR, 2022.
12

---

[41] Xue Bin Peng, Marcin Andrychowiez, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and cutomation (ICRA), pages 3803-3810. IEEE, 2018.
[42] Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yees Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Conference on Robot Learning, pages PMLR. 2020.
[43] Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre- Yves Oudeyer. Automatic curriculum learning for deep rl: A short survey. arXiv preprint urXiv:2003.0400 044664 2020.
[44] Edward Powley, Swen Gaudl, Simon Colton, Mark Nelson, Rob Saunders, and Michael Cook. Automated tweaking of levels for casual creation of mobile games. Computational Creativity and Games Workshop, 2017.
[45] Mare Rigter, Minqi Jiang, and Ingmar Posner. Reward-free curricula for training robust world models. In International Conference on Learning Representations, 2024.
[46] Sebastian Risi and Julian Togelius. Increasing generality in machine learning through procedural content generation. Nature Machine Intelligence, 2(8):428 436, 2020.
[47] Jan Robine, Mare Höftmann, Tobias Uelwer, and Stefan Harmeling. Transformer- based world models are happy with 100k interactions. In International Conference on Learning Representations. 2023.
[48] Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pages 91-100. PMLR, 2022.
[49] Alexander Rutherford, Michael Beukman, Timon Willi, Bruno Lacerda, Nick Hawes, and Jakoh Foerster. No regrets: Investigating and improving regret approximations for curriculum discovery. Advances Neural Information Processing Systems, 37: 37:16071- 2024.
[50] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbecl. High-dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations, 2024.
[51] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint
2017.
[52] David Silver. Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panncershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
[53] David Silver, Thomas Hubert, Julian Schritwweseer, loannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel. al. Mastering chess and shogi by self play with a general reinforcement learning algorithm. arXiv preprint 1rXiv:1712.01815. 2017.
[54] Ruben Smelik, Tim Tutenel, Klaas Jan De Kraker, and Rafael Bidarra. Integrating procedural generation and manual editing of virtual worlds. In Proceedings of the 2010 Workshop on Procedural Content Generation in Games, pages 1 1-8, 2010.
[55] Josh Tohin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
[56] Georgios Tzannetos, Parameswaran Kamalaruban, and Adish Singla. Proximal curriculum with task correlations for deep reinforcement learning, arXiv preprint 240.0444,, 2024.
[57] Georgios Tzannetos, Bárbara Gomes Ribeiro, Parameswaran Kamalaruban, and Adish Singla. Proximal curriculum for reinforcement learning agents. arXiv preprint 22X::22344412877 2023.
[58] Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft using multi multi-agent ageett reinforcement learning. nature, 575(7782):350-33 354, 2019.
[59] Rui Wang. Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint a01iv.190..11533, 2019.
13

---

[60] Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li. Jeffrey Clune, and Kenneth Stanley. Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In International conference on machine learning, pages 9940-9951. PMLR, 2020.
[61] Mengjiao Yang. Yilun Du, Kamyar Ghasemipour. Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. arXiv preprint arXiv:2310.06114 1(2)6. 2023.
[62] Amy Zhang. Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint 1tff::180000777 2018.
[63] Kai Zhu and Tao Zhang. Deep reinforcement learnnng based mobile robot navigation: A review. Tsinghua Science and Technology, 26(5):674 691, 2021.
14

---

A
Related Work
This research presents a new method for Unsupervised Environment Design (UED) [1] area of research locused on improving the generalisation of policies trained using reinforcement learning and enabling open- ended learning. Domain Randomisation (DR) can be categorised as the most basic form of UED, where environment parameters are simply randomised naively for training. DR has shown to be effective in applications such transferring policies from simulations to real world robotics deployments [55] 41 2). Prioritised Level Replay (PLR) 2 [5] augments domain randomisation by maintaining replay buffer of levels that have been scored as effective for training. PLR has been employed in diverse of domains, such
training World Models 4S or meta-reinforcemment learning 24 Additional work has examined dealing with distribution shift between training and deployment (26). Whilst PLR relies on replaying levels from a fixed random generator, other work has focused on generating new levels through evolving prior levels [40] POET (59 60 examines co-evolving both the environment levels, but also population of agents playing those levels. More similarly to g.
DRGen, other existing work focuses explicitly learning level generator
PAIRED introduced the formalisation for URD, framing the level design problem as minimax regret game between student and teacher. PAIRED relies on the performance difference between a protagonist and antagonist agent to score level suitability, and uses these scores to train level generator: CLUTRR R improves on PAIRED by learning latent representation of levels to reduce the challenge of learning the generator. ADD S has used diffusion model generate levels instead. More recent work has examined using non-regret based
scoring of levels [57 56 such as learnability [49 37
where ACL aims to provide an automatic curriculum to enable learning of increasingly challenging tasks. Unlike UED, ACL often relies on specific knowledge of the target task [14 34
UED can be seen a form of Automatic Curriculum Learning (ACL) 16],
This work also relates to the field of Procedural Content Generation (PCG)
32 46
PCG has focused on level
design for games, such as with terrain generation S4 task design 1
31 or puzzle setting 33 4. Much of
this work has focused level design for human play, and some of this work relies specific user input for level generation 33 7 44. Methods using RL for training level generators 22
require hand-designed.
environment specific generation rewards, which differs from the Nomain-agnostee level scoring used in UED.
Unlike previous UED work, DEGen relies on a teacher that generates the environment based on the student's current observations. This shares similarities with existing work in World Models [19]. Whilst the DEGen teacher is able control the observations of the student, the environment is designed explicitly disallow infeasible observations be generated. World Models are instead trained example trajectories, and the fcasibility of generated observations implicitly learnt. These learnt models are capable of training agents without access to the real environment [47 (36] 20). Large open-ended world models [6] 23 [ may potentially
enable the training of highly generally capable agents. Recent work has examined applying UED methods. specifically PLR, to training agents in world models [1]]
15

---

B Detailed Experimental Setup
B.1 Environment Details
B.1.1
Minigrid
We the standard minigrid implementation from exisiting UED work I|| are shown in Figure g
25
40
49
Examples of levels
Observations: The agent, depicted in red, observes 5x5 square front access absolute direction, e.g. whether it facing North, South, East or West.
Environment State
(b) Agent Observation
Figure 6: Minigrid Agent Observations
The agent receives reward rg it reaches the goal. The size of this reward is determined by the number steps taken to reach the goal T, and the parameter T max. specifying the maximum number of steps before an episode terminates.
Rewards:
T
Ty
0.9
(14)
Tiaax
Actions: At each step, the agent can either move forward, turn left or turn right.
B.1.2 Key Minigrid
The key minigrid implementation is identical to the standard minigrid level, except for the the addition a key and locked door.
Observations: The agent is additionally able to observe whether it has collected the key.
Rewards:
The reward remains identical to the standard minigrid, with reward only being received on reaching
the goal. Note that there are no specific rewards associated with collecting the key unlocking the door.
Actions: The agent has additional use action. The agent will pick up the key if it reaches the grid square the key is In order to unlack the door, the agent must select the action when has the key and the door directly ahead agent.
(a) Locked Door
(b) use action
(c) Unlocked Door
Figure 7: Key Minigrid Door Unlocking
16

---

B.1.3 DEGen Teacher
Observations: The teacher's observations are an augmented version of the student agent. Similarly to the student, the teacher observes 5x5 square in front of the student. However, this is augmented by an overlayed 5x5 square that indicates which squares have yet to be generated.
(a) Environment State
(b) Teacher Observation
Figure DEGen Teacher Observations
Actions: At each step, the teacher is able to fill any ungenerated grid square the student is currently observing. The teacher action a is composed of two sub acttinnions (a1, 42) where as selects which cell will be filled, and d2 selects what the cell will be filled with, e-g. wall, empty, goal, key, door. We action masking to ensure that only previously ungenerated cells filled, as well enforce that only goal, key and door can be placed for each level.
The teacher rewards are determined by the chosen regret approximation. These dense rewards are calculated using Equation 6
Rewards:
Dynamics: As the environment is rolled out, at each step, cither the teacher or the student will act. If there are ungenerated cells in the agent's s observation, the teacher must fill all ungenerated cells currently observed before the student is able to take another action.
As both the student and teacher have partial observability, both policy networks include LSTM layer, actions are conditioned on previous observations. teacher conditioned all previous observations. whereas the student is conditioned only on the fully-generated observations.
KL Regularisation
work 35 has shown that entropy regularisation necessary when using reinforcement learning to train teacher for UED. The policy gradient loss is augmented with an entropy loss
Lentropy
Bent
-H (111s))
(15)
Entropy regularisation biases the policy towards more uniform distribution of action probabilities. However, as there can only be maximum of single goul, key and door per level, the distribution of cell objects will be the vast majority cells will be either wall empty. Therefore rather than including an
highly non-uniform, entropy loss for the az sub-action, we instead add regularisation with fixed categorical distribution ((22). detined with the parameter Pg. such that
3p
((2)) Cat(a))
a (pr Prey Pas Pos po).
Pw
(16)
Lemtopu + KL ESNT H Fai (116) + DEe ( (1)((1))))))))
(17)
For all experiments, used Py 0.01.
17

---

B.2 Hyperparameters
Full code
and instructions
how to run
be
found
httpa: bttta://github..
All
existing
methods were
trained
using implementations based
on
JaxUED
available
at
https://github cow com/Dramacow/jjaxuell and
SFL
49
available at
shown in Table wwwpa//atteerrr//eaaaaaaoeaemeee
for-learnabilty Learning hyperparameters are and the replay UED hyperparameters shown in T
com/
Table I: Learning Hyperparameters.
Parameter
Minigrid
Key Minigrid 13x13
17x17 and 21x21
Student PPO
Number of Updates
30000
15000
7
0.995
AGAR
0.95
PPO number of steps
512
PPO epochs
4
PPO minibatches per epoch
4
PPO clip range
0.04
PPO parallel environments
256
Adam learning rate
5e-4
2.4e-4
Anneal LR
yes
no
Adam
le-5
PPO max gradient norm
0.5
PPO value clipping
yes
value loss coefficient
0.5
entropy coefficient
le-3
Hidden dimension size
256
Teacher PPO
Y
0.998
0.99
AGAE
0.95
PPO epochs
4
PPO minibatches per epoch
4
PPO clip range
0.2
Adam learning rate
le-3
Anneal LR
yes
no
Adam
le-5
PPO max gradient norm
0.5
PPO value clipping
yes
value loss coefficient
0.5
entropy coefficient
5e-2
Hidden dimension size
256
Num Teacher Steps (Initial Gen)
60
18

---

Table 2: UED Hyperparameters.
Parameter
Minigrid
Key Minigrid
PLR
Replay rate, p
0.5
Buffer size, K
8000
Prioritisation
Rank
Temperature, 3 1.0
staleness coefficient
0.3
ACCEL
Number of Edits
20
Buffer size, K
8000
Prioritisation
Rank
Temperature, 3
1.0
SFL Batch Size
25000
Rollout Length L
20000
Update Period T
100
Buffer Size K
1000
Sample Ratio
0.5
B.3
Compute Details
For all experiments, each run was on Nvidia A40. We show the mean compute time for both domains and cach UED method in Table
Table 3: Compute Time.
Compute Time (hh:mm)
Method
Minigrid
Key Minigrid
DR
11:16 + 00:02
12:01 = 00:02
Initial Gen
13:39 00:01
13:49 = 00:02
PAIRED
00:02
23:33 = 00:01
SFL
10:53 = 00:01
10:48 # 00:02
PLR
06:53 = 00:02
06:53 # 00:01
ACCEL
05:43 = 00:02
05:45 = 00:01
DEGen
25:49 = 00:00
25:53 = 00:01
19

---

B.4 Zero-shot Transfer Levels
Figures and TO show the hand designed levels used for evaluating zero-shot performance. The minigrid levels taken directly from JED [TO]. The key minigrid levels have been modified that the student required unlock door to reach goal. Note that have chosen include FourRooms Key levels rather than modified versions of the labyrinth levels, as the key would trivially be on the path to the goal for these labyrinth levels.
SixteeniRooms
RioteenRooms2
Labyrinth
LabyrinthFlipped
Labyrinth2
StandardiMaze
StandardMaze2
StandaroMaze3
C
Figure 9: Hand designed evaluation levels for minigird
SixteenRooms Key
SikteerRcams2 Key
FourRoams, Key
FourRaems2 Key
PH
FourRooms 3. Key
StandardM aze Key
StandardlMaze2 Key
StancardMaze3 Key
E
D
F
1
Figure 10: Hand designed evaluation levels for key minigird
20

---

B.5 Tabular Results
Minigrid Results
Table 4: Minigrid Solve Rate (1)
Level
Initial Gen MNA
DR
SFL
PLR MaxMC
ACCEL MaxMC
SixteenRooms
0.93 = 0.06
1.00 = 0.00
1.00 0.00
1.00 = 0.00
1.00 + 0.00
SixteenRooms2
0.48 = 0.14
1.00 = 0.00
1.00 =.00
1.00 # 0.00
1.00 + 0.00
Labyrinth
0.04 + 0.04
0.51 0.16
0.15 0.10
0.69
Labyrinthliippd
0.11 + 0.10
0.15
0.13
0.14
0.13
Labyrinth2
0.05 + 0.04
0.10
0.12
0.75
0.81
Stan
0.28 + 0.11
0.09
1.00
0.98
0.00
Maazd
0.30 # 0.10
0.06
0.05
0.12
1.00
StandardMaze3
0.56
0
1.00
0.98
1.00 0.00
Mean
0.34 # 0.05
0.76 + 0.05
0.88 + 0.04
0.83 # 0.03
0.86 0.02
Table 5: Minigrid Solve Rate (2)
Level
PLR PVL
ACCEL PVL
PLR MNA
ACCEL MNA
DEGen MNA
SoxteenRooms
1.00 0.00
1.00 0.00
1.00 0.00
1.00 = 0.00
1.00 = 0.00
SixteenRooms2
0.95 + 0.04
0.88 # 0.05
1.00 2 0.00
1.00 + 0.00
1.00 = 0.00
Labyrinth
0.71 = 0.09
0.86 0.04
0.88 0.06
0.68 + 0.09
0.98 = 0.02
LabyrinthFlipped
0.63 + 0.11
0.07
0.73 0.11
0.91 = 0.09
Labyrinth2
0.51 0.07
0.08
0.06
0.95 0.02
0.79 779
StandardMaze
0.63 0.09
0.06
0.98
1.00 0.00
1.00 = 0.00
StandardMaze22
0.10
0.88
0.06
0.99
1.00 = 0.00
StandardMaze3
0.91 # 0.05
0.99
1.00 0.00
0
1.00 = 0.00
Mean
0.75 0.03
0.87 0.87 0.03
0.92 0.02
0.91 1 0.01
0.96 = 0.03
21

---

Key Minigrid 13x13 Results
Table 6: Key Minigrid 13x13 Solve Rate (1)
Level
Initial Gen MNA
DR
SFL
PLR MaxMC
ACCEL MaxMC
SixteenRooms_Ke Key
0.18 # 0.12
0.59 0.12
0.58 + 0.12
0.41 # 0.11
0,83 # 0.04
SixteenRooms2_KK Key
0.12 # 0.12
0.13
0
0.19 0.05
0.83 # 0.08
YourRooms_Ker
0.03 # 0.02
0.98 0.13
0.10 0.08
0.96 # 0.02
FourRooms2_ Key
0.05 # 0.04
0.86
0.79 0.07
0.98 0.02
3ourRooms3 Key
0.29 = 0.14
0.11
0.06
0.06
0.95 + 0.05
StandardMaze Key
0.01 0.01
0.09
0.12
0.07
0.10
StandardMaze2 Key
0.00 = 0.00
0.11
0.14
0.11
0.26 0.07
StandardMaze3 Key
0.00 0.00
0.36
0.30 0.14
0.09 0.06
0.75 + 0.05
Mean
0.08 + 0.04
0.52
0.68 + 0.04
0.29 = 0.03
0.72 + 0.02
Table 7: Key Minigrid 13x13 Solve Rate (2)
Level
PLR PVL
ACCEL PVL
PLR MNA
ACCEL MNA
DEGen MNA
SixteenRooms Key
0.15 + 0.10
0,55 + 0.04
0.03
0.99 0.01
1.00 + 0.00
SoxtcenRooms2 Key
0.00 + 0.00
0.10
0.06
0.88 0.10
1.00 0.00
FourRooms Key
0.00
0.60 0.12
0.04
0.02
0.94
FourRooms2 Key
0.06 0.00
0.06
1.00
0.99
1.00
FourRooms3_nn Key
0.00 +
0.09
0.05
0.02
1.00
StandardMaae Key
0.000
0.00
0.16
0.85
0.03
StandardMaze2 Key
0.00
0.02
0.13
0.75
0.63
StandardMazze__Ke Key
+ 0
0.30 0.09
0.78
0.89
0.94 0.05
Mean
0.03 # 0.01
0.45 + 0.01
0.84 0.01
0.90 + 0.03
0.93 + 0.02
Key Minigrid 17x17 Results
Table Key Minigrid 17x17 Solve Rate
Level
PLR MNA
ACCEL MNA
DEGen MNA
SixteenRooms. Key
0.99 0.01
0.86 + 0.05
1.00 0.00
SoxteenRooms22 Key
0.86 0.03
0.40 # 0.13
0.95 0.05
FourRooms Key
0.80 + 0.11
0.64
0.94 0.05
FourRooms2 Key
0.92 0.05
0.10
0.95 0.04
FourRooms3_KK Key
0.88 + 0.07
0.09
0.94 0.04
StandardMaze Key
0.69 + 0.13
0.19
0.23 0.05
StandardMaaze2_KK Key
0.29 + 0.09
0.21 + 0.07
0.50 2 0.07
StandardMaze33_Key
0.73 0.09
0.83 0.06
0.73 0.09
Mean
0.77 0.04
0.57 + 0.03
0.78 0.03
22

---

Key Minigrid 21x21 Results
Table 9: Key Minigrid 21x21 Solve Rate
Level
PLR MNA
ACCEL MNA
DEGen MNA
SixteenRooms_KKe Kee
0.60 + 0.11
0.60 = 0.12
1.00 = 0.00
SixteenRooms2_ Key
0.40 0.09
0.45 0.11
0.99 0.01
FourRooms jeey
0.41 0.10
0.99 = 0.01
FourRooms2_KK Key
0.93 0.04
0.59
0.93 0.05
FourRooms3 Key
0.35 0.12
0.09
0.98 + 0.02
StandardMaze Key
0.11
0.03
0.56 0.11
StandardMaze2 Key
0.26
0.00
0.39 + 0.12
StandardMaze3 Key
0.38 0.11
0.16
0.61 0.11
Mean
0.43 + 0.05
0.31 + 0.04
0.80 0.03
23

---

C Sokoban Environment
We have additionally performed experiments in Sokoban-style environment. As in standard Sokoban, the agent aims to get all boxes to their storage locations. The agent receives sparse reward when completing level inversly proportional how many steps were required complete the level. For this domain, similarly minigrid the agent has 5x5 observation space ahcad the agent, and an action space of move forward, turn left or turn right. The agent also has a reser action that resets the agent and boxes to their starting locations.
We used 9x9 levels for training for DR, PLR and SFL, the random level generator generated levels that had 15 walls, and between and 10 boxes. For DEGen, the generator could fill each newly observed cell in the environment with emply square/wallbeo//ttragg location/box storage location. For all methods, used identical hyperparameters the minigrid environment (See Table
C.1 Sokoban Results
From the results shown in Figure 11 and Tables 10 and various UED methods. We see that ACCEL using MaxMC the best performing method, although comparable performance achieved using DEGen. Sokoban, we see that MaxMC outperforms MNA methods, with a substantial performance difference when used with ACCEL. Sokoban presents unique challenges compared to the previous environments used in this work. Primarily, the majority of randomly-sampled levels tend to be impossible, given that only takes one box storage location be unreachable make the entire level unsolvable. However, is also likely a high proportion the solvable seddomly-sampllee levels will be difficult.
large range in the performance of the
replay-based
As all impossible levels will necessarily score zero with both MaxMC and MNA. We hypothesise that MaxMC may he a better metric than MNA domains where difficult levels represent a higher proportion of the non-zero scoring randomly-sampled levels. The high performance of ACCEL in Sokoban is likely due to the clear difficulty scaling that be achieved with ACCEL-like level evolution. In our implementation of ACCEL, one possible mutations is to add or remove a box/storage-locall pair. This enables gradual difficulty evolution in Sokoban, which less likely in the minigrid enviroments. As such, ACCEL is highly effective at generating curriculum.
DEGen
PLR MNA
DR
ACCEL
PVL
MaxMC
0.40
0.6
0.35
0.5
0.30
Return
0:25
Mean Solve
0.20
0.3
0.15
0.2
0.00
1.0
2.0
2.5
3.0
PPO Update Step
PPO Update Step
104
(a) Mean Return
(b) Mean Solve Rate
Figure 11: Sokoban zero-shot performance on hand-designed test set, showing mean and standard еπrоr across 8 runs.
We do however see in Tables 10 and and 11 that a number levels, those marked in italics, are not solved by any method. This suggests that there is room for future work to enable zero-shot performance on more difficult levels, and that Sokoban may be an interesting environment for future UED research.
24

---

C.2 Sokoban Zero-shot Transfer Levels
For the zero-shot transfer set, we have used the first 20 Sokoban Jr levels that do not exceed 13x13 in size.
Sokoban _r___l
SokobanJra Jr_1_02
Sokoban_ _r___a_________ Sokoban_Jr_
Sokoban Jr_1_os
Sokoban Jr_1_06
Sokoban Jr_1_11111 1_07
Sokoban Sa_e_rr_1__00 1_08
Sokoban 1r____0
Sokoban $_1_12
20
Sokoban 1r_1_1
Sokoban Laa_s__11111 1_16
Sokoban Jr_1__7
Sokoban_ir.
Sokoban _r_11191111
Sokoban_J_2 _r___20
Sokoban Jr___11
Sokoban Jr Sokoban_J__1. v_1_23
Sokoban
Figure 12: Hand designed evaluation levels for sokoban
25

---

C.3
Sokoban Tabular Results
Table 10: Sokoban Solve Rate (1)
Level
DR
SFL
PLR MaxMC
ACCEL MaxMC
Sokoban ________________________________ 1_01
0.00 + 0.00
0.23 0.15
1.00 0.00
1,00 = 0.00
Sokoban_ik Baaian_J____ Jr_1____0___00 1_02
0.00 + 0.00
0.18 # 0.12
0.94
0.98 + 0.02
Sokoban 1..._100
0.00
0.00
0.46
0.83
Sokoban Jr 1r_4
0.00
0.00
0.16
0.30
Sokoban_Jr___00
0.00
0.00
0.42
0.45
Sokoban_Jra _ 1r6
0.00
0.00
0.14
0.75
Sokoban ____________ Ir_1_77
0.00
0.00
0.10
0.96
Sokoban Jr 1_08
0.00
0.00
0.21
0.50
Sokoban Jr_I_00 1_9
0.00
0.00
0.65
=
Sokoban I_
0.00
0.00
0.00
0.29
Sokoban Jr 15
0.00
0.00
0.00
0.00 = 0.00
Sokoban Jr 16
0.00
0.00
0.03
0.96
Sokoban Jr 17
0.00
0.00
0.000
0.00 + 0.00
Mikoban_Jr_1__
0.00
0.00
0.04
0.16
_ra__in_Jr___99
0.00
0.00
0.00
0.13
Lokoban_Jr_1_2
0.00
0.000
0.00
0.00 0.00
Sokoban_Jr_1_22
0.000
0.00
0.06
1.00 = 0.00
Sokoban 1__1_2
0.00
0.00
0.14
0.64 0.10
Sokoban Jr_1_23
0.00 = 0.00
0.00
0.00 0.00
0.00 + 0.00
Sokoban_Jr.
122
0.00
0.00
0.003
0.14 0.05
Mean
0.00
0.02 = 0.01
0.36 + 0.01
0.51 = 0.02
Table 11: Sokoban Solve Rate (2)
Level
PLR PVI.
ACCEL PVL.
PLR MNA
ACCEL MNA
DEGen MNA
Sokoban_Irz 1_00
1.00 + 0.00
0.99 = 0.01
0.98 + 0.02
0.89 0.11
1.00 + 0.00
Sokoban Jr 1_22
0.24 # 0.05
0.46 0.06
0.98 0.02
0.49
1.00 # 0.00
Sokoban Jr_1_03
0.07
0.11
0.15
0.49 # 0.14
Sokoban Jr 1r_1_G0 04
0.00 0.00
0.04
0.14
0.35 0.09
Sokoban_Jr Jr__J___00000 1_05
0.10
0.01
0.12
0.18
8.24
Sokoban Jr
06
0.03
0.03
0.10
0.46
0.94 0.03
Sokoban Jr 1 07
0.19
0.51 0.116
0.09
0.19
1.00 0.00
Sokoban 1 08
0.00
0.00
0.15
0.00
0.50 0.15
Sokoban Jr
09
0.14
0.13
0.06
0.18
0.85
Sokoban 1 12
0.00
0.00
0.00
0.00
0.30
Sokoban Jr_1_11
0.00
0.00
0.00
0.00
0.00 0.00
Sokoban 1
0.25
0.06
0.03
0.60
0.95
Sokoban |
0.00
0.00 0.00
0.00
0.00
0.00 + 0.00
Sokoban 1_1
0.00
0.03
0.00
0.00
0.69
Sokoban 19
0.00
0.00
0.00
0.00
0.10 0.06
Sokoban 1arobban_Jr__200
0.00
0.00
0.00
0.00
0.00
Sokoban Jr 21
0.59
0.56
0.76
0.68
0.99
Sokoban 12
0.06 0.00
0.09
0.00
0.10
Sokoban Jr 23
0.00 + 0.00
0.00 0.00
0.00
0.00
0.00
Sokoban 24
0.00
0.01 0.01
0.04
0.00
0.09 0.03
Mean
0.13
0.16
0.36
0.21 + 0.02
0.49 = 0.02
26

---

D
Additional Results
D.1 MNA and Existing Methods
To directly examine the effectiveness of MNA compared existing regret metics, show zero-shot performance ACCEL and PLR using MNA, and MaxMC.
Minigrid
Figures and we illustrate relative performance each of these metries in the standard minigrid domain and show that MNA clearly outperforms other metrics, whether using either PLR or ACCEL.
MaxMC
1.0
0.8
Mean Solve Rate
0.4
0.6
Mean
0.4
0.2
0.0
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Update Step
1c4
Update Step
1e4
(a) Mean Return
(b) Mean Solve
Figure 13: Minigrid comparison of PLR performance using different metrics
MNA ACCEL
PVL- -ACEEL
SARMC-ACCER
0.6
Mean Solve Rate
0.2
0.1
PPO Update Step
1c4
PPO Update Step
1c4
(a) Mean Return
(b) Mean Solve Rate
Figure 14: Minigrid comparison of ACCEL performance using different metrics
27

---

Key Minigrid
Figures 15 and T6 we compare the same methods but on the key minigrid domain instead. Here, we again see that MNA outpertorms existing methods including a substantial performance improvement when using PLR.
MNA PLR
PVL
MaxMC PLR
DR
0.7
1.0
0.8
Mean Solve Rate
0.6
0.3
0.4
0.2
0.5
2.0
3,0
0.0
1.0
1.5
2.0
2.5
3.0
PPO Update Step
PPO Update Stop
(a) Mean Return
(b) Mcan Solve Rate
Figure 15: Key Minigrid comparison of PLR performance using different metrics
MNA ACCEL
PVL-ACCEL
SALMC-ACCCE
DR
0.5
Mean Solve Rate
0.4
0.6
Moan
0.4
0.2
0.1
0.0
0.0
0.0
0.5
1.0
1.5
2.0
3.0
0.0
1.0
2.0
2.5
3.0
PPD Update
1c4
Update Step
1e4
(a) Mean Return
Mean Solve Rate
Figure 16: Key Minigrid- comparison of ACCEL performance using different metrics
28

---

D.2 DEGen and Existing Regret Approximations
In order illustrate the ineffectiveness of existing regret approximations when used optimisation objectives for training teacher, show the relative performance of DEGen using MNA, PVL and MaxMC. Figures 17 show that MNA consistently outperforms PVL and MaxMC. We also see here that using teacher trained using PVL and MaxMC results in at best equivalent, but generally worse, performance compared to naive domain randomisation.
0.7
1.0
0.8
Mean Solve Rate
0.6
Moan
0.3
0.4
0.2
0.2
1.0
1.5
2.0
3,0
1.0
1.5
2.0
2.5
3.0
PPO Update Step
PPO Update Step
1e4
(a) Mean Return
(b) Mean Solve Rate
Figure 17: Minigrid comparison of DEGen performance using different metrics
MNA DEGen
PVL DEGen
MaxMC DFGen
DR
0.6
0.5
Mean Solve Rate
0.1
0.0
1.0
2.0
3.0
0.0
1.0
2.0
2.5
3.0
PPD Update Step
1e4
PPO Update Stop
1e4
(a) Mean Return
(b) Mean Solve Rate
Figure 18: Key Minigrid comparison of DEGen performance using different metrics
29

---

D.3 DEGen vs Initial Gen
In Figures 19 and 20, show the performance of DEGen compared the performance of generator that generates the full level prior student rollouts. We include both standard level generator Initial Gen, identical the PAIRED generator in JaxUED [0], as well as Initial Gen (Rand), which randomly places the agent the level before the rest of the level is constructed. We see that both methods performance worse than domain randomisation.
DCGen MMA
Intiil Gen MNA
Iribaal Gen (Rand) MMA
0.5
Mean Solve Rate
Mean
0.2
0.1
0.0
0.0
0.0
0.5
1.0
1.5
2.0
3.0
0.0
1.0
2.0
2.5
3.0
PPO Update Step
1c4
PPO Update Step
164
(a) Mean Return
(b) Mean Solve Rate
Figure 19: Minigrid comparison of Initial Gen and DEGen
MNA
Gen (Rard) MhA
0.6
0.5
Mean Solve Rate
0.2
0.1
1.0
2.0
3.0
0.0
1.0
2.0
2.5
3.0
PPO Update Step
le4
PPO Update Stop
1e4
(a) Mean Return
(b) Mean Solve Rate
Figure 20: Key Minigrid comparison of Initial Gen and DEGen
30

---

D.4 PAIRED
Finally, we examine the performance of DEGen compared to PAIRED [1] In standard minigrid, we that PATRED performs very similarly domain randomisation, and than DEGen. Additionally, we in the key minigrid domain the limitations the PAIRED regret approximation. As PAIRED relies the antagonist's performance to approximate the best possible level return, high scoring levels require the antagonist to perform well. However, if a level is challenging due to some obstacle the student has not previously encountered, it is likely that the antagonist will also perform poorly, given it has been trained on the same set of levels the student. Therefore, the PATRED generator is unlikely generate levels requiring the antagonist so as student agent has not encountered levels similar to zero-shot hand-designed levels that require the key, zero-shot performance is extremely poor.
the key,
MNA DEGen
PAIRED
DR
Mean Solve Rate
0.4
0.6
Mean
0.4
0.2
0.0
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
PPD Update
1e4
Update Stop
1e4
(a) Mcan Return
Mean Solve Rate
Figure 21: Minigrid comparison of PAIRED and DEGen
MNA DEGen
PAIRED
DR
0.6
Mean Solve Rate
0.2
PPO Update Step
1e4
PPO Update Step
1e4
(a) Mean Return
(b) Mcan Solve Rate
Figure 22: Key Minigrid comparison of PAIRED and DEGen
31

---

D.5
Training Level Examples
We have included examples of levels generated by each method in the repository httpa://giithb
http:///wwaaa...mmtmmttttaataooa were all sampled from the final training step.
These levels
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in abstract are accurately reflected in the description our method and our results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be pereeived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Iustification:
We have discussed the limitations of this work Section
Guidelines:
The answer NA means that paper has no limitation while answer means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust results are violations these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
grounds for rejection,
Theory assumptions and proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper contains theoretical results.
Guidelincs:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
33

---

model The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
Formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The full method used is outlined in Sections B and 4 and further detail is provided in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is datasct and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, contribution novel architecture, describing the architecture fully might suffice, or the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., the case of a large language model), releasing checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require relcasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (c.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reprodueibility. In the case of closed-source source models, it may be that access to the model is limited in some way (c.g.. to registered users). but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open faithfully reproduce the main experimental results, as described supplemental material?
the data and code. with sufficient instructions
Answer:
[Yes]
Justification: We include the code and instructions in the supplementary material
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurTPS code and data submission guidelines hhttpa://nips.cc/pppe
for more details.
While we encourage release of code and data, we understand that might not possible,
so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source henchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : ://nis//..////////// cc/public/ Quiider//CodeSubbllssssooPo..... for more details.
34

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) recommended, but including URLs to data and code is permitted.
6. Experimental sefting/details
Question: Does the paper specify all the training and test details (c.g,, data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All addtional hyperparameters and experiment details are included in Appendix B.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: All results are taken over multiple seeds with standard error
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should given (e-g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error than state that they not verified.
if the hypothesis of Normality of errors
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). crror bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, docs the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
Details on compute and time provided in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
35

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform. every respect, with NeurIPS Code Ethics hhttps://eeuree http://eurripsscppppililiiicceep cccpppllicc/tahaccuuecelllllllad
Answer:
[Yes]
Justification:
Our research contorms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics,
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e-g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: Our work is not expected to have direct socictal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considcrations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated relcase of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible relcase of data or models that have a high risk for misuse (e.g. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not present any of these risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
36

---

12. Licenses for existing assets
Question: Are creators or original owners assets (e.g., code, data, models), used the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected"'
Answer: [Yes]
Justification: All prior work that has been used has been cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, ppaperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged reach out asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have provided details on training, as well as code and documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training. license, limitations.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection. curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This work does not involve research with human subjects.
Guidelines:
37

---

The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
recognize that procedures this may vary significantly between institutions and Iocations, and we expect authors adhere to the NeurIPS Code of Ethics the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
Declaration LLM usage
Does the paper describe the usage of LLMs if it important, original, or non-standard component of the core methods in this rescarch? Note that if the LLM is used only for writing, editing, or formatting purposes and does impact the core methodology, seientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not used any important, original or non-standard components of this reserach.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, or non-standard components.
Please refer to our LLM policy httppp////orrpppp..//////ooeeeer wwt/oo/ooeeenne////////// should should not be described.
LLM) for what
38


Reviews:
[{"review_id": "ReLjmtA4wO", "reviewer": "Reviewer_7GXd", "summary": "The authors present two methods for improving UED. First, they introduce a partial environment generation method that only produces parameters of an explored level as the agent explores it. Second, they introduce a new approximation of regret for UED that promotes more challenging levels as a regret counterpart of level replay methods that promote past challenging and diverse levels. The authors evaluate their methods on the minigrid environment and a modification that requires the agent to find a key to open a door. Both improvements yield an algorithm that outperforms standard baselines for UED on larger levels.", "questions": "- Some details of dynamic environment generation confuse me. Specifically, the environment generator dynamically produces a partial environment based on what the current agent observes. However, minimax regret-based methods often require a protagonist and antagonist agent. These agents could interact with different parts of the environment. How does this interact with partial environments when computing regret? Or can this method only be applied to methods with one agent?\n\n- In the derivation of MNA, the authors assume a deterministic environment. Is this a general assumption? Why or why not? Is MNA limited to deterministic environments or could be generalized?\n\n- The dynamic environment generation seems to be only applicable in domains where strong partial observability is present. What about fully observable domains? Is there still a benefit?\n\n- In the paper, the minigrid levels are closely connected to observations. However, consider environments like doom, where there is significant computation to produce the observations. It seems like the inverse function of mapping from an partial observations to the free parameters needed to be filled in by the environment generation problem to be highly non-trivial. How would the authors propose adapting their method to more complex entangling of observation spaces and environment free parameters?", "limitations": "- The domain in the paper is not representative of all domains that could be used in the work. (For example, with respect to stochasticity or or continuous environments.) The current writing does not discuss domains where the methods presented could not be applied, nor gives an avenue for applying dynamic environment generation to more complex domains. I encourage the authors to expand the limitations section more. It's not clear to me how to apply the method and receive benefit on domains other than mini grid. Many domains seem like they would have no benefits (such as those with full observability or observability requiring knowledge of all environment free parameters).", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "- The paper provides a clear review of the UED problem in the context of reinforcement learning and curriculum learning’s history.\n- I feel figure 1 is not as impactful as it could be. It would be better if the first figure gave an overview of the proposed method, or contrasted the approach with existing methods. This is clear and easy to do for dynamic environment generation (as done in Figure 2). For the MNA, a figure highlighting the intuition of the new regret approximation would improve the paper.\n- The paper fails to cite the seminal paper in this space “Paired Open-Ended Trailblazer” from Wang et. al. Many of the ideas in the PAIRED paper formalized concepts of this work.\n- The authors only test their proposed methods on the minigrid domain. To convince readers of the paper that the results generalize, more domains are needed.\n- This paper feels like two separate papers stuck together. Instead of two thorough papers, we have two half-papers without thorough evaluation. Both dynamic environment generation and the new proposed regret method are complicated enough, and produce sufficiently independent benefits, that I think splitting these contributions into two papers would be better. I'd rather see two separate contributions that explore each avenue more thoroughly. I'd particularly be interested in how dynamic environment generation could be applied to environments with complex entanglement of the observation space and environment parameterization. Solving that issue seems essential for the method to be generally applicable.\n- The authors mention the problem of small changes in levels resulting in large changes in difficulty. This is a well studied problem in the Procedural Content Generation community. I refer the authors to “The Unexpected Consequence of Incremental Design Changes” from Sturtevant et. al. as a starting point.\n- The authors mention small changes in levels producing large changes in difficulty, and give a few examples in the paper, but they don’t evaluate their method on environments where this effect occurs strongly. If this is a core argument in the paper, there should be some evidence that the method improves performance under these conditions. I recommend taking the snakebird environment from Sturtevant et. al. (see above) or Sokoban as an evaluation as both have been shown to have these properties.\n- There has been some work in scaling environment generation via neural cellular automata. This seems like a natural baseline to compare dynamic environment generation against as the paper has results on 66x66 minigrid mazes. See “Arbitrarily Scalable Environment Generators via Neural Cellular Automata” from NeurIPS 2023. A simple baseline would be to replace the evolutionary environment generator of ACCEL with NCAs optimized via CMA-ES with a regret objective.\n- The presentation of experimental results could be more thorough. For example, it is unclear how many trials were used in the experiments. It is also not stated what the error bars in the plots represent. Finally, there is no table listing the quantitative results of all experiments.\n- Overall, if a claim is made in the paper, experiments should be designed to support the claim. Domains should be catered to study the proposed benefits of the method. I feel this paper has a disconnect between claims made and the experiments run.", "was_revised": true, "final_justification": "The authors did a good job of addressing my technical concerns during the rebuttal period. The original manuscript only contained one experimental domain (mini-grid), and although the core idea behind the paper is interesting, the experiments were not thorough enough  for a NeurIPS paper. The original manuscript also failed to highlight some major limitations of the proposed work. Additionally, I had some confusion on why the two techniques needed to exist in one paper.\n\nTo address these concerns, the authors added one of my suggested domains (Sokoban) and shared preliminary one-shot results. The new domain greatly strengthens the paper as Sokoban is a domain where small changes greatly affect the outcome and these changes occur quite frequently in the environment. This better supports the arguments made in the paper. The preliminary results also show a wider gap between the proposed method and baselines. I think this better justifies the proposed approach. While we didn't get to see the final results before the rebuttal period ended (which would be unreasonable), I do think the authors will be able to make a fair comparison before the camera ready period. If the paper is accepted, the authors should feel comfortable publishing negative results in this domain as it would highlight current limitations. The outcome of the additional experiments should not affect acceptance. In other words, the authors should include accurate results and not feel pressured to make their proposed method work beat the baselines.\n\nThe authors rewrote a portion of the limitations section to highlight the limitations raised by reviewers not addressed in the current manuscript. These included problems related to multiple agents, stochastic environments, and complex mappings between observations and environment representation. I think this is important because future work could address these limitations and cite this work as evidence these are open and important problems. While I would like to see a work address the complex mapping issue when doing dynamic environment generation, it is perhaps best left to future work so that the community at large can try and address it.\n\nMy main negative not addressed during the rebuttal period is lack of scholarship from the broader environment generation field. After a small literature review of procedural content generation, I found many paper studying dynamic level generation from agent feedback and methods that scale to large game environments. I believe this is a scholarship problem with the UED community at large rather than a problem with specific authors. Level generation has been studied widely for decades, but many of the proposed methods have not been applied to UED settings where RL agent training is in the loop. However, I do think the paper should cite these related ideas and provide context to the novelty of the proposed work, which is only novel in a UED context. The authors should also consider citing the POET paper as it inspired this research area.", "rebuttal": "Thank you for your review, we really appreciate the detailed and thorough response you have given, especially your insight into similar work in the Procedural Content Generation community that we had not previously considered. \n\n## Combining MNA and DEGen\n\nWhilst we understand why you have suggested  presenting MNA and DEGen as two separate papers, we feel these two contributions are too closely linked for this. One of the core goals of this research was to enable denser reward signals for RL-training a level generator to reduce the difficulty of credit assignment. Previous work relying on a generator that fully generates the level initially only allowed for a single sparse reward at the final generator time step. By implementing DEGen, we can instead assign rewards throughout the level generator trajectory. In order to maximally leverage this advantage, a dense regret approximation was necessary. However in practice, we found that existing regret approximation metrics, such as PVL and MaxMC, performed poorly when used to train a DEGen adversary. This indicates that although these metrics outperform naive DR when used to curate levels generated through random level generation/mutation, when directly optimised for, these metrics perform poorly. This directly necessitated an improved regret approximation (MNA) that performs well in both with the curation-based methods, and with an RL-trained level-generator like in DEGen.\n\n## Evaluation Environments\n\nThe suggestions of Snakebird and Sokoban as environments that exhibit large changes in difficulty with small changes to the environment was useful. Based on this suggestion, we have begun to run experiments in Sokoban with some initial promising results. However, we do also believe that the provided key-based minigrid environment also strongly exhibits this property. Given two near-identical levels, but with one having a door blocking the path to the goal, and one without, a relatively small change of environment has induced a large change in difficulty. It is not simply that the optimal solution in the former will be longer, but to solve the former, the agent will have had to learn how to find the key, and how to use the key to unlock the door (i.e. learn a new class of solution). \n\n## Experimental Details\n\nEach experiment was run over eight seeds, and the error bars shown are the 95% confidence intervals of these seeds. We can also include a detailed table of the quantitative results in the Appendix.\n\n# Questions\n\n***Some details of dynamic environment generation confuse me …***\n\nDEGen is usable for methods with multiple agents. If both agents are running in parallel in the environment: after each protagonist/antagonist step, any ungenerated parts of the environment that are observed by either agent can be generated by the adversary. We have done some experimentation with the PAIRED objective with the DEGen adversary, so it is definitely possible to use methods that require multiple agents. \n\n***In the derivation of MNA, the authors assume a deterministic environment …***\n\nIn its current form, MNA is limited to deterministic environments given Eqn 6 does not necessarily hold in stochastic environments. However, as far as we are aware, existing UED work only focuses on deterministic environments. Stochastic environments greatly increase the difficulty of approximating the optimal return, as it becomes difficult to determine whether a given trajectory resulted in a high return because of strong actions or good luck with environmental randomness. E.g. a metric like MaxMC would likely break down in stochastic environments as it would score highly on levels that have a high probability of low reward and a low probability of high reward based purely on environment dynamics. \n\n***The dynamic environment generation seems to be only applicable in domains where strong partial observability is present …***\n\nDEGen will only differ to methods that fully-generate the level up front in domains that are strongly partially observable. However, we feel there are sufficient domains where this holds for DEGen to have benefits. For example, robotics domains relying on local, limited range sensors. Also, many games will fall into this category, such as Crafter [1], or domains like side-scrolling platformers. \n\n***In the paper, the minigrid levels are closely connected to observations …***\n\nIt is a limitation of DEGen that in certain environments it may be more difficult to map environment parameterisations to agent observations. We will add a discussion of this to the limitations section of the paper.  \n\n\n[1] Hafner, Danijar. \"Benchmarking the spectrum of agent capabilities.\" arXiv preprint arXiv:2109.06780 (2021)."}, {"review_id": "FXqGiaEOJX", "reviewer": "Reviewer_dzPV", "summary": "* Introduces (1) a new regret approximation for the UED setting, Maximized Negative Advantage (MNA), in order to better identify challenging levels; and (2) Dynamic Environment Generation for UED (DEGen) to enable UED to scale to larger environment sizes", "questions": "* I wonder if the DEGen idea is directly compatible with a diffusion-based generator? I.e. where each diffusion step clarifies the environment more and more; of course, this increases the complexity of training the generator, probably; but might be a more scalable approach in the long-term (assuming the initial diffusion step produces a valid level)", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n* The paper is well-written; easy to follow; properly contextualizes related work\n* Accurate (and sensitive, in the upper-difficulty region) regret approximations are indeed much-needed; a fundamental challenge with UED\n* The DEGen idea is fantastically creative and novel; hadn't considered this kind of approach for improving the problem of credit assignment for UED, but is natural, and—\n* The MNA and DEGen results are both convincing\n\nWeaknesses\n* It's not clear to me how general the DEGen idea is; I can imagine it being difficult to define the constraints in more realistic settings, where it is not so easy to tell which causal variables have been interacted with yet, and can be generated on the fly\n* None major", "was_revised": true, "final_justification": "I stand by my initial review and therefore maintain my current score.", "rebuttal": "Thank you for your review. We very much appreciate your positive response to our work. \n\nWe do agree that there may be some limitations with DEGen mapping the agent’s observations to the relevant environment parameters, but we believe that DEGen represents a promising starting point for future work.\n\n# Questions\n\n***I wonder if the DEGen idea is directly compatible with a diffusion-based generator …***\n\nThis definitely is an interesting approach for future work. There has already been UED work [1] using diffusion-based generation for generating whole levels, so applying this to generating partial levels seems like a natural extension. Thank you for the suggestion. \n\n[1] Chung, Hojun, et al. \"Adversarial environment design via regret-guided diffusion models.\" Advances in Neural Information Processing Systems 37 (2024): 63715-63746."}, {"review_id": "bv6fzIvJCy", "reviewer": "Reviewer_D6YH", "summary": "This paper proposes to improve unsupervised environment generation methods. It does this using two ideas: 1) to avoid the level generator struggles with credit assignment just have it generate the level around the agent and 2) a new approximation for regret. It shows that as UED as scaled to larger environments, this method begins to outperform the baselines.", "questions": "- In line 187, why does decreasing the horizon of this V function increase the likelihood of conservatism? Is it purely that you're decreasing the number of random variables you're taking a max over?\n- Why does figure 5 use return instead of solve rate?\n- Did you list your hyperparameter tuning procedure for all methods in the paper somewhere? Did I just miss it?", "limitations": "yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I thought this was a really good paper. The ideas are clear and understandable, the results seem to be significant and the methods can be straightforwardly implemented. They test against standard, strong baselines. I have very little to say because I think the paper is very solid.\n\n### Serious Weaknesses\n- Please put your hyperparameter tuning procedure for your algorithms and the baselines into the supplement or paper. Without this it is hard to tell if the baselines were really afforded their due in tackling the problems under consideration.\n\n### Less serious weaknesses / suggestions\n- It probably would make the paper stronger to study more environments besides minigrid as it would help make the case that the technique is more general. It won't change my score, it's just a suggestion. \n\n\n### Nitpicks (you don't necessarily have to follow them and it won't change my score but I think you should make these changes to make your paper better)\n- Going from equation 7 to equation 8 is a bit of a leap, you could hold the reader's hand more here.\n- I think in 178 you should be clearer that this is the true value function", "was_revised": true, "final_justification": "My original score was provided based on not observing any significant errors and the results seeming to be a significant improvement. The authors answered my questions so I maintain my score.", "rebuttal": "Thank you for your review. We very much appreciate your positive response to our work.\n\n## Hyperparameter Tuning\n\nFor hyperparameter tuning, we have relied on previous work [1] using the same implementations for our baselines. Full details are shown in the Appendix of [1], and we will state this in the final version of the paper\n\n## Additional Environments\n\nReviewer 7GXd suggested some additional environments that may be suitable. We have done some preliminary experiments in Sokoban with promising initial results, and so may be able to include these in a final version of the paper. \n\n# Questions\n\n***In line 187, why does decreasing the horizon of this V function increase the likelihood of conservatism …***\n\nReducing the horizon results in a reduced number of different value approximations for the state (see Eqn 7), and so the greater likelihood of conservatism is just to do with a reduced number of variables over which you are taking a max over. \n\n***Why does figure 5 use return instead of solve rate?***\n\nWe have only included one of return and solve rate in Figure 5 for the sake of space and chose return because it not only reflects the proportion of times the agent solves the level but also how quickly the level is solved. However, we do acknowledge that it may be less interpretable to the reader so we will include the solve rate plots in the Appendix, but qualitatively, these solve rate plots show the same patterns as seen in the return plots.\n\n\n[1] Rutherford, Alexander, et al. \"No regrets: Investigating and improving regret approximations for curriculum discovery.\" Advances in Neural Information Processing Systems 37 (2024): 16071-16101."}, {"review_id": "QQwSmt9r1K", "reviewer": "Reviewer_6nip", "summary": "This paper proposes Dynamic Environment Generation for Unsupervised Environment Design with a new regret approximation. This method outperforms existing methods in some cases.", "questions": "- It is confusing that the **zero-shot performance** has rewards curves with the updating steps of parameters. In many RL works that focus on generalization and transferring, zero-shot performance will be shown with success rates in unseen new tasks without updating any parameters of the whole agent. For instance, in the citation [8] of this work, they utilize 'percent successful trials in environments used to test zero-shot transfer'.\n- The method DEG may have good consistency because it utilizes a mask to build partial observation environments for simulating the change and dynamics in the environments. However, this method will lose diversity. In environment generation, consistency causes overfitting while diversity causes instability. This paper requires further discussion on the effectiveness of its method.\n- Many of the test experiments are hand-designed. It will undermine the credibility of the results.\n- In both the appendix and the main text, the results seem to diverge from the claim in the paper. Because the performance differences between the baselines are minimal in minigrid but are quite different in key-minigrid. The task of key-minigrid has bottleneck states and requires strong exploration capabilities. The outperformance owing to that the curricula built by partial observation can give guidance to encourage to explore, not the intrinsic advantage of the method itself. These are not suitable comparisons.", "limitations": "Yes but not sufficient.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The focused problem is important and this paper demonstrates good clarity of expression.\n\nWeakness:\n- This paper lacks sufficient analyses of the effectiveness of the method.\n- Parts of the setup in the experiments are confused.", "was_revised": true, "final_justification": "The author has addressed most of my concerns satisfactorily. I will raise the score to ‘Borderline Accept’.", "rebuttal": "# Questions\n\n***It is confusing that the zero-shot performance has rewards curves …***\n\nTo provide further clarity, the zero-shot return plots are showing the return and solve rate of the agent on the unseen test set as the agent is trained using a specific UED method. The agent is not being trained on the unseen levels - it is being trained using the levels generated/curated by the UED method. Plots showing zero-shot performance throughout training are standard in UED literature [1,2,3,4,5]\n\n***The method DEG may have good consistency because it utilizes a mask to build partial observation …***\n\nWe found that introducing some randomness to the environment initialisation - as outlined in Section 3 of the paper - was sufficient to ensure level diversity throughout training. We did not observe any instability induced by this diversity. We would appreciate a follow-up question if this does not fully address your concerns.\n\n***Many of the test experiments are hand-designed …***\n\nWhilst there could be issues with using hand-designed test sets, it is the standard practice in UED literature [1,2,3,4,5] to show zero-shot performance. \n\n***In both the appendix and the main text, the results seem to diverge from the claim in the paper …***\n\nWe disagree with our results diverging from claims of the paper. We agree that the key-minigrid task does require strong exploration capabilities but our results show that MNA and DEGen are capable of facilitating a curriculum that enables the agent to learn this exploration, given they both outperform existing baselines. As you suggest, “The outperformance owing to that the curricula built by partial observation can give guidance to encourage to explore”, but this is an intrinsic advantage of the method. We would appreciate a follow-up question if you have further issues with this not being a suitable comparison.\n\n\n[1] Chung, Hojun, et al. \"Adversarial environment design via regret-guided diffusion models.\" Advances in Neural Information Processing Systems 37 (2024): 63715-63746.\n\n[2] Parker-Holder, Jack, et al. \"Evolving curricula with regret-based environment design.\" International Conference on Machine Learning. PMLR, 2022.\n\n[3] Rutherford, Alexander, et al. \"No regrets: Investigating and improving regret approximations for curriculum discovery.\" Advances in Neural Information Processing Systems 37 (2024): 16071-16101.\n\n[4] Jiang, Minqi, et al. \"Replay-guided adversarial environment design.\" Advances in Neural Information Processing Systems 34 (2021): 1884-1897.\n\n[5] Mediratta, Ishita, et al. \"Stabilizing unsupervised environment design with a learned adversary.\" Conference on Lifelong Learning Agents. PMLR, 2023."}]

Author rebuttal:

Meta-review: The paper had two accepts, one borderline accept and one strong reject (at the decision deadline). The authors addressed many of the points in the strong reject review, yet the reviewer has not replied to the authors or justified their score on time. I therefore chose to ignore this review predicting that the reviewer will most likely increase their score (as they did post the decision deadline). Given that the majority of reviews recommended acceptance, and found the paper to be well motivated with creative ideas and good experiments, I recommend accepting it.
Decision: Accept (poster)