Title: Improving Regret Approximation for Unsupervised Dynamic Environment Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: UED

Reinforcement Learning
Abstract: Unsupervised Environment Design (UED) seeks to automatically generate training curricula for reinforcement learning (RL) agents, with the goal of improving generalisation and zero-shot performance. However, designing effective curricula remains a difficult problem, particularly in settings where small subsets of environment parameterisations result in significant increases in the complexity of the required policy. Current methods struggle with a difficult credit assignment problem and rely on regret approximations that fail to identify challenging levels, both of which are compounded as the size of the environment grows. We propose Dynamic Environment Generation for UED (DEGen) to enable a denser level generator reward signal, reducing the difficulty of credit assignment and allowing for UED to scale to larger environment sizes. We also introduce a new regret approximation, Maximised Negative Advantage (MNA), as a significantly improved metric to optimise for, that better identifies more challenging levels. We show empirically that MNA outperforms current regret approximations and when combined with DEGen, consistently outperforms existing methods, especially as the size of the environment grows. We have made all our code available here: \url{https://github.com/HarryMJMead/Dynamic-Environment-Generation-for-UED}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Dynamic Environment Generation for Unsupervised Environment Design with a new regret approximation. This method outperforms existing methods in some cases.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The focused problem is important and this paper demonstrates good clarity of expression.\n\nWeakness:\n- This paper lacks sufficient analyses of the effectiveness of the method.\n- Parts of the setup in the experiments are confused.\n\nQUESTIONS:\n- It is confusing that the **zero-shot performance** has rewards curves with the updating steps of parameters. In many RL works that focus on generalization and transferring, zero-shot performance will be shown with success rates in unseen new tasks without updating any parameters of the whole agent. For instance, in the citation [8] of this work, they utilize 'percent successful trials in environments used to test zero-shot transfer'.\n- The method DEG may have good consistency because it utilizes a mask to build partial observation environments for simulating the change and dynamics in the environments. However, this method will lose diversity. In environment generation, consistency causes overfitting while diversity causes instability. This paper requires further discussion on the effectiveness of its method.\n- Many of the test experiments are hand-designed. It will undermine the credibility of the results.\n- In both the appendix and the main text, the results seem to diverge from the claim in the paper. Because the performance differences between the baselines are minimal in minigrid but are quite different in key-minigrid. The task of key-minigrid has bottleneck states and requires strong exploration capabilities. The outperformance owing to that the curricula built by partial observation can give guidance to encourage to explore, not the intrinsic advantage of the method itself. These are not suitable comparisons.\n\nLIMITATIONS:\nYes but not sufficient.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to improve unsupervised environment generation methods. It does this using two ideas: 1) to avoid the level generator struggles with credit assignment just have it generate the level around the agent and 2) a new approximation for regret. It shows that as UED as scaled to larger environments, this method begins to outperform the baselines.\n\nSTRENGTHS AND WEAKNESSES:\nI thought this was a really good paper. The ideas are clear and understandable, the results seem to be significant and the methods can be straightforwardly implemented. They test against standard, strong baselines. I have very little to say because I think the paper is very solid.\n\n### Serious Weaknesses\n- Please put your hyperparameter tuning procedure for your algorithms and the baselines into the supplement or paper. Without this it is hard to tell if the baselines were really afforded their due in tackling the problems under consideration.\n\n### Less serious weaknesses / suggestions\n- It probably would make the paper stronger to study more environments besides minigrid as it would help make the case that the technique is more general. It won't change my score, it's just a suggestion. \n\n\n### Nitpicks (you don't necessarily have to follow them and it won't change my score but I think you should make these changes to make your paper better)\n- Going from equation 7 to equation 8 is a bit of a leap, you could hold the reader's hand more here.\n- I think in 178 you should be clearer that this is the true value function\n\nQUESTIONS:\n- In line 187, why does decreasing the horizon of this V function increase the likelihood of conservatism? Is it purely that you're decreasing the number of random variables you're taking a max over?\n- Why does figure 5 use return instead of solve rate?\n- Did you list your hyperparameter tuning procedure for all methods in the paper somewhere? Did I just miss it?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
* Introduces (1) a new regret approximation for the UED setting, Maximized Negative Advantage (MNA), in order to better identify challenging levels; and (2) Dynamic Environment Generation for UED (DEGen) to enable UED to scale to larger environment sizes

STRENGTHS AND WEAKNESSES:
Strengths
* The paper is well-written; easy to follow; properly contextualizes related work
* Accurate (and sensitive, in the upper-difficulty region) regret approximations are indeed much-needed; a fundamental challenge with UED
* The DEGen idea is fantastically creative and novel; hadn't considered this kind of approach for improving the problem of credit assignment for UED, but is natural, and—
* The MNA and DEGen results are both convincing

Weaknesses
* It's not clear to me how general the DEGen idea is; I can imagine it being difficult to define the constraints in more realistic settings, where it is not so easy to tell which causal variables have been interacted with yet, and can be generated on the fly
* None major

QUESTIONS:
* I wonder if the DEGen idea is directly compatible with a diffusion-based generator? I.e. where each diffusion step clarifies the environment more and more; of course, this increases the complexity of training the generator, probably; but might be a more scalable approach in the long-term (assuming the initial diffusion step produces a valid level)

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present two methods for improving UED. First, they introduce a partial environment generation method that only produces parameters of an explored level as the agent explores it. Second, they introduce a new approximation of regret for UED that promotes more challenging levels as a regret counterpart of level replay methods that promote past challenging and diverse levels. The authors evaluate their methods on the minigrid environment and a modification that requires the agent to find a key to open a door. Both improvements yield an algorithm that outperforms standard baselines for UED on larger levels.\n\nSTRENGTHS AND WEAKNESSES:\n- The paper provides a clear review of the UED problem in the context of reinforcement learning and curriculum learning’s history.\n- I feel figure 1 is not as impactful as it could be. It would be better if the first figure gave an overview of the proposed method, or contrasted the approach with existing methods. This is clear and easy to do for dynamic environment generation (as done in Figure 2). For the MNA, a figure highlighting the intuition of the new regret approximation would improve the paper.\n- The paper fails to cite the seminal paper in this space “Paired Open-Ended Trailblazer” from Wang et. al. Many of the ideas in the PAIRED paper formalized concepts of this work.\n- The authors only test their proposed methods on the minigrid domain. To convince readers of the paper that the results generalize, more domains are needed.\n- This paper feels like two separate papers stuck together. Instead of two thorough papers, we have two half-papers without thorough evaluation. Both dynamic environment generation and the new proposed regret method are complicated enough, and produce sufficiently independent benefits, that I think splitting these contributions into two papers would be better. I'd rather see two separate contributions that explore each avenue more thoroughly. I'd particularly be interested in how dynamic environment generation could be applied to environments with complex entanglement of the observation space and environment parameterization. Solving that issue seems essential for the method to be generally applicable.\n- The authors mention the problem of small changes in levels resulting in large changes in difficulty. This is a well studied problem in the Procedural Content Generation community. I refer the authors to “The Unexpected Consequence of Incremental Design Changes” from Sturtevant et. al. as a starting point.\n- The authors mention small changes in levels producing large changes in difficulty, and give a few examples in the paper, but they don’t evaluate their method on environments where this effect occurs strongly. If this is a core argument in the paper, there should be some evidence that the method improves performance under these conditions. I recommend taking the snakebird environment from Sturtevant et. al. (see above) or Sokoban as an evaluation as both have been shown to have these properties.\n- There has been some work in scaling environment generation via neural cellular automata. This seems like a natural baseline to compare dynamic environment generation against as the paper has results on 66x66 minigrid mazes. See “Arbitrarily Scalable Environment Generators via Neural Cellular Automata” from NeurIPS 2023. A simple baseline would be to replace the evolutionary environment generator of ACCEL with NCAs optimized via CMA-ES with a regret objective.\n- The presentation of experimental results could be more thorough. For example, it is unclear how many trials were used in the experiments. It is also not stated what the error bars in the plots represent. Finally, there is no table listing the quantitative results of all experiments.\n- Overall, if a claim is made in the paper, experiments should be designed to support the claim. Domains should be catered to study the proposed benefits of the method. I feel this paper has a disconnect between claims made and the experiments run.\n\nQUESTIONS:\n- Some details of dynamic environment generation confuse me. Specifically, the environment generator dynamically produces a partial environment based on what the current agent observes. However, minimax regret-based methods often require a protagonist and antagonist agent. These agents could interact with different parts of the environment. How does this interact with partial environments when computing regret? Or can this method only be applied to methods with one agent?\n\n- In the derivation of MNA, the authors assume a deterministic environment. Is this a general assumption? Why or why not? Is MNA limited to deterministic environments or could be generalized?\n\n- The dynamic environment generation seems to be only applicable in domains where strong partial observability is present. What about fully observable domains? Is there still a benefit?\n\n- In the paper, the minigrid levels are closely connected to observations. However, consider environments like doom, where there is significant computation to produce the observations. It seems like the inverse function of mapping from an partial observations to the free parameters needed to be filled in by the environment generation problem to be highly non-trivial. How would the authors propose adapting their method to more complex entangling of observation spaces and environment free parameters?\n\nLIMITATIONS:\n- The domain in the paper is not representative of all domains that could be used in the work. (For example, with respect to stochasticity or or continuous environments.) The current writing does not discuss domains where the methods presented could not be applied, nor gives an avenue for applying dynamic environment generation to more complex domains. I encourage the authors to expand the limitations section more. It's not clear to me how to apply the method and receive benefit on domains other than mini grid. Many domains seem like they would have no benefits (such as those with full observability or observability requiring knowledge of all environment free parameters).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-21"}