Title: Imagined Autocurricula
Venue: NeurIPS 2025
Accepted: yes
Keywords: World Models

Reinforcement Learning

UED

Generalization
Abstract: Training agents to act in embodied environments typically requires vast training data or access to accurate simulation, neither of which exists for many cases in the real world. Instead, world models are emerging as an alternative–leveraging offline, passively collected data, they make it possible to generate diverse worlds for training agents in simulation. In this work, we harness world models to generate “imagined” environments to train robust agents capable of generalizing to novel task variations. One of the challenges in doing this is ensuring the agent trains on useful generated data. We thus propose a novel approach IMAC (Imagined Autocurricula) leveraging Unsupervised Environment Design (UED), induces an automatic curriculum over generated worlds. In a series of challenging, procedurally generated environments, we show it is possible to achieve strong transfer performance on held-out environments having trained only inside a world model learned from a narrower dataset. We believe this opens the path to utilizing larger-scale, foundation world models for generally capable agents.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Imagined Autocurricula (iMAC), a novel method for training reinforcement learning agents. The method leverages offline data to generate diverse imagined environments using a trained diffusion model and employs Prioritized Level Replay to create an emergent curriculum that adapts to the agent’s capabilities.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nThe paper is clear and well-written. The introduction sets up the problem context and motivation effectively. The proposed method is explained logically, with a helpful architectural overview in Figure 1.\n\n## Weaknesses\n\nThe quality of the diffusion model depends on PPO being able to solve the tasks during data collection. It is unclear if the final agent is able to solve qualitatively different tasks after being trained with iMac.\n\nThe paper claims to demonstrate \"open-ended learning in learned world models\". This is a powerful claim, as open-ended learning often implies the generation of tasks with ever-increasing and potentially unbounded complexity. Given that the world model is fixed after training on a finite dataset, the space of generated environments is necessarily constrained by what the model has learned.\n\nA potential weakness is the complexity of the method with many moving pieces.\n\nAnother potential weakness is the very high computational cost—180 GPU days for the main experiments —which may be a barrier to reproducibility and adoption. However, the authors are transparent about this in the limitations section.\n\nQUESTIONS:\nThe performance of IMAC is fundamentally tied to the quality of the generative world model, which in turn depends on the offline dataset used for its training. The paper uses a \"mixed\" dataset of expert, medium, and random trajectories. How does IMAC's performance degrade if the world model is trained on a less diverse or lower-quality dataset?\n\nThe authors should discuss the OMNI-EPIC paper, as it appears to make similar claims. A discussion comparing the two approaches would help to clarify the novel contributions of this paper.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an offline model-based RL method that integrates unsupervised environment design with a generative world model, to train agents capable of generalizing to unseen tasks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The paper introduces a well-motivated and novel approach that leverages generative world models to train agents with strong generalization capabilities. Applying unsupervised environment design when generating imaginary rollouts seems like a powerful idea and has the potential to be used in large scale foundation world models.\n- The paper is clearly written. It is easy to understand its main idea and algorithm details.\n- The proposed method shows significant improvements in generalization performance.\n\n### Weaknesses\nWhile the proposed method combines existing components, its contribution would be clearer with a more in-depth justification of the core design choices. In particular, explaining why these components were selected over other viable alternatives would help clarify the novelty and applicability of the method.\n- Justification for the use of a 2D UNet architecture\n  - The authors adopt 2D UNet architecture following DIAMOND [1]. However, recent advances in video models often use 3D DiT architecture [2, 3]. Although an exhaustive architectural comparison may be infeasible, referencing and briefly discussing recent alternatives would help contextualize this design choice.\n- Justification for using prioritized level replay (PLR)\n  -  There exist recent UED algorithms that may be applicable in the proposed framework. For instance, ACCEL [4] could be implemented by varying the noise schedule, and ADD [5] could be integrated naturally by guiding the diffusion world model. \nAdditionally, although the authors employ PVL for prioritization score, alternative regret estimation strategies such as MaxMC [6] are available and sometimes yield stronger performance. A discussion of these alternatives would enhance the technical depth of the paper.\n- Limited baselines\n  - Among baselines, only transformer-based methods can be seen as model-based approaches. Given that the proposed method leverages a diffusion world model, incorporating baselines such as Diffuser [7] and its variants would allow for a more comprehensive evaluation.\n- Experiments\n  - While the authors conduct extensive experiments in Procgen benchmark, I'm concerned about whether the proposed method works well in other domains. Specifically, while the authors utilize random exploration data to ensure broad state coverage, there are many cases where random exploration data is unavailable (e.g. real world robotics tasks). \n\n### References\n\n[1] Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, François Fleuret, \"Diffusion for World Modeling: Visual Details Matter in Atari\", NeurIPS, 2024.  \n[2] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Yuxuan Zhang, Weihan Wang, Yean Cheng, Bin Xu, Xiaotao Gu, Yuxiao Dong, Jie Tang, \"CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer\", ICLR, 2025.  \n[3] NVIDIA, \"Cosmos World Foundation Model Platform for Physical AI\", arXiv, 2025.  \n[4] Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, Tim Rocktäschel, \"Evolving Curricula with Regret-Based Environment Design\", ICML, 2022.  \n[5] Hojun Chung, Junseo Lee, Minsoo Kim, Dohyeong Kim, Songhwai Oh, \"Adversarial Environment Design via Regret-Guided Diffusion Models\", NeurIPS, 2024.  \n[6] Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob Foerster, Edward Grefenstette, Tim Rocktäschel, \"Replay-Guided Adversarial Environment Design\", NeurIPS, 2021.  \n[7] Michael Janner, Yilun Du, Joshua B. Tenenbaum, Sergey Levine, \"Planning with Diffusion for Flexible Behavior Synthesis\", ICML, 2022.\n\nQUESTIONS:\n- In table 1, what is the maximum achievable return for each task? It is unclear whether the reported scores represent meaningful improvements, as the absolute values appear relatively low.\n- Can the proposed approach be extended to settings without explicit reward signals, which are common in real-world applications?\n- Is the method applicable to continuous action spaces, or is it limited to discrete environments?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Imagined Autocurricula, a model-based offline reinforcement learning method that employs Prioritized Level Replay to exploit valuable states for agent learning. By focusing on initial states with the highest learning potential and utilizing a dynamic imagining horizon, this method demonstrates superior generalization to held-out levels in five Procgen environments.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\nS1) The proposed method is well-motivated.\n\nS2) The writing is clear and easy to follow.\n\nS3) Compared to other baselines, the proposed method achieves a notable performance gain on five challenging Procgen environments.\n\n# Weaknesses\n\nW1) **Task coverage.** In Line 35-36, the authors claim that they attempt to bridge the gap between previous MBRL methods that focus on single environments and those utilizing Internet-pretrained world models for training generalist agents. However, the proposed method is trained and evaluated on only five tasks with similar goal-reaching objectives, which is not sufficient to convincingly demonstrate its generalization capabilities. Additionally, the proposed method is also only able to generalize to different levels within the same Procgen environment, a setting that has been explored by previous methods.\n\nW2) **Differences to previous works.** It appears that the proposed world model primarily follows the previous DIAMOND world model. It would be helpful if the differences could be elaborated.\n\nW3) **Lacking competitive baselines.** As a model-based method, the proposed method is mainly compared with some vanilla baselines in Table 1. While more advanced baselines (like [1-4]) and world models (like DreamerV3, TWM, STORM, IRIS, and DIAMOND mentioned by the authors in the related work).\n\nW4) **Missing visualization.** It would be great to show some predictions from the world model.\n\nW5) **Wrong reference.** In Line 312, the reference for STORM points to an unrelated paper. The correct one should be [5].\n\nW6) **A related reference.** [6] also proposes prioritized imagination for MBRL.\n\n---\n\n[1] Rethinking Decision Transformer via Hierarchical Reinforcement Learning. ICML 2024\n\n[2] Elastic Decision Transformer. NeurIPS 2023\n\n[3] IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies. arXiv 2023\n\n[4] AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimization. arXiv 2024\n\n[5] STORM: Efficient Stochastic Transformer based World Models for Reinforcement Learning. NeurIPS 2023\n\n[6] Pre-Trained Video Generative Models as World Simulators. arXiv 2025\n\nQUESTIONS:\nQ1) Any reason for choosing A2C as the agent learning algorithm?\n\nLIMITATIONS:\nPlease see the Weaknesses part.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces IMAC (Imagined Autocurricula), a novel approach combining world models with automatic curriculum learning to train rl policies. The key innovation is using Prioritized Level Replay (PLR) to create an emergent curriculum over \"imagined\" environments generated by a world model trained solely on offline data. The paper trains a diffusion world model from raw offline data, then uses it to generate diverse rl training scenarios. PLR automatically selects which imagined rollouts to prioritize based on learning potential, creating a natural curriculum that adapts to the agent's improving capabilities. This results in SOTA results vs. comparable models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper presents a successful combination of learned world models with unsupervised environment design (UED), opening a new research direction for training generally agents with online data, without simulators..\n- IMAC demonstrates substantial improvements (up to 48% on Jumper, 35% on Maze) over both model-free offline RL methods and fixed-horizon world model baselines, with consistent gains across all tested environments.\n- The approach automatically discovers effective curricula without manual design, naturally progressing from simple to complex scenarios.\n\nWeaknesses:\n- The approach only evaluates on 5 discrete Procgen environments with relatively simple visual observations; generalization to continuous control, high-dimensional state spaces, or real-world robotics remains unexplored.\n- Additional ablations on the bottlenecks posed by the world model, and scalability would be nice.\n\nQUESTIONS:\n- How well does this approach work under distribution shift between the input data used to learn the world model and the tasks that need to be completed? In a way the world model is now the weakest link. How well does it generalize?\n- Does the complexity of this approach scale with the complexity of the environment? It would be great to see some experiments where the environment is an ever more difficult function to learn, and seeing how the model becomes a bottleneck, and if this is feasible to scale to highly complex real world robotics environments.\n- What happens if we use the curriculum developed by one policy in one training run and use it to train another model from scratch? Is the online nature of the curriculum's development important or Is the curriculum good even when consumed offline.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}