Title: Dyn-O: Building Structured World Models with  Object-Centric Representations
Venue: NeurIPS 2025
Accepted: yes
Keywords: World Model

Object-centric representation
Abstract: World models aim to capture the dynamics of the environment, enabling agents to predict and plan for future states. In most scenarios of interest, the dynamics are highly centered on interactions among objects within the environment. This motivates the development of world models that operate on object-centric rather than monolithic representations, with the goal of more effectively capturing environment dynamics and enhancing compositional generalization. However, the development of object-centric world models has largely been explored in environments with limited visual complexity (such as basic geometries). It remains underexplored whether such models can be effective in more challenging settings. In this paper, we fill this gap by introducing Dyn-O, an enhanced structured world model built upon object-centric representations. Compared to prior work in object-centric representations, Dyn-O improves in both learning representations and modeling dynamics. On the challenging Procgen games, we demonstrate that our method can learn object-centric world models directly from pixel observations, outperforming DreamerV3 in rollout prediction accuracy. Furthermore, by decoupling object centric features into dynamic-agnostic and dynamic-aware components, we enable finer-grained manipulation of these features and generate more diverse imagined trajectories. The code of Dyn-O can be found at: https://github.com/wangzizhao/dyn-O.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an object-centric world model for better capturing the dynamics of the environment. The world model is also designed to be able to decouple object features into dynamics-agnostic and dynamics-aware components. The authors conduct experiments on the challenging Procgen benchmark and demonstrate their method's superior prediction accuracy compared to DreamerV3.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1.  The paper incorporates pretrained computer vision models like SAM2 into world model learning. This is a promising direction, as leveraging large-scale pretrained models for representation learning in reinforcement learning remains underexplored. This work narrows the gap in this area.\n\n2. The paper is well-structured, with clear illustrations and plots that effectively communicate the proposed method. The visualisations of reconstructed observations are helpful in understanding the model’s behaviour. \n\n**Weakness:**\n\n1. The primary goal of world models is to improve agent performance, yet the paper does not evaluate whether the learned representations actually benefit policy learning. While the method demonstrates improved reconstruction accuracy, this does not necessarily translate to better RL performance.\n    - Is the static-dynamics disentanglement truly beneficial for agents?\n    - If the agent can learn a good policy without such disentanglement, the additional complexity may be unjustified.\n    - Could the gains simply come from increased model capacity?\n\n    Without policy learning experiments, the practical utility of the proposed method remains unclear.\n\n\n\n2. The experiments rely on trajectories generated by a pretrained PPG agent, raising concerns about applicability in online settings:\n    - Does the method work in an online RL setting like DreamerV3? Online world models must learn from dynamically changing data distributions, and it is unclear whether the proposed approach would remain stable.\n    - Could training collapse when learning from scratch? Pretrained policies provide high-quality exploration data, but in online RL, poor initial exploration might lead to degenerate representations.\n\n3. The paper does not explicitly describe whether reconstruction metrics are computed on training or unseen test trajectories. Is the reported accuracy based on training rollouts? If so, the improvements could stem from overfitting rather than better generalisation.\n\nQUESTIONS:\n1.  The paper would be significantly more convincing if it included experiments evaluating the agent’s performance, either in a model-based or model-free setting. Currently, the focus on reconstruction accuracy leaves open whether the learned representations actually improve downstream policy learning. I would be willing to accept if this point is covered.\n\n2. The paper evaluates the method on 7 Procgen environments but does not justify how these were chosen from the full benchmark suite. A brief explanation (e.g., diversity of difficulty, visual complexity, or relevance to object-centric learning) would clarify the experimental design.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of learning a world model that decomposes observations into static parts and dynamic objects. The proposed method proceeds as follows: First, it encodes images using the COSMOS tokenizer [A], obtaining a fixed set of K latents representing individual objects via slot attention [B]. These latents are then used to generate reconstructed images. The overall network is trained to minimize reconstruction error, a standard approach in object-centric representation learning [B; C]. The method further improves upon this framework by leveraging a pre-trained SAM2 model [Ravi et al., 2024], utilizing its segmentation masks as attention masks for slot attention to ensure accurate assignment of individual objects to their corresponding slots.\n\nWith this learned object-centric representation, the method trains a world model to predict, given a set of slots and actions, the next-frame slots, a reward, and the probability of episode termination.\n\nThe method is evaluated on the Procgen dataset [Cobbe et al., 2020], which comprises procedurally generated 2D video game environments.\n\n[A] Agarwal, Niket, et al. \"Cosmos world foundation model platform for physical ai.\" arXiv preprint arXiv:2501.03575 (2025).\n\n[B] Locatello, F. et al. “Object-Centric Learning with Slot Attention.” Advances in Neural Information Processing Systems (NeurIPS), 2020.\n\n[C] Engelcke, Martin, et al. \"Genesis: Generative scene inference and sampling with object-centric latent representations.\" arXiv preprint arXiv:1907.13052 (2019).\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n* I like the paper’s premise that decomposing the static world and individual objects (i.e., learning a disentangled representation) facilitates effective world model learning.\n* The methodology’s representation is clearly presented and straightforward to follow.\n* The approach’s use of an object-centric latent space for world modeling is innovative and compelling, offering a novel perspective on structured representation learning.\n\n### Weaknesses\n\n* ***Experimental Validation***:\nThe paper states: “However, the development of object-centric world models has largely been explored in environments with limited visual complexity (such as basic geometries). It remains underexplored whether such models can be effective in more challenging settings.” I fully agree with this assessment; however, the evaluation on the Procgen dataset [A], which comprises procedurally generated 2D video game environments, mirrors the limited complexity of prior datasets [D]. Therefore, this paper's experimental validation does not address whether the proposed methodology is effective in more challenging, real-world environments. More critically, the method is tested solely on Procgen, without comparing its performance against baselines on datasets commonly used in prior work [B, C]. This omission prevents firm conclusions about whether the proposed methodology advances the field relative to existing approaches. Additionally, the world model’s experimental evaluation is limited beyond the object-centric representation. Could the authors clarify why broader datasets, such as the Arcade Learning Environment (ALE) [E], were not included?\n\n\n* **Unclear Relation to Prior Art**:\nThe paper presents two core contributions: object-centric representations and the world model. However, it lacks clarity on how these components compare, conceptually and empirically, to prior work. While the paper references DreamerV3 as a baseline, it reports results only on Procgen, whereas DreamerV3 demonstrates performance across multiple datasets, Procgen, ALE [E] and several others with greater diversity than 2D games. This limited comparison undermines the paper’s claim of advancing the state of the art.\n\nOverall, this work shows promise, but it requires significantly broader experimental validation to warrant presentation at a venue like NeurIPS.\n\n\nReferences:\n\n[A] Cobbe, K. et al. “Leveraging Procedural Generation to Benchmark Reinforcement Learning.” International Conference on Machine Learning (ICML), 2020.\n\n[B] Greff, K. et al. “Multi-Object Representation Learning with Iterative Variational Inference.” International Conference on Machine Learning (ICML), 2019.\n\n[C] Kabra, R. et al. “SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition.” Advances in Neural Information Processing Systems (NeurIPS), 2021.\n\n[D] Girdhar, R. and D. Ramanan. “CATER: A Diagnostic Dataset for Compositional Actions and Temporal Reasoning.” arXiv:1910.04744, 2019.\n\n[E] Bellemare, M. G. et al. “The Arcade Learning Environment: An Evaluation Platform for General Agents.” Journal of Artificial Intelligence Research, 2013.\n\nQUESTIONS:\n* The first sentence in the abstract is: \"World models aim to capture the dynamics of the environment\". Is this a fair statement? Intuitively, the first thing world model must learn is the static aspect of the world. Not to say learning dynamics isn't important -- it absolutely is -- but this emphasis doesn't sound right to me.\n\n* Abstract states: \"the dynamics are highly centered on interactions among objects within the environment\". Could the rebuttal clarify this? I am confused by this statement.\n\nLIMITATIONS:\nThe paper does not discuss limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper introduces a new method for learning world model with object-centric representations. Given an input video, the method first detect objects in the video, and then extracts latent feature representations are each object. These latent representations are used in a state-space model to predict their next representations given an action. The method further decomposes the object latent representations into a static part and a dynamic part. Experiments are conducted on the Procgen dataset to verify the world model.

STRENGTHS AND WEAKNESSES:
Positives:

-	The paper has introduced several novel ideas in my opinion such as using the state-space model for prediction, and decomposition of a static and a dynamic components of the object slot representations.

-	The paper also carefully designs what loss functions to use for learning the world model.

-	The experimental results show improvement compared to DreamerV3.

Negatives:

-	The experimental evaluation of the method seems not very thorough. For example, the comparison is only against DreamerV3. There is no other world model evaluated.

-	The experimental environments are all about games. More sophisticated setups can be considered such as using real-world videos.

-	Some ablation studies can be added. For example, how important is using the state-space model? What if some other models are used?

QUESTIONS:
Please see weaknesses above regarding questions.

LIMITATIONS:
No limitations discussed in the paper.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

comments: SUMMARY:
In this paper the authors present the Dyn-O model that learns world models by both learning object representations and a dynamics model to transition between those representations. Crucially, Dyn-O segments object representations into static properties that cannot change (e.g., the object sprite) and dynamics properties that can (e.g., position). Dyn-O is tested on seven different Procgen environments, outperforming DreamerV3 – an alternate world learning model – and on one environment disentangles the static and dynamic features appropriately

STRENGTHS AND WEAKNESSES:
This work presents an interesting idea for world model, especially in segmenting out the static and dynamic features to better capture real-world object structure. While relatively simple environments, using seven Procgen tasks is good for capturing diverse dynamics. However, the results report data that is either only aggregated across environments, or listed for a single environment at a time, making it difficult to assess how general the results are across different types of environments and dynamics

QUESTIONS:
My main question is how the findings differ and/or generalize across the different Procgen environments. For instance, Table 1 reports results aggregated for all environments, but it would be useful to assess how Dyn-O performs vs Dreamer or the non-object-centric version across all environments. Similarly, evaluating the object-centric representations is only done on BigFish and the static/dynamic disentanglements only on CoinRun – how do these findings generalize to the other environments?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-02