Dyn-O: Building Structured World Models with  Object-Centric Representations
Authors: Zizhao Wang
Kaixin Wang
Li Zhao
Peter Stone
Jiang Bian
Venue: NeurIPS 2025 poster ()
Keywords: World Model
Object-centric representation
Primary area: reinforcement_learning

Abstract: World models aim to capture the dynamics of the environment, enabling agents to predict and plan for future states. In most scenarios of interest, the dynamics are highly centered on interactions among objects within the environment. This motivates the development of world models that operate on object-centric rather than monolithic representations, with the goal of more effectively capturing environment dynamics and enhancing compositional generalization. However, the development of object-centric world models has largely been explored in environments with limited visual complexity (such as basic geometries). It remains underexplored whether such models can be effective in more challenging settings. In this paper, we fill this gap by introducing Dyn-O, an enhanced structured world model built upon object-centric representations. Compared to prior work in object-centric representations, Dyn-O improves in both learning representations and modeling dynamics. On the challenging Procgen games, we demonstrate that our method can learn object-centric world models directly from pixel observations, outperforming DreamerV3 in rollout prediction accuracy. Furthermore, by decoupling object centric features into dynamic-agnostic and dynamic-aware components, we enable finer-grained manipulation of these features and generate more diverse imagined trajectories. The code of Dyn-O can be found at: https://github.com/wangzizhao/dyn-O.

Dyn-O: Building Structured World Models with Object-Centric Representations
Zizhao Wang Kaixin Wang? Li Zhao? Peter Stone 4.
Jiang Bian? 'Sony Al
'The University of Texas Austin "Microsoft Research Asia zizhao. wang@utexas. adu,pstoneeee utexas. edu (kaixwang, 1izo, jiang. bian) eeeeemiirroooo wwm.irosso..cmmm com
Abstract
World models aim capture the dynamics of the environment, enabling agents predict and plan for future states. In most scemarios of interest, the dynamics are highly centered on interactions among objects within the environment. This moti- vates the development of world models that operate on object-centric rather than monolithic representations. with the goal of more effectively capturing environment dynamics and enhancing compositional generalization. However, the development of object-centric world models has largely been explored in environments with limited visual complexity (such as basic geometries). It remains underexplored whether such models can effective more challenging settings. this paper, we fill this gap by introducing Dyn-O, enhanced structured world model built upon object-centric representations. Compared to prior work in object-centric representa- tions, Dyn-O improves in both learning representations and modeling dynamics, On the challenging Proegen games, we demonstrate that our method can learn object- centric world models directly from pixel observations, outperforming Dreamer V3 in rollout prediction accuracy. Furthermore, by decoupling object-centric features and dynamic-awan
into jyynamic-gnoosie manipulation of these features and generate more diverse imagined trajectories. The code of Dyn-O can be found at: https: https:://githul wtt.thhw..ccccceeeeoeeoeeeeeeeet wwm/wanggizhao/dyy...oooo
components, enable finer-grained
Introduction
World models have emerged powerful tools for simulating environment dynamics, enabling agents to predict and plan for the future [16 17 20
21 39 A common design in these models to map high- nigh-dimensional observations (e.g.. images) into latent features and then model the environment's dynamics in this latent space. However, these latent features are typically monolithic, encoding the entire scene as whole without accounting for its internal compositional structure. Yet, interactions in most environments are inherently object-centric. This observation motivates the development of object-centric world models, which can offer improved efficiency, interpretability, and compositional generalization.
Building object-centric world models involves two key steps: 1) learning object-centric represen- tations and 2) modeling dynamics on top of them, as illustrated in Figure []] For the former, the goal is encode features associated with each object present in the observation. For the latter, the dynamics model should account for the different representation space, in contrast to monolithic representation, the input and the prediction target a object-centric representations. While several prior approaches have explored object-centric world models, they primarily focus on simple environments consisting of basic shapes
13
or rely on externally provided compositional
"work done during an internship at Microsoft Research Asia
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

OC Feature Encoder
Dynamics Model
object-centric features
object-centric features @ t+1
Figure A high-level overview of the object-centric (OC) world model framework. The latent features are not monolithic or patch-based, but instead are bound to the objects present in the scene.
signals such as language 46
remains underexplored whether object-centric world models can be learned purely from trajectories to capture complex dynamics in more challenging, complex settings.
As step toward bridging this gap, we introduce Dyn-O, novel object-centric world model with improved designs for both object-centric representation learning and dynamics modeling over prior work. To handle complex visual observations, we draw inspiration from prior work on learning object-centric representations from real-world videos (e.g.. SOLV (I) and adopt an autoencoder-style architecture embed observations into object-associated features, referred as slots. Specifically, we adopt pre-trained Cosmos encoder [12,, which offers two key advantages compared to the DINO encoder [34] used in SOLV: (1) improved representation quality, and (2) access to pretrained visual decoder, eliminating the need to train decoder from scratch for reconstructing pixel observations. the extracted slot features, we incorporate priors from a high- performing pretrained segmentation model, SAM2 (35). While this incorporation significantly improves performance, it also introduces substantial computational overhead, particularly during inference. To mitigate this issue, we use scheduling strategy that gradually reduces reliance on the segmentation mask during training, enabling the model to maintain performance inference time without requiring the mask.
To further enhance the quality
The extracted slot features are modular in nature, with each slot associated with distinct object. To model transition dynamics in this slot space, we aim for the world model to respect this modular structure. To this end, we adopt state-space model (SSM) based on the Mamba architecture (15). borrowing the idea from SlotSSM [23]. Unlike SlotSSM, which tightly integrates the slot encoder and decoder within each transformer block of the SSM, we use the pretrained slot representation module introduced above and apply the SSM solely for dynamics modeling. This decoupling of representation learning and dynamics modeling aligns with recent trends in the world model literature ( 30 Additionally, we explore disentangling each object's slot feature into static component capturing time-invariane properties (e.g., texture) and a dynamic component encoding time-varying properties (e.g., position). This disentanglement enables fine-grained manipulation of slot features. For instance, we can modify the static feature of one object while keeping its dynamics unchanged, allowing for diverse data generation during world model rollouts.
We evaluate Dyn-O seven Procgen environments. Our experiments indicate that Dyn-O learns high-quality object-centric representations, generalizable world models, and disentangled static and dynamic representations. We summarize our contributions as follows.
We propose novel object-centric representation learning method by leveraging segmentation masks with a dropout schedule, enhancing representation quality while keeping efficient inference.
We propose a novel object-centric world model that uses state-space models as backbones and outperforms monolithic models in both rollout quality and generalization.
We propose novel object-centric representation that disentangles each object's representation into static and dynamic features, allowing the generation of diverse rollouts by altering static attributes while preserving dynamic behavior.
2
Related Work
World Models Accurately and efficiently modeling the world's dynamics has been long-standing topic in reinforcement learning [39]. A learned model of a simulated environment can be used to

---

facilitate planning or generate imagined data for training policy. In recent years, world models have seen great progress powered by advances in deep neural networks, Ha and Schmidhuber introduce generative world model where the dynamies is learned entirely in a latent feature space. In similar fashion, the Dreamer line of work [77 has further advanced the performance of world models. More recently, several works have explored using sequence modeling techniques to train world models within token representation space
30].
Our works differs in that we build a world model on top of an object-centric representation, while prior work mainly focuses on monolithic representations. Closely related our work are those that also consider the compositional nature of the world. Specifically, HOWM [45] and Cosmos [36] design object-centric world models to address the compositional generalization problem. DreamWeaver Weaver [ uses a novel Recurrent Block-Slot Unit to discover compositional representations and generate compositional future simulations. However, those approaches mainly focus on small-scale diagnostic environments such as 2D block pushing, without validating in more complicated environments. RoboDreamer [46] learns a compositional world for robot manipulation tasks, but their method relies on language and leverages its natural compositionality. In comparison, our method learns the object-centric world model purely from agent trajectories.
Object-Centric Representations Object-centric representations have gained increasing attention in recent years 6 [ 22 28 37). Instead of encoding an image observation as single monolithic latent vector, object-centric representation learning aims to decompose a scene into objects and to learn different latent representations for each object. A key milestone in this direction was Slot Attention [28]. which has since become the foundation for many subsequent methods that learn object-centric features in an unsupervised manner from images 22
or videos D 10 25
Our work builds on the SOLV framework (. introducing improvements to enhance the extraction of object-specific latent features. Another closely related approach the Slot State Space Model (SlotSSM) [23]. which uses state-space models to capture long-range temporal dependencies in structured way. However, has only been validated on domains with limited complexity, such as basic shapes. Moreover, SlotSSM tightly couples slot representation learning and dynamics modeling, training them jointly. In contrast, we first learn strong slot feature encoder and then build object-centric world models on top of it.
Disentangling Static and Dynamic Features The disentanglement between static and dynamic features in sequences has been studied in computer vision [5] 31 38). DSVAE 44 divides the latent representation into the static factors and the dynamic factors and learns them jointly with the
ELBO objective. However, as formalized by Locatello et al. (27). unsupervised disentanglement
impossible to achieve without inductive biases. C-DSVAE 3 further adds contrastive learning to encourage disentanglement, yet its learned dynamic features highly depend on the chosen data augmentations and can still capture static information. ContextWM [4T] incorporates the notion of time-invariant context and time-varying latent variables into world models. However, using the same ELBO objectives as DSVAE, may still learn entangled representations.
3 Method
From a high level perspective, Dyn-O models the enviromment's 's dynamics in an object-centric manner, enabled by two learning phases (Figure 2p.
Object-centric representation learning (Section 3.1p: b: Dyn-O learns an object-centric representa-
tion in which coherent objects are each encoded into independent features, referred to as slots. This factorized representation, in contrast to a monolithic scene-level encoding, leverages the natural compositionality of objects in the world.
Dynamics learning (Section 3.2): Dyn-O adopts State Space Model (SSM) to predict transitions from the current slots to the next-step slots, conditioned on the action. Each object's 's slot feature is further decoupled into static and dynamic components enable fine-grained manipulation.
3.1 Extracting Object-Centric Representations
In the first learning phase, Dyn-O learns an object-centric representation from image observations. In contrast to monolithic representation that mixes all objects' information, this factored representa-
3

---

(a)
Reconstructed image
(b)
h2
Decoder
SSM
Self-Attn
Pred
Slot Decoder
SSM
Slot Attention
21-1
Dyn-O Encoder
Encoder
Downsample
Attn
Cross
Attn
Cross
ddb
Image
SAM mask
Figure 2: Components of Dyn-O: (a) object-centric representation learning; (b) dynamics learning. Modules marked with * are fixed, while others are learnable. The "Dyn-O Encoder" in (b) corre- sponds to the lower half of (a), which maps the image the latent slot feature 2. See Section 3 for details.
tion enables Dyn-O to modify each object independently and generate novel object combinations. Formally, shown in Figure 2]a), Dyn-O learns an encoder Enc O + Z to extract an object- centric representation z E 2 from observations o, where the representation consists of slots, ................................ 2 2K], each corresponding to object. Here, K is predefined hyperparameter, and when the number of objects in a scene smaller than K, some slots may remain unused.
During encoder learning, inspired by SOLV M. instead of training from scratch with raw pixels, Dyn- learns on top the high-quality features extracted by the Cosmos tokenizer [[2]. Given an input frame R ffxWx3 Dyn-O first applies the Cosmos encoder (CosmosEnc) to extract patch-level features E NNx,, where N denotes the number of patches in the frame. It then initializes K slots Eiit E T xd. from a set of learnable vectors. These slots compete to bind to objects using iterative Slot-Attn f). The learning signals for slot extraction arise from reconstructions a decoder (Slot-Dec) reconstructs features from the slots, which are then used to reconstruct the observation via the Cosmos decoder as ô = CosmosDec(f) f). That is,
Slot Attention
28 producing
f CosmosEnc(o))
2 = Slot-Attm Finit: f);
f Slot-Dec(z).
Ô = CosmosDec(f).
The Slot-Attn and Slot-Dec modules trained jointly by minimizing the reconstruction error as follows, while the Cosmos encoder-decoder remains frozen:
Estol = fl fl + 11o - 6||2.
(1)
Empirically, we observe that this fully unsupervised training objective usually results in inaccurate object-slot bindings (see examples in Section 4.I).. To improve the quality of object-centric repre- sentations, leverage the prior provided by Foundational segmentation models. Specifically, use a pre-trained SAM2 model [35) to generate a segmentation mask m E {0, 1) H x and use it an attention mask during slot attention, segmented object and constraining it to only attend to patches from that object.
Slot Slot-Attn( -tttnnnnttttt..ta (Amit m), binding each slot one
While the segmentation mask enhances object-slot binding, it also introduces substantial computa- tional overhead, especially during inference. This dependency could limit the practicality of Dyn-O. For example, when extracting slots for an agent interacting with the environment in online setting, we would need to run SAM2 inference every environment step, which would be prohibitively expensive compared to the typical cost per step. To address this issue, during encoder lcarning,

---

(a) Static-dynamic disentanglement
(b) Static feature learning
(c) Dynamic feature learning
Ct
Ct'
Mc
from the same slot?
pull together
M
Mc
Md
Mz
24
push apart
2
Disc
Md
X
Dyn
Mc
M
Zt
Ut
Vt+1
Ct
Figure 3: Illustration of (a) the overall design for disentangling slot features in dynamics modeling, and the training procedures for (b) static features and (c) dynamic features. // indicates coo-graaient operation.
introduce an annealing schedule that gradually reduces the reliance on the segmentation mask, aiming climinate its use by the end of training. Initially, the segmentation mask is always used to guide slot extraction. As training progresses, the probability of not using the mask increases according to a loga- rithmic schedule w.r.t. the number of network updates, log(1 + updates) log(1+ +1++# total updates). This dropout schedule allows the encoder to achieve the best of both worlds learns high-quality representations with the initial guidance of segmentation masks, while enabling efficient inference by phasing out the need for them.
3.2 World Model with Object- Centric Representations
After learning object-centric representations as described above, the next phase Dyn-O focuses training world model to reason actions age, the world model predicts the next slots, the reward, and whether the episode terminates as 444 fr, de Dyn(zgı, ag.).
about object interactions. Given the history of slots 2gg and
World Model For the world model Dyn, we adopt state-space models (SSMs) for their strength capturing long-range temporal dependencies. Meanwhile, the model needs to account for the permutation equivariance among slots - if the input slots are randomly permuted, the slots' prediction should follow the same permutation. Therefore, shown in Figure we design Dyn as follows. We first apply self-attention to extract information about object interactions:
Self Self-Attn(q
kv [=]...., K aa)),
where position encoding is used maintain permutation invariance. Next, each slot is processed by a shared SSM to update its hidden state, which is then used by Dyn-O to predict the next slots, reward, and whether the episode terminates using separate prediction networks as follow:
h = SSM(h ={), e Pred(h++1)
A Cross-Attmn Tel: kv = [1 11))))s
d Cross-Attn(g dea, kv [1.41 MK
where h is the hidden state of the SSM that tracks past information for the k-th slot, and rets and dets learnable query tokens used to extract reward and termination signals from the hidden states. Finally, the world model is optimized by minimizing the following prediction loss (Alg, I 4)
Lwm Σ
سلق 2441 1
Σ
rxll CE(di,d))
where CE stands for cross-entropy loss for binary classification.
Denti--Dynamic Disentanglement The object-centric representation learned in Section 3.1 enables the creation of scenes with novel object combinations. However, when synthesizing data, it is often desirable to only modify object appearance (e.g., colors) while preserving their dynamic properties
5

---

(e.g., positions). For example, modifying dynamic information could result in invalid scenes, such as a table being moved while the objects originally on it remain suspended in midair.
To this end, as shown in Figure aa), Dyn-O disentangles each slot = into two components: static features e TRd= that captures time-invariant properties and dynamic features v² IRd. that encodes time-variant properties. This decomposition allows us to modify an object's static features while keeping its dynamic features unchanged, generating novel scenarios with consistent dynamics. Formally, Dyn-O maps slots to static and dynamic features with two separate neural networks: ct = (M((2*) and Eh (z)) We describe how these features are learned below.
Static features are intended to capture time-inve invariint properties. Thus, for an object present at timestep t, its static feature should remain the same across all timesteps. Therefore, natural training objective is to minimize the difference between static features across all timesep pairs. However, this objective alone admits degenerate solution where all static features collapse collapse, Dyn-Q incorporates contrastive learning to encourage feature diversity, as illustrated in Figure bb): static features should be similar if they belong to the same slot, and distinct otherwise.
constant. To prevent this
Dyn-O therefore learns static features by optimizing the following loss (Alg, I line 5):
cos(c), ott cos(ct, ct) + EzEe cos(ct
Sttt =
Σ1 k. time invariance
lc
Cy
112
log
contrustive learning
where cos denotes cosine similarity, and the second term corresponds to the InfoNCE loss [33. For each ct, the positive sample e₁ comes from the same slot different timestep, while the negative samples Ö are drawn from other slots in the batch.
On the other hand, Dyn-O learns dynamic features by reconstructing the slot content while ensuring their disentanglement from static features. Specifically, Dyn-O uses reconstruction network M, to output &* = M₂ (sg(c") ) where sg denotes gradgian,nnt,, preventing the static feature ch from being updated by the reconstruction loss. (For simplicity, in this paragraph, we omit the subscript in 2, c and r). However, minimizing reconstruction loss alone may lead the dynamic features to encode time-invariant information well, bypassing the need for ck To avoid this effect, Dyn-O promotes disentanglement by minimizing the mutual information Σ., [cc', uff) via adversarial training, as illustrated in Figure Ci.. A discriminator Disc trained to distinguish whether a pair of static and dynamic features comes from the same slot. While Disc minimizes the discrimination loss, the dynamic feature extractor M, is trained to maximize it, thereby reducing the static information encoded in dynamic features [14). For stability, we adopt the Wasserstein distance as the discrimination loss and apply LeCam regularization [40] (Alg. lines 6-7):
Layn
Σ || 1 Disc(c
(4)
reconstruction
disentanglement
Laiss
Σ Disc(c Disc(c ,a
+ LeCam(Disc)
(5)
regularization
discrimination
where (c ) are from the same slot, and (c* d'') are from different slots.
4 Experimental Evaluation
We evaluate Dyn-O to answer the following questions: QI: Can Dyn-O learn accurate object-centric representations (Sec. 4 Yes. Q2: Can Dyn-O learn accurate world models (Sec. 4.3 Yes. Q3:
Are static and dynamic features learned by Dyn-O O truly disentangled (Sec.
4.4 Yes.
Our experiments are conducted Procgen 7 a set of pvocedurally-generaae 2D video game environments. We use 7 Procgen environments: bigfish, coinrun, caveflyer, dodgeball, jumper, ninja, and starpilot. each Procgen environment, a PPG policy [8] is trained and used to collect an offline dataset of IM transitions from the first 200 levels for the learning of all methods.

---

Algorithm Dyn-O World Model Learning
1: Collect dataset of (ot, de, Tt, 0t+1). Initialize object-centric encoder (Enc), mappings static features (M,), dynamic features (M,), and slots (M2), the world model (Dyn), and the discriminator (Disc).
2: Train object-centric encoder Enc with Eq. (
3: Train the world model with joint optimization:
4: Update the world model Dyn by minimizing prediction losses in Eq.
5: if learn static-dynamic disentanglement then
6: Update static features from Me by enforcing time-invariann in Eq.
7: Update Disc with Eq. (4) to estimate mutual information.
8:
Update dynamic feature from M, via reconstruction M, and disentanglement loss in Eq. 5
Method
FR-ARI (1)
Oracle
0.96
Dyn-O (ours)
0.80
SOLV
0.54
Oracle Dyn-O (ours)
SOLV
(d) slot-object binding accuracy
Figure 4: Evaluation of the object-centric representation learning bigfish. (a) Using segmentation masks during inference, Oracle achieves the most accurate slot-object binding, but also suffering from the high computational overhead. (b) By using segmentation masks only during training. Dyn-O learns to accurately assign each fish to one slot (shown as colored patches), which avoiding inference-time overhead. contrast, learning without the guidance of segmentation masks, SOLV often inaccurately splits fish into multiple separate slots.
4.1 Evaluating Object-Centric Representation
As Dyn-O's world model is learned top of the object-centric representations, the quality of learned representation is critical to its prediction accuracy. We compare Dyn-O's representation learning against the following ablations:
Oracle: our method but always using the segmentation mask during both training and inference.
SOLV MI:
the same our method but does
segmentation mask for training and inference.
As shown in Fig. compared to SOLV that often splits an object into multiple slots, Dyn-O achieves more accurate object-slot binding, assigning each object to single slot. We also evaluate the foreground adjusted rand index (FG-ARI) which a widely used metric in the object-centric literature that measures the similarity of the discovered objects masks ground-truth masks. Again, Dyn-O outperforms SOLV, suggesting the benefit of using segmentation masks to guide representation learning.
4.2 Evaluating World Model Accuracy
The promise of Dyn-O is learn accurate and and generalizable world models based on object-centric representations. Therefore, most critical evaluation of our work focuses on the world model quality, and we compare Dyn-O against the following baselines and ablations:
DreamerV3 119 one of the state-of-the-art model-based RL methods.
Dreamweaver an object-centric world model designed to discover hierarchical and composi-
tional representations.
Dyn-O without object-centrie representations (denoted as Dyn-O w/o OC): method but without object-centric representations. The dynamics model is learned on top 14x14 patch-level features extracted by Cosmos encoder, where each patch is treated "slot".
For fair comparison, we use the same frozen Cosmos tokenizer for DreamerV3 and Dreamweaver Dyn-O during world model learning.
7

---

t=0
Dyn-O
40
180
Dreamerv3
w w OC
Dyn-O
2-3
t-5
t+7
t-10
b-15
t-20
t-0
Dyn-O
Dreame NV
Dyn-O w OC
t-5
t-10
t-15
t-20
Figure 5: Dyn-O generates more accurate rollouts than Dreamer in bigfish and coinrun. In each environment, the first row displays trajectory collected in the real environment. The second row depicts the prediction inside the world model by Dyn-O (ours). The third and fourth rows show the prediction of Dreamer and Dyn-O w/o OC respectively. In bigfish, our method keeps consistent prediction for each fish until the 15th step, while baselines lose track of multiple small fish before the 10th step. Similarly, in coinrun, compared to baselines, Dyn-O generates predictions with clearer floor and boxes.
To evaluate the generalizability of each method, the learned world model generate 20-step rollouts in 500 unseen levels. The evaluation metric covers pixel-level quality (PSNR) as well as temporal and spatial coherence (FVD, LPIPS, and SSIM). The results are shown Table [ where we average the evaluation at the last step across all environments. The results for each environment can be found Appendix B.3 Dyn-O significantly outperforms dreamer which uses monolithic representation, demonstrating the advantage of predicting in the object-centric representation space. Meanwhile, in contrast Dyn-O w/o OC whose latent representation consist of 196 patch-level "slots", Dyn-O only uses 31 or 47 object-level slots (where each object-level slot has same dimension as a patch-level "slot") and achieves higher performance, demonstrating the superior efficiency of object-centric representations. The results for Dreamwaver can be found in Appendix B.3 To complement the analysis with qualitative examples, Fig. shows the generated rollouts of Dyn-O and its comparison with baselines.

---

Original awaaped
Swap Agent
Swap Floor
T=0
10
15
20
Figure 6: Dya-O generates dynamically consistent rollouts after exchanging static features. In the top two rows, swap the static features of the avatar (the small agent in the center of the image) between two initial states and generate 30-step rollouts using Dyn-O. The results show that only the avatar's color changes, while all other objects remain unchanged. the bottom two rows, we exchange the static features the floor, and Dyn-O consistently swaps their colors while keeping all other objects intact.
Table Quantitative results for rollout trajectories, averaged across all environments.
Method
LPIPS ())
FVD (L)
SSIM (t)
PSNR (t)
DreamerV3
0.42
692.5
0.56
15.70
Dyn-O w/o OC
0.41
538.4
0.53
16.10
Dyn-O (ours)
0.33
361.3
0.62
16.34
In addition to Procgen, we further compare Dyn-O against DreamerV3 on the CLEVR dataset 24 and two ALE environments [4) and the results are Appendix B.3
4.3 Evaluating Representation Effecctvvneess for Policy Learning
Next, we evaluate whether the learned object-centric representations can facilitate policy learning. To do so, we fix the representation backbone and train only the policy head using the PPG algorithm on three Procgen games: Bigfish, Starpilot, and Coinrun, with three random seeds for each game. We use the "easy" difficulty setting with unlimited number of levels.
We compare the performance of policies using object-centric representations learned by Dyn-O against policies using representations directly output by the Cosmos encoder, and the policy performance (measured as reward) are shown in Table. 3 Policies using object-centric representations achieve much higher reward than policies with raw patch-level features, demonstrating the effectiveness of Dyn-O's representations for downstream task learning.
Table 2: Probing accuracy (f), percentage (%), coinrun privilege properties, shown the and standard deviation Static features have much higher prediction accuracy than dynamic features for static properties (i.c., RGB values), while dynamic features have higher accuracy on dynamic properties (i.c., position and area), demonstrating that Dyn-O O achieves effective static dynti- disentanglement
Static Properties G value
Dynamic Properties position
value
B value
x position
area
slots
83,5 - 0.0
81.7 0.0
89.3 0.0
91.8 + 0.0
94.5 + 0.0
97.1 0.0
dynamic features
47.7 = 7.3
45.7 + 7.3
47.0 8.9
78,1 + 5.1
75.2
25.2 6.6
83.3 3.1
static features
67.3 + 1.4
70.9 + 1.9
81.7 2.2
34,9 1.2 1.2
4.0
75.3 + 0.4
random features
25.8 0.0
27.3
23.2 + 0.0
29.3 +
28.3
73.3 + 0.0

---

Table 3: Reward of policies using different representations, measured by the mean and standard deviation across three random seeds.
Representation
Bigfish
Starpilot
Coinrun
Cosmos encoder
0.90 + 0.02
8.80 + 0.47
8.00 + 0.18
Dyn-O (ours)
7.67 4.02
19.19 0.88
8.95 + 0.38
4.4 Evaluating Static-Dynamic Disentanglement
To examine whether the static and dynamic features learned by Dyn-O are disentangled, We probe the model with petrroomment-rivilggeeee information, evaluating whether static features only capture time- invariant properties and whether dynamic features only capture time-varying properties. Specifically, use SAM to segment out objects in 10K transitions and extract the following properties for each object as probing targets: area (i.e., the number of pixels), xy positions, and average RGB values across object pixels. Then we compute each object's slot, static feature, and dynamic feature as probing inputs. During probing, we discretize each property into 10 bins and use a linear classifier to predict the target. To establish reference point for interpretation, we also perform the same prediction using randomly initialized features, representing the performance expected when no meaningful information is available in the inputs. Table 2 shows the prediction accuracy of all features in the coinrun environment (see results for other environments in the Appendix). Dyn-O shows strong disentanglement results, particularly as the prediction accuracy of static features for position and area is close to that of random features, indicating that static features capture little to no information about dynamic properties.
We further illustrate the disentanglement between static and dynamic features with qualitative examples. Fig. 6 presents rollouts generated after swapping static features between initial states. As shown, rollouts from the same initial states (the first and third rows) remain nearly identical, except for the color changes in the avatar and floor. The same pattern observed in the second and fourth rows, demonstrating that Dyn-O preserves dynamics while modifying only static properties.
5 Conclusion
We present Dyn-O, a world modeling method that builds on object-centric representations. By lever- aging segmentation masks during training with a schedule, Dyn-O learns high-quality object-centric representations while avoiding the overhead of segmentation computation at inference time. Addition- ally, Oyn-O further disentangles each object feature into static, time-invariant components (e.g., color) and dynamic, time-variant components (e.g.. position), allowing it to generate diverse predictions. Dyn-O predicts in latent space, with a pretrained decoder used for visualization. Exploring more sophisticated decoding methods, such as diffusion models, could further enhance the visual quality of generated rollouts.
Acknowledgements
The majority of this work has taken place in the Learning Agents Research Group (LARG) the Artificial Intelligence Laboratory. The University of Texas at Austin. LARG research is supported in part by the National Science Foundation (FAIN-2019844 Research (N00014-24-1 1-25500,, Army Research Office (W911NF-11-20)) W91 25-1-655))) DARPA (Cooperative Agreement HR0011252006 on Ad Hoc Teamwork), Lockheed Martin, and Good Systems, a research grand challenge at the University of Texas at Austin. The views and conclusions contained this document are those of the authors alone. Peter Stone serves as the Chief Scientist of Sony AI and receives financial compensation for that role. The terms of this arrangement have been reviewed and approved by the University of Texas Austin in accordance with its policy objectivity in research.
NRT-2125858), the Office of Naval
W91 1011NNF-33-2000
10

---

References
[1] G. Aydemir, W. Xic, and F. Güney. Self-supervised Object-centric Learning for Videos. In Advances in Neural Information Processing Systems, 2023.
[2] J. Back, Y.-F. Wu, G. Singh, and S. Ahn. Dreamweaver: Learning compositional world representations from pixels. arXiv preprint arXiv: 2501.14174, 2025.
[3] Bai, W. Wang, and C. P. Gomes. Contrastively disentangled sequential variational autoencoder. Advances in Neural Information Processing Systems, 34:10105-10118 2021.
[4] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013.
[5] N. Berman, Naiman, I. Arbiv, G. Fadlon, and O. Azencot. Sequential disentanglement by extracting static information from a single sequence element. arXiv preprint arXiv:2406.18131, 2024.
[6] C. P. Burgess, L. Matthey, N. Watters, R. Kabra, I. Higgins, M. Botvinick, and A. Lerch- ner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv: 1901. 2019.
[7] K. Cobbe, C. Hesse, J. Hilton, and J. Schulman. Leveraging procedural generation benchmark reinforcement learning, arXiv preprint arXiv: 2019.
[8] K. W. Cobbe, J. Hilton, O. Klimov, and J. Schulman. Phasic policy gradient. In International Conference Machine Learning, pages 1020-20217 PMLR, 2021.
[9] I. Cohen, K. Wang, B. Kang, and S. Mannor. Improving token-based world models with parallel observation prediction. In Forty-first International Conference on Machine Learning, ICML. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. [tLLittts///ppperevviee net/ pet/foruumiddLLfpsDDttttt
[10] G. F. Elsayed, A. Mahendran, S. van Steenkiste, K. Greff, M. Mozer, and T. Kipf. Savi++: To- wards end-to-end object-centric learning from real-world videos. Neural Information Processing Systems, 2022. doi: 00.48550/arriw.2206.7700 2776.7776.
[11] M. Engelcke, A. R. Kosiorek, O. P. Jones, and I. Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. arXiv preprint arXiv: 2019.
[12] N. et. al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575. 2025.
[13] S. Ferraro, P. Mazzaglia, T. Verbelen, and Dhoedt. Focus: Object-centric world models for robotics manipulation. arXiv preprint arXiv: 2307.02427, 2023.
[14] Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. In Interna- tional conference on machine learning, pages 1180-1189. PMLR, 2015.
[15] A. Gu and T. Dao. Mamba: Linear- time sequence modeling with selective state spaces, arXiv preprint arXiv: 2312.00752, 2023.
[16] D. Ha and J. Schmidhuber. Recurrent world models facilitate policy evolution. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianche and R. Garnett, ed- itors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018 /ttss newrips..... cttpcc/ppppppepeeeeeeeee http:/ae...llleeeeee//////////// 22ec6 Paper. Paper pdff
[17] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: 1912.01603, 2019.
[18] D. Hafner, Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models. International Conference on Learning Representations, 2020.

---

[19] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse domains through world models, arXiv preprint arXiv:2301.04100 2023.
[20] D. Hafner, J. Pasukonis. J. Ba, and T. Lillicrap. Mastering diverse control tasks through world models. Nature, 640(8059) Apr. 2025.
[21] N. Hansen, H. Su, and X. Wang. Td-mpe2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16825 2023.
[22] J. Jiang, F. Deng, G. Singh, and S. Ahn. Object-centric slot diffusion. Neural Information Processing Systems, 2023. doi: 10.48550/arXiv.2203..
[23] J. Jiang. F. Deng, G. Singh, M. Lee, and S. Ahn. Slot state space models. arXiv preprint arXiv: 2406. 12272, 2024.
[24] J. Johnson, B. Hariharan, L. Van Der Maaten, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2901- 2910, 2017.
[25] T. Kipf, G. Elsayed, A. Mahendran, A. Stone, S. Sabour, G. Heigold, R. Jonschkowski, A. Dosovitskiy, and K. Greff. Conditional object-centric learning from video. arXiv preprint arXiv:2111.12594. 2021.
[26] B. Liu, X. Liu, X. Jin, P. Stone, and Q. Liu. Conflict-averse gradient descent for multi-task learning, Advances in Neural Information Processing Systems, 34, 2021.
[27] F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. Chal- lenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 41 14 4124. PMLR, 2019.
[28] F. Locatello, D. Weissenborn, T. Unterthiner. A. Mahendran, G. Heigold, J. Uszkoreit, A. Doso- vitskiy, and T. Kipf. Object-centric learning with slot attention, 2020.
[29] V. Micheli, Alonso, and F. Fleuret. Transformers are sample-efficient world models. Interna- tional Conference on Learning Representations, 2022. doi: 10.48550//rXii2220..0555 000088.
[30] V. Micheli, E. Alonso, and F. Fleuret. Efficient world models with context-aware tokenizziion.... In Forty-first International Conference on Machine Learning, 2024. URL httpt///////oeerreiiiee met//oruumidd:BWWnee
[31] D. Miladinović, M. W. Gondal, B. Schölkopf, J. M. Buhmann, and S. Bauer. Disentangled state space representations. arXiv preprint D::Xiv:99000025555 2019.
[32] A. Nakano, M. Suzuki, and Y. Matsuo. Interaction- based disentanglement of entities for object- centric world models. In The Eleventh International Conference on Learning Representations, 2023. URL
[33] A. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 7807003748 2018
[34] M. Oquab, T. Darcet, Moutakanni, H. Vo, M. Szafraniec, Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, R. Howes, P.-Y. Huang, H. Xu, V. Sharma, S.-W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, P. LLbatut, A. Joulin, and P. Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
[35] N. Ravi, Gabeur, Y.-T. Hu, Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, E. Mintun, J. Pan. K. V. Alwala, N. Carion, C.-Y. Wu, R. Girshick. P. Dol- áár, and C. Feichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint wwXi::2408.00714 2024. URL https: org/abs/ hgg/abss/408 2408 000044
[36] A. Sehgal, A. Grayeli, J. J. Sun, and S. Chaudhuri. Neurosymbolic grounding for compositional world models. International Conference on Learning Representations, 2023. doi: 10.48550/ 1iiv.2310.12690
12

---

[37] M. Seitzer, M. Horn, A. Zadaianchuk, D. Zietlow. T. Xiao, C.-J. Gabnonl T. He, Z. Zhang, B. Schölkopf, T. Brox, and Locatello. Bridging the gap to real-world object-centric learning. arXiv preprini arXiv: 14860, 2022.
[38] M. C. Simon, P. Frossard, and C. D. Vleeschouwer. Sequential representation learning via static-dynamic conditional disentanglement. In European Conference on Computer Vision, pages 110-126. Springer, 2024.
[39] R. Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning proceedings 1990, pages 216-224. Elsevier, 1990.
[40] H.-Y. Tseng, L. Jiang, C. Liu, M.-H. Yang, and W. Yang. Regularing generative adversarial networks under limited data. In CVPR, 2021.
[41] Wu, Ma, C. Deng, and Long. Pre-training contextualized world models with in-the-wild videos for reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[42] Z. Wu, N. Dvorikk, K. Greff, Kipf, and A. Garg. Slotformer: Unsupervised visual dynamics simulation with object-centric models. International Conference Learning Representations, 2022. doi: 10488550k2Xi.2210.0566..
[43] Z. Wu, J. Hu, W. Lu, I. Gilitschenski, and A. Garg. Slotdiffusion: Object-centric generative modeling with diffusion models. Advances in Neural Information Processing Systems, 36: 50932-50958, 2023.
[44] L. Yingzhen and S. Mandt. Disentangled sequential autoencoder. In International Conference on Machine Learning, pages 5670-5679, PMLR, 2018.
[45] L. Zhao, Kong, R. Walters, and L. L. Wong. Toward compositional generalization in object- oriented world modeling. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 26841-26866 PMLR, 17-23 Jul 2022. URL http::tts///// ptttps://prrceeddings mlr pross/v162/zhao222 html
[46] S. Zhou, Y. Du, J. Chen, Y. Li, D. Yeung, and C. Gan. Robodreamer: Learning compositional world models for robot imagination. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https //openreview. net (ttp://foorummiidddadnnmmaaaoe
13

---

Table 4: The Architecture and Hyperparameters of Encoder Learning.
Name
Value
bigfish 31
caveflyer 63
coinrun 31
dodgeball
jumper 31
ninja 47
starpilot 47
slots slot dimension
31
2.56
slot attention iterations
image size patch size optimizer learning rate data size
[224, 224, 31
16
Adam
4e-4
1M
epoch batch size
15
64
Cosmos model
Cosmos-0.1 --Tooeeeeieiie-1166666666
Table 5: The Architecture and Hyperparameters of Dynamics Learning.
Name
Value
dynamic feature dimension
256
static feature dimension
256
self-attention # layers
self attention model size
512
sell-attention # heads
SSM # layers
2
SSM model size
512
SSM duaae
64
SSM doom
optimizer
Adam
learning rate
le-4
batch size
32
A
Method Details
A.1
Encoder
The architecture and hyperparameter used during encoder training shown in Table. 4
A.2
Dynamics
During dynamic feature training, the slot reconstruction loss and the disentanglement loss may have conflicting gradient and hinder model learning. To mitigate this issue, we leverage gradient modification 26 to enhance the balance between two objectives.
The architecture and hyperparameter used during encoder training are shown in Table 5
B Experiment Details
B.1
Environment Details
Among the 16 Procgen environments, we selected environments based on their relevance to object- centric learning and visual complexity. Specifically, we looked for environments with moving objects, dynamic layouts, minimal sensitivity color. Based on these criteria, we excluded maze-like like environments such Maze, Heist, Chaser, and Miner, as well as Plunder, where the ship's 's color plays critical role. Among the remaining games, we randomly selected while ensuring diversity across game types, given resource constraints.
14

---

Table 6: slot-object binding aceuracy, measured by FR-ARI (1).
Environments
Oracle
Dyn-O (ours)
SOLV
bigfish
0.96
0.80
0.54
coinrun
0.33
0.27
0.10
dogehall
0.79
0.48
0.17
starpilot
0.86
0.47
0.49
average
0.74
0.51
0.33
(a) Oracle
(b) Dyn-O (ours)
SOLV
(d) Oracle
(e) Dyn-O (ours)
(I) SOLV
gg) Oracle
(h) Dyn-O (ours)
(i) SOLV
(i) Oracle
(k) Dyn-O (ours)
(I) SOLV
Figure 7: Qualitative evaluation of the object-centric representation learning in bigfish, coinrun,
dodgeball, and starpilot.
B.2 Evaluating Object-Centric Representation
The object-centric representation evaluation for 4 procgen environments are shown in Table. and Fig. By leveraging segmentation masks only during training, Dyn-O significantly outperforms SOLV in all environments, in terms of slot-object binding accuracy.
15

---

Table Rollout accuracy for each Procgen environment at 20-th timestamp, measured as mean and standard error.
Environment
Metric
DreamerV3
Dreamerwaver
Dyn-O w/o
Dyn-O (ours)
LPIPS (4)
0.39 0.01
0.52 0.01
0.36 0.01
0.25 0.01
FVD (U)
567.20 + 11.18
831.17 = 15.55
248.94 + 16.32
126.88 + 6.42
bigfish
SSIM (T)
0.73 + 0.01
0.68 + 0.00
0.68 + 0.01
0.76 = 0.01
PSNR ())
19.34 + 0.14
19.09 + 0.12
20.16 + 0.27
20.28 0.30
LPIPS (J)
0.53 + 0.01
0.59 + 0.01
0.61 = 0.01
FVD (J)
1104.50 18.87
966.07 28.71
877.74 20.99
caveflyer
SSIM (P)
0.44 0.01
0.26 0.01
0.26 0.01
PSNR (
11.99 + 0.12
10.93 0.19
10.82 = 0.13
LPIPS (L)
0.34 0.01
0.36 + 0.01
0.28 0.01
FVD (
530.31 + 10.51
583.11 + 16.07
266.13 + 6.16
coinrun
SSIM (1)
0.60 + 0.01
0.47 + 0.01
0.62 0.01
PSNR (T)
14.22 + 0.25
12.61 0.14
13.56 = 0.19
LPIPS (U)
0.42 + 0.01
0.15 + 0.01
0.14 0.01
FVD (L)
903.51 + 20.44
481.55 + 20.83
372.50 15.07
dodgeball
SSIM ())
0.50 0.01
0.72 0.01
0.76 0.01
PSNR (
16.10 +
20.33 0.12
20.88 0.15
LPIPS (L)
0.48 0.01
0.49 0.01
0.45 0.01
FVD (L)
423.27 + 14.26
521.32 + 15.85
313.57 + 9.73
ninja
SSIM (T)
0.45 + 0.01
0.33 + 0.01
0.54 0.01
PSNR (T)
12.58 1 0.17
12.80 1 0.10
11.95 = 0.12
LPIPS (L)
0.37 + 0.01
0.50 + 0.01
0.50 + 0.01
0.22 - 0.01
FVD (L)
626.47 + 14.95
735.24 = 21.48
429.36 + 15.46
211.27 12.70
starpilot
SSIM (7)
0.69 0.01
0.68 1 0.00
0.72 0.01
0.76 0.01
PSNR (1)
19.99
19.48 1 0.13
19.76 0.23
20.55 0.13
Table 8: Rollout accuracy for CLEVR and ALE environments at 20-th timestamp, measured as mean and standard error.
Environment
Metric
DreamerV3
Dyn-O (ours)
LPIPS (L)
0.34 1 0.00
0.31 +: 0.00
FVD (L)
1676.18 + 40.92
446.57 = 8.55
CLEVR
SSIM (f)
0.86 0.00
0.88 0.00
PSNR (t)
21.37 + 0.09
22.63 = 0.06
LPIPS (L)
0.12 0.00
0.09 0.01
FVD (L)
217.84 - 3.76
187.76 + 8.04
Atari Skiing
SSIM (t)
0.91 0.00
0.93 + 0.00
PSNR (1)
25.46 + 0.13
26.39 + 0.16
LPIPS (L)
0.04 + 0.00
0,02 + 0.00
FVD (L)
237.90 12.15
218.68 + 15.68
Atari Boxing
SSIM
0.93 0.00
0.95 0.00
PSNR (1)
30.00 0.14
29.46 - 0.09
B.3 Evaluating World Model Accuracy
The world model accuracy for other procgen environments are shown Table. 7 and Fig. Fig. 10 In most environments, Dyn-O significantly outperforms baselines in term of prediction accuracy.
Meanwhile, we further compare Dyn-O against DreamerV3 on the CLEVR dataset 24 and two
ALE environments 4 (Sking and Boxing), and the results are shown in Table. g
16

---

Dyn-O
DreamerV3
Dyn-O w/o OC
F10
EIIS
t=20
Figure 8: 20-step rollouts in dodgeball. 1st row: ground-truth, 2nd row: Dyn-O (ours), row: DreamerV3, and 4th row: Dyn-O w/o OC. Dyn-O significantly outperforms dreamer, with sharp player shape and accurate predictions of threw balls.
0000000
t-0
Dyn-O
DreamerV3
Dyn-O w/o OC
t
E-1
t=15
t=20
Figure 20-step rollouts in jumper. 1st row: ground-truth 2nd row: Dyn-O (ours), 3rd row: DreamerV3, and 4th row: Dyn-O w/o OC. Dyn-O significantly outperforms dreamer, with sharp wall and player trail until 10th timestamp.
B.4 Evaluating Static-Dynamie Disentanglement
The probing accuracy for other procgen environments are shown in Table. 9 12 The same privilege information may belong to different properties in different environments, which label out in the tables. In some environments, the same privilege information can be static properties of some objects and can dynamic for other objects. In such case, mark such privilege information "mixed".
17

---

0RD
0000
00DO0
Dyn-O
00000
1110
DreamerV3
DON.
0D0R
ODDOR
20000
Dyn-O w/o OC
11111
11
m1TT.
ShOUp
mET
E-1
t=20
Figure 10: 20-step rollouts in ninja. 1st row: ground-truth, 2nd row: Dyn-O (ours), 3rd row:
DreamerV3, and 4th row: Dyn-O w/o OC. Dyn-O significantly outperforms dreamer, with sharper wall shape at 15-th timestamp.
Table 9: Probing accuracy (f), in percentage (%), on bigfish privilege properties.
mean R values static
mean G values static
mean B values static
position dynamic
y position static
area static
slots
74.7 0.0
77.9 0.0
83.8 0.0
97.7 0.0
98.3 = 0.0
100,0
dynamic features
49.3
+ 3.0
48.4 2.4
47. 3.7
94.0 1.9
45.2 5.1
95.9 1.2
static features
65,8 + 4.2
67.6 = 4,8
77.5 1.2
29.3 0.4
88.8 0.7
100.0
random features
37.6 + 0.0
31.8 = 0.0
37.3 + 0.0
19.2 0.0
20.8 + 0.0
88.6 + 0.0
NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. [NA]
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Table 10: Probing accuracy (f), in percentage (%), on dodgeball privilege propertics.
mean R values static
mean G values static
mean B values static
position mixed
position mixed
static
slots
90.2 0.0
91.7 + 0.0
91.2 + 0.0
98.7
0.0
98.7 + 0.0
99.8 0.0
dynamic features
66.0 + 1.6
62.4
1.2
61.1
+ 1.8
55.4
57.7 +
95.3 1.7
static features
73.1 + 1.2
74.2
- 1.4
75.5
+ 1.7
74.9 = 2.4
70.4 + 2.9
99.3 + 0.0
random features
53.3 1 0.0
35.5 = 0.0
42.9 + 0.0
20.1 = 0.0
19.7 1 0.0
90.1 4: 0.0
18

---

Table 11: Probing accuracy (1), in percentage (%), ninja privilege properties.
mean R values static
mean G values static
mean B values static
x position dynamic
position dynamic
area
dynamic
slots
87.1 + 0.0
80.7 = 0.0
86.1 + 0.0
95.0 = 0.0
96.6 + 0.0
97.3 + 0.0
dynamic features
60.7 + 9.7
51.3 + 8.2
60.6 + 9.0
86.3 3.0
88.1 + 0.3
87.6 + 2.2
static features
77.6 + 2.9
62.2 0.4
79.6 + 0.4
35,9 =
37.8 1.1
82.2 1.3
random features
32.9 + 0.0
25.4 = 0.0
31.3 + 0.0
25.0 = 0.0
19.5 = 0.0
80.5 0.0
Table 12: Probing accuracy (1), in percentage (%), starpilot privilege properties.
mean R values static
mean G values static
mean B values static
x position dynamic
y position static
area static
slots
71.5 + 0.0
79.1 = 0.0
71.0 + 0.0
97.2 = 0.0
97.5 + 0.0
99.5 1 0.0
dynamic features
55,6
+ 7.2
66.5 : 6,4
55.3 6.2
94.8 0.9
77.0 14.2
97.9 1 1.7
static features
55.9
1.1
70.7 1.1
50.5 1 1.8
26.8
76.6 3.2
1
random features
35.9
- 0.0
50.6 = 0.0
36.3 - 0.0
18.3 = 0.0
22.7 = 0.0
92.0 - 0.0
Please provide short (1-2 sentence) justiffcattt right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", is perfectly acceptable to answer "[No] provvided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for dataset we used"). "[No] " or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification elaborate. All supporting evidence supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
appear either in the main paper the
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Our main claims can be supported by our experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long clear that these goals not attained by the paper.
19

---

2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss this in the appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification: Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
20

---

Justification:
We will provide code and instructions to make sure the results are reproducible.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will make the code available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: wwciidicggiieeeeeee
/
http://oooouummiiiiseeee ssionooiicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
21

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide these details in Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We provide this in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide this in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
22

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The authors have reviewed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We don't 't foresee such risks.
23

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credited existing assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?

---

Answer:
[NA]
The paper does not involve crowdsourcing nor research with human subjects Guidelines:
Justification:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non stanndaddrrd components.
Please refer to our LLM policy (https://neut) /cC//a/reeeeccc//222//////////// for what should or should not be described.
25


Reviews:
[{"review_id": "ijTn59e97T", "reviewer": "Reviewer_AdL1", "summary": "In this paper the authors present the Dyn-O model that learns world models by both learning object representations and a dynamics model to transition between those representations. Crucially, Dyn-O segments object representations into static properties that cannot change (e.g., the object sprite) and dynamics properties that can (e.g., position). Dyn-O is tested on seven different Procgen environments, outperforming DreamerV3 – an alternate world learning model – and on one environment disentangles the static and dynamic features appropriately", "questions": "My main question is how the findings differ and/or generalize across the different Procgen environments. For instance, Table 1 reports results aggregated for all environments, but it would be useful to assess how Dyn-O performs vs Dreamer or the non-object-centric version across all environments. Similarly, evaluating the object-centric representations is only done on BigFish and the static/dynamic disentanglements only on CoinRun – how do these findings generalize to the other environments?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "This work presents an interesting idea for world model, especially in segmenting out the static and dynamic features to better capture real-world object structure. While relatively simple environments, using seven Procgen tasks is good for capturing diverse dynamics. However, the results report data that is either only aggregated across environments, or listed for a single environment at a time, making it difficult to assess how general the results are across different types of environments and dynamics", "was_revised": true, "final_justification": "In light of the author's responses to me and other reviewers that provided more results, I am raising my review to a 4. I still have some concerns about the limitations of the problem set tested (e.g., some of the results in the responses are on subsets of the games -- possibly due to the limited response period) but the extensions here have started to address them", "rebuttal": "Thank you for your valuable comments and feedback\\! We're glad to hear that you found our idea interesting. Below, we address your questions.\n\n> `Q1:` My main question is how the findings differ and/or generalize across the different Procgen environments. For instance, Table 1 reports results aggregated for all environments, but it would be useful to assess how Dyn-O performs vs Dreamer or the non-object-centric version across all environments. Similarly, evaluating the object-centric representations is only done on BigFish and the static/dynamic disentanglements only on CoinRun – how do these findings generalize to the other environments?\n\n`R1:` Due to space limit, the results for each Procgen environment are deferred to the **supplementary materials**. Specifically,\n  * **Dynamics rollout accuracy**: As shown in **Table 6** (Page 15), Dyn-O outperforms both Dreamer and the non-object-centric baseline in most environments. Qualitative results are provided in Fig. 8 \\- 10\\.  \n  * **Object-centric representation quality**: As shown in **Table 5** (Page 14), Dyn-O outperforms SOLV in 3 out of 4 environments. Corresponding qualitative results can be found in Fig. 7\\.  \n  * **Static/dynamic disentanglement**: As shown in **Tables 7 \\- 10**, Dyn-O successfully disentangles static and dynamic features. In particular, for static privileged properties, static features achieve significantly higher probing accuracy than dynamic features, and vice versa.\n\nWe will add clarification in the main text to help reduce potential confusion."}, {"review_id": "xKLSWtolFI", "reviewer": "Reviewer_EH1k", "summary": "The paper introduces a new method for learning world model with object-centric representations. Given an input video, the method first detect objects in the video, and then extracts latent feature representations are each object. These latent representations are used in a state-space model to predict their next representations given an action. The method further decomposes the object latent representations into a static part and a dynamic part. Experiments are conducted on the Procgen dataset to verify the world model.", "questions": "Please see weaknesses above regarding questions.", "limitations": "No limitations discussed in the paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Positives:\n\n-\tThe paper has introduced several novel ideas in my opinion such as using the state-space model for prediction, and decomposition of a static and a dynamic components of the object slot representations.\n\n-\tThe paper also carefully designs what loss functions to use for learning the world model.\n\n-\tThe experimental results show improvement compared to DreamerV3.\n\nNegatives:\n\n-\tThe experimental evaluation of the method seems not very thorough. For example, the comparison is only against DreamerV3. There is no other world model evaluated.\n\n-\tThe experimental environments are all about games. More sophisticated setups can be considered such as using real-world videos.\n\n-\tSome ablation studies can be added. For example, how important is using the state-space model? What if some other models are used?", "was_revised": true, "final_justification": "The authors provided more experimental comparison and ablation studies in their rebuttal.", "rebuttal": "We sincerely appreciate your insightful comments. We're glad to hear that you find our ideas novel and recognize the careful design of the loss functions. Below, we address the questions you raised.\n\n> `W1`: The experimental evaluation of the method seems not very thorough. For example, the comparison is only against DreamerV3. There is no other world model evaluated\n\n`R1`: Thank you for the suggestion\\! During the rebuttal period, we further compared our Dyn-O with **Dreamweaver** \\[1\\], a recent object-centric world model. The experiments are currently in progress, and we will update the results as soon as they are available.\n\n\\[1\\] Baek, Junyeob, et al. \"Dreamweaver: Learning Compositional World Models from Pixels.\" arXiv preprint arXiv:2501.14174 (2025).\n\n> `W2`: The experimental environments are all about games. More sophisticated setups can be considered such as using real-world videos.\n\n`R2`: We agree that evaluating on real-world videos would further demonstrate the applicability of our object-centric world model and represents an exciting direction for future work. One avenue we plan to explore is evaluating Dyn-O on **robotic datasets** such as Bridge-v2 \\[2\\] and RT-1 \\[3\\] to assess its performance in more realistic settings. However, due to time and compute constraints, we were unable to include these experiments during the rebuttal period. We intend to pursue them in future extensions of this work.\n\n\\[2\\] Walke, Homer Rich, et al. \"Bridgedata v2: A dataset for robot learning at scale.\" Conference on Robot Learning. PMLR, 2023.  \n\\[3\\] Brohan, Anthony, et al. \"Rt-1: Robotics transformer for real-world control at scale.\" arXiv preprint arXiv:2212.06817 (2022).\n\n> `W3`: Some ablation studies can be added. For example, how important is using the state-space model? What if some other models are used?\n\n`R3`: Thank you for the suggestion\\! We conducted an ablation study on the backbone model for Dyn-O in the **Bigfish** environment, replacing the **state-space model** (SSM) with a **transformer**. As shown below, the SSM significantly outperforms the transformer.\n\n**Table 1**: Rollout accuracy at 20-th timestep in **Bigfish**, measured as mean and standard error\n\n| Method | LPIPS (↓) | FVD (↓) | SSIM (↑) | PSNR (↑) |\n| :---- | :---: | :---: | :---: | :---: |\n| Dyn-O SSM | **0.25 ± 0.01** | **126.88 ± 6.42** | **0.76 ± 0.01** | **20.28 ± 0.30** |\n| Dyn-O Transformer | 0.39 ± 0.00 |  235.09 ± 3.23 | 0.73 ± 0.00 | 17.78 ± 0.08 |"}, {"review_id": "zP40PcKS2z", "reviewer": "Reviewer_eNWE", "summary": "This paper addresses the challenge of learning a world model that decomposes observations into static parts and dynamic objects. The proposed method proceeds as follows: First, it encodes images using the COSMOS tokenizer [A], obtaining a fixed set of K latents representing individual objects via slot attention [B]. These latents are then used to generate reconstructed images. The overall network is trained to minimize reconstruction error, a standard approach in object-centric representation learning [B; C]. The method further improves upon this framework by leveraging a pre-trained SAM2 model [Ravi et al., 2024], utilizing its segmentation masks as attention masks for slot attention to ensure accurate assignment of individual objects to their corresponding slots.\n\nWith this learned object-centric representation, the method trains a world model to predict, given a set of slots and actions, the next-frame slots, a reward, and the probability of episode termination.\n\nThe method is evaluated on the Procgen dataset [Cobbe et al., 2020], which comprises procedurally generated 2D video game environments.\n\n[A] Agarwal, Niket, et al. \"Cosmos world foundation model platform for physical ai.\" arXiv preprint arXiv:2501.03575 (2025).\n\n[B] Locatello, F. et al. “Object-Centric Learning with Slot Attention.” Advances in Neural Information Processing Systems (NeurIPS), 2020.\n\n[C] Engelcke, Martin, et al. \"Genesis: Generative scene inference and sampling with object-centric latent representations.\" arXiv preprint arXiv:1907.13052 (2019).", "questions": "* The first sentence in the abstract is: \"World models aim to capture the dynamics of the environment\". Is this a fair statement? Intuitively, the first thing world model must learn is the static aspect of the world. Not to say learning dynamics isn't important -- it absolutely is -- but this emphasis doesn't sound right to me.\n\n* Abstract states: \"the dynamics are highly centered on interactions among objects within the environment\". Could the rebuttal clarify this? I am confused by this statement.", "limitations": "The paper does not discuss limitations.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n* I like the paper’s premise that decomposing the static world and individual objects (i.e., learning a disentangled representation) facilitates effective world model learning.\n* The methodology’s representation is clearly presented and straightforward to follow.\n* The approach’s use of an object-centric latent space for world modeling is innovative and compelling, offering a novel perspective on structured representation learning.\n\n### Weaknesses\n\n* ***Experimental Validation***:\nThe paper states: “However, the development of object-centric world models has largely been explored in environments with limited visual complexity (such as basic geometries). It remains underexplored whether such models can be effective in more challenging settings.” I fully agree with this assessment; however, the evaluation on the Procgen dataset [A], which comprises procedurally generated 2D video game environments, mirrors the limited complexity of prior datasets [D]. Therefore, this paper's experimental validation does not address whether the proposed methodology is effective in more challenging, real-world environments. More critically, the method is tested solely on Procgen, without comparing its performance against baselines on datasets commonly used in prior work [B, C]. This omission prevents firm conclusions about whether the proposed methodology advances the field relative to existing approaches. Additionally, the world model’s experimental evaluation is limited beyond the object-centric representation. Could the authors clarify why broader datasets, such as the Arcade Learning Environment (ALE) [E], were not included?\n\n\n* **Unclear Relation to Prior Art**:\nThe paper presents two core contributions: object-centric representations and the world model. However, it lacks clarity on how these components compare, conceptually and empirically, to prior work. While the paper references DreamerV3 as a baseline, it reports results only on Procgen, whereas DreamerV3 demonstrates performance across multiple datasets, Procgen, ALE [E] and several others with greater diversity than 2D games. This limited comparison undermines the paper’s claim of advancing the state of the art.\n\nOverall, this work shows promise, but it requires significantly broader experimental validation to warrant presentation at a venue like NeurIPS.\n\n\nReferences:\n\n[A] Cobbe, K. et al. “Leveraging Procedural Generation to Benchmark Reinforcement Learning.” International Conference on Machine Learning (ICML), 2020.\n\n[B] Greff, K. et al. “Multi-Object Representation Learning with Iterative Variational Inference.” International Conference on Machine Learning (ICML), 2019.\n\n[C] Kabra, R. et al. “SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition.” Advances in Neural Information Processing Systems (NeurIPS), 2021.\n\n[D] Girdhar, R. and D. Ramanan. “CATER: A Diagnostic Dataset for Compositional Actions and Temporal Reasoning.” arXiv:1910.04744, 2019.\n\n[E] Bellemare, M. G. et al. “The Arcade Learning Environment: An Evaluation Platform for General Agents.” Journal of Artificial Intelligence Research, 2013.", "was_revised": true, "final_justification": "After the rebuttal, I no longer have doubts regarding the scope of experimental validation, and I believe that the paper passes the bar.", "rebuttal": "Thank you for your valuable and insightful feedback. We're glad to hear that you found our methodology and representation clearly presented and easy to follow. We're especially encouraged that you consider our use of an object-centric latent space for world modeling to be innovative and a meaningful contribution to structured representation learning. Below, we address your comments and questions.\n\n> `W1`: Experimental Validation: The paper states: “However, the development of object-centric world models has largely been explored in environments with limited visual complexity (such as basic geometries). It remains underexplored whether such models can be effective in more challenging settings.” I fully agree with this assessment; however, the evaluation on the Procgen dataset \\[A\\], which comprises procedurally generated 2D video game environments, mirrors the limited complexity of prior datasets \\[D\\]. Therefore, this paper's experimental validation does not address whether the proposed methodology is effective in more challenging, real-world environments.\n\n`R1`: We agree with the reviewer that real-world videos are the best benchmark for evaluating object-centric world models. However, we would like to clarify that, while Procgen consists of 2D video game environments, it offers significantly greater visual and dynamic complexity compared to prior benchmarks such as **CATER** \\[D\\] and **CLEVR** \\[2\\]. Specifically, Procgen features:\n\n* **diverse shapes and textures** \\- while most objects in prior benchmarks use a single color and basic shape, as shown in Fig 7 and 9, Procgen objects have more complex shapes (like spaceships, trails) and color combinations (like explosion).  \n* **cluttered scenarios** \\- in prior benchmarks, most scenarios have \\<10 objects, while in Procgen, environments like Starpilot can have \\~50 objects.  \n* **complex dynamics** \\- in prior benchmarks, most movements are constrained to linear motion with constant velocity. In contrast, Procgen features complex motions (like jump) and mechanisms (like starship explosion).\n\nTherefore, evaluation on Procgen meaningfully advances the study of object-centric world models toward more challenging settings, and exploring real-world videos remains an exciting avenue for future work.\n\n\\[1\\] Girdhar, R. and D. Ramanan. “CATER: A Diagnostic Dataset for Compositional Actions and Temporal Reasoning.” arXiv:1910.04744, 2019\\.  \n\\[2\\] Johnson, Justin, et al. \"Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2017\\.\n\n> `W2`: More critically, the method is tested solely on Procgen, without comparing its performance against baselines on datasets commonly used in prior work \\[B, C\\]. This omission prevents firm conclusions about whether the proposed methodology advances the field relative to existing approaches. Additionally, the world model’s experimental evaluation is limited beyond the object-centric representation. Could the authors clarify why broader datasets, such as the Arcade Learning Environment (ALE) \\[E\\], were not included?\n\n`R2`: Thank you for the helpful feedback\\! During the rebuttal period, we conducted additional experiments by running our Dyn-O model on the **CLEVR** dataset. Regarding ALE, we believe Procgen serves as an enhanced benchmark, offering comparable or even greater complexity, with the added advantage of being designed with **generalization** in mind. As a result, models trained on Procgen are **less prone to overfitting**. That said, we were able to run experiments on two ALE games, **Boxing** and **Skiing**, during the rebuttal period. Detailed results for both the CLEVR and ALE experiments are provided in our response to the next comment.\n\n> `W3`: Unclear Relation to Prior Art: The paper presents two core contributions: object-centric representations and the world model. However, it lacks clarity on how these components compare, conceptually and empirically, to prior work. While the paper references DreamerV3 as a baseline, it reports results only on Procgen, whereas DreamerV3 demonstrates performance across multiple datasets, Procgen, ALE \\[E\\] and several others with greater diversity than 2D games. This limited comparison undermines the paper’s claim of advancing the state of the art.\n\n`R3`: Thank you for the suggestion\\! Following your recommendation, we conducted experiments comparing Dyn-O and DreamerV3 on **CLEVR** and two **ALE** games. As the results below show, Dyn-O outperforms DreamerV3 in ALE, highlighting the advantages of object-centric world models. The experiments on CLEVR are currently in progress, and we will update the results as soon as they are available.\n\n**Table 1**: Rollout accuracy at 20-th timestamp in **CLEVR**, measured as mean and standard error\n\n| Method | LPIPS (↓) | FVD (↓) | SSIM (↑) | PSNR (↑) |\n| :---- | :---: | :---: | :---: | :---: |\n| Dyn-O | running |  |  |  |\n| DreamerV3 | running |  |  |  |\n\n**Table 2**: Rollout accuracy at 20-th timestamp in **Atari Skiing**, measured as mean and standard error\n\n| Method | LPIPS (↓) | FVD (↓) | SSIM (↑) | PSNR (↑) |\n| :---- | :---: | :---: | :---: | :---: |\n| Dyn-O | **0.09 ± 0.01** | **187.76 ± 8.04** | **0.93 ± 0.00** | **26.39 ± 0.16** |\n| DreamerV3 | 0.12 ± 0.00 | 217.84 ± 3.76 | 0.91 ± 0.00 |  25.46 ± 0.13 |\n\n**Table 3**: Rollout accuracy at 20-th timestamp in **Atari Boxing**, measured as mean and standard error\n\n| Method | LPIPS (↓) | FVD (↓) | SSIM (↑) | PSNR (↑) |\n| :---- | :---: | :---: | :---: | :---: |\n| Dyn-O | **0.02 ± 0.00** | **218.68 ± 15.68** | **0.95 ± 0.00** | 29.46 ± 0.09 |\n| DreamerV3 | 0.04 ± 0.00 | 237.90 ± 12.15 | 0.93 ± 0.00 | **30.00 ± 0.14** |\n\n> `Q1`: The first sentence in the abstract is: \"World models aim to capture the dynamics of the environment\". Is this a fair statement? Intuitively, the first thing world model must learn is the static aspect of the world. Not to say learning dynamics isn't important \\-- it absolutely is \\-- but this emphasis doesn't sound right to me.\n\n`R4`: We agree that a world model can capture both the dynamic aspects (transition dynamics) and the static aspects (representation) of an environment. We wrote this statement because most prior work on world models has focused on the dynamic aspect, while the representation side, such as learning object segmentations and image understanding, has been more extensively explored in the broader deep learning and computer vision communities. We are happy to revise this statement based on your suggestions.\n\n> `Q2`: Abstract states: \"the dynamics are highly centered on interactions among objects within the environment\". Could the rebuttal clarify this? I am confused by this statement.\n\n`R5`: Thank you for your feedback\\! By this statement, we mean that most changes in the environment (its dynamics aspect), are either caused by objects or affect them. For example, in the Procgen Coinrun game, changes in the game screen are primarily driven by the agent’s movement or interactions with other objects, such as collecting coins, getting hurt by enemies, or being blocked by chests. We are happy to revise the statement according to your suggestions."}, {"review_id": "utye1DvVb0", "reviewer": "Reviewer_caxu", "summary": "This paper proposes an object-centric world model for better capturing the dynamics of the environment. The world model is also designed to be able to decouple object features into dynamics-agnostic and dynamics-aware components. The authors conduct experiments on the challenging Procgen benchmark and demonstrate their method's superior prediction accuracy compared to DreamerV3.", "questions": "1.  The paper would be significantly more convincing if it included experiments evaluating the agent’s performance, either in a model-based or model-free setting. Currently, the focus on reconstruction accuracy leaves open whether the learned representations actually improve downstream policy learning. I would be willing to accept if this point is covered.\n\n2. The paper evaluates the method on 7 Procgen environments but does not justify how these were chosen from the full benchmark suite. A brief explanation (e.g., diversity of difficulty, visual complexity, or relevance to object-centric learning) would clarify the experimental design.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1.  The paper incorporates pretrained computer vision models like SAM2 into world model learning. This is a promising direction, as leveraging large-scale pretrained models for representation learning in reinforcement learning remains underexplored. This work narrows the gap in this area.\n\n2. The paper is well-structured, with clear illustrations and plots that effectively communicate the proposed method. The visualisations of reconstructed observations are helpful in understanding the model’s behaviour. \n\n**Weakness:**\n\n1. The primary goal of world models is to improve agent performance, yet the paper does not evaluate whether the learned representations actually benefit policy learning. While the method demonstrates improved reconstruction accuracy, this does not necessarily translate to better RL performance.\n    - Is the static-dynamics disentanglement truly beneficial for agents?\n    - If the agent can learn a good policy without such disentanglement, the additional complexity may be unjustified.\n    - Could the gains simply come from increased model capacity?\n\n    Without policy learning experiments, the practical utility of the proposed method remains unclear.\n\n\n\n2. The experiments rely on trajectories generated by a pretrained PPG agent, raising concerns about applicability in online settings:\n    - Does the method work in an online RL setting like DreamerV3? Online world models must learn from dynamically changing data distributions, and it is unclear whether the proposed approach would remain stable.\n    - Could training collapse when learning from scratch? Pretrained policies provide high-quality exploration data, but in online RL, poor initial exploration might lead to degenerate representations.\n\n3. The paper does not explicitly describe whether reconstruction metrics are computed on training or unseen test trajectories. Is the reported accuracy based on training rollouts? If so, the improvements could stem from overfitting rather than better generalisation.", "was_revised": true, "final_justification": "See the comment", "rebuttal": "Thank you for your thoughtful and constructive comments. We’re glad that you recognize the potential of leveraging large-scale pretrained computer vision models like SAM2 within world model learning, and appreciate our efforts to narrow the gap in this underexplored area. We also thank you for acknowledging our paper’s well-organized presentation as well as its effective illustrations and visualizations. Below, we address your comments and questions.\n\n> `W1`: The experiments rely on trajectories generated by a pretrained PPG agent, raising concerns about applicability in online settings:  \n> * Does the method work in an online RL setting like DreamerV3? Online world models must learn from dynamically changing data distributions, and it is unclear whether the proposed approach would remain stable.  \n> * Could training collapse when learning from scratch? Pretrained policies provide high-quality exploration data, but in online RL, poor initial exploration might lead to degenerate representations.\n\n`R1`:\n* We would like to clarify that our focus is not on online reinforcement learning, but rather on building world models based on object-centric representations. In earlier experiments, we found that training the entire pipeline online was prohibitively resource-intensive, so we decided not to pursue that direction. Notably, many existing works also focus on the **offline setting** \\[1-4\\], so we believe our approach still offers meaningful contributions to the field.  \n  We agree that investigating the stability of the proposed method under dynamically changing data distributions would be insightful. However, doing so would require substantial engineering efforts to significantly optimize our pipeline. We view this as a valuable direction for future work.  \n* Thank you for the interesting question\\! First, we would like to clarify that the training data we used were **not collected by an expert policy**, but rather by **different intermediate checkpoints** saved during RL training. As a result, the dataset contains a significant portion of suboptimal exploration data.  \n  To further investigate this, we conducted an ablation experiment in the Bigfish environment, using only data collected from a **random policy** to train the representation. This was done to assess whether representation collapse would occur. The results are summarized in the table below. We can see using random exploration data leads to **slightly worse**, but not significantly degraded, performance.\n\n\n**Table 1**: Rollout accuracy of **Dyn-O trained with different data** at 20-th timestep in **Procgen Bigfish**, measured as mean and standard error\n\n| Data Collected by | LPIPS (↓) | FVD (↓) | SSIM (↑) | PSNR (↑) |\n| :---- | :---: | :---: | :---: | :---: |\n| PPG Policy Mixture | 0.25 ± 0.01 | 126.88 ± 6.42 | **0.76 ± 0.01** | 20.28 ± 0.30 |\n| Random Policy | 0.26 ± 0.02 | 137.90 ± 12.15 | 0.70 ± 0.01 | 20.17 ± 0.07 |\n\nWe believe the extent to which representation quality is affected depends heavily on the properties of the environment. In many of the environments we use, most objects remain visible even under random policies, making representation collapse unlikely. However, in particularly challenging environments that require extensive exploration (e.g., Montezuma’s Revenge), such collapse may occur. That said, effective exploration in such environments is itself a broader and open research question, which we consider beyond the scope of our current work.\n\n\\[1\\] Jiang, Jindong, et al. \"Object-centric slot diffusion.\" arXiv preprint arXiv:2303.10834 (2023).  \n\\[2\\] Wu, Ziyi, et al. \"Slotdiffusion: Object-centric generative modeling with diffusion models.\" Advances in Neural Information Processing Systems 36 (2023): 50932-50958.  \n\\[3\\] Wu, Ziyi, et al. \"Slotformer: Unsupervised visual dynamics simulation with object-centric models.\" arXiv preprint arXiv:2210.05861 (2022).  \n\\[4\\] Singh, Gautam, Yi-Fu Wu, and Sungjin Ahn. \"Simple unsupervised object-centric learning for complex and naturalistic videos.\" Advances in Neural Information Processing Systems 35 (2022): 18181-18196.\n\n> `W2`: The paper does not explicitly describe whether reconstruction metrics are computed on training or unseen test trajectories. Is the reported accuracy based on training rollouts? If so, the improvements could stem from overfitting rather than better generalisation.\n\n`R2`: We would like to point out that, **Line 236** of our paper describes that the evaluation is performed on unseen test levels. Therefore, the observed improvements are unlikely to result from overfitting.\n\n> `Q1`: The paper would be significantly more convincing if it included experiments evaluating the agent’s performance, either in a model-based or model-free setting. Currently, the focus on reconstruction accuracy leaves open whether the learned representations actually improve downstream policy learning. I would be willing to accept if this point is covered.\n\n`R3`: Thank you for the great suggestion and for your willingness to accept our paper if this point is addressed\\! This is indeed an important aspect that could further strengthen our paper. During the rebuttal period, we conducted additional policy learning experiments as recommended. Specifically, to evaluate whether the learned representations facilitate policy learning, we **fixed the representation backbone and trained only the policy head using the PPG algorithm** on three Procgen games: **Bigfish**, **Starpilot**, and **Coinrun**. We used the “easy” difficulty setting with an unlimited number of levels.\n\nWe compared the performance of **object-centric representations** against the representations directly output by the **Cosmos encoder**. Due to NeurIPS policy, we are unable to include figures at this stage, so we provide a table below summarizing the results at different training steps. Each game is trained with three seeds, and some runs are still in progress. We will update the results during the discussion phase once training is complete, and include the full figures in the final version. Based on the results so far, the learned representations lead to clearly improved policy learning performance.\n\n**Table 1**: Episodic returns in **Starpilot**\n\n| Steps | 2M | 4M | 6M | 10M | 25M |\n| ----- | :---- | :---- | :---- | :---- | :---- |\n| Object-centric | **7.81** | **15.06** | **17.13** | running | running |\n| Cosmos | 3.34 | 4.48 | 5.33 | 7.54 | running |\n\n**Table 1**: Episodic returns in **Bigfish**\n\n| Steps | 5M | 8M | 10M | 15M | 25M |\n| ----- | :---- | :---- | :---- | :---- | :---- |\n| Object-centric | **3.77** | **4.94** | running | running | running |\n| Cosmos | 0.95 | 0.84 | 0.88 | 0.91 | running |\n\n**Table 1**: Episodic returns in **Coinrun**\n\n| Steps | 5M | 8M | 10M | 15M | 25M |\n| ----- | :---- | :---- | :---- | :---- | :---- |\n| Object-centric | **8.74** | **8.44** | running | running | running |\n| Cosmos | 7.88 | 7.45 | 7.54 | 8.02 | running |\n\n> `Q2`: The paper evaluates the method on 7 Procgen environments but does not justify how these were chosen from the full benchmark suite. A brief explanation (e.g., diversity of difficulty, visual complexity, or relevance to object-centric learning) would clarify the experimental design.\n\n`R4`: Thank you for pointing this out. We selected 7 Procgen environments based on their **relevance to object-centric learning** and **visual complexity**. Specifically, we looked for environments with **moving objects**, **dynamic layouts**, and **minimal sensitivity to color**. Based on these criteria, we excluded maze-like environments such as Maze, Heist, Chaser, and Miner, as well as Plunder, where the ship's color plays a critical role. Among the remaining games, we randomly selected 7 while ensuring diversity across game types, given resource constraints. We would be happy to evaluate additional games if the reviewer deems it necessary, and we will include the above rationale in the next version."}]

Author rebuttal:

Meta-review: The paper introduces Dyn-O, a structured world model built on object-centric representations that disentangle static and dynamic features. Experimental results indicate that the approach improves rollout prediction and compositional generalization in visually complex environments.

Reviewers found the core idea innovative and the paper well-written, but significant concerns were raised due to limited experimental validation—confined mostly to the Procgen benchmark and a lack of evidence that the model's improved prediction accuracy translated to better performance on any downstream reinforcement learning task. The rebuttal was critical to alleviate these major concerns, and all reviewers were in agreement towards acceptance. The AC agrees with the consensus.

The authors are highly encouraged to incorporate the expanded evaluation (comparisons to new baselines and on different datasets, demonstration of improved sample efficiency, etc) in the camera ready version of the paper.
Decision: Accept (poster)