Dreamweaver: Learning Compositional World Models from Pixels
Authors: Junyeob Baek
Yi-Fu Wu
Gautam Singh
Sungjin Ahn
Venue: ICLR 2025 Poster (2025)
Keywords: compositional world models
unsupervised object-centric learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: we propose Dreamweaver, a neural architecture designed to discover hierarchical and compositional representations from raw videos and generate compositional future simulations.

Abstract: Humans have an innate ability to decompose their perceptions of the world into objects and their attributes, such as colors, shapes, and movement patterns. This cognitive process enables us to imagine novel futures by recombining familiar concepts. However, replicating this ability in artificial intelligence systems has proven challenging, particularly when it comes to modeling videos into compositional concepts and generating unseen, recomposed futures without relying on auxiliary data, such as text, masks, or bounding boxes. In this paper, we propose __Dreamweaver__, a neural architecture designed to discover hierarchical and compositional representations from raw videos and generate compositional future simulations. Our approach leverages a novel Recurrent Block-Slot Unit (RBSU) to decompose videos into their constituent objects and attributes. In addition, Dreamweaver uses a multi-future-frame prediction objective to capture disentangled representations for dynamic concepts more effectively as well as static concepts. In experiments, we demonstrate our model outperforms current state-of-the-art baselines for world modeling when evaluated under the DCI framework across multiple datasets. Furthermore, we show how the modularized concept representations of our model enable compositional imagination, allowing the generation of novel videos by recombining attributes from previously seen objects. [cun-bjy.github.io/dreamweaver-website](https://cun-bjy.github.io/dreamweaver-website/)

Published as a conference paper at ICLR 2025
DREAMWEAVER:
LEARNING COMPOSITIONAL WORLD MODELS FROM PIXELS
Junyeob Back' KAIST
Yi-Fu Wu Rutgers University
Gautam Singh Rutgers University
Sungjin Ahn"
KAIST, New York University
2022 2025
ABSTRACT
Humans have an innate ability to decompose their perceptions of the world into objects and their attributes, such as colors, shapes, and movement patterns. This cognitive process enables us to imagine novel futures by recombining familiar con- cepts. However, replicating this ability in artificial intelligence systems has proven challenging, particularly when it comes to modeling videos into compositional con- cepts and generating unseen, recomposed futures without relying auxiliary data, such as text, masks, or bounding boxes. In this paper, we propose Dreamweaver, a neural architecture designed to discover hierarchical and compositional representa- tions from raw videos and generate compositional future simulations. Our approach leverages a novel Recurrent Block-Slot Unit (RBSU) to decompose videos into their constituent objects and attributes. In addition, Dreamweaver uses a multi-future- frame prediction objective to capture disentangled representations for dynamic concepts more effectively as well as static concepts. In experiments, we demon- strate our model outperforms current srate-of-the-us baselines for world modeling when evaluated under the DCI framework across multiple datasets. Furthermore, we show how the modularized concept representations of our model enable com- positional imagination, allowing the generation of novel videos by recombining attributes from previously seen objects. cun-bjy github.ioddremmwaavrrvww.o
2XX. [cs.CV] 10 10 Apr
INTRODUCTION
The primary function of the brain is believed to be the construction of an internal model of the world from sensory inputs like vision-a concept often referred to world models (Ha & Schmid- huber, 2018). This construction involves developing two fundamental processes: knowing and thinking (Summerfield, 2022; Lake al., 2017: Fodor al., 1975). The knowing function entails how to compositionally structure and encode knowledge from experiences through representation learning. The thinking function enables the utilization of this encoded knowledge for abilities such as reasoning, planning. imagining, and causal inference (Goyal & Bengio, 2022; Schölkopf et al., 2021a). Due its generative and inferential nature, the thinking function necessitates some form of generative learning (Parr & Friston, 2018; Kurth-Nelson et al., 2023; Schwartenbeck et al., 2023).
key aspect distinguishing the world models in humans and currently is compositionaalty (Smolen- sky et al., 2022; Schölkopf et al., 2021c; Lake et al., 2017; Goyal & Bengio, 2022; Greff et al., 2020; Behrens et al., 2018), the capability to understand or construct complex concepts as a composition of simpler concepts. Recent studies in neuroscience suggest that humans can understand and adapt to various novel situations because the brain supports compositional representation and genera- tion (Kurth-Nelson et al., 2023; Schwartenbeck et al., 2023; Behrens al., 2018; Bakermans et al., 2023). However, current world models in Al are limited in this ability for the following reasons.
Most of compositionality in AI is currently approached via language-conditionee image/video generation models such as DALL-E (Ramesh et al., 2021; 2022) and Sora (Cho et al., 2024; Millière). These models do offer degree of compositionality (e.g., generating an image of an avocado chair) through language as a medium, which is convenient for human interaction. However, achieving
"Correspondence wnsd ajtm@@aaakaaa st..cc. kr and sungji n. ahn@kai at.

---

Published as a conference paper at ICLR 2025
Menory of Reusable Modiler Corcepts comee
Sensoyy Japur Stream
Ansgined World
Percaived Concagts
Recomblied Concopts
Composable World Perceiver
Composable World Simulator
Figure I: Overview of the Dreamweaver Framework. Our aim is to take sequential unstructured sensory stream and bind the low-level information into abstract modular concepts to build memory of reusable concepts, called concept library all without text and in an unsupervised way. These concepts include both static factors such as color and shape as well as dynamic factors such as direction and speed of motion. Finally, we seek to recombine these concepts, e.g., in novel configuration, and imagine unseen world.
compositionality more broadly, without relying on language, remains challenge. This is crucial because there is vast amount of world knowledge that is difficult to accurately articulate through language. Another limitation of this language-dependent approach is that it addresses only half of the problem. This is because the most challenging part of the knowing problem. namely discovering a composable knowledge representation from unstructured raw sensory inputs, is sidestepped as the core structure provided through the conessoompoiii structure inherent in language, rather than being discovered. To the best of our knowledge, this crucial ability of extracting composable world knowledge from video without language and enabling compositional imagination, has yet to be realized in machine learning.
In this paper, we take the first step toward this challenge. Specificially, we focus on unsupervised learning of compositional world models from image sequences in way support both the com- positional knowing and thinking processes. We approach this challenge from the perspective of Object-Centrre Learning (OCL) (Greff et al., 2020). OCL aims to learn to develop compositional and modular representation from visual observations in an unsupervised way. However, existing OCL approaches lack some key properties required for truly compositional world model. For instance, most OCL methods, such as Slot Attention (SA) (Locatello et al., 2020) and its subsequent models (Singh et al., 2022a; Kipf et al., 2021; Singh al., 2022b; Seitzer et al., 2022) maintain monolithic, entangled representation for representing an object while lacking a compositional structure within an object. This issue has recently been addressed by SysBinder (Singh al., 2023; Wu et al., 2024) by introducing a block-slot representation that offers nested compositional structure, referred as blocks, within an object slot. These works (Singh et al., 2023; Wu et al., 2024) showed that this block-slot representation enables the generation of novel scene imaginations by recombining static blocks concepts in an out-of-distribution manner.
However, since these models are designed for static images, it is unclear whether they can be extended for dynamic sequence modeling and serve as a foundation for image-based compositional world models. A key uncertainty and challenge is whether dynamic concept blocks, such as direction (e.g.. 'to the right') ) and speed (c.g., 'fast' or 'slow'), can emerge solely from observational learning. Our key hypothesis is that object-centric representation learning alone is insufficient. Instead, both block-slot representation and a temporally predictive objective function are that were not utilized in previous works. If successful, this proof-of-concep! approach would represent a step toward the grand challenge of generating novel videos without relying on language, through the composition of both static and dynamic block-slot concepts, such as "flying green rabbit," as shown in Figl.
We achieve this goal by introducing neural architecture named Dreamweaver. Dreamweaver encodes a temporal context window of T past images using a novel Recurrent Block-Slov Slot Unit (RBSU). The RBSU represents its state as a set of slot states, each updated independently. This is then passed through block-slot bottleneck, mapping the potentially entangled monolithic slot state into composition of independent blocks. Since a block vector can only obtain its value through attention to the prototype concept library, this process can also be seen as implementing an attraction process at each step. At the end of the temporal encoding, Dreamweaver predicts the observation at a future time step from the latest block-slot representation. We found that this predictive reconstruction objective is indeed crucial in making the dynamic concept abstractions emerge. In experiments, Dreamweaver outperforms state-of-the-e the-arr object-centric methods under the DCI framework across several datasets.
2

---

Published as a conference paper at ICLR 2025
RBSU
Minimize Prediction
Poctctyp
Dreamweaver
Decoder
RBSU
RBSU
S
ST
Block Slots,
Appi
Orsdict
Block Slots
CNN
CNN
Prster
Steta
NT+LT-K
Figure 2: Model Architecture. Left: The Recurrent Block-Slot Unit (RBSU) is recurrett unit designed for processing sequences where each item is set of vectors. RBSU maintains and updates Block- Slots, which represent compositional and semantic concepts such shape, color, and motion direction. Right: The Dreamweaver model encodes video inputs into Block-Slot representations, which pass through series of RBSUs with recurrent structure. It then predicts future frames by decoding the extracted Block-Slots using a transformer decoder, training to minimize the predictive objective.
Moreover, demonstrate how RBSU's modularized concept representations enable compositional imagination, generating novel videos that combine attributes from previously seen objects.
The main contributions of the paper compositional world modeling from pixels. We introduce a novel recurrent module, named Recurrent Block-Slot Units. Our model is the first in object-centric learning that can learn both static and dynamic composable concepts in an unsupervised way. By contrast, previous works were modeling only static images and able learn only static concepts. We also found that using predictive imagination loss is crucial in achieving this, in addition to the architectural inductive bias of block- slot representation. We found that previous temporal object-centric models cannot develop abstraction of dynamic concepts due to their autoencoding objective. Finally, for the first time in this area, demonstrate the ability Dreamweaver to simulate a future from an out-of-distribution configuration of discovered concepts. While our model shares common limitation of current state-of-the-an object-centric learning models it is not yet applicable to highly complex scene images- we that the success of this proof-of-concep represents significant step toward the grand challenge of composing novel videos without relying on language.
follows: We introduce the Dreamweaver model for
we believe
RECURRENT BLOCK- BLOCK-SLOT UNITS
Recurrent Block-Slot Unit or RBSU a general-purpose recurrent unit for sequence modeling that propose in this work. Given input sequence Ej...., Er, where each item in the sequence E, E LR Dupe a collection of T. vectors, RBSU works by processing the sequence recurrently. starts with initial state representation Sg. and for each item E, in the input sequence, applies the RBSU to update the state from S to S,
So Initialize()
S, RBSUUE, S.-1)
Importantly, RBSU maintains a disentangled state representation S, called the block-slot representa- tion or simply block-slots. The block-slot representation is collection of N vectors called slots i.c., S. NDMD We denote each n-th slot S, as 8t,, E TR MD. Furthermore, each slot is internally disentangled and constructed by concatenating M vectors of size / called blocks. We denote the m-th block within a slot St, as St,n. E R_ For modeling multi-object video inputs, a slot St, can represent object while block sm within the slot can represent intra-object reusable concept e.g., color, shape, or direction of motion.
2.1 BOTTOM-UP ATTENTION
In the first step in RBSU, the N slots of the previous time-step S act queries and attend over the current input features E, to obtain bottom-up readour vectors U, R NxNxM XMD, Following Singh et al. (2023); Locatello et al. (2020), this performed via inverted attention and renormalization
3

---

Published as a conference paper at ICLR 2025
(Wu et al., 2023a) follows:
q -(E,))
Aind Eis Arn
A, sof
Annd
U.
- v(E)),
VMD
Next, in the readout U, we split each n-th row vector un E RMD into equal readout chunks, where each m-th chunk is denoted by u. The chunks shall be used to update their corresponding block in the block-slot state representation.
2.2 INDEPENDENT BLOCK ATTRACTOR DYNAMICS
The next step in RBSU is update each block in the current block-slot state representation via per-block independent recurrent modules. Importantly, we incorporate attractor dynamics within these recurrent modules facilitate the convergence of each block's state reusable representation. We perform the following per-block operations:
GRU and MLP. First, each previous block state S,-1,n.m is independently updated using its corre- sponding readout chunk u by applying a GRU to incorporate the bottom-up information. The resulting block then fed to MLP with residual connection as follows:
St,nm GRU pm (Sm-1, mm.mm
St,n.m St,n.m MLP om (L.(5t.,.)))
We maintain separate GRU and MLP weights for each m encourage modularity between blocks. Here, the st,,, and St.n, denote the intermediate block states after applying GRU and the residual ML.,, respectively.
Block-Slot Attractor Dynamies. Since GRU and MLP alone are found insufficient for making block's state reach a reusable attractor state (Singh et al., 2023), we explicitly incorporate a learned memory of prototypes and make each block perform dot-product attention over this learned memory to retrieve a state as follows:
St.mm C. )} Vd
m
softmax N Njewwype
Cm-
To maintain modularized representations from GRU and MLP, utilize separate library of prototypes C. Nowoees x for each m.. Notably, each library is initialized with Nprontypes learnable vectors, which start with random values and are subsequently learned through backpropagation to capture emergent semantic concepts.
2.3 BL.OCK INTERACTION
Finally, although we maintain independent information processing pathways, for improved flexibility, it is desirable to let the blocks in S, interact: S, BlockInteraction(SS) To implement this interac- tion step, we first flatten the slots into a collection of NM block vectors, feed them to a single-layer transformer, and then reshape the output back to the original shape S, NxM
3 DREAMWEAVER: COMPOSITIONAL WORLD MODEL VIA PREDICTIVE IMAGINATION
In this section. we propose and describe novel compositional world model called Dreamweaver using the proposed RBSU. Broadly, Dreamweaver takes video frames XI,..., input, constructs the world state in terms of composable tokens, and predicts future video frames ST+1,.... *T+K:
xr+1,..., * Dreamweaver( .x, XT),
where 7' is temporal context window size and the frames belong to RC Dreamweaver is implemented as an encoder decoder architecture:
Sr = fx ..)
*y = go (5 (Sr
where the encoder fo encodes the video Xi...., Xy into slots Sy leveraging RBSU. Given the slots Sy and step indicator k, where k is the relative temporal index of the target frame, the decoder go predicts the target frame & & via an autoregressive image transformer (Chen al., 2020; Singh al., 2022a;b; 2023).
4

---

Published as a conference paper at ICLR 2025
3.1 ENCODING VIA SPATIOTEMPORAL CONCEPT BINDING
In the encoder of Dreamweaver, each input frame X, is processed by CNN to output feature map. Next, add positional embeddings to this feature map, flatten it, and feed an MLP to obtain E, E IRLDD. This is similar to Locatello et al. (2020); Singh et al. (2022b; 2023). The sequence E Ei...., ET is then processed by the RBSU to produce slots ,
Es = (X,)
S. t RBSU (E,
Here, we obtain the initial block-slot state So by sampling it from a learned Gaussian distribution. That is, for all n ,,...,N, N, we sample So,n NN Tø), where n& T E RMD are learned parameters.
3.2 DECODING VIA AUTOREGRESSIVE IMAGE TRANSFORMER
The extracted slots Sr are decoded leveraging an autoregressive image transformer to predict the future frames XT+1.-.. - K- Similar to the powerful architectures used in (Singh et al., 2022a;b; 2023), our transformer decoder does not directly predict the target images in pixel space but instead predicts their discrete token representation. We employ Discrete VAE (dVAE) (Ramesh et al., 2021; Singh et al., 2022a) to transform an image into a sequence of integer tokens , 4T, E V and vere-versa, where V denotes vocabulary.
To train the transformer, we adopt the standard practice in language modeling (Vaswani et al., 2017; Singh et al., 2022a) and perform parallelized training of the autoregressive transformer via causal masking. That is, we first map each token 2T+ E V to an embedding eT+*, E TR. by retrieving the token's embedding from learned dictionary and adding positional encoding 1T+ = Dictionary o (2)+ + Pe,r. Next, we provide the embeddings eT+k,I -T to the transformer decoder that utilizes beginning-of-sequenne generate the next-token log probabilities for each input token:
follows:
(BOS) token and causal masking to
vy--T+ O Transformero (BOSo,k @T+kk1: "TT,,,:...,,T,,,,// cond Sr),
Here, the notation O Hunne RIV denotes the predicted log probabilies over V for the 1-th token of the image at time-step furge given the slots inferred at time-step fsource- Additionally, the BOS token serves dual purpose: provides the initial input to the transformer and acts indicator, representing the relative temporal index k. We maintain a dedicated BOS token for each k, denoted BOS The detailed method for the conditional prediction of the k-th frame is described in Appendix B.2.
step
3.3 TRAINING OBJECTIVE
The complete model is trained by minimizing cross-entropy loss averaged over all values of k 1,..., K, and all token indices 1,...,L'.
K
L'
L Dreamweaver (0,
Σ k=1
KL
In practice, we train the d'VEE jointly with the Dreamweaver by using a combined loss L(0, Dreamweeaee (0, Φ) + LavAn (0, Φ). For more details about the dVAF, see Appendix section B.
Need for a Predictive Objective. The purpose of introducing a predictive objective contrast to simple reconstruction objective commonly used in prior works (Kipf et al., 2021; Singh et al., 2022b; Elsayed et al., 2022) is two-fold: i) The predictive objective incentivizes RBSU to capture not just static factor primitives, such color or shape, but also dynamical primitives, such as direction or speed, since the latter is necessary when performing future prediction but much less important when the goal is to merely reconstruct the present frame. ii) The ability to generate future frames naturally provides a world model that can be utilized to roll out long future trajectories by autoregressively feeding each generated frame back into the model.
4 RELATED WORKS
Learning Object-centric Compositional Representations. Our work is influenced by recent research in unsupervised object-centric representation learning (Locatello et al., 2020; Engeleke et al.,
5

---

Published as a conference paper at ICLR 2025
2020; Burgess et al., 2019; Greff et al., 2017; 2019; 2020; Van Steenkiste et al., 2018; Zoran et al.,
2021; Veerapaneni et al., 2019: Ding et al., 2020; Seitzer et al., 2022; Lowe et al., 2022; Sajjadi et al.,
2022), where multi-object scenes are decomposed into entity- representations corresponding to the objects in the scene. In particular, our method is built upon the family of Slot Attention -aasdd models (Locatello al., 2020; Singh et al., 2022a; Wu et al., 2023b; Jiang al., 2023) and related to several Slot Attention for Video models (Kipf et al., 2021; Elsayed et al., 2022: Singh et al., 2022b; Bao al., 2023; Zadaianchuk al., 2023; Singh et al., 2024; Jiang et al., 2024). Unlike our method, these models are trained with reconstruction objective instead of a predictive objective and generally do not support imagination of future frames. Furthermore, these slot-based models only decompose the scene into object-level representations, whereas our model discovers concept-level representations in the form of block-slot representations. While previous work (Singh et al., 2023; Wu et al., 2024) also learned static concept-level representations from images, our method is the first to be applied to videos and additionally capture dynamic concepts. Another line of research learns bounding boxes for the objects in the scene, decomposing the objects into where and what representations (Eslami et al., 2016; Crawford & Pineau, 2019b; Kosiorek et al., 2018; Lin et al., 2020b; Crawford & Pineau, 2019a; Jiang et al., 2019; Lin et al., 2020a). However, these methods also do not further decompose into concept-level representations. Lastly, several works (Wu et al., 2021; 2022) support future frame prediction by training a transformer on set of pretrained object-representations (Lin et al., 2020b; Kipf et al., 2021; Singh et al., 2022b). Our method instead is trained end-to-end, which allows dynamic concepts to be captured in the representations and naturally reuses them predict future frames. Additional related works are detailed in Appendix A.
5 EXPERIMENTS
We evaluate the following questions: (1) How effectively can Dreamweaver infer modular con- cepts-both static and dynamic-from RGB videos without any supervision? (2) Can Dreamweaver generate new videos by composing novel configurations of the inferred concepts? (3) How well does Dreamweaver's representation support out-of-distribution (OOD) generalization on downstream reasoning tasks? Finally, we conduct an ablation study to evaluate various design choices.
Datasets. We experiment on five datasets spanning two axes of complexity: Dynamical Complexity and Visual Complexity. Along the axis of dynamical complexity, our datasets Moving-Sprites Moving- CLEVR, and DowinngCLEVRRIE exhibit low dynamical complexity i.e., objects move uniformly in a specific direction throughout the video (e.g.. up, down, left, right). On the other hand, our datasets Sancing-Sprice and Dancing-CLEVE CLEVR exhibit high dynamical complexity, featuring multi- step "dance" patterns (e.g., a clockwise square dance pattern: right - down -> left - up, as shown in Figure 14) multi-step color-change patterns (see Figure 15). Along the axis of visual complexity, we have levels: (i) The simplest are Moving-Sprites and Dancing-Sprites with 2D sprites black canvas. (ii) Moving-CLEVR CLEVR and Dancing- -LEVR significantly increase visual rubber objects, realistic lighting, and shading. (iii) In line with previous work (Singh et al., 2023; Wu et al., 2024), the highest level of visual complexity is in Moving-CLEVRRe where objects have complex textures on them.
complexity, with 3D scenes featuring solid-colored 3D
Baselines. We compare our model against three unsupervised representation learning baselines: RSSM (Hafner et al., 2018 STEVE (Singh et al., 2022b), and SysBinder (Singh et al., 2023). RSSM is widely used world modeling framework that represents video frame via single-vector representation. STEVE is a state-of-the-art method that represents each video frame as a set of per-object latent vectors (called slots). STEVE uses Slot Attention (Locatello et al., 2020) to represent scenes in a structured manner by spatially binding objects into slot representations. Like STEVE, SysBinder also provides a slot representation of each video frame; however, it further disentangles a slot as a concatenation of several blocks elocks-each block representing one factor e.g., color, shape, etc. Since SysBinder was originally designed for image data, it can only be applied independently per video frame and is not capable of utilizing the temporal context. Therefore, for a fair comparison with our model, we modify SysBinder for videos to obtain a stronger baseline as follows. We equip SysBinder with recurrent encoder (Kipf et al., 2021) to handle multiple frames. We refer to this modified model as SysBinder for simplicity. For all baselines, we use the same length of conditioning frames our model. The baselines are trained with reconstruction objective.

---

Published as a conference paper at ICLR 2025
(a) Moving Sprites
(b) Moving CLEVR
(c) Moving CLEVRTes
Fue Mode
INSTEVE
BydBinder
DDRSSM
0.4
0.2
012
6.2
(e) Dancing-Sprites
(f) Dancing-CLEVR
0.4
0.4
0.2
0.2
Figure 3: DCI Performance. We compare our model with the baselines in terms of Disentanglement (D), Completeness (C), Informativeness (I), and Informativeness Seaiiveeness-Dnmmic (I-D). I-D is the informativeness score for dynamic concepts only (e.g.. the direction motion or dance pattern, etc.) to evaluate how effectively the models capture dynamic concepts.
5.1 UNSUPERVISED MODULAR CONCEPT DISCOVERY FROM VIDEOS
Metries. For quantitative evaluation of learned representations, we use the DCI (Eastwood & Williams, 2018) framework evaluate Disentanglement (D), Completeness (C), and Informativeness (I). Note that the DCI is computed using ground truth object factors that not only include the static factors e.g., color or shape, but also the dynamical factors e.g., direction of motion, "dance" pattern, etc. Additionally, we also compute a metric called Informativeness- -Dyeamic (I-D), which is the informativeness score evaluated only on the dynamic factors.
DCT Performance. In Figure 3, we compare the DCI performance of our model with the baselines. We note that our model consistently surpasses the other baselines across all datasets, achieving the highest scores in Disentanglement (D), Completeness (C), and Informativeness (I).
Emergence of Dynamical Factor Representation. Importantly, we also note Figure that our model strongly surpasses all other baselines in terms of the I-D score which captures how informative our representation is about the dynamic factors. Our model achieves scores more than twice those of the other baselines. This strong performance points to the effectiveness of the predictive objective which incentivizes our model to capture dynamical information. As such, to the best our knowledge, ours is the first unsupervised representation learning model capable of representing both static and dynamic concepts while simultaneously providing disentanglement.
Visualizing the Feature Space of Blocks. We visualize the semanties of each block's 's feature space in Appendix D.I. We can see that our model allocates specific regions of the feature space of specific blocks capture specific factor values e.g., the star shape, the yellow color, etc. We also qualitatively that our model captures the dynamical factor values such as the up&down dance and the horizontal sliding movement.
5.2 COMPOSITIONAL IMAGINATION
In this section, we demonstrate how our block-slot representation can be recomposed into novel configurations and be used to generate compositionally novel videos.
Setup. We feed the initial frames of video (called the context frames) into our pre-trained RBSU encoder. We take the block-slot representation from the last context frame and manipulate it in one of the following two ways: (1) We can perform a factor swap by taking two slots corresponding to distinct objects, selecting the blocks that correspond to specific factor (e.g., color), and swapping them. (2) We can perform a factor change by taking the block-slot representation from a source video, selecting the block that captures certain factor e.g.. block that represents certain "dance" movement and using to replace a block in the block-slot representation of the video that we seek to manipulate. The manipulated block-slot representation fed pre-trained Dreamweaver decoder
7

---

Published as a conference paper at ICLR 2025
rollect
Inpux
Compositional Imagination
Compositional Imagination
color swap
e
shape swap
change dance
*
uplidow
change direction
change dance
*
3-bsp-CCM
CN
change speed
change dance
Moving-Sprites
Dancing-CLEVA
Figure Compositional Imagination. We show compositionally novel videos generated by Dreamweaver. this visualization, we (1) infer block-slot representation given an initial context video, (2) perform manipulations on the inferred block-slot representation, and (3) perform rollout starting from the manipulated block-slot representation. At the top, also visualize the rollout that would have occurred had no manipulation been done the representation. Left: For the Moving Sprites dataset, visualize manipulations such as swapping color and shape, changing of direction of motion specific object, and changing speed of movement of a specific object. Right: For the Dancing-CLEP CLEVR dataset, we visualize manipulations such as swapping the object shapes and changing the dance patterns.
to generate future video frames. We autoregressively feed the predicted frames back into the encoder as context frames to perform rollout. Details about how we ascertain the correspondence between ground truth factor and its representative block are provided in Appendix C.5.
Results. In Figure 4, we can see that our approach successfully generates novel videos through the compositional manipulation of our block-slot representations. For instance, in the Mlovng--Sprite video, we could create novel object appearances by swapping color or shape blocks (Examples and 2), and alter the objects' motion trajectories and velocities by modifying direction speed blocks (Examples and 4). Additionally, the more complex Dancing-CLEW demonstrate the ability to imagine different types of object dynamics. such as transitioning from color-changing dynamics to lifting up and down movement dynamics, highlighting the flexibility of method (Examples 2 and 4). For examples of compositional imagination, refer to Appendix block configurations in
experiments, we
D.3. Furthermore, our model effectively generalizes to compositional imagination tasks, preserving quality without degradation. See the detailed results in Appendix D.2.
5.3 COMPOSITIONAL SCENE PREDICTION AND REASONING
To evaluate the quality of the learned representations, we construct downstream task that requires predicting the future states of the objects in the scene and reasoning among the factors of the objects.
Task. The task is to take the latent representation of the last frame of a given context video and predict a target value corresponding to range of future frames at various offsets e.g., O,...,5. 5. The target value corresponding to each possible value of each ground truth factor following Wu et al. (2024). (2) We then take the maximum value of each ground truth factor across objects and sum these maximum values to obtain the target value. To solve this task well, the learner must, either explicitly or implicitly, predict the future factor values of each object and compare these values to determine the maximum value.
frame is defined follows: (1) We first assign an integer number
Probe. We train a downstream probe that takes as input the latent representation of the last context frame from each pre-traine pre-trained model and predicts the target value for range of frame offsets.
Setup. We evaluate the probing performance on the Sancing-Spritee and Dancing-CLe CLEVR datasets. For Dancing-Spriter we use the shape, color, and position of each object as the ground truth factors
8

---

Published as a conference paper at ICLR 2025
Dancing- Sprrttes
Dancing-Spriter (OOD)
Dancing CLEVR
100
Accuracy (in %)
80
50
40
Mudd STEVE
20
SysBinder -a RSSM
0
Frame Offset
Frame Offset
Frame Offset
Figure 5: Compositional Scene Prediction and Reasoning. We compare our model with baselines in terms of prediction accuracy for different frame offsets. A frame offset of zero corresponds to the last context frame and a frame offset of one corresponds to the first predicted frame after the context frames, and
that determine the target value. Since we provide 3 context frames for this dataset and each object moves in 4-frame dance sequence, the models must capture the dance patterns to be able to accurately infer the positions. To evaluate diitriiiiitiitttion generalization, we further create a Dancing-Sprite (OOD) task where the test set consists only of objects with factor combinations that are not seen during training. For the Dancing-CLEVe dataset, we use the dynamic object movement type as a ground truth factor in addition the shape, color, and position.
Probing Performance. Figure 5, we can see that Dreanweaver consistently outperforms the baselines all datasets, indicating the learned block-slot representations conducive to solving this relational reasoning task.
Importance of Predictive Objective. On Dancing-Sprites SysBinder can accurately predict the target value at frame offset 0 but the performance degrades for larger frame offsets. This shows the usefulness of Dreamweaver's predictive objective- while SysBinder also decomposes the scene to block-slot representation corresponding to the factors of the object, it is trained with a reconstruction objective instead of predictive objective and thus cannot accurately predict the target value for frames outside of the context frames. For Jancing-CLEVRE we see all models besides Dreamweaver fail, even at frame offset zero. This is because the dynamic object movement type part of the ground truth factors determining the target value, and as we saw from the DCI analysis, the bascline models do not adequately capture these dynamical factors well.
Generalization Ability. Dreamweaver and SysBinder generalize well to the Dancing Sarnng-Sriit-OOOO OOD dataset, while the performance of STEVE and RSSM decrease significantly, illustrating the out- of-distribution capability of the block-slot representations. Lastly, we note that the performance of Dreamweaver generally maintained even as the frame offset is increased beyond the training prediction length. For Mancing-CLEEV even though we only train the model to predict two frames, we see that the learned representations can be used to predict up to five frames, further indicating that the dynamics of the scene captured well by the model.
5.4 ABLATION STUDY
We conduct series of ablations on the different architectural components of Dreamweaver as well as several of the key hyperparameters.
Analysis of Architectural Components. Figure 6 (a) shows the results of ablating several architec- tural components on the Mancing-CLEVE dataset. We see that using the concept memory only on performs similarly to our model, which uses the concept
the last slot iteration memory after every slot iteration, although there slight drop in capturing dynamic concepts (I-D). Removing the concept memory completely (No-CM), however, results in a sharp drop in disentanglement, showing the importance of using a shared concept memory for our model. Lastly, removing the predictive objective and instead training reconstruction (N--rrddictiv) results in worse performance across all metrics. In particular, I-D drops significantly, showing the importance of the predictive objective for capturing dynamic concepts.
Effect of Number of Blocks and Concept Memory Size. Figures (b-c) show the results of different numbers of blocks and concept memory sizes on the CLEVR-Hap dataset. We the performance saturating once the number of blocks is large enough, indicating the robustness of this parameter as long as it is set large enough capture the complexity of the dataset. On the other hand, we observe

---

Published as a conference paper at ICLR 2025
(a) Architectural Components Analysis
(b) Effect of Number of Blocks
(c) Effect of Concept Memory Size
Our Model Irddttidctto
In
DEM
BEM 16
Ink 61
Bn 135
BMNe-CM
In 206
Ix 512
0.5
0u4
(d) Effect of Context Length
(c) Effect of Prediction Length
Bur.,
Lu
Dee.
Iut, m
04-
04
(041 CLEVI
(422) Dancing Sprins
[+4] Dandin CLEVR
(+43 Dancing Sprins
Figure 6: Ablation Study Results. (a) Architectural ablations for the predictive objective and concept memory on the Dancing- CLEVR dataset. (b-c) Varying number of blocks and concept memory size the CLEVR Hard dataset. (d-e) Varying context length and prediction length on the Dancing- CLEVR and Dancing Sprites datasets.
Bancing-CLEVRR
that the model's performance is sensitive to the size of the concept memory. Specifically, increasing the concept memory size leads to better disentanglement, but excessively large sizes can degrade performance. This suggests that there is an optimal range for the concept memory size, and finding this balance crucial for achieving good disentanglement while maintaining overall performance.
Effect of Context Length and Prediction Length. Figures 6 (d-e) show the results of varying context and prediction length on the Dancing-CLii CLEVR and Pancing-Sprites datasets. We observe that performance generally improves with longer context lengths, particularly in terms of disentangiement and informativeness. For Pancing-Sprites there is significant drop in performance when the context length is less than two, aligning with our design intention that requires observing more than three frames to determine dance patterns accurately, while performance declines again when the context length exceeds four frames. This may be because the dance patterns in this dataset repeat every 4 frames, so a longer context length will not provide more information to the model. We also find that increasing prediction length larger than 2 has negative affect on performance, particularly for I-D on the Dancing-CLe CLEVR dataset. Since the decoder is shared for the prediction across timesteps, this may be due the limited capacity of the decoder to represent longer-term predictions.
LIMITATIONS & CONCLUSION
In this paper, we introduced Dreamweaver, novel neural architecture for unsupervised learning of compositional world models from videos. The key component of Dreamweaver is the Recurrent Block-Slot Unit, which encodes a temporal context window of past images and maintains set of independently updated block-slot states, enabling the emergence of abstraction for both static and dynamic concepts. Our model outperforms state-of-the-an object-centric methods and demonstrates the ability to generate videos through novel compositional imagination. This is the first time such compositional imagination has been shown in object-centric learning from videos.
While Dreamweaver represents significant advancement in the unsupervised learning of composi- tional world models from visual data, it does have some limitations that future work may address. First, extending Dreamweaver to probabilistic model could improve its ability to manage uncertainty and generate more diverse and realistic videos. Second, although Dreamweaver has demonstrated the emergence of static and dynamic concepts, exploring the emergence of other abstract concepts, such as numbers, could be fascinating. Lastly, our model shares the general limitation of current ofte--of-the-ar object-centric learning, which is not yet applicable to very complex scene images. Addressing these areas could lead to improved world models.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of our research. To this end, we intend to make all resources, including the code and datasets specifically designed for this work, publicly available. Prior to release, we will thoroughly verify all implementations and empirical results to guarantee their accuracy and reliability.
ETHICS STATEMENT
Although we address fundamental issue in modern deep learning, we are not aware of immediate societal concerns. While carrying deep potential, the practical applications of Dreamweaver remain in developmental stages, indicating that its immediate impact on broader technology or industry sectors is yet to be fully realized. However, future extensions that extend this framework to real-world settings should be mindful of such impact. Also, the environmental impact of training transformer models at scale should be considered in future extensions.
ACKNOWLEDGEMENT
This research was supported by the Global Research Development Center (GRDC) through the Cooperative Hub Program 2SS-2024-00436165) and by the National Research Foundation of Korea (NRF) through both the Young Researcher Program (No. (12RICCCC100443) and the Brain Pool Plus Program (No. 202HHID342403103365) funded by the Ministry Science and ICT.
REFERENCES
Rim Assouel, Lluis Castrejon, Aaron Courville, Nicolas Ballas, and Yoshua Bengio. Vim: Variational independent modules for video prediction. In Conference on Causal Learning and Reasoning, pp. 70-89. PMLR, 2022.
Jacob JW Bakermans, Joseph Warren, James CR Whittington, and Timothy EJ Behrens. Constructing future behaviour in the hippocampal formation through composition and replay. bioRxiv, pp. 2023 04, 2023.
Zhipeng Bao, Pavel Tokmakov, Yu-Xiong Wang, Adrien Gaidon, and Martial Hebert. Object discovery from motion cetion-guided tokens. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22972-229811 2023.
Timothy EJ Behrens, Timothy H Muller, James CR Whittington, Shirley Mark, Alon B Baram, Kimberly Stachenfeld, and Zeb Kurth-Nelson. What a cognitive map? organizing knowledge for flexible behavior. Neuron, 100(2):490-509 2018.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv 2019.
preprint
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML 2020: 37th International Conference on Machine Learning, volume pp. 1691-1703, 2020.
Joseph Cho, Fachrina Dewi Puspitasari. Sheng Zheng, Jingyao Zheng, Lik-Hang Lee, Tae-Ho Ho Kim, Choong Seon Hong, and Chaoning Zhang. Sora as an agi world model? a complete survey on text-to-video generation. arXiv preprini wrXiv:2403.051311 2024.
Eric Crawford and Joelle Pincau. Exploiting spatial invariance for scalable unsupervised object tracking, arXiv preprint 1xX1v:1911.00033, 2019a.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of AAAI, 2019b.

---

Published as a conference paper at ICLR 2025
David Ding, Felix Hill, Adam Santoro, Malcolm Reynolds, and Matthew M. Botvinick. Attention over learned object embeddings enables complex visual reasoning. In Neural Information Processing Systems, 2020.
Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu. Pieter Abbeel, Joshua B Tenenbaum, et al. Video language planning. arXiv preprint arXiv:2310.10625. 2023.
Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference Learning Representations, 2018.
Gamaleldin F. Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C. Mozer, and Thomas Kipf. SAVi++: Towards end-to-end object-centric learning from real-world videos. In Advances in Neural Information Processing Systems, 2022.
Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In International Conference on Learning Representations, 2020.
SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, and Geoffrey Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, pp. 3225-3233, 2016.
Jerry A Fodor et al. The language of thought, volume 5. Harvard university press Cambridge, MA, 1975.
Anirudh Goyal and Yoshua Bengio. Inductive biases for deep learning of higher-level cognition. Proceedings of the Royal Society A, 47802266):202100688 2022.
Anirudh Goyal, Alex Lamb, Phanideep Gampa, Philippe Beaudoin, Sergey Levine, Charles Blundell, Yoshua Bengio, and Michael Mozer. Object files and schemata: Factorizing declarative and procedural knowledge dynamical systems. arXiv preprint arXiv:2006. 16225, 2020.
Anirudh Goyal, Aniket Didolkar, Nan Rosemary Ke, Charles Blundell, Philippe Beaudoin, Nicolas Heess, Michael C Mozer, and Yoshua Bengio. Neural production systems. Advances in Neural Information Processing Systems, 34:25673-25687, 2021a.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and
Bernhard Schölkopf. Recurrent independent mechanisms. ArXiv, abs/1909. 10893, 2021b.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber, Neural expectation maximization. In Advances in Neural Information Processing Systems, pp. 6691-6701, 2017.
Klaus Greff, Raphaël Lopez Kaufmann, Rishab Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. arXiv preprint arXiv: 2019.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208 2020.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint 1803 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint 181 1,0455,, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning 2019.
behaviors by latent imagination. arXiv preprint
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193 2020.
Alain Hore and Djemel Ziou Image quality metrics: Psnr vs. ssim. 2010 20th international conference on pattern recognition, pp. 2366-2369. IEEE, 2010.
12

---

Published as a conference paper at ICLR 2025
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world mod- els with scalable object representations. In International Conference on Learning Representations, 2019.
Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-centric slot diffusion. arXiv preprint arXiv:2303. 10834, 2023.
Jindong Jiang, Fei Deng. Gautam Singh, Minseung Lee, and Sungjin Ahn. Slot state space models. arXiv preprint arXiv:2406.122 12272, 2024.
Laurynas Karazija, Iro Laina, and Christian Rupprecht. Clevrtex: texture-rich benchmark for unsupervised multi-object segmentation. arXiv preprint 1rXiv:2111.10265, 2021.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6988, 2014.
Thomas Kipf, Gamaleldin Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional Object-Centric Learning from Video. arXiv preprint arXiv:2111.12594 2021.
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. Advances in Neural Information Processing Systems, pp. 8606-8616 2018.
Zeb Kurth-Nelson, Timothy Behrens, Greg Wayne, Kevin Miller, Lennart Luettgau, Ray Dolan, Yunzhe Liu, and Philipp Schwartenbeck. Replay and compositional computation. Neuron, 111(4): 454-469, 2023.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
Kwonjoon Lee, Huiwen Chang. Lu Jiang, Han Zhang, Zhuowen Tu, and Ce Liu. Vitgan: Training gans with vision transformers. arXiv preprint arXiv:2107.04589, 2021.
Anson Lei, Frederik Nolte, Bernhard Scholkopf, and Ingmar Posner. Compete and compose: Learning independent mechanisms for modular world models. 2024.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Jindong Jiang, and Sungjin Ahn. Improving generative imagination in object-centric world models. In International Conference on Machine Learning, pp. 41 4124, 2020a.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations, 2020b.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention, 2020.
Sindy Lowe, Phillip Lippe, Maja R. Rudolph, and Max Welling. Complex-valle valued autoencoders for object discovery. arXiv preprint 025:::2200000075, 2022.
Kanika Madan, Nan Rosemary Ke, Anirudh Goyal, Bernhard Schölkopf, and Yoshua Bengio. Fast and slow learning of recurrent independent mechanisms. arXiv preprint arXi::2105.087/00 2021.
Raphaël Millière. "are video generation models world simulators?".
Thomas Parr and Karl Friston. The anatomy of inference: generative models and brain structure. Frontiers in computational neuroscience, 12:44 12:420836, 2018.
13

---

Published as a conference paper at ICLR 2025
A. Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- Conditional Image Generation with CLIP Latents. arXiv preprint arXiv:204.006125, 2022.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation, 2021.
Mehdi S. M. Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 9512-9524. Curran Asso- ciates, Inc., 2022. URL https: httpp:://ppccceednnnggee//// wttp://eeeeeiin..........ee.e.ee neurips cc/paper wilesspapppe 2022 file/ htt.a///ccS.cccaaaaaeeeoooaaaeea oappaaaacceecccppppe--ooooeeeeee pdf.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. arXiv preprint urXiv:/206.644 2012.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 10963) 2021a.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 10965):612-634, 2021b.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings the IEEE, 109(5):612266) 62-644, 2021c.
Philipp Schwartenbeck, Alon Baram, Yunzhe Liu, Shirley Mark, Timothy Muller, Raymond Dolan, Matthew Botvinick, Zeb Kurth-Nelson, and Timothy Behrens. Generative replay underlies compo- sitional inference in the hippocampal-prefioonn
circuit. Cell, 186(22):4885 4897, 2023.
Atharva Sehgal, Arya Grayeli, Jennifer J. Sun, and Swarat Chaudhuri. Neurosymbolic grounding for compositional world models. In The Twelfth International Conference on Learning Representations, 2024. URL https: wwwpreviee.....e/////./.. net forum?ouummmmmm id= wKwpdd=KKKDDDDDNN
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Gabriel,, Tong He, Zheng Zhang, Bernhard Scholkopf, Thomas Brox, and Francesco Locatello. Bridging the gap real-world object-centric learning. arXiv preprint ar 12X::2209.1486600 2022.
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. In International Conference Learning Representations, 2022a. URL https: hwpp///ppew/e..////////e. net. het/forum? id=hoOXVOWe3oh.
Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. arXiv preprint arXiv:2205.14065, 2022b.
Gautam Singh, Yeongbin Kim, and Sungjin Ahn. Neural Systematic Binder. International Conference on Learning Representations, 2023.
Gautam Singh, Yue Wang, Jiawei Yang, Boris Ivanovic, Sungjin Ahn, Marco Pavone, and Tong Che. Parallelized spatiotemporal binding. arXiv preprint arXiv:2402. 12.40...7777, 2024.
Paul Smolensky, R Thomas McCoy, Roland Fernandez, Matthew Goldrick, and Jianfeng Gao. Neurocompositional computing human and machine intelligence: A tutorial. Microsoft Technical Report MSR-TR- 2022, 2022.
Christopher Summerfield. Natural General Intelligence: How understanding the brain can help us build AI. Oxford University Press, 2022.
Sjoerd Van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. arXiv preprint arXiv: 1802. 10353, 2018.
14

---

Published as a conference paper at ICLR 2025
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 3998-6008, 2017.
Rishi Veerapaneni, John D Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua B Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. arXiv preprint 1rXiv:1910.122 12827, 2019.
Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. Generative video transformer: Can objects be the words? In International Conference on Machine Learning, pp. 11307-11318 PMLR, 2021.
Yi-Fu Wu, Klaus Greff, Gamaleldin Fathy Elsayed, Michael Curtis Mozer, Thomas Kipf, and Sjoerd van Steenkiste. Inverted-attention transformers can learn object representations: Insights from slot attention. 2023a.
Yi-Fu Wu, Minseung Lee, and Sungjin Ahn. Neural language of thought models. In The Twelfth International Conference on Learning Representations, 2024, URL https: https:://peenev.tte net/ forum? 3dm m.m.
Ziyi Wu, Nikita Dvomik, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsupervised visual dynamics simulation with object-centric models. arXiv preprint arXiv:2210.0586/ 2022.
Ziyi Wu, Jingyu Hu, Wuyue Lu, Igor Gilitschenski, and Animesh Garg. Slotdiffusion: Object-centric generative modeling with diffusion models. arXiv preprint arXiv:2305. /281, 2023b.
Wei Yu, Wenxin Chen, Songheng Yin, Steve Easterbrook, and Animesh Garg. Modular action concept grounding semantic video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3605-3614, 2022.
Andri Zadaianchuk, Maximilian Seitzer, and Georg Martius. Object-centric learning for real- world videos by predicting temporal feature similarities. Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS 2023), 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018
Linfeng Zhao, Lingzhi Kong, Robin Walters, and Lawson L.S. Wong. Toward compositional gener- alization in object-oriented world modeling. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 26841-26864. PMLR, 17-23 Jul 2022. URL https: http:///occeeeddeeeeeeee www.eeiiiininnn...e.e.m www.mmrr.rss....... 62/ zhao22b. html.
Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Yan Yeung, and Chuang Gan. Robodreamer: Learning compositional world models for robot imagination. arXiv preprint arXiv:2404. 2237777 2024.
Daniel Zoran, Rishabh Kabra, Alexander Lerchner, and Danilo Rezende. Parts: Unsupervised seg- mentation with slots, attention and independence maximization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10439-10447, 2021.
15

---

Published as a conference paper at ICLR 2025
A ADDITIONAL RELATED WORKS
Learning Compositional Mechanisms. Unlike model is built on slot-based models, there is another line of work for learning compositional representations motivated by the Independent Causal Mechanisms principle and the Sparse Mechanism Shift hypothesis (Schölkopf al., 2012; 2021b). RIMs architecture family (Goyal et al., 2021b; 2020; Madan et al., 2021; Assouel et al., 2022) decomposes state space representations into separate recurrent components that operate independently. with sparse interactions among them. These models are designed to learn a set of reusable mechanisms that are selectively activated through sparse communication, enabling them to capture recurring concepts across frames. Similarly, our model also learns reusable and modularized representations, however, our representations are always used and more structured by introducing a hierarchical structure, where these blocks are encapsulated within individual slot representations. Lastly, NPS (Goyal et al., 2021a) model, as a recent work of RIMs, learns a set of independent mechanisms capturing the interaction between objects, while ours is still limited in this ability.
Compositional World Models. Previous work has studied world models from the perspective compositional generalization (Zhao et al., 2022; Sehgal et al., 2024; Zhou al., 2024). In particular, Cosmos (Sehgal et al., 2024) a framework for object-centric world modeling that leverages a frozen pretrained vision-language foundation model decompose objects into symbolic attributes. Du et al. (2023); Zhou et al. (2024); Yu et al. (2022); Cho et al. (2024) similarly relies on language to enable compositional generalization. Our model, on the other hand, is able to learn a compositional world model from pixels without the use language input or pretrained foundation models. COMET (Lei al., 2024) also a related model that learns disentangled modes of interaction between objects, but does not learn attribute representations of the objects.
B ADDITIONAL MODEL DETAILS
B.1 IMAGE TOKENIZATION VIA DISCRETE VAE
Since leverage autoregressive image transformer to predict the frames X1+1y-..5 -++1,- train a discrete Variational Autoencoder (dVAE) to generate discrete token representations of these frames and act as prediction targets. The dVVE consists of an encoder fAVAE and a decoder gø dVAB The dVAE encoder takes an image as input and outputs a sequence of patch-level tokens or latent codes 21,1,.... 2t,4' where 2 E I, while the dVAE decoder takes a sequence of tokens as input and decodes it to an image. The dVAE encoder and decoder are trained with an autoencoding objective as follows:
Zt, I,,
5 JOVVE (x+)
&
90
dVAE
LavAr(O, () =1[x, xx1l2.
where we use Gumbel-Softmax relaxation for the discrete latent codes to facilitate training and a simple squared error as the reconstruction loss.
B.2 CONDITIONAL PREDICTION WITH STEP INDICATORS
We train our model to predict multiple future frames x.+1s... X
which encourages the encoder fa to extract consistent dynamic features by preventing the model from overfocusing on image generation. However, predicting all frames simultaneously is inefficient. Instead, we improve efficiency by randomly sampling one frame X++* to predict during each training step. This is achieved through conditional prediction using a single transformer decoder go and a step indicator in-
BOS token as a step indicator. In Dreamweaver, the Beginning-of-Seeree -f-euuuncce (BOS) token in the transformer decoder functions step indicator k. Specifically, initialize distinct BOSa, token for each step and input the corresponding BOSa, token, based on the index sampled during each training step, the first token for autoregressive generation.
Self-modulation with step indicators. In more complex and realistic datasets, relying solely on the BOS token limits conditional prediction performance. To address this, employ the melf-modulation technique (Karras et al., 2019; Lee et al., 2021). Specifically. we enhance the transformer decoder go by replacing every layer normalization with self-modulated layer normalization (SLN), computed as
16

---

Published as a conference paper at ICLR 2025
follows:
Sh, we.k) Te(W,,))) he
+ (e(we.k
where he is the input from the previous layer, μ and denote the mean and variance of the inputs within the layer, while "/e and Be compute the adaptive normalization parameters. Wa,& is a learnable latent vector used for guiding explicit modularized generation.
C ADDITIONAL IMPLEMENTATION DETAILS
C.I TRAINING AND IMPLEMENTATION DETAILS
We utilized images with a 64x64 resolution for all datasets except Doving-CLEVRTer which uses a 128x128 resolution. Each model was trained on NVIDIA GeForce RTX 4090 GPUs with 24GB of memory. The Dreamweaver model underwent 400,000 iterations of training, taking 20 hours for the Moving-Spriten and Moving-Si Dovinn-Spprite--00 Moving-CI.N CLEVR, and Mancing-CLEVV datasets, and 48 hours for Moving-CLEVVRTes Additionally, we employed the self-modulation technique for conditional prediction exclusively the Moving- CLEVRTex dataset.
datasets, 30 hours for the Bancing-Sprites,
C.2 HYPERPARAMETERS
Table details the hyperparameters employed for the various datasets in our Dreamweaver exper- iments. We trained all models using the Adam optimizer (Kingma & Ba, 2014) with 31 set to 0.9 and 32 set to 0.999. Furthermore, we utilized the architecture and hyperparameters of the backbone image encoder specified in Singh al. (2023).
Table 1: Hyperparameters our model used our experiments. We use shortened version the dataset name, omitting prefixes such as "Moving-" or "Dancing-" unless they are necessary to distinguish hetween datasets.
Dataset
Module
Hyperparameter
Moving-Sprites
Dancing- Spritts
CLEVR
CLEVRTex
General
Batch Size
24
24
24
48
Training Steps
400K
400K
400K
400K
Image Size
64 x
64 64
64 x 64
128 x 128
Context Length, T
2
2
Prediction Length, K
2
2
Grad Clip (norm)
0.5
0.5
0.5
0.5
RBSU
# Iterations
3
3
# Slots
5
# Prototypes
64
128
# Blocks
8
8
Block Size
96
96
96
96
Learning Rate
0.00005
0.00005
0.00005
0.00005
Discrete VAE
Patch Size
4 x 4
4 4
4 4
x
Vocabulary Size
4096
4096
4096
4096
Temp. Start
1.0
1.0
1.0
1.0
Temp. End
0.1
1.0
0.1
0.1
Temp. Decay Steps
60K
60K
60K
60K
Learning Rate
0.0003
0.0003
0.0003
0.0003
Transformer Decoder
# Layers
8
8
# Heads
4
Hidden Size
192
192
Dropout
0.1
0.1
0.1
0.1
I Learning Rate
0.0003
0.0003
0.0003
0.0005
17

---

Published as a conference paper at ICLR 2025
C.3 BASELINES
In this paper, we train three baseline models: STEVE, SysBinder, and RSSM. For STEVE, we use the original architecture and hyperparameters, modifying only the slot size to 64 dimensions. For SysBinder, we adapt the architecture to handle video data with recurrent structure and align the hyperparameters with those of our model for more accurate comparison, including block size, the number of prototypes, and the number of blocks. For RSSM, we implement the world model from DreamerV1 (Hafner al., 2019), using 32 dimensions for both deterministic and stochastic latents, while adhering to the other recommended hyperparameters.
For implementation, we utilize the following open-source resources:
STEVE (Singh al., 2022b): http://gttaabbooom//gghhaaaattte
SysBinder (Singh et al., 2023): https///gihhbb.ooommiinggaaaaaae
RSSM (Hafner al., 2020): https///giiubbbboommuuugiphhghee
C.4 DOWNSTREAM MODEL ARCHITECTURE
For the downstream experiments on all models except for RSSM, transformer architecture with layers, 4 heads, 0.1 dropout, and model dimension of 128. We use the output of a learned class token predict the target label.
For RSSM, we use an MIP to make the prediction. While we experimented with matching the number of parameters with the transformer model, we found better performance using a smaller network with 4 layers and hidden dimension of 128 and ReLU activation.
During training, we freeze the up-stream models and train with a learning rate of 3e-4.
C.5 IDENTIPYING BLOCK-FACTIE CORRESPONDENCE.
To swap or change factor, it important identify which block index corresponds to which object factor. For this, we adopt the following procedure. (1) Take a large batch of videos with per-object factor labels given. (2) Extract block-slot representation from the videos using pre-trained RBSU encoder. (3) Match slots with ground truth labels via Hungarian matching using mask overlap. (4) Train a probe to predict ground truth factor labels from the block-slot representation. Use probing methods that provide feature importance (e.g.. LASSO or Decision Trees). (5) Manually inspect the feature importances to identify which block(s) represent a specific ground truth factor.
D ADDITIONAL. EXPERIMENT RESULTS
D.1 VISUALIZATION OF CAPTURED CONCEPTS
Figure and illustrates the concepts represented by each block within learned block- represen- tations. To achieve this, we gather block representations with the same index and apply clustering methods, such as k-means, following the approach Singh et (2023).
For instance, in the Moving-Sprites dataset, block 0 captures shape concepts such as Star and rquare, block captures color concepts such as Yellow and Red, and block captures dynamic concepts such Lift Up & Down and Slide Leftside & Rightside motion. Similarly, in the Dancing-CLEVE dataset, block captures shape concepts, block 0 captures color concepts, and block 4 captures dynamic concepts.
D.2 OUT-OF-DISTRIBUTION COMPOSITIONAL IMAGINATION
In Section 5.2, we demonstrate that model is capable of synthesizing novel videos by recombining block-slot representations. In this section, we further investigate whether Dreamweaver can generate videos featuring objects with compositionally didtt--disttiittioon (OOD) properties. To this end, introduce the Moving-Spriinn -OD dataset, a modified version of the Moving-Spriis dataset. In this version, the model is trained with only 90% of the possible (shape, color, moving direction)
18

---

Published as a conference paper at ICLR 2025
A
[cluster Yellow]
[cluster Square]
[cluster 1: Up Down]
O
C
[cluster 8: Cyan]
[cluster Circle]
[cluster Left Right]
Block 2 represents 'colors'
Block represents 'shapes'
Block 3 represents 'movements'
Figure 7: Visualization Captured Concept in Moving-Sprites dataset.
Time
Time
[cluster 1: Yellow]
[cluster 1: Sphere]
[cluster 1: Lift UpwDown]
Tima
Time
[cluster Blue/Cyan]
[cluster Box]
[cluster 8: Color Patterns]
Block 8 represents 'colors'
Block represents 'shapes'
Block represents 'dynamics'
Figure 8: Visualization of Captured Concept in Dancing- CLEVR dataset.
combinations, deliberately withholding the remaining 10% to test the model's ability to generalize to unseen combinations.
19

---

Published as a conference paper at ICLR 2025
To evaluate -ustr--ddit (OOD) compositional imagination, adhere to the methodology outlined in Section 5.2. As a result, our model successfully generates novel video frames that feature objects with properties not present in the training dataset. For instance, shown in Figure 9, we intentionally modify a block-slot representation of an object characterized by (brown, square, move left) replacing the brown block with cyan block, which is an unseen combination. Consequently. we observed that our model extends its compositional imagination capabilities novel, compositionally unseen objects. Additional examples are provided in Figure 9.
object seen during training into (cyan, square, move left.) by
Cyon circke clown
lime
brown left
Dercamte Colora
Combinations
Combinations
unseen
Figure 9: Out-of-Distributioo Compositional Imagination Example.
D.3 ADDITIONAL EXAMPLES OF COMPOSITIONAL IMAGINATION TASKS
To further illustrate the concept of compositional imagination, we provide additional examples of tasks that demonstrate this capability. Some examples from Moving- Moving-CLEVRTees CLEVRTex dataset are visualized in Figure 10.
burn-in
rollout
burn-in
compositional imagination
Original
Appearance Swap
burn-in
compositional imagination
burn-in
compositional Imagination
<=a
Backgrouud Swap
Dynamic
Figure 10: Compositional imagination examples on Moving-CLEV In this visualization, we demon- strate the generation of compositionally novel videos on visually more complex and textured scenes than the previously tested datasets. Top-Left: We show two original videos from the dataset. Top-Right: We swap the block representations of object texture between two objects Bottom-Leff We swap background blocks the two videos after context phase and roll-out. Bottom- Right: We swap the synamics-capturing capturing blocks of two objects across the two videos after the context phase and roll-out.
videos after the context phase and roll-out.
20

---

Published as a conference paper at ICLR 2025
D.4 ADDITIONAL QUANTITATIVE RESULTS FOR FUTURE IMAGINATION
In addition to the remarkable qualitative outcomes of compositional imagination, we present quantita- tive results in Table 2 to provide a more comprehensive understanding of our work. We computed MSE, LPIPS (Zhang al., 2018), and PSNR (Hore & Ziou, 2010) evaluate compositional imag- ination, where the blocks are manipulated in the final context frame and the remaining video is rolled out. Specifically, for each test video, we performed the following manipulation: we swapped the blocks for each intra-object factor across all object pairs in the video. Alongside, for each of these manipulations, we generated ground truth videos also. To assess the quality of the imagined compositions, we calculated MSE, LPIPS, and PSNR metrics by comparing the compositionally imagined videos to their respective ground truth videos.
Table 2: Quantitative Results for Compositional Imagination Performance. We calculate this metric by averaging over 20 different video scenarios, automatically conducting all semantically possible combinations of slot manipulations per each vidco scenario.
Dataset
Moving-Sprites
Bancing-CLEVE
MSE
1261
1484
LPIPS
0.288
0.395
PSNR
30.42
33.83
D.5 GENERALIZATION TO ENTIRELY UNSEEN CONCEPTS
In order further evaluate the out-o--disti generalization ability of Dreamweaver, additional experiments on the Dancing Sprites dataset for the downstream task from Section 5.3 with novel shapes and dance patterns that were not seen when training the underlying baseline and Dreamweaver models. In this setup, we use the same pre-trained models as used in Section 5.3, but in- troduce novel shapes (OOD Shapes) and 4 novel dance patterns (OOD Dynamics) when training and evaluating the downstream probe. For OOD Shapes, we introduce the following shapes: hexagon, octagon, star_5, star_6 6, spoke_4, spoke_6. 6. For OOD Dynamics, we show the novel dance patterns in Figure 11. We show the results of these experiments in Figure 12. We see that Dreamweaver still outperforms the baselines in this setting and we do not see much performance degradation when compared to the in distribution setting (Figure 5 (left)). This indicates that the block-slot representations can be useful for downstream tasks even on data with static and dynamic concepts not previously during pre-training.
triangle up
triangle down
triangle right
triangle left
2
2
4
3
4
2
Figure 11: Visualization of Unseen Dance Patterns in Dancing-Sprites (OOD Dynamics) Experiments
E DETAILS OF DATASET DESIGN
We provide detailed information that is carefully considered in our dataset design. For a comprehen- sive understanding of our design, refer to the visualized overview in Appendix 13.
21

---

Published as a conference paper at ICLR 2025
Dancing-Spriter (OOD Shapes)
Dancing-Spriie Spriites (OOD Dynamics)
Accuracy (in %)
80
80
60
40
20
Our Model STEVE
20
SysBinder
RSSM
%
Frame Offset
Frame Offset
Figure 12: Downstream Performance with Entirely Unseen Concepts
Visual Complexity
Dynamics Complexity
Moving-Sprites
Moving- CCERn
Moving-CLE CLEVRTex
Bancing-Sprites
Dancing-CLEV
Figure Overview of Designed Datasets
E. SIMPLE DYNAMIC DATASETS
For the datasets with simple dynamics, incorporate sliding movements into the existing static datasets. Moving-Sprites features three fixed-sized objects with 12 shapes and colors sliding in directions 4 different speeds within 2D scene (detailed in Table 3). Moving-CLEVR consists of two or three fixed-sized objects with 3 shapes and 8 colors sliding in 4 directions at a constant speed within a 3D scene (detailed in Table 4). Moving-CLEVRIIE CLEVRTex includes two or three objects of different sizes, each with one of 4 shapes and 10 complex textures, moving textured ground with the same dynamics as Moving CLEVR (detailed in Table 5). Within an episode, multiple objects simultaneously evolve under their corresponding dynamics.
E.2 ADVANCED DYNAMIC DATASETS
For the datasets with advanced dynamics, we introduce additional complexity beyond sliding move- ments. Dancing-Sprites includes three fixed-size objects derived from shapes and 10 colors, each performing one of 4 patterned dances (detailed in Table 6). Each dance pattern comprises a sequence of movement steps, requiring models to understand dynamics over longer context frames. Dancing- CLEVR builds upon Moving-CLe CLEVR by adding two extra motions: 'lift motion', where objects move vertically instead of sliding, and 'color pattern dynamics', where each object follows one of distinct color-changing patterns over time (detailed in Table 7). Similar to simple dynamic dataset, multiple objects simultaneously evolve under their corresponding dynamics within an episode.
22

---

Published as a conference paper at ICLR 2025
Table 3: Primitive Object Factors Moving-Sprites. In addition to their combinations of primitive properties, all objects in the Sroving-Sprices datasets have fixed size of 0.22.
Shape
Color (RGB)
Moving Direction
Speed
square
(0, 0, 255)
Up
0.025
triangle
(0, 128,
Down
0.05
star_4
(255, 255, 0)
Left
0.075
star_5
(255, 0, 0)
Right
0.1
star_6
(0, 255, 255)
circle
(255, 192, 203)
pentaagnn
(165, 42, 42)
hexagon
octagon
spoke 4
spoke spoke_6
Table 4: Primitive Object Factors in Moving-CLEVR, In addition their combinations of primitive properties,
all objects Moving-CLLE CLEVR datasets have a fixed size of 1.6, made of rubber material, and move constant speed.
Shape
Color (RGB)
Moving Direction
Cube
(87, 87, 87)
Forward
Sphere
(173, 35, 35)
Backward
Cy1 1 inder
(42,75, 215)
leftside
(29, 105, 20)
Rightside
(129, 74, 25)
(129, 38, 192)
(41, 208, 208)
(255, 238, 51)
Table Primitive Object Factors Moving-CLEVEI All ohjects Moving-CLEVRTex slide at fixed speed. For materials, we use free textures provided by Polyhaven, You can directly download through weweete__eeneeeeeeeeeeeeeeeeeeee ion (Karazija et al., 2021) code (link).
Shape
Materials
Size
Moving Direction
Cube
whitemarble
1.6
Forward
Sphere
polyhavee_leather_eed
2.0
Backward
Cylinder
polyyhaven_aactoyy_all
deftside
Torus
pol collyhaven_ccaakeecccccreeeee
Ri Rightside
pol polyy_haven_stony__lir.eattt
polyhaver.paiateed
polyhaven_ran raw
bollyharen_larrg__rreyeiiiie
23

---

Published as a conference paper at ICLR 2025
Table 6: Primitive Object Factors in Dancing-Sprites Along with their combinations of primitive properties, all objects the Dancing-Sprin Sprites datascts maintain fixed size of 0.22 and move at constant speed of 0.15, For more details on dance patterns, see Figure 14.
Shape
Color (RGB)
Pattern
square
(0, 0, 255)
Clockwise_$qpun
triangle
(0, 128, 0)
Soanter_Clockee cclcckiise_quurr
star_4
(255, 255, 0)
Clockwise __l___iamoddd
circle
(255, 0, 0)
Muuttr__lockwisse Diamond
pentagon
(0, 253, 255)
spoke_5
(255, 192, 203)
(165, 42, 42)
(0, 255, 0)
(255, 0, 255)
(255, 165, 0)
clockwise square
counterelockuise square
2
clockuise diauonnd
counterelockuise diamord
4
Figure Visualization of Patterns Dancing-Sprites. There are four distinct dance patterns, each composed of a sequence four movements. Specifically, slockwise_Square follows Nleft, down,
follows [right, down, left, upl; cennter_llockwieeee_ee right, upl; clockwise_Diaae amonnd follows [downright, downleft, upleft, upright]; Counter Centter_llccwieee Diamond follows [dowlleft, downright, upright, upleft].
Table 7: Primitive Object Factors in Dancing-CLEVR, This dataset retains all object properties from Moving- CLEVR, with the addition of new dynamic patterns. The maximum height of the lifting motion is twice the object's height. For more details on color patterns, see Figure 15.
Shape
Color (RGB)
Dynamics
Cube
(87, 87, 87)
Slide Forward
Sphere
(173, 35, 35)
Slide Backward
Cylinder
(42,75, 215)
Slide leftside
(29, 105, 20)
Slide Rightside
(129, 74, 25)
Color Pattern: Cl
(129, 38, 192)
Color Pattern: CCM
(41, 208, 208)
Color Pattern: 3-hep CW
(255, 238, 51)
Color Pattern: 3-hop cow
Lift Up
Lift Down
24

---

Published as a conference paper at ICLR 2025
CW
CCW
3-hop
Cw
3-hop
CCW
(a) Color Map
(0) Color-changing Pottems
Figure 15: Visualization of Color-changing Patterns in Bancing-CLEVE The dataset includes four distinct color-changing patterns, illustrated in (b). Each sequence generated following the color map in (a): (1) Clockwise Selection (CW), (2) Counter- -lonter- Selection (CCW), (3) 3-hop Clockwise Selection (3-hop CW), and (4) 3- 3-hop
Selection (3-hop CCW).
25


Reviews:
[{"review_id": "XaE35DUds7", "reviewer": "Reviewer_aMJV", "summary": "This paper proposed a neural architecture called Dreamweaver designed to discover hierarchical and compositional representations from raw videos. The core contribution of this work is  a novel Recurrent Block-Slot Unit (RBSU) to decompose videos into their constituent objects and attributes. Through experiments, the author showed that Dreamweaver can learn a disentangled representation and generate new videos by recombining attributes from different objects.", "questions": "All of my question are provided in the Weaknesses section", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Developed a new module Recurrent Block-Slot Unit (RBSU) to decompose videos.\n\n2. Well-written, easy to follow\n\n3. Experimental results show that Dreamweaver can learn different attributes and freely combine them to generate varied videos.", "weaknesses": "1. Missing related works:  Some related works [1,2,3] also discuss how to use RNNs for composition video generation or use slot attention to learn disentangled representations. The authors could also include these in the related work section.\n\n2. Insufficient comparison: This paper claim to be the first work that can learn both static and dynamic composable concepts in an unsupervised way. But I think Slotformer and Slotdiffusion[3] can do the same decomposition and are not inclued for comparsion. The authors could additionally supplement this part with experiments or explain why it is unfair to compare with SlotFormer and similar methods.\n\n3. Better datasets: CLEVR and Sprites are still too simple. The authors could experiment on more complex datasets, such as Kubric[4] , to better demonstrate the effectiveness of the method.\n\n\nReference:\n\n[1]. Goyal, A., Lamb, A., Hoffmann, J., Sodhani, S., Levine, S., Bengio, Y. and Schölkopf, B., 2019. Recurrent independent mechanisms. arXiv preprint arXiv:1909.10893.\n\n[2]. Yu, W., Chen, W., Yin, S., Easterbrook, S. and Garg, A., 2022. Modular action concept grounding in semantic video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3605-3614). \n\n[3]. Wu, Z., Hu, J., Lu, W., Gilitschenski, I. and Garg, A., 2023. Slotdiffusion: Object-centric generative modeling with diffusion models. Advances in Neural Information Processing Systems, 36, pp.50932-50958.\n\n[4]. Greff, K., Belletti, F., Beyer, L., Doersch, C., Du, Y., Duckworth, D., Fleet, D.J., Gnanapragasam, D., Golemo, F., Herrmann, C. and Kipf, T., 2022. Kubric: A scalable dataset generator. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 3749-3761).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gbQKUNfvvO", "reviewer": "Reviewer_ixLS", "summary": "This paper presents a neural network model named Dreamweaver, designed to learn compositional world representations from videos in an unsupervised manner, without the need for auxiliary data such as text or labeled masks. It utilizes a novel Recurrent Block-Slot Unit (RBSU) to extract modular representations of objects and their attributes, including both static and dynamic attributes. By training to predict future frames rather than reconstructing them, Dreamweaver can generate future video sequences based on learned compositional features and performs exceptionally well in world modeling and compositional reasoning tasks across various datasets.", "questions": "Although the model can predict short-term dynamic scenes, as the prediction time extends, the generated frames may gradually deviate from the true trajectory. For example, in the Dancing-CLEVR example 3 shown in Figure 4, the last frame's blue sphere is slightly deformed and enlarged. How does the model's performance change as the prediction time increases?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Dreamweaver learns compositional world representations without relying on auxiliary data such as text or labeled masks.\n\n2. The proposed RBSU captures both static factors (such as shape) and dynamic factors (such as motion direction), allowing the model to generate new video sequences by recombining learned object attributes.\n\n3. Dreamweaver performs well in new object configurations and arrangements outside the training set, demonstrating strong adaptability.\n\n4. By predicting future frames, Dreamweaver enhances its ability to represent dynamic concepts, outperforming models trained using reconstruction objectives.", "weaknesses": "1. The architecture of Dreamweaver relies on complex Recurrent Block Slot Units (RBSUs) and self-regressive Transformer decoders, requiring significant computational resources and memory, especially when processing long video sequences or higher resolution videos.\n\n2. Due to the use of Discrete VAE (dVAE) for image token representation, Dreamweaver may be limited in video generation quality, particularly in applications that require fine visual details. For example, in the Moving-Sprites experiment shown in Figure 4, when objects in the video overlap, the shapes in the generated video frames may become slightly distorted.\n\n3. Although the model can generalize to simple object configurations out of distribution, its generalization ability may be limited in complex scenes. The model's performance on real-world videos with high visual complexity is unknown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v7xRgVO0Pk", "reviewer": "Reviewer_aS2J", "summary": "1. The authors propose a novel method Dreamweaver for learning composable concepts(static and dynamic) from videos in an unsupervised way.\n2. The authors introduce motion into existing 2d and 3d static datasets at two different complexities (simple and advanced)\n3. The authors demonstrate the effectiveness of their method on their datasets along three axes of comparison; concept discovery, compositional generation and out-of-distribution generalisation.", "questions": "see weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The authors are the first to introduce a method to learning dynamic composable concepts from videos in an unsupervised way on top of static composable concepts while maintaining disentanglement.\n2. The authors introduce a novel module Recurrent Block Slot Unit to model dynamic concepts.\n3. Instead of the traditional reconstruction objective, the authors use a predictive objective to model dynamic concepts better.\n4. The authors demonstrate the effectiveness of their method on their datasets along three axes of comparison; concept discovery, compositional generation and out-of-distribution generalisation.", "weaknesses": "1. The compositional imagination evaluation only has qualitative results which while interesting is not very informative about the model's performance relative to the other baselines. Some comparative, quantitative results should help here. For example, the authors can holdout a set of combinations in their dataset during training and evaluate the fidelity and consistency of the imagined results for these unseen combinations using standard generation quality evaluation metrics like FVD (Cobbe et al 2019), FID (Heusel et al 2017) etc.\n\n2. The OOD results contain novel factor combinations. But do they contain entirely unseen object shapes, dynamics etc. ? if no, unless there is a specific challenge prohibiting such an evaluation, these results should also be informative about the model's out of distribution generalisation. For example, generalisation to entirely new object shapes, object textures/colors and object motion could be evaluated.  \n\n3. The authors have only evaluated their results on their own datasets. Some results on existing datasets like MOVI or CLEVRER should be very interesting to have and would help contextualise the model's performance. Such an evaluation should provide insights into the challenges of extending the proposed approach to more complex scenes with more diverse objects, motions and occlusions. If there are technical challenge prohibiting such evaluations, the authors should clearly explain what those issues may be.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "paoW1LC8q0", "reviewer": "Reviewer_ZQtq", "summary": "This paper studies the problem of discovering shared visual concepts from videos and recomposing the concepts to unseen videos. The authors propose a novel neural architecture, Dreamweaver, operating on video object-centric representations. In Dreamweaver, a novel Recurrent Block-Slot Unit (RBSU) decomposes videos into objects and attributes and a multi-future-frame prediction loss captures disentangled representations to form both dynamic and static concepts. Experiments demonstrate that Dreamweaver can outperform current SOTA baselines on DCI scores of multiple datasets. The visualization experiments are conducted to show the compositional imagination ability of Dreamweaver as well.", "questions": "1.\tDid the authors try increasing the number of prototypes? I notice that currently, the datasets use different numbers of prototypes. Is the number of prototypes an important hyperparameter for Dreamweaver? The assumption of knowing the number of prototypes in the dataset makes the setting less realistic. Hence it would be important to show whether the method can have reasonable performance when the predefined number is not equal to the total number of rules in the dataset.\n2.\tCould Dreamweaver be extended to discover and simulate multi-object physical concepts like object collision? It seems that the objects cannot interact with each other in current settings.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper is well-written. The authors propose a method that learns a list of static and dynamic abstract concepts from videos. The learned concepts are interpretable, and Dreamweaver can imagine unseen videos by applying the learned concepts to new objects. The authors conduct extensive experiments to show the effectiveness of the proposed method compared with baselines.", "weaknesses": "My major concern is the generalizability of the proposed method. Several design choices prevent Dreamweaver from discovering latent rules beyond the datasets used in the paper. Dreamweaver assumes the videos only involve single-object moving, which significantly constrains the possible dynamic Dreamweaver can represent. It is unclear how the architecture can be modified to relax this assumption. The model also assumes a predefined set of prototypes that will be seen during training. The only generalization we would see in the test split is a novel combination of prototypes and objects, which is not surprising in slot-based architectures trained on synthetic datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)