Reusable Slotwise Mechanisms
Authors: Trang Nguyen
Amin Mansouri
Kanika Madan
Nguyen Duy Khuong
Kartik Ahuja
Dianbo Liu
Yoshua Bengio
Venue: NeurIPS 2023 poster (2023)
Keywords: Out-of-Distribution Generalization
Slotwise
Visual Reasoning
Video Prediction
Reusable Mechanism
Dynamics modeling
TL;DR: Improving dynamics modeling with reusable mechanisms and an efficient communication among object-centric representations

Abstract: Agents with the ability to comprehend and reason about the dynamics of objects would be expected to exhibit improved robustness and generalization in novel scenarios. However, achieving this capability necessitates not only an effective scene representation but also an understanding of the mechanisms governing interactions among object subsets. Recent studies have made significant progress in representing scenes using object slots. In this work, we introduce Reusable Slotwise Mechanisms, or RSM, a framework that models object dynamics by leveraging communication among slots along with a modular architecture capable of dynamically selecting reusable mechanisms for predicting the future states of each object slot. Crucially, RSM leverages the Central Contextual Information (CCI), enabling selected mechanisms to access the remaining slots through a bottleneck, effectively allowing for modeling of higher order and complex interactions that might require a sparse subset of objects. Experimental results demonstrate the superior performance of RSM compared to state-of-the-art methods across various future prediction and related downstream tasks, including Visual Question Answering and action planning. Furthermore, we showcase RSM’s Out-of-Distribution generalization ability to handle scenes in intricate scenarios.

Reusable Slotwise Mechanisms
Trang Nguyen"
Amin Mansouri
Mila Quebec Al Institute
Mila - Quebec AT Institute
FPT Software Al Center
Université de Montréal
Montreal, Canada
Montreal, Canada
trang nguyen@mila, quebec
mansouri@min quebec
Kanika Madan
Khuong Nguyen FPT Software Al Center Ho Chi Minh City, Vietnam khuongnd6@fpt.coo
Kartik Ahuja
Mila Quebec Al Institute
Mila Quebec Al Institute
Université de Montréal
Montreal, Canada
Montreal, Canada madankan@mila. quebec
kartik ahu j la quebec
Dianbo Liu
Yoshua Bengio
Mila Quebec AI Institute
Mila Quebec Al Institute
National University of Singapore Montreal, Canada
Université de Montréal, CIFAR AI Chair
Montreal, Canada
dianbo. liu@mila. quebec
yoshua. o quebec
Abstract
Agents with the ability to comprehend and reason about the dynamics objects would be expected to exhibit improved robustness and generalization in novel scenarios. However, achieving this capability necessitates not only an effective scene representation but also an understanding of the mechanisms governing in- teractions among object subsets. Recent studies have made significant progress in representing scenes using object slots. In this work, introduce Reusable Slotwise Mcchanisms, or RSM, framework that models object dynamics by lever- aging communication among slots along with modular architecture capable of dynamically selecting reusable mechanisms for predicting the future states of each object slot. Crucially, RSM leverages the Central Contextual Information (CC!), enabling selected mechanisms to access the remaining slots through a bottleneck, effectively allowing for modeling of higher order and complex interactions that might require a sparse subset of objects. Experimental results demonstrate the superior performance of RSM compared state-of-the-an methods across various future prediction and related downstream tasks, including Visual Question Answer- ing and action planning. Furthermore, we showcase RSM's Out-of-Distribute generalization ability to handle scenes in intricate scenarios.
Introduction
Accurate prediction of future frames and reasoning over objects is crucial in various computer vision tasks. These capabilities are essential for constructing comprehensive world models in applications like autonomous driving and reinforcement learning for robots. Traditional deep learning-based representation learning methods compress entire scenes into monolithic representations, lacking
"Work done under an internship at Mila Quebec Al Institute and FPT Software Al Residency Program. Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

compositionality and object- understanding. As a result, these representations struggle with systematic generalization, interpretability. and capturing interactions between objects. This limitation leads to poor generalization performance as causal variables become entangled in non-trivial ways.
There has been growing interest in slot-based and modular representations that decompose scenes into individual entities, deviating from fixed-size monolithic feature vector representations [19] 33
16 [7 15 18 32 28 37 39 30]. These novel approaches offer significantly more flexibility when dealing with environments that comprise multiple objects. By employing an encoder that segments a scene into its independent constituent entities instead of compressing information into a fixed-size representation, these methods allow for greater flexibility and parameter sharing when learning object-centric representations, and their compositional nature enables better generalization. Compositional and object-centric representations can be effectively utilized alongside complex world models that accurately capture the interactions and dynamies of different entities in a scene.
These world models, when presented with proper representations, in principle, can model the transition functions that relate latent causal factors across consecutive time steps of rollout. While monolithic blocks are still used occasionally with object-centric methods [44), recent attempts have incorporated similar inductive biases related the object-centricity of images in modeling interactions. Structured world models and representations seem truly promising for systematically generalizing to novel scenes. Structured world models would ideally decompose the description of the evolution of scene into causal and independent sub- modules, making it easy to recombine and repurpose those mechanisms in novel ways to solve challenges in unseen scenarios. Such separation of dynamics modeling makes structured world models more adaptable to distribution shifts, only the parameters of few mechanisms that have changed in new environment would have to be retrained, and not all of the parameters in the case of monolithic model {
major class of such structured world models aims at baking in some inductive bias about the nature of object interactions. On one extreme, there have been studies that employ Graph Neural Networks (GNNs) to capture object dependencies through dense connections, while on the other hand, there has been contrasting work aiming at modeling the dynamics through only pairwise interactions. We believe, however, that ideally, an agent should be able learn, select, and reuse a set of prediction rules based on contextual information and the characteristics of each object.
In this work, argue that the assumptions made in previous attempts at learning the dynamics among slots may be insufficient in more realistic domains. To address these limitations, we propose Reusable Slotwise Mechanisms (RSM), novel modular architecture incorporating set of deep neural networks representing reusable mechanisms on top of slotwise representations [31 [0]. Inspired by the Global Workspace Theory (GWT) in the cognitive neuroscience of working memory 2 3. we introduce the concept of Central Contextual Information (CCI), which allows each reusable mechanism, i.e., a possible explanation of state evolution, to access information from all other slots through a bottleneck, enabling accurate predictions of the next state for a specific slot. The CCT's bottleneck amounts a relaxed inductive bias compared the extreme cases of pairwise or fully dense interactions among slots. Finally, comprehensive experiments demonstrate that RSM outperforms the state-of-the-art in various next-step prediction tasks, including independent and identically distributed (iid) and
(OOD) scenarios.
In summary, the presented work makes the following contributions:
1. RSM: A modular dynamics model comprising a set of reusable mechanisms that take as input slot representations through attention bottleneck and sequential slot updates.
2. RSM strengthens communication among slots in dynamics modeling by relaxing inductive biases in dynamics modeling to be not too dense or sparse but depend on a specific context.
3. RSM outperforms baselines in multiple tasks, including next frames prediction, Visual Question Answering, and action planning in both iid and OOD cases.
2 Proposed Method: RSM Reusable Slotwise Mechanisms
2.1 RSM Overview
We introduce RSM, modular architecture consisting of set of M Multilayer Perceptrons (MLPs) that act as reusable mechanisms, operating on slotwise representations to predict changes in the slots.

---

rollout steps
K
Transition Model
steps temporal window
Decode
CCI
Encode
Encede
T burn-in frames
(a) Spatial-temporal based Future Prediction Pipeline
(b) RSM Intuition
Mech
Mech
Mech
Medh
Mech
CCI
Slot
Slot 2
Step
Step
Step
sequeatial
Thistorical stepe
Ast
Mechanisms Assignment Matrix
Step 4: Update the predicted next state to the system
F'ri
(c) Computational Flow in RSM from step to f
Figure 1: The future prediction pipeline (Figure I(a), RSM Intuition (Figure 1(b) and Computational Flow (Figure [(c)). Colored circles represent slots, with the dashed border denoting changes in a slot. The Central Contextual Information (CCI) derived from all object slots context, assists in selecting mechanism for slots, and acts as an input of mechanisms. Slots are sequentially updated in four steps: (1) Compute CCI by unrolling slots (updated and non-updated)
the past steps, (2) Select a mechanism based on CCI and the slot of interest, (3) Predict the next state by the selected mechanism's 's dynamics, and (4) Update the predicted slot and prepare for the next object's turn.
What sets RSM apart from other architectures incorporating the CCI buffer, which enhances its adaptability when dealing with environments characterized by varying levels of interaction complexity among objects by allowing the propagation of information about all other slots. Unlike previous approaches 18 16, 17 23 we enable a sparse subset of slots to transmit contextual information through a bottleneck for updating each slot. This inductive bias becomes helpful environments where higher-orde complex interactions need to be captured by reusable mechanisms as the CCI effectively modulates the complexity of the mechanisms and accommodates those that require one or larger subset of slots.
two slots, as well as those that rely
The training pipeline of RSM, which is visualized in Figure ((a) is designed predict K rollout steps of N object slots, based on T burn-in frames with a temporal window of maximum J history steps for each prediction step. Prediction of the rollout begins by processing the given T burn-in frames to obtain the N slot representations for each of the I' burn-in steps using an object-centric model. For any rollout step after the burn-in frames, the slots will be fed to the model as additional context to predict the state (slots) at + 1. The model takes output the slots at t + 1. The sequential updating
window of T T' previous steps
this input and updates the slots sequentially of slots means that updates from (according to some ordering) slots can influence the prediction of later slots within prediction step, as illustrated in Figure 1(b) sequential way of updating slots breaks the symmetries in mechanism selection, and also allows for more expressive transition function, similar to how autoregressive models enable encoding of rich distributions. The prediction process is repeated until the slots for K rollout steps are predicted.
worth highlighting that

---

2.2 Computational Flow in RSM
This section describes the computational flow of RSM in more detail along with 4-step process that will be repeated for all slots within time-step I, in sequential manner, as illustrated in Figure l(c) and Algorithm T in the Appendix. The following are the main components of the architecture, where dg and deci denote the dimension of slots and the CCI, respectively:
1. MultiheaadAttenttioll
: [R((++)x)))xd + Rd. followed by projection o(-)
IR, + Rdss that computes the CCI, denoted as cci E R& from all of the N slots in the past T steps concatenated. Keys, queries, and values all come from slots, so the CCI is not affected by the order of slot representations.
2. The set of M reusable mechanisms {gI..... :......M) where gi(+) Rdu+dd - Rd, are repre- sented by independently parametrized MLPs implicitly trained to specialize explaining different transitions. Each such g:(-) takes as input one slot concatenated with the CCI.
3. (+)) : Rdattd R- that takes as input the CCI and the slot of interest st. It computes a categorical distribution over the possible choices of mechanisms for si and outputs sample of that distribution to be used for updating s:.
Considering the N slots per each of the T steps in the temporal window before t, 5. :/ St-++1 ....,,...,,,, s,, sf. with -" = T + 1, RSM predicts the next state of slots, denoted as st+1 using the following 4-step process, which is sequentially applied to each of the slots. Suppose an ordering has been fixed over the slots for rollout, and according to that ordering, for some 0 < n S N, we have that n - slots have been updated to their predicted values at ++1 and are denoted by s/1 $1+1...... next state for s7 (e.g. the blue slot in Figure I(c)
s/1:N
Below, we explain the process of computing the
Step 1. Compute the CCI: We first append s s/1:N to the current temporal window achieve !l:N +1 The duplicated sf=:N serves as a placeholder, which will be overwritten with the predicted values in subsequent steps. Subsequently, as presented in Equation [ Multihead (ttention()) takes 1:N sy1 as input before passing through Φ(-) to produce the central contextual information cci.
cci @[Muulihendd
Step 2. Select mechanism for si: ((()) takes arguments as inputs, cci from Step and s?, to output the logits of a categorical distribution TI:M over M possible choices of mechanisms. We employ hard Gumbel-softmax layer with Straight-through Trick [33 22 on top of gi's outputs, as described in Equation 2 to select the mechanism.
TI:M = Sounbl--ofttax((((cc s)))
(2)
Step 3. Predict the changes of s!:: Let AsT denote the change of of from t to t + 1. As" is presented in Equation 3 where k (rggmx((TI..))
A gN gu s()
(3)
Step 4.
Update and sync si+1: 8Th1 is then computed by adding the predicted transformation from the previous step and replaces the value of 8 in the slots buffer, as described in Equation
c-1 = De + A
/1:N The process above is repeated for all slots at time I, to obtain the next state (slots) prediction 8
3 Experiments Setup
This study evaluates RSM's dynamics modeling and generalization capabilities through video pre- diction, VQA, and action planning tasks. We aim to provide empirical evidence supporting the underlying hypotheses that guided the architectural design of RSM.
Hi: Slots communication through the CCI and reusable mechanisms reduces information loss during prediction, resulting in accurate prediction of future rollout frames (Section 4.1

---

N2: RSM effectively handles novel scenarios in the downstream tasks (Section 4.2 cially enhancing OOD generalization (Section 4.3)
espe
H3: The synergy between the CCI and the disentanglement of objects dynamics into reusable mechanisms is essential to RSM (analyses in Section 4.4 and ablations in Section 4.5)
In the following subsections, describe the experiments focusing the transition of slots over rollout steps with pre-trained object-centric models, Additionally, Appendix D provides experiments and analyses with an end-to-end training pipeline.
3.1
Environments
OBJ3D (29]
contains dynamic scenes of a sphere colliding with static objects. Following Lin et al. use 3 to 5 static objects and one launched sphere for interaction.
29
Wu et al. 44
CLEVRER 45 shares similarities with OBJ3D, but additionally has multiple moving objects in various directions throughout the scene. For the VQA downstream task, CLEVRER offers four question types: descriptive, explanatory, predictive, and counterfactual, among which, in the spirit of improving video prediction, we focus on boosting the performance on answering predictive questions which require an understanding of future object interactions.
PHYRE [4] is a 2D physics puzzle platform where the goal is strategically placing red objects such that the green object touches the blue purple object. Bakhtin al. 4 design templates that describe such tasks with varying initial states. Subsequently, they define (1) within-lemplate protocol where the test set contains the same templates as training, and (2) cross-template protocol that tests on different templates than those in training. We report results both on within-template as iid and on cross-template obtain the OOD evaluation.
Physion 7 a VQA dataset that assesses a model's capability in predicting objects' movement and interaction in realistic simulated 3D environments in eight physical phenomena.
For further details and data visualization, we refer the readers to Appendix B
3.2 Baselines
We compare RSM against three main baselines: SlotFormer 44 Switch Transformer [4] denoted SwitchFormer, and NPS [17). We show the efficacy of relaxing the inductive bias on communica- tion density among slots in RSM by contrasting with dense communication methods (SlotFormer and SwitchFormer) and a pair-wise communication method (NPS). Likewise, comparing RSM to SlotFormer highlights the role of disentangling objects' dynamics into mechanisms, while comparing to SwitchFormer and NPS emphasizes the vital role of communication among slots via the CCI. A Additionally, we compare to SAVi-Dyn [44), which is the SOTA on CLEVRER. In other experiments, we compare to SlotFormer (current SOTA).
In the tables, we present our reproduced SlotFormer (marked by "!") and our re-implemented SwitchFormer and NPS (marked by "x"), alongside SAVi-Dyn reported by Wu al. 44 2
3.3 Implementation Details
Following Wu al. 44 we focus on the transition of slots and take advantage of the pre-trained object-centric encoder-decoder pair that convert input frames into slots and vice versa. We use the pre-trained weights of SAVi and STEVE provided by Wu et al. [44), including SAVi (26) for OBJ3D, CLEVRER, and PHYRE; and STEVE 40 for Physion.
We present the best validation set configuration of RSM for each dataset, along with fine-tuning results and model size scaling in Appendix [ In summary, (1) OBJ3D and CLEVRER include 7 mechanisms, while PHYRE and Physion use 5, and (2) the number of parameters in RSM is slightly lower than that of SlotFormer in corresponding experiments. Additionally, we re-implemented SwitchFormer and NPS with a similar number of parameters as in RSM and SlotFormer.
We adapt the code for Switch Transformer and NPS to ensure consistency of experimental setups and evaluations with SlotFormer and RSM (See Appendix C
5

---

Table 1: Future frame prediction quality OBJ3D and CLEVRER. Bold scores indicate the best performance, with the RSM consistently outperforming baselines by a remarkable margin.
OBJ3D
CLEVRER
Method
SSIM
LPIPS
1004
SSIMI
LPIPS 1004
ARIT
FG-ARI1
FG-mloUf
SAVi-Dyn
0.91
12.00
0.89
19.00
8.64
64.32
18.25
NPS
0
8.24
0.89
12.51
62.84=0.2
30.39 64.62
SwitchFormer'
0.91
8.09
0.88+0.3
14,2840.1
60.61
59.3240.3
28.9420.2
SlotFormer'
0.90
8.32
0.88
13.09
63.38 +0.3
62.91 40.2
29.68
RSM (Ours)
0.92
7.88 10.1
0.91 1
11.96220.1
67.72=0.2
66.15-00.2
32.73
Rollout step=
10
20
30
40
49
Remarks
Gold
Accurate dynamics Sharpness preserved
RSM (Ours)
0.
Incorrect dynamics
NPS
Changed shapes
Incorrect dynamics
SlotFormer
0.
Lost sharpness
Incorrect dynamics
SwitchFormer
Figure 2: Comparison of rollout frames in OBJ3D. RSM renders frames with precise dynamics and holds visual quality, even complex actions in steps 20 to 30. In contrast, the baselines produce ones with artifacts (yellow boxes) and inaccurate dynamics (red boxes). Best viewed in video format.
Experimental Results
We report mean and standard deviation across 5 different runs. Video visualizations of our experiments are provided in our repository See also Appendix A on the reproducibility of our results.
4.1 Video Prediction Quality
To demonstrate Hi, we provide the video prediction quality on OBJ3D and CLEVRER in Table | and Figure In general, RSM demonstrates its effectiveness in handling object dynamics in the long-term future prediction over baselines.
Evaluation Metrics We evaluate the quality and similarity of the predicted rollout frames using SSIM [42] and LPIPS [66] metrics. Since the range of LPIPS metric small, we report the actual values times 100, denoted LPIPS, x000 facilitate comparisons among the methods. Additionally, also assess the performance using ARI, FG-ARI, and FG-mIoU metrics, which measure clustering similarity and foreground segmentation accuracy of object bounding boxes and segmentation masks. We evaluate the model's ' performance averaged over unrolling 44 steps on OBJ3D and 42 steps on CLEVRER with six burn-in frames in both datasets.
Table exhibits that RSM outperforms other approaches and achieves the highest scores across evaluation metrics for both datasets. Notably, compared to SlotFormer, RSM improves LPIPS x 100 by 0.46 points in OBJ3D, 1.12 points in CLEVRER, and increases FG-mloU by 3.05 points in CLEVRER. Following RSM, NPS consistently ranks second in performance among the baselines.
These results are supported by Figure which illustrates the rollout frames in OBJ3D. RSM's outputs' accurate rollout predictions with high visual fidelity demonstrate the efficacy of slot communication by having less error accumulated along time steps than any of the baselines. is worth emphasizing that RSM excels in handling a significant series of complex movements in steps 20-30, particularly
github..om/rrangnn//RSM

---

Table 2: VQA performance CLEVRER and Physion. Despite surpassing human perfor-
mance in Physion, RSM outperforms baselines in both datasets. All scores are in percentage.
Method
CLEVRER
Physion
per opt. f
per ques.
Obs.
Dyn.
Gap t
Human
74.7
NPS*
95.3#0.3
93.8-00.2
65.6
+0,6
SwitchFormer"
92.8+0.3
90.4+0.2
66.2#0.11
+1.2
65.0
SlotFormer
96.1 =0.2
93.3#0011
66.9
+1.9
RSM (Ours)
96,8-t0.1
94.3
68.1 10.0
+3.1
Table 3: Action planning task in PHYRE, RSM outperforms all baselines in both iid and OOD,
PHYRE-B
NPS
SwitchFormer'
SlotFormer'
RSM
iid (within-template)
80.52+10
78.2741.9
76.4
82.89
OOD (cross-template)
42.63#1.3
48.36+144
42.46+1.7
57.37#1.4
during a sequence of rapid collision of object pairs. In contrast, we find that the baselines struggle with complex object movements during this period, leading to inaccuracies in predicting the dynamics towards the end. RSM effectively maintains visual quality by predicting the changes of slots instead of predicting the next state of slot. This approach allows RSM to handle action-free scenarios well and significantly reduce error accumulation by facilitating null transitions that preserve slot integrity.
Furthermore, RSM demonstrates flexible slot communication with relaxed inductive biases interaction density, enabling it to adapt to environments comprising mechanisms with varying levels of complexity. In contrast, we find that NPS with sparse interactions faces difficulties with close- by objects and rapid collisions (seen in OBJ3D), while SwitchFormer with dense communication struggles with distant objects (as in CLEVRER).
4.2
Downstream Tasks: Visual Question Answering and Action Planning
4.2.1 Visual Question Answering task
To demonstrate H2, we validate the performance of future frames generated by the models on the VQA task in CLEVRER and Physion, and the action planning task in PHYRE in the next section. The general pipeline to solve the VQA task with the predicted rollout frames from the given input frames. Specifically, we employ Aloe (II] as the base reasoning model on top of the unrolled frames in CLEVRER. Likewise, in Physion, we adhere to the official protocol by training a linear model on the predicted future slots detect objects' contact. In Physion, from human participants
also include the results obtained ( for reference. Likewise, we collect the results from observed frames (Obs.), which are obtained by training the VQA model on top of pre-trained burn-in slots and compare them to the performance of rollout slots (Dyn.).
In Table 2 RSM consistently outperforms all three baselines in VQA for CLEVRER and Physion. On CLEVRER, RSM achieves the highest scores of 96.8% (per option) and 94.3% (per question), surpassing SlotFormer and NPS. In Physion, RSM shows notable improvement, with a 3.1% increase from 55. in Obs. to 68.1% in Dyn., outperforming all other baselines, indicating the benefit of enhancing the dynamics modeling to improve the downstream tasks. However, RSM still far from human performance in Physion, showing room for further research into this class of algorithms.
4.2.2 Action Planning task
We adopt the approach from prior works [36 and construct a task success classifier as an action scoring function. This function is designed as simple linear classifier, which considers the predicted object slots, to rank pre-defined set of 10,000 actions from 4. executed accordingly. We utilize AUCCESS, which quantifies the success rate over the number of attempts curve's Area Under Curve
7

---

Step=0
4
11" I1
119
Figure 3: Action planning task in PHYRE. RSM strategically positions red ball at step to prevent the green ball from falling onto the glass by causing collision that alters the original trajectory of the green ball and causes make contact with the blue floor (indicated by the arrow).
Rollout
Pred
Slot 1
Slot 2
Slot 3
Slot 4 Slot 5
Slot 6 Slot 7 Slot 8
Mechanism
EEP
t
Mechanism
t 3
Mechanism
t t= 5
Mechanism
Figure 4: The underlying mechanism assignment in PHYRE. Mechanisms assigned each slot at to obtain the updates at +++ 1. RSM disentangles objects' dynamics into reusable mechanisms, which can be expressed as Collision (2), Moving left right (3), Idle (4), and Falling (5).
(AUC) for the first 100 actions. We report the average score over ten folds, with the best performance among five different runs on each fold.
The first line in Table@iindiccaes the action planning results of the proposed RSM and baselines in the iid setting (within-Iemplate protocol). RSM achieves the highest performance compared to baselines, indicating the critical role of efficient slots communication in complex tasks like action planning. In addition, Figure 3 shows successful case of RSM solving the planning task by strategically placing the red object at step O, causing collision between the green and blue objects at the end.
4.3 Out-of-Distribution (OOD) Generalization
To provide more evidence for H2, we resume analyzing the performance of the action planning task in PHYRE in two OOD scenarios:
OOD in Templates Performance cross-template protocol indicated in the last line in Table Overall, RSM demonstrates strong generalization and has the smallest gap between iid and OOD performance compared to the baselines. The cross-template is a natural method evaluate the OOD generalization in PHYRE since scenes in the train and test sets are in distinct templates and contain dissimilar object sizes and objects in the background further details and visualizations.
We refer the reader Appendix for
OOD in Dynamics We introduce modifications to PHYRE dataset generation that introduce distribution shifts to the dynamics of the blue objects. Particularly, the blue objects are static floors during training. However, the test set, the object can be assigned as blue ball, inheriting all dynamics as red or green ball. As result, the blue ball appearing only in the test set is considered strong OOD case, as the model does not have chance construct the blue ball and its dynamics during training. Presented in Figure although revealing some distortion in shape, RSM demonstrates the ability generalize the movements of the ball-shaped object intensive OOD cases.
4.4 The Ability to Disentangle Object Dynamics into Mechanisms
To demonstrate H3, we conduct qualitative analysis on the underlying mechanisms assignment within the four steps of Figure 3 and visualize results in Figure 4 While there is explicit assignment

---

Training
Exclusions
Rollout
Step=0
2
4
Gold
Pred*
Pred
Mcchanism
Fall
Fall
Collide h. Movec Move Nloveh.M. Move Move
Figure 5: Ablation on OOD objects' dynamics. The blue objects are static during training (blue floor) and can move during testing time (blue balls). RSM responds correctly to the blue ball's dynamics. Pred are predictions by model trained with the standard training set, Pred are predictions by model trained with the modified training data, and Mechanism are mechanisms assigned for the blue object's slot when obtaining Pred. h.Move denotes the horizontal movement.
Gold
RSM
RSM
RSM3
RSMg
RSM
LPIPS, 100l
7.88
15.32
9.36
13.72
8.43
Step 10
Step 49
Figure 6: Ablation studies in OBJ3D. The original design of RSM always demonstrates the superior performance and the accurately predicted future frames compared to the modified versions, including breaking the usages of the CCI in step 2 (RSM22) and step 3 (RSM)3), randomly selecting mechanisms (RSMg), and parallel update of slots (RSMp). Best viewed in video format (See our repository).
of roles to mechanisms during training, we can infer their functionality at convergence based on slot visualizations and patterns of activation follows: Mechanism 2 handles collisions, which can be observed in the collision between the green and red balls in step and that of the red ball with the right-side wall step 5 (blue boxes). Mechanism 3 controls the horizontal movements, observed in the green ball from steps 5 to 10 (green boxes). Mechanism acts as an idleness mechanism. Mechanism 5 handles downward free-fall motion, observed from steps to 3 of the two balls. Mechanism 1 does not seem to be doing anything meaningful, and this could be due to the model using more capacity than it requires model the dynamics in this environment.
The inferred functions provide the following insights into understanding the efficacy of RSM. Firstly, RSM successfully disentangles the dynamics into reusable mechanisms, as described in Section 2.1 Secondly, RSM properly assigns such mechanisms to slots throughout the rollout steps, which not only helps preserve the accuracy of prediction but also emphasizes the effectiveness of communication among slots in deciding mechanisms for one another. The automatic emergence of null mechanism (mechanism 4) is also worth highlighting, which significantly helps reduce the error accumulation in action-frre free settings, such as idle objects in the background.
4.5 Ablation Studies
To provide more evidence for Has we conduct ablations to understand the individual effects of (1) the CCI, (2) mechanisms and their disentanglement, and (3) the sequential update of slots in RSM, and visualize the results in Figure Overall, the ablations confirm the superiority of the original RSM design compared to all variations, highlighting a significant disparity in the absence of CCI.
RSM masks out the CCI step 2 at inference. We observe that lack CCI leads to a degenerate selection of mechanisms for slots, with 4 out of 5 object slots being controlled by mechanism 3. RSMg masks out the CCI in step 3 during inference. We have found that CCI not only captures comprehensive spatial- spatial-tempot temporal information but also provides guidance regarding specific movement

---

details, including directions. In particular, even when the correct mechanism is assigned (e.g. moving backwards), the slots become confused about the exact direction of movement. Additionally, slots encountered a color-related issue in the subsequent steps.
RSM& randomly assigns mechanisms to slots by randomized mechanism index, k, to replace the distribution и in Equation 2 We observe that the launched ball moves in the wrong direction and exits the scene early, while other objects shake in their positions, underscoring the importance of correctly assigning mechanisms to slots.
RSMp is the parallel version of RSM that modifies the model (-) in Equation 2 to assign mechanisms to N slots simultaneously. in both training and inference time. We find that RSMp stands out promising model, as it demonstrates a notable LPIPS performance; however, it is essential to note the partially inaccurate dynamics caused by the weaker communication among slots.
5 Related Work
Modular dynamics models. In the domain of modular neural networks, RIMs 18 pioneered the exploration of modularity for learning dynamics and long-term dependencies in non-stationary settings. However, RIMs suffer from conflating object and mechanism concepts, limiting their effectiveness. SCOFF [16] introduced object files (OF) and schemata to address this limitation, but struggles with generalizing out-of-distribution (OOD) scenarios. Key distinetions between RSM and SCOFF are as follows: SCOFF schemas can handle only one OF at time, while RSM allows multiple slots to be input to reusable mechanisms using an attention bottleneck. SCOFF and RIMS use sparse communication among OFs for rare events involving multiple objects, whereas RSM leverages the CCI to activate suitable reusable mechanisms. NPS [17] incorporates sparse interactions directly into its modular architecture, eliminating the need for sparse communication among slots or OFs. Their "production rules" handle rare events involving multiple objects by taking one primary slot and a contextual slot as input. A recent benchmark (3] evaluates causal discovery models and introduces a modular architecture with dense object interactions, similar to GNN- based methods, but assigns separate mechanisms to each object. Among the class of algorithms with less modularity in their dynamics model, R-NEM [1 was a pioneering unsupervised method for modeling dynamics using learned object-centric latent space through iterative inference (see also 13) which along similar approaches (16), (5) used Graph Neural Networks (GNNs) to model pairwise interactions and differ from our work in two key aspects. Firstly, we do not rely on GNNs to model interactions, as dense interactions are not always realistic in many environments. Secondly, we focus on learning a set of simple and reusable mechanisms that can be applied flexibly different scenarios, rather than compressing information through shared node and edge updates in GNN.
Unsupervised learning of object-centric representations. To decompose a scene into meaningful sub-parts, there have been lots of recent works on unsupervised learning of object-centric representa- tions from static images 48] 20 31 47]. videos [35 27 25 12 44 and theoretical results on the identifiability of such representations [34 2] Although lots of these methods work very well in 31 to be consistent with the baselines.
practice, we decided to proceed with slot attention
6 Conclusion
In this study, we developed RSM, novel framework that leverages an efficient communication protocol among slots to model object dynamics. RSM comprises a set of reusable mechanisms that take as input slot representations passed through bottleneck, the Central Contextual Information (CCI), which are then processed sequentially to obtain slot updates. Through comprehensive empirical evaluations and analysis, we show RSM's 's advantages over the baselines in various tasks, including video prediction, visual question answering, and action planning tasks, especially in OOD settings. Our results suggest the importance of CCI, which integrates and coordinates knowledge from different slots for both mechanism assignment and predicting slot updates. We believe there is a promise for future research endeavors in exploring more sophisticated stochastic attention mechanisms for information integration, aligning with the principles of higher-level cognition, coping with a large number of object slots, and enabling uncertainty quantification in the predictions. In Appendix F we discuss the limitations of this work and future directions.
10

---

Acknowledgement
We gratefully acknowledge the support received for this research, including from Samsung, CIFAR and NSERC. This research was partly enabled by computational resources provided by Mila Quebec AI Institute and The Digital Research Alliance of Canada (formerly known as Compute Canada). Each member involved in this research is funded by their primary institution. Amin Mansouri acknowledges support from Microsoft. Additionally, we sincerely thank the anonymous reviewers for their valuable comments and contributions to improve this work.
References
[1] Kartik Ahuja, Jason Hartford, and Yoshua Bengio. Weakly supervised representation learning with sparse perturbations. arXiv preprint arXiv: Arxiv-2206.01122 2022.
[2] Bernard J. Baars. Global workspace theory of consciousness: toward a cognitive neuroscience of human experience. In Steven Laureys, editor, The Boundaries of Consciousness: Neurobiology and Neuropathology, volume 150 of Progress in Brain Research, pages 45-53, Elsevier, 2005. doi: http::/dbb.org//1..0016660001/ 50000 BtLhhtts:://ww sciencedirect com/ http://awweo/rrrcaoooo////////00
[3] Bernard Baars. The Global Workspace Theory of Consciousness. John Wiley and Sons, Ltd, 2017. ISBN 9781119132588
[4] Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. Phyre: A new benchmark for physical reasoning. 2019.
[5] Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray Kavukcuoglu. Interaction networks for learning about objects, relations and physics. In NIPS, 2016.
[6] Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonnalez, Vinícius Flo- Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Çaglar Gülçehre, H. Francis Song, Andrew J. Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey R. Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matthew M. Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. CoRR, abs/1806.01261 2018. URL htt httt:://axxi> org/ abs 1806.1106...00 01261
[7] Daniel M. Bear, Elias Wang, Damian Mrowca, Felix J. Binder, Hsiau-Yo Yu Fish Tung, R. Pramod, Cameron Holdaway, Sirui Tao, Kevin A. Smith, Fan-Yun Sun, Li Fei-Fei, Nancy Kanwisher, Joshua B. Tenenbaum, Daniel L. K. Yamins, and Judith E. Fan. Physion: Evaluating physical prediction from vision in humans and machines. CoRR, abs/2106.08261, 2021. URL https: /taps:///arxiii org/ abs/ 2206 2110..006001111111
[8] Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sébastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. A meta-transfer objective for learning to disentangle causal mechanisms. In (CLR'2020, arXiv:1901. 10912, 2019.
[9] Jack Brady, Roland S. Zimmermann, Yash Sharma, Bernhard Schölkopf, Julius von Kügelgen, and Wieland Brendel. Provably learning object-centric representations. arXiv preprint arXiv: 2305. 2023.
[10] Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matthew M. Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposi- tion and representation. CoRR, 2019.
[11] David Ding, Felix Hill, Adam Santoro, Malcolm Reynolds, and Matthew Botvinick. Atten- tion over learned object embeddings enables complex visual reasoning. In A. Beygelzimer, Dauphin, Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. http:ppp///pppeeeeeeeeeeeeee/rrs

---

[12] Gamaleldin Fathy Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael Curtis Mozer, and Thomas Kipf. SAVi++: Towards end-to-end object-centric learning from real-world videos. 2022.
[13] S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, Koray Kavukcuoglu, and Geoffrey E. Hinton. Attend, infer, repeat: Fast scene understanding with generative models. CoRR, abs/1603.08575, 2016. URL httlht/tp//aiit org/ abs/ 1101.66300000 08575
[14] William Fedus. Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL
https://arxiv. org/ abs 03961
[15] Anirudh Goyal and Yoshua Bengio. Inductive biases for deep learning of higher-level cognition. CoRR, 1.tp/20011150000 2020.
[16] Anirudh Goyal, Alex Lamb, Phanideep Gampa, Philippe Beaudoin, Sergey Levine, Charles Blundell, Yoshua Bengio, and Michael Mozer. Object files and schemata: Factorizing declarative and procedural knowledge in dynamical systems. arXiv preprini arXiv:2006. 16225, 2020.
[17] Anirudh Goyal, Aniket Didolkar, Nan Rosemary Ke, Charles Blundell, Philippe Beaudoin, Nicolas Heess, Michael Mozer, and Yoshua Bengio. Neural production systems. CoRR, abs/2103.01937, 2021.
[18] Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. 2021.
[19] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. CoRR, abs/1410.5401, 2014.
[20] Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. Advances in Neural Information Processing Systems, 2017.
[21] Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. Tracking the world state with recurrent entity networks. 2017.
[22] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmant In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:// openreview hwt.:eeetfoooaeeeee
[23] Nan Rosemary Ke, Aniket Didolkar, Sarthak Mittal, Anirudh Goyal, Guillaume Lajoie, Stefan Bauer, Danilo Rezende, Yoshua Bengio, Michael Mozer, and Christopher Pal. Systematic evaluation of causal discovery in visual model based reinforcement learning. 2021.
[24] Thomas Kipf, Elise van der Pol, and Max Welling. Contrastive learning of structured world models. In International Conference on Learning Representations, 2020.
[25] Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. ICLR, 2022.
[26] Thomas Kipf, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object- centric learning from video. In International Conference on Learning Representations, 2022. teeee//oorumm..-a///esss...
[27] Shiqian Li, Ke Wu, Chi Zhang, and Yixin Zhu. On the learning mechanisms in physical reasoning. ArXiv, abs/2210.02075. 2022.
[28] Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indmn)) Building a longer and deeper rnn. 2018.
[29] Zhixuan Lin, Yi-Fu Wu, Skand Peri, Bofeng Fu, Jindong Jiang, and Sungjin Ahn. Improving generative imagination in object-centric world models. In Proceedings of the 37th International Conference Machine Learning, ICML'20. JMLR.org, 2020.
12

---

[30] Dianbo Liu, Vedant Shah, Oussama Boussif, Cristian Meo, Anirudh Goyal, Tianmin Shu, Michael Curtis Mozer, Nicolas Manfred Otto Heess, and Yoshua Bengio. Stateful active facili- tator: Coordination and environmental heterogeneity in cooperative multi-agent reinforcement learning. ArXiv, abs/2210.03022, 2022.
[31] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. N0, 2020.
[32] Kanika Madan, Nan Rosemary Ke, Anirudh Goyal, Bernhard Schölkopf, and Yoshua Bengio. Fast and slow learning of recurrent independent mechanisms. 2021. URL attps://arxiv brggabbs21105.000
[33] Chris J. Maddison, A. Mnih, and Y. Teh. The concrete distribution: A continuous relaxation of discrete random variables. International Conference On Learning Representations, 2016.
[34] Amin Mansouri, Jason Hartford, Kartik Ahuja, and Yoshua Bengio. Object-centric causal representation learning. NeurlPS Workshop on Symmetry and Geometry, 2022.
[35] Adeel Pervez, Phillip Lippe, and Efstratios Gavves. Differentiable mathematical programming for object-centric representation learning. arXiv preprint arXiv: Arxiv-22 (0000-2/00021159, 2022.
[36] Haozhi Qi, Xiaolong Wang, Deepak Pathak, Yi Ma, and Jitendra Malik. Learning long-term vi- sual dynamics with region proposal interaction networks. In International Conference on Learn- ing Representations, 2021. URL
net. wwt.:eeetfooueeee htt/fformm??d
X 4AkcdöRe
[37] Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, and Tim Klinger. Routing networks and the challenges of modular and compositional computation. CoRR, abs/1 12774, 2019.
[38] Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. Relational recurrent neural networks, 2018. https:/t:///arxxi.....
[39] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- gated mixture-of- experts layer. CoRR, abs/1701.06532. 2017.
[40] Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances Neural Information Processing Systems, 2022. URL
wtwp:eewwwwwewweffrfeeeeeee met/forrmmiddaee IM88MTUE
[41] Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and J. Schmidhuber. Relational neural ex- pectation maximization: Unsupervised discovery of objects and their interactions. International Conference On Learning Representations, 2018
[42] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-612 2004. doi: 10. 109/TIP.2003 200....000..1886... 819861.
[43] Ziyi Wu, Nikita Dvorikk, Klaus Greff, Jiaqi Xi, Thomas Kipf, and Animesh Garg. Slotformer: Long-term dynamic modeling in object-centric models. 2022.
[44] Ziyi Wu, Nikita Dvoriik, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsuper- vised visual dynamics simulation with object-centric models. In The Eleventh International Conference on Learning Representations, 2023. URL |RLttttss: http/operevvee.nee//eeeee
id=TFbwV6IOVLg
[45] Kexin Yi, Chuang Gan Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. Clevrer: Collision events for video representation and reasoning. In International Conference on Learning Representations, 2020. URL httpttsst///ppreeeeeeee
13

---

[46] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as perceptual metric. In 2018 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 386-595, Los Alamitos, CA, USA, jun 2018. 8. IEEE Com- puter Society. doi: 10. N0.100//VVP.220 20..2201..0000 8.00068. Utt.https://doi .eeeccoptterssciet
jun
org/10. 1109/
1009/CVPR. CVPR. 2018. 800068
[47] Yan Zhang, David W. Zhang, S. Lacoste-Julien, G. Burghouts, and Cees G. M. Snoek. Unlocking slot attention by changing optimal transport costs. ARXIVOOG,, 2023. doi: 10.48550/arXiv. 2301.13197.
[48] Mingde Zhao, Zhen Liu, Sitao Luan, Shuyuan Zhang, Doina Precup, and Yoshua Bengio. A consciousness-insee -nnpreee planning agent for model-based reinforcement learning. Advances in neural information processing systems, 1569-181,,,, 2021.
14

---

A Reproducibility
Each experiment is trained on 4 VI00-GPUs with 12 CPUs, using distributed data-parallel training strategy. The number of parameters and the average training time over runs are summarized in Table In addition, Table providess the necessary configurations reproduce our work, including the setting related to datasets and the training process that follows the prior work [44), and RSM's design that achieves the best tuning results.
Regarding the rollout frames K and the video length V in Table 5 we predict and consider K future frames from the last burn-in steps for training, whereas, produce the total of V frames in the inference time, including 7 burn-in and V T rollout steps. other words, equals T plus the actual rollout frames inference time. Regarding the training process, employ the Adam optimizer with an initial learning rate of x 10 and employ a cosine decay schedule to We further diseuss the RSM design in Appendix C
Table 4: Summary of the number of parameters and training duration. "M" stands for millions. "h" stands for GPU hours.
RSM
NPS
SwitchFormer
SlotFormer
OBJ3D
Num. Params
0.76M
0.99M
0.82M 20h
0.82M
Training Duration
21h
25h
21h
CLEVRER
Num. Params
3.1M
4.06M
3.22M 72h
3.22M
Training Duration
82h
94h
86h
PHYRE
Num. Params
5.13M 29h
5.98M
6.38M
6.38M
Training Duration
33h
28h
30h
Physion
Num. Params
5.61M 32h
6.7M
6.41M 30h
6,41M
Training Duration
41h
34h
Table 5: Summary of experiments' configuration, including the configuration of datasets, training process, and RSM.
OBJ3D
CLEVRER
Physion
PHYRE
Frame Size
64 X 64
64 X 64
128
128
128
128
Num. Slots
6
7
Slot Size d,
128
128
192
128
Burn-in Frames T
6
15
15
Temporal Window
6
15
15
6
Rollout Steps K
10
10
10
10
Video Length V
6+44
15+42
15+35
1+14
Batch Size
128
128
128
64
Num. Epochs
200
80
25
50
Object-centric Model
SAVi
SAVi
STEVE
SAVi
Loss Weight
1.0
1.0
0.0
0.0
Num. Mechanisms M
7
7
5
5
Num. Layers of v--)
I
2
2
Num. Layers of Mechanism
3
3
3
B Dataset Collection
We follow the collection and pre-processing of the dataset, including video length and the dataset splits, as done by Wu et [44] addition, we provide datasets visualizations in Figure and the visualization of templates in PHYRE in Figure g
15

---

Algorithm Reusable Slotwise Mechanisms
N: number of slots
M: number of mechanisms
7: number of history length
T: number of burn-in frames
K: number of rollout steps
dg: slot dimension
does: central contextual information dimension
Input: J... (
-:-1.1................1......... f1-++1 st. 87. ................................ sf .8) E R((xN)xx, with T"
st
T + 1: unrolled N slots of the previous T steps to time
Output: J:N TTTKK predicted N slots in the next K steps from time T + to
Variables in RSM
cci Rdoe; the Central Contextual Information (CCI).
s TRd:; the slot of interest, which is slot n1th time f.
p E RM; the Gumbel distribution over M choices of selecting mechanisms.
AsT E Rd: changes of the n-th slot from time to + 1.
Components in RSM:
Wq. Wk. W. RR - Rdd denote query, key, and value projection layers transforming the unrolled 7 in the attention mechanism.
I:N
MultiheadAttention() [((++))xN)xx, + Rd: apply self-attention on the s
<(-) R de computes the central contextual information by passing the outputs of the MultiheadAtttentinn through a nonlinear transformation (MLP).
<6.)) dd. - R M computes the unnormalized probability of selecting mechanism from M possible choices by taking the CCI and slot of interest as input and feeding that to an MLP.
Set of M mechanisms gj (-) Rddertd, + Rd, j (1... M ): predict the changes of each slot based on the CCI and current state of the slot. These are also realized with MLPs.
for each in TT.. K) do
Step 0: Prepare slots buffer concat(s s J:N st J.N) {B((+++)xN)xd
each si J.NN with n E E..... do
Step 1: Compute the central context
cci = MoMMltihead
W (s))). Wx(s)-M (s
Step 2: Select a mechanism for slot si
p Gumbel-max( wwt.maaaaa///////mae (concat(cci, s(
Step 3: Apply the selected mechanism to slot si Note that is one-hot-like distribution.
Ast 9, (conccat,,ci,, s) Vj (I...,,
AsT SM Es
Step 4: Update the slots buffer with the new value of 8
sh+1 Si + As
end for
end for return 1:N TT1.TTKK
16

---

Step=
0
6
12
18
24
30
36
42
48
OBJ3D
Step=
14
18
22
26
30
34
38
42
9
8
CLEVRER
Step=
10
15
20
25
30
35
40
45
50
Physion
Step=
2
3
4
5
6
7
8
PHYRE
Figure 7: Dataset visualization steps.
Figure 8: Pairs of PHYRE scenes in the same template with similar objects in the background and differences in objects' positions.
OBJ3D We collect the OBJ3D dataset from the official GitHub repository
CLEVRER We directly download the CLEVRER dataset from the official website
PHYRE We use the PHYRE-1B version, which sets the number of red balls to dataset generated based on the instructions provided on official GitHub page
The PHYRE
Physion We directly download the Physion dataset from the official GitHub page
C
Implementation Details
C.1 Loss Functions
The following is the training objective that follows the prior work
44
We employ slot reconstruction loss to train the rolled out future frames prediction, as described in Equation 5 where n is the slot index, 87 is the predicted rollout slot, and 87 the pre-trained slot (that is used as the target slot).
K
N
Ls K.N
Σ
18744
- on 1|
(5)
k=
Experiments using SAVi as the object-centric model also the image reconstruction loss, as frozen decoder, and TT is the ground truth image.
described Equation 6 where faee
gelthbb.om//hixxua--ini-----W
clevrer.csail clevrer.ssal.mit.edu/ mit.ed/
http:://ww......o.o.ooooooooooon the whl--phh-oossassseee
17

---

Experiments using STEVE as the object-centri model can still employ the image reconstruction loss; however, we do not conduct such experiments with image reconstruction loss due to the dramatically extended training time. In PHYRE, we do not utilize the image reconstruction loss L, due to the large image size that could affect the training time and the simplicity of PHYRE's objects compared to other environments in this work.
K
-1= K
Σ I|faco(ST+s) - xT++|l
(6)
The overall objective function is the weighted sum of the above losses, as presented in Equation 7
L= Ls + XC1
Note that when using slot and image reconstruction losses as presented in Equation S and Equation 6 the model will fail in the end-to-end training since the overall objective function is constrained to pre-trained slots (s7++) and well-trained decoder (faee(-)). Motivated by this observation, in Appendix H we provide the end-to-end training objective to compare RSM and other approaches
abilities in objects' dynamics modeling from scratch.
C.2 Models Architecture
The distinction among methods is in the process of predicting the next state based on the input of past states. In this section, we provide a detailed description of how both the baselines and the proposed RSM approach the task of the next state prediction. Specifically, we examine two key aspects: (1) how slots communicate with cach other and (2) how the predictions computed, revealed through the design specifics of each method.
C.2.1 Baselines
SlotFormer consists of 3 main parts: (1) the Multi-Layer Perceptron (MLP) input projection layer, (2) a Transformer architecture layer, and (3) the MLP output projection layer. First, the unrolled past states (the sequence of r x N slots) passed through the input projection layer before being processed by the Transformer model. Afterward, the output projection produces the next state from the Transformer's output. Through this process, slots densely communicate with each other in all three parts of SlotFormer. In addition, the entire next state of slots is directly generated by the models. In this work, utilize the publicly available implementation of SlotFormer.
SlotFormer 474
SwitchFormer from Switch Transformer [4].
The Switch Transformer is variant of the Transformer architecture designed to improve efficiency and scalability. It achieves this through the usage of dynamic routing and adaptive computation. This architecture employs a "switch" module that intelligently routes tokens to different layers based on their content.
In this work, we create SwithFormer that integrates the Switch Transformer implementatior into the SlotFormer codebase and replace the vanilla Transformer by Switch Transformer. In this way, SwitchFormed follows the same strategy as in SlotFormer, which conducts the dense communication among slots and directly predicts the entire state of slots.
NPS 17 NPS is a framework that combines neural networks and production systems for object modeling that integrates neural networks into the production system. In traditional production systems, rules are used to represent knowledge and guide the system's behavior. In the case of NPS, they conduct set of rules to handle the pair-wise interaction of slots. The two slots involved in an interaction, which are called the primary and contextual slots, are selected through attention mechanisms. In addition, the official design of NPS for object dynamics' modeling, inspired by Kipf et al. (24,, predicts the changes of the primary slot within a time step instead of the entire slots. Afterward, the predicted next state of slots is the sum of the current state and the predicted slots changes.
In this work, integrate the official NP the SlotFormer's codebase for consistency in the training pipeline and sharing the pre-trained object-centric model. Besides, NPS includes an MLP
mn. wallaamm.aarrmnssoorerrrsiee//ee
github.com/anirudh9) 19/ crapiiaa_roouuutoooyyee systems
18

---

slot encoder that requires fixed imput size. However, the temporal window size in PHYRE increases from to 6, leading to the difference in the imput size of the starting and later steps. Therefore, the beginning of the rollout prediction process of PHYRE, we duplicate the burn-in frame to have fixed six steps window size along the rollout process.
C.2.2 RSM
We design RSM as a framework for dynamics modeling with relaxed inductive bias in the com- munication density of slots that enables a subset of slots involved in communication, based on a particular context through the CCL. RSM consists of three main elements described in Section 2.1 (1) the multi- head self-attention that computes the CCI, (2) the (b(+) that estimates the suitable mechanism, and (3) a list of mechanisms. In terms of the multi-head self-attention, We employ 4-head architecture for multi-head self-attention, where the hidden size of the Feed-forward Networks is x d4. We design ( as MLP with the number of hidden layers being tuned using the validation set (See Appendix D.3). Similarly, each individual mechanism is designed as MLP layers with tuned number of hidden layers. All mechanisms share the same architecture but have separate weights. In addition, the total number of parameters when considering all mechanisms is constrained be the same across when the number of mechanisms varies, meaning that as the number of allocated mechanisms increases, the number of parameters per each mechanism decreases (further investigated in Appendix D.3). Like NPS, the mechanism predicts the changes in slot within two consecutive steps instead of the entire slot. Last but not least, we (and NPS) omit the input and output projection layers as SlotFormer and SwitchFormer.
C.2.3 Downstream tasks
CLEVRER VQA Inheriting from the baseline of Wu et al. 44], we employ Aloe as the VQA model that concatenates the predicted rollout slots and the processed question (represented as language tokens) before passing them through a stack of Aloe Transformer encoder predict the answers.
Physion VQA In the VQA task of Physion, the objective is determine whether the red object will come into contact with the yellow object once the dynamics of all the objects have been completed. Since the task does not involve any language processing. we construct an MLP model that takes the rollout slots as input. The MLP processes these slots and produces binary prediction, indicating whether the red object and the yellow object "touched" or "did not touch".
PHYRE Readout In the PHYRE action planning task, an action involves determining the size and position of the red ball. In our approach, we utilize the set of 10,000 predefined actions introduced by 4 and train a readout model to determine if a given action can solve the task. To construct the readout model, we draw inspiration from Wu et al. 44 that design 2-layer MIP model on top of the encoded states. The readout model takes the predicted rollout states as input. To process these states, we employ an encoder, which differs depending on the specific model variant used, In SlotFormer, Transformer is used, while in SwitchFormer, Switch Transformer is employed. An MLP is operated as the encoder in the NPS model, and a Multi-head Self-Attention mechanism utilized in the RSM model. Once the encoder has processed the states, the classifier generates a binary output indicating whether the task has been solved. This output serves as an inference for the task's 's solvability.
D Further Discussion on Experiment Results
D.1 Rollout Future Prediction
Figure 9 depicts the rollout frames generated by RSM alongside the baselines. Notably, RSM excels in producing robust future frames that accurately capture the dynamics of objects while maintaining visual fidelity. Nevertheless, we have encountered a challenge in generating objects with sharpness within the CLEVRER dataset. SlotFormer marks partially incorrect object's t's dynamics in this case.
D.2 Discussion on the Action Planning task in PHYRE
We encountered difficulties in reproducing the action planning results of SlotFormer, even when using their provided checkpoints. In Table we report the iid result of 76.4 for SlotFormer, which 5.6 points lower than the officially reported value. To investigate this issue, explored potential reasons
19

---

Rollout step=
14
18
22
26
30
34
38
42
46
Gold
RSM
SlotFormer
Figure 9: Comparison of rollout frames in CLEVRER.
6.56
Num. Params (M)
6
5.05 5.13.11354
5.35
5.35
5.75
5.37 5.61
5.96
5
4
3.0 3.37
3-
2.57
2-
1-
0.810.770.760.75
OBJ3D
CLEVRER
PHYRE
Physion
Environments
Figure 10: The scaling of RSM's parameters with different amounts of mechanisms.
and identified the following possible factors: (1) Instability of results: The official GitHub page of SlotFormer acknowledges that the results in this specific task may exhibit instability. This suggests that achieving consistent and reproducible results with SlotFormer can be challenging and (2) data discrepancies: The PHYRE dataset, being regenerated rather than downloaded from a common source, introduces variations in the computing configuration. These data collection and processing differences may contribute to the disparities observed between our results and the reported values in Wu al. 43
D.3
Finetuning Results in RSM
In Figure we present the results of hyperparameter fine-tuning on CLEVRER and Physion datasets. This section explores the impact of the number mechanisms, the expansion of vt-)) parameters, and the structure of the mechanism models. The term number of layers in this analysis refers to the hidden layers within the MLP structure that maps the input dimension the output dimension the respective models.
In terms of the number of mechanisms, have observed that having either large small value for M significantly worsens the results and leads to high variance. However, we have also discovered that the tasks can be solved with relatively few mechanisms, even when dealing with diverse object movements. To validate the design principle stated in Appendix C.2.2) which proposes an inverse relationship between the number of the mechanism's 's parameters and the number of mechanisms, we conducted experiments denoted as 10", which replicates mechanisms to achieve a total of 10 mechanisms without reducing the number of parameters, as compared the best configuration (the first boxes). Our findings indicate that the replicated configuration (10*) achieves a slightly lower score than the best configuration, and the additional mechanisms are not selected by the v)(+) models.
In terms of the number of layers ()).). the challenge to map a 2 x d, vector to compact vector hidden layers yields favorable results in terms
of size M. Our findings suggest that using of achieving high score. However, we find a complication in identifying consistent patterns for fine-tuning v-(-) across different datasets.
gethub.com/paiiaa//lofffoom
20

---

LPIPS 400
16-
14-
Num. Mechanisms
Num. Layers of
Num. Layers of Mechanism
(a) CLEVRER. The lower score indicates better performance.
68
Dyn. (%)
66-
61
62-
10*
Num. Mechanisms
Num. Layers of v
Num.
Layers of Mechanism
(b) Physion. The higher score indicates better performance.
Figure 11: The finetuning results in CLEVRER and Physion. The first box of each subplot the configuration that achieves the best performance, whereas other boxes are performance that has different number mechanisms, layers of the to model, or layers of each mechanism from the best setting. The plotted values are the mean and standard deviation over 5 different runs corresponding to each configuration. See text for more details.
Lastly, when considering the mechanism structure, we have noticed that increasing the number of parameters allows a single mechanism achieve moderately decent score, although a high score. Consequently, the vi(-) model tends to choose only one mechanism for all cases, resulting in lower score when assigning
layers to the mechanism.
E Experiments on End-to-end Training Pipeline
We propose an additional experiment to verify the methods' ability to model objects' dynamics from scratch, additionally, in action-conditioned environments. We establish end-to-end training pipeline that comprehensively assesses the models' effectiveness in the entire process of extracting slots from frames, handling the objects' action-conditioned dynamics, and finally decoding slots back into frames.
RSM generally demonstrates superior ability model objects' dynamics and produce meaningful slots compared to the baselines.
E.1 Experiment Setup
Environment:
This dataset consists of objects arranged in 5 x 5 grid. At each time step, single frame and an action are provided. The action specifies one object and one manipulation from the set of UP, RIGHT, DOWN, and LEFT. The challenge of this task is to determine the feasibility of the given action and predict the resulting frame. For example, an action of moving an object to the LEFT executable if no objects are obstructing the left side of the target object. Otherwise, the frame remains unchanged.
Encoder and Decoder architecture In this experiment, we follow the encoder proposed by Kipf al. (24) that contains a simple CNN-based Object Extractor to extract N feature maps from the input frame that is followed by an MLP-based Object Encoder to encode feature maps to vector representations of objects, i.e. the object slots. Afterward, we exploit a decoder architecture for slot-based visual prediction, consisting of N slot decoders with separate weights. Each slot is decoded an RGB reconstruction, and the final frame reconstruction is obtained by summing the reconstructions of all slots.
21

---

Training Objective. This experiment follows the Contrastive loss setup as Kipf et al. [24] that uses the prediction result of the transition model form the positive hypothesis, whereas, sampling random input states in the same batch forms the opposing hypothesis. The target slots, s/41 are obtained by passing the target next frame through the Encoder, s111 s!:a indicates the random slots in the training batch.
**1NN
I:N denotes the predicted slots, and
MSE($ *1:N), H = MSE(3) s*1:N)
(8)
Contrastive Loss : - + max
We consider the sum of two BCE loss terms training Decoder, L1 and L2. L1 is applied on zip- obtained by passing 8t I:N through Decoder, which is expected to be close to the input frame z4. L2 is applied on 2t+1- obtained by passing st41 through Decoder to achieve the prediction of next frames reconstruction, which is desired to be close to the target next frame I++1-
z = Decoder (st 1:NN
2+++ Decoder(s2 11:N) (5+11 L2 = BCE( 1,
A1:N
L1 = BCE (1 2t),
(9)
BCE Loss Decoder
L1 + L2
E.2 Experimental and Analytical Results
Step=
0
2
3
4
5
Action=
5-R
3-U
2-L
I-U
5-D
Gold
RSM
Mech=LEFT
Mech=RIGHT
Mcch=UP
RSM
RSM33
Figure 12: Observation mechanisms assignment and performance in 5 rollout steps in 2D Shapes. In the last three rows, present the reconstruction by only changing slot with mechanism applied to that slot over steps, while other slots are untouched.
Disentangling objects transition dynamics to mechanisms. In Figure 12] we study the role of each mechanism in RSM. The analysis shows that RSM produces a reasonable reconstruction compared to the ground-truth -trtthh frame and encourages the mechanisms to distinguish themselves in their roles. In this sample, we can infer the slots corresponding to: 1: red circle, 2: blue triangle, 3: green square, purple circle, and 5: yellow triangle. Likewise, we observe the mechanism assignment each step and list the role of each mechanism (5 mechanisms in this experiment) specialized as the following: Mcchanism 1: RIGHT, Mechanism 2: UP, Mechanism 3: LEFT, Mechanism 4: DO NOT MOVE, and Mechanism 5: DOWN.
Looking deeper into the reconstruction results, RSM takes advantage of the CCI assign a suitable mechanism for each slot in all scenarios, considers the particular situation, and reacts accordingly to the same action given the specifics of each context. For instance, we observe that with the same action UP given in step 1 on the green rectangle and step 3 on the red round, RSM recognizes the situations which the object is allowed to move or is blocked by the upper wall, respectively, then applies the movement at step while not modifying objects at step 3 and generates the correct reconstruction in both eases. similar example shown in the row corresponding to mechanism at step -+ 4 when the green object does not move UP and remains at the same position since the red object blocks it.
22

---

Input
Extracted feature maps
Gold Pred
Decoded slots
RSM (Ours)
NPS
SwitchFormer
SlotFormer
Figure 13:
Comparison of extracted feature maps from a scene and reconstructions in 3D Cubes. RSM deals with object slots better than baseline in both slot extraction and slot decoding phases.
Step=
3
4
5
6
9
10
Groundtruth
RSM (Ours)
NPS
SlotFormer
SwitchFormer
Figure 14: Reconstruction comparison on 3D Cubes dataset
The ability to decompose a frame into slots. We analyze RSM's slot-centric representation learning ability. In Figure [3] we illustrate a comparison of the extracted feature maps with a size of 10 x 10, which are constructed by the Encoder model and the reconstructed slots and the frame with size of 3 x 50 x 50, acquired by the SlotDecoder models that receive the input as the predicted next state. We find that RSM decomposes the input frame into separated slots and keeps each object in the same slot until the decoding phase. In contrast, the baselines do not capture all objects but produce noisy feature maps (SlotFormer and SwitchFormer), or put the same object in two slots and identify two objects in another slot (NPS).
We observe that combining the CCI with the sequential updates that encourage slots to observe the modification of each other benefits from recognizing the overlap of objects. More specifically, RSM only generates part of the object in case that object is covered by another one (e.g., the green and blue cubes in slot 2 and slot 4 are partially covered by the yellow cube). On the other hand, the basclines overlook the condition of exccutability of action and generate overlapped objects in some cases (e.g., the green, blue, and yellow cubes in SlotFotmer overlapping one another). Lastly, SwitchFormer produces blurry objects and incorrectly predicts objects' dynamics.
Reconstruction in 3D Cubes. One of the challenges of generating reconstructions in 3D Cubes is to recognize the visibility order of objects. Figure 14 illustrttes RSM's 's strength in communication among slots to obtain the order information, as well as its ability to generate the proper movement of slots and produce an accurate reconstruction compared to the ground truth. In contrast, SlotFotmer misses that kind of information from the beginning steps and renders the blue and green objects inside each other. Besides, other methods lose the information about some objects and produce inaccurate reconstructions at the end, witnessing huge gap from the preceding steps up until step 10.
23

---

Limitations and Future Works
While RSM has demonstrated robustness for modeling objects' dynamics in various tasks in both iid and OOD settings, there is a lot of room to expand this work to overcome the following limitations:
1. Sensitivity to Hyperparameters: RSM requires tuning the number and size of mechanisms. Future research could explore automated methods for determining optimal values and enhancing RSM's adaptability across tasks and scenarios.
2. Time complexity: The sequential update of slots can become computationally expensive in larger systems with many slots (e.g., exceeding 10,000). Although this problem does not affect our work (see Appendix (A) future research should also explore parallelization techniques to improve computational efficiency by assigning mechanisms and predicting the next states simultaneously for large-scale scale systems.
3. Environments in this study are all observable. Future work should explore wider range of observable and unobservable environments to gather deeper insights into the working mechanism of RSM in such domains.
24


Reviews:
[{"review_id": "32vzHc2VFU", "reviewer": "Reviewer_DY7c", "summary": "This work presents a novel framework to model object dynamics by leveraging communication slots in a modular architecture. The method uses Central Contextual Information (CCI) to allow information exchange among existing slots. The authors demonstrate the efficacy of their method's superior empirical performance on VQA, action planning, and future frame prediction. ", "questions": "See weaknesses. ", "limitations": "See weaknesses. ", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using central contextual information is novel and interesting and seems to provide clear empirical benefits over prior work. \n2. The authors have clearly outlined the primary contributions and have carefully designed ablations to support their claim. ", "weaknesses": "One question I have for the authors is how this method would be extended to a real scene where the entities that are interacting cannot be clearly separated. For instance, in CLEVRER or Obj3D, it's known ahead of time that the 3-5 objects would be moving around and interacting with each other. But in a real scene, finding that might get very challenging. I would be interesting in hearing author's perspective about that. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable reviews. We deeply appreciate the time and effort you invested in evaluating our work.\n\nWe are glad to see `novel and interesting`, `clear empirical benefits`, `clearly outlined contributions`, and `carefully designed ablations` in your reviews.\n\n---\n> **One question I have for the authors is how this method would be extended to a real scene where the entities that are interacting cannot be clearly separated.** For instance, in CLEVRER or Obj3D, it's known ahead of time that the 3-5 objects would be moving around and interacting with each other. But in a real scene, finding that might get very challenging. I would be interesting in hearing author's perspective about that.\n\nWe appreciate the insightful comment and acknowledge it is a valid concern. However, we would like to emphasize that RSM is particularly suited to be extended to such scenarios, the reason being that RSM works directly with slots and does not care about their number. And the problem of not knowing the number of objects beforehand has been studied in the literature. Most notably,  going towards more realistic environments, we would use superior object-centric architectures such as Slot Attention [1], SAVi, SAVi++ [2,3], where they have successfully scaled to real-world self-driving datasets [3]. So the task of object discovery in real scenes has been extensively studied, and with enough computing, we could scale to real-world scenes as well. Regarding the variable number of objects, please refer to [4] where a Dirichlet Process Mixture Model is employed for adaptively changing the number of mixtures required to represent the observed data in a continual learning setting. Since the methods in [1,2,3] basically constitute a learnable clustering scheme, we believe [4] could be a reasonable candidate for adaptively choosing the number of such clusters/mixtures (in our case, objects) in real scenes.\n\nTurning our attention to an extension of our work on a more complex dataset, we present additional experimental results from the MOVi-C dataset—a rich repository for video prediction featuring realistic scenes and intricate interactions. \n**Please refer to   Figure 1 in the supporting document for visualizations and the \"1. Additional experiment on the realistic dataset - MOVi-C\" section in Global Rebuttal**.\n\n\n\nWe hope our explanations and the initial experimental results in the more realistic datasets will help address your concerns and help you with your assessment.\n\n---\n[1] Object-Centric Learning with Slot Attention - Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, Thomas Kipf \n\n[2] SAVI Conditional Object-Centric Learning from Video - Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, Klaus Greff \n\n[3] SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos - Gamaleldin F. Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C. Mozer, Thomas Kipf \n\n[4] A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning - Soochan Lee, Junsoo Ha, Dongsu Zhang, Gunhee Kim\n "}, {"review_id": "GekB2sYJ6e", "reviewer": "Reviewer_NsYp", "summary": "The paper proposed Reusable slot-wise mechanisms wherein the authors introduce \"Central Contextual Information\" -- a bottleneck that captures the global context and helps in choosing which slot to attend to.\nThrough experiments on several simple and relatively complex datasets are shown with reconstruction, VQA and planning tasks to show the benefit of the reusable mechanisms.\n", "questions": "See *Weaknesses*", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. I find the introduction a bottleneck CCI to summarize the context of past $T$ frames to be intuitive and novel.\n\n2. The method shows significant improvement in OOD action planning tasks which is impressive!\n", "weaknesses": "3. **CCI**: Have authors tried treating each slot independently as input to the transformer with positional encoding instead of concatenating past $T$ time slots? \n\n4. In Fig 1(c), is should the $t$ in blue circle (drawn adjacent to the CCI bubble) be $s_t$. If so please clarify in the figure as $t + \\delta s_t $ in Step 3 is confusing as one represents time and another represents the predicted change in slot representation.\n\n5. **Memory Constraints**: The CCI (Step 1) has a huge memory constraint of $(\\tau+1) \\times N \\times d_s$ which is $7 \\times 6 \\times 128 = 5376$ for OBJ3D dataset (which has the smallest memory requirements), and $16\\times 6 \\times 192 = 18432$ which is huge. This appears to be a huge factor given that object-centric models are already memory expensive when it comes to applications such as RL.\n\n6. **OOD in dynamics**: How does RSM behave when it sees an OOD dynamics for a particular object. For example, in the OBJ3D dataset (or any other simplistic dataset with enough variations in objects and their dynamics) if a green object is static throughout during the training, but during testing scenario if the green object moves would RSM reuse the mechanism corresponding to the motion of object? If time permits, I would like to see a simple experiment on thise on any of the datasets (which ever is faster to train/test). Please reach out to me if you need more clarification on this.\n\n7. **Inference Time**: Can the authors provide comparison between all the baselines and RSM with regards to inference time on the different datasets evaluated?\n\n\n**[Minor comments which I have not considered for rating of the paper]** -- The authors need not reply to these 2 comments below.\n\n8. I think reconstruction results shown using SSIM or MSE on most simplistic datasets such as OBJ3D or CLEVERER where the values are in the 90s aren't really indicative of how useful the representations are for downstream tasks. It appears that VQA task in CLEVERER is also pretty simplistic and has reached mid-90s -- it would be better the community in general moves to addressing complex tasks of planning, RL and hard-VQA (such as Physion VQA task as shown in the paper).\n\n9. **Readability of methodology section**: In section 2.1 (RSM Overview), it would be helpful to re-iterate what the purpose of having CCI is. The only mention of CCI is in the last section of introduction, so explicitly having it as a part of the RSM overview would be helpful.\n\n\n----\n**Rationale for rating**\nI find the contributions of this work to be novel and experimentation to be through and hence I lean towards accepting the paper. However, I would like the authors to address my questions before I finalize my decision. In addition to this, if authors have clarifying questions regarding comment (6), I would request them to reach out to me early on during the rebuttal phase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable reviews. We deeply appreciate the time and effort you invested in evaluating our work.\n\n\nWe are happy to see `intuitive and novel`, `significant improvement`, and `impressive`  in your reviews.\n \n  \n\n>3.  **CCI**: **Have authors tried treating each slot independently as input to the transformer with positional encoding instead of concatenating past  $T$  time slots?**\n\nWe have not tried doing so, and a major reason was the observation in one of our baselines, SlotFormer [1]. There they explicitly pass slots as tokens to a transformer and positional encoding is added temporally across the previous $T$ steps, yet even with positional encoding, they concatenate all slots across the span of $T$ frames as input to their transformer. Given their success with concatenation, we also opted for doing so, and did not use positional embedding alone.\n    \nFinally, we prioritized our compute budget during the rebuttal phase to address your concern wrt comment 6. However, we would certainly explore the effect of using positional encoding alone vs. concatenation before the final version. \n\n\n---\n    \n> 4.  **In Fig 1(c), is should the  $t$  in blue circle be  $s_t$.**  \n\nThank you for your question, and sorry for the confusion on $s_t$ notation in Figure 1c.\n\nWe had opted for a notation where any colored circle with a solid border represents a particular slot at the time that is given inside the circle (i.e., $t$ or $t+1$), but you are right, and it can be confusing when such slots appear alongside $\\Delta s_t$, we should update all colored circles with solid borders to contain symbols such as $s^i_t$, denoting slot number and its time.\n\nWe have updated Figure 1 in our latest version. Thank you.\n\n---\n\n> 5.  **Memory Constraints**: The CCI (Step 1) has a huge memory constraint of  ($\\tau$+1)×$N$×$d_s$  which is  7×6×128=5376  for OBJ3D dataset (which has the smallest memory requirements), and  16×6×192=18432  which is huge. \n    \nThank you for your helpful observation.  \n\nWe would like to mention that this memory constraint could, to some extent, be addressed by choosing $N$ more wisely. One way, when it comes to more realistic scenes, is to segment a scene comprising numerous objects into patches (with possible overlapping) where the objects inside the patch are unlikely to interact with those outside; this will keep the dependence on $N$ to a constant rate.\n\nLastly, we would appreciate it if the reviewer could help us realize in which settings our method could constitute a potential memory-related problem since even with batches as big as 512, the memory consumption of this block seems to be ~100MB at most, which doesn't cause a critical hurdle in our experiments. One possible future work to try is to build a bank of memory and use the memory retrieval method as in Goyal et al. ICLR 2022, in which, in each step, appropriate information from the large memory pool can be retrieved as a context. We would appreciate the reviewer's insight in case we've missed an angle.\n\n---\n\n> 6.  **OOD in dynamics**: How does RSM behave when it sees an OOD dynamics for a particular object.  \n    \nThank you for your insightful suggestion regarding the  OOD  evaluation strategy. We conducted additional experiments for OOD dynamics and presented our approach and findings in response to your question. **Please refer to  Figure 4 in the supporting document for visualizations and Section \"3. Additional evaluation in OOD dynamics\" in the Global Rebuttal for more details**.\n\n\nIn conclusion, our findings underscore RSM's potential to generalize and reuse objects' motion mechanisms across different colors. However,  there are still limitations in mechanism prediction in OOD scenarios, particularly when confronted with strong OOD cases, as blue balls appear only in the test set.\n\n\n\n\n\n---\n> 7.  **Inference Time**: Can the authors provide comparison between all the baselines and RSM with regards to inference time on the different datasets evaluated?\n\n\nWe conduct the analysis to compare the inference duration (in seconds) among methods on the same 1 Tesla V100-SXM2 GPU. The reported values are the average inference time per batch, where the average is taken over the batches in the test set. The batch size is the same within one dataset but could be different across the different datasets. \n\n\n| Method         | OBJ3D                 | CLEVRER               | Physion               | PHYRE                 |\n|----------------|-----------------------|-----------------------|-----------------------|-----------------------|\n| **SlotFormer** | 0.1655 ± 0.012       | 0.1889 ± 0.009       | 0.0349 ± 0.014       | 0.5322 ± 0.017       |\n| **SwitchFormer** | 0.2636 ± 0.007       | 0.2634 ± 0.011       | 0.1336 ± 0.009       | 0.6334 ± 0.012       |\n| **NPS**        | 0.1639 ± 0.020       | 0.1877 ± 0.014       | 0.0807 ± 0.005       |  **0.4733** ± 0.010    |\n| **RSM**        |  **0.1526** ± 0.014   |  **0.1833** ± 0.016  |  **0.0324** ± 0.012   | 0.4794 ± 0.020       |\n\nAccording to the table, we can see that RSM takes the shortest duration for inference for most of the datasets or is on par with the rest of the baselines in the rest of the datasets.\nHowever, it is worth mentioning that the part which accounts for most of the inference time is the sequential prediction of rollout steps (that all models encounter), not the prediction of slots' next step (which is different among models). \n\n\n---\n[1] SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models - Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, Animesh Garg\n"}, {"review_id": "G3xobDH50u", "reviewer": "Reviewer_8CU7", "summary": "This paper presents a study on future prediction using object-centric representations and disentangled and reusable mechanisms (RSM) that govern the interactions between objects. The authors' key contribution lies in the proposal of Central Contextual Information (CCI) representation, which captures the interactions between objects observed in the past. The CCI is encoded using multi-head attention layers with all object-centric representations from previous observations and a bottleneck projection layer. It aids the model in selecting a single mechanism to predict object transitions from the current state to the next state. Experimental results demonstrate that the introduced RSM outperforms several competing baselines in tasks such as future frame prediction, visual question-answering, and action planning in relatively simple environments.", "questions": "o) The usage of the non-differentiable Gumbel-max layer needs clarification. If it is indeed used, how are the gradients backpropagated to the layer $\\psi$? If not, is it a typo?\n\no) In Equ. (3), if only one mechanism is selected, why does the formulation still require a sum operation to combine all other transitions predicted by other mechanisms?\n\no) Have the authors experimented with using the Softmax operation in Equ. (2) instead of the hard selection provided by the Gumbel-softmax?\n\no) Can the authors demonstrate the scalability of the visualization of mechanism assignments/decompositions with more scenarios? For example, can they provide additional visualizations of mechanism decompositions with different initial states/transitions on different datasets such as OBJ3D or CLEVRER?\n\no) In Line 101, when mentioning the projection $\\phi$, is it referring to a linear layer or an MLP (multi-layer perceptron)?", "limitations": "-) As mentioned in the Weaknesses, the experiments conducted in this study are limited to relatively unrealistic environments with toy objects and simple interactions. As a result, it remains uncertain whether the proposed approach would maintain its effectiveness in a more realistic environment or when applied to real-world data.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "+) The motivation behind the proposed CCI is clearly explained, with the authors using multi-head attention layers to capture interactions between objects in past observations. The subsequent bottlenecking of the attention layers’ output into the CCI enables the model to learn a concise representation that encapsulates meaningful information about object interactions.\n\n+) The selection process for predicting state transitions is intriguing and surprisingly effective. The use of Gumbel-softmax (assuming it is Gumbel-softmax instead of Gumbel-max as mentioned in the manuscript) allows the model to decompose the potential mechanisms governing state changes.\n\n+) The visualizations in Figures 4 and 5 provide compelling evidence of the model's ability to decompose mechanisms into discrete and reusable components. Figure 4 demonstrates the model's capability in decomposing mechanisms, while Figure 5 showcases how the CCI encodes essential information for determining outcomes following object interactions. \n\n+) The experimental results support the efficacy of the proposed RSM, as it outperforms several competing baselines, including SlotFormer, SwitchFormer, and NPS, across various tasks such as future frame prediction, visual question-answering (which relies on future visual states), and action planning.", "weaknesses": "-) The assumption that there is only one mechanism responsible for the state change of an object may not always hold true. Factors such as the object's original momentum and collisions with other objects can influence its behavior. Therefore, the use of Gumbel-softmax (again, assuming it is Gumbel-softmax instead of Gumbel-max as mentioned in the manuscript) in Equation (2) might impose overly strong restrictions on modeling interaction mechanisms.\n\n-) There appears to be a misinterpretation regarding the mention of the Gumbel-max layer in Step 2. The Gumbel-max layer, without the Straight-Through (ST) Gumbel Trick, is not differentiable and is primarily used to enable the categorical sampling using unnormalized log probabilities by the reparameterization trick. The references cited at Line 122 also pertain to Gumbel-Softmax, not Gumbel-max. For more details, please refer to the PyTorch implementations of the Gumbel-Softmax layer: https://pytorch.org/docs/stable/generated/torch.nn.functional.gumbel_softmax.html#torch.nn.functional.gumbel_softmax or the TensorFlow API: https://www.tensorflow.org/probability/api_docs/python/tfp/distributions/RelaxedOneHotCategorical.\n\n-) There is a discrepancy at Line 269 where the authors mention the visualization of disentangled mechanisms in Figure 5. However, Figure 5 is not found in the main submitted manuscript but in the supplementary material (Page 13). The mention of Figure 5 at Line 269 implies it should be present in the main paper, potentially violating the Formatting instructions.\n\n-) While the proposed RSM outperforms competing baselines on various datasets and tasks, it is worth noting that the experiments are conducted in relatively unrealistic environments with toy objects and simple interactions. It would be beneficial to demonstrate the effectiveness of the proposed approach (object-centric representations + RSM) on more realistic datasets such as MOVi-B, C, D, E, and F.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable reviews. We deeply appreciate the time and effort you invested in evaluating our work.\n\nWe are happy to see `clearly explained motivation`, `intriguing and surprisingly effective`, and `impressive`  in your reviews.\n\n---\n\n> 1. **The usage of the non-differentiable Gumbel-max layer needs clarification.**\n> \n\nWe apologize for any confusion regarding using the Gumbel-Softmax in our paper. \nWe would like to clarify the matter as follows.\n\nFirst of all, we agree that we should have mentioned the \"Straight-Through Gumbel Trick\" in our paper. That is exactly what we are using to backpropagate gradients in training when sampling from a categorical distribution. We employ `F.gumbel_softmax()` with the `hard=True` option (where F refers to `torch.nn.functional`, coherent with your referenced implementation of Gumble-softmax). This enables hard selection from the categorical distribution during the forward pass, and backpropagates the gradients using the straight-through trick in the backward pass.\n\nOnce again, we are sorry for the confusion. In light of your feedback, we have made the necessary adjustments to our paper to explicitly include the \"Straight-Through Gumbel Trick\" terminology, clarifying our use of Gumbel-Softmax.\n\n\n\n\n\n---\n> 2. **In Equ. (3), if only one mechanism is selected, why does the formulation still require a sum operation to combine all other transitions predicted by other mechanisms?**\n\nThank you for your question. Equ. (3) just presents a notation convention. The summation has only one non-zero term, and we wrote the equation as such for completeness in case one decides to use a softmax instead of our hard selection. But again, the summation in Equ. (3) contains only one non-zero element. We're sorry if this has confused; we would change it for the final version to avoid any confusion.\n\n---\n> 3. **Have the authors experimented with using the Softmax operation in Equ. (2) instead of the hard selection provided by the Gumbel-softmax?**\n\nThank you for your question. \nOur work has intentionally refrained from conducting experiments with the Softmax operation (`F.softmax()`  or `F.gumbel_softmax(hard=False)`)  for a specific rationale. Our primary objective is to achieve the disentanglement of mechanisms, enabling them to independently and autonomously address distinct tasks. In line with this, our design philosophy emphasizes the independence of mechanisms, thereby avoiding significant information sharing among mechanisms. As such, the Softmax operation is not aligned with our adopted direction and causes information leakage among the slots.\n\nNevertheless, we acknowledge the merit of incorporating a comparative perspective. In this regard, we have compiled a comparison between the performance of RSM when utilizing the Gumbel-Softmax with hard selection and Softmax operations during the training phase. The following table offers an overview of this comparison:  \n\n| Method | OBJ3D (LPIPS$_{\\times 100 } \\downarrow$) |  PHYRE-fold0 (AUCCESS $\\uparrow$)|\n|--|--|--| \n| **RSM with Gumbel-softmax and hard selection** | **7.88** ± 0.1 |**82.14** ± 0.2| \n|**RSM with Softmax** | 9.65 ± 0.2 | 74.31 ± 0.5 | \n\nAccording to the table, we could see that RSM with Gumble-softmax and hard selection produces much better performance than one with Softmax, indicating the importance of learning independent mechanisms in RSM.\n\n---\n> 4. **Can the authors demonstrate the scalability of the visualization of mechanism assignments/decompositions with more scenarios?  such as OBJ3D or CLEVRER?**\n\nSure. **Please refer to Figures 2 and 3  in the supporting document for visualizations and Section \"2. Mechanisms assignment in OBJ3D and CLEVRER\" in the Global Rebuttal for the discussion**.\n\n ---\n> 5. **In Line 101, when mentioning the projection $\\phi$, is it referring to a linear layer or an MLP (multi-layer perceptron)?**\n\nIt refers to a linear layer mapping  $\\mathbb{R}^{d_s} \\rightarrow \\mathbb{R}^{d_{cci}}$.\n\n---\n\n> -) **The assumption that there is only one mechanism responsible for the state change of an object may not always hold true.**  \n\nThank you for your insightful comment. This is an important observation, and we are aware of the limitation and have tried ideas to explore how it affects performance. As shown earlier, the softmax selection of the mechanisms does not improve performance. Alternatively, we could employ multiple loops of the sequential slot updates,   allowing multiple mechanisms to alter a slot's state. However, we did not observe any surprising advantages compared to the original RSM, and the results stayed on par with them. However, this observation does not refute the possibility that more realistic scenes might benefit from such modifications or other ways that we haven't tried for having multiple mechanisms change a slot's content. Therefore, we remain keen on exploring the extent of this limitation in more realistic scenarios.   \n \nOn the other hand,  we observe that mechanisms' behavior can deal with complex scenarios, such as colliding with another object when moving or moving while spinning. More details for these cases are  in the answer to your question 4 above. \n \n\n ---\n> -) There is a discrepancy at Line 269 where the authors mention the visualization of disentangled mechanisms in Figure 5.  \n\nWe are so sorry for the confusion on Figure 5's position. \nFigure 5 should definitely be in the main text. We have moved it back to the main text in our latest version.\n\n---\n> -) **It would be beneficial to demonstrate the effectiveness of the proposed approach (object-centric representations + RSM) on more realistic datasets such as MOVi-B, C, D, E, and F.**\n\nThank you for your great suggestion. We have experimented on **MOVi-C** and got some initial results.\n**Please refer to   Figure 1 in the supporting document for visualizations and the \"1. Additional experiment on the realistic dataset - MOVi-C\" section in the Global Rebuttal for the discussion**.\n"}, {"review_id": "agXVoeF9yA", "reviewer": "Reviewer_TEgG", "summary": "This paper presents an architecture (Reusable Slotwise Mechanism or RSM)  for modelling the temporal dynamics of objects based on a slot-represent.\nThe main idea is to extract a global context (Central Contextual Information or CCI)  from the past frames and use this to stochastically select one of a small discrete set of (learned) mechanisms for each slot and time-step. The mechanism computes the update of a slot,  based on its current representation as well as the CCI. The slots are updated sequentially. It is shown anecdotally that they specialize to particular object behaviors/dynamics (such as idle, free-fall and collision) and are reused across slots and time. \nRSM is compared to several strong baseline methods on several tasks including dynamics modelling, question answering, and task planning. It demonstrates significant improvements over the baselines with a comparable amount of parameters and compute.\n ", "questions": "* My main request would be training-time ablations for $RSM_{!2}$ and $RSM_{!3}$ with quantitative results, since they would help buttress the claims about the importance of the CCI. \n* Similarly, the paper claims in various places that the CCI has advantages over pairwise interactions. Is there evidence from a fair comparison of the two? Communication is pairwise in NPS, but is that the only relevant difference? \n* How big is $d_{cci}$? The paper claims that it acts as a bottleneck, yet the CCI is recomputed for every slot from a self-attention among all previous slots. So unless $d_{cci}$ is at least smaller than $d_s$ I do not see in what sense it constitutes a bottleneck. \n* Are all mechanisms equally important? I suspect that main advantage of RSM might be the ability to put objects firmly in an idle mode. ", "limitations": "The paper discusses some limitations in the supplementary. I see no necessity for a discussion of negative societal impact. ", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The studied problem of modelling temporal dynamics from raw video is an important one, and there still is a lot of headroom.\n* The idea behind RSM is simple and well motivated. \n* RSM clearly improves upon the baselines and produces SOTA results on several relevant benchmarks. It thus presents an important step towards unsupervised learning of dynamics models from video.\n* The presentation is clear and the paper + supplementary contain all relevant information for reproducing the experiments.", "weaknesses": "To evaluate the different parts of the RSM architecture, the paper presents 4 ablations in section 4.5. Unfortunately the only quantitative results in Fig 5. the present fell out of the main paper into the appendix.\n\nBut if I understand correctly there is a more severe problem: it seems from the text that ablations RSM!2, RSM!3 and RSMk  are **inference time** ablations. That means during training the model had access to CCI in steps 2 and 3 and has learned to choose a mechanism. It is thus not at all surprising that changing any part of that during inference will deteriorate the performance. Thus, all that these ablations demonstrate is that the model has not actively learned to ignore the CCI and the mechanisms. \nIt would be much more informative to see how the model performs if these modifications are also applied during training. That could for example help answer the much more interesting question \"How much does having access to CCI during step 2 help the model choose the right mechanism\". For this very reason ablation 4 is much more informative. \nIn my opinion the such training-time ablations are necessary to properly support the claims about the role and importance of CCI and the mechanisms. \n\n\n\nOther minor weaknesses:\n* the differences between RSM and the baselines could be made clearer. There is some information in Sec 3.2 but it doesn't clarify what exactly is different. \n* The citations especially in the introduction and related work seem a bit biased with many citations to Goyal et al while leaving out other relevant and influential work in the area such as:\n  - Eslami, S. M., Heess, N. & Weber, T. Attend, infer, repeat: Fast scene understanding with generative models. Adv. Neural Inf. Process. Syst. (2016)\n  - Burgess, C. P. et al. MONet: Unsupervised Scene Decomposition and Representation. arXiv [cs.CV] (2019)\n  - Greff, K., Kaufman, R. L. & Kabra, R. Multi-object representation learning with iterative variational inference. International (2019)\n  - Battaglia, P. W. et al. Relational inductive biases, deep learning, and graph networks. arXiv [cs.LG] (2018)\n  - Greff, K., van Steenkiste, S. & Schmidhuber, J. On the Binding Problem in Artificial Neural Networks. arXiv [cs.NE] (2020)\n  - Schölkopf, B. et al. Toward Causal Representation Learning. Proc. IEEE 109, 612–634 (2021)\n\n* Sec 2.2 feels a bit repetetive with the enumeration of the three main components of RSM and the list of four steps in RSM mostly explaining the same things. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the time and effort you invested in evaluating our work.\n\nWe are glad to see `simple and well motivated`,  `clearly improves `,  and `presentation is clear `  in your reviews.\n\n---\n> **training-time ablations for $RSM_{!2}$  and  $RSM_{!3}$  with quantitative results.** \n\nWe   agree that the training-time ablations for $RSM_{!2}$  and  $RSM_{!3}$  are helpful.\n\nFirst, we would like to repeat the   purpose behind  the ablations $RSM_{!2}$ and $RSM_{!3}$. CCI is   in steps 2 and 3 in a concatenation, a   question arises: Is the CCI really influence     these steps, or is the greater weight being placed on the other component of the concatenation? Therefore,  we mask out the CCI in these steps to verify.\n\nNext, we present the results of our training-time ablations on OBJ3D across 5 random seeds. Ablations with a `'`  for  training-time, while those without   `'`  for   inference-time.  \n| Method |  LPIPS$_{\\times 100}\\downarrow$|\n|--|--| \n| **$RSM$** | **7.88**±0.1 | \n|**$RSM_{!2}$** | 15.32±0.4 |  \n|**$RSM_{!3}$** | 9.36±0.2 | \n|**$RSM_{!2}'$** | 17.3±0.9 | \n|**$RSM_{!3}'$** | 9.70±0.2 |  \n\nWe could see that:\n -  **$RSM$** achieves the best performance over the ablations. \n - **$RSM_{!2} '$** experiences challenges when using   one slot as input for mechanism selection, then tends to favor a single mechanism after a few epochs consistently. This stresses the vital role of the CCI  in   information diffusion among slots during mechanism selection.\n  - **$RSM_{!3}'$** outperforms $RSM_{!2}'$ but lags behind RSM by a large margin.  \n\t  * Using a single slot as mechanism input is not entirely detrimental if CCI aids in recognizing mechanism roles. Therefore, it does not produce a really bad performance.  \n\t  * Nonetheless,  Section 4.5 underscores that CCI also holds contextually   data guiding slot movements (e.g., left, right, or some angle). Therefore, the mechanism still needs the CCI  for a more accurate prediction.   \n\t  * Furthermore, the mechanism selection process generates a probability   for M mechanisms' indices, which must convey the entire system context due to its limited size. That's why the CCI   in the mechanism's input helps deliver system context.\n\nThese findings show the key role of CCI in   slots' communication.\n\n---\n>  **Communication is pairwise in NPS, but is that the only relevant difference?**\n\nYes, the pairwise communication in NPS is the only relevant difference. The rest is conceptually equivalent for RSM and NPS, such as the training pipeline, object-centric models, and configurations.\n\n---\n> **How big is  $d_{cci}$?** The paper claims it acts as a bottleneck, yet the CCI is recomputed for every slot from a self-attention among all previous slots. \n\nThank you for your question. We choose $d_{cci}=d_s$ in this work. \n\nWe call CCI a bottleneck not merely because of its dimension; rather, CCI is summarizing the information from all slots, i.e., $N\\times d_s$, in a vector of size $d_{cci}=d_s$. Moreover, across rollout steps, the system communicates and joins to update the CCI. That is what we mean by a bottleneck. It is indeed recomputed from a self-attention among all previous slots, but it constantly summarizes the whole state of all earlier slots as they change in a fixed-size vector, squeezing the information.\n\nWe hope this helps clarify what we denote as a bottleneck.\n\n---\n> **Are all mechanisms equally important?** \n\nThank you for  noticing this vital point. \n\nAll mechanisms have an equal chance of getting selected at the beginning, as there are no pre-defined roles. They share the same architecture and are  initialized randomly. \n\nThrough training, it gains the ability to decompose the transitions into separate mechanisms with varying importance. Certain environments may require specific actions, we expect RSM to **autonomously  recognize** these characteristics and adjust the frequency of mechanisms accordingly.\n\nThis table is the ratio of mechanisms selected in test time over 3 datasets with a single seed.\n\n| Mechanisms  | OBJ3D | CLEVRER|  PHYRE|\n|--|--|--|-- |\n| **1** | 0.435 |  0.037| 0.163\n|**2** | 0.107 | 0.370  |  0.057\n|**3** | 0.045 | 0.076 |   0.144\n|**4** | 0.096 | 0.129 |   0.461\n| **5** | 0.205 | 0.212  |  0.173\n| **6** | 0.044   |   0.126|        -| \n| **7** | 0.064 | 0.047 |   -\n\nThe 7 mechanisms in OBJ3D can be identified as: Idle (1), Moving fast (2), Moving and spinning (3), Collision (4), Redundant   (5,6), and Moving slow (7).\n\nThe 7 mechanisms in CLEVRER can be identified as: Redundant   (1,6), Idle (2), Collision (3), Moving slow (4), Moving fast (5), Moving and spinning (7).\n\nThe 5 mechanisms in PHYRE can be identified as: Redundant    (1), Collision (2), Moving left or right (3), Idle (4), and Falling (5).\n\nPlease refer to **Section \"2. Mechanisms assignment in OBJ3D and CLEVRER\" in the Global Rebuttal** for discussing the **Redundant Mechanisms**.\n\n> **I suspect that main advantage of RSM might be the ability to put objects firmly in an idle mode.**\n\nYes, we think recognizing the idle status of objects is an important ability. However, 2 points are also necessary for a successful prediction: (1) the accurate assignment of all other mechanisms  and (2) the accurate prediction of the `Idle` mechanism for a smooth communication of the slots.\n\n\n---\n> the only quantitative results in Fig 5. the present fell out of the main paper into the appendix. \n\nWe are so sorry for the confusion on Figure 5's position.  We have moved it back to the main text in our latest version.\n\n> The differences between RSM and the baselines could be made clearer.  \n> \n> The citations  seem a bit biased with many citations to Goyal et al \n> \n> Sec 2.2 feels a bit repetetive with the enumeration of the three main components of RSM and the list of four steps.  \n    \nWe truly appreciate your suggestions. We will certainly fix these points in the final version and make  adjustments for a fairer presentation of the prior work. \n\n"}]

Author rebuttal:
We would like to say thank all reviewers for their time and efforts to review our work.

The following sections are explanations related to the one-page PDF visualization.

The outline is as follows:
1. **Additional experiment on the realistic dataset - MOVi-C** as suggested by reviewers DY7c and 8CU7.
2. **Mechanisms assignment in OBJ3D and CLEVRER** as suggested reviewer 8CU7.
3. **Additional evaluation in OOD dynamics** as suggested by reviewer TEgG.



---
### 1. Additional experiment on the realistic dataset - MOVi-C

**Please refer to   Figure 1 in the PDF for visualizations**

The following is a quick summary of our configurations on MOVi-C:
- Dataset: Training was conducted on approximately 9000 videos, with testing performed on 1000 videos.
- Object-centric Training: We employed a StoSAVi (a stochastic version of SAVi [1], introduced in SlotFormer [2]) model equipped with 7 slots, trained over 50 epochs.
- RSM Training: Our RSM model underwent 20 epochs of training, utilizing 10 burn-in frames as input and predicting 14 rollout frames.




According to the initial experiment result in **Figure 1**,  RSM exhibited the ability to manage complex scenes and interactions, as showcased in the MOVi-C dataset. An example is in RSM's accurate prediction of the dynamics, such as the motion of the white can to the right and the downward trajectory of the lion toy.

Due to the time limitations for the rebuttal period, we cannot work on a larger dataset or fine-tune both object-centric and RSM models carefully, thus causing low-quality visuals such as losing small objects and object details. Nonetheless, it is worth noticing the potential of RSM in modeling dynamics in complex scenes from these initial results. These initial results also motivate us to further work on this family of realistic datasets.     

[1] SAVI Conditional Object-Centric Learning from Video - Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, Klaus Greff 
[2] SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models - Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, Animesh Garg


---
### 2. Mechanisms assignment in OBJ3D and CLEVRER
**Please refer to Figures 2 and 3 in the PDF for visualizations**

According to the mechanism assignments in the analysis, we could see:
- It is interesting to see that the environments share similar motions and interactions as OBJ3D and CLEVRER share a similar set of mechanisms, such as `Moving and spinning`, `Moving fast`, and `Moving slow`.
- With some complex motions like `Moving and spinning`, a separated mechanism is appointed (by the model during training).

Below, we provide the ratio of selected mechanisms over the test set for reference:
| Inferred Mechanism | OBJ3D | CLEVRER|  
|--|--|--|
|**Idle** | 0.435 | 0.370
|**Moving Fast** | 0.107 | 0.212
|**Moving Slow** | 0.064 |  0.129
|**Collide** |0.096 | 0.076
|**Moving and Spinning** | 0.045|  0.047
|**Redundant Mechanisms** |0.249 | 0.163

The Redundant Mechanisms (2 mechanisms in particular) we believe are artifacts of having more mechanisms and slots employed than there are needed for the environment. We observe that these mechanisms usually attend to slots that contain no object, and in fact, removing such mechanisms only affects the performance very slightly, further confirming their redundancy, and this artifact could be expected because we do not tune the number of slots and mechanisms per each environment. The rest of the mechanisms demonstrate meaningful transitions, as can be seen in the new visualizations.


---
### 3. Additional evaluation in OOD dynamics
**Please refer to   Figure 4 in the PDF for visualizations** 


We introduce modifications to the PHYRE dataset, specifically altering the characteristics of blue objects between the train and test sets as follows:
- **Modified train set**: Blue objects in the train set are **static** since they represent blue floors or walls.
- **Modified test set**: Blue objects in the test set are allowed to **move**, simulating scenarios involving blue balls or glasses.

Particularly,  our approach involves extracting templates containing blue balls and glasses from the training set and transferring them to the modified test set. Notice that the modified train set includes the ball objects in red, green, white, and gray colors, as well as the glass objects in gray color. In PHYRE, it's impossible to have, for instance, a red object that is NOT a ball to be static in the dataset. 



In response to the question:" `Would RSM reuse the mechanism corresponding to the motion of the object?`", we offer the following insights:

- **Yes**. RSM exhibits the capacity to reuse mechanisms based on object motion appropriately. For instance, the blue ball is assigned the "**Fall**" mechanism from steps 0 to 2, the "**Collide**" mechanism from steps 2 to 3, and the "**horizontal moving**" mechanism (leftward) in the subsequent steps. These mechanism assignments effectively capture the object's dynamics compared to the target frames. 
- RSM demonstrates the ability to generalize object motion across different colors and the ability to reuse mechanisms in OOD cases by assigning appropriate mechanisms to the blue ball, corresponding to its motions over steps (falling, colliding, and moving left).
- However, since the blue balls appear only in the test set could be considered a strong OOD case (the model has not had a chance even to reconstruct a blue ball-shaped object); the predicted next frames exhibit some limitations in comparison to predictions trained on the standard training set, such as distortions in the shape of the blue ball and incorrect position of the red ball from step 4 after the collision of the blue and red balls.


Meta-review: This paper received mixed reviews in the initial round of review. However, the authors have done a very good job addressing most of the comments and points of feedback. Given this effective rebuttal, there was a consensus by all reviewers to accept this paper. I have read the reviews and the rebuttal, and given these I recommend accepting the paper.
Decision: Accept (poster)