PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
Authors: Angel Villar-Corrales
Sven Behnke
Venue: ICML 2025 poster (2025)
Keywords: object-centric video prediction
object-centric learning
inverse dynamics learning
action planning
Primary area: general_machine_learning->representation_learning
TL;DR: We propose PlaySlot-- an object-centric video prediction model that infers object representations and latent actions from unlabeled video sequences and uses them to forecast future video frames.

Abstract: Predicting future scene representations is a crucial task for enabling robots to understand and
interact with the environment. However, most existing methods rely on videos and simulations
with precise action annotations, limiting their ability to leverage the large amount of avail-
able unlabeled video data. To address this challenge, we propose PlaySlot, an object-centric
video prediction model that infers object representations and latent actions from unlabeled
video sequences. It then uses these representations to forecast future object states and video
frames. PlaySlot allows the generation of multiple possible futures conditioned on latent actions,
which can be inferred from video dynamics, provided by a user, or generated by a learned action
policy, thus enabling versatile and interpretable world modeling. Our results show that PlaySlot
outperforms both stochastic and object-centric baselines for video prediction across different environments. Furthermore, we show that our inferred latent actions can be used to learn robot behaviors sample-efficiently from unlabeled video
demonstrations. Videos and code are available on our project website.

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centiee Video Prediction and Planning
Angel Villar-Corrales'
Sven Behnke
Abstract
Ref. Video
InvDyn
Predicting future scene representations is eru- cial task for enabling robots to understand and interact with the environment. However, existing methods rely videos and simulations with precise action annotations, limiting their ability to leverage the large amount avail- able unlabeled video data. To address this chal- lenge, we propose PlaySlot, an object-centric video prediction model that infers object rep- resentations and latent actions from unlabeled video sequences. It then uses these representa- tions to forecast future object states and video frames. PlaySlot allows the generation of multi- ple possible futures conditioned on latent actions, which can be inferred from video dynamics, pro- vided by user, generated by a learned action policy, thus enabling versatile and interpretable world modeling. Our results show that PlaySlot outperforms both stochastic and object-centric baselines for video prediction across different en- vironments. Furthermore, we show that our in- ferred latent actions can be used to learn robot be- haviors sample-effiiinnly from unlabeled video demonstrations. Videos and code are available our project website.
2
Scene Parsing
cOCVP
Z
cOCVP
Z
COCVP
Learned Objects
Figure PlaySlot parses an image X₁ into its object com- ponents S1. It then predicts multiple future object states and frames with an object-centric video prediction module (cOCVP) conditioned on latent actions Z, which can be inferred from a reference video with our InvDyn module, provided as input, or generated by a learned action policy.
Humans naturally achieve such understanding by parsing their environment into background and multiple sepa- rate objects, which interact with each other and can be recombined to form more complex entities (Johnson, 2018; Kahneman et al., 1992). Neural networks equipped with such compositional inductive biases have the ability to learn structured object-centric representations with desir- able properties such as robustness (Bengio et al., 2013; Dit- tadi al., 2022), generalization novel compositions (Gr- eff al., 2020), transferability to novel tasks (Zhang et al., 2022), and sample efficiency (Mosbach al., 2025), among others.
1. Introduction
Accurate and flexible world models are crucial for au- tonomous systems reason about their surroundings, pre- dict possible future outcomes, and plan their actions effec- tively. Such models require structured representation of the world that supports generalization, robustness, and con- trollability, especially in complex and dynamic scenarios.
Building on these foundations, the field of object-centric learning has made great advances recent years, progress- ing from learning object representations in simple syn- thetic images (Locatello al., 2020; Burgess et al., 2019) and videos (Kipf et al., 2022: Elsayed et al., 2022), to- wards more complex real-world scenes (Seitzer et al., 2023; Zadaianchuk et al., 2024). Recently, the field of object- centrie video prediction combines these learned object rep- resentations with forward-dyname models and has shown great promise for multiple downstream applications such
Autonomous Intelligent Systems, Computer Science Insti- tute VI Intelligent Systems and Robotics, Center for Robotics and the Lamarr Institute for Machine Learning and Artificial In- telligence, University of Bonn, Germany Correspondence Angel Villar-Corrale Corrales <villar@@ uis.uni- iissuui-bonni bonn.de>.
Proceedings of the 4 International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Vidco Prediction and Planning
as modeling object dynamics CVillar-Corrales et al., 2023;
Locatello al., 2019). Early approaches aimed learn object representations from synthetic images (Locatello et al., 2020; Singh et al., 2021; Biza et al., 2023) or videos (Kipf et al., 2022; Creswell et al., 2021; Singh et al., 2022) by minimizing a reconstruction objective. To learn meaningful representations from real data, recent slot-based methods leverage weak supervision (Elsayed et al., 2022; Bao et al., 2023), large pretrained transform- ers (Seitzer et al., 2023; Aydemir et al., 2023; Zadaianchuk et al., 2024), or diffusion models (Jiang et al., 2023; Wu et al., 2023b) These object-centric representations benefit multiple downstream tasks such as reinforcement learning for robotic manipulation (Mosbach et al., 2025; Ferraro et al., 2023) or visual-question-answees (Mamaghan et al., 2025).
Wu al., 2023a) action planning (Yoon et al., 2023; Mosbach et al., 2025). However, such models are currently limited to deterministic enviromments or rely on videos and simulations with precise action labels to forecast scene dy- namics, limiting their ability to leverage unlabeled video data and serve as world models for robotic applications.
In this work, we propose PlaySlot, a novel method for con- trollable video prediction using object-centric representa- tions. PlaySlot learns in a self-supervised manner from video to infer object representations, called slots, and la- tent action embeddings, which are computed using our proposed InvDyn module to capture the scene dynamics. PlaySlot then predicts future video frames conditioned on the inferred object slots and latent actions. At inference time, illustrated on Fig. 1, PlaySlot parses the observed environment into set of object slots, each of them repre- senting a different object in the image. Then, PlaySlot fore- casts future object states and frames conditioned on past object slots and latent actions, which can be inferred from a video sequence using our proposed InvDyn module, pro- vided by a human, or generated by learned action policy.
Object-Centric Video Prediction Object-centric video prediction aims to model the object dynamics and inter-
actions in a video sequence with the goal of forecasting future object states and video frames. Several methods ad- dress this task using different architectural priors, including RNNs (Zoran et al., 2021; Assouel et al., 2022), transform- ers ( (Villar- Corrales et al., 2023; Wu et al., 2023a; Daniel & Tamar, 2024; Meo et al., 2024) or state-space models (Jiang et al., 2024), attaining a remarkable prediction accuracy on synthetic datasets. Recently, some methods improve the controllability of object-centric video prediction models by conditioning the prediction process on actions (Mosbach et al., 2025) or language captions (Wang et al., 2024; Villar- Corrales et al., 2025; Jeong et al., 2025). However, fore- casting future object states without supervision in complex environments still remains an open challenge.
In our experiments, we demonstrate that PlaySlot learns rich and semantically meaningful action space, enabling accurate video prediction while providing high levels of controllability and interpretability. We show how PlaySlot effectively captures precise robot actions and seamlessly scales to scenes with multiple moving objects or to real- world robotics data, outperforming several controllable video prediction baselines. Moreover, we show that the latent actions inferred by PlaySlot enable sample-efficient learning of robot behaviors from unlabeled demonstrations.
In summary, our contributions are as follows:
Learning Latent Actions from Unlabeled Videos Videos provide abundant information about dynamics and activities, but often lack the action labels necessary for learning behaviors from video. To address this challenge, some methods train a latent policy directly from observa- tions by learning discrete latent action space and sampling the actions that minimize reconstruction error (Edwards et al., 2019; Struckmeier & Kyrki, 2023). Another group of methods, to which PlaySlot belongs, learns inverse dynam- ics from unlabeled videos by predicting latent actions given pairs of observations, and uses them for learning behav- iors for video games and robot simulations (Ye et al., 2022; Brandfonbrener et al., 2024; Schmidt & Jiang, 2024), as pretraining for Vision-Language-Accte Action models (Ye al., 2025) or for learning robot policies (Cui al., 2024; Tian et al., 2025).
We propose Play an object-centric video predic- tion model that infers object representations and latent actions from unlabeled videos, and uses them to fore- cast future object states and video frames.
PlaySlot outperforms several video prediction models across diverse robotic environments, while showing superior interpretability and control capabilities.
The object representations and latent actions inferred by PlaySlot can be used to learn robot behaviors from unlabeled video demonstrations sample efficiently.
2. Related Work
Unsupervised Slot- Object-Centric Learning Slot-based object-centric representation methods aim to parse in an unsupervised manner an image or video into a set of Ng latent vectors called slots, where each of them binds to different object in the scene (Greff et al., 2020;
Latent action models have also been used for conditional video prediction. The most similar method to ours is CADDY (Menapace et al., 2021: 2022), which learns la- tent actions from a collection of unlabeled videos from
2

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
C-C mm
Ssoo DIJDU-DA
Lvo
Object Render
Object Render
Object
Object Render
Object Render
Object Reader
Render
$
S
Sri
Conditional Object- Centric Predictor
Conditional Object- Centric Predictor
2
2
ZT
27
Dyv
$
LnvDyn
ST-1
InyDyn
S
S7
2
2,
S
2
Sy Sy-1
STi
S
Scene Parsinn
Scene Parsing
Scene Parsing
Scene Parsing
Scene Parsing
InvDyn
Policy
10--1 TH
1.
X1
X
X
X
X
Inverse Dynamics
User Input
Learned Policy
a) PlaySlot training
b) Play inference
Figure 2: Overview of PlaySlot training and inference processes. (a) PlaySlot is trained given unlabeled video sequences by inferring object representations S, and latent actions 2,, and using these representations to autoregressively forecast future video frames and object states. (b) PlaySlot autoregressively forecasts future frames conditioned on a single frame X, and latent actions which can be inferred from observations, provided by a user, or output by a learned action policy.
a single domain and uses the latent actions as condition- ing signal for predicting future frames. At inference time, CADDY maps user inputs to the latent space for playable video generation. Building upon this same principle, Ge- nie (Bruce et al., 2024) proposes foundation world model for playable video generation on diverse environments. However, both CADDY and Genie operate on holistic scene representations, which are limited for tasks that re- quire relational reasoning, often struggle model object relationships and interactions, and require human supervi- sion to generalize to scenes with multiple moving agents.
InvDyn. Finally, the object rendering module decodes the object slots render object images and masks, which can be combined via weighted
render video frames.
At inference time, shown in Fig. 2b), PlaySlot autore- gressively predicts multiple possible sequence continua- tions conditioned on the initial object slots and latent action embeddings, which can be estimated by InvDyn, provided by user, or generated by learned action policy.
3.1. Object-Centric Representation Learning
PlaySlot employs SAVi (Kipf et al., 2022), a recursive encoder-decoder model with a structured bottleneck of Nss permutation-equivarii
3. PlaySlot
object slots, parse a sequence of video frames X into their object components S S, I- Ns Dg The slots So are sampled from learned distri- bution and recursively refined to bind to the objects in the video frames. At time step 4, SAVi encodes the correspond- ing input X, into feature maps h, E TR 1. L.,,,, which are fed to Slot Attention (Locatello et al., 2020) to iteratively refine the previous slot representations conditioned on the current features. Slot Attention performs cross-attention between the image features and slots with the attention weights nor- malized over the slot dimension, encouraging competition between slots so as to represent each feature location:
We propose PlaySlot, a novel framework for control- lable object-centric video prediction from unlabeled video sequences. Fig. 2a) illustrates the training process in PlaySlot, well its main four components. Namely, given T' video frames X1.2, our model employs as Scene Parsing module (Sec. 3.1) that decomposes these im- ages into object representations, called slots, Sir = (y), where S, (s! (s!....,s:')) E x is the set Ds-dimensional object slots parsed from frame X,, For each consecutive pair of frames, PlaySlot employs an Inverse Dynamics (InvDyn) module (Sec. 3.2) in order estimate latent action embeddings 2, that encode the ac- tions taken by the agents in the scene between every con- secutive pair of frames. The Conditional Object-Centric Predictor (cOCVP) (Sec. 3.3) forecasts future object states conditioned on past slots and latent actions estimated by
softmax (g(S (((S--1)k(hh)77 k(h,)" Ni
R NEXT.
VDs
where 9 and are linear projections. The slots are then independently updated via shared Gated Recurrent
3

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
Unit (Cho al., 2014) (GRU) followed by a residual MLP:
3.2.2. INVDYN MODULE
And
We propose two variants of our inverse dynamics module for inferring latent actions from object-centric representa- tions. InvDyns processes object slots S, alongside learn- able token [ACT] using a transformer encoder fz, pro- ducing a single latent action 2, that represents the agent's behavior. This formulation is particularly well-suited for single-agent environments. In contrast, InvDynM processes each slot with a shared MLP, producing Ng latent action embeddings 2. = - 2,N3,, ). where each vector rep- resents the action of specific object in the scene. Be- low we explain the process for computing latent actions using InvDyns, which follows similar procedure to that of InvDynM.
S. GRU(A e Sh)), An,
~1= An.i
where U is a linear projection. The steps described in Equa- tions (1) and (2) can be repeated multiple times with shared weights to iteratively refine the slots and obtain an accurate object-centric representation of the scene.
To map the object representations back images, SAVi in- dependently decodes each slot in S, with Spatial Broad- cast Decoder (Watters et al., 2019) (DsAvi to render an ob- ject image and mask, which can be normalized and com- bined via a weighted sum to render the reconstructed frame:
o m = (ssva
(3) Following Menapace et al. (2021), we adopt a probabilistic formulation where InvDyn predicts the posterior distribu- tion of scene dynamics, modeled Gaussian:
Ng. 0 m with m softmax(m"))
X,
Ng
Pdr: Fa 2 f2(S,, [ACT]).
(6)
The distribution of latent actions â, is then defined as the difference between the distributions of scene dynamies from two consecutive time steps:
3.2. Learning Inverse Dynamics
In general, future frames depend not only on previous ob- servations, but also on other variables, such robot ac- tions. We propose inverse dynamies module (InvDyn) that estimates, given the object slots from two consecutive time steps, latent action embeddings à E R R that encode the actions taken by the agents between such time steps:
P2 Md++1 Md:: ozi
2. ~ N G2T) with
(7)
d + a
2
from which latent actions 2, are sampled.
To prevent the model from trivially encoding future states into 2, we regularize the latent action space by enforcing information bottleneck. Specifically, we constrain the latent action space to be low dimensional, i.c., D2 << Ds. Furthermore, as discussed in Sec. 3.2.1, we parameterize the latent actions as the sum of a discrete action prototype Pj and continuous action variability embedding v., where p. is obtained by vector-quantiznng the latent actions 2, (p. = VQ(2.)). We empirically verify that the information bottleneck enforced by vector quantization achieves com- parable performance to the one proposed by (Menapace et al., 2021), while requiring significantly fewer hyper- parameters.
2. = S...).
(5)
3.2.1. ACTION PARAMETERIZATION
The parameterization of the latent actions determines the complexity of transitions that can be modeled, well as the degree of control that we have over the predictions. On the one hand, learning a finite set of latent actions enables controllable video prediction while limiting the complexity of the dynamics that such actions can explain. On the other hand, continuous latent vectors offer greater flexibility to model complex transitions between frames, but at the cost of reduced interpretability and control.
Our hybrid latent action parameterization ensures that our InvDyn module encodes only the essential dynamics, ef- fectively capturing the agent's interaction with the scene while learning semantically meaningful action prototypes. Moreover, this hybrid factorization improves the control- lability and interpretability of the prediction process while maintaining the ability to model complex scene dynamics.
As a compromise between these two approaches, inspired by Menapace et al. (2021), we propose a hybrid parame- terization of the latent action 2n as the sum of a discrete P, and a continuous V, component: 2, = Pt + v.. The discrete component P.. denoted as action prototype, deter- mines the high-level action taking place (c.g. move left, go up), whereas the continuous action variability Yi captures stochastic and fine-grained variations, allowing the model to interpolate between prototypes and account for environ- mental uncertainty. This hybrid design enables the model to learn expressive dynamics while maintaining control and interpretability.
3.3. Conditional Object- Centric Prediction
We employ a transformer-bate based (Vaswani et al., 2017) module to autoregressively predict future object slots con- ditioned on past object states and latent actions.
4

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
Our proposed predictor, cOCVP, is transformer encoder with Pred layers. At each time step i, cOCVP takes in- put all previous slots Si.,, action prototypes Pı:r and vari- ability embeddings V1:t, all of which are first linearly pro- jected into a shared token dimensionality. The slots are then conditioned by adding them with the corresponding projected action prototype and variability embeddings. Ad- ditionally, we incorporate sinusoidal positional encodings such that all slots from the same time step receive the same encoding, thus preserving the inherent permutation equiv- ariance of the objects.
actions in the simulator for evaluation purposes, and the core training process of PlaySlot is fully unsupervised, op- erating entirely on unlabeled, action-free videos.
Our approach shares similarities with Schmidt & Jiang (2024), who learn policies from video using discrete latent actions in simple game environments. In contrast, PlaySlot leverages a more expressive action representation and conditional object-centric decoder, enabling the learning of more complex robot behaviors in realistic environments.
3.5. Training
cOCVP forecasts the future slots S by jointly modeling the object dynamics and interactions from the past object slots conditioned on the inferred latent actions. This pro- cess is summarized as:
We differentiate three different training stages in PlaySlot. We first train SAVi parse video frames into object-centric representations by minimizing reconstruction loss:
S.+1 :S.C + fp(P..) + (v(v..)),
(8)
CSAV Σ (Sssn(EsAM(X))) x,112.
where fs. fo and fv are learned linear layers.
The prediction process can be initiated from the slots of single reference frame S, and the corresponding inferred latent actions 21. This process is repeated autoregressively, with the predicted slots being appended to the input at each subsequent time step. allowing the generation of future ob- ject representations for desired number of time steps T.
where εSAwi and DSAVi correspond to the scene parsing and object rendering modules, respectively.
Second, given the pretrained SAVi model, jointly train InvDyn and cOCVP by minimizing a combined loss:
3.4. Learning Behaviors from Unlabeled Videos
PPhySlot
Ling + AstorCSton + AveLve,
(10)
Using a pretrained InvDyn module, we aim to learn a latent policy from unlabeled video expert demonstra- tions-without the need for action or reward information. For this purpose, we first use InvDyn to annotate dataset of unlabeled expert demonstrations with latent actions that capture the underlying scene dynamics. Subsequently, we train latent policy model fa via behavior cloning. regress- ing the inferred latent actions from the object slots observed at the corresponding time step.
Lime = Xx - Xel13,
(11)
Lst SS, (X:)
(12)
LvQ = ssg [2x] - P.ll + 0.25 ||2 sglp,,ll. (13)
where sg the stop-gradient operator. Limg measures the future frame prediction error, Lsim aligns the predicted ob- ject slots with the actual object-centric representations, and Lvq encourages the learning of meaningful action proto- types while regularizing the latent actions to align with their prototypes (Van Den Oord & Vinyals, 2017). We do not employ teacher forcing, enabling the predictor model to learn handle its own imperfect predictions.
At inference time, starting from a single observation X PlaySlot computes the corresponding object slots S and uses the learned policy predict a latent action 21, which is decomposed into an action prototype P1 vQ(&1 and variability embedding
2 P1. These representa- tions are fed to cOCVP to forecast the subsequent set of object slots $2. This process repeated autoregressively. allowing the learned behavior to unfold within the model's 's latent imagination.
Finally, the policy model fa and action decoder D, trained to regress the inferred latent actions à and ground truth actions a, respectively:
To enable the execution of latent actions in a simulator, we introduce action decoder D, that maps the predictions of the latent policy f= to weal-world actions. This module, implemented as a three-layer MLP, is trained on small set of eetion-labeled data to translate latent actions inferred by InvDyn into the corresponding executable actions. Impor- tantly, the action decoder only needed to execute latent
(S.) 2elb
(14)
T LD. =
Σ t=1
[(D, - arll
(15)
5

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
Table 1: Quantitative evaluation of several object-centric (OC) and controllable (Cont.) video prediction models. Given six seed frames, all models predict the subsequent 15 frames. PlaySlot achieves the best results in datasets that require modeling object interactions (BlockPush) or feature multiple moving objects (GridShapes), while maintaining competitive performance on ButtonPress and Sketchy. Best two results are highlighted in boldface and underlined, respectively.
BlockPush
ButtonPress
GridShapeszo0i
Sketchy
Model
OC Cont.
PSNR
SSIM
LPIPS|
PSNRT
SSIM LPIPS|
PSNRT
SSIM LPIPS|
PSNR
SSIM
LPIPS
SVG
X
20.96
0.898
0.096
32.23
0.950
0.011
38.10
0.988
0.002
24.24
0.813
0.076
CADDY
X
/
21.18
0.901
0.090
24.58
0.853
0.028
39.16
0.986
0.006
23.65
0.809
0.059
SlotFormer
V
X
17.22
0.729
0.134
19.52
0.762
0.111
19.74
0.795
0.149
21.17
0.751
0.091
OCVP
X
17.26
0.751
0.134
19.55
0.762
0.115
18.86
0.791
0.154
21.23
0.751
0.092
PlaySlot
21.41
0.890
0.066
26.03
0.878
0.025
54.09
0.996
0.001
24.43
0.815
0.063
4. Experiments
4.2. Video Prediction
4.1. Experimental Setup
We compare PlaySlot for video prediction with the object- centric baselines SlotFormer (Wu al., 2023a) and OCVP- Seq (Villar-Corrable et al., 2023), the stochastic video prediction model SVG-LP (Denton & Fergus, 2018) and the playable video generation model CADDY (Menapace et al., 2021). For fair comparison, all models are trained with six seed frames to predict the subsequent eight, and evaluated for 15 predictions. For CADDY, PlaySlot and SVG, we predict future frames conditioned on latent ac- tions vectors inferred from the ground truth sequence. Additionally, on BlockPush and ButtonPress datasets all models are trained using sequences with random explo- ration policies, and evaluated on expert demonstrations.
1. DATASETS
We evaluate our method on four environments with distinct characteristics. Further details are provided in Appendix C.
ButtonPress This environment, based on WerlWorld (Yu et al., 2020), features Sawyer robot arm that must press red button. This environment depicts a non-object centric task involving complex shapes and textures.
This environment, inspired by Li et al. (2020), features a robot arm and a table with multiple uni-colored cubes. The robot must push the block of distinct color into the location specified by red target. This task evaluates relational reasoning and modeling of object collisions.
BlockPush:
We evaluate the quality of the predicted frames using PSNR, SSIM (Wang et al., 2004) and LPIPS (Zhang et al., 2018). A quantitative comparison of the methods is pre- sented in Tab. 1. As expected, deterministic object-centric models (i.e. SlotFormer and OCVP) perform poorly, as they cannot infer the agent's actions and simply average over multiple possible futures. Our proposed method out- performs all other models on both the BlockPush and Grid- Shapes datasets, demonstrating PlaySlot's 's superior ability to forecast future video frames in environments involving multiple object interactions and moving agents, respec- tively. On Sketchy, which features real robotics videos, we observe that PlaySlot, CADDY and SVG achieve sim- ilar performance, with PlaySlot slightly outperforming the baselines. While the margin of improvement is mod- est, these results suggest that object-centric models like PlaySlot are well-suited for controllable video prediction and can serve world models in real robotic scenarios.
GridShapes: This dataset features two simple 2D shapes moving in grid-like patterns on a colored background. The shapes randomly change direction, introducing stochastic- ity to their motion. This dataset benchmarks a model's abil- ity to jointly predict the motion of multiple moving agents.
Sketchy: Sketchy (Cabi al., 2020) is real-world robotics dataset in which robotic gripper interacts with di- verse objects of different shape and color. This dataset eval- uates the model performance on real-world robotics data.
4.1.2. IMPLEMENTATION DETAILS
Our model is implemented in PyTorch (Paszke et al., 2017) and trained on single NVIDIA A100 GPU. PlaySlot uses SAVi (Kipf et al., 2022) with 128-dimensional object slots, as well a convolutional encoder and spatial broadcast decoder as scene parsing and rendering modules, respec- tively. The conditional predictor and inverse dynamics modules are transformer encoders with four layers and a token dimension of 256. For ButtonPress, BlockPush and Sketchy we use the InvDyns variant with eight distinct 16- dimensional action prototypes, whereas for GridShapes we use InvDyn with five 8-dimensional prototypes. Further implementation details are provided in Appendix B.
Fig. 3 depicts qualitative comparison of the best perform- ing methods on the ButtonPress and BlockPush datasets, respectively. On the ButtonPress dataset, as shown in Fig. 3a), all methods accurately model the motion of the robot arm. However, on the more complex BlockPush task, depicted in Fig. 3b), SVG and CADDY fail to model the object collisions, leading to blurriness and vanishing ob- jects. In contrast, PlaySlot maintains sharp object represen-
6

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centrie Video Prediction and Planning
2
3
11
16
2
3
11
16
PlaySlot
PlaySlot
SVG
SVG
CADDY
CADDY b)
a)
Figure 3: Qualitative comparison (a) ButtonPress and (b) BlockPush datasets. PlaySlot accurately predicts the scene dynamics, whereas baselines fail to predict object interactions, leading to blurriness and disappearing objects.
PSNR 1
LPIPS 4
Table 2: Quantitative comparison of PlaySlot with variants using continuous and discrete latent actions, as well as an oracle model with access to the ground truth actions.
10
40-
10-2
10-3
Results
20
10
RobotDB
Model Variant
PSNRT
SSIM
LPIPS.
2
Num. Objects
Num. Objects
PlaySlot
26.64
0.944
0.016
Random
w/ Cont. 2 w/ Discrete & w/ GT Actions
26.24
0.924
0.019
PlaySlot InvDyna
SVG
Exploration
20.60
0.849
0.040
PlaySlot InvDyns
CADDY
27.77
0.955
0.016
Figure 4:
Quantitative results on the GridShapes dataset with different numbers of objects. PlaySlot outperforms the baselines, particularly for higher number of objects.
PlaySlot
21.41
0.890
0.065
Expert Demos.
w/ Cont. 2 w/ Discrete à
22.00
0.900
0.061
18.00
0.791
0.109
w/ GT Actions
22.30
0.904
0.058
tations and correctly models interactions between objects, leading to accurate frame predictions. Further qualitative evaluations are provided in Appendix E.
Action Representation: Tab. 2 we compare the hybrid latent action representation used in PlaySlot with three dif- ferent variants that use continuous latent actions, discrete set of latent actions, and oracle variant with access to ground truth actions. Additionally, we evaluate the models on two different BlockPush variants, including set with random exploration sequences, similar to the training dis- tribution, and another set featuring expert demonstrations.
4.3. Model Analysis
Impact of Number of Moving Objects: We evaluate the performance of our method for different numbers of mov- ing objects. For this purpose, we train two PlaySlot variants with the InvDyng and InvDyng inverse dynamics mod- ules, respectively, and compare them with the SVG and CADDY baselines several variants of the GridShapes dataset featuring a different number of objects, ranging from one to five moving shapes. The results are shown in Fig. 4. CADDY and PlaySlot with InvDyns, which en- code scene dynamics using single latent action, perform strongly when forecasting one two objects, but experi- ence sharp drop in performance as the number of objects increases. SVG scales to multiple moving objects but codes all dynamics into a single distribution, limiting its flexibility and control. In contrast, PlaySlot with InvDyn, uses latent action per object, seamlessly scaling to a large number of moving agents by individually modeling the mo- tion of each object, thus outperforming all baselines.
On the random exploration set, PlaySlot with a hybrid la- tent action performs comparably the oracle, highlighting the ability of our InvDyn module to infer latent dynam- ics from unlabeled sequences. However, PlaySlott' 's perfor- mance drops on the expert demonstrations, with the variant using unconstrained continuous latent actions outperform- ing it. We attribute this to the large discrepancy between the action distribution of expert sequences and the training data, which challenges the generalization of the learned tion prototypes. In both cases, the variant with only discrete latent actions performs the worst, likely due to its limited flexibility in modeling the fine-grained dynamics of robot motion. Nonetheless, our hybrid action representation re- mains competitive, achieving performance close to the best methods despite the distribution shift.

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centree Video Prediction and Planning
2
3
6
11
16
3
11
16
Inferred Dynamics
Inferred Dynamics
Slot Masks
'Go Down' Act.
'Go Right' Act. 2
'Go Up" Act.
'Go Up" Act. 4
'Open' Act.
'Go Left' Act. 7
Figure 6: Qualitative results on real-world roboties se- quence. PlaySlot accurately predicts possible futures ditioned on a single reference frame and latent actions.
Figure 5: PlaySlot predictions given different latent ac- tions, including inferred inverse dynamics and three action prototypes. PlaySlot learns accurate object-centric repre- sentations and semantically consistent action prototypes.
4.4. Learning Behaviors from Expert Demonstrations
We evaluate the quality of PlaySlot's object-centric repre- sentations and inferred latent actions for a downstream be- havior learning task. To this end, we train a policy model and an action decoder, described in Sec. 3.4, imitate ButtonPress and BlockPush behaviors from a limited set of expert demonstrations. Furthermore, we train the models using three different random seeds and show the mean suc- cess rate and its standard deviation.
Learned Action Prototypes: Fig. 5 illustrates the effect
of different action prototypes learned by PlaySlot on the BlockPush dataset. Given single seed frame, we forecast 15 frames by repeatedly conditioning the predictor the same action prototype. We also visualize the predictions obtained using the latent actions inferred by our inverse dy- namics module from the ground truth sequence. PlaySlot effectively learns to infer precise robot actions from visual observations, as well as the physics of interacting objects. Additionally, Fig. shows that PlaySlot learns consistent and semantically meaningful actions, such moving the robot towards the right (action 2), left (action 7), or up- wards (action 4). Finally, the instance segmentation maps, obtained by assigning a different color to each slot mask, demonstrate how PlaySlot distinctly represents each object into separate slot, clearly disentangling each individual block, the robot and the background.
ButtonPress Behavior We compare our learned pol- icy, which imitates the ButtonPress behavior, against or- acle baselines with access to all expert demonstrations. These models directly regress the ground-truth actions from object-centric slots (Slot Oracle), ResNet feature (ResNet Oracle), and from feature maps output by the CADDY encoder (CADDY Oracle), respectively. In contrast, PlaySlot autoregressively predicts latent actions within its latent imagination before decoding them into ex- ecutable actions. We also evaluate modified CADDY variant with lightweight policy and action decoder that maps observed features to real-world actions.
Real-World Robotic Videos: We validate the applica- bility of PlaySlot to real-world robotic videos using the Sketchy (Cabi et al., 2020) dataset. Fig. 6 presents a quali- tative assessment demonstrating PlaySlot's 's ability to accu- rately infer the scene's 's inverse dynamics from a real robotic demonstration. Starting from single reference frame, our model reconstructs the ground truth sequence by predict- ing the robot's 's movement, rotation, and gripper opening. Furthermore, PlaySlot learns semantically meaningful and temporally consistent action prototypes that capture diverse robot behaviors such as opening the gripper (action 6), moving downwards (action 2), or upwards (action 4).
As shown in Fig. 7a), PlaySlot consistently outperforms CADDY across all training regimes. With as few as 200 demonstrations, PlaySlot matches the performance of the CADDY Oracle, which has direct access to more expert demonstrations and ground-truth actions. As the number of demonstrations increases, PlaySlot continues improve, approaching the performance of the oracle models, while CADDY struggles to generalize. These findings highlight the effectiveness of PlaySlot's object-centric representa- tions in learning robot behaviors from limited expert data. Additionally, we observe that slot-based models outper-
8

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
ButtonPress Behavior
2
10
100
Success Rate [%)
60
Sim. Actions
20
10
100
200
500
900
# Expert Demonstrations
ResNet Orac.
CADDY Orac.
CADDY
Latent Pred.
Slot Orac.
PlaySlot
BlockPush Behavior
100
Success Rate [%)
Sim. Actions
50
b)
Figure 8: Predicted frames using latent actions generated by the learned latent policy, and sequences simulated by executing the decoded latent actions for a) ButtonPress and b) BlockPush learned behaviors.
0200 500
1000
2000
4500
b)
Expert Demonstrations
SOLD
DreamerV3
PlaySlot
Figure 7: Success rate (%) as function of available expert demonstrations on the ButtonPress and b) BlockPush en- vironments. Object-centric models (Slot Oracle, PlaySlot and SOLD) outperform their counterparts. PlaySlot con- sistently improves with more demonstrations.
starting from single reference frame, autoregressively generates latent actions using the policy model and predicts future scene states in the latent space. The bottom row (la- beled Sim. Actions) visualizes the simulated execution of the decoded latent actions in the respective environments. PlaySlot learns to solve both tasks within its latent imag- ination, successfully reasoning over object properties and producing coherent sequences of latent actions that can be decoded into executable motions.
form their holistic counterparts, further emphasizing the advantage of structured object-centric representations for learning robot behaviors.
BlockPush Behavior We evaluate policy learned using PlaySlot the challenging BlockPush task, which requires reasoning over object properties. Our learned policy imi- tates the behavior from a limited number of expert demon- strations, with only approximately 80% being successful. The imperfect nature of the expert data adds an extra layer of difficulty the learning process.
5. Conclusion
We introduced PlaySlot, novel framework for control- lable object-centric video prediction. PlaySlot parses video frames into object slots, infers the scene's inverse dynam- ies, and predicts future object states and video frames by modeling the object dynamics and interactions, con- ditioned on inferred latent actions. Through extensive experiments, we demonstrated that PlaySlot learns a se- mantically rich and meaningful action space, allowing for accurate video frame predictions. Our method outper- forms several baseline models, offering superior control- lability and interpretability. We further demonstrated that the learned object-centric representations and inferred la- tent actions can be utilized to predict future frames with precise robot control, while also enabling learning complex robot behaviors from limited, unlabeled demonstrations outperforming methods that rely on holistic scene encod- ing. These results highlight PlaySlot as versatile and in- terpretable world model, well-suited for wide range of tasks in autonomous systems.
We compare PlaySlot to model-based reinforcement learning baselines with holistic (DreamerV3 (Hafner et al.,
2023)) and object-centric (SOLD (Mosbach et al., 2025)) latent spaces, both of which learn the robot behavior by interacting with the environment. As shown in Fig. 7b).
PlaySlot closes the gap with the baseline models as the number available demonstrations increases, demonstrat- ing its ability generalize from limited data.
Visualization of Learned Behaviors Fig. 8 illustrates the learned latent policies in the (a) ButtonPress and (b) BlockPush enviromments, respectively. The top row in each sequence (labeled Latent Pred.) shows predicted trajecto- ries within PlaySlot's latent imagination, where the model,

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
Acknowledgment
Bruce, J., Dennis, M. D., Edwards, A., Parker-Holder, J., Shi, Y., Hughes, E., Lai, M., Mavalankar, A., Steiger- wald, R., Apps, C.,, et al. Genie: Generative interactive environments. In International Conference on Machine Learning (ICML), 2024.
This work was funded by grant BE 2556/16-2 (Research Unit FOR 2535 Anticipating Human Behavior) of the German Research Foundation (DFG). Computational re- sources provided by the German Al Service Center WestAI.
Burgess, C. P., Matthey, I.., Watters, N., Kabra, R., Hig- gins, I, Botvinick, M., and Lerchner, A. Monet: Unsupervised scene decomposition and representation. 2019.
Impact Statement
This paper presents work whose goal to advance the field of machine learning by introducing PlaySlot-an object- centric video prediction model that forecasts future video frames conditioned inferred object-centric representa- tions and latent actions, enhancing its interpretability and controllability, as well as learning representations that can be used for action planning. This advancement is particu- larly relevant for domains such as robotics and autonomous systems, where understanding and predicting complex en- vironments essential for correct and safe operation. However, the deployment of such models in real systems requires careful consideration for ethical and safety chal- lenges, such as biases in the training data or lack of trans- parency in the decision-making process.
Cabi, S., Colmenarejo, S. G., Novikov, A., Konyushkova, K., Reed, S., Jeong, R., Zolna, K., Aytar, Y., Budden, D., Vecerik, M., et al. Scaling data-driven roboties with reward sketching and batch reinforcement learning. In Science and Systems (RSS), 2020.
Robotics:
Cho, K. van Merriënboer, B., Gulcehre, C.,, Bahdanau, D., Bougares, F, Schwenk, H., and Bengio, Y. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Conference on Empiri- cal Methods in Natural Language Processing (EMNLP), 2014.
Creswell, A., Kabra, R., Burgess, C., and Shanahan, M. Unsupervised object- transition models for 3D par- tially observable environments. Advances in Neural In- formation Processing Systems (NeurIPS), 2021.
References
Assouel, R., Castrejon, I.., Courville, A., Ballas, N., and Bengio, Y. Vim: Variational independent modules for video prediction. Conference on Causal Learning and Reasoning, pp. 70-89, PMLR, 2022.
Cui, Z. J., Pan, H., Tyer, A., Haldar, S., and Pinto, L. DynaMo: In-Domain Dynamies Pretraining for Visuo- Motor Control. In Advances in Neural Information Pro- cessing Systems (NeurIPS), 2024.
Aydemir, G., Xie, W., and Guney, F. Self-supervised object-centric learning for videos. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Daniel, T. and Tamar, A. DDLP: Unsupervised Object- centric Video Prediction with Deep Dynamic Latent Par- ticles. Transactions on Machine Learning Research (TMLR), 2024.
Bao, Z., Tokmakov, P., Wang, Y.X.,, Gaidon, A., and Hebert, M. Object discovery from motion-guides to- kens. In IEEE/CVF Conference on Computer Vision and Pallern Recognition (2023), 2023.
Denton, E. and Fergus, R. Stochastic video generation with a learned prior. In International Conference on Machine Learning (ICML), 2018
Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE Trans- actions on Pattern Analysis and Machine Intelligence (TPAMI), 35(8): 2013.
Dittadi, A., Papa, S., De Vita, M., Schölkopf, B., Winther, O., and Locatello, F. Generalization and robustness im- plications in object-centric learning. In International Conference on Machine Learning (ICML), 2022.
Biza, O., Van Steenkiste, S., Sajjadi, M. S., Elsayed, G. F., Mahendran, A., and Kipf, T. Invariant slot attention: Object discovery with slot-centric reference frames. In International Conference Machine Learning (ICML), 2023.
Edwards, A., Sahni, H., Schroecker, Y., and Isbell, C. Im- itating latent policies from observation. In International Conference on Machine Learning (ICML), 2019.
Elsayed, G. F., Mahendran, A., van Steenkiste, S., Greff, K., Mozer, M. C., and Kipf, T. SAVi++: Towards end- to-end object-centric learning from real-world videos. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Brandfonbrener, D., Nachum, O., and Bruna, J. Inverse dy- namics pretraining learns good representations for mul- titask imitation. In Advances in Neural Information Pro- cessing Systems (NeurIPS), volume 36, 2024.
10

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
Ferraro, S., Mazzaglia, P., Verbelen, T., and Dhoedt, B. Fo- cus: Object-centric world models for robotics manipu- lation. In Advances in Neural Information Processing Systems Workshops (NeurIPSw), 2023.
Mamaghan, A. M. K., Papa, S., Johansson, K. H., Bauer, S., and Dittadi, A. Exploring the effectiveness of object-centric representations in visual question answer- ing: Comparative insights with foundation models. In International Conference on Learning Representations (ICLR), 2025.
Greff, K., Van Steenkiste, S., and Schmidhuber, J. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208, 2020.
Menapace, W., Lathuiliere, S., Tulyakov, S., Siarohin, A., and Ricci, E. Playable video generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Hafner, D.,
Pasukonis, J., Ba, J., and Lillicrap, T. Mastering diverse domains through world models. arXiv preprint arXiv:2301. 34044,4444, 2023.
Menapace, W. Lathuilière. S., Siarohin, A., Theobalt, C., Tulyakov, S., Golyanik, V., and Ricci, E. Playable en- vironments: Video manipulation in space and time. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Jeong, Y.. Chun, J.. Cha, S., and Kim, T. Object-centric world model for language- eogiddee-guieed manipulation. arXiv preprint arXiv:2503.06 70, 2025.
Jiang, J., Deng, F., Singh, G., and Ahn, S. Object-centric slot diffusion. In Advances in Neural Information Pro- cessing Systems (NeurlPS), 2023.
Mco, C., Nakano, A., Licä, M., Didolkar, A., Suzuki, M., Goyal, A., Zhang, M., Dauwels, J., Matsuo, Y., and Ben- gio, Y. Object-centric temporal consistency via condi- tional autoregressive inductive biases. arXiv preprint arXiv:2410.15728, 2024.
Jiang, J., Deng, F., Singh, G., Lee, M., and Ahn, S. SlotSSMs: Slot State Space Models. In Advances in Neural Information Processing Systems (NeurlPS), 2024.
Mosbach, M., Niklas Ewertz, J., Villar- Vilral-Crralle, A., and Behnke, S. SOLD: Slot Object-Centric Latent Dynam- ies Models for Relational Manipulation Learning from Pixels. In International Conference on Machine Learn- ing (ICML), 2025.
Johnson, S. P. Object perception. In Oxford Research Ency- clopedia of Psychology. Oxford University Press, 2018.
Kahneman, D., Treisman, A., and Gibbs, B. J. The review- ing of object files: Object-specific integration of infor- mation. Cognitive psychology, 24(2):175-219) 1992.
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in PyTorch. In Inter- national Conference on Neural Information Processing Systems Workshops (NeurIPSw), 2017.
Kingma, D. P. and Ba, J. A method for stochastic optimiza- tion. In International Conference on Learning Represen- tations (ICLR), 2015.
Kipf, T., Elsayed, G. F., Mahendran, A., Stone, A., Sabour, S., Heigold, G., Jonschkowski, R., Dosovitskiy, A., and Greff, K. Conditional Object-Centric Learning from Video. In International Conference on Learning Rep- resentations (ICLR), 2022.
Schmidt, D. and Jiang, M. Learning to act without ac- tions. In International Conference on Learning Repre- sentations (ICLR), 2024.
Seitzer, M., Horn, M., Zadaianchuk, A., Zietlow, D., Xiao, C.-J., He, T., Zhang, Z., Schölkopf, B., Brox, T., et al. Bridging the gap to real-world object- centric learning. In International Conference on Learn- ing Representations (ICLR), 2023.
T.,
Li, R., Jabri, A., Darrell, T., and Agrawal, P. Towards practical multi-object manipulation using relational re- inforcement learning. In IEEE International Conference on Robotics and Automation (ICRA), 2020.
Singh, G., Deng, F., and Ahn, S. Illiterate dall-e learns to compose. arXiv preprint 10X.v:2110.11405. 2021.
Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., and Bachem, O. Challenging common as- sumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Learning (ICML), 2019.
Singh, G., Wu, Y.FF, and Ahn, S. Simple unsuper- vised object-centric learning for complex and naturalis- tic videos. In Advances in Neural Information Process- ing Systems (NeurIPS), 2022.
Locatello, F., Weissenborn, D., Unterthiner, T., Mahen- dran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. Object-centrie learning with slot attention. In In- ternational Conference on Neural Information Process- ing Systems (NeurIPS), 2020.
Struckmeier, O. and Kyrki, V. ILPO-MP: Mode Priors Pre- vent Mode Collapse when Imitating Latent Policies from Observations. Transactions on Machine Learning Re- search (TMLR), 2023.

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Vidco Prediction and Planning
Tian, Y., Yang, S., Zeng, J., Wang, P., Lin, D., Dong, H., and Pang, J. Predictive inverse dynamics models are scalable learners for robotic manipulation. In Interna- tional Conference on Learning Representations (ICLR), 2025.
Ye, W., Zhang, Y., Abbeel, P., and Gao, Y. Become pro- ficient player with limited data through watching pure videos. In International Conference on Learning Repre- sentations (ICLR), 2022.
Yoon, J., Wu, Y.-F., Bae, H., and Ahn, S. An investiga- tion into pre-training object-centric representations for reinforcement learning. In International Conference Machine Learning (ICML), 2023.
Todorov, E.,, Erez, T., and Tassa, Y. Mujoco: A physics en- gine for model-based control. In IEEE/RST International Conference on Intelligent Robots and Systems (IROS), 2012.
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evalu- ation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2020.
Van Den Oord, A. and Vinyals, O. Neural discrete repre- sentation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, Atten- tion is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Zadaianchuk, A., Seitzer, M., and Martius, G. Object- centric learning for real-world videos by predicting tem- poral feature similarities. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2024.
Villar-Corrales A., Wahdan, I., and Behnke, Object- centric video prediction via decoupling of object dynam- ics and interactions. In IEEE International Conference on Image Processing (ICIP), 2023.
Zhang, C., Gupta, A., and Zisserman, A. Is an object- centric video representation beneficial for transfer? In Asian Conference Computer Vision (ACCV), 2022.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep fea- tures as perceptual metric. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
Cillar-Corrales, A., Plepi, G., and Behnke, S. Object- centric image to video generation with language guid- ance. arXiv preprint arXiv:2502. 1655, 2025.
Wang, X., Li, X., Hu, Y. Zhu, H., Hou, C., Lan, C., and Chen, Z. TIV-Diffusio:: Towards object-centric move- ment for text-driven image to video generation. In AAAI Conference Artificial Intelligence, 2024.
Zoran, D., Kabra, R., Lerehner, A., and Rezende, Parts: Unsupervised segmentation with slots, attention and independence maximization. In IEEE/VV Interna- tional Conference Computer Vision (ICCV), 2021.
Wang, 7., Bovik, A. A.C Sheikh, H. R., and Simoncelli, E. Image quality assessment: from error visibility to struc- tural similarity. IEEE Transactions on Image Processing (TIP), 13(4))600-613 612, 2004.
Watters, N., Matthey, L., Burgess, P., and Lerchner, A. Spatial broadcast decoder: simple architecture for dis- entangled representations in VAEs, 2019. URL https: / //openzeview net/forunt 1d=
Wu, Z., Dvoriik, N., Greff, K., Kipf, T., and Garg, A. Slot- Former: Unsupervised visual dynamics simulation with object-centric models. In International Conference on Learning Representations (ICLR), 2023a.
Wu, Z., Hu, J., Lu, W., Gilitschenski, I., and Garg, A. Slot- diffusion: Object-centric generative modeling with dif- fusion models. Advances in Neural Information Pro- cessing Systems (NeurIPS), 2023b.
Ye, S., Jang, J., Jeon, B., Joo, S., Yang, J., Peng, B., Man- dlekar, A., Tan, R., Chao, Y.WW., Lin, B. Y., et al. Latent action pretraining from videos. In International Confer- ence on Learning Representations (ICLR), 2025.
12

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
A. Limitations & Future Work
We recognize two main limitations that currently limit the scope our PlaySlot framework to simple tabletop robotic scenarios, preventing from generalizing to more complex domains.
Limited Decomposition Model The first limitation arises from the SAVi object-centric decomposition model used at the core of our framework. SAVi achieves great decomposition performance datasets with objects of simple shapes and textures. However, it fails to generalize to complex real-world robotic scenarios, hence limiting the current scope of PlaySlot to robotic tabletop simulations, or simple real-world environments as in Sketchy.
Single Latent Action The second limitation lies at the representational capability of our latent actions and action proto- types. robotic scenarios, several actions often happen simultaneously, such as moving and rotating the robot, as well as opening or closing the gripper. Our current latent action representation jointly models the scene's 's inverse dynamics, en- coding together all actions into a single latent space. This entangled representation limits our ability to control the agents in the scene with greater precision.
Future Work In future work, we plan extend our proposed PlaySlot framework with more capable decomposition models, such DINOSAUR (Seitzer et al., 2023) or SOLV (Aydemir et al., 2023), well scale our inverse dynamics and predictor models. Furthermore, we can employ factorized latent action vectors, which represent in disentangled manner different actions that happen simultaneously, such as moving the robot arm and opening the gripper. We believe that this architectural modifications will enable us to use PlaySlot on more complex robotic simulations and perform real- world robotic experiments.
B. Implementation Details
In this section, we describe the network architecture and training details for each of the components in our PlaySlot framework. Our models are implemented in Py PyTorch (Paszke et al., 2017) and are trained on single NVIDIA A100 GPU,
B.1. Object-Centric Learning
We closely follow Kipf et al. (2022) for the implementation of the SAVi object-centric decomposition model, which we employ as scene parsing and object rendering modules. We strictly adhere to the architecture of their proposed CNN-based image encoder ESAN., slot decoder DSAN. transformer-based dynamics transition module, and Slot Attention corrector. We use a variable number of 128-dimensional object slots, depending of the dataset. Namely, we employ eight object slots on BlockPush, four slots on ButtonPress, three slots on GridShapes, and eight slots on Sketchy. On all datasets, we sample the initial object slots So from a Gaussian distribution with learned mean and covariance. Furthermore, we use three Slot Attention iterations for the initial video frame to obtain good initial object-centric decomposition, and single iteration for subsequent frames, which suffices to recursively update the slot state given the newly observed image features.
B.2. Inverse Dynamics Module
We propose two variants of our InvDyn module.
InyDyng: InvDyng jointly processes the object slots from single time step S, along with an additional token [ACT] using a transformer encoder. We use a four-layer transformer encoder with -56-dimensional tokens, four 64-dimensional heads and hidden dimension of 1024. This module aggregates information from the object slots into the [ACT] token, and outputs single latent action 2, that captures the agent's action, making well-suited for single-agent environments.
InvDyn InvDynM independently processes each object slot with a shared MLP, thus generating Ns latent action em- beddings, each representing the action of specific object in the scene. This design makes InvDyn well-suited for environments with multiple moving agents. We employ two-layer MLP, featuring a ReLU nonlinear activation and layer normalization.
As described in Sec. 3.2.2, the generated latent actions ê, are vector-quantized to assign them to their corresponding action prototype P:- On the ButtonPress, BlockPush, and Sketchy datasets, we use the InvDyng variant with eight different 16-dimensional action prototypes, whereas for GridShapes we use InvDyng with five distinct eight-dimensional action
13

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Vidco Prediction and Planning
prototypes. Following common practice, update the action prototypes using the exponential moving average updates of cluster assignment counts (Van Den Oord & Vinyals, 2017).
B.3. Conditional Object-Centric Predictor
Our conditional object-centric predictor (cOCVP) is inspired by the transformer-based SlotFormer (Wu et al., 2023a) architecture. Our cOCVP module features four layers, 256-dimensional tokens, eight 64-dimensional attention heads, and hidden dimension of 1024.
To enable predictions conditioned on the inferred latent actions, cOCVP maps the action prototypes РI:+. variability em- beddings Viat and object slots S1 into the token dimensionality. The projected object slots are then conditioned by adding them with the projected action prototype and variability embedding from the corresponding time step. Furthermore, fol- lowing Wu et al. (2023a), we augment these representations with temporal sinusoidal positional encoding, such that all tokens from the same time step receive the same encoding, thus preserving the inherent permutation equivariance of the objects.
B.4. Policy Model and Action Decoder
The policy model fa follows similar architecture InvDyns. fx four layer transformer that jointly processes the objects slots from single time step S, and an additional token [ACT] in order to regress latent action.
The action decoder is three-layer MLP with a hidden dimension of 128 that maps latent action vectors into real-world actions.
B.5. Training Details
SAVi Training: SAVi is trained for object-centric decomposition using the Adam optimizer (Kingma & Ba, 2015), a batch size of 64, sequences of length eight frames, and base learning rate of 10-4, which is linearly warmed-up for the first 4000 steps, followed by cosine annealing for the remaining of the training process. Moreover, we clip the gradients to a maximum norm of 0.05.
InvDyn and cOCVP Training: We jointly train our InvDyn and cOCVP modules given pretrained SAVi decomposition model. These modules are trained with the Adam optimizer (Kingma & Ba, 2015), batch size of 64, and base learning rate of 2 10-4, which decreases during training with cosine annealing schedule. To stabilize the training, we clip the gradients to a maximum norm of 0.05. We set the loss weights to Ammg = 1, Asta = 1, and \vo 0.25.
ft and D, Training: We train the fa and D2 modules given pretrained and frozen SAVi, InvDyn and cOCVP modules. These modules are trained with the Adam optimizer (Kingma & Ba, 2015), batch size of 64, and fixed learning rate of 2 x 10-4
C. Dataset Details
C.1. BlockPush
This environment, inspired by Li et al. (2020) and simulated using MuJoCo (Todorov et al., 2012), features robot arm on a tabletop interacting with multiple uni-colored blocks. We use two different dataset variants.
The first variant consists of a robot controlled by a random exploration policy, moving in the environment with minimal meaningful object interactions. This easily simulated dataset includes 20,000 training sequences and 2,000 validation sequences. We use this dataset variant for training SAVi, as well for jointly training our inverse dynamics and conditional predictor modules.
The second variant contains smaller subset of expert demonstrations where the robot is tasked to push the block of distinct color to a target location marked with red sphere. This task evaluates the capabilities of an agent to reason about object relations and model object collisions. We collect 5,000 expert demonstrations using a pretrained policy (Mosbach et al., 2025) with a success rate of &s 80% for this pushing task. We split this dataset into 4,500 training sequences, which are used for training the policy model and action decoder; and 500 evaluation sequences that are used for benchmarking the prediction models.
14

---

PlaySlot:
Learning Inverse Latent Dynamics for Controllable Object-Centric Vidco Prediction and Planning
C.2. Press
This environment, based on MetaWorld (Yu et al., 2020) features Sawyer robot arm tasked with pressing red button. Unlike BlockPush, it involves mon-object-centric task with complex shapes and textures. As before, we use two different dataset variants.
The first variant contains 10,000 sequences, split into 9,000 training and 1,000 validation videos, with the robot controlled by a random exploration policy. We use this dataset variant to train SAVi, as well as our inverse dynamics and conditional predictor modules.
The second variant includes small subset of expert demonstrations where the robot successfully presses the red button. We collect 1,000 expert demonstrations using expert policy, from which 900 used for training the policy model and action decoder, and 100 demonstrations are used for benchmarking the prediction models.
C.3. GridShapes
This dataset features one more simple 2D shapes moving in grid-like pattern on top of colored background. The shapes can be either ball, triangle or square, and have random color. These shapes move up, down, left, right or remain still, and revert their motion when an image boundary is reached, thus emulating a bouncing effect. To introduce some stochasticity into the motion, the shapes randomly change direction with a predefined probability of 0.25. We train and evaluate the models on several variants of the GridShapes dataset featuring different number of objects, ranging from one single moving shape, to five objects moving independently in the same sequence. This simple dataset serves as benchmark to evaluate a model's s ability to jointly predict the motion of multiple moving agents in the scene.
C.4. Sketchy
Sketchy (Cabi et al., 2020) is a real-world robotics dataset featuring videos in which a robotic gripper interacts with diverse objects of different shape and color on tabletop scenario. We use the human demonstration sequences provided in the dataset, resulting in a total of 394 sequences with an average length of 80 frames. For each sequence, use the observations taken with the front right and front left cameras, and discard the cropped images. We resize the video frames to an image size of 64 x 64 and use 316 and 78 sequences for training and evaluation, respectively.
D. Baselines
In our experiments, we compare our approach with different baseline models, including the stochastic and playable video prediction models SVG (Denton & Fergus, 2018) and CADDY (Menapace al., 2021), as well as the object-centric video prediction models SlotFormer (Wu et al., 2023a) and OCVP (Villar-Corrales al., 2023). To ensure fair comparison, balance the number of learnable parameters and compute requirements for all methods. Tab. 3 lists the number of learnable parameters in our proposed PlaySlot as well as for all baselines.
Table 3: Number of learnable parameters for PlaySlot and baselines.
Model
Trainable Parameters (M)
SVG
18.84
CADDY
9.34
SlotFormer
3.97 (2.96 Predictor + 1.01 SAVi)
OCVP
4.86 (3.85 Predictor 1.01 SAVI)
PlaySlot
8.49 (3.19 InvDyn 4.27 cOCVP 1.01 SAVI)
SVG
SVG (Denton & Fergus, 2018) is generative model for video prediction that captures both deterministic dynamics and stochastic variations in video sequences. It combines variational autoencoder (VAE) with recurrent neural networks (RNNs) to model stochastic temporal dynamics. SVG represents probabilistic framework, where the next frame X is generated based on the previous frame X, and a latent sample 2, drawn from a latent distribution. For a fair comparison with PlaySlot, we use SVG's posterior module to infer latent vectors using future video frames, and use those latent vectors
15

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Vidco Prediction and Planning
guide the video prediction process. In our experiments, we adapt the original implementation' and an SVG variant with learned prior, VGG-like encoder and decoders, and two recurrent predictor layers.
Main Difference with PlaySlot: SVG operates with holistic scene representations, whereas our proposed method employs structured object-centric representation. Furthermore, SVG encodes the stochastic scene dynamics into a single contin- uous latent vector. In contrast, PlaySlot follows a hybrid approach in which the latent action vectors are composed of an action prototype and action variability embeddings, making the prediction process more controllable and interpretable.
D.2. CADDY
CADDY (Menapace et al., 2022) is a recurrent encoder-decodee model designed for playable video generation, enabling user-controllable future video prediction. CADDY infers latent actions that encode the agent's s actions between consecutive pairs of frames. These latent actions are parameterized with discrete one-hot action label, which determines the high-level action taking place; and high-dimensional action variability embedding, which describes the variability of each action and captures the possible non-determinism in the environment. We adapt the original implementation" for our experiments.
Main Difference with PlaySlot: CADDY operates with holistic scene representations, i.e. CNN features, whereas our proposed method employs structured object-centric representation. Moreover, despite both methods using a hybrid parameterization of the latent actions, they differ in their implementation. CADDY learns a discrete one-hot action label by minimizing multiple regularization objectives, including an action matching loss and several Kullback-Leibler (KL) divergences losses. In contrast, PlaySlot employs a discrete set of high-dimensional action prototypes, which are learned via vector-quantization of the latent space. We empirically verify that both approaches achieve comparable performance. However, our vector quantization approach requires significantly fewer
and is easier to tune.
D.3. SlotFormer
SlotFormer (Wu al., 2023a) is object-centric video prediction model that builds upon slot-based representations, First, it uses Slot Attention to parse videos frames into object-centric latent representations, called slots. SlotFormer then employs transformer-based autoregressive predictor module, which jointly processes all input slots in order to forecast future object representations. Finally, the predicted slots are decoded into object images and video frames. In our experiments, we adapt the original implementation".
Main Difference with PlaySlot: SlotFormer forecasts future slots in an unconditional manner, thus not being able to model stochastic environments or agents such as robots. PlaySlot addresses this challenge by inferring the scene inverse dynamics and using them condition the prediction process.
D.4. OCVP
OCVP (Villar- Corrales et al., 2023), similar SlotFormer, is slot-based object-centric video prediction model. Differing from SlotFormer, OCVP leverages specialized decoupled attention mechanisms, relational and temporal attention, which model the object interactions and dynamics, respectively. In our experiments, use the OCVP-Seq variant with default settings adapted from original implementation'.
Main Difference with PlaySlot: OCVP forecasts future slots in an unconditional manner, thus not being able to model stochastic environments or agents such as robots. Play addresses this challenge by inferring the scene inverse dynamics and using them to condition the prediction process.
phttps://gitee ht//://hhbbbcmm//eeeett////ggg
peattps://itthin http///tt..c.olllllaaaaaaaaaaaae
16

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
E. Additional Results
E.1. Effect of the Number of Actions
In Tab. we evaluate on the BlockPush dataset multiple PlaySlot variants using a different number learned action prototypes. We show that using eight learned action prototypes achieves the best video prediction performance, while learning concise semantically meaningful set of action prototypes.
Table 4: Evaluation of PlaySlot variants on the BlockPush dataset using different number of learned action prototypes.
BlockPush
# Actions
PSNRT
SSIM
LPIPS.
21.26
0.886
0.071
8
21.41
0.890
0.066
10
21.36
0.889
0.067
15
21.38
0.889
0.067
E.2. Learned Behaviors from Expert Demonstrations
In this section, we present additional experiments and results demonstrating PlaySlot's ability to learn robot behaviors from unlabeled expert demonstrations. Sec. 1.2.1, we compare policies learned with PlaySlot under two different evaluation protocols. In Sec. E.2.2, we compare PlaySlot with LAPO (Schmidt & Jiang, 2024), a recent model for behavior learning from unlabeled demonstrations. Finally, Sec. E.2.3 provides additional rollout visualizations from PlaySlot's policy, along with simulated executions of the predicted action sequences.
1. 1. IMAGINED ROLLOUT vs. SIMULATION
In Fig. 9, we compare two different protocols for evaluating the behaviors learned with PlaySlot on the ButtonPress and BlockPush environments. Namely:
w/ Latent Imagination The learned behavior unfolds within the model's latent imagination, where future states are predicted with PlaySlot. To assess the performance of the policy, the predicted sequence of latent actions is decoded and exccuted in the simulator, with the final assesses how well the learned representations and policies generalize within the internal model's constraints.
or failure-used as the performance measure. This protocol
The latent actions output by the policy are directly decoded and executed the simulator, which generates the subsequent environment state. This process is repeated iteratively for a fixed number of steps or until the task is suc- cessfully completed. This protocol evaluates the quality of both the learned policy and action representation independently of the world model's accuracy, providing a more direct measure of the policy's 's effectiveness in an interactive setting.
w/ Simulation
As shown in Fig. 9, success rates improve for both evaluation protocols the number of available expert demonstra- tions increases. Notably, with few as 500 and 2000 expert demonstrations, PlaySlot w/ Simulator matches-ant even
ButtonPress Behavior
BlockPush Behavior
100
65
Success Rate [%)]
Success Rate [%)
50
65
35
20
30
0
0 10 50 100
200
500
900
200 500
1000
2000
4500
Expert Demonstrations
# Expert Demonstrations
a)
ResNet Oracle
PlaySlot w/ Simulator
b) ResNet Oracle
PlaySlot w/ Simulator
Slot Oracle
PlaySlot w/ Latent Imag.
- Slot Oracle
PlaySlot w/ Latent Imag.
Figure 9: Success rates (%) of policies learned with PlaySlot under evaluation protocols the a) ButtonPress and b) BlockPush environments. Simulation-based rollouts outperform latent imagination, but both approaches improve with more available expert demonstrations and surpass the ResNet Oracle.
17

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
ButtonPress Behavior
BlockPush Behavior
100
100
Success Rate [%)]
Success Rate [%
75
70
50
25
30
100
200
500
900
0.200
500
1000
2000 # Expert Demonstrations
4500
# Expert Demonstrations
a)
ResNet Oracle
PlaySlot
b)
SOLD
ResNet Oracle
PlaySlot
Slot Oracle
LAPO
DreamerV3
Slot Oracle
LAPOO
Figure 10: Success rate (%) as a function of available expert demonstrations on the a) ButtonPress and b) BlockPush envi- ronments. PlaySlot outperforms LAPO, especially on the challenging BlockPush environment, which requires reasoning about object properties.
outperforms the ResNet Oracle baseline on the ButtonPress and BlockPush environments, respectively. This suggests that PlaySlot can effectively learn robust and performant robot behaviors from limited expert data when evaluated in an interactive setting.
Unsurprisingly, the latent imagination-based rollouts perform slightly worse than those executed in the simulator. This discrepancy arises because errors in the learned world model can compound over time, causing deviations from real- world dynamics. Nonetheless, the performance gap remains moderate, with the imagination-base rollouts succeeding approximately 70% and 50% of the scenes on the ButtonPress and BlockPush environments, respectively.
E.2.2. COMPARISON WITH LAPO
LAPO (Schmidt & Jiang, 2024) a recent model proposed for learning world model, an inverse dynamics model, and latent action policy from unlabeled videos. In contrast to the object-centric representations employed by PlaySlot, LAPO relies on feature maps output by convolutional encoder.
ButtonPress Behavior In Fig. 10a), we compare policies learned by PlaySlot and LAPO on the ButtonPress task, using a varying number of expert demonstrations. As reference baselines, we include two oracle models trained with access to all available expert demonstrations and to ground-trun truth actions: the Slot Oracle, which uses object-centric representations, and the ResNet Oracle, which uses ResNet feature maps.
Both PlaySlot and LAPO perform comparably on the ButtonPress task, which requires limited object-centric reasoning. Nonetheless, PlaySlot achieves slightly better sample-eficiennyy and consistently higher performance than LAPO across most data regimes. Notably. both methods approach the performance of the Oracle models (which have access to ground- truth actions during training) with a limited number of expert demonstrations. This is especially notable for PlaySlot, which matches the performance of the ResNet oracle with as few as 500 demonstrations.
BlockPush Behavior On the challenging BlockPush task, which requires reasoning over object properties and relations, both models learn policies from varying number of expert demonstrations, with only 80% depicting successful task completions-adding further difficulty due to noisy supervision.
As reference baselines, we include two model- based reinforcement learning methods, using holistic (DreamerV3 (Hafner et al., 2023)) and object-centric (SOLD (Mosbach et al., 2025)) latent spaces, both of which learn the robot behavior by directly interacting with the environment and using action and reward information. Additionally, we include two oracle behavior cloning baselines trained on expert demonstrations, with access to ground-truth actions, and leveraging object- centric representations (Slot Oracle) or ResNet feature maps (ResNet Oracle).
Fig. 10b) shows that PlaySlot consistently outperforms LAPO by a large margin across all data regimes, demonstrating much stronger sample-effiiency and substantially higher final performance. While LAPO struggles to scale with the number of demonstrations, PlaySlot shows steady improvement, narrowing the gap between latent action models and baseline methods trained with access to ground-truth actions. Notably, PlaySlot even outperforms the ResNet Oracle model with as few as 2000 noisy expert demonstrations, and closely approaches the performance of the strong Slot Oracle baseline.
18

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centrie Video Prediction and Planning
2
3
11
16
2
3
11
16
PlaySlot
PlaySlot
SVG
SVG
CADDY
CADDY
a)
Figure 11: Qualitative comparison on (a) ButtonPress and (b) BlockPush datasets. Our method accurately predicts the scene dynamies, whereas the baselines fail to model object physics and interactions, leading to disappearing objects and failing to predict the pushing of the cyan block the target location.
Additionally, observe that slot-based object-centric models outperform their holistic counterparts, i.e. SOLD outper- forms DreamerV3, Slot Oracle outperforms ResNet Oracle, and PlaySlot outperforms LAPO; further emphasizing the advantage of structured slot-based latent spaces.
These results highlight the benefits of object-centric representations for sample-efficient behavior learning, especially in tasks that require understanding object properties and their relations. By parsing the scene into individual objects, PlaySlot able to generalize more effectively from limited and noisy demonstrations and infer complex behaviors, which are often challenging for models relying on monolithic, holistic representations.
E.2.3. VISUALIZATIONS OF LEARNED BEHIVVIRSS
ButtonPress Behavior: Fig. 12 illustrates PlaySlot's learned behavior in the ButtonPress environment. The top row in each sequence (labeled as Latent Behavior) shows predicted trajectories within PlaySlot's latent imagination, where the model autoregressively generates actions using the policy model and predicts future scene states in the latent space. The bottom row (labeled Sim. Actions) depicts the simulated exccution of the decoded latent actions in the environment. PlaySlot learns to solve the task within its latent imagination, successfully reasoning about the required action sequences before translating its latent actions into executable motions. Fig. 12b) shows failure case where PlaySlot predicts within its latent imagination trajectory that leads successfully pressing the button. However, accumulated errors in the world model and during action decoding cause the simulated execution to miss the button.
Fig. 13 illustrates PlaySlot's learned behavior in the BlockPush environment. The top row in each sequence (labeled as Latent Behavior) shows predicted trajectories within PlaySlot's latent imagination, where the model autoregressively generates actions using the policy model and predicts future scene states in the latent space. The bottom row (labeled Sim. Actions) depicts the simulated execution of the decoded latent actions in the environment. PlaySlot learns solve the task within its latent imagination, successfully reasoning about object properties and generating precise sequence of latent actions, which can be decoded into executable motions. Fig. 13c) shows failure case where PlaySlot controls the robot to interact with the correct block, but fails to place it in the target location.
BlockPush Behavior:
E.3. Qualitative Results
In the following sections, we provide additional qualitative results all datasets, well as qualitative comparisons with baselines. Further qualitative results and animations are provided in the project website.
1. COMPARISON WITH BASELINES
Fig. 11 depicts a qualitative comparison between SVG, CADDY and PlaySlot on the ButtonPress and BlockPush datasets, respectively. On the ButtonPress dataset, as shown in Fig. 1 la), all methods accurately model the trajectory the robot arm. However, on the complex BlockPush task, depicted in Fig. 11b), SVG and CADDY fail model the object collisions,
19

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Vidco Prediction and Planning
failing to move the block of distinet color to the target location, and leading blurriness and disappearing objects. In contrast, PlaySlot maintains sharp object representations and correctly models interactions between objects, leading to accurate frame predictions.
E.3.2. BL.OCKPUSH DATASET
Fig. 14 shows two qualitative comparisons between PlaySlot, CADDY and SVG on the BlockPush dataset. Our proposed method, which explicitly models object interactions, preserves sharp object representations and accurately predicts future frames. In contrast, SVG and CADDY, which rely on holistic scene features for forecasting, struggle to model object collisions, resulting in blurry predictions and disappearing objects.
Fig. 15 shows the predicted video frames, slot masks, and objects representations a BlockPush sequence. PlaySlot parses the scene into precise object images and masks, which can be assigned a unique color to obtain a segmentation of the scene. Our approach decomposes the BlockPush environment using eight object slots, where one slot represents the background, five slots to different blocks, one slot to the red target, and one slot to the rohot arm. The sharp object images and masks demonstrate that PlaySlot encodes into each slot features from the corresponding object. This allows our method directly reason about object properties, dynamics and interactions, allowing for accurate future frame predictions.
Fig. 16 depicts the effect each action prototype learned by PlaySlot the BlockPush dataset. PlaySlot learns consistent semantically meaningful action protolypes that control the robot arm move in a specific direction. We note that some actions prototypes, e.g. action 5 and 7, perform semantically similar actions but with different velocities.
E.3.3. BUTTONPRESS DATASET
Fig. 17 shows a comparisons between PlaySlot, CADDY and SVG on the ButtonPress dataset. All methods successfully predict the ground truth sequence by inferring and modeling the robot's dynamics,
Fig. 18 shows PlaySlot's predictions and object representations on ButtonPress sequence. We visualize the ground truth sequence, the predicted frames, segmentation obtained by assigning different color to each slot mask, well as the object reconstructions for four slots. Play assigns one slot to the background, one slot for box and red button, and two slots for different parts of the robot arm.
Fig. 19 depicts the effect each action prototype learned by PlaySloton the ButtonPress dataset.
E.3.4. GRIDSHAPPES DATASET
Fig. 20 depicts the effect each action prototype learned by PlaySlot on a variant of GridShapes with three shapes. PlaySlot successfully captures the five possible object movements up, down, left, right, and stay-predicting future frames by modeling the motion of each object independently. However, we observe that PlaySlot sometimes generates artifacts when shapes reach the image boundaries. We attribute this to the training data, where objects change direction upon reaching the boundary mimicking a bouncing effect. This leads to poor prediction performance when conditioning the model to predict outside its training distribution.
E.3.5. SKETCHY DATASET
Fig. 21 shows PlaySlot's predictions and object representations on a Sketchy sequence. We visualize the ground truth sequence, the predicted frames, segmentation obtained by assigning different color to each slot mask, well as the object reconstructions for four slots. PlaySlot assigns two slots to the workspace and background, two slots for each part of the robot gripper, and two slots for different objects present in the scene.
20

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
2
4
10
13
16
Sim. Actions Latent Behavior
e
13
17
21
Sim. Actions Latent Behavior
b)
Figure 12: Predicted frames using latent actions from the learned policy, and sequences simulated by executing the decoded latent actions for two ButtonPress scenarios. a) PlaySlot successfully plans the trajectory within its latent imagination and translates latent actions into executable motions. b) Play fails to decode its latent actions into correct robot motion, missing the button.
21

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
2
3
9
Latent Behavior
Sim. Actions
Latent Behavior
Sim. Actions
10
13
16
Sim. Actions Latent Behavior
Figure 13: Predicted frames using latent actions from the learned policy, and simulation computed by executing the decoded
latent actions for three BlockPush scenarios. a) & b) PlaySlot identifies the block of distinet color and generates executable latent actions push the target location. PlaySlot identifies the block but fails push it into the target location.
22

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
6
9
13
16
PlaySlot
SVG
CADDY
b)
9
PlaySlot
SVG
CADDY
b)
Figure 14: Qualitative comparison on BlockPush. Our method accurately predicts the scene dynamics and object interac-
tions, whereas the baselines fail model object collisions, leading blurry disappearing objects.
23

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
Predictions
Slot Masks
Object
Object
Object
Object 4
Object 5
Object 6
Figure 15: PlaySlot predictions and object representations BlockPush sequence. We visualize the ground truth quence, the predicted frames, segmentation obtained assigning different color to each slot mask, well as the object reconstructions for five slots. PlaySlot assigns one slot to the background, one slot for the robot, one slot for the red target, and the remaining slots to the blocks.
24

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
9
13
16
GT
Inferred Dynamics
"Up & Left' Action
'Right'
Action
'Stay'
Action
"Up & Right'
Action
'Left' Action
'Up' Action
'Left' Action
'Up & Right Action
Figure 16: PlaySlot predictions conditioned on different latent actions, including the inferred inverse dynamics, as well as each action prototype learned on BlockPush. We generate a sequence by repeatedly conditioning the prediction process on a single action prototype. The model learns action prototypes that control the robot to move consistently on a specific direction.
25

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
t=1
9
PlaySlot
SVG
CADDY
Figure 17: Qualitative comparison on ButtonPress. All methods successfully reconstruct the ground truth sequence by inferring the robot's trajectory.
26

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
Predictions
Slot Masks
Object
Object
Object 3
Object 4
Figure 18: PlaySlot predictions and object representations ButtonPress sequence. We visualize the ground truth sequence, the predicted frames, segmentation obtained by assigning different color to each slot mask, well as the object reconstructions for four slots. PlaySlotassigns one slot to the background, one slot for box and red button, and two slots for different parts of the robot arm.
27

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
8
13
16
GT
Inferred Dynamics
Action 1
Action 2
Action 3
Action 4
Action 5
Action 6
Action 7
Action 8
Figure 19: PlaySlot predictions conditioned on different latent actions, including the inferred inverse dynamics, as well as each action prototypes learned on ButtonPress. We generate sequence by repeatedly conditioning the prediction process on a single action prototype. The model learns action prototypes that control the robot to move consistently on a specific direction.
28

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
f
7
9
11
13
Inferred Dynamies
'Up' Action
'Right"
Action
'Down' Action
'Stay" Action
'Left'
Figure 20: PlaySlot predictions conditioned on different latent actions, including the inferred inverse dynamics, well as each action prototypes learned on the GridShapes dataset. We generate a sequence by repeatedly conditioning the prediction process on a single action prototype. The model learns the five possible actions and achieves sharp predictions by forecasting the motion each object individually.
29

---

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
Predictions
Slot Masks
Object
Object
Object
Object 4
Object 5
Object 6
Figure 21: PlaySlot predictions and object representations on Sketchy sequence. We visualize the ground truth sequence, the predicted frames, segmentation obtained by assigning different color to each slot mask, well the object recon- structions for four slots. PlaySlot assigns two slots to the workspace and background, two slots for each part of the robot gripper, and two slots for different objects present in the scene.
30


Reviews:
[{"review_id": "XnVb7ZLquQ", "reviewer": "Reviewer_R5dN", "summary": "The papers proposes PlaySlot - a novel approach to controllable video prediction that builds on previous work on object-centric learning and latent action learning. Unlike previous approaches to video prediction based on object-centric learning, PlaySlot incorporates the InvDyn module for inferring latent actions between observations, which allows PlaySlot to learn in a self-supervised manner, enabling controllable generation (or planning) of multiple possible futures conditioned on different latent actions. In contrast to previous latent action learning approaches (such as CADDY or LAPO), PlaySlot infers object representations as slots and models the dynamics in terms of them via the transformer-based predictor. The authors use SAVi to infer slots and adapt the architecture of forward dynamics and inverse dynamics models to work specifically with slots. Latent actions are factorized into deterministic action prototypes and action variability continuous embeddings. Using simple robotic manipulation tasks (ButtonPress, BlockPush) and a synthetic dataset (GridShapes), the authors empirically demonstrate that PlaySlot outperforms considered baselines in terms of prediction quality. Experiments also show that the resulting latent actions are meaningful and consistent, allowing for sample-efficient fine-tuning with a small number of ground truth actions.\n\n# update after the rebuttal\n\nI was initially reluctant to raise the score much, but during the rebuttal the authors addressed most of my concerns either by providing experimental evidence or by describing planned changes to the text. Although the changes are large and change the claims, I consider them positive. I also saw the seriousness of the authors during the rebuttal, which gives me confidence that all changes will be included in the final version of the paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive review and for finding our proposed approach novel, exciting and potentially useful.\n\nBelow we address your questions and main comments. We will address other comments (e.g. remove exaggerated claims and add missing references) in the final version of the paper. Furthermore, we will reduce the number of qualitative evaluations on the main text, and include the sample-efficient Behavior Learning experiment instead.\n\n\n**Lack of quant. evaluation and baselines on Sketchy**\n\nWe quantitatively evaluate PlaySlot, as well as SVG and CADDY on the real-robot Sketchy dataset. The results are:\n\n| | **PSNR** | **SSIM** | **LPIPS** |\n| --- | --- | --- | --- |\n| **SVG** | *24.24*| *0.813* | 0.076 |\n| **CADDY** | 23.65 | 0.809 | **0.059** |\n| **PlaySlot** | **24.43** | **0.815** | *0.063* |\n\nWe observe that all methods achieve an overall similar performance on this dataset, with PlaySlot slightly outperforming the baselines. We show qualitative comparisons and prediction rollouts for all methods in the following link: https://anonymous.4open.science/r/Rollouts-7F68/ \n\nDespite not outperforming the baselines by a significant margin, these results show that object-centric models like PlaySlot can be applicable for playable video prediction and as a world model on real robot data. In future work, we plan to extend our method with more expressive encoders and decoders (see Appendix A) to apply PlaySlot on more complex real robotic scenarios.\n\n\n\n**Lack of LAPO baseline for behavior learning**\n\nWe are currently working on training and training LAPO on our BlockPush environment. We hope to provide a comparison between PlaySlot and LAPO by the end of the rebuttal period.\n\n\n**Some additional differences with LAPO/LAPA**\n\nLAPO & LAPA are primarily designed and optimized for latent-action pretraining in the context of learning policies for 2D games (LAPO) or robot behaviors (LAPA). However, these models are not designed for autoregressive future prediction, limiting their effectiveness as world models.\n\nIn contrast, PlaySlot incorporates a powerful object-centric world model, which enables the generation of future trajectories conditioned on latent actions.\n\n\n\n**Does ablation on action representations include variants without stochasticity (e.g. continuous part)? Is there really any benefit to sampling?**\n\nOur ablation includes a comparison with an oracle model that does not perform any sampling. However, the remaining PlaySlot variants (hybrid, continuous, discrete) sample latent actions $\\hat{\\mathbf{z}_t}$ from a learned Gaussian.\n\nInspired by literature in stochastic video prediction, e.g. SVG (Denton & Fergus 2018), CADDY (Meanapace et al. 2021) or SLRVP (Franceschi et al. 2020), we use a stochastic formulation to learn a structured latent space that captures uncertainty and enables controllable video prediction.\n\n\n**Were these experiments reported? ',.. information bottleneck enforced by vector quantization achieves comparable performance to the one proposed by CADDY while...'**\n\nNo, these experiments were not included in the paper. We conducted early experiments where both variants performed comparably and decided to stick with our approach. \nThe method proposed by Menapace et al. involves numerous hyper-params (e.g., many loss weights and Gumbel-Softmax temperature scheduling), which impact performance. A poor choice of these hyper-params often leads to suboptimal training runs where multiple latent actions represent the same motion, or where multiple action prototypes did not encode any meaningful representations.\nWe found that tuning the hyper-params for all datasets would be challenging and expensive. Therefore, we favored our approach, which is more practical.\n\n\n**Training and Compute Hyper-parameters**\n\nTo ensure a fair comparison, we balance the number of learnable params and compute requirements for all methods. Below, we report the number of trainable params for each model:\n SVG:  18.84M; CADDY:  9.34M; SlotFormer:  3.97M (2.96M for predictor & 1.01M for SAVi);  OCVP: 4.86M  (3.85M for predictor & 1.01M for SAVi); PlaySlot: 8.49M  (3.19M for InvDyn, 4.27M for cOCVP & 1.01M for SAVi).\n\nRegarding hyper-param tuning, we use the following structured approach:\n - For CADDY and SVG we start with the hyper-params from the BAIR dataset and manually refine them for our datasets.\n - For SlotFormer, OCVP and PlaySlot, we use the default OCVP hyper-params as a baseline and further adjust them for optimal performance on our datasets.\n\nWhile we acknowledge that a more exhaustive hyper-param search could yield better results, conducting extensive searches for all methods was infeasible due to compute budget and hardware constraints.\n\nRegarding the number of random seeds. We run the behavior learning experiments (Appendix E.2) with 3 different seeds and report now the mean success rate and standard deviation. The results on ButtonPress are provided in: https://anonymous.4open.science/r/Rollouts-7F68/", "claims_and_evidence": "There are two main claims in the paper (direct quotes from the text):\n\n> 1. PlaySlot outperforms several video prediction models across diverse robotic environments, while showing superior interpretability and control capabilities.\n> 2. The object representations and latent actions inferred by PlaySlot can be used to learn robot behaviors from unlabeled video demonstrations sample efficiently.\n\nFirstly, it seems to me that the experiments presented do not allow the authors to claim superiority \"across different robotic environments\", as only two simple robotic environments were used, and GridShapes is not in the robotic domain. Furthermore, on ButtonPress PlaySlot does not outperform the baseline SVG (and this is just one of many tasks in MetaWorld, and not the most difficult), and even on BlockPush, which is specifically designed to favour object-centric representations, PlaySlot does not outperform the baseline by much. However, it is difficult to judge here as no standard deviation or confidence intervals were provided. Similarly, for the most interesting benchmark - real-world robot videos (Sketchy) - only PlaySlot results are provided, without the baselines and metrics.\n\nSecondly, there is currently no evidence in the main text that PlaySlot can be used to learn robot behavior in a sample efficient manner, as only a qualitative evaluation is provided. However, there are such experiments in the Appendix, and they are quite convincing, at least for the environments tested. However, they don't include LAPO as a baseline, which I find odd given its large influence on latent action learning research and its overall simplicity. What if, despite its simplicity, it learns latent actions more suitable for efficient fine-tuning even on BlockPush?\n\nThus, I think that the current main claims are not sufficiently supported by the evidence provided. The authors should either provide more evidence (more than one environment for each type, clearly state number of random seeds used, provide metrics of variation, baseline metrics on Sketchy, bring experiments showing sample-efficient fine-tuning into the main text and compare with the LAPO), or remove the exaggerated claims to match the experiments provided. I understand that there may be limitations in computational resources, but this does not justify overclaiming. I still find the proposed approach novel, exciting and potentially useful. I may increase the score if some of the comments are addressed.\n\nSome other minor claims:\n\n> We further show how PlaySlot effectively captures precise robot actions and seamlessly scales to scenes with multiple moving objects or to real-world robotics data, outperforming several stochastic and controllable video prediction baselines\n> \n\nAs noted above, there are no comparisons with baselines on real world robotics data.\n\n> However, both CADDY and Genie operate on holistic scene representations, which are limited for tasks that require relational reasoning, often struggle to model object relationships and interactions, and require human supervision to generalize to scenes with multiple moving agents.\n> \n\nCurrently there are no references in the text which support this claim. \n\n> This approach shares similarities with Schmidt & Jiang (2024). However, whereas their method learns policies for simple games with a small discrete set of actions …\n> \n\nIf I'm interpreting the numbers correctly in the Appendix A.4 from the LAPO paper, they used latent actions of dimension 128, which were then reshaped to 8 discrete ones, each with a codebook size of 64. Which gives a total of 64^8 actions. I wouldn't call that small.", "methods_and_evaluation_criteria": "Despite the fact that the evaluation is limited to a few simple domains, they are fully suitable for potentially verifying the claims of the article. The benchmarks considered include domains that require non-object and object-centred reasoning, as well as real-world robotics data. The experiments also use standard and widely used metrics for video prediction.\n\nHowever, the evaluation itself is questionable. Nowhere in the paper (including the appendix) is it stated how many random seeds were used for the experiments. Results are also reported without standard deviation or confidence intervals (for video prediction experiments as well as for performance after fine-tuning with ground truth actions).\n\nThe hyperparameter search process is not explicitly documented. Was there one, or did all methods use the default hyperparameters? If so, was it explicitly considered that all methods should use the same enumeration budget? Were the methods explicitly balanced in the number of weights and gradient updates? I think the authors should state this explicitly and include a table with the number of trainable weights for each method.", "theoretical_claims": "The paper does not present any new theoretical results or claims. After careful examination, I do not see any flaws or errors in the formulas used to present the approaches, e.g. SlotAttention and other components of the PlaySlot pipeline.", "experimental_designs_or_analyses": "Apart from the concerns discussed in 'Methods and evaluation criteria', I think the only concern with experimental design is too much focus on qualitative comparisons.\n\nFirst, it's pretty hard to tell if it's not cherry picking, which is quite possible even without explicit intent. Second, it's quite repetitive, better to demonstrate it once and use the remaining space for more valuable experiments (e.g. sample-efficient fine-tuning with ground truth actions).\n\nFinally, while visual quality is important for video generation, it is less important for world modelling and planning (unless the goal is to train policies in image space rather than compact latent space, but this is rare). A world model can be globally inaccurate and still be better for planning and downstream performance than a more accurate one (see [1]).\n\nTherefore, to support the claim that PlaySlot captures precise robot actions useful for planning and fine-tuning, which is what most practitioners would value, it's necessary to include fine-tuning experiments in the main part of the paper.\n\nAs for the ablations, I don't have any concerns. I think they show the right things and actually improve the understanding of the proposed method.\n\nReferences:\n\n1. Lambert, N., Amos, B., Yadan, O., & Calandra, R. (2020). Objective mismatch in model-based reinforcement learning. *arXiv preprint arXiv:2002.04523*.", "relation_to_broader_scientific_literature": "PlaySlot is a novel combination of object-centric learning, i.e. SAVi, and latent action learning, pioneered by the seminal work on Imitating Latent Policies from Observation (ILPO) [1], and recently significantly improved by LAPO [2] and LAPA [3]. PlaySlot, on the other hand, borrows many design decisions from CADDY [4]. However, the empirical evidence provided is rather weak and does not allow to conclude that the PlaySlot approach is definitely a better way for latent action learning, e.g. compared to LAPA, which also uses the robotics domain. Without such evidence, the main contribution is rather incremental.\n\nReferences:\n\n1. Edwards, A., Sahni, H., Schroecker, Y., & Isbell, C. (2019, May). Imitating latent policies from observation. In *International conference on machine learning* (pp. 1755-1763). PMLR.\n2. Schmidt, D., & Jiang, M. (2023). Learning to act without actions. arXiv preprint arXiv:2312.10812.\n3. Ye, S., Jang, J., Jeon, B., Joo, S., Yang, J., Peng, B., ... & Seo, M. (2024). Latent action pretraining from videos. *arXiv preprint arXiv:2410.11758*.\n4. Menapace, W., Lathuiliere, S., Tulyakov, S., Siarohin, A., & Ricci, E. (2021). Playable video generation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (pp. 10061-10070).", "essential_references_not_discussed": "To my knowledge, paper cites all the relevant papers essential for understanding the context and key contributions.", "other_strengths_and_weaknesses": "Strengths:\n\n- The paper is clearly written and presents the method in a way that's easy to understand.\n- The method is novel in the sense that, to my knowledge, there is no other work that combines object-centred representations with latent action learning.\n- The related work section provides all the necessary context and does not overlook important papers in both fields.\n\nWeaknesses:\n\n- As discussed in the sections above, the empirical evidence is weak.\n- The evaluation protocol is not clearly explained. Information on number of random seeds, standard deviations or confidence intervals, hyperparameter search is not provided.", "other_comments_or_suggestions": "I do not have any other comments or suggestions.", "questions_for_authors": "1. Does ablation on action representations include variants without sampling/stochasticity (e.g. continuous part)? Is there really any benefit to sampling? LAPO claims that it can capture stochasticity, and they do not sample latent actions.\n2. Where these experiments reported?\n    \n    > We empirically verify that the information bottleneck enforced by vector quantization achieves comparable performance to the one proposed by (Menapace et al., 2021), while requiring significantly fewer hyper-parameters.\n    >", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EZ9OsFeCEL", "reviewer": "Reviewer_Cnpu", "summary": "This work introduces a novel approach for the video prediction task using object-centric representations. It proposes an InvDyn module to learn latent action embeddings and a conditional object-centric predictor to forecast future object slots. Extensive experimental results demonstrate superior performance compared to previous methods, including evaluations on a real-robot video dataset.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the positive review, acknowledging our experimental design, highlighting that our method is novel and interesting, and that our paper is well-written.\nBelow we address some of your questions and add clarifications about certain weaknesses: \n\n**Clarification about the lower performance on ButtonPress.**\n\nThe Metaworld Button-Press task is inherently non-object-centric, i.e., there is no need to reason about object properties and interactions to press the button. Therefore, no large gains can be obtained from using a structured object-centric representation of the scene.\nNevertheless, we demonstrate that our PlaySlot model is general and still remains competitive in non-object-centric scenarios, ranking second among the evaluated methods on ButtonPress.\n\n\n**What are the key differences between the proposed InvDyn module and Menapace et al. (2021)?**\n\nThere are two main differences:\n - *Scene representation*: CADDY models scene dynamics using a holistic representation, while our InvDyn model leverages object slot representations, enabling finer-grained object-centric reasoning.\n\n - *Action Parameterization*: Both methods represent actions differently, leading to distinct modeling choices.\n\nA more detailed discussion of these differences is provided in the Appendix D.2.\n\n\n**About inputs provided by a human to PlaySlot**\n\nPlaySlot parameterizes an action with a discrete action prototype, which represents the action taking place, and an action variability embedding, which allows to interpolate between different motions or modify the action prototypes (e.g. changes in speed or direction).\nTo allow a user to control PlaySlot, we first qualitatively measure the effect of each learned action prototype and then assign a semantic meaning (e.g. ‘move left’) to each action prototype according to the motion it represents.\nAt inference time, a user can specify actions by selecting the corresponding action prototypes, while setting the action variability embedding as zero.\n\n\n**How can this approach generalize to more real-world domains and complex scenarios? I believe this remains the main bottleneck for slot-based methods.**\nRecent works, such as DINOSAUR (Seitzer et al. ICLR 2023) or SlotDiffusion (Wu et al NeurIPS 2023), show that slot-based models can be extended to handle more realistic data by using large self-supervised pretrained encoders, or powerful diffusion-based decoders.\nAs discussed in the Limitations & Future Work section (Appendix A), we plan to explore in future work these architectural designs to learn robot behaviors in more complex robotic scenarios", "claims_and_evidence": "Yes, the results strongly support the claims.", "methods_and_evaluation_criteria": "The method is well-founded, and the evaluation follows standard practices.", "theoretical_claims": "No theoretical claims are proposed in this work.", "experimental_designs_or_analyses": "The experimental design is well-structured, but it lacks an analysis of the low performance on the ButtonPress dataset.", "relation_to_broader_scientific_literature": "This work presents an improvement in video prediction using object-centric learning.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. The method is novel and interesting.\n2. The experiments are well-designed and robust.\n3. The paper is well-written.\n\nWeaknesses:\n1. Clarification on ButtonPress performance: In Table 1, the lower performance on the ButtonPress task requires further clarification.\nComparison with prior work: In Section 3.2, what are the key differences between the proposed InvDyn module and Menapace et al. (2021)?\n2. Handling multiple input types during inference: In Figure 1, can multiple input types be combined during inference? I understand the other two types, but what would a human input look like?\n3. What format does it take?\n4. How is it processed, considering that the outputs from the InvDyn module or a learned action policy are probabilistic and implicit, whereas human inputs (e.g., \"move up\" or \"move down\") are explicit, discrete, and symbolic?", "other_comments_or_suggestions": "No.", "questions_for_authors": "Overall, this is a solid paper. However, I have one question that many researchers in this field often face: beyond the relatively simple block-based scenarios (even on the Sketchy dataset), how can this approach generalize to more real-world domains and complex scenarios? I believe this remains the main bottleneck for slot-based methods.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RiEkp2G5tI", "reviewer": "Reviewer_19cQ", "summary": "This paper introduces PlaySlot, an object-centric video prediction model that learns inverse latent dynamics for controllable future frame forecasting and can be used in downstream tasks. PlaySlot infers object representations and latent actions from unlabeled video sequences instead of action annotations, leveraging these representations to predict possible futures. To achieve that, the model integrates an inverse dynamics module to capture scene dynamics and a conditional object-centric predictor for forecasting. Through experiments on various environments, the authors demonstrate that PlaySlot outperforms both stochastic and object-centric baselines in video prediction accuracy, while enabling robot behavior learning from demonstrations.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for  the  constructive comments. We are delighted that you consider our experimental section and model analysis as a strength.\nBelow we address the highlighted weaknesses and questions, and clarify some misunderstandings.\n\n**The action decoder still requires ground-truth actions for training, which contradicts the claim of unsupervised learning of robot behaviors from unlabeled demonstrations.**\n\nWe respectfully disagree with this statement and would like to provide further clarification.\n\nThe action decoder is only needed to decode and execute latent actions in the simulator, both for qualitative evaluations (e.g. Fig. 7 ‘Sim. Actions’) and quantitative evaluation (i.e. measuring success rate as in Appendix E.2.).\nNevertheless, PlaySlot learns robot behaviors in a completely unsupervised manner within its latent imagination. The policy model is trained in isolation to predict latent actions inferred by the inverse dynamics model from unsupervised expert demonstrations, without relying on ground-truth actions entirely.\n\nIn summary, PlaySlot learns robot behaviors and to infer inverse dynamics from unlabelled demonstrations in an unsupervised manner, and the ground-truth labels are solely needed to train an action decoder for evaluation purposes only.\n\n\n**Regarding potential information leakage when using future frames.**\n\nWe follow the same evaluation protocol as CADDY, where the model is first used to infer latent actions using future frames, and then it uses those same latent vectors to forecast/reconstruct the future video frames given the inferred latent actions and a small number of seed frames.\nThis protocol is consistently applied across all controllable and stochastic models (CADDY, SVG, and PlaySlot), ensuring a fair comparison.\n\nFurthermore, we mitigate information leakage by enforcing an information bottleneck on the latent actions, ensuring they capture only scene dynamics rather than the full target scene state. This prevents the model from directly encoding future frame details into the latent actions.\n\n\n**Equation 7: Can you explain why the latent action is the difference between two consecutive embeddings?**\n\nThe latent action is modeled as the difference between two consecutive scene dynamics’  embeddings, thus capturing the state transformation between frames.\n\nSince each embedding independently encodes information from its respective frame, their difference isolates the temporal change, which corresponds to the performed action.\n\nAdditionally, we assume that action embeddings are Gaussian-distributed and independent, ensuring that their difference also follows a Gaussian distribution, facilitating a probabilistic formulation for action prediction.\n\n\n\n**How is the meaning of latent actions mapped to numbers as shown in Figs 5 & 6?**\n\nThe numbers correspond to the action prototype that represents such movement. We simply add a text-label describing the action (e.g. ‘move up’) to make visualizations clearer. \n\n\n\n**Since the model uses a pretrained SAVi, does each frame’s representation solely encode visual information from that frame? If so, would two consecutive frames be insufficient to infer an action?**\n\nYes, each frame’s representation encodes information from that frame alone. Therefore, we need our Inverse Dynamics module to infer a latent action that encodes the scene dynamics between each pair of consecutive frames.\nWe indeed only use two consecutive frames to infer the latent action executed between them.\n\n\n**Weakness: GridShapes seems visually too simple.**\n\nGridShapes is indeed a visually simple dataset. However, to correctly solve this task, a method must be able to jointly model the motion of multiple moving objects.\nWe find out (Fig. 4) that PlaySlot successfully models the motion of multiple (e.g. 5) objects, whereas baselines such as CADDY fail for more than two objects despite the visual simplicity of the dataset.\nThis highlights the potential of object-centric world models for dynamic scenarios with multiple moving agents.\n\nIn the following link, we show qualitative rollouts on GridShapes with 2 & 3 objects. We can see how the CADDY baseline fails to jointly model the motion of 3 shapes, whereas PlaySlot solves the task.\n - https://anonymous.4open.science/r/Rollouts-7F68/ \n\n\n**Weakness: Expert data is required for behavior learning.**\n\nSimilar to latent action models, e.g. Genie (Bruce et al. ICML 2024) or LAPO (Schmidt & Jiang. ICLR 2024), PlaySlot indeed requires expert demonstrations to learn robot behaviors.\nHowever, as shown in Appendix E.2, we demonstrated that PlaySlot can learn behaviors from unlabelled video demonstrations (without access to simulators and without action information) in a sample efficient manner.\nTo make this clearer, we will include the experiments showing sample-efficient behavior learning into the main text for the final version.", "claims_and_evidence": "Although the authors claim that the inferred latent actions enable sample-efficient learning of robot behaviors from unlabeled video demonstrations, the action decoder still requires ground-truth actions for training, which contradicts the claim of a fully unsupervised approach.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "In the comparison presented in Table 1, PlaySlot infers latent actions using future frames before utilizing them for predicting subsequent frames, which raises concerns about potential information leakage. This approach may grant an advantage over baseline methods that do not have access to future frames, calling into question the fairness of the comparison.", "relation_to_broader_scientific_literature": "This paper builds on the work in object-centric learning, particularly methods that decompose visual scenes into structured representations using slot-based models such as SaVi. Additionally, it relates to object-centric video prediction, which has been explored in works such as SlotFormer and OCVP, but differs by emphasizing controllability through inferred actions.  PlaySlot extends these approaches by incorporating inverse latent dynamics, drawing inspiration from prior research in learning latent actions from unlabeled videos.", "essential_references_not_discussed": "As far as I am concerned, the paper discusses the most relevant prior work.", "other_strengths_and_weaknesses": "### Strength\n- The authors provide extensive experiments and model analysis.\n- The authors explore action-conditioned object-centric video prediction.\n\n### Weakness\n- The main weaknesses are listed in the previous sections.\n- While the paper claims novelty in combining object-centric video prediction with latent action models, both object-centric prediction and latent action learning in conditional video prediction have been previously explored.\n- Expert data is required for behavior learning.\n- The GridShapes environment seems visually too simple.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Equation 7: Can you explain why the latent action is the difference between two consecutive embeddings?\n2. How is the meaning of latent actions mapped to action numbers, as shown in Figures 5 and 6?\n3. Since the model uses a pretrained SAVi, does each frame’s representation solely encode visual information from that frame? If so, would two consecutive frames be insufficient to infer an action?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F63FUWxvYk", "reviewer": "Reviewer_nim7", "summary": "The paper tackles the problem of learning controllable object-centric video prediction from actionless videos. The proposed method, PlaySlot, combines slot-attention-based object-centric representations with a previous slot dynamics prediction module (OCVP) conditioned on learned latent actions (inverse module). A latent policy is also learned to infer latent actions at inference time. The model is evaluated on robotic datasets, two simulated and one real-world (Sketchy).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive review. We are happy that you find our claims clear and our experimental design sound.\nFor the final version, we will try to improve the description of the latent action modules and we will rename the subsection to “Unsupervised Slot-based Object-centric Learning”.\nBelow we address your specific questions and comments:\n\n**Predictions on Sketchy don’t seem very “stable”. Is there an explanation for that?**\n\nThis is indeed an interesting observation. We attribute this behavior to two main factors:\n - *Simplicity of rendering module*: PlaySlot (following SAVi) uses a shared simple CNN to decode slots into object images and masks, which only interact with each other via a weighted sum. We hypothesize that a more expressive rendering module that jointly decodes all slots, e.g. transformer-based or diffusion-based, could overcome this limitation.\n\n - *Error-Accumulation*: Our object-centric predictor is an autoregressive transformer, which can be susceptible to error accumulation over long prediction horizons. Since each predicted frame conditions the next, small inaccuracies in early steps can propagate and amplify, leading to noticeable object drift or flickering. One interesting future research direction would be to introduce temporal regularizations to minimize this effect.\n\nNevertheless, we observe that baselines such as SVG or CADDY also suffer from lack of temporal consistency on Sketchy: https://anonymous.4open.science/r/Rollouts-7F68/ \n\n\n**Is the sampled Gaussian latent action quantized? How is the $\\mathbf{v}$ modeled? Is it the residual?**\n\nThe latent actions $\\mathbf{\\hat{z}}$ are indeed sampled from a learned Gaussian distribution.\nAdditionally, we decompose each latent action into an action prototype $\\mathbf{p}$ and a variability embedding $\\mathbf{v}$. This hybrid parameterization helps us learn semantically meaningful action prototypes, while also being able to model more complex actions (e.g. interpolations or modifications) by using the variability embedding.\nNamely:\n - $\\mathbf{p}$ is obtained by vector quantizing the latent action: $\\textbf{p} = VQ(\\mathbf{\\hat{z}})$\n - The variability $\\mathbf{v}$ is the residual: $\\mathbf{v} = \\mathbf{\\hat{z}} - \\mathbf{p}$\n\nThese specifics on how $\\textbf{p}$ and $\\textbf{v}$ are computed are described in Sec. 3.2.2 Lines 187-192-Col.2, but we will make this also explicit in Sec. 3.2.1. \n\n\n**Use of real actions to train the action-decoder:**\n\nPlaySlot learns to infer latent actions and learns robot behaviors in a completely unsupervised manner from actionless videos.\nAn action decoder is trained to map latent actions to real-actions, but this module is only needed to execute latent actions in the simulato for evaluation. See detailed answer to reviewer 19cQ.\n\nWe will try to make this clearer in the final version of the paper.\n\n\n**How does one choose between the inverse model variants?**\nIf we know that there is a single agent in the scene (e.g. tabletop robotic scenarios), we use the InvDynS variant.\nWhen multiple objects (> 2) are moving, we show that using a single latent action is not powerful enough, and explicitly modelling the action of each agent/object using InvDynM proves to be beneficial.\nTherefore, we use InvDynS on the robotic scenarios, and InvDynM on GridShapes.\n\n\n**No quantitative results on the real-world Sketchy dataset or comparison with baselines on that dataset.**\nWe now include a quantitative evaluation of PlaySlot and baselines on Sketchy.\nSee detailed answer to review R5dN. \n\nAdditionally, we provide videos in the following link featuring qualitative comparisons with the baselines on Sketchy and all other datasets: https://anonymous.4open.science/r/Rollouts-7F68/ \n\n\n**Marginal improvement over baselines**\n\nWhile pixel-wise metrics are indeed close in some cases, our method still demonstrates notable advantages. In particular, on BlockPush, PlaySlot achieves the best results, highlighting its ability to model object dynamics and interactions effectively. Furthermore, PlaySlot achieves the overall best performance on Sketchy (see table in response to reviewer R5dN) and remains competitive in non-object-centric scenarios such as ButtonPress.\n\nIt is important to note that BlockPush contains object interactions in every test scene, where the robot must push a block to a target location. Among the compared methods, only PlaySlot consistently captures these interactions and generates coherent sequence predictions.\n\nMoreover, while small object sizes in BlockPush may limit the impact of object localization errors on quantitative video metrics, this does not diminish the benefits of object-centric decomposition.\n\nAdditionally, beyond immediate gains in pixel-based scores, our InvDyn module, in combination with a structured slot-based representation, enables sample-efficient behavior learning, as detailed in App. E.2. This makes PlaySlot particularly valuable in settings where data efficiency is crucial.", "claims_and_evidence": "* Claims are clear – designing a controllable video prediction model.\n* Evidence: very marginal improvement over baselines (except the simplistic GridShapes dataset), especially over the “holistic” baselines (like CADDY). It doesn’t seem that the object-centric decomposition provides significant gains, except for certain cases where the “holistic” approaches don’t model object-interactions, but these cases seem rare as evident by the close pixel-metrics (also, hard to tell without video rollouts of the baselines how severe the issue is). \n* Evidence: no quantitative results on the real-world Sketchy dataset or comparison with baselines on that dataset.\n* Evidence: in the provided video rollouts, the predictions (e.g., on Sketchy) don’t seem very “stable”, objects randomly move/wiggle There are no video rollouts of the baselines.", "methods_and_evaluation_criteria": "* The proposed method and idea make sense for the problem; however, the method is not entirely clear.\n* The benchmarks make sense, but they are too simple to draw a conclusion regarding the performance of the proposed method.", "theoretical_claims": "There are no theoretical claims in this paper.", "experimental_designs_or_analyses": "* The experimental design is sound.\n* There are ablations.\n* I raised some questions under the questions section and had several concerns under “Claims and Evidence”.", "relation_to_broader_scientific_literature": "* The key contribution of this paper is complimenting a slot-based object-centric video-prediction model with a latent action module, which enables handling stochastic transitions (e.g., an action performed by a robot).\n* In that sense, there is a contribution to the community; however, the chosen benchmark includes simple simulated datasets and one real-world dataset (with no comparisons to baselines). \n* The performance of the method is not significant with respect to the baselines.", "essential_references_not_discussed": "The paragraph “Unsupervised Object-Centric Learning” under the Related Work section should be renamed to “Unsupervised Slot-based Object-centric Learning” as it completely ignores other unsupervised OC families, such as the patch-based family (e.g., SPACE-https://arxiv.org/abs/2001.02407) and the particle-based family (e.g., DLP-https://arxiv.org/abs/2205.15821). The other approaches don’t use the term “slots”, so if you decided to ignore the previous work in unsupervised OC, use the correct terms. While it is true that the object-centric community is mostly focused on slot-based models, I find it a bad habit to ignore the other types of object-centric models.", "other_strengths_and_weaknesses": "**Strengths**:\n* Open-source code.\n\n**Weaknesses**:\n* The paper is hard to follow and includes obscure descriptions of the components (especially regarding the latent actions modules).\n* No video rollouts of the baselines.", "other_comments_or_suggestions": "Add a reference in the main text to limitations and future work in the appendix.", "questions_for_authors": "* In the provided video rollouts, the predictions (e.g., on Sketchy) don’t seem very “stable”, objects randomly move/wiggle. Is there an explanation for that? Maybe the latent action and slots are not entirely disentangled (I would expect static objects to remain static unless interacted with).\n* Latent action design choices: to my understanding, the sampled Gaussian latent action is quantized? I find this an odd design choice. How is $v_t$, the “action variability embedding” modeled? This information is missing in Section 3.2.1 (or have I missed it?). According to lines 244-251, at inference, $v_t = \\hat{z}_t - p_t$. Is that what happens at train time as well? $v_t$ is the residual?\n* I thought the whole-point was to learn from actionless videos, but it seems that you regress the ground-truth real actions with an action-decoder? This information is obscured from the reader until Section 3.5. So does this approach require labeled actions? Or is it used for evaluation or task-related (Section 4.4) purposes only? It is unclear from the text how the latent actions are learned, and what is the use of the action decoder that requires labeled real actions?\n* How does one choose between the inverse model variants? Is it just trial-and-error? Shouldn’t the M variant work-well for all cases?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received overall positive reviews, with reviewers finding the proposed PlaySlot framework to be a novel and meaningful contribution, combining object-centric video prediction with latent action learning in a way that enables controllable future forecasting from actionless videos. Several strengths were noted, including the clear writing and thoughtful integration of inverse dynamics with slot-based representations. Concerns were raised about limited evaluations (e.g., only simple robotic environments, missing baselines on real-world data), but the authors addressed these in the rebuttal and committed to revising the claims to better reflect the scope of the experiments.

The AC agrees with reviewers that PlaySlot represents a valuable step forward in controllable video prediction. The authors are encouraged to clarify evaluation details, moderate the paper's claims, and incorporate the discussion from the discussion phase into the camera-ready version.
Decision: Accept (poster)