TextOCVP: Object-Centric Video Prediction with Language Guidance
Authors: Angel Villar-Corrales
Gjergj Plepi
Sven Behnke
Venue: Accepted by TMLR ()

Abstract: Understanding and forecasting future scene states is critical for autonomous agents to plan and act effectively in complex environments. Object-centric models, with structured latent spaces, have shown promise in modeling object dynamics and interactions in order to predict future scene states, but often struggle to scale beyond simple synthetic datasets and to integrate external guidance, limiting their applicability in robotic environments. To address these limitations, we propose TextOCVP, an object-centric model for video prediction guided by textual descriptions. TextOCVP parses an observed scene into object representations, called slots, and utilizes a text-conditioned transformer predictor to forecast future object states and video frames. Our approach jointly models object dynamics and interactions while incorporating textual guidance, enabling accurate and controllable predictions. TextOCVP’s structured latent space offers a more precise control of the forecasting process, outperforming several video prediction baselines on two datasets. Additionally, we show that structured object-centric representations provide superior robustness to novel scene configurations, as well as improved controllability and interpretability, enabling more precise and understandable predictions.

Published in Transactions on Machine Learning Research (02/2026)
TextOCVP: Object-Centric Video Prediction with Language Guidance
Angel Villar- Corrales
villar@ais. mni-born,ee de
Autonomous Intelligent Systems group, Lamarr Institute for Machine Learning and Artificial Intelligence and Center for Robotics, University of Bonn, Germaney
Gjergi Plepi!
plepi@ais. uni-boren, de
Autonomous Intelligend Systems group, Lamarr Insttuute for Machine Learning and Artificial Intelligence and Center for Robotics, University of Bonn, Germancy
Sven Behnke
behnke@uni-bone de
Autonomous Intelligent Systems group. Lamarr Institute for Machine Learning and Artificial Intelligence and Center for Robolics, Universily of Bonn, Germony
Reviewed on OpenReview:
https: openreveew. net/ forum? Xd:.
Abstract
Understanding and forecasting future scene states is critical for autonomous agents to plan and act effectively in complex environments. Object-centric models, with structured latent spaces, have shown promise in modeling object dynamics and interactions in order to pre- dict future scene states, but often struggle to scale beyond simple synthetic datasets and to integrate external guidance, limiting their applicability robotic environments. To ad- dress these limitations, we propose TextOCVP, an object-centric model for video prediction guided by textual descriptions. TextOCVP parses an observed scene into object represen- tations, called slots, and utilizes text-conditioned transformer predictor to forecast future object states and video frames. Our approach jointly models object dynamics and interac- tions while incorporating textual guidance, enabling accurate and controllable predictions. Tex1OCVP's structured latent space offers more precise control of the forecasting process, outperforming several video prediction baselines on two datasets. Additionally, we show that structured object-centric representations provide superior robustness to novel scene config- urations, as well improved controllaiility and interpretability, enabling more precise and understandable predictions. Videos and code available on our project website
1 Introduction
Understanding and reasoning about the environment is essential for enabling autonomous systems to better comprehend their surroundings, predict future events, and adapt their actions accordingly. Humans achieve those capabilitics by perceiving the environment as a structured composition of individual objects that interact and evolve dynamically over time (Kahneman et al. 1992). Neural networks equipped with such compositional inductive biases have shown the ability to learn structured object-centric representations of the world, which enable desirable properties, such out-of-distrribttion generalization (Dittadi et al. 2022 compositionality (Greff et al. 2020), sample efficiency (Mosbach et al. 2025).
Recent advances in unsupervised object-centric representation learning have progressed from extracting object representations synthetic images (Locatello et al.) 20200 Lin et al. 2020 to modeling objects in
video Kipf et al. 2022 Singh et al. 2022 and scaling to real-world scenes (Seitzer et al. 2023). These
denotes equal contribution.

---

Published in Transactions on Machine Learning Research (02/2026)
a) Object-Centric Parsing
b) Text-Conditioned Video Prediction
.ut the green block, in the cyan bowl.
X₂
X
X T
1
S
Learned Objects
S
Scene
TextOcVP
Parsing
1
.Put the green block;
1. in _e_the__e _te_reeee_____eee bowl._"
Figure 1: Overview of TeXtOCVP. (a) Our model parses reference frame X₁ into its object components S1. (b) Our TeXPOCVP predictor jointly models object dynamics and interactions guided by text, generating future object states and frames that align with the provided textual instructions.
developments have enabled object-level dynamics modeling for future prediction and planning, Notably, approaches like SlotFormer (Wu et al., 2023) or OCVP Villar-Corrales et al., 2023) introduced object- centric prediction models that explicitly model spatio-temporal relationships between objects, shifting away from image-level approaches that ignore scene compositionality. Despite these advancements, current object- centric methods struggle with complex object appearances and dynamics, and lack mechanisms to incorporate external guidance, thus limiting their scalability and broader applicability.
To address these challenges, propose Tex1OCVP, novel object-centric model for video prediction guided by lextual instructions, illustrated in Fig. 1 Given reference image and text instruction, TextOCVP extracts object representations and predicts their evolution using a text-conditioned object-centric trans- former. This prodictor forecasts future object states by explicitly modeling their dynamics and interactions over time, while integrating textual information via lext-lo-slot atlention mechanism. By jointly modeling temporall object relationships and incorporating textual guidance, TextOCVP predicts future object states and frames aligned with the input instruction.
We evaluate approach for lext-condittoned video prediction through extensive experiments on two dis- tinet datasets. Our results show that TextOCVP outperforms several other text-conditioned video prediction methods by effectively leveraging structured object-centric representations, particularly on scenes featuring multiple moving objects. Furthermore, we conduct an in-depth analysis OCxtO verify the effective- ness of object-centric representations for text-guided video prediction. Specifically, we demonstrate that its structured latent space enables accurate and controllable video prediction by aligning language instructions with the corresponding objects, outperforming baselines that rely on holistic scene representations. Beyond generation quality, TextOCVP offers improved interpretability and exhibits strong robustness to novel scene configurations, including varying numbers of objects or previously unseen colors.
In summary, our contributions are as follows:
We propose TextOCVP, a text-guided video prediction model, featuring a text-conditioned object- centric predictor that integrates textual guidance into the prediction process via a text-lo-slot at- tention mechanism.
Through extensive evaluations, we show that TextOCVP outperforms other existing text-condittoned vidco prediction models by leveraging object-centric representations.
We demonstrate that TexLOCVP is controllable, seamlessly adapting to diverse textual instructions, while exhibiting interpretability and robustness to novel setups.
2 Related Work
2.1 Object- Cenett-C Learning
Representation learning the ability to extract meaningful features from data- often improves model perfor- mance by enhancing its understanding of the input space (Bengio et al. 2013 Object-centric representation
2

---

Published in Transactions on Machine Learning Research (02/2026)
methods aim to parse an image video into set of object components in an unsupervised manner. These
objects are typically represented as unconstrained embeddings (called slots) (Locatello et al. 2020 Kipl
et al.
2022
Singh et al. 2022), patch-based representations (Lin et al. 2020), factored latent vectors Grefft
ict al.
2019 or object prototypes (Villar-Corrles & Behnke 2022 Monnier ct al.
2020 These methods
have demonstrated promise in learning object representations across diverse domains, ranging from synthetic
images (Locatello al. 2020 Lin et al., 2020) videos (Kipf et al.
2022 Singh et al., 2022), and real-world
scenes (Seitzer et al. 2023 Zadaianchuk al. 2024). The learned object representations benefit downstream
tasks, such as reinforcement learning (Mosbach et al. 2025 or visual-question answering (Wu et al. 2023
2.2 Video Prediction
Video prediction (VP) is the task of forecasting the upcoming T video frames conditioned on the preceding C' Several methods have been proposed to address this task, using 2D convolu- 2018). recurrent neural networks Vilar-Corrales et al.] 2022 Wang et al. 2022. transformers (Rakhimov
seed frames (Oprea et al. 2020
tions
(Gao et al. 2022
Chiu et al. 2020). 3D convolutions (Tulyakov et al.
(RNN8) (Denton& & Fergus
2018
et al.
2021
Ye & Bilodeau
2022 or diffusion models HHppe et al. 2022
Ho et al. 2022
2.2.1 Object-
Video Prediction
Object-centric VP presents structured approach that explicitly models the dynamics and interactions of individual objects to forecast future video frames. These methods typically involve three main steps: parsing seed frames into object representations, predicting future object states using a dynamics model, and rendering video frames from the predicted object representations. Various approaches have addressed this task using different architectural priors, such as RNNs (Creswell et al. 2021 Nguyen et al. 2024) or transformers (Wu et al., 2021 2023; Villar-
et al.) 2023 Daniel & Tamar 2024). Despite promising results, these models are limited to simple deterministic datasets or rely on action-condin notion-conditioning (Mosbach et al. 2025 Villar-Corrales & Behnke 2025). In contrast, model forecasts future frames conditioned past object slots and lext descriptions.
2.2.2 Text-Conditioned Video Prediction
Text-conditioned VP models leverage text descriptions to provide appearance, motion and action cues that guide the generation of future frames. This task was first proposed by Hu et al. (2022), who utilized a VQ-VAE to encode images into visual token representations, and modeled the scene dynamics with axial transformer to jointly process visual tokens with text descriptions. Similarly, approaches like TVP (Song
ct al. 2024) and MMVG (Fu et al. 2023) address this task using RNNs or masked transformers, respectively.
More recentlyy several methods leverage diffusion models for text-guided VP (Blaltmann et al. 2023 Ni el al.,
2023 Chen et al. 2024 Xing et al., 2024 Chen et al. 2023 Gu et al. 2024). These approaches encode video frames into discrete token sequences via pretrained quantized autoencoders, and leverage pretrained language models to guide the diffusion-based generation with text features. To further improve temporal coherence and semantic alignment, several diffuusio--bssed hods leverage specialized conditioning strategies {Xing)
ett al. 2024 Chen et al., 2023) or attention mechanisms (Gu et al. 2024), among others. While effective, these models operate on holistic or spatial representations, and require large-scale compute and data. In contrast, TextOCVP adopts a more structured and efficient approach, explicitly modeling object dynamics using slot representations, where each object in the scene represented by a distinct embedding.
Concurrently with our work, Wang et al. (025 and Jeong et al. (2025) also combine object-centric learning with autoregressive diffusion models and transformers for text-guided video prediction simple synthetic datasets. In contrast, we evaluate on more complex robotic simulations, and perform an in-depth analysis of the properties of object-centric representations for text-conditioned video prediction.
3 Method
We propose TextOCVP, a novel object-centric model for sext-conditioned video prediction Given an initial reference image X, and text caption C, OCVP generates the T subsoquent video frames X which maintain a similar appearance and structural composition as the reference image, and follow the motion described in the text caption.
3

---

Published in Transactions on Machine Learning Research (02/2026)
X
X
XT+I
Video Render
Video Render
Video Render
S
S
Sr+t
Text-Conditionel Object-Centric Centric Prediction
S
S2
S
Text Encoder
Scene Parsing
Object Representations
ST
Put the gray) block in the brown boul'""
X1
Figure 2: Overview of TextOCVP. Our model parses the reference frame X, into object representations S1. The text-conditioned object-centric predictor models object dynamics, incorporating information from the description to predict future object states S2.+11, which can decoded into video frames X2:+1-
TextOCVP, which is illustrated in Fig. implements object-centric approach, in which the reference frame X₃ is first decomposed with scene parsing module (Sec. 3.1 into set of Ns D-dimensional object representations called slots $1 E NaxD where each slot represents single object in the image. The object slots are fed to text-conditioned transformer predictor (Sec. [S2c. which jointly models their spatio-temporal relations, and incorporates the textual information from the caption as guidance for predicting the future object slots S2.7+1. Finally, the predicted slots are decoded to render future video frames (Sec. 3.3
We propose two different TextOCVP variants, which differ in the underlying object-centric decomposition modules. Specifically, TexLOCVPsav. leverages SAVi (Kipl al., 2022), whereas Tex1OCVPprNo extends DINOSAUR (Seitzer al. 2023 for recursive object-centric video decomposition and video rendering.
3.1 Scene Parsing
The scene parsing module decomposes video sequence X1 into set of cormutationiivaaiinn object representations called slots S = S,), with S, = (s)...., s,,...,,,,,,,, NN*,, where each slot E Rº represents a single object. For scene parsing, we adopt the recursive object-centric video decomposition framework from Kipf et al. (2022).
At time step t, the corresponding input frame X, encoded with feature extractor module into set of Di-dimensional feature maps h, RLXD; representing L spatial locations. The feature extractor is volutional neural network in our TexLOCVPsav variant and DINO-pretrained vision transformer (Caron) et al. 2021 in TextOCVP These feature maps are processed with Slot Attention (Locatello et al, 2020), which updates the previous slots S,-1 based on visual features from the current frame following an iterative attention mechanism. Namely, Slot Attention performs cross-attention, with the attention weights normalized over the slot. dimension, thus encouraging the slots to compete to represent parts of the input. It then updates the slots using a Gated Recurrent Unit (Cho al. 2014] (GRU). Formally, Slot Attention updates the previous slots S, by:
q(S: ((S:-1)k(h))
A softmax N'
NaxL
(1)
VD
4

---

Published in Transactions on Machine Learning Research (02/2026)
Adj Ei=1 A
S: GRU(W. S.-1) with Wia
where k, q and v are learned linear projections that map input features and slots into common dimension. The output of this module is set of slots S, that represents the objects of the input frame.
3.2 Text-Conditioned Object-Centric Predictor
Our proposed text-conditioned predictor module, il- lustrated in Fig. 3 autoregressively forecasts future object states conditioned on the object slots from the reference frame S and text description C. This design enables TextOCVP to predict temporally con- sistent object dynamics that are grounded not only in visual context, but also in high-level semantic intent specified via natural language.
S2
xNp
MLP
Layer Norm
To condition the prediction process, the text descrip- tion C is encoded into text token embeddings using encoder-only transformer. These lokens provide global semantic prior guiding predictions at every time step. We experiment with different encoder vari- ants, including vanilla transformer encoder (Vassani et al. 2017) and pretrained T5 (Raffel et al. 2020). At time step i, the predictor receives as imput the corresponding text emboddings C, as well the pre- vious object slots Sia, which are initially mapped via MLP to the predictor token dimensionality. Ad- ditionally, these tokens are augmented with tem- poral positional encoding, which applies the same sinusoidal positional embedding to all tokens from the same time step, thus preserving the inherent permutalionnequiiariinne of the objects.
Textual Description
Text Encoder
Text-to Slot Attention
Layer Norm
Text Enncoder
Custom TF Encoder
Multi- Head Self-Attention
Layer Norm
Text Encoder
Pre-trained T5 Encoder
S2
S3
Figure 3: Overview of TextOCVP's text-conditioned object-centric video predictor transformer.
Each layer of our predictor module mirrors the transformer decoder architecture Vaswani et al. 2017 First, self-attention layer enables every slot to attend to all other object representations in the sequence, modeling the spatio-temporal relations between objects. Subsequently, a text-to-slot cross-attention layer enhances the slot representations by incorporating relevant features from the text embeddings, guiding the prediction process
align with the motion and dynamics described the textual caption. Finally, an MLP is applied for each token. This process is repeated in every predictor layer, resulting in the predicted object slots of the subsequent time step S,+1. Furthermore, we apply residual connection from S, to S which improves the prediction temporal consistency. This process is repeated autoregressively to obtain slot predictions for T subsequent time steps.
3.3 Video Rendering
The video rendering module decodes the predicted slots $, to render the corresponding video frame X, We leverage two variants of the video rendering module, for our TextOCVPsAv: and TextOCVPpiNo variants.
TextOCVPsAv: Decoder This variant independently docodes each slot in S, with CNN-based Spatial Broadcast Decoder (Watters et al. 2019), rendering an object image of and mask m² for each slot s. The object masks are normalized across the slot dimension, and the representations are combined via a weighted sum to render vidco frames:
Ng 0 with m softmax(m") Ng
X,
m
(3)
5

---

Published in Transactions on Machine Learning Research (02/2026)
DeXtOCVPDINO Decoder This decoder variant decodes the object slots in two distinct stages. First. following DINOSAUR (Seitzer al. 2023), an MI.P-hased Spatial Broadcast Decoder (Watters et al.) 2019 is used to generate object features along with their corresponding masks. Similar to the TextOCV decoder, the object masks are normalized and combined with the object features in order to reconstruct the encoded features hú E RLxDs, In the second stage, the reconstructed features h2 are arranged into grid format and procossed with a CNN decoder generate the corresponding video frame X,.
3.4 Training and Inference
Our proposed TextOCVP is trained in two different stages.
Object-Centric Learning We first train the scene parsing and video rendering modules for parsing video frames into object-centric representations by minimizing a reconstruction loss. In the TextOCVPsavi variant (CsAvi), these modules are trained by reconstructing the input images, whereas in TextOCVPpiNo (CDINO) they are trained by jointly minimizing an image and feature reconstruction loss:
CSAVI
1 T X X
Σ
DDINO
1 T E
[ X x hell
(4)
Predictor Training Given the pretrained scene parsing and rendering modules, we train our Tex1OCVP predictor for rext-conditioned video prediction using a dataset containing paired videos and text descriptions. Namely, given the object representations from a reference frame S and the textual emboddings C, the predictor forecasts subsequent object slots $2, which are decoded into a predicted video frame X2. This process is repeated autoregressively, i.e. the predicted slots appended to the input in the next time step, in order to generate the set of slots for the subsequent T time steps. This autoregressive training, in contrast to teacher forcing, enforces our predictor to operate with imperfect inputs, leading to better modeling of long-term dynamics at inference time. TextOCVP predictor trained by minimizing the following loss:
T+1 Σ
TexLOCVP T
(Aing + Astot t=2
(5)
with Cing 111 -1 111 and Cslot = ||S, S
(6)
where Limg measures the future frame prodiction error, and Lswe enforces the alignment of the predicted object slots with the actual inferred object-centric representations.
Inference At inference time, TextOCVP receives input single reference frame and language instruc- tion. Our model parses the seed frame into object slots and autoregressively predicts future object states and video frames conditioned on the given textual description. By modifying the language instruction, TextOCVP can generate a new sequence continuation that performs the specified task while preserving a consistent scene composition.
4 Experiments
4.1
Experimental Setup
4.1.1
Datasets
We evaluate TextOCVP for text-conditioned video prediction the CATER. and CLIPort datasets:
CATER Girdhar & Ramanan 2020) is a dataset of long video sequences showing 3D objects in motion, each paired with a descriptive caption. We used the CATER-hard variant (Hu et al., 2022), containing 30,000 sequences with 64 x 64 frames leaturing two to eight objects, two of which follow the scripted motion.
CLIPort (Shridhar et al. 2022 is a robot manipulation dataset featuring vidco-caption pairs. We employ 21,000 336 x 336 sequences of the Put-Blocks Bowl task, where each scene contains six objects, either a block a bowl, on 2D table plane, and robot arm. The caption describes placing block into bowl.

---

Published in Transactions on Machine Learning Research (02/2026)
Table 1: Quantitative evaluation on CATER and CLIPort datasets for prediction horizons of and
19. TextOCVP outperforms the baselines. Best two results are shown in bold and underlined, respectively.
(a) Quantitative evaluation CATER.
CATER
CATER
Method
PSNR
SSIM
LPIPSI
JEDi
PSNR
SSIM
LPIPS!
JEDi
OCVP
et al.
2023
29.08
0.874
0.078
4.16
28.11
0.854
0.101
8.08
Non-OC
29.68
0.874
0.092
3.04
28.39
0.849
0.112
8.62
SEER Gu et al.
22.05
0.723
0.245
11.23
16.05
0.535
0.299
17.29
MAGE
Hu et a.
2022
34.91
0.877
0.108
3.46
34.76
0.871
0.111
5.88
TextOCVP (Ours)
32.98
0.922
0.036
2.16
31.29
0.902
0.044
5.09
(b) Quantitative evaluation on CLIPort.
CLIPorti-9
CLIPorti-19
Method
PSNR1
SSIMI
LPIPS|
JEDi
PSNR
SSIM1
LPIPS.
JEDi
Non-OC
23.44
0.901
0.184
8.13
20.14
0.872
0.210
13.23
SEER Gu et al. 2024
21.01
0.887
0.141
6.80
11.30
0.622
0.331
8.29
MAGEDINO (Hu et al. 2022
23.72
0.940
0.064
2.11
22.27
0.931
0.075
2.59
TextOCVP (Ours)
26.99
0.950
0.062
1.36
23.88
0.931
0.078
2.23
4.1.2 Baselines
We benchmark TextOCVP against established text-guided video prediction baselines and analyse key ar- chitectural design choices. To assess the impact of text conditioning, we compare TextOCVP with OCVP- Seq (Villar-Corralss et al. 2023), an unconditional object-centric prediction model. To evaluate the role of object-centric representations, we introduce OCVP variant (Non-OO) replacing slot representations with single holistic embedding. We further compare TextOCVP with three transformer- diffusion- text-conditee cext-conditioned prediction models: MAGE (Hu et al. 2022), MAGEDINO, and SEER (Gu et al. 2024 Further details on bascline models are provided in Appendix
4.1.3
Implementation Details
All models are implemented Torch and trained on a single NVIDIA A6000 (48Gb) GPU. OCXPOCVPsM SAVE closely follows Kipf et al. (2022) for scene parsing and video rendering. TextOCVPpiNo uses DINOv2 (Oquab) et al. 2024) as image encoder, a four-layer MLP Spatial-Broadeast Decoder (Watters et al. 2019 to decode slots into object features and masks, and CNN decoder to map the reconstructed scene features back to images. On CATER, we use the TextOCVPsAv variant with eight 128-dimensional object slots, whereas on CLIPort we employ the OextOCPPiiNo variant with ten 128-dimensional slots. Our predictor module is an eight-layer transformer with 312-dimensional tokens, eight attention heads, and a hidden dimension of 1024. Further experimental details including datasets, baselines, evaluation metrics, and implementation details are provided in Appendices
4.2 Results
4.2.1 CATER Results
On CATER, we train the models predict nine future frames given single reference frame and text caption. In Table Ia report quantitative evaluations CATER using the same setting in training, i.c. predicting T = 9 frames, well as when predicting T = 19 future frames. In both settings, TextOCVP outperforms all other models, demonstrating superior perceptual quality.
Fig. La qualitatively compares TextOCVP and MAGE on CATER. TextOCVP generates a sequence that exhibits coherent object trajectories and accurate motion grounded the input textual instruction. This alignment made possible by the explicit object-centric design, which disentangles the scene into structured object representations and allows the model to reason about each object's dynamics independently. Tn

---

Published in Transactions on Machine Learning Research (02/2026)
'the medium green metal sphere is aliding to (2, 1). small metal cube picked up and placed (-3, 1
2
10
20
30
Blurred
Changed
MAGE
Tex
(a) Text-guided video prediction result CATER. Top depicts the ground truth frames. TextOCVP predicts sharp and accurate frames, whereas MAGE blurs and misses objects.
put the blue block in the gray bowl
20
30
40
50
Missing
MAGE
Tex1OCVP
(b) Qualitative evaluation on CLIPort for text-driven video prediction. Top row depicts the ground truth frames. TextOCVP successfully completes the pick-and-place task, whereas in MAGFDINo the moved block disappears.
Figure 4: Qualitative comparison of TextOCVP and baseline methods text-guided video prediction.
contrast, MAGE predictions feature multiple errors and artifacts, including missing objects, blurry contours, and significant changes on object shapes. These issues arise from the difficulty of modeling object-specific dynamics in pixel or token space without explicit object-level abstraction.
These results demonstrate the importance of structured, object-centric representations in capturing the compositional and controllable nature of physical scenes, which especially critical in tasks involving fine- grained object manipulations.
4.2.2 CLIPort Results
Table [ reports quantitative results for text-guided video prediction on CLIPort. TextOCVP outperforms all baselines when evaluated on the same sotting as training (i.c. 1 1 9). For longer prodiction horizons,
both MAGEpINO and TextOCVP perform well, with TextOCVP achieving the best scores on JEDi (Lo perceptual video metric evaluating motion quality and realism.
et al. 2025
8

---

Published in Transactions on Machine Learning Research (02/2026)
Table 2: Ablation studies on key design choices in TextOCVP, including predictor depth 2ap, text encoder [2c.. and slot count (2d).
choice (2b, residual connections
(a) Effect of number of layers (Np) pre-
(b) Impact of different text encoders, including a transformer encoder
dictor module.
(TF) and two 15 variants, on CATER and CLIPort.
CATER:--
CATER1-19
CLIPorti-1
Np
S
LPIPS
JEDi
Text Enc.
SSIM
LPIPSI JEDi,
SSIMI LPIPS} JEDI
0.908
0.045
2.26
TF
0.903
0.045
5.39
0.911
0.043
2.20
Frozen T5
0.902
0.044
5.09
0.931
0.078
2.23
0.922
0.036
2.16
FT T5
0.901
0.043
5.14
0.928
0.082
2.67
(c) Effect of residual connection in predictor.
(d) Effect of the number of object slots (Ns).
CLIPorti-9
CLIPort
Residual
SSIM1
LPIPS!
JEDi
# Slots
SSIM1
LPIPS1
JEDi
X
0.946
0.066
1.58
8
0.946
0.079
5.62
0.950
0.062
1.36
10
0.950
0.062
1.36
In our qualitative evaluations, we observe that TexP generates the most accurate sequence predictions given the reference frame and language instruction. Fig. 4b compares TextOCVP with MAGEDINO over a long prediction horizon of 50 frames, corresponding to the full completion of the given task. MAGEDINO fails to complete the task outlined in the textual description, it misses the target block after several prediction time steps. In contrast, TextOCVP successfully completes the instructed task, maintaining coherent object trajectorics and consistent scene dynamics throughout the sequence.
Nevertheless, we observe that TexLOCVP often generates visual artifacts and lacks fine-grained textures. While these imperfections can affect its quantitative scores, TextOCVP's object-centric structure enables robust long-term prediction and precise estrucctionfollovinng behavior.
4.3
Model Analysis
4.3.1
Ablation Studies
We perform several ablation studies to support and validate the architectural choices of our model compo- nents and their impact on TextOCVP's video prediction performance. The results are presented in Table 2 and analysed below across four main design axes. Further results and analysis are provided in Appendix F
We study the effect of increasing the number of transformer layers (Np) in the predictor module. We observe that prediction quality improves with model depth, with Np = 8 yielding the best performance across all metrics on the CATER datasct. This suggests that deeper predictor models enable more accurate modeling of object. dynamics. However, we did not explore beyond eight layers due to the substantial increase in computational cost, training time and parameter count.
Number of Layers (Table 2a
We evaluate the effect of introducing residual connection in the predictor module, where the updated slot representation is defined as $ S + fS.) Adding this residual path improves performance across all metrics the CLIPort dataset, particularly in improving the temporal consistency of predicted slots. This result is consistent with prior findings "Villar-Coraal et al, 2023), which show that residual updates are beneficial for iterative refinement in structured latent spaces.
Residual Connection (Table 2c)
We evaluate the performance three different text encoders, including a lightweight transformer trained from scratch (TF), a frozen T5 encoder, and fino-tuned (FT) T5. While all three perform similarly on CATER, the frozen T5 text encoder achieves the best results on CLIPort, especially on perceptual metrics such JEDi and LPIPS. Interestingly, fine-tuning the T5 encoder led to
Text Encoder (Table 2b)

---

Published in Transactions on Machine Learning Research (02/2026)
Table 3: Quantitative evaluation on two CLIPort test sets with unseen colors and larger number of objects. We report the absolute video prediction performance and the relative drop (in parentheses) compared to the original evaluation setting. Best result is marked in bold.
(a) Evaluation on CLIPort test set with objects of colors unseen during training.
CLIPorti-9
CLIPorti-19
Method
SSIM
LPIPSL
SSIM
LPIPSI
MAGEDINO (Hu et al. 2022)
0.985 (-0.5%)
0.076 (-19%)
0.924 (-0.7%)
0.087 (-16%)
TextOCVP (Ours)
0.946 (-0.4%)
0.066 (-6.4%)
0.927 (-0.4%)
0.083 (-6.4%)
(b) Evaluation on CLIPort test set with larger object count than training scenes.
CLIPorti-9
CLIPort1-19
Method
SSIM
LPIPS}
SSIMT
LPIPS1
MAGEDINO (Hu et al. 2022 TextOCVP (Ours)
0.929 (-1.2%)
0.088 (-37.5%)
0.920 (-1.2%)
0.094 (-25.3%)
0.936 (-1.5%)
0.076 (-22.6%)
0.921 (-1.1%)
0.090 (-15.4%)
slightly degraded performance, likely due to overfitting or interference with the pretrained representations. We therefore use the frozen T5 encoder in all main experiments to balance performance and model efficiency.
Finally, we explore the impact of varying the number of slots (Ns) used to represent the scene. Although each CLiPort scene can be described with eight slots representing six objects, the robot arm, and the background- we find that using ten slots results in notably better performance. This observation suggests that the additional slots can function as internal registers, supporting attention routing or acting as a form of cache that aids with internal model computations (Darcet et al. 2024).
Number of Slots (Table 2d
4.3.2 Model Robustness and Generalization
We evaluate the robustness and generalization of OCX and MAGEDINO on two CLIPort evaluation sets: one involving color variations in the text instructions that not encountered during training (unseen- color), and another featuring scenes with more objects than observed during training, cight instcad of six (more-objects). These experiments assess model robustness under distribution shifts and test the ability to generalize to unseen disual-linguise combinations and novel scene configurations. Results are summarized in Table 3 reporting both the absolute performance and the relative drop compared to the standard (seen-color or six-object) settings.
On the unseen-color benchmark (Table 3al) TextOCVP consistently outperforms MAGEDINO across all metrics and prediction horizons, showing significantly smaller degradation in perceptual similarity (LPIPS). This indicates stronger robustness to novel color-object combinations. In the more-objects setting (Table 3b TextOCVP again maintains superior performance with a notably smaller decline in LPIPS, demonstrating resilience to higher scene complexity and unscon object counts. Qualitative examples in Appendix F 6 further support these findings.
Overall, these results highlight the advantages of structured object-centric representations, which provide greater generalization and robustness in vidco prediction tasks, particularly in contrast to holistic scene representations that struggle with novel scene compositions.
4.3.3 Interpretability
Fig. 5a visualizes the text-to-slot attention weights averaged across heads for different slots in a CATER soquence, highlighting how textual instructions guide the model's prodictions. Slots corresponding to de- scribed objects attend strongly the relevant tokens in the language, such as actions or target coordinates. To further dissect this behavior, Fig. shows the attention distribution of individual text-to-slot attention
10

---

Published in Transactions on Machine Learning Research (02/2026)
the cone is sliding to (-3, -3)
the snitch is picked up and placed to (-1, 2)
t ==
5
10 15
20
25
30
Slot 3
1.0
0.5
the
cone is
sliding
to
3
3
the
snitch is the picked
and is placed picked up
up
to
-1
0.0
Slot 4
1.0
0.5
0.0
the
cone
is
sliding
to
33
snitch
and placed
to
-1
2
Slot 6
1.0
0.5
0.0
the cone
sliding
is
to
-3
33
the
snitch
is
picked
up
and
placed
to
-1
(a) Texl-to-slot attention weights, averaged across heads, for different objects in CATER sequence.
Scene
Attended Object
'the small red rubber cube is rotating. the small gray cylinder is rotating."
1.0
1.0
Head 1
Head 2
0.5
0.5
0.0
0.0
the
small
red
rubber
cu
be
rotating
is
the
small
gray
metal
cylinder
rotating
is
the
small
red
rubber
cu
bo
rotating
is
the
small
gray
metal
cylinder
rotating
is
1.0
1.0
Head 3
Head 4
0.5
0.5
0.0
0.0
the
small
red
rubber
cu
be
rotating
is
the
small gray metal cylinder
rotating
is
the small
red
rubber
cu
be
rotating
is
the
small
gray
intaal cylinder
rotating
is
(b) Text-to-slot attention weights between object slot and the textual tokens for four different heads.
Figure 5: Visualization of text-to-slot attention in TexOOCVP (a) Slots attend to relevant text tokens, grounding objects to their described motions. (b) Distinct attention heads focus on complementary textual cues, such as object attributes and actions.
11

---

Published in Transactions on Machine Learning Research (02/2026)
15
30
large purple cone picked up and placed to (2, 3). the small gold metal snitch 1s picked and placed to (-1, 1).
'the large purple rubber cone sliding to -3 ) the small gold metal snitch is rotating.
Original Caption
Changed Actions
1)
'the medium cyan rubber sphere picked and placed to (-2, 2). the medium purple
"the large purple rubber cone is eliding to (-1, 1). the small gold metal snitch is rotating. the mediun purple is picked up and placed to (-1, -3).
cone is sliding to (2, 3)."
Changed Objects
More
iii)
iv) Moving
& Actions
Objects
(a) Qualitative evaluation of TextOCVP's controllability on CATER. Top row shows the ground truth sequence. We underline the changed actions with respect to the original caption. TexP demonstrates fine-grained control by predicting different sequence continuations from the same reference frame, each conditioned on a different instruction.
1=1
10
20
30
40
'put cyan block the broun bool'
Original
Caption
"put the blue block in the brown bool
Changed
Block
(b) Qualitative evaluation of TextOCVP's controllability CLIPort. Top row shows ground truth frames. Tex- tOCVP correctly generates sequences where the robot picks up the correct block and places into the specified bowl.
Figure 6: Qualitative evaluation of 'eextOCVPss controllability on CATER and CLIPort datasets.
heads for slot representing rotating red cube. Different heads specialize in distinct attributes of the text, including the object's 's shape (Head 2), size (Head 1), and the described motion (Head 4).
Together, these results show that the text-to-slot attention mechanism effectively aligns textual information with object-centric representations. This enables accurate video prediction conditioned on natural language, and provides a degree of interpretability by revealing which parts of text influence each object representation.
4.3.4 Controllability
A key objective of text-guided video prediction is provide fine- ane-ngrined control over the prediction process via language instructions that specify the relevant objects and their actions.
12

---

Published in Transactions on Machine Learning Research (02/2026)
Fig. 6a illustrates TextOCVP's controllability CATER. Starting from the same reference frame, we generate multiple sequence continuations by varying the natural language instruction. These variations include changing the target objects and their actions, as well as instructions that specify greater number of actions than seen during training. As shown in Fig. 6a TextOCVP successfully identifies the relevant objects and executes the described actions. Notably, it distinguishes between two nearly identical purple cones, despite their identical shapes and color, and generates sequences consistent with the specified motions.
A similar experiment on CLIPort is shown in Fig. 6b Given single frame with multiple colored blocks and bowls, modifying the instruction determines which block the robot arm picks and the destination bowl. In both cases, TextOCVP correctly selects the specified block, places it in the instructed bowl, and adapts the arm trajectory to the described action.
These results show that key object attributes, such as size or color, are effectively captured through the text-to-slot attention mechanism. This enables accurate, per-object motion forccasting and highlights the benefit of combining object-centric representations with language guidance for controllable video predic- tion. Further quantitative and qualitative evaluations of TextOCVP's controllability are provided in Appen-
dices F.5 and F. F.10 respectively.
5
Conclusion
In this work, we presented Tex1OCVP, an object-centric model for text-conditioned video prediction. Given single input image and a natural language description, TextOCVP generates future frames by parsing the scene into slot-based object representations and modeling their dynamics conditioned the text instruction. This is accomplished through lext-conditioned object-centric transformer that predicts future object states by modeling pratiottemporal relationships between objects while incorporating textual guidance. Through extensive evaluations, we demonstrated that TextOCVP outperforms other existing approaches for text- driven vidco prediction from a single frame, highlighting our model's ability to predict over long prediction horizons and adapt its predictions based on the provided description. Moreover, we validated our architec- tural choices through ablation studies, highlighting the importance of combining textual and object-centric information, and demonstrating strong robustness and interpretability. With its structured latent space and superior controllability, TeXtOCVP offers promising step toward controllable object-centric manipulation in simulated robotic environments, supporting more efficient planning, reasoning and decision-making.
Acknowledgment
This work was funded by grant BE 2556/16-2 (Research Unit FOR 2535 Anticipating Human Behavior) of the German Research Foundation (DFG). Computational resources were provided by the German AT Service Center WestAl.
References
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspec- lives. IEEE Trunsactions on Paltern Analysis and Machine Intelligence (TPAMI), 2013.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In JEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Haoxin Chen, Menghan Xia, Yingqing Hc, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. CiatoCrafterl: Open diffusion models for high-quality video generation. arXiv preprint ar Xiv:2330. 19512, 2023.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Xi Chen, Zhiheng Liu, Mengting Chen, Yutong Feng, Yu Liu, Yujun Shen, and Hengshuang Zhao. LivePhoto: Real image animation with text-guided motion control. In European Conference Computer Vision (ECCV), 2024.
Hsu-kuang Chiu, Ehsan Adeli, and Juan Carlos Niebles. Segmenting the future. IEEE Robotics and Au- tomation Letters (RA-L), 5(3):2202 4209, 2020.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcchre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets. In International Conference on Learning Representations (ICLR), 2020.
Antonia Creswell, Rishabh Kabra, Chris Burgess, and Murray Shanahan. Unsupervised object-based tran- sition models for 3D partially observable environments. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Tal Danicl and Aviv Tamar. DDLP: Unsupervised object-centric video prodiction with deep dynamic latent particles. Transactions on Machine Learning Research (TMLR), 2024.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In International Conference on Learning Represenlations (ICLR), 2024.
Emily Denton and Rob Fergus. Stochastic video generation with learned prior. In International Conference on Machine Learning (ICMI.), 2018.
Andrea Dittadi, Samuele Papa, Michele De Vita, Bernhard Schölkopf, Ole Winther, and Francesco Locatello. Generalization and robustness implications in object-centric learning. Tn International Conference Machine Learning (ICML), 2022.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In IEEE/CVP Conference on Computer Vision and Pattern Recognàtion (CVPR), 2021.
Tsu-Jui Fu, Licheng Yu, Ning Zhang, Cheng- Yang Fu, Long-Chy? Su, William Yang Wang, and Sean Bell. Tell me what happened: Unilying text-guided video completion via multimodal masked video generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognátion (CVPR), 2023.
Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Z Li. SimVP: Simpler yet. better video prediction. In IEEE/CVE Conference on Computer Vision and Pottern Recognition (CVPR), 2022.
Rohit Girdhar and Deva Ramanan. CATER: A diagnostic dataset for compositional actions and temporal reasoning. In International Conference on Learning Representations (ICLR), 2020.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freiteer et al. The "something some- thing' video database for learning and evaluating visual common sense. In IEEE International Conference on Computer Vision (ICCV), 2017.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew M. Botvinick, and Alexander Lerchner. Multi-object representation learning with itcrative variational inference. In International Conference on Machine Learning (ICML), 2019.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint XXtu:2012.05203 2020.
Xianfan Gu, Chuan Wen, Wcirui Ye, Jiaming Song, and Yang Gao. Secr: Language instructed video prediction with latent diffusion models. In International Conference on Learning Representations (ICLR), 2024.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems (NeurPPS), 2022.
Tobias Höppe, Arash Mehrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi. Diffusion models for video prodiction and infilling. Transactions on Machine Learning Research (TMLR), 2022.
Yaosi Hu, Chong Luo, and Zhenzhong Chen. Make it move: Controllable image-to-video generation with text descriptions. In IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), 2022.
Youngjoon Jeong, Junha Chun, Soonwoo Cha, and Taesup Kim. Object-centric world model for language- guided manipulation. arXiv preprint arXi::2503.061 061700, 2025.
Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-centric slot diffusion. In Advances in Newral Information Processing Systems (NeurIPS), 2023.
Daniel Kahneman, Anne Treisman, and Brian J Gibbs. The reviewing of object files: Object-specific inte- gration of information. Cognâtive Psychology, 24(2):175-25 1992.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Confer- ence on Learning Representations (ICLR), 2015.
Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. In International Conference on Learning Representations (ICLR), 2022.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. SPACE: unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations (ICLR), 2020.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Ge Ya Luo, Gian Mario Favero, Zhi Hao Luo, Alexia Volicoeur-Martinen and Christopher Pal. Beyond FVD: Enhanced evaluation metrics for video generation quality. In International Conference on Learning Representations (ICLR), 2025.
Tom Monnier, Thibault Groueix, and Mathieu Aubry, Deep transfornatioon-invriaal clustering. Advances in Neural Information Processing Systems (NewrIPS), 2020.
Malte Mosbach, Jan Niklas Ewertz, Angel
and Sven Behuke. SOLD: Reinforcement learning with slot object-centric latent dynamics. In International Conference on Machine Learning (TCML), 2025.
Trang Nguyen, Amin Mansouri, Kanika Madan, Khuong Duy Nguyen, Kartik Ahuja, Dianbo Liu, and Yoshua Bengio. Reusable slotwise mochanisms. In Advances in Neural Information Processing Systems (NeurIPS). 2024.
Haomiao Ni, Changhao Shi, Kai Li, Sharon X Huang, and Martin Renqiang Min. Conditional image- vi-eeo generation with latent flow diffusion models. In IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
Sergiu Oprea, Pablo Gonzalez,, Alberto Garcia- Garcia-Gareia, John Alejandro Castro-Vargas, Sergio Orts- Escolano, Jose Garcia-Rodrigues and Antonis Argyros. A review on deep learning techniques for video prediction. JEEE Transactions on Pattern Analysis and Machine Intelligence CTaeeancceePPAAAA (TPAMT), 2020.
Maxime Oquab, Timothóe Darcet, Théo Moutakanni, Huy Vo, Mare Szafranice, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision. Transactions Machine Learning Research (TMLRR, 2024.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 21 (140):11667 2020.
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent video transformer. In International Joint Conference Computer Vision, Imaging and Computer Graphics Theory and Applications (VISTGRAPP), 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Maximilian Scitzer, Max Horn, Andrii Zadaianchuk, Dominík Zictlow, Tianjun Xiao, Carl-Johann Simon- Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, et al. Bridging the gap to real-world object-centric learning. In International Conference on Learning Representations (ICLR), 2023.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. CLIPort: What and where pathways for robotic manipu- lation. In Conference on Robot Learning (CoRL), 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015.
Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. In Advances in Newral Information Processing Systems (NewrIPS), 2022.
Xue Song, Jingjing Chen, Bin Zhu, and Yu-Gang Jiang. Text-driven video prediction. ACM Transactions on Multimedia Computing, Communications and Applications (TOMM), 2024.
Sergey Tulyakov, Ming-Yo Yu Liu, Xiaodong Yang, and Jan Kautz. MocoGAN: Decomposing motion and content for video generation. In IEEE/CVF Conference on Computer Vision and Paltern Recogrétion (CVPR), 2018
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. FVD: A new metric for video generation. In International Conference on Learning Representations Workshops (ICLR- W), 2019.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Angel Corralle and Sven Behnke. Unsupervised image decomposition with chase-correlation networks. In International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP), 2022.
Angel Villar- CoraalCs and Sven Behnke. PlaySlot: Learning inverse latent dynamics for controllable object- centric vidco prediction and planning. In International Conference on Machine Learning (ICML), 2025.
Angel Villar- Corrales,, Ani Karapetyan, Andreas Boltres, and Sven Behnke. MSPred: Video prediction multiple ceato-l scales with hicrarchical recurrent networks. British Machine Vision Conference (BMVC), 2022.
Angel Villar- Corrales, Ismail Wahdan, and Sven Behnke. Object-centrie video prediction via decoupling of object dynamics and interactions. In IEFF International Conference on Image Processing (TCIP), 2023.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Xingrui Wang, Xin Li, Yaosi Hu, Hanxin Zhu, Chen Hou, Cuiling Lan, and Zhibo Chen. TIV-Diffusion Towards object-centric movement for text-driven image to video generation. In AAAT Conference Artificial Intelligence, 2025.
Yunbo Wang, Haixu Wu, Jianjin Zhang, Zhifeng Gao, Jianmin Wang, S Yu Philip, and Mingsheng Long. PredRNN: A recurrent neural network for spatiotemporal predictive learning. IEEE Transactions Pullern Analysis and Machine Intelligence (TPAMI), 45(2):2208 2225, 2022.
Zhou Wang, Alan C Bovik, Hamid R. Sheikh, and Eero P Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions Image Processing (TPAMI), 13(():600-6)) 612, 2004.
Nicholas Watters, Loic Malthey, Christopher P Burgess, and Alexander Lerchner. Spatial Broadcast Decoder: A simple architecture for learning disentangled representations in VAEs. arXiv preprint orXiv:1901.07017, 2019.
Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. Generative video transformer: Can objects be the words? In International Conference on Machine Learning (ICML); 2021.
Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg. SlotFormer: Unsuperviscd visual dynamics simulation with object-centric models. In International Conference on Learning Representalions (ICLR), 2023.
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. DynamiCrafter: Animating open-domain images with video diffusion priors. In European Conference on Computer Vision (ECCV), 2024.
Ye and Meeilaume-Alexandie Bilodeau. VPTR: Efficient transformers for video prediction. International Conference on Pattern Recognition (ICPR), 2022.
Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Object-centric learning for real-world videos by predicting temporal feature similaritics. Advances in Neural Information Processing Systems (NewrIPS), 36, 2024.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
17

---

Published in Transactions on Machine Learning Research (02/2026)
A Limitations and Future Work
A.1 Limitations
While TextOCVP demonstrates promising results for text-guided object-centric vidco prodiction, it presents some limitations, which we plan to address in future work:
Prediction Artifacts TextOCVP occasionally generates artifacts in the predicted frames, such as blur- riness, inconsistent object appearances, lack of textured details, or visual artifacts in the background. We believe that these limitations stem from the video rendering module, which might lack the representational power reconstruct precise image details from the object-centric latent space representation.
Limited Temporal Consistency While TextOCVP produces plausible predictions, we observe that its temporal consistency degrade when forecasting for long prediction horizons (T 30), occasionally resulting in object jittering or instability. We attribute this limitation to the fact that Tex1OCVP is trained to predict only up to nine future frames (T'= = 9) and only optimizing reconstruction losses, which do not penalize such temporal inconsistencics.
Inherent Limitations of Slot-based Models Slot-based object-centric models, while effective for disen- tangling scene structure and producing interpretable object-level representations, come with several inherent limitations. They often struggle to represent fine textures, objects with intricate geometry, small visual details, tightly interlocking and highly deformable objects, these phenomena exceed the gramularity and capacity of single slot embedding. In such cases, relevant information is often spread across multiple slots, leading to oversegmentation issues, or compressed into overly coarse latent, leading to imperfect and blurred reconstructions, lack of representation detail, or unstable object binding. More fundamentally, current slot-based architectures typically operate at a single spatial scale, which prevents them from natu- rally encoding part-whole hierarchies, capturing object structure at different levels of detail, or representing complex compositional structures. Addressing these challenges through hierarchical object-centric parsing, adaptive slot resolution, or richer generative decoders remains an important future direction for object-centric learning and video prediction.
A.2 Future Work
In future work we aim address these limitations. Our TextOCVP model is designed with a modular architecture, enabling for both improvements and flexible swapping of components such the parsing, predictor, video rendering modulcs- scamlessly improve the entire pipcline.
To address the prediction artifacts, we plan to extend our TextOCVP framework with more powerful decoder modules, such as autoregressive transformers (Singh et al., 2022) or diffusion models (Jiang et al. 2023 as well as scale our predictor module. Furthermore, we plan to incorporate temporal discriminators Clark et al. 2020 to improve the temporal consistency of the predicted video frames.
We believe that exploring these architectural modifications can overcome the aforementioned limitations and will enable the scaling of TextOCVP to real-world robotic environments.
B Implementation Details
We employ TextOCVPsAvi for the experiments on CATER and TextOCVPpINo for experiments on CLIPort. Below discuss the implementation details for each of these variants.
B.1 TextOCVPDINO
The TextOCVPpiNo variant consists of our proposed text-conditioned predictor module and an object- centric decomposition module that extends the DINOSAUR (Seitzer al.) (023) framework (Extended DINOSAUR) for recursive object-centric video decomposition and video rendering.
18

---

Published in Transactions on Machine Learning Research (02/2026)
Text-Conditions Predictor The predictor is composed of Np 8 identical layers, each containing 8- head attention mechanisms and an MLP with a single hidden layer of dimension 1024 and ReLU activation function. Furthermore, the predictor uses an embedding dimensionality of 512, context window size of ten frames, and applies a residual connection from the predictor input to its output.
Text Encoder TextOCVP DINO leverages pretrained and frozen small version of T5 encoder (Raffe) et al. 2020 which consists of six T5 blocks. This text encoder uses a vocabulary with size 32,120 128.
Scene Parsing The scene parsing module generates Ng- 10 slots of dimension 128. As feature extractor, use DINOv2 ViT-Base (Oquab et al., 2024 featuring 12 layers, using a patch size of 14, and producing patch features with dimension D2- 768. The Slot Attention corrector module processes the first video frame with three iterations in order to obtain a good initial object-centric decomposition, and single ileration for subsequent frames, which suffices to recursively update the slot representation. The initial object slots Sy are randomly sampled from a Gaussian distribution with learned mcan and covariance. We use a single Transformer encoder block as the transition function, which consists of four attention heads and an MLP with a hidden dimension of 512.
Video Rendering The video rendering module consists of two distinct decoders. First, a four-layer MLP- based Spatial Broadcast Decoder (Watters et al. 2019) with hidden dimension 1024 reconstructs the patch Teatures from the slots. Then, CNN-based decoder reconstructs full-resolution images from these features. It consists of four convolutional layers, each using upsampling. A final convolutional layer and bilinear interpolation are applied map the outputs to the RGB channels and spatial dimensions of the image.
kernels, ReLU activation function and bilinear
Training We train our model for object-centric decomposition using video sequences of length five frames for 1000 epochs. We use batch size of 16, the Adam optimizer (Kingmna & Ba 2015), and base learning rate of 4 10-4, which linearly warmed-up for the first 10000 steps, followed by cosine annealing for the remaining of the training process. Moreover, we clip the gradients predictor module is trained given the frozen and pretrained object-centric decomposition model for 700 epochs to predict the subsequent nine frames using single seed frame. The predictor is trained using the same
maximum norm of 0.05. The
as for object-centric decomposition. In the predictor loss function LexxoovP, we set Amng = 1 and Astot =1 1.
B.2 TextOCVPsAV
OCAPaiii uses the same text-conditioned predictor and text-encoder architectures as TextOCVPpiNo. but employs SAVi (Kipf et al. 2022) as the object-centric decomposition module.
Scene Parsing The scene parsing module generates Nş = 8 slots of dimension 128. Following Kipl et al. 2022), we use as feature extractor a lour-layer CNN with ReLU activation function, where each convolutional layer features 32 5 x kernels, stride = 1, and padding = 2. The Slot Attention corrector follows the same structure as in TextOCVPpINO..
Video Rendering Following Kipl et al. 2022), we utilize a CNN-based Spatial Broadcast Decoder (Wal- ters et al. 2019) with four convolutional layers with 32 kernels of size 5 stride 1, and padding = A final convolutional layer maps from the hidden 32-channel representation to four output channels (RGB + alpha mask).
Training We train our model for object-centric decomposition using video sequences of length ten frames for 1000 epochs, using batch size of 64, and an initial learning rate 10-4, which is warmed up for 2500 steps, followed by cosine annealing for the remaining of the training process. Moreover, we clip the gradients maximum norm of 0.05. The predictor module is trained given the frozen and pretrained object-centric decompasition model for 1400 epochs to predict the subsequent nine frames using single seed frame. The
19

---

Published in Transactions on Machine Learning Research (02/2026)
Table Number of learnable parameters in TextOCVP and baselines for experiments on CLIPort.
Model
# Parameters
TeX
33.76M
Non-OC
34.16M
MAGEDINO
32.11M
SEER
405.89M
predictor is trained using the same Super-paramett as for object-centric decomposition. In the predictor loss function LICOOCV, set Amng and ASlot
C Baselines
We employ five different basclines to compare against our TextOCVP model for the task of text-conditioned video prediction on the CATER and CLIPort datasets. To emphasize the importance incorporating textual information, we include a comparison with OCVP-Seq WVillar-Corral et al. 2023), recent object-centric video prediction model that does not utilize text conditioning. Additionally, centric TextOCVP variant (Non-OC) that processes the input image into single high-dimensional slot representation, instead of multiple object-centric slots, thus allowing us to evaluate the effect of object-centric representations. Moreover, we compare TextOCVP with three popular text-conditioned video prediction basclines that do not incorporate object-centricity: the transformer- based MAGE (Hu et al. 2022) model
evaluate a non-object-
and its MAGEDINO variant, and the caluusion-basedd SEER (Gu et al. 2024 model. We train these baselines on CATER and CLIPort closely following the original implementation detail:"])
Table lists the number of learnable parameters in our proposed Tex1OCVP as well as for the baseline models CLIPort. TextOCVP, Non-OC and MAGEDINO employ comparable number of parameters, thus ensuring fair comparison. SEER employs a pretrained latent diffusion model, which already requires a significantly larger number of parameters, and adapts it for the task of text-guided image-to-vided generation.
C.1
MAGE and MAGEDINO
MAGE is an autoregressive text-guided video prediction framework that utilizes a VQ-VAE (Van Den Oord et al. 2017) encoder-decoder architecture to learn efficient visual token representations. A cross-attention module aligns textual and visual embeddings to produce spatially-aligned motion representation termed Motion Anchor (MA), which is fused with visual tokens via an axial transformer for video generation. For experiments on CATER, we use a codebook size of 512 x 256 with a downsampling ratio of four, whereas on CLIPort we use a codebook size of 512 1024.
To ensure fair comparison with TeXtOCVP on CLIPort, we replace MAGE's standard CNN encoder and decoder with the DINOv2 VIT encoder and CNN decoder used in our Tex1OCVP model. We refer to this modified version as MAGEDINO Table 5 presents comparison between the original MAGE model and MAGEDINO CL.IPort. The results demonstrate that MAGEDINO significantly outperforms the original variant, enabling a fair comparison with TextOCVP and other baselines on this benchmark.
MAGE and MAGEDINO share several architectural similarities with our proposed approach using similar encoder and docoder modules, text-conditioning, and an autoregressive transformer for prodiction. However, these models differ from TeXtOCVP in two fundamental ways:
Scene representation: TextOCVP operates object-centric slot representations, whereas MAGE and MAGEDINO rely on holistic, dense scene tokens (vector-quantized image latents). Whereas slots provide factorized latent structure with one representation per object, the holistic VQ-token grid entangles ob-
http:///gthuu..c///onggg///
20

---

Published in Transactions on Machine Learning Research (02/2026)
Table 5: Comparison of MAGE variants on CLIPort. MAGEDINO clearly outperforms the original MAGE
variant across all metries.
Method
PSNR
SSIM 1
LPIPS
MAGE
7.116
0.453
0.713
DAGO
23.723
0.940
0.064
ject information across many spatial tokens without explicit boundarios. This distinction enables clean comparison belween object-centric and holistic auloregressive models.
Mechanism for text conditioning: Both methods use cross-attention to incorporate language guidance, but differ in how textual information interacts with the prodictor. MAGE and MAGEDINO compute a single global Motion Anchor via one cross-attention step between text and image latents, and inject this global signal uniformly into all decoding steps. In contrast, Text applies text-to-slot cross-attention within every transformer block, allowing the model to repeatedly integrate and select the textual information most relevant at each processing stage.
C.2 SEER
SEER is a diffusion-based model for languago-guided video prediction. It employs Inflated 3D U-Net derived from pretrained text-lo-image 2D latent diffusson model (Rombach et al. 2022). exlending it along the temporal axis and integrating temporal attention layers to simultaneously model spatial and temporal dynamics. For the language conditioning module, SEER introduces a novel Frame Sequential Text (FSText) Decomposer, which decomposes global instructions generated by the CLIP text encoder (Radford jtt al. 2021) into frame-specific sub-instructions. These are aligned with frames using a dransformer-basel temporal network and injected into the diffusion process via cross-attention layers. We initialize SEER from checkpoint pretrained on the Something- -Smettin- V2 dataset (Goyal et al. 2017), and further fine-tune it for few epochs. We observed that incorporating a text loss enhanced SEER's performance, while other hyper-parameters were kept consistent with its original implementation.
C.3 Non-OC
Non-OC is variant of our proposed TextOCVP model in which the slot-based object-centric latent repre- sentations are replaced with a single, nign-dimensiooal slot embedding. This design allows us to isolate the contribution object-centric structure in the latent space.
Non-OC mirrors the Tex1OCVP architecture, using the same visual backbone, text-guided autoregressive predictor and decoder, and it is trained with identical hyper-parameters and training schedule. The only difference lies in the scene parsing module: instead of slot attention, Non-OC applies additional convo- lutional block followed by average pooling to produce a single latent vector per frame. This results in one 512- 112-dimensional embedding, in contrast to the set of 128-dimensional object slots used in the object-centric model.
D Datasets
D.1 CATER
CATER Girdhar & Ramanan 2020 is datasct that consists of long video sequences, cach described by ual caption. The video scenes consist of multiple 3D geometrie objects in a 2D table plane, which is split into a 6 x grid with fixed axis, allowing the exact description of object's positions using coordinates. The text instruction describes the movement of specific objects through four atomic actions: 'rotate', 'pick-place', 'slide', and 'contain'. The caption follows a template consisting of the subject, action, and an optional object end-point coordinate, depending the action. The movement of the objects starts at the same time
21

---

Published in Transactions on Machine Learning Research (02/2026)
step. Furthermore, the initial positions are randomly selected from the plane grid, and the camera position
is fixed for every sequence.
In our work, we employ CATER-hard, which is a complete version of the CATER dataset, containing 30000 video-caption pairs, with video frames resized to 64 x 64. It inchudes possible objects: cone, cube, sphere, cylinder, or snitch, which a special small object metallic gold color, shaped like three intertwined tori. Furthermore, every object is described by its size (small, medium, or large), material (metal or rubber), and color (red, blue, green, yellow, gray, brown, purple, cyan, gold if the object is the snitch), and this description is included in the textual caption. Every atomic action is available. The 'rotate' action is afforded by cubes, cylinders and the smitch, the 'contain' action is only afforded by the conos, while the other two actions are afforded by every object. Every video has between 3 and 8 objects, and two actions happen to different objects at the same time. The vocabulary size is 50.
D.2 CLIPort
CLIPort (Shridhar et al. 2022] is a robot manipulation dataset, consisting of video-caption pairs, iee. long videos whose motion is described by a textual vidco caption. There are many variants of the CLIPort datasot, but we focus on the 336 x 336. Every video contains 6 objects block or a bowl, and there is at least one of them in every sequence. The starting position of each object is random, with the only constraint being that it must be placed on the table. Each video describes the action of the robot arm picking a block, and putting it in a specific bowl. The video caption follows the template 'put the [color] block in the [color] bowl'. Each individual object in the scene has a different color. In the train and validation set, the block and the bowl that are part of the caption can have one of the following colors: blue, green, red, brown, cyan, gray, or yellow, while in the evaluation set with unseen colors they can have blue, green, red, pink, purple, white, or orange color. The other 4 objects, called distractors, can have any color. During video sequence, it can be possible that the robot arm goes out of frame, and comes back in later frames, thus requiring the model to leverage long range dependencies. The vocabulary size is 15.
variant. We generate 21000 video-caption pairs with resolution 2D table plane, and robot arm. Objects can be either
E Evaluation Metrics
To measure TextOCVP's video prediction performance and compare it with existing approaches, we evaluate the visual quality of predicted video frames using popular image- and video-based metrics.
PSNR and SSIM (Wagg et al. 2004 measure pixel-wise and statistical differences between the predicted and ground-truth video frames, respectively.
LPIPS Zhang et al. 2018) is a perceptual metric that measures the visual similarity between two images based on deep features from pretrained neural networks, usually VGG (Simonyan & Zisserman 2015). Unlike pixel-wise metrics, LPIPS compares activations from multiple layers and captures differences in texture, structure, and semantics, thus aligning closely with human perception.
While these metrics focus frame-level visual quality, video prediction evaluation requires quantifying temporal consistency and motion rcalism. JEDi [Luo et al.) 2025) mcasures the quality and realism of generated videos by comparing feature distributions of generated and real videos, capturing both visual fidelity and motion dynamics. We prefer JEDi over FVD (Unterthiner et al.) 2019), as JEDi is less sensit.vee to small evaluation datasets, and more stable in low-motion synthetic scenarios.
In our evaluations and comparisons with baselines, we favor the LPIPS and JEDi scores, which correlate well with human perception, while reporting other metrics for completeness.
22

---

Published in Transactions on Machine Learning Research (02/2026)
==1
2
5
GT
SAVi
Ext. DINO
Ext. DINO Masks
Figure 7: Comparison between SAVi and our Extended DINOSAUR (Ext. DINO) decomposition modules for reconstruuctnng a CLIPort sequence. We visualize the reconstructed frames, as well as the slot. masks obtained by Extended DINOSAUR. SAVi fails to reconstruct most objects, whereas Extended DINOSAUR accurately reconstructs the scene, while representing each object.
F
Additional Results
F.1 SAVi vs. DINOSAUR
Current object-centric approaches for video prediction are limited to relatively simple synthetic datasets, and struggle to scale beyond scenes featuring simple 3D shapes with simple deterministic motion. We attribute this limitation primarily to the object-centric modules used for learning object representations. Motivated by this oon, we extend the recent DINOSAUR (Seitzer et al. 2023 framework to recursively process vidco data and reconstruct vidco frames from their corresponding object-centric representations.
To demonstrate the significance of the object-centric module in scaling to more complex datasets, compare both SAVi and our Extended DINOSAUR trained on CLIPort. As illustrated in Fig. 7 SAVi struggles to accurately represent the objects on the table, missing multiple objects and changing their shape and color. In contrast, the Extended DINOSAUR model successfully reconstructs the scene, closely resembling the input, while accurately representing each object. The visual features extracted by the DINOv2 (Oquab) et al. (2024) encoder contain high-level semantic information, and during training, the slots are specifically optimized efficiently encode this information. This design enables the model scale and handle more complex object-centric video data effectively.
23

---

Published in Transactions on Machine Learning Research (02/2026)
Table 6: Average inference (Inf.) time for T and T = 9 frame predictions on CATER and CLIPort.
CATER Inf. [ms]]
CLIPort Inf. [ms]
Model
7==1
9
T=
7' 9
MAGE
15.4 = 1
111.1 + 11
34.4 + 3
198.8 + 2
Tex1OCVP 13.6
78.6
7
21.1 +
109.9 + 2
Table 7: Evaluation at prediction horizons T = 9 and 19. Tex1OCVP is the best performing among all compared methods both on CATER and CLIPort, followed by MAGE/MAGEDIN-- Best two results are shown in bold and underlined, respectively.
CATER1 19
CATER
CLIPort -9
CLIPorti
Method
PSNR, SSTM, JPIPS,JEDi; PSNR; SSIM, JEDII PSNR- SSIM, LPIPS.JEDi, PSNR; SSTM, LPIPS,JEDL
OCVP
29.08
0.874
0.078
4.16
28.11
0.854
0.101
8.08
Non-OC
29.68
0.874
0,092
3.04
28.39
0.849
0.112
8.62
23.44
0.901
0.184
8.13
20.14 14
0.872
0.210
13.23
SEER
22.05
0.723
0,245
11.23
16.05
0.535
0.299
17.29
21.01
0.887
0.141
11.30
0.622
0.331
8.29
MAGE
34.91
0.877
0.108
3.46
34.76
0.871
0.111
5.88
23.72
0.940
0.064
2.11
22.27
0.981
0.075
2.59
TextOCVP
32.98
0.922
0.036
2.16
31.29
0.902
0.044
5.09
26.99
0.950
0.062
1.36
23.88
0.078
2.23
F.2
Computational Efficiency
In Table f we report the average inference time for T and T =9 frame predictions on CATER and CLIPort using NVIDIA-A6000-SGGb GPU.
During inference, TextOCVP is significantly faster than MAGE/MAGEDINO achieving x 40% lower latency. This efficiency stems from our model's object-centric design, which operates on a small number of object slots, in contrast to the larger number of spatial tokens employed by MAGE.
F.3 Quantitative Comparison
In the main paper, we present quantitative evaluations the CATER and CLIPort datasets using three evaluation metrics, shown in two separate tables. For completeness, report in Table text-guided video prediction evaluation of TextOCVP and multiple baseline models on CATER and CLIPort using four distinct evaluation metrics for different prediction horizons.
Our proposed OCXP outperforms baselines on both the CATER and CLIPort datasets, surpassing the nest-best method, MAGE/MAGEDINO by clear margin. Most notably, TextOCVP consistently achieves the best LPIPS and JEDi scores, demonstrating superior frame-wise visual quality, well video-level fidelity and motion realism.
F.4 Object-Centric Centric Evaluation
A key advantage of object-centric approaches in video prediction is their ability to generate segmentation masks alongside frame predictions. For TextOCVP, we derive predicted segmentation masks by applying an argmax operation over all object slot masks produced by the docoder during prediction and subsequently filtering the resulting masks by assigning values with small magnitude the background. These predicted masks are evaluated against ground- truth segmentation masks using the Intersection over Union (IoU) metric.
We compare our model against two basclines on CLIPort. The Copy-Seed bascline simply replicates the pre- dicted segmentation mask from the seed frame across all frames, while the OC-module-onil variant generates segmentation masks by feeding ground-truth frames directly into the object-centric module, ie., Extended DINOSAUR. As shown in Table 8] TextOCVP outperforms both basclines across prodiction horizons of T and T 19 frames, demonstrating the model's ability to model object dynamics, maintaining spatial consistency and object coherence over time. The predicted masks of OCXPOCVP align more closely with the
24

---

Published in Transactions on Machine Learning Research (02/2026)
Table 8: Object-centric evaluation on CLiPort. Comparison of Intersection over Union (loU) scores between
predicted and ground truth segmentation masks for different prediction horizons (T 9 and T 19).
CLIPort
CLIPort1-11
Method
IoU1
IoUt
Copy-Seed
0.540
0.525
OC-modulc-only
0.554
0.553
TeXOOCVP 0.573
0.569
Table 9: Quantitative evaluation of TextOCVP's controllability on CL.Port. We report the mean pick- and-place success rates (averaged over 5 runs) for the original evaluation set and three instruction variants differing in the target bowl. TextOCVP maintains consistent performance across variants, reflecting strong robustness to instruction changes.
Pick-and-Place success rate
Method
Original set
Variant 1
Variant
Variant
Mean variants
TextOCVP
0.83
0.86
0.80
0.81
0.82
ground-truth segmentations, benefiting from better temporal stability throughout the prediction horizon. In we illustrate TextOCVP's object-centrie behavior on CATER and CLIPort by showing the predicted segmentation masks, as well as different per-object predictions.
Figs. 11
and 12
F.5
Quantitative Evaluation of Controllability
In Sec. 4.3.4 we qualitatively show how Tex1OCVP is able adapt its predictions based on the language instruction it receives as input. We further make an initial attempt quantitatively evaluate the controlla- bility of TextOCVP.
Starting from CLIPort evaluation set of 100 sequences, create three language instruction variants for each sequence, differing in the specified target bowl. Given the same initial frame, TextOCVP then generates future predictions conditioned on the adapted instructions. To assess performance, we ground the slot masks produced by the video rendering decoder their corresponding objects in the scene and compute two key distances: (1) between the masks the robot arm and the specified block, and (2) between the picked block and the target bowl. These measures are used to estimate the pick-and-place success rate. A generated sequence considered successful only if the robot arm remains sufficiently close to the correct block over multiple frames and the block is significantly close to the target bowl toward the end of the sequence.
As shown in Table g TextOCVP demonstrates consistent success rates across different instruction varia- tions, indicating strong robustness and fine- ane-grained controllability. Given identical starting scene, the model effectively adapts its predictions to the changing text instructions, successfully placing the block into the specified bowls in most of the cases.
We note that equivalent experiment could not be performed with MAGEDINO, it does not generate object masks during prediction.
F.6 Robustness to Number of Objects
In Sec. 1.3.2 we quantitatively assess the generalization and robustness of TextOCVP in video prediction tasks involving novel scene compositions. Our results highlight the benefits object-centric representations over holistic scene-based approaches.
This finding is further illustrated in Fig. 9 which presents qualitative comparisons of video generations for scenes containing eight objects, in contrast to the six-object configurations seen during training. As
25

---

Published in Transactions on Machine Learning Research (02/2026)
Table 10: Impact of visual artifacts on TextOCVP's performance on CLIPort. Cropping the artifact-prone bottom part of the predicted frames leads notable improvement TextOCVP's performance.
CLIPort1-9
CLIPorti-11
Image View
SSIM1
LPIPS}
SSIM
LPIPS}
Full Image
0.950
0.062
0.931
0.078
Excluded Bottom
0.953
0.050
0.932
0.069
observed, model correctly predicts sequences following the motion described in the text instructions, whereas MAGEDINO fails to generate accurate sequences according to the descriptions.
These results further demonstrate the effectiveness of object-centric representations for video prediction, as TextOCVP is able to generalize to scenes with more objects by simply increasing the number of slots. This flexibility is enabled by initializing object slots via sampling from a learned Gaussian distribution, allowing the use of variable number of slots at test time while breaking symmetry and preserving permutation invariance. Although the object slots do not specialize during training, they reliably bind to meaningful entities through iterative attention during inference. This results in robust and scalable scene decomposition, enabling accurate modeling of complex scenes with varying object counts.
F.7 Impact of Visual Artifacts
As already discussed in the main paper, TeXtOCVP occasionally generates visual artifacts on the CLIPort dataset, most noticeably in the bottom-center region of the frame, where blurry patches often appear.
To assess the impact of these visual artifacts on TextOCVP's quantitative performance, evaluate model after removing the last bottom rows from the predicted frames an area that contains only background pixels and the artifacts. As shown in Table 10 removing these rows leads to improved results, particularly for the perceptual LPIPS metric. These findings verify that TextOCVP generates future frame predictions that closely follow the text description, and that its overall performance is underestimated due to localized visual artifacts.
F.8 Impact of Video Rendering Module
In the previous section (Appendix(.7) F.7 7 we described how visual artifacts affect TextOCVP's performance on the CLIPort dataset. We argue that these artifacts mostly originate from limitations the video rendering module, and that they can be mitigated by adopting a more expressive decoder architecture.
To support this claim, conducted an additional experiment CLIPort in which we replaced the simple CNN dleoder-describeer described in Appendix B.T with more expressive alternative, while keeping the rest of TextOCVP's architecture unchanged. The new decoder, inspired by VQGAN (Esser et al. 2021), integrates convolutional, residual, non-local attention, and upsampling blocks, and contains nearly twice as many learnable parameters as the original module (~ 25M vs. ~ 13M).
TextOVP models using both decoder variants are qualitative compared in Fig. 10 which shows that the more expressive decoder substantially reduces the visual artifacts observed in the original model's predictions. In Table II we quantitatively compare both decoder variants. The more expressive decoder model achieves noticeably better LPIPS and JEDi scores, demonstrating superior perceptual frame and video quality.
Overall, these findings indicate that employing a more capable decoder can mitigale some of Tex1OCVPs visual limitations. They also highlight the benefits of our modular architecture, which enables such im- provements to be incorporated scamlessly without modifying the underlying predictor or representation. Exploring more powerful rendering modules such as diffusion-basse based decoders (Jiang et al., (023) or refin- is promising direction for future work.
ing the VQGAN-style architecture FEsser al. 2021
26

---

Published in Transactions on Machine Learning Research (02/2026)
Comparison of TVP with a variant using a more expressive video rendering module on CLIPort. The stronger, VQGANN-ispired [Esser et al. 2021 decoder leads improved perceptual quality.
Table 11:
CLIPort
CLIPort1-11
TextOCVP Decoder
LPIPS!
JEDi
LPIPS|
JEDi
Simple CNN
0.062
1.36
0.078
2.23
VQGAN-based
0.043
0.86
0.065
2.22
0.58
0.16
0.56
0.14
LPIPS
0.12
0.54
0.10
0.52
0.08
0.50
0.06
0.48
10
15
20
25
30
35
40
5
10
15
20
25
30
35
40
Predicted time-step
Predicted time-step
(a) LPIPS (4) metric across prediction horizons
(b) IoU (f) metric across prediction horizons
Figure LPIPS and slot-mask IoU metrics across prediction horizon of 40 future frames. The plots show both the average values (bold line) well as the standard deviation (shaded areas).
F.9 Evaluation of Predictions over Time
In Fig. present plots of LPIPS and slot-mask IoU scores over prediction horizons of up to 40 future frames. These curves show that our model's predictions steadily deviate from the ground truth as the pre- diction horizon increases, reflecting the natural accumulation of errors inherent to autoregressive prediction.
We also note that quantitative metrics such as LPIPS or IoU become less informative when predicting beyond 20-30 future frames. At these prediction horizons, even slight differences in object velocity or trajectory compound and lead to large discrepancies in frame-wise metrics, despite still producing qualitatively consistent futures. This behavior is typical of autoregressive video models and reflects the sensitivity of pixel-level scores to small lemporal deviations, rather than qualitative failure of the predicted dynamics.
F.10
Additional Qualitative Evaluations
Fig. shows example where evaluate TextOCVP and MAGEDINO for text-guided video prediction over long prediction horizon of 50 frames. MAGEDINO fails to complete the task outlined in the textual de- scription, as it stops generating consistent robot motion after 30 frames. In contrast, TextOCVP successfully predicts future frames where the robot completes the pick-and-place task.
Figs. and show qualitative evaluations on CATER in which both MAGE and TextOCVP successfully prodict sequence continuations following the instructions from the textual description.
Fig. 16 illustrates example where MAGE fails to generate correct sequence, while TextOCVP successfully completes the task described by the text.
Figs. and I8 show examples of TextOCVP's control over the predictions. In both sequences, TextOCVP generates a correct sequence given the text instructions, and seamlessly adapts its generations to a modified version of the textual instructions.
27

---

Published in Transactions on Machine Learning Research (02/2026)
'utt the gray block in the brown bowl' ?
=
5
10
20
30
40
MAGEDINO
TextOCVP
a)
'put the yellow block in the gray bowl' >
t = 1
10
20
30
40
50
MAGEDINO
TextOCVP
b)
Figure 9: Qualitative evaluation of MAGEDINO and Tex1OCVP CLIPort sequences with more objects than those seen during training (eight instead of six). TextOCVP correctly generates sequences according to the text instructions, whereas MAGEDINO misses the target bowl.
28

---

Published in Transactions on Machine Learning Research (02/2026)
ptt the red block in the brown bowl'
5
10
20
30
10
4
TextOCVP w/ VQGAN dec.
TextOCVP w/ original dec.
a)
'put the yellow block in the gray bowl'
t = 1
10
20
30
40
50
TextOCVP w/ VQGAN dec.
Tex1OCVP w/ original dec.
b)
Figure 10: Qualitative comparison between original OeX implementation and a OeXP variant with a more expressive, VOGAN-inspired, video rendering module. While both variants correctly predict the described motion, the new decoder improves the predicted frames' quality by mitigating most of visual artifacts, especially on the bottom-center region and in the robot arm.
29

---

Published in Transactions on Machine Learning Research (02/2026)
'the large purple rubber come is picked up and placed to (2, 3)
the small gold metal snitch is picked up and placed to (-1, 1).²
t = 1
10
15
20
30
Predictions
Slot Masks
Object
Object
Object 3
Object 4
Figure 11: TextOCVP's object-centric behavior on a CATER sequence. The first row shows the ground fruth sequence, followed by Tex1OCVP's predicted frames and segmentation masks. The subsequent rows display the reconstructed objects from four of the predicted slots across various time steps, highlighting the ability of TextOCVP to model the dynamics of individual objects in the scene through slot representations.
30

---

Published in Transactions on Machine Learning Research (02/2026)
put the cyan block in the red bowl' ,
1
1
10
15
20
35
GT
Predictions
Slot Masks
Object
Object 2
Object 3
Object
Figure 12: TexLOCVP's object-centric behavior CLIPort sequence. The first row shows the ground truth sequence, followed by TextOCVP's predicted frames and segmentation masks. The subsequent rows illustrate the represented objects from four of the predicted slots across various time steps. Although only eight slots are required for this dataset (six objects, one robot arm, and one background), we use ten slots in CLIPort experiments which proved beneficial; the two extra slots represent background. We emphasize that the object segmentations computed at the patch level, thus the pixelated appearance in the visualizations.
31

---

Published in Transactions on Machine Learning Research (02/2026)
'put the gray block in the brown bowl'
t=1
20
30
40
50
MAGEDINO
TextOCVP
Figure 13: Qualitative result on CLIPort. Top row shows ground truth frames. TextOCVP completes the pick-and-place task, whereas MAGEDINO fails to predict the robot motion.
32

---

Published in Transactions on Machine Learning Research (02/2026)
'the large brown metal cone is picked up and containing the medium red rubber cone. the small gold metal snitch is rotating.
5
10 15
20
30
MAGE
TextOCVP
Figure 14: Qualitative evaluation on CATER. Both MAGE and TexPOCVP successfully generate a sequence following the instructions from the textual description.
the medium green rubber cone is picked up and containing the small gold metal snitch.
the large purple rubber cone is picked up and placed to (-1, 3) , 5
10
15
20
30
MAGE
TeXtOCVP
Figure 15: Qualitative evaluation on CATER. Both MAGE and TextOCVP successfully generate a sequence that illustrates the motion described in the text, but MAGE's predictions are of lower resolution.
33

---

Published in Transactions on Machine Learning Research (02/2026)
'the large yellow rubber cone is sliding to (2, 3) the small gold metal snitch is picked up and placed to (-3, 1).² 5
10
15
20
30
MAGE
TextOCVP
Figure 16: Qualitative evaluation on CATER. MAGE fails to generate a sequence that accurately follows the motion described in the text. Specifically, the yellow cone docs not slide as expected, and artifacts such as the merging of two small objects are introduced. On the other hand, the sequence generated by TextOCVP is closely aligned with the ground truth, accurately capturing the motion of the objects.
34

---

Published in Transactions on Machine Learning Research (02/2026)
10
20
30
40
50
9
'put the blue block in the brown bowl'
Original
Caption
'put the blue block in the red bowl
Changed Bowl
Figure 17:
Qualitative evaluation of TextOCVP controllability CLIPort. TextOCVP correctly generates sequence where the robot picks up and places the block specified in the textual instruction.
10
20
30
40
50
'put the green block in the cyan bowl"
Original
Caption
'put the green block in the red bowl
Changed Bowl
Figure 18: Qualitative evaluation of Tex1OCVP controllability CLIPort. Tex1OCVP correctly generates sequence where the robot picks up and places the block specified in the textual instruction.
35


Reviews:
[{"review_id": "4V8PyuL5xK", "reviewer": "Reviewer_PJwh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents TextOCVP, a new object-centric approach for video prediction guided by textual descriptions. Past work tends to struggle to generalize and scale beyond simple scenes and don't afford easy ways to integrate additional information like language which has become a more popular input modality for robotics (e.g. VLAs). A text-to-slot attention mechanism is leveraged to allow text to attend to the image/video data for textual guidance. TextOCVP is tested on two benchmarks and outperforms the baselines.\n\nStrengths:\n- Paper is well written and articulated\n- The presented ideas/tricks are great (such as text-to-slow attention) and would be particularly valuable to the CV and robotics community\n\n\nWeaknesses:\n- There aren't many ablations on TextOCVP itself and more of comparisons quantitatively and qualitatively with some previous baselines. This makes it difficult to determine the contributions of each proposed method and whether they do make a significant difference or not.\n- Only synthetic datasets are tested, there are real world robotics video datasets that could be tested.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Appropriate benchmarking is provided to compare TextOCVP against baselines with TextOCVP outperforming in terms of SSIM, LPIPS, and JEDi. Appropriate figures are provided to demonstrate the interpretability and controllability aspects of TextOCVP. Attention maps are visualized along with the input text/image sequences and show this well. They also show good results in terms of how object-centric features improve TextOCVP over the MAGE baseline.", "audience_yn": "Yes", "audience_explanation": "The paper tackles a highly relevant problem of world modeling with applications to robotics. The demonstration of the ability to integrate more multi-modal inputs makes this paper more interesting as robotics data tends to be highly multi-modal and will only grow in diversity and complexity in the future. For those who work more on the computer vision side of things, the approach to integrating text would be of interest to others in that field as well.", "requested_changes": "While the results shown are great, it would be appreciated to see a proper ablation on whether the object-centric loss/part of the model is really necessary. The example shown in Figure 4a highlights it well, but is just a single example.\n\nMoreover, only synthetic datasets are tested, however there are plenty of real world datasets in e.g. the OpenX Embodiment dataset work that could be compared against which can help better ground this paper.", "broader_impact_concerns": "N/A", "additional_comments": "N/A", "extra_scores": {}, "extra_text": {}}, {"review_id": "6BfYQ7CsDK", "reviewer": "Reviewer_AP1H", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents TextOCVP, an object-centric model for text-guided video prediction. The method parses a reference frame into slot-based object representations and uses a text-conditioned autoregressive transformer to predict future object dynamics and render video frames consistent with a natural-language instruction. The authors evaluate two variants (based on SAVi and DINOSAUR), benchmark against existing text-conditioned video prediction methods (MAGE, SEER, MAGEDINO), and conduct ablations on model depth, text encoders, and slot count. They further analyse robustness to distribution shifts and visualise text-to-slot attention to argue interpretability and controllability.\n\n**Key strengths**\n* The paper is clearly written, well structured, and easy to follow.\n* The overall idea (slot-level conditioning with language) is compelling and practically useful, particularly for robotics.\n* Experiments on CATER and CLIPort are extensive and show meaningful improvements over baselines.\n* The method provides fine-grained control via natural language and demonstrates robustness to unseen colours and object counts.\n* The ablation studies are thorough and informative.)\n\n**Key weaknesses**\n* The conceptual distinction between TextOCVP and prior models (especially MAGE-DINO style token-based conditioning) is not articulated clearly. My issue here is not necessarily about whether the contribution is incremental or novel but rather about specific insight as to where TextOCVP differs from its baselines and whether performance changes due to the text-to-slot encoder.\n* The interpretability analysis (attention visualisations) is descriptive but not explanatory; the paper misses an opportunity to provide deeper insights into why slot-level conditioning helps. For instance, it makes sense that the attention mechanisms learns a correlation between coordinates and object position as coordinates vary across scenes while the other tokens remain fixed, essentially being a discrete bottleneck. All these phenomena require a bit more explaining in my opinion and would make the paper immensely more valuable.\n* The method inherits common limitations of slot-based models (like disentangled representations), but these are not discussed.\n* It is unclear which improvements stem from the object-centric representations versus stronger architectural components (deeper transformer, DINO encoder), since these factors are somewhat confounded.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The empirical claims regarding prediction quality, controllability, and robustness are supported by extensive experiments on two relevant datasets. The evaluation setup is well explained, ablations are thoughtful, and comparisons to strong baselines (MAGE, SEER, MAGEDINO, OCVP) are fair and comprehensive. \n\nThe qualitative results are meaningful and correspond to the claims.\nHowever, while the experimental evidence convincingly demonstrates performance gains, some conceptual claims, such as the interpretability of the text-to-slot alignment or the inherent generalisation benefits of object-centric representations are stated but not deeply substantiated. The paper could be strengthened by providing more mechanistic insight into how and why the slot representations align with text attributes beyond the attention heatmaps.\n\nOverall, the empirical evidence is strong, but the conceptual justification could benefit from deeper analysis.", "audience_yn": "Yes", "audience_explanation": "The intersection of object-centric learning, video prediction, and language grounding is highly relevant to the TMLR community, particularly for researchers working on goal-conditioned RL, robotics, and structured representations. The method is practical, addresses a real limitation of prior models (lack of controllable prediction), and introduces a structured alternative to recent diffusion-based approaches. Additionally, the CLIPort evaluation and focus on controllability make the work valuable for robotics and model-based decision-making tasks.", "requested_changes": "1. Clarify the conceptual distinction between TextOCVP and MAGE/MAGEDINO-style models.\nThe paper currently frames the contribution as object-centric vs. holistic, but does not clearly articulate why slot-level text conditioning is fundamentally advantageous. A more explicit conceptual argument would greatly strengthen the contribution.\n\n2. Provide deeper insight into the mechanisms behind text-to-slot alignment.\nThe attention visualizations are interesting but remain descriptive. The authors should include either:\na hypothesis grounded in representational learning (e.g., lower entropy slot space, conditional independence structure), or\nan empirical study probing the specialization of slots or attention heads.\nWithout this, the interpretability claims are underspecified.\n\n3. Discuss limitations of slot-based models.\nThe method inherits known issues from slot-based models. A short discussion acknowledging failure modes would make the work more balanced and informative. The authors touched on this in their results discussion (blurriness, lack of fine-grained detail), but like other parts of the paper they feel barely addressed. The authors do not necessarily have to run additional experiments for this but a deeper interpretation would help the reader take away more than from reading a simple technical report", "broader_impact_concerns": "I do not see major ethical risks beyond those inherent to generative video models.", "additional_comments": "The paper is overall strong: well presented, empirically validated, and relevant to TMLR’s audience. My main concern is not with correctness or technical adequacy but with conceptual clarity and depth of insight. Strengthening the discussion around why slot-level conditioning helps, and what mechanisms underlie the observed behaviours, would elevate the contribution from a well-executed system paper to a more enlightening scientific result.", "extra_scores": {}, "extra_text": {}}, {"review_id": "UwdyaxZEjv", "reviewer": "Reviewer_q6ZE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a video prediction model that leverages object-centric representations conditioned on natural language instructions.\n\nContributions:\n- A modular framework that parses frames into object slots and predicts future states using a text-conditioned transformer.    \n- Empirical demonstration that object-centric representations significantly outperform holistic baselines in out-of-distribution scenarios, specifically regarding unseen object colors and higher object count.    \n- Quantitative and qualitative evidence that the model can precisely follow fine-grained text instructions for robotic manipulation tasks.    \n\nStrengths:\n- The integration of text-to-slot attention is well-motivated and provides interpretability.    \n- The contribution for handling complex textures in CLIPort is valuable.    \n- The paper provides strong empirical support for the long-held hypothesis that object-centric biases facilitate combinatorial generalization.    \n\nWeaknesses:\n- Evaluation is limited to synthetic and simulation datasets, despite claims regarding applicability to \"robotics\".    \n- The method suffers from visual artifacts and temporal instability at longer horizons (T>30).", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors provide rigorous comparisons against strong baselines. The claims regarding robustness are well-supported by Table 3. The claims regarding controllability are supported by both qualitative visualizations in Fig. 6 and the ablation studies in Table 9. \n\nWhile the visual fidelity is not photorealistic (as acknowledged in the limitations), the evidence supports the core claims about control and generalization dynamics.", "audience_yn": "Yes", "audience_explanation": "The paper is relevant to researchers in computer vision, robot learning, and representation learning. Specifically, the Object-Centric Learning community will be interested in the method for conditioning slots on text.", "requested_changes": "I recommend the following adjustments to strengthen the submission. None are strictly critical for acceptance, but addressing them would improve the paper's precision and impact.\n\n- The introduction claim the method addresses limitations limiting applicability in \"robotics\" and enables \"world models for robotics\". However, the experiments use CLIPort, which is a simulation environment. Please soften the language to specify \"robotic simulation\" or \"simulated manipulation environments\" to avoid overclaiming real-world applicability, or explicitly discuss the domain gap challenges in the limitation section.   \n- The paper notes that artifacts likely stem from the video rendering module. A brief discussion (or a small experiment if feasible) comparing the current decoder with a stronger alternative would be valuable.\n- The limitations section mentions jittering after 30 frames. It would be helpful to see a plot of prediction error (e.g., LPIPS or slot position error) over time (T=1 to 50) in the appendix. \n- For the SEER baseline, ensure the training compute/data regime is explicitly compared. Diffusion models often require more data. Clarifying if SEER was under-trained or if this represents a fundamental efficiency advantage of TextOCVP would strengthen the results analysis.", "broader_impact_concerns": "None. The paper focuses on a fundamental method for video prediction in simulated environments. The current work does not raise ethical concerns requiring a specialized statement beyond standard considerations.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)