Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling
Authors: Tal Daniel
Carl Qi
Dan Haramati
Amir Zadeh
Chuan Li
Aviv Tamar
Deepak Pathak
David Held
Venue: ICLR 2026 Oral (2026)
Keywords: World Model
Self-supervised
unsupervised
object-centric
video prediciton
video generation
imitation learning
latent particles
vae
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: a self-supervised object-centric world model that learns keypoints, and masks directly from videos, supports multi-modal conditioning, scaled to real-world multi-object datasets

Abstract: We introduce Latent Particle World Model (LPWM), a self-supervised object-centric world model scaled to real-world multi-object datasets and applicable in decision-making. LPWM autonomously discovers keypoints, bounding boxes, and object masks directly from video data, enabling it to learn rich scene decompositions without supervision. Our architecture is trained end-to-end purely from videos and supports flexible conditioning on actions, language, and image goals. LPWM models stochastic particle dynamics via a novel latent action module and achieves state-of-the-art results on diverse real-world and synthetic datasets. Beyond stochastic video modeling, LPWM is readily applicable to decision-making, including goal-conditioned imitation learning, as we demonstrate in the paper. Code, data, pre-trained models and video rollouts are available: https://taldatech.github.io/lpwm-web

Published as a conference paper at ICLR 2026
LATENT PARTICLE WORLD MODELS: SELF- SUPERVISED OBJECT-CENTRIC STOCHASTIC DYNAM- ICS MODELING
Daniel', Carl Qi Dan Haramati', Amir Zadeh', Chuan Li Aviv Tamar Deepak Pathak- David Held"
Carnegie Mellon University, PT Austin, 'Brown University, Lambda, Techhion tdaniel @andrew. cmu. edu
ABSTRACT
We introduce Latent Particle World Model (LPWM), a self-supervised object- centric world model scaled to real-world multi-object datasets and applicable in decision-making. LPWM autonomously discovers keypoints, bounding boxes, and object masks directly from video data, enabling it to learn rich scene de- compositions without supervision. Our architecture trained end-to-end purely from videos and supports flexible conditioning on actions, language, and im- age goals. LPWM models stochastic particle dynamics via a novel latent ac- tion module and achieves tate--of-the-art results on diverse real-world and syn- thetic datasets. Beyond stochastic video modeling, LPWM readily applicable including coal-conditioned imitation learning, strate in the paper. Code, data, pre-trained models and video rollouts are available: https: :ettss:/ttan htt///aaaaaaccceeeeeaecae github. io/ 1pwm web
decision-
demon-
96
Wasks
FG
36
RRE
Br Age
A/h
db
GiT
1PVAD
DYAE
Pla
LPwM
LPWi
Wove the blue cabe disgonal to the red dircle
latokt-astion conditooooooiiooonnn condittoned vidco prodiction
Figure Self-supervised object-centric world modeling with LPWM. Top: latent particle decomposition. Bottom left: language-condttioned video generation. Bottom right: Iatentaactioncoomdiione video prediction.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Recent years have witnessed remarkable progress in the visual fidelity of general-purpose video generation models (Blattmann al. 2023 Yang al. 2024b). Driven by vast datasets and expan- sive computational resources, these sofdlssofteen built on scalable architectures like Transform- ers (Vaswani al., 2017 have achieved unprecedented realism. However, their success comes at steep computational cost: training requires thousands of GPU hours (Zhu et al.] (024), and, due to their reliance on diffusion processes (Ho et al.) 2020), inference remains slow and resource- intensive, limiting practical applications. This has sparked an important question: can we lever- age the strengths of these generative models for decision- masiinn? For instance, by turning them into world models- dyyemis- predictors that can be externaly controlled by action goal signals, tasks such as robotic planning (Yang et al. 2023: Zhu et al., 2024) become possible. Despite their strengths in producing high-fidelity videos, these models' resource demands be prohibitive. In parallel, recent work (Haramati et al.) [2224] Qi et al., 2025) demonstrates that incorporating in- ductive biases and leveraging more compact models can enable efficient, robust performance complex multi-object decision-making tasks. Motivated by this, our work aims to bridge these two directions-by by introducing an efficient, self-supervised, object-centric world model for video pre- diction, and decision-making in real-world and simulated, multi-entity environments.
Consider the illustrative example in Figure where the dynamics of two moving objects are described alongside the caption: "The blue ball is moving diagonally towards the green square." Text representations typically rely on semantic tokenization into words or subword units, which underpins the success of large language models (Rad- ford et al., 2018). contrast, image representations pre- dominantly use "patchifying" diiiiiyyiinnividdinnnn the image into a fixed grid of patches without regard for semantic con- tent (Dosovitskiy al., 2020). While this patch-based approach enables scalability and generality, it lacks the semantic intuitiveness of object-centric decompositions that can enhance the model's ability to capture mean- ingful object interactions and relationships, crucial for understanding complex scenes, and align more naturally with language representations. Inspired by the "what- where" pathway in the human visual system (Goodale & Milner 1992) and recent neuroscience findings (Pick- ering & Clark 2014 Nau et al.) 2018 [Barnvvlli et al. 2025). which suggest humans leverage internal visual- spatial world models for planning and action, combin- ing object-centric representations with world models is promising direction towards more effective decision- making and lasion-language integration.
"The ioke boll is moving diagonaly tswaads thts green squars"
TTokovi eation cttfioo/RRorrseettatiin
Twrchiffing
SamaaviclDkeen-ccrttt
Fine
Titd
movi
Aui
frse
di duegonociyy
nowwardss
groes
Squuare
Figure 2: Representation discrepancy. Text is typically tokenized into seman- tically meaningful units such as words or subwords, whereas image representa- tions most often constructed by di- viding the image into a fixed grid of patches ("patchifying") that do not ex- plicitly encode semantic content.
Building on this premise, recent research has focused introducing inductive biases in the form of Daniel & Tamar (2022a)) which
object-centric representations, namely Deep Latent Particles (DnP, have shown strong empirical benefits across range of doomann--incudinn video prediction (Daniel & Tamar 2024), reinforcement learning (RL) (Haramati et al. 2024). imitation learning (Qi et al., 2025). and microscopy (Goldenberg al. 2025). These approaches demonstrate that, when applica- ble, object-centric representations can lead to improved downstream performance, even with smaller model sizes and less data. However, their success has so far been largely confined to specific datasets and environments, typically involving simulated scenes or simple real-world settings with isolated objects, limited camera motion, or single-agent interactions. Scaling object-centric models han- dle the complexity of real-world multi-object environments remains a substantial challenge. While patch-based representations remain dominant for large-scale, general- purpose visual modeling, on- going advances indicate that object-centric approaches offer clear advantages for decision decision-making making tasks whenever the problem structure allows. The present work aims to advance this direction by developing scalable, efficient world modeling grounded in object-centric decomposition.
2

---

Published as a conference paper at ICLR 2026
In this work, we introduce the Latent Partiele World Model (LPWM), the first self-supervised object- centric world model capable of end-to-end training on complex real-world video data. Building upon the DLP-based video prediction framework DDLP (Daniel & Tamaa 2024), we eliminate the requirement for explicit particle tracking and propose a novel context module that predicts distribu- tions over latent actions for each particle. This approach enables stochastic dynamics sampling and enables scalability to complex environments. LPWM is trained exclusively from video observations and supports optional conditioning on actions, language, images, and multi-view inputs. Its design can accommodate range of decision-making applications, including unconditional video predic- tion pretraining and coal-conditioned imitation learning. By integrating object-centric representa- tions with scalable stochastic dynamics modeling, LPWM advances the development of efficient and interpretable visual world models.
Our contributions are summarized as follows: (1) We propose a self-supervised object-centric world model with a novel latent action module that supports multiple conditioning types, including ac- tions, language, images, and multi-view inputs; (2) We achieve state-of-the-art performance in object-centric video prediction on diverse real-world and simulated multi-object datasets, and (3) We demonstrate the applicabilitty of our model to imitation learning on two complex multi-object environments, highlighting its utility for decision-makee making tasks.
RELATED WORK
This section provides an overview of related literature relevant to latent object-centric video predic- tion and world modeling. To the best of our knowledge, LPWM is the first self-supervised object- centric model that can be trained solely from videos, supports multi-view training, and enables diverse conditioning modalities, including actions, language, and goal images. Since existing method shares this unique combination of capabilities, we briefly review several adjacent and com- plementary lines of work to highlight the context and novelty of our contributions. A more detailed survey of keypoints, latent actions, and decision- decision-making methods is provided in Appendix [A.6]
General video prediction and latent world models: Classic approaches encode images into latent spaces and predict future states with recurrent dynamics, often using convolutional encoders and RNNs (Finn et al., 201 Ha & Schmidhuber) (2188). Recent work has improved long-horizon prediction with discrete latent variables (Hafner et al., 20205), hierarchical architectures (Wang al. 2022). self-attention (Micheli et al. (2024), and language conditioning (Nematollahi et al., 2025). but most methods model frames holistically and lack explicit object decomposition, resulting in blurry or unstable predictions. Video diffusion models (Zhu et al. 2024) achieve high fidelity, but remain computationally intensive and do not incorporate object-centric biases.
Unsupervised object-centric latent video prediction and world models: Unsupervised object- centric video prediction methods learn latent dynamics on decomposed scene elements, typically using patch-, slot-, or particle-based representations. Patch-based approaches (e.g., G-SWM (Lin) et al. 2020a)) represent objects using local latent attributes and typically model joint dynamics with RNNs and interaction modules. These methods rely on post-hoc matching object proposals across frames for temporal consistency,
with unordered object representations- their scalability to complex or real-world video datasets. Slot-based approaches (Locatello et al., 2020) typically represent scenes set of slots: ceemmutation-ivvriin latent vectors encoding spatial and appearance information for objects. These approaches generally adopt a two-stage train- ing strategy: slot decomposition is first learned independently, followed by a separate dynamics model trained on the inferred slots using RNNs (Nakano et al., 2023) or Transformers (Wu et al.) 2022b Villar-Corales et al. 2023). In practice, slot-based methods suffer from inconsistent de- compositions, blurry predictions, and convergence issues (Seitzer et al.) (002).. Particle-based approaches, introduces as DLP (Daniel & Tamar (002a),, provide compact, interpretable object representations using seypoint-based latent particles with extended attributes. DDLP (Daniel & Tamar 2024) jointly trains a Transformer dynamics model and the particle representation, allowing stable object-centric decomposition and improved modeling of complex scenes. However, DDLP relies on particle tracking and sequential encoding, which restricts parallelization and stochasticity. Our proposed LPWM model is a direct extension to this lineage. LPWM eliminates the need for explicit tracking, enabling parallel encoding of all frames, trains end-to-end, and integrates a latent action distribution for stochastic world modeling. This allows the model to capture transitions such
3

---

Published as a conference paper at ICLR 2026
object occlusion, appearance, random movements (e.g., agents grippers), and supports com- prehensive conditioning via actions, language, or goal images-ade advancing particle- basieee-based modeling to the world model regime and addressing unsolved limitations of previous work.
Video prediction and world models with latent actions: Several recent works have introduced la- tent cationss-llobe latent variables representing transitions between consecutive frames- to learn controllable or playable environments from videos. Models like CADDY (Menapace et al., (201)) and Genie (Bruce et al. 2024) learn discrete latent actions by quantizing inverse module outputs. and conditioning dynamics on these codes in a two-stage training scheme. AdaWorld (Gao et al., 2025) proposes continuous latent action space with strong KL regularization. PlaySlot (Villar Corrales & Behnke (025) augments slot-based object-centric video prediction with discrete latent action conditioning, demonstrating benefits of object-level decomposition for controllable model- ing. In contrast, our particle-based LPWM learns continuous, per- per-particle latent actions end-to-end with dynamics, naturally capturing stochastic multi-object interactions. LPWM'ss learned latent pol- icy enables sampling latent actions during inference without external input, supporting stochastic video generation. It also supports diverse ienaitioninn-includii goal-conditioning-mmale well-suited for post-hoe policy learning and control, demonstrated in experiments.
Table summarizes key differences between sulf--upe object-centric video prediction and world modeling methods.
Obj--Conticc Rep.
Cond.
SCALOR
Tamg elall
2019
Patch
RNN
Patch
STOVE
Kueann cl m 2019
Patch
RNN
Transformer RNN
GATSBI
Min cl al
2001
Paich-Keypt
Slots
STEDIE
Nakabo el
M023
Slots
RNN
Slots
OCVP
Vilber Corriles
2023
Slots
Transformer
2025
Transfommer
SOLD
Musbuch c/ 2023
Slots
Transformer
PlaySki
Discrete
Transfoomer
DLP Dantil & Tamar 20122
Particles
GNN
Daniel
Particles
Transformer
1PWM(OUs))
Particles
Coni. (per)
Transformer
Table I: Comparison of object-centric video prediction and world modeling methods across key dimensions and representation types. Please refer to Table 4 for an extended comparison.
BACKGROUND
Variational Autoencoders (VAEs, (Kingma & Welling 2014)): VAEs are likelihood- based latent variable models that maximize the evidence lower bound (ELBO) on the data log-likeling ligllikelihood:
log po()) 2 (=lux) [l 2)] - K (x(
ELBO(x),
where (2(xx) (the encoder) approximates the intractable posterior, and pe (z)2) (the decoder) mod- els the likelihood. Typically, qo. Pg. and the prior p22) are Gaussian distributions, enabling efficient training via the reparameterization trick. Minimizing the negative ELBO decomposes into recon- struction loss and KL regularization term.
Temporal VAEs (ee et al.) 2018: Ha & Schmidhuber 2018) extend this framework to sequential data (e.g., videos) by training to maximize the sum ELBOs over timesteps. Here, the prior for each latent at timestep is conditioned on previous latents through a dynamics model, i.e., (2'))
E, K for video prediction.
enables learning temporally coherent latent dynamics suitable
Deep Latent Partieles (DLP, Daniel Tamar (2022a; 2024)): a VAE-based self-supervised object-centric representation for images. Each image is modeled set of M foreground la- tent particles alongside a single background particle. A foreground particle is defined as 2g = [2p, Ze: 2d, 2t. 2y] 6 where each component encodes a disentangled stochastic attribute: po- sition Zp ~ (Np,, E R²², representing the 2D keypoint coordinates; scale 2, ~ NN o?) RR? representing bounding-bor box size; depth 2d ~ N(ua, oz) E R, specifying compositing or- der (indicating which particles appear in front of others within the rendered scene); transparency
4

---

Published as a conference paper at ICLR 2026
2 Beta(a, E [0, 1], controlling visibility; and visual features 2, ing appearance of the local region around the particle. The particle attributes are illustrated in Fig- ure il the Appendix. The background is represented by a single particle 2bg ~ [Nihh zgg E Rd fixed at the image center and modeling background visual features. DLP additionally learns an al- pha channel mask per particle as part of reconstruction, enabling pixel-space foreground-backgroundd decomposition. For a detailed overview of DLP and its components, as well as improvements in- troduced in this work, see Appendix A.3 Figure presents example decompositions of DLP on various datasets used this work.
NN of) E ER encod-
Non-latent (observed) temporal variables are denoted with subscripts indicating the tem- poral index, c.g., 2t, while latent variables use superscripts, e.g., zf. For latent particles, the super- script index denotes the particle number within the set, and subscripts refer to attribute types. For example, zpr. denotes the position (keypoint) attribute p of particle m at timestep t, whereas 2ig represents the background particle at timestep I.
Notations:
4 LATENT PARTICLE WORLD MODELS (LPWM)
Our objective is to construct a world model a dynamics model F(L:TT-1,C) II-T+--1 that, given a sequence of I image observations Io E ==*"xHH* (where C is the number of chan- nels, H and W are image height and width), and optionally conditioning inputs e (actions, language, etc.), generates a rollout of future predictions /7: E RTxCCHXW in an autoregressive man- ner. As modeling directly in pixel space is high-dimensional and sample inefficient, we propose an end-to-end latent world model, Latent Particle World Models (LPWM), which combines a compact self-super object-centric latent representation, based on DLP, with a novel learned dynamics module that operates over particle latents. The model is trained end-to-end such that the representa- tion trained to be predictable by the dynamics module.
The Latent Particle World Model (LPWM)) consists of four components, jointly trained end-to- end as a VAE: the ENCODER Eg, the DECODER Dø. the CONTEXT K. and the DYNAMICS Fş. The pipeline, illustrated in Figure 3 proceeds as follows: input frames are encoded into particle sets by the ENCODER, decoded back to images by the DECODER for reconstruction loss, then processed by the CONTEXT module to sample latent actions, which are combined with particles in the DYNAMICS module to predict next-step particles states and compute per per-particle KL. Below, we summarize the the con-
role of each module and describe, in the main text, the core novel conribbutions-particllaa text and dynamics modules. For completeness, extended component details, minor implementation modifications of DLP and design choices are provided in Appendix A.4
corresponds to the VAE's approximate posterior (2(2(z)) It takes as input an image frame and outputs a set of latent particles: E = 1,) = H m.t M - 26g) Each frame 4e is represented by M foreground latent particles {= 1. where each particle originates from per- par--atch learned keypoint (see Appendix A.4.1), and one background particle zhg- Unlike DDLP, particle filtering to a subset L S M is deferred to the decoder to preserve particle identities and eliminates the need for explicit tracking, requirement in DDLP that necessitated sequential frame encoding. In contrast, the proposed approach enables encoding all frames in parallel. Fore- ground particles are parameterized as afg E gg+das where the first six dimensions capture explicit attributes (e.g., spatial coordinates, scale, transparency), and the remaining dabj dimensions repre- sent uppearance features as described in Section The background particle is defined as 2bg E Rd,, where dog denotes the latent dimension of the background visual features. These features are en- coded from a masked version of the original image, in which regions corresponding to visible fore- ground particles are masked out, as illustrated in Figure Pin the Appendix.
ENCODER & (Appendix A.4.1):
Jm=
DECODER D, (Appendix A..ee.) corresponds to the VAE's likelihood po(x)>). It takes as input a set of L < M foreground particles together with background particle, and reconstructs an image frame: De([L2 1.1 J1=0 2 7. Here, L can be less than M to allow particle filtering before rendering, based on transparency confidence measures (Daneel & Tamar 2024), reducing memory usage without compromising reconstruction quality. Each particle is decoded independently into an RGBA (RGB + Alpha channel) glimpse i. E R 5x54, where S is the glimpse size, representing the reconstructed appearance of particle I. The alpha mask (Alpha channel) is modulated by the transparency and depth attribute of each particle, and the decoded glimpse is then placed into the full-
5

---

Published as a conference paper at ICLR 2026
resolution canvas to create Zig. The background is decoded from 2hg using standard upsampling network to produce fgg. and the final reconstructed image is stitched according to i = a @ Big + (1 - a) O Tbg. where a is the effective mask obtained from the compositing process.
CONTEXT Nsp: We now present the main novel component added to the DLP framework the CONTEXT module designe to model stochastic dynamics in actionless videos. In such videos, scene dynamics are not fully determined by initial frames (e.g.. a ball beginning to roll (Lin et al.] 2020a) where initial conditions fully determine future dynamics) but can also be influenced by ex- ternal signals such as actions (e.g., a robotic gripper (Ebert et al. 2017a))
Commonly, stochastic transitions are captured by introducing latent actions (Menapace al. 2021
Bruce et al. [024]] Gao et al.) 2025 Villa-Corrales & Behnke| 2025). Typically, a latent action Ze learned through an autoencoding scheme: an inverse model infers 2 Kinn (4++1, 4,) from
consecutive frames, and decoder reconstructs the future frame 1++1 Do z trained via construction loss. To prevent = from trivially memorizing 4++1. strong regularization is applied through vector quantization bottleneck (Bruce et al., 2024 Ye et al. (025) or KL-regularrzztton to a fixed prior (Giao et al., 2025). However, these approaches use a global latent action vector rep- resenting all changes between frames, limiting their ability model local dynamics in multi-entity scenes (e.g., independent enemy movements in Maric or secondary contact events in robotics). A global action vector cannot naturally disentangle these local dynamics.
In this work, we introduce the CONTEXT module Kg,, a novel per-particle mechanism for latent action modeling. Unlike prior work (Villar-Corrals & Behnke 2025 Gao al., (025). we model
a latent action for each particle, directly governing the transition from 2, to 2; m,,+1 Regular- ization is not imposed via a fixed prior, but instead learned through latent policy, which models the distribution of latent actions conditioned on the current state. This per-particle formulation en- ables the representation of multiple, simultaneous interactions, and allows stochastic sampling of latent actions at inference time, capturing multimodality (e-g., moving left or right from the same state). The proposed module explicitly separates the modeling of latent actions (which encapsulate the stochastic aspects) from the dynamics prediction.
Formally, the CONTEXT module takes as input sequence of particle sets across T + 1 frames, optionally conditioned on external signals {at]T- (e.g.. control actions, goal images, or language instructions). It outputs sequence of per-particle latent contexts:
M K. m. 1m= 2/g. c = (H<= m= 26 T-
The CONTEXT module is implemented as a causal spatio-temporal transformer( (Zhu et al. 2024), Appendix A.4.5) which jointly processes particles across space and time while ensuring autore- gressive temporal dependencies. It is composed of two complementary heads: (1) Latent inverse dynamics p (=t 2f+1 ,...,, 2", (2), which predicts the latent action responsible for the tran-
sition between consecutive states; (2) Latent policy p polley 2 .2.....,2,,, 24),,), which models the distribution of latent actions conditioned on the current state.
The latent policy serves as a prior that regularizes the inverse dynamics via a KL-divergence penalty in the VAE objective (Appendix A.4.6). Specifically, the latent actions are modeled as Gaussian distributions, 2e ~ N(He, o²), parameterized by the context module. At training time, latent actions obtained through the inverse dynamics head, ensuring consistency with observed transitions. At inference time, latent actions can instead be sampled directly from the latent policy prior, enabling stochastic rollouts of the world model. Conditioning on external signals (global actions, language instructions or image-based goals) within the latent context module maps global scene-level signals into per-pare per-particle latent actions. For instance, given a language instruction, Nr learns to translate it into per-particle latent actions that drive the dynamics towards satisfying the instruction. When no external conditioning is provided, KG simply infers latent actions from past particle trajectories. When conditioned on goal image or a language instruction, sampling from the latent policy can be further utilized for planning in the particles space, as we demonstrate later in the experiments section. In Appendix A.4.3 we describe how the global action, language and image condition- ing mechanisms are implemented. Finally, we note that the proposed novel CONTEXT module is broadly applicable to general- purposs, non-obiect-centric architectures utilizing patch- based rep- resentations, as demonstrated in the experiments section. The CONTEXT module is illustrated in Figure 3

---

Published as a conference paper at ICLR 2026
DYNAMICS Fe: The dynamics module implements the VAE's autoregressive dynamics prior Pg(2* e ..... =)). It predicts the particles at the next timestep conditioned on the current particles and their corresponding latent actions provided by the context module:
F& ([[=).))) mm= +og: = =1
( M=D [6g))==1
It is implemented as causal spatio-temporal transformer, where particles are conditioned on their corresponding latent actions via AdaLN (Zhu et al. 2024). The module outputs distribution param- eters serving as the prior in the KL-divergence between the encoder posterior and dynamics prior
Unlike DDLP (Daniel& & Tamar 2024). LPWM retains all M encoded particles with their identities (patch origins) across timesteps, removing the need to track particles over time. This results in an implicit regime where particles can move certain region around their origin, balancing between two extremes: patch-based methods (e.g., VideoGPT (Yan et al. 202T)), where particles are fixed patches with evolving features, and object-centric particle models (Daniel & Tamar 2024), which track subset of free-moving particles with explicit attributes that traverse the entire canvas. We
discuss tracking limitations and the implications of this regime in Appendix A.4.4
Context
ge+1
Dynamics
Prior
Laher (hriar)
Lamet Zeevered Drnvenies (rsstertar
Fg
KL
ptrle'. .a!"3
Context Kgp
g!+1
o
X N
Add koom
Head
Add &
Por Encoder
Particle Peceder
Partidle Eweode
Partidl Decoder
ITpur Controkt
Fipur
Kgp
Raion
Ico2
Condition
Figure 3: Latent Particle World Model architecture. Left: Input frames are encoded into particle sets by the ENCODER and decoded back to images by the DECODER. The CONTEXT module then pro- cesses the particles sample latent actions, which are combined with the particles in the DYNAMICS module to predict next-step partiele states. Right: The CONTEXT module models the per-particle latent action distribution. During training, we use the latent inverse dynamics head, while at infer- ence, the latent policy is employed for sampling.
Optimization and Training Details: LPWM, following DDLP, is trained by maximizing poral ELBO, or minimizing the sum of reconstruction errors and KL-divergences, decomposed into static term for the first frame and a dynamic term for subsequent frames: LLPWM
-ET- ELBO(2, = 1e) = Cstatic + Ldynamic. The static term covers the single-frame setting, computing per-particle KL with respect to fixed priors, and adds regularization parency. The dynamic term includes KL losses for both latent actions and predicted future particles. All KL terms evaluated in closed-form. Both losses also include frame- wise reconstruction loss: pixel-wise MSE for simulated datasets, or MSE and LPIPS (Hoshen et al., (211)) for real-world data. key difference from DDLP is that KI contributions
particle trans-
masked using the particle transparency attribute (Lin et al. 2020a). so only visible particles affect the KI. loss. Full loss details are provided Appendix|A.4.. A.4.6 For all experiments, the dimension of the latent actions set dax
Models are optimized end-to-end with Adam (Kingma & Ba 2014) with a learning rate of 8 x 10 and
implemented in PyTorch (Paszke et al. 2017)) Hyperparameter details are in Appendix A.9 Code
and pretrained models are available at https:::. / tiggthhu.... com/ ta malalaatcchh. hteech/ppww. 1pwm
'The priors for the first timestep particles are fixed hyperparameters, consistent with DL.P's single-image
setup.

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTS
We design our experimental suite with the following key goals: (1) benchmark LPWM on uncon- ditional and conditional video prediction across real-world and synthetic datasets; (2) analyze the impact of LPWM's s design choices through ablation studies; and (3) demonstrate a practical imita- tion learning application on diverse, multi-object, long-horizon tasks and environments.
5.1 SELF-SUPERVISED OBJECT-CENTRE VIDEO PREDICTION AND GENERATION
We evaluate LPWM across multiple video prediction settings, including unconditional, action- conditioned, and language-conditionee scenarios. Additional demonstrations of image condition- ing and multi-view training, particularly for goal-conditioned imitation learning, presented in Section
Datasets:
we evaluate approach a diverse set of datasets, spanning real-world and simu- lated domains with varying dynamics and interaction densities. Simulated datasets include OBJ3D, featuring dense interactions and deterministic 3D physics (Lin et al., 2020a). PHYRE, a sparse in- teraction 2D physical reasoning benchmark with deterministic dynamics (Bakhtin et al. 2019). and Mario, a stochastic 2D dataset with dense interactions from expert Super Mario Bros gameplay videos (Smirnov et al., (021). Real-world datasets encompass robotic datasets such as Sketchy with sparse stochastic interactions (Cabi et al. 2019), BATR and Bridge featuring dense, stochas- tic robotic manipulation with and without language instructions (Ebert et al.) 2017a Walke et al.) 2023). and ManguageTable featuring language-guided, dense object rearrangements (Lynch) al. 2023 Unless stated otherwise, all datasets are trained 128 x 128 resolution. We pro- Vide a detailed description each dataset in Appendix A.7
Baselines: Our main baseline is non-object-centric patch-based dynamics VAE (DVAE) world model, where "particles" correspond to fixed grid patch embeddings matching the number of LPWM particles, M. This baseline shares the same architecture and parameter count LPWM and sup- ports identical conditioning but lacks explicit attribute modeling. video generation models using patch-based tokenization (Yan et al.; [202T] Yang al. (024b)). Un- like pre-trained or quantized patch embeddings, ours are learned end-to-end like LPWM's parti- cles, with higher latent dimension to offset the absence of object-centric structure. When ap- plicable, we also compare against recent object-centric video prediction methods, including the blot- PlaySlot (Villar-Corrales deterministic dynamics datasets, the patch- based G-SWM (Lin et al.) 2020a), the slot-based Slot- Former/OCVP (Wu al. 2022b Villar-Corrales et al. 2023), and the particle-based DDLP (Daniel
closely resembles large-scale
Behnke 2025) for weant-actioncoonditinn
tasks; and for
& Tamaa 2024.. Extended baseline details are provided in Appendix A.8
video prediction and datasets with deterministic dynamics, we report standard visual similarity meeticcsPPSNE SSIM (Wang al. 2004), and LPIPS (Zhang) ct al., 2018 compare generated sequences against ground truti For stochastic video genera- tion, compute the Fréchet Video Distance (FVD, Unterthiner et al. (2018) Hu (2023) to evaluate the distributional similarity between generated and real video sets.
Metrics: For iieet-action-condiiinnn
Results: LPWM outperforms all baselines on LPIPS and FVD metries across stochastic dynamic datasets under various conditioning settings (Table 2 It effectively preserves object permanence throughout generation (Figure[T) and models complex object interactions, unlike competing methods that exhibit blurring or deformation. LPWM also supports multi-modal sampling, producing diverse plausible rollouts from identical initial conditions (see Appendix A. and videos). Compared to the slot-based PlaySlot baseline, which suffers from object drifting and blurry reconstructions due global latent actions and limited number of slots, LPWM's s per-particle latent actions and low- dimensional representation scale effectively to many-object scenarios. DVAE, a cat---ojjecceettee baseline, performs well on synthetic data but lacks robustness on real-world datasets, highlighting the advantages of object-centric modeling. Finally, we demonstrate that a compact LPWW model trained on BAIR-64 matches larger video generation models in FVD (89.4, Table D) emphasizing how object-centric inductive biases enable superior modeling of object interactions beyond what
Our evaluation follows the DDLP protocol Daniel & Tamar 2024 using the open-source PIQA li-
brary (Rozet 2022 for perceptual metrics.
8

---

Published as a conference paper at ICLR 2026
scale alone achieves. Extended results in Appendix A.I0 and vidcos are available:
https:
//taldatech. qithub. 10/ 1pwn- pwb-
Daraet
Skotchy-U
Bh18-0
Nario-U
Setting
20,0, 6.p LMIK] FVD.
16
I.P
2:200.1
8.P:
PANKT
SSIMT
PSNRT
SSIMF
LAIS,
PVD]
PANKT
SSIME
TAPS
FVDJ
DVAE
2575133
01I3 14000
26122 0,000000 0URE
164.41
43.38
0:90
0:087-005
277.41
PlayStot
22,34300
080+2009
0.75+000
17.56+1.50
0,57+0005 0,43--1103
16,38+2778
0,68=01
0,314=009
LPWM (Ours)
23,41338
09110.06
0070100 85.45
25.0611.52 0.8910000 0.00211022
16391
27.50:15.2
0:95:1004
0.035-0022 195.95
Daraset
Sketcay-A
LanguageTaoler
Language77bbe--
Bri-dge-..
Settng
C. 44
20, a 1.p: 15
7: 20, 6 1,
(: 24,46 1.
PSNR
SSIM"
LPIPS
PSNRT
SSIMT
LPIPS
FVD1
FVD1
DVAE
25.33444.2
0.89
0.111=006
29.29:4..28 29.5+5.06
0.94:00.44 0.96:0004
0.02
26.78
146.85
LPWM (Ouis)
27.06+t4.22
0.89
0
0.03==0.22
15.96
47.78
Table Quantitative results on latent-action-condiiie conditioned (L) video prediction. FVD is reported for stochastic generation by sampling from the latent policy. is the training horizon, is the conditional frames inference, and p is the predicted frames at inference.
(U), centioncondittoone (A), and language-
Ablation Analysis: We conduct series of ablation studies our design choices, including global versus per per-particle latent actions, the dimensionality of the latent action vector, and types of posi- tional embeddings. As detailed in Appendix A. our results demonstrate that per-particle latent actions are essential for achieving strong performance and that the model is robust to latent ac- tion dimension near the effective particle dimension (6 + dobj). Furthermore, embedding positional information via AdaLN outperforms standard additive positional embeddings.
5.2 IMITATION LEARNING WITH PRE-TRAINED LPWM
Pre-training LPWM on actionless video datasets enables it predict video dynamics using latent actions, suggesting that these latent actions capture actionable information. Assuming access to ground-truth actions and that the latent actions effectively encode dynamics, learning a simple map- ping from latent actions to true actions may suffice to derive policy, which we verify next. For- mally, once paired video-action trajectories (Iu:T, 0o:T-1) are available (e-g., collected post-hoc), pre-trained LPWMs can be adapted to goal-conditioned imitation tasks using image-based goals (de- tails in Appendix [A.4.3). For each trajectory, image sequences are encoded by frozen LPWM to produce per-particle latent actions {= Jm= via the latent inverse dynamics head plr =
Policy learning maps latent actions to global actions using a simple, compact, two-layer attention pooling transformer (Haramati et al. 2024). At inference, given a goal image and rollout horizon k. LPWM is autoregressively unrolled for k 1 steps to generate particles and their k latent actions. The trained mapping network outputs a sequence of global actions, which are executed sequen- tially in the environment (Zhao et al. 2023);; this process is repeated until the maximum number of environment steps is reached. Implementation and training details are described in Appendix(A..
Task
EC Diffusion Policy
EC Diffuser
LPWM (Ours)
Task
GCIVL
HIQl
LPWM (Ours)
1 Cube
88.7 + 3
94.8 15
92.7 02.7
task1
84 14
80 1:6
100 1:0
Cubes
38.8
91.7 3
74
task?
18
19
3 Cubes
66.8 1 17
89.4 25
62.1 1 4.4
taak3
16
61 611
89 #9
Table 3: Imitation learning results (success rates) on PandaPush (left) and OGieech-Sccene (right). Environments: PandaPush challenges manipulation of up to three cubes observed from two cam- era views, while OGBench- Serencch-ceeese Scene evaluates long-horizon planning involving diverse objects such as drawers and buttons. We train a single LPWM per environment that encompasses all tasks, whereas for PandaPush, the baselines train separate policies for each task, effectively giving them an advantage by optimizing individually for each task.
Results: Table summarizes performance compared the two best baselines in each (full results in Tables 12 and 13p. On FandaPush. LPWM outperforms all baselines except EC Diffuser and matches its performance on the 1 1-cube task. We employ the multi-view LPWM variant here, model- ing particle dynamics from multiple views simultaneously. highlighting the framework's 's flexibility (see Appendix AA.455.. On OCBench, despite the challenge of highly suboptimal, unstructured "play" data hindering behavioral cloning, our method achieves strong results on tasks involving up to four atomic behaviors (task1 and task3), outperforming all baselines on these. For task4 and task5, all methods fail (with the exception of HIQL attaining 20% success rate task 4). Although we employ a relatively simple policy, LPWM demonstrates competitive performance, un- derscoring its potential for decision- applications. Figure 4 visualizes example imagined

---

Published as a conference paper at ICLR 2026
trajectory alongside environment execution OGBench, and rollout videos available the project website. Full results, baseline details and more visualizations are in Appendix A.100.4
Ihaagine
Gcal
Exccute
Figure 4: LPWM generated goal-conditioned imagined trajectories (top) and actual environment exccutions (bottom) through Icarned mapping actions OGBench- Scene. The imagined trajectories closely match the actual executions, demonstrating LPWM's predictive accuracy.
CONCLUSION
We introduced Latent Particle World Model (LPWM), advancing self-supervised object-centric world modeling for real-world data. LPWM discovers keypoints, bounding boxes, and masks in fully sulf-supe fashion, decomposing scenes into latent particles whose temporal evolution modeled by novel latent action and dynamics modules. This design enables state-of-the-art stochas- tie object-centric video generation while flexibly supporting action, language, and image condi- tioning, as well multi-view inputs. LPWM shows strong potential for decision-making tasks, including imitation learning as demonstrated here.
Limitations: LPWM presently depends on datasets exhibiting small camera motion and recurring scenarios, such as robotics or video games; not yet applicable to general-pune purpose large-scale video data. Future work could address scaling to diverse datasets, unified multi-modal condition- ing (e.g., simultaneous action, language, and image signals), and integration with explicit reward modeling for reinforcement learning.
10

---

Published as a conference paper at ICLR 2026
7 ETHICS STATEMENT
This work introduces a video generation model evaluated on both simulated and real-world robotics datasets. As discussed in the limitations above, our model is demonstrated on general-purpost video data or applied to sensitive content. All datasets used are either publicly available or collected in controlled, non-sensitive environments. We do not foresee ethical or societal risks arising from this work as presented; however, as with any generative model, future extensions to broader or less- controlled domains should carefully consider potential misuse and ensure responsible deployment.
REPRODUCIBILITY STATEMENT
We strive facilitate reproducibility and transparency self-supervised object-centric world mod- cling. To this end, we provide code excerpts throughout the appendix. along with extended imple- mentation details and the full list of hyperparameters in Appendix Source code and pre-trained model checkpoints available. These resources aim to make it straightforward for others to build upon our framework.
ACKNOWLEDGMENTS
This work is supported by ONR MURI N00014-24-1-2748 and ERC grant Bayes-RL (101041250). Views and opinions expressed those of the authors only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency (ERCFA). Neither the European Union nor the granting authority be held responsible for them.
REFERENCES
Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. Advances in Neural Information Processing Systems, 37:58757-58799 2024.
Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan. Fitvid: Overfitting in pixel-level video prediction. arXiv preprint arXiv:2106.13192 2021.
Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. Phyre: new benchmark for physical reasoning. Advances Neural Information Processing Systems, 32, 2019.
Irina Barnaell,, Simone Viganò, Daniel Reznik, Patrick Haggard, and Christian F Doeller. Hippocampal-entorhinnl cognitive maps and cortical motor system represent action plans and their outcomes. Nature Communications, 16(1):4139, 2025.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems, 29, 2016.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Macicj Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023.
Alexandre Brown and Glen Berseth. Segdac: Segmentation-drinn driven actor-critic for visual reinforce- ment learning. arXiv preprint arXiv:2508.09322 2025.
Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Barker-Holder Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative inter- active environments. In Forty-first International Conference on Machine Learning, 2024.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and represen- tation. arXiv preprint arXiv: 1901v:190..11900 2019.

---

Published as a conference paper at ICLR 2026
Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, et al. Scaling data- driven robotics with reward sketching and batch reinforcement learning. arXiv preprint arXiv: /909. 12200, 2019.
Jun Cen, Chaohui Yu, Hangjie Yuan, Yuming Jiang, Siteng Huang, Jiayan Guo, Xin Li, Yibing Song, Hao Luo, Fan Wang, et al. Worldvla: Towards autoregressive action world model. arXiv preprint arXiv:2506.1533, 2025.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation complex datasets. arXiv preprint arXiv:1907.06571, 2019.
Jeremy A Collins, Loránd Cheng, Kunal Aneja, Albert Wilcox, Benjamin Joffe, and Animesh Garg. Amplify: Actionless motion priors for robot learning from videos. arXiv preprint arXiv:2506. 14198, 2025.
Eric Crawford and Joelle Pincau. Spatially invariant unsupervised object detection with convolu- tional neural networks. In AAAI, 2019.
Zichen Cui, Hengkai Pan, Aadhithya Iyer, Siddhant Haldar, and Lerrel Pinto. Dynamo: In-domain dynamics pretraining for visuo-motor control. Advances Neural Information Processing Sys- tems, 17:33933-339611 2024.
Tal Daniel. Ddlp: Unsupervised object-centric video prediction with deep dynamic latent particles. com/taldatech//dll 2024.
Tal Daniel and Aviv Tamar. Unsupervised image representation learning with deep latent parti- cles. In Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. 4644 4665. PMLR, 17-23 Jul 2022a.
Tal Daniel and Aviv Tamar. Unsupervised image representttinnnnnoonn learning with deep latent particles. https://githui: http:///ww...aaaaeeeeeeeaaaaaaar
2022b.
Tal Daniel and Aviv Tamar. DDLP: Unsupervised object-centric video prediction with deep dynamic latent particles. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
Aram Davtyan, Sepehr Sameni, and Paolo Favaro. Efficient video prediction via sparsely condi- tioned flow matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23263-23274, 2023.
Antoine Dedieu, Joseph Ortiz, Xinghua Lou, Carter Wendelken, Wolfgang Lehrach, J Swaroop Guntupalli, Miguel Lazaro-GGediile and Kevin Patrick Murphy. Improving transformer world models for data-efficient rl. arXiv preprint arXiv:2502.01591, 2025.
Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In International conference on machine learning, pp. 1174-1183. PMLR, 2018.
Aniket Didolkar, Andrii Zadaianchuk, Anirudh Goyal, Mike Mozer, Yoshua Bengio, Georg Mar- tius, and Maximilian Seitzer. Zero-shot object-centric representation learning. arXiv preprint 2wXiv:2408.09162 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. CoRL., 12(16):23, 2017a.
12

---

Published as a conference paper at ICLR 2026
Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. In CoRL, pp. 344-356, 2017b.
Ashley Edwards, Himanshu Sahni, Yannick Schroecker, and Charles Isbell. Imitating latent policies from observation. International conference on machine learning, pp. 1755-1763, PMLR, 2019.
Gamaleldin Fathy Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael Cur- tis Mozer, and Thomas Kipf. SAVi++: Towards end-to-end object-centric learning from real- world videos. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Gener- ative scene inference and sampling with object-centric latent representations. arXiv preprint arXiv:1907.1305: 2019.
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object representations without iterative refinement. arXiv preprint arXiv:2104.09958. 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- lion, pp. 12873-12883, 2021.
Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learn- ing as goal-conditioned reinforcement learning. Advances in Neural Information Processing Sys- tems, 35:35603-3565 356200, 2022.
Stefano Ferraro, Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, and Sai Rajeswar. Represent- ing positional information in generative world models for object manipulation. arXiv preprint arXiv:2409. 12005, 2024.
Stefano Ferraro, Pietro Mazzaglia, Tim Verbelen, and Bart Dhoedt. Focus: object-centric world models for robotic manipulation. Frontiers in Neurorobotics, 19:1585386, 2025.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. Advances in neural information processing systems, 29, 2016a.
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference Robotics and Automation (ICRA), pp. 512-519. IEEE, 2016b.
Shenyuan Gao, Siyuan Zhou, Yilun Du, Jun Zhang, and Chuang Gan. Adaworld: Learning adaptable world models with latent actions. arXiv preprint arXiv:2503. 18938, 2025.
Xiaojie Gao, Yueming Jin, Qi Dou, Chi- Ming Fu, and Pheng-Amn Heng. Accurate grid keypoint learning for efficient video prediction. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 9908-5915 IEEE, 2021.
Ofri Goldenberg, Tal Daniel, Dafei Xiao, Yael Shalev Ezra, and Yoav Shechtman. Physics-informed generative model for 3d localization microscopy. bioRxiv, pp. 2025-07, 2025.
Xinrui Gong, Oliver Hahn, Christoph Reich, Krishnakant Singh, Simone Schaub-Mes Meyer, Daniel Cremers, and Stefan Roth. Motion-refined dinosaur for unsupervised multi-object discovery. arXiv preprint 2509 2025.
Melvyn A. Goodale and A. David Milner. Separate visual pathways for perception and action. Trends in Neurosciences, 15:20-25, 1992.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. International Conference on Machine Learning, pp. 2424-22244 2433. PMLR, 2019.
David Ha and Jürgen Schmidhuter. World models. arXiv preprint arXiv: /803. 10/22, 2018.
13

---

Published as a conference paper at ICLR 2026
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream control: Learn- ing behaviors by latent imagination. In International Conference on Learning Representations, 2020a.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint arXiv:2010.02119 2020b.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04 2023.
Dan Haramati, Tal Daniel, and Aviv Tamar. Entity-centric reinforcement learning for object ma- nipulation from pixels. In The Twelfth International Conference on Learning Representations, 2024.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. bAE: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6855 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances neural information processing systems, 35:8633- 8646, 2022.
Tobias Höppe, Arash Mehrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi. Diffusion models for video prediction and infilling. arXiv preprini arXiv:2206.0777 07696, 2022.
Yedid Hoshen, Ke Li, and Jitendra Malik. Non-adversarial image synthesis with generative la- tent nearest neighbors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2019.
Junyao Hu. Common metries video quality. bttps::/github... http::///tttaub..cco///////aaaap metrics on cice___o__viee__uu__e____e haadee_quality/ 2023.
Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations. In Forty-second International Conference on Machine Learning, 2025.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances in neural information processing systems, 28:2017-2025, 2015.
Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of object landmarks through conditional image generation. In Proceedings of the 32nd International Con- ference on Neural Information Processing Systems, pp. 4020-4031, 2018.
Joel Jang, Seonghyeon Ye. Zongyu Lin, Jiannan Xiang, Johan Bjorck, Yu Fang, Fengyuan Hu, Spencer Huang. Kaushil Kundalia, Yen-Chen Lin, et al. Dreamgen: Unlocking generalization in robot learning through video world models. arXiv preprint arXiv:2505. 12705, 2025.
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, 2022.
Youngjoon Jeong, Junha Chun, Soonwoo Cha, and Taesup Kim. Object-centric world model for language-guided manipulation. arXiv preprint arXiv:2503.06170, 2025.
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. arXiv preprint arXiv: /tti::/9000238.. 2019.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
14

---

Published as a conference paper at ICLR 2026
Nikola Jukić, Tim Lebailly, and Tinne Tuytelaars. Object-centric pretraining via target encoder bootstrapping. In The Thirteenth International Conference on Learning Representations, 2025.
Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matt Botvinick, Alexander Lerchner, and Chris Burgess. Simone: View-invariant, lemporally-abstraated object representations via unsupervised video decomposition. Advances in Neural Information Process- ing Systems, 34:201 34:20146-20155 46-20159, 2021.
Ioannis Kakogeorgiou, Spyros Gidaris, Konstantinos Karantzalos, and Nikos Komodakis. Spot: Self-training with patch-order permutation for object-centric learning with autoregressive trans- formers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 22776-22786, 2024.
Yunji Kim, Seonghyeon Nam, In Cho, and Seon Joo Kim. Unsupervised keypoint learning for guiding class-conditional video prediction. Advances in neural information processing systems, 32, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 12 2014.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Thomas Kipf. Gamaleldin Elsayed. Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. arXiv preprint arXiv:2111.12594 2021.
Albina Klepach, Alexander Nikulin, Ilya Zisman, Denis Tarasov, Alexander Derevyagin, Andrei Polubarov, Nikita Lyubaykin, and Vladislav Kurenkov. Object-centric latent action learning. arXiv preprint arXiv:2502.00960) 2025.
Jannik Kossen, Karl Stelzner, Marcel Hussing, Claas Voelcker, and Kristian Kersting. Struc- tured object-aware physics prediction for video modeling and planning. arXiv preprint 1rX1v:1910.022255 2019.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In International Conference on Learning Representations, 2022. URL https:
openreview wwtpirevveew.ee/fforrooee mer/forum" (emm::ddddddnnammm
Tejas Kulkarni, Ankush Gupta, Catalin lonescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised learning of object keypoints for perception and control. arXiv preprint arXiv:19066 1883, 2019.
Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. Cevs: Context-aware comtxxtla synthesis. Advances in Neural Information Processing Systems, 34:14042-14055, 2021.
controllable video
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:1804.00 0155000..11,,,... 2018.
Seungjae Lee, Yibin Wang, Haritheja Etukuru, H Jin Kim, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Behavior generation with latent actions. In Forty-first International Conference on Machine Learming. 2024.
Yunzhu Li, Antonio Torralba, Animashree Anandkumar, Dieter Fox, and Animesh Garg. Causal discovery in physical systems from videos. arXiv preprint arXiv:2007.00631, 2020.
Zhixuan Lin, Yi-Fu Wu, Skand Peri, Bofeng Fu, Jindong Jiang, and Sungjin Ahn. Improving gen- erative imagination in object-centric world models. In International Conference Machine Learning. pp. 6140-6149, PMLR, 2020a.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang. and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. arXiv preprint arXiv:2001.02407 2020b.
15

---

Published as a conference paper at ICLR 2026
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centrie learning with slot atten- tion. arXiv preprint wwX.v:2006.1505: 2020.
Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan. Transformation-Bee based adversarial video prediction on acage-scale data. arXiv preprint arXiv:2003.0400:: 2020.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pp. 1113- 1132. Pmlr, 2020.
Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters, 2023.
Xin Ma, Yaohui Wang, Xinyuan Chen, Gengyun Jia, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Qiao. Latte: Latent diffusion transformer for video generation. Transactions on Machine Learning Research, 2025. ISSN 2835-8856
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin. Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance gpu-based physics simulation for robot learning, 2021.
Willi Menapace, Stéphane Lathuilière, Sergey Tulyakov, Aliaksandr Siarohin, and Elisa Ricci. Playable video generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10061-10070, 2021.
Willi Menapace, Stéphane Lathuilière, Aliaksandr Siarohin, Christian Theobalt, Sergey Tulyakov, Vladislav Golyanik, and Elisa Ricci. Playable environments: Video manipulation in space and time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1884-359,, 2022.
Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world mod- els. In The Eleventh International Conference Learming Representations, 2023.
Vincent Micheli, Eloi Alonso, and François Fleuret. Efficient world models with context-aware tokenization. International Conference Machine Learning, pp. 35623-35638. PMLR, 2024.
Cheol-Hui Min, Jinseok Bae, Junho Lee, and Young Min Kim. Gatsbi: Generative agent-centric spatio-temporal object interaction. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 3074-3083, 2021.
Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin P Murphy, and Honglak Lee. Unsupervised learning of object structure and dynamics from videos. Advances in Neural Infor- mation Processing Systems, 32, 2019.
Malte Mosbach, Jan Niklas Ewertz, Angel Villar-Corrales, and Sven Behnke. Sold: Slot object- centric latent dynamics models for relational manipulation learning from pixels. arXiv preprint arXiv:2410.08822. 2024.
Akihiro Nakano, Masahiro Suzuki, and Yutaka Matsuo. Interaction-bassd disentanglement of en- tities for object-centric world models. In The Eleventh International Conference on Learning Representations, 2023.
Charlie Nash, João Carreira, Jacob Walker, Iain Barr, Andrew Jaegle, Mateusz Malinowski, and Peter Battaglia. Transframer: Arbitrary frame prediction with generative models. arXiv preprint arXiv:2203.09499, 2022.
Matthias Nau, Joshua B Julian, and Christian F Doeller. How the brain's 's navigation system shapes visual experience. Trends in cognitive sciences, 22(9):810-825 2018.
16

---

Published as a conference paper at ICLR 2026
Iman Nematollahi, Branton DeMoss, Akshay L Chandra, Nick Hawes, Wolfram Burgard, and Ing- mar Posner. Lumos: Language-conditiong conditiond imitation learning with world models. arXiv preprint 1:Xi::2503.103700 2025.
Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine Hiql: Offline goal-
conditioned rl with latent states as actions. Advances in Neural Information Processing Systems, 36:34866-34899 2023.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. In International Conference on Learning Representations (ICLR), 2025.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
William Peehles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4195-4205, 2023.
Martin John Pickering and Andy Clark. Getting ahead: forward models and their place in cognitive architecture. Trends in Cognitive Sciences, 18:451-4560 2014.
Carl Qi, Dan Haramati, Tal Daniel, Aviv Tamar, and Amy Zhang. EC-diffuser: Multi-object ma- nipulation via entity-centric behavior generation. The Thirteenth International Conference on Learming Representations, 2025.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training. 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21 21(140):1-67, 2020.
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent video transformer. arXiv preprint arXiv:2006. 10704, 2020.
Zhongwei Ren, Yunchao Wei, Xun Guo, Yao Zhao, Bingyi Kang, Jiashi Feng, and Xiaojie Jin. Vide- oworld: Exploring knowledge learning from unlabeled videos. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 29029-29033 2025.
François Rozet. Pytorch image quality assessment.
com/
Francois-rozet/piie 2022.
Alexander Rubinstein, Ameya Prabhu, Matthias Bethge, and Seong Joon Oh. Are we done with object-centric learning? arXiv preprint arXiv:2504.07002. 2025.
Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetić, Mario Lučić, Leonidas Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. arXiv preprint arXiv:2206. 220.009220.. 2022.
Nedko Savov, Naser Kazemi, Mohammad Mahdi, Danda Pani Paudel, Xi Wang, and Luc Van Gool. Exploration-driven generative interactive environments. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 27597-27607, 2025.
Dominik Schmidt and Minqi Jiang. Learning to act without actions. arXiv:2312.10812, 2023.
arXiv preprint
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, and Francesco Lo- catello. Bridging the gap to real-world object-centric learning. In The Eleventh International Conference on Learning Representations, 2023.
Gaurav Shrivastava and Abhinav Shrivastava. Continuous video process: Modeling videos as contin- uous multi- multi-diimensonnal processes for video prediction. arXiv preprint arXiv:2412.0492. 2024.
17

---

Published as a conference paper at ICLR 2026
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns compose. arXiv preprint arXiv:2110.11405. 2021.
Gautam Singh, Yeongbin Kim, and Sungjin Ahn. Neural block-slot representations. arXiv preprint arXiv:2211.01177, 2022a.
Gautam Singh. Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for com- plex and naturalistic videos. arXiv preprint arXiv:2205. /4065, 2022b.
Gautam Singh, Yeongbin Kim, and Sungjin Ahn. Neural systematic binder. In The Eleventh Inter- national Conference on Learning Representations, 2023.
Dmitriy Smirnov, Michael Gharbi, Matthew Fisher, Vitor Guizilini, Alexei A. Efros, and Justin Solomon. MarioNette: Self-super sprite learning. Conference on Neural Information Pro- cessing Systems, 2021.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In NIPS, 2015.
Yeon-Ji Song, Suhyung Choi, Jaein Kim, Jin-Hwa Kim, and Byoung-Tai Tak Zhang. Unsupervised dynamics prediction with object-centric kinematics. arXiv preprint arXiv:2404. 0..4404.842., 2024.
Aleksandar Stanić and Jürgen Schmidhuber. R-sqair: relational sequential attend, infer, repeat.
arXiv preprint arXiv:191000523: 2019.
Jennifer Sun, Serim Ryou, Roni Goldshmid, Brandon Weissbourd, John O Dabiri. David Anderson, Ann Kennedy, Yisong Yue, and Pietro Perona. Self-supervised keypoint discovery in behavioral videos. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2171-21888 2022.
Yang Tian, Sizhe Yang, Jia Zeng, Ping Wang, Dahua Lin, Hao Dong, and Jiangmiao Pang. Pre- dictive inverse dynamics models are scalable learners for robotic manipulation. arXiv preprint 10Xiv:24/2.15109, 2024.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint 181.00111, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Rishi Veerapaneni, John D Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. In Conference on Robot Learning, pp. 1439-1456. PMLR, 2020.
Angel Villar-Corrales Playslot: Learning inverse latent dynamics for controllable object-centric video prediction and planning.
2025.
Angel Villar-Corrales and Sven Behnke. Playslot: Learning inverse latent dynamics for controllable object-centric video prediction and planning. In International Conference on Machine Learning (ICML), 2025.
Angel Villar- Corrale,,,,,,,, Ismail Wahdan, and Sven Behnke. Object-centric video prediction via de- coupling of object dynamics and interactions. In Internation Conference on Image Processing (ICIP), 2023.
Angel Villar-Corrales. Gjergi Plepi, and Sven Behnke. Object-centric image to video generation with language guidance. arXiv preprint 15X2v.2502.11655, 2025.
Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn. Xunyu Lin, and Honglak Lee. Learning to generate long-term future via hierarchical prediction. international conference on machine learning, pp. 3560-3569. PMLR, 2017.
18

---

Published as a conference paper at ICLR 2026
Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoe V Le, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Vikram Voleti, Alexia Selicoeuur-aattinenr and Chris Pal. Mevd-masked conditional video diffusion for prediction, generation, and interpolation. Advances in neural information processing systems, 35:23371-233663 2022.
Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruuhh, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning (CoRL), 2023.
Jacob Walker, Ali Razavi, and Aäron van den Oord. Predicting video with vqvae. arXiv preprint arXiv:2103.01150, 2021.
Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal goal-reaching reinforce- ment learning via quasimetric learning. In International Conference on Machine Learning, pp-
36411-36430. PMLR, 2023.
Xingrui Wang, Xin Li, Yaosi Hu, Hanxin Zhu, Chen Hou, Cuiling Lan, and Zhibo Chen. Tiv- diffusion: Towards object-centric movement for text-driven image video generation. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 7988-7996, 2025a.
Yunbo Wang, Haixu Wu, Jianjin Zhang, Zhifeng Gao, Jianmin Wang, Philip Yu, and Mingsheng Long. Predrun: A recurrent neural network for spatiotemporal predictive learning. IEEE Trans- actions Pattern Analysis and Machine Intelligence, 2022.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibilitt to structural similarity. IEEE transactions on image processing, 13(4):600- 612. 2004.
Zizhao Wang, Kaixin Wang, Li Zhao, Peter Stone, and Jiang Bian. Dyn-o: Building structured world models with object-centric representations. arXiv preprint arXiv:2507.03298, 2025b.
Marissa Weis, Kashyap Chitta, Yash Sharma, Wieland Brendel, Matthias Bethge, Andreas Geiger, and Alexander S Ecker. Benchmarking unsupervised object representations for video sequences. J. Mach. Learn. Res., 22:183-1, 2021.
Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Comput., 1:270-280. jun 1989.
Bohan Wu, Suraj Nair, Roberto Martin,Martin, Li Fei-Fei, and Chelsea Finn. Greedy hierarchi- cal variational autoencoders for large-scale video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2318-2328 2021a.
Chenfei WU, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre-training for neural visual world creation. In European conference on computer vision, pp. 720-736. Springer, 2022a.
Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. Generative video transformer: Can objects be the words? In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11307-11318. PMLR, 18-24 Jul 2021b.
Ziyi Wu, Nikita Dvorikk, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsupervised visual dynamics simulation with object-centric models. arXiv preprint 2x.iv:2210.058666 202.b.
Amber Xie, Oleh Rybkin, Dorsa Sadigh, and Chelsea Finn. Latent diffusion planning for imitation learning. arXiv preprint arXiv:2504. 16925, 2025.
Zhengtong Xu, Qiang Qiu, and Yu She. Vilp: Imitation learning with latent video planning. JEEE Robotics and Automation Letters, 2025.
19

---

Published as a conference paper at ICLR 2026
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint vrXiv:2104.0015/ 2021.
Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. arXiv preprint arXiv:2310.06114, 1(2)6, 2023.
Mingyu Yang, Junyou Li, Zhongbin Fang, Sheng Chen, Yangbin Yu, Qiang Fu, Wei Yang, and Deheng Ye. Playable game generation. arXiv preprint arXiv:2412.00887, 2024a.
Zhuoyi Yang. Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint 2iti:::2000000072,, 2024b.
Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Se June Joo, Jianwei Yang, Baolin Peng, Ajay Man- dlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, Lars Liden, Kimin Lee, Jianfeng Gao, Luke Zettlemoyer, Dieter Fox, and Minjoon Seo. Latent action pretraining from videos. In The Thir- teenth International Conference on Learning Representations, 2025.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint 1rXiv:1910.01442, 2019.
Jiwen Yu, Yiran Qin, Xintao Wang, Pengfei Wan, Di Zhang, and Xihui Liu. Gamefactory: Creating games with generative interactive videos. arXiv preprint Stti::2500.08322. 2025.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming- Ming-Hsuan Yang. Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. arXiv preprint arXiv:2212.05 2022.
Andrii Zadaianchuk, Georg Martius, and Fanny Yang. Self-supervised reinforcement learning with independently controllable subgoals. In Conference on Robot Learning, pp. 384 394. PMLR, 2022.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Weipu Zhang, Gang Wang, Jian Sun, Yetian Yuan, and Gao Huang. Storm: Efficient stochastic transformer based world models for reinforcement learning. Advances Neural Information Processing Systems, 36:27147-27166 2023.
Weipu Zhang, Adam Jelley, Trevor Mclnroe, and Amos Storkey. Objects matter: object-centric world models improve reinforcement learning in visually complex environments. arXiv preprint arXiv:2501. 16443, 2025.
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 13705, 2023.
Fangqi Zhu, Hongtao Wu, Song Guo, Yuxiao Liu, Chilam Cheang, and Tao Kong. Trasim: Learning interactive real-robot action simulators. arXiv preprint arXiv:2406. 14540, 2024.
Daniel Zoran, Rishabh Kabra, Alexander Lerchner, and Danilo J. Rezende. Parts: Unsupervised seg- mentation with slots, attention and independence maximization. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pp. 10439-10447, October 2021.
20

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 LARGE LANGUAGE MODELS (LLMS) ASSISTANCE DISCLOSURE
We used large language models (LLMs) to assist polishing the writing and improving grammar on sentence level. All suggestions were reviewed and approved by the authors.
A.2 PRELIMINARIES: SPATIAL SOFTMAX (SSM) AND SPATIAL TRANSFORMER NETWORK (STN)
We first review two core building blocks of the Deep Latent Particles (DLP, Daniel & Tamar 2022a 2024) model, an object-centric latent representation with disentangled attributes, before formally defining DLP. The Spatial Softmax (SSM. Jakab et al., 2018, Finn et al. 2016b) is commonly used for self-supervised extraction of keypoints from feature maps. The Spatial Transformer Network (STN, Jaderberg al., 2 provides a differentiable mechanism for spatial transformations: given set of keypoint locations, it enables the model to extract localized patches from the image and to recompose the image from such patches using parameterized affine transformations.
Spatial Softmax (SSM). The spatial softmax, also known as the soft-argmax, can be viewed as dif- ferentiable relaxation of the argmax operator: rather than selecting a single coordinate, it computes the expected coordinate under a probability distribution. Given a heatmap H R x typically obtained from CNN feature maps of an image or patch, the softmax function is applied over the spatial dimensions normalize H into probability distribution H. Each entry haj H) then represents the probability of a keypoint being located at position (i.j). From this distribution, the mean coordinate (Na, Hyy and the covariance values of og. and day: following Sun et al. 2022 are computed:
Mx Σ I, H(i,)),
My Σ Vi Σ H(i,)),
Σ (xi px) )z) (H(())))
og Σ (ys nby HH((i,)
0
Σ (x) - pz) - (i).).
azg
Here, E2H(i, H(i.5) and Σ, H(i.j) correspond to the marginal distributions along each axis. The co- ordinate grids {xt) and {ı,} are defined as normalized continuous values, typically spanning [-1, 1] across the width and height, rather than raw pixel indices. The process is illustrated in Figure 5 and provide PyTorch-ttyle code in Figure 6 Intuitively, sharply peaked activations yield low variance values, typically corresponding to salient structures such as objects, corners, or edges. In contrast, broadly spread activations tend to produce high covariances, which characteristic of background or less informative regions. Thus, covariance values provide a natural criterion for de- tecting and filtering meaningful locations. Since the SSM is fully differentiable, the heatmaps are optimized end-to-end through the reconstruction objective, encouraging the model to attend to the most informative regions of the scene.
Spatial Transformer Network (STN). A Spatial Transformer Network (STN; Jaderberg et al., 2015) is learnable module that performs spatial transformations on input data fully differen- tiall manner. Such transformations include translation, scaling, rotation, and more general warping. In our context, we focus on the core differentiable operation underlying STNs: grid sampling.
Given an input image I E CxCxx an affine transformation matrix
a11
tr
=
@21
022 by
21

---

Published as a conference paper at ICLR 2026
SSM
H
(1,30)))1(300)
CNN
0zy(3()
03(1))
O (3()
Figure 5: Spatial-softmax. Given heatmap H HxW the softmax function is applied over the spatial dimensions normalize H into a probability distribution H. These values then used to compute the expected coordinate values for each axis, and their covariance.
def praticl__oftmox theotmap, kp_range= (-1, 111:
Spatiel Softmox Marginalization Args:
keypoint detection.
heatmaps [3, K, H, H] input heatmaps
kp_raage: coordinate zange for nermalization (default: (-1, 11)
Returns:
kpt 2) expected keypoint coordinatas x]
vart [B, K, 3] variance estimates [var_y, var_X, cov_yx]
10
11
batch_size, n_ height, width hhatmap.shapa
12
13
# 1. Platten and apply softmax
14
logizs heatmap. view (batch_aize, n_kp, -1) + [H, 8, B+M]
torch, fomax (logita, dim=-1)
16
scores accres. lbatch_eize, n_kp, height, width) [B, 3, E, X]
18
# 2. Create coordinate axea
19
y_axis torch.1insped. Linspace (kg_range(01 kp kp_range(11, height,
devicerscores.devi.e .type_as (scores) _xis torch, Linapace (kp_ (0), kp_ kp_range(1], [1], width, device-scores.der
21
_tyee_a tacores)
23
Marginalize over dimensions
25
sm__ coores.aun 1) [B, K, R marginalize over width SE_M soores.sumddime
26
[B, E, R]
marginelize ever height
28
# 4. Compute expected coordinates
kp_y Lorch, sumtenJh y_axis, dim=-1)
X]
30
kp_x torch. sum x_xxis, dim -1 [B, X]
kp torch. ([kp_y, kp_x), dim--1) [H, x, 2]
32
33
# 5. Compute variance: Var(X) EIX12] (BIx1)
y_ (scores (y_axta..umquuoozet
(3) -1)) [B, R]
35
var_y = ly_aq kp_y ** 21 ml
37
x_$9 (scores (x_exis. uns.uuueeueeee222 ))) sum (-2, -1)) [8, K]
38
var_x [x_9q kp_x a 2).cllap_min(le--)
39
40
# 6. Compute covariances dev (X [[Xx] - [[x]8[Y)
xy (scorea ly_axta. unsqueeze un
(-21) aum (-2, -112
42
cov_yx xy kp_y x kp_x [B, K]
var torch. (var_y, var_x, cov_yxl, dimm-1) 44
[B, K, 3]
Figure 6: PyTorch-style code of Spatial Softmax for keypoint detection.
22

---

Published as a conference paper at ICLR 2026
maps normalized target coordinates (x',8" E 1-1, in the output grid to source coordinates (x")y") in the input:
2
x
A
y
The resulting sampling grid G {(x", y²"} specifies where fetch pixels from the input image.
The grid sampling operation then computes the transformed image î via bilinear interpolation:
Î(x', 34) I(t.)) max(0, f.j
|x" 51) max 1
-Iy" -1))-
This interpolation ensures that the transformation is diffcrentiable with respect to both the sampling locations and the input image. DLP, this mechanism is used in two ways:
1. Encoding:
extracting glimpses from the input image, parameterized by each particle's attributes (position and scale).
2. Decoding:
stitching back the reconstructed image from the decoded particle glimpses.
Thus, the encoder and decoder construct particle-specific affine transformations, generate the corre- sponding sampling grids, and warp the images or patches via differentiable grid sampling. Impor- tantly, both grid generation and sampling are natively implemented in modern frameworks such as PyTorch (Paszke et al. 2017). A minimal PyTorch-ttyle code snippet provided in Figure
def spatiel_ttrensfon image, 2_pos, s_scale, oat_dime, inversenrelse, epa-le-9, padding_modere "zeros"
}:
Differentiable spatial transform using grid sampling.
Args:
image: [B, C, H, K] input tensor.
s_post [B, 2] position (tx, ty).
10
t_scales (8, 2] scale factors (sx, syl.
11
out oum_dims: (8, e, H_out, [_out] desired output size.
inverse: if False (default), encoding transform Himage glimpse)
13
15 True, decoding transtorm (glispse -> image).
eps: small constant for padding mode: padding for out-of-bounds sampling.
stability.
15
17
# 1. Construct 2x3 affine transform matrix
18
theta = toroh (image. 2, 3, dovicosimago.devien
20
scaling thetal:, 0] s_scalel;,
(e_scale(:, eps)
22
thetal;, 1, 1] s_scalel;, 0] if not inverse else 1 / (s_scalel:, 0] aps)
24
A translation
25
theta(:, 4, 2] ?_poa(:, 1] if not inverse else -z_pos(:, 1) / (z_acalej;, 1] + ops) 2_poali, if
thetali,
-R_posli, (s_scale eps)
27
#.2. Constsuct grid apply bilinear sampling
29
grid = saaffino_grridthhee return F.grie_aanple (image, grid, modew' bilder"bllnear" Lligg_cozneess-allee paddinggnndeepadddinn.....o
Sorch.Size
al coigg_corneor
30
31
Figure 7: PyTorch-style code for differentiable warping with encoding (image-rglimpse) and de- coding (glimpse-image
A.3 DEEP LATENT PARTICLES (DLP)
We now formally define deep latent particles (DLP) (Daniel & Tamar 2024) and their attributes. In the following sections, describe how they are learned and used for dynamics modeling.
foreground latent particle, illustrated in Figure g defined as
[Zp 2s. Zds Zt, 24] TR6+da,
2ig
23

---

Published as a conference paper at ICLR 2026
where each component corresponds to disentangled stochastic attribute: position 2p. scale 2 depth 2d, transparency 2t, and visual features 2y. The background is represented by a single abstract particle Tng, fixed at the center of the image and parameterized only by dng background features. Formally,
Dog ~ N (Hug. oog) E RRd
We now detail the role of each attribute:
Position 2p E 2 encodes the spatial location of the particle, i.e., its (x. coordinates within [-1.]] Following object-centric models such as G-SWM (Lin et al. 2020a) and SCALOR (Jiang al. 2019), Zp modeled Gaussian (Mpp oy) In DLP, the prior for 2p derived from SSM over patches, which ensures that positions carry explicit spatial meaning.
Scale 2g E R²: defines the particle's height and width (i.e., bounding box size). modeled as (Hs, 0g) and passed through Sigmoid activation to constrain values to [0, 1].
Depth za specifies the relative ordering of particles when reconstructing the scene, and mod- eled as ((P,, aÿ). While determines the compositing order of decoded objects, it does not necessarily correspond to physical 3D depth, since DLP is trained on monocular RGB inputs.
Transparency E [0, 1]: controls whether and to what extent particle contributes to the recon- structed image. value of 2, 0 corresponds to a fully transparent (inactive) particle, 2, = 1 to fully visible particle, and intermediate values capture partial visibility. Unlike many object-centric models that use a Bernoulli "presence" variable, we model transparency with a Beta distribution, 2, (etaaa,, be). This has two key advantages: (1) the Beta distribution is continuous and repa- rameterizable, enabling stable gradient-based optimization without discrete relaxations, and (2) naturally supports intermediate values, making possible to represent partially occluded or semi- transparent objects. Moreover, like Gaussian and Bernoulli, the Beta distribution has a closed-form KL divergence that can be easily plugged in the VAE'S objective function.
Visual features 2f E Rdas, Zbg E Rd encode the appearance of foreground particles, i.e., the keypoint's surrounding region, and the background, respectively. Both modeled as Gaussian latents: 2f ~ NN of) and Fbg N(bbg, og)).
Position (KP)
Zp E R²
Scale (BB)
Z, E R
Transparency
EΦ 2y E Rdob) Appearance Features
2, E [00
Depth
Za E IR
Figure 8: A Deep Latent Particle. Each component of latent particle corresponds to disentangled stochastic attribute: position žp, scale %., depth 2a, transparency 21, and visual features 2y. Further details are provided in Section (3 of the main text and Section A.3 of the Appendix.
A.4 LATENT PARTICLE WORLD MODELS EXTENDED METHOD DETAILS
In this section, we provide a detailed overview of our method.
Our goal is to design world model, i.e., dynamics model F(Jor-1,e)) fy-r4r-1 that takes in a sequence of image observations In.T-1 E RTxCxHXVV (a video). where is the number of im- age channels (typically 3 for RGB) and H and W the height and width of the images respectively,
24

---

Published as a conference paper at ICLR 2026
and optionally sequence of conditioning signals (e.g., action sequence or language instruction), and prediets sequence of future observations autoregressively Ir:r+r-1 RTXCxH*W. We note that our world model need not be conditioned on c and can be trained only with videos, i.e. = 0, as explain later in the section. As the original image pixel space is high-dimensional, propose an end-to-end latent world model, termed Latent Particle World Models (LPWM), which learns a compact self-supervised object-centric latent representation for the images, based an improved version of the Deep Latent Particles (DLP, Daniel & Tamar 2022a; 2024) representation, DLPv3, and a novel dynamics module learned over the latent particle space. The model trained end-to-end such that the representation is trained to be predictable by the dynamics module, and as such does not require pre-trained image tokenization of any sort.
The Latent Particle World Model (LPWM) consists of four components, jointly trained end-to- end as a variational autoencoder (VAE, Kingma & Welling 2014): the ENCODER Eg. the DECODER Dg. the CONTEXT Kg and the DYNAMICS FE. The general pipeline operates as follows: input frames are first encoded by the ENCODER into sets of particles, which are then decoded by the DE- CODER to reconstruct images and compute the reconstruction loss. The resulting sequence of latent particles is passed to the CONTEXT module, which generates distributions over latent actions. The sampled latent actions- module to predict the next-step particle states, where the KI. is computed per-particle. Below, we provide a high-level overview of each component and a detailed description is provided in subse- quent sections. A high-level schematic of the encoding and decoding process is shown in Figure 9 and an overview of the full architecture is shown in Figure 3
with the partieles themselves
processed by the DYNAMICS
FG
Troposak
Enander
Figure 9: Encoding and decoding particles in DLP. Input image is first used to generate keypoint proposals, that, jointly with input image are used to encode a set of particles by the ENCODER, which is then decoded by the DECODER. Further details are provided in Section of the main text and Section of of the Appendix.
ENCODER (Section A
corresponds to the VAE's approximate posterior (0(2/x). It takes as input an image frame and outputs a set latent particles:
Egp = It) = [[= m. M -
Each frame I, is represented by M foreground latent particles { m. 1- 3= where each particle orig- inates from per-patch learned keypoint (see Section Section[A.4.I] and one background particle afg. Unlike DDLP, particle filtering to subset L < M is deferred to the decoder to preserve particle identities and avoid explicit tracking in downstream modules. Foreground particles are parameterized as
mm zfg
E [R B
where the first six dimensions capture explicit attributes (e.g.. spatial coordinates, scale, trans- parency), and the remaining daty dimensions represent appearance features. The background particle defined as
2ng E Rhe
25

---

Published as a conference paper at ICLR 2026
with dog encoding the global appearance of the background.
DECODER (Section A.4.2):
corresponds to the VAE's likelihood p,(x]z). takes as input a set of L S M foreground particles together with background particle, and reconstructs image frame:
Do([(=gg ) =bg) = Tt.
Here, denotes the number of foreground particles provided to the decoder, which can smaller than the M particles produced by the encoder. Particles can be filtered based their confidence e.g., using their variance scores (Daniel & Tamar 2024), or transparency values prior to rendering the frame, with the purpose of reducing the memory footprint without degrading the reconstruction performance.
: novel mechanism for modeling latent actions, i.e., the transitions that move a particle from 2 to 2; These latent actions provide the dynamics model with per-particle context, capturing events such as the stochastic movement of gripper in robotic video or the appearance of a new object in the scene. Throughout the paper, we use the terms latent context and latent actions interchangeably. Formally, the module takes as input a sequence particle sets across T + frames, optionally conditioned on external signals {ct) (e.g., control actions or language instructions), and outputs a sequence of per-particle latent contexts:
CONTEXT (Section A.4.3)
Kigh (1l(zzg
m. Jm=o zig: er o)) = (1 m. M-11 m 1
The CONTEXT module consists of two complementary heads:
Latent inverse dynamics piT (zf 1 2' 20, a)), which predicts the latent action
responsible for the transition between consecutive states.
Latent policy p given the current state.
policy (=t | ,",...,,",c,!! ct). which models the distribution of latent actions
In practice, the model does not output particles directly, but instead produces the parameters their predictive distribution (e.g.. Gaussian means and variances). The latent policy acts prior that regularizes the inverse dynamics via a KL-divergence term in the VAE objective (Section (A.4.6). During training, latent actions are inferred through the inverse dynamics head. At inference time, actions can instead be sampled from the policy prior, enabling stochastic rollout of the world model.
corresponds the VAE's autoregressive dynamics prior : ....,."). It predicts the next-step particles conditioned on the current particles and their associated latent actions:
DYNAMICS (Section A.4.4):
Fe(1 M Jm=0 2ng: 2TD
11 mm=0 2yy)
Similarly the context module, the model outputs the parameters of the distributions, which serve as the prior in the KL-divergence calculation part of the VAE training objective.
LOSS (Section A.4.6 Latent Particle World Models are trained by maximizing temporal ELBO, which decomposes into static term (for the first frame) and a dynamic term (for subsequent frames). For brevity, we omit the particle index m, and note that both dynamics and context losses are summed over all M particles.
Static ELBO. For the initial frame To. we optimize
Estatic
Lree (20, .ro) 3KL (1s((=9) 20) p(=2)) + Brog Lreg(
where 20 denotes the set of particle attributes and features. The KL. is computed masked form, where each particle's contribution is weighted by its transparency attribute 2t", such that particles with z!" zs 0 (inactive) have negligible effect. The transparency regularizer is defined as
M-1
Grog = Σ (z)))
m==
which penalizes the total transparency values across particles and thus encourages sparse explana- tions of the scene, i.e., only small subset of partieles remain active.
26

---

Published as a conference paper at ICLR 2026
Dynamic ELBO. For frames 2 1, the loss is
Ldynamic Σ
(3res (t)
Bayn KU KL(20(= go( xxx) pe 22 xx22))
Bexx KL (ptty (Et (25 polliyy (2f 2 259)].
where z denotes latent context (action) variables. The dynamics KL is masked as above, while the context KL is not, allowing context variables to also explain transitions between active and inactive states.
Priors. The static prior parameters are fixed: Gaussian means and covariances for attributes and visual features, and (a, b) parameters of a Beta distribution for transparency (see Appendix A.9).
Reconstruction. The reconstruction term is defined pixel-wise MSE in simulated environments, and as a perceptual loss in real-world data:
2 2122.
for simulated environments,
Leee
a + [1)()) (()) for real-world datasets,
where o(.) denotes VGG features as LPIPS Hoshen et al. 2019). and 7 = controls the
perceptual loss contribution.
In the following sections, we describe the technical and implementation details each component, with the differences from DLPv2 and DDLP (Daniel & Tamar 2024) highlighted in italies. In Section 1. we compare the proposed DLPv3 to DLPv2 and DLP on image reconstruction in the single-image setting, demonstrating the effect of these modifications.
A.4.1 ENCODER Eg.
We now describe the image encoding process from pixels latent particles. The encoder operates per-frame (i.e., non-temporally; all frames are processed independently in parallel) and serves as the posterior of the VAF. The scheme largely follows DLPv2 (Daniel & Tamar 2024); for completeness, we provide the details here and highlight the modifications introduced in DLPv3.
DLP learns object-centric particle representation by disentangling position from appearance within a conditional VAE framework (Sohn et al., 2015). Specifically, keypoint proposals are first generated represent candidate partiele positions, after which additional attributes such scale and transparency are extracted from regions centered around these proposals. The overall encoding steps are illustrated in Figure 10
The encoder's 's role is to produce the posterior distribution over latent particles for a given image. Formally, models the approximate posterior
q (=) (x) (0(za|z)) q0(2o; Rs, zd, 2z [r, za) x (a(22 Ap, zn);
where 2a denotes the keypoint proposals and zo the offsets that together form the particle positions 2p 2a + 2.- This modular factorization improves performance: keypoint proposals from the spatial softmax (SSM) layer tend capture regions of interest but are not guaranteed align with object centers. Offsets, in contrast, are predicted by neural network and can accurately adjust proposals object centers-a property that is crucial for modeling object dynamics. The encoding process is hierarchical and involves of 3 steps: (1) keypoint proposals patch-based network with a spatial softmax layer generates candidate particle locations; (2) attribute encoding offsets, scales, depths, and transparencies are inferred for each particle, and (3) appearance encoding foreground particle features and global background features are extracted. We now describe each step in detail.
Keypoint proposals. Given input image Cx divide it into M non-overlapping patches of size D D (typically D E {8, 16)). Each patch is processed by the proposal encoder (s(=al>)) (z).. lightweight convolutional neural network (CNN), followed by spatial softmax (SSM)
27

---

Published as a conference paper at ICLR 2026
Teregwend Parbiles
Praperial
Ap
Fauder
Swilgrooe noriile
Decepond
Aag
Zog
Liaage
Figure 10: Encoding particles in DLP. The encoding process involves 4 steps: (1) keypoint propos- als a patch-based network with a spatial softmax layer generates candidate particle locations; (2) attribute encoding offsets, scales, depths, and transparencies are inferred for each particle; (3) ap- pearance encoding foreground particle features, and (4) background encoding global background features extracted from the keypoint-masked image.
layer, which produces single keypoint proposal Za per patch. In DLPv2, these proposals filtered down to smaller set of size L using the variance of the SSM distribution. In DLPv3, we instead postpone filtering until after other particle attributes have been estimated, which yields more reliable selection. In LPWM, filtering is deferred even further: particles are never removed in the encoder, but instead filtered the decoder, ensuring that positional identity preserved for down- stream dynamics modeling. Nevertheless, for the single-image training setting. where only object- centric decomposition learned and no dynamics are modeled, we retain the option of applying encoder-side filtering.
Attribute encoding. To infer the position offset 2o- scale 2g. depth 2d. and transparency 24 of each particle, extract glimpses of size S S, where $ > D, centered at the keypoint proposals A using an STN 3 These glimpses are processed by the attribute encoder (o(2o, 2s. 2d, 24 (r., 2a), implemented small CNN followed by fully connected layer, which outputs the distribution parameters described in Section A.3 for each particle. The encoder's weights are shared across all particles.
Appearance encoding. The visual features of each particle are extracted with the appearance encoder qo(2y|ar 2p. 2.). As the attribute stage, an STN is used to obtain glimpses size S, but here the transformation is conditioned on both the particle position 2p 2, t2, and the learned scale Es, rather than fixed patch ratio. This allows the glimpse size to adapt when objects are smaller or larger than the nominal S x S region. Since all stages rely STN, the entire pipeline remains fully differentiable. The appearance encoder has the same architecture as the attribute encoder and outputs the Gaussian distribution parameters for the particle's visual features. In addition to object particles, we allocate a background particle anchored at the image center. Its features 25g are inferred by a dedicated background encoder 90 Na(2bgla l2, zp) which operates masked version of the input image. Specifically, the posterior keypoints Σp. along with their transparencies 2t, are used to generate M masks of size S x S, each masking out the region corresponding a particle, leaving the background regions visible for encoding.
DLPy3 encoding modifications. In DLPv3, we introduce several changes to the encoding process aimed improving stability and performance:
Thhe affine transformation in this stage uses fixed scale corresponding to the patch size S, predefined hyperparameter (typically 0.125 0.25 of the image size; assume square images with 1/ W).
28

---

Published as a conference paper at ICLR 2026
1. Depth via particle attention. Instead of predicting the depth attribute 2d during the at- tribute encoding stage, we introduce an attention layer applied after attribute encoding. This attention layer takes input all particles, including the background particle, and out- puts the depth values {= J mm=0 The motivation is that relative depth is inherently global property, best estimated by jointly considering all particle positions and features rather than independently.
M
2. Residual appearance encoding. The same attention layer is also used to refine appearance features. Specifically, the appearance encoding stage, we first compute a deterministic feature embedding ž for each particle. The attention layer then outputs residual Azj and variance o7 such that the final appearance distribution is
g0 Ir 2p; 2,) = [N šy + Azy, o7)
This residual modeling improves performance by allowing the network to adjust features based on contextual information from all particles.
3. Stable transparency parameterization. In DLPv2, the Beta distribution parameters (a, b) for transparency were modeled as exp(ya), b = exp(Bs). where Ya, Uh are the outpus of the network, which could lead to excessively large concentration values and unstable training. In DLPv3, we reparameterize them
Timx (Ja) + Tmin (1 (ya)), b = mmxx a (yb) Timi (1-o(
where o(.) is the sigmoid function, "min 10-4, and Mmax 100. This constrains (a, b) to a bounded range, leading to smoother and more stable optimization.
A.4.2 DECODER Da
The decoder architecture is designed to mirror the object-centric structure of the latent representa- tion. Each particle is decoded into a localized appearance patch, positioned and scaled according to its spatial attributes, while transparency and depth resolve visibility and occlusions. This composi- tional design parallels classical graphics pipelines, local rendering, spatial transformation, and alpha compositing, but is learned end-to-end from data, enabling the model to reconstruct complex scenes in structured and interpretable manner.
The decoder models the likelihood
(o 2) p0 Big (2ps 2ss Zd, 24, 2 2bg)
and is composed of a particle decoder and background decoder, illustrated in Figure I1
Particle decoder. Each particle is decoded independently into an RGBA (RGB + Alpha) glimpse i E SxS4 representing the reconstructed appearance of particle The particle decoder consists of fully connected layer followed by small upsampling CNN that maps the latent feature vector into this glimpse.
The alpha channel encodes soft segmentation mask for the particle. The depth =a and transparency z, attributes modulate this mask, determining both the effective visibility and the compositing order of the particle. The spatial attributes (Zp, z.) specify the particle's 's position and scale, and are applied to the decoded glimpse using Spatial Transformer Network (STN) to place it into the full-resolution canvas sig.
The transparency and depth factorization process, which governs the stitching of multiple particles,
summarized in Figure 12
Background decoder.
The background is decoded from 2ng using standard VAE-style network: a fully connected layer followed by an upsampling CNN produces ings and the final reconstructed image is produced according
= aig + (1 a) @ dbg.
where is the effective mask obtained from the compositing process (Figure 12
29

---

Published as a conference paper at ICLR 2026
FG
Foreground Partiles
{z)
Ter tide
Decoder
Backey pverd
7b
Detader
Backegroowd Yartidle
BG
Figure 11: Decoding particles in DLP. Each particle is decoded into a localized appearance patch, positioned and scaled according to its spatial attributes, while transparency and depth resolve visi- bility and occlusions. The background is decoded with standard upsampling CNN-based network. Finally, the foreground and background components are stitched using the effective alpha masks.
def Factos_alpie heeto___lppaa_mpppeapaaa_...o (apphapha_bbb rgb_obj, z_t, R_d1:
alpha_obis [3, L, wl, per-particle alphe maps
rgb_abit [2, 3, h, u], per-particle RCB patches
[3, 1), transparency attributes
=_d:
[s, 1], depth attributes
Apply transpazenoy
alpha_obj - alpha_obj A 2_t
10
Moak ROs chonnels with alpha rgba_obj alpha_obj rgb_ob1
11
13
Depth-based importance map inportance alpha_ob] sigmold! 2_d)
15
Normalize importance weights
17
importance importance (Importance, sum (dim=1, keepdimsTrue) 16-8)
18
Composite objects according to importance
20
jbjecta_canvas {rgba_ob} importance) .aun (dim-11
22
sackground mask
bg_mask (alpha_obj impartanae) summddim=))
24
25
zeturn ob jects_canvess bg_maak
Figure 12: PyTorch-style pseudocode of the transparency and depth factorization used for composit- ing particles.
In DLPv3, defer the particle filtering process to the decoder stage. Instead of rendering all M particles, we only render a subset of L M particles, where filtering is based on the particles' positional variance and transparency. This design serves purposes: (i) preserving the full set of particles after encoding, which is important for downstream dynamics modeling. and (üi) reducing computational and memory cost during rendering, since many particles may be inactive or redundant. The filtering procedure is described below.
30

---

Published as a conference paper at ICLR 2026
Particle filtering. During training. we retain the top-L particles with the lowest positional vari- ance (i.e., highest spatial confidence). Formally, the positional variance of a particle is defined as
V(2) = + Dzy +
Σ JE(0,1)
log logoo.J.
where o22, oy. and Jzy denote the variance and covariance terms from the spatial softmax proposal, and o2 is the variance of the offset distribution predicted by the attribute encoder along axis j. We use the log-variance for the offset term to account for scale differences between the empirical vari- ance from the spatial softmax and the learned offset uncertainty. This choice is motivated by prior work (Daniel & Tamar 2022a), which demonstrated that particles with low positional variance tend to correspond to salient and meaningful parts of the scene, such as objects or object parts. At infer- ence time, where purticles are generated autoregressively, we simply discard all purticles with zero transparency, i.e, 2t = O.
A.4.3 CONTEXT Kip
We now present the main novel additional component to the DLP framework the CONTEXT mod- ule dey-designed to address the problem of stochastic dynamics in actionless videos. In such videos, scene dynamics are not fully determined by the first frames (c.g., ball starting to roll in OBJ3D (Lin al. 2020a) CLEVRER YYi et al. 2019)), but also by external signals such actions (e.g., a robotic gripper moving in the BAIR dataset (Ebert et al. ))17
A common approach capture such stochastic transitions is to introduce latent actions (Menapace
ct al.,
2021 Bruce et al. 2024 Gao et al.] 2025 Villar-Corrales & Behnke 2025). Typically, a autoencoding scheme: inverse model infers z Icinn (Je+1; It)
latent action Ze is learned
from two consecutive frames, and a decoder reconstructs the future frame f (T, =g), with training driven by reconstruction loss. avoid degenerate solutions where z memorizes 1++1 1, Re strongly regularized, either via vector-quantization (VQ) bottleneck (Bruce et al. 2024 Ye al.) 2025) or a variational bottleneck with KL-regularization to a fixed prior (Gao et al. 2025). Crucially, in these designs, the latent action is global: single vector encodes all changes between two frames. While this aligns with how agents are typically controlled (e.g.. joint positions in robotics, discrete actions video games), is limited in multi-entity settings. For example, in Mario, enemies move independently of the player's true action space, and in robotics, contact events can induce secondary object interactions. A global action vector cannot naturally disentangle these local dynamics.
In this work, we introduce the CONTEXT module K.g., a novel per-particle mechanism for latent action modeling. Unlike prior work (Villar-Corrale & Behnke 2025 Gao al. (025). we model
latent action for each particle, directly governing the transition from 2; m..t to z M.TF1 Regulariza- tion is not imposed via a fixed prior, but instead learned through a latent policy, which models the distribution of latent actions conditioned on the current state. This per-particle formulation enables the representation of multiple, simultaneous interactions, and allows stochastic sampling of latent actions inference time, capturing multimodality (e.g., moving left or right from the same state).
Finally, we extend Kug to support external conditioning signals such as global actions (e.g., ground- truth gripper controls), language instructions, or image-based goals. Importantly, conditioning within the latent context module maps global scene-level signals into per-particle latent actions. For instance, given a language instruction, XC.. learns to translate it into per- per-particle latent actions that drive the dynamics towards satisfying the instruction. When no external conditioning is provided, KGp simply infers latent actions from past particle trajectories.
Formally, the CONTEXT module takes input sequence of particle sets across T + 1 frames, optionally conditioned on external signals {ct]T" (e.g., control actions, goal images, or language instructions). It outputs sequence of per-particle latent contexts:
(1 mn M-1
mi m- Im= 2Lgg ce c (H< m== 2c.hg T-1
The CONTEXT module is implemented which jointly processes particles across space and time while ensuring autoregressive temporal de- pendencies. It is composed of two complementary heads:
causal spatio-tempont transformer (huu et al. 2024),
31

---

Published as a conference paper at ICLR 2026
Latent inverse dynamics P (2' , ,.',,...,,, at), which predicts the latent action responsible for the transition between consecutive states.
Latent policy p conditioned on the current state.
palicy (sf | 2",...,," 2", ce), which models the distribution of latent actions
The latent policy serves as prior that regularizes the inverse dynamics via KL-divergence penalty in the VAE objective (Section A..46.. Specifically, the latent actions are modeled Gaussian dis- tributions, 2c ~ [Me: 2²), parameterized by the context module. At training time, latent actions are obtained through the inverse dynamics head, ensuring consistency with observed transitions. At inference time, latent actions can instead be sampled directly from the latent policy prior, enabling stochastic rollouts of the world model. When conditioned on goal image or a language instruc- tion, sampling from the latent policy can be further utilized for planning in the particles space, as we demonstrate in the experiments section (Section 5.2 The CONTEXT module is illustrated in Figure
3
We now describe how different optional conditioning mechanisms are implemented.
Action conditioning. Given sequence of T global actions { = {a' {a') (e.g., robotic grip-
per joint positions), each action is projected to the transformer's 's inner dimension and then repeated for all M input particles, such that the same global action conditions every particle at timestep t. Conditioning is applied via adaptive layer normalization (AdaLN, (Peebles & Xie (2023)), enabling global actions to modulate the particle representations consistently across the scene.
Language conditioning. Given language instruction, we embed its K tokens with pretrained TS-large model (Raffel et al. 2020 to obtain sequence of embeddings c; {1x { These em- beddings are projected to the transformer's inner dimension and appended to the M particle embed- dings along the sequence dimension, resulting in M + K inputs every timestep = O...., T 1. The joint set of particle and language embeddings is processed by self-attention, after which the language embeddings discarded. We found this self-attention conditioning more effective than cross-attention, consistent with prior work in video generation (Yang et al.) (0224b).
K-1
Goal image conditioning. Given a goal image Ig. encode it into M particles (plus background particle) using the particle encoder ENCODER. The resulting goal particle set c = {=g m= is re- peated across all T timesteps and used to condition the corresponding particles {2²} through AdaLN. This allows each particle to be guided toward its goal state in temporally consistent manner.
A.4.4 DYNAMICS Fs
The dynamics module implements the VAE's autoregressive dynamics prior P5(=" | | predicts the particles at the next timestep conditioned on the current particles and their corresponding latent actions provided by the context module:
..... 2º). It
Fe (1ll m. M-1 0 zige = = = 11 J 26g)) 1==1'
mm.t s 11
Here = denotes the latent actions at timestep f. The dynamics module Fg is implemented a causal spatio-temporal transformer, where particles are conditioned on their corresponding latent actions through adaptive layer normalization (AdaLN (Zhu al., 2024)).
As in the other components, Fe outputs distribution parameters that serve as the prior in the KL- divergence term between the posterior encoder E.p and the dynamics prior
Differently from DDLP (Daniel & Tamar 2024), LPWM does not rely on tracking a subset of par- ticles across timesteps. Instead, it keeps the entire set of M encoded particles along with their identities (i.e., the patches they originated from). This induces a particle-grid regime: each parti- cle is constrained to move only within a local region around its original patch center, and when it reaches the limits of this region, its features are transferved to nearby particles. This mechanism illustrated in Figure 13
'The priors for the particles in the first timestep are fixed hyperparameters, consistent with the single-image training setup of DLP.
32

---

Published as a conference paper at ICLR 2026
Partide-Grri
Green sortislé models squars
Yelew partuls takess ever Tite squers festwees
visu part ragjea
Particle-grid regime. Each particle is constrained move only within a local region around its original patch center, and when it reaches the limits of this region, its features are trans- ferred to nearby particles.
Figure 13:
This design balances between two extremes. On one side are patch-based approaches (e.g., VideoGPT (Yan et al.) 2021)), where "particles" are fixed at patch centers and only patch fea- tures evolve over time. On the other side are object-centric particle models (Daniel & Tamar (2024), where subset of free-moving particles with explicit attributes (e.g.. position) can traverse the entire canvas, but their identities must be tracked across time. The latter assumption may hold in controlled settings- instance, videos with deterministic dynamics and moderate frame rates- but it fails in more general video data where actions stochastic events occur.
As discussed in Daniel Tamar (2024). relying on tracking introduces two key limitations: (1) the tracking algorithm assumes suffciently small frame-to-frant displacements, which constrains the model to certain frame rates; and (2) since the tracked subset of particles is fixed, the model cannot naturally represent events such new objects entering the scene without additional mecha- nisms (Lin et al. 2020a).
In contrast, purely patch-based dynamics models avoid these issues by predicting only fixed patch features without explicit attributes (e.g,, keypoints). While more generul, such models struggle to capture fine-grainee object interactions and relations (Lin al. 2020a Ww al. [2022b] Daniel & Tamar 2024). LPWM, through its particle- grid design, aims to combine the generality of patch- based models with the expressivity of object-centric particles.
A.4.5 TRANSFORMER ARCHITECTURE AND MULTI- VIEW
Spatio-temporal transformer. Given a temporal input sequence of particle sets with shape [B.T, M, DJ. where B is batch size, T is temporal horizon, M is the number of particles, and D the embedding dimension, standard transformer block applies multi-head attention over all T x M tokens, resulting in quadratic computation cost. To reduce this, LPWM employs memory- efficient spatio-temporal attention mechanism (Ma et al. 2025 Zhu et al. 2024 (see Figure [4), which decomposes each spatio-temporal block into two stages: (I) spatial block that processes all M particles at each timestep independently (B x T, M, DD. and (2) temporal block that captures temporal dependencies for each particle across time ([B M,T, T, D)).
For conditioning, we use adaptive layer normalization (AdaLN) (Peebles & Xie 2023 Given a condition vector e and intermediate feature z, AdaLN modulates transformer block activations
a1 022, B1, B2, 71, 7/2 MLP(c),
2 2 + 01 Sclf-Attentio((r: R.SNorm + 81),
z z + a2 MLP(12 RMSNorm(3) + 32))
This mechanism is used to incorporate positional and temporal information within the transformer in addition other conditional inputs such actions, language tokens images. During training, we use teacher forcing (Williams & Zipser 1989) while inference is performed autoregressively.
Multi-view support. We extend LPWM to jointly train multiple camera views by synchronizing particle dynamics across views. Multi-viee training is crucial for mriision-making tasks with oc- clusions, such multi-object manipulation (Haramati et al. 2024 Qi et al. 2025), where object
33

---

Published as a conference paper at ICLR 2026
[B.T.M.01
Feed Forward
Norm
Multi-Head Attention
[B.T.MD]]
Norm
Temporal Attention Block
[8 M,7.0]
x
[B x M,
AdaLN
Spatial Attention Block
Feed Forward
[B x D]
Add Norm
[B.T,MDD]
Multi-Head Attention
Input Particles
Add Norm
T.M.D]
Input
Input
Particlas
Cond
[B.T.M.D]
Figure 14: Spatio-temporal transformer block. It consists of (1) spatial block that independently processes all M particles at each timestep, with shape [B x T, M, D]. and (2) a temporal block that captures each particle's 's temporal dependencies across the full horizon, with shape [B M, T, D). Here, B is batch size, T is temporal horizon, M is number of particles, and D is the embedding dimension.
hidden in one view may be visible in another, enabling agents to form more complete representa- tion of the scene. To achieve this, images from V views each encoded into M particles, which are then concatenated into a single set of V M particles. Each particle is augmented with view embedding to indicate its origin. The latent action and dynamics modules process the entire multi- view particle set jointly, allowing particles from different views to attend to each other and integrate information across viewpoints.
A.4.6 LOSS C
Latent Particle World Models are trained as variational autoencoder (VAE), with the objective of maximizing the temporal evidence lower bound (ELBO), as in prior VAE-based video prediction models (Denton & Fergus 2018 Lin et al. 2020a Daniel & Tamar 2024). The ELBO decomposes into a reconstruction term and a Ki-divergence term between the posterior particle distributions and the prior predicted by the dynamics model. We further distinguish between two regimes: static ELBO, where the KL-divergence computed against fixed priors for the first timestep (equivalent the single-image DI.P objective), and dynamic ELBO, where the prior is given by the autore- gressive dynamics predictions. Formally,
T-1
L.PWM = =- ELBO(Z, = 12) = Lstatic + Ldynamic.
(2)
We next detail the formulation of the static and dynamic components.
Static ELBO: Building on DLPv2 (Daniel & Tamar 2024), given an input image x E BCCHHW the loss in DLPv3 defined as:
L (x, + BKL (x) + Broy (zrg
(3)
where è is the reconstructed image, Live Ex, 2) is the reconstruction loss, the posterior particle distribution parameters, (x) is the KL--divergence between the poste-
1f M 1m 2bg
34

---

Published as a conference paper at ICLR 2026
rior and fixed priors, & is the transparency attribute of each particle, ( is regularization loss applied over the transparency values, and ŠKL: Breg are scalar hyperparameters balancing the losses (Higgins et al. 2017). For the single-image setting (i.e., temporal dynamics), Ssttii the only objective. Below we detail the Kl.-divergence and regularization terms.
KL-divereence loss LKL: For all M particles, we compute the KL-divergence of each attribute distribution with respect to its fixed prior, except for the background particle which only has visual features. In DLPv3, we adopt the masked KL-divergence (Lin et al.) [020a],, where the mask is defined by the transparency attrihute (e.g., particles with 2t 0 do not contribute to the KL. term):
M-1
CRL(Z)=
Σ
Σ
KL(go(=0it |x, Paso (z)) 2t"
m=0 (ossdd)
+ KL(go(zz"
| xx,"")) + By (Lo I |,)) o at
+ By KL(gg(Zbg x zp) Pbg(2))
(4)
where o, s, d denote the offset (zo), scale (2z), and depth (zd) attributes, 2n is the keypoint proposal, 2, is the transparency, 2, = + 2, is the final particle position, 2f is the visual features attribute, and 3y is fixed hyperparameter balancing explicit and visual attributes (3y = 0.01 in all experiments). Note that the KL for the transparency attribute is not masked.
Transparency regularization Lreg: In DLPv3, to prevent the trivial solution where all particles remain active (2+ 1) and sit patch centers (i.e., patch-based decomposition), we apply L2 penalty on transparency:
M-
Creg
(2
(5)
m=0
This penalty encourages sparsity transparency so that only subset of particles is active (2r 0). Since inactive particles do not contribute to reconstruction, the remaining active particles must cover more of the scene and are thereby incentivized move off patch centers and lock onto salient ob- jects. This yields more object-centric decomposition and reduces per-particle appearance variance decoding (e.g., a moving ball better captured by a single active particle than by several fixed patches). In practice, we set Preg
9KL.
Dynamic ELBO.
The dynamic component of the ELBO consists of three terms: the frame recon- struction loss, the particle dynamies KL, and the context KL:
Ldynamic
Bayn KL
xx)) ll (e(( 2 , -0:t-1)] KL(gas>"
(6)
+ Boxx KL p (at 20041) l1 polliyy (2i 20its)
where Liee is the reconstruction eтrоr as defined in the static ELBO. Here, denotes all particles and their attributes (including the background particle), while 2e denotes the latent actions. The coefficients Aayn and 8ex weight the two KI. terms (in practice we set fex Bayn). Note that for brevity we omitted the particle index m; in practice, the summation is carried out over all M particles for both the dynamics and context losses.
For the particle dynamics KL, we adopt the same masked formulation as the static ELBO, without distinguishing between explicit attributes and visual features. For the context KL, however, we do not apply masking: latent actions must also capture discrete events where particles switch between inactive (z, 0) and active (2, 1). Optimizing this loss end-to-end regularizes the posterior particle distributions to remain predictable under the learned particle dynamics and latent action models. Intuitively, the context KI. enforces agreement between the inverse model of latent actions
35

---

Published as a conference paper at ICLR 2026
(which infers actions from observed transitions) and the policy prior (which proposes actions given the current state), ensuring coherent action conditocod dynamics.
Finally, we specify the choice of priors and reconstruction losses used in practice.
Priors: For the fixed static prior distribution parameters, we define means and covariances for Gaussian distributions over all attributes and visual features, and (a,b) b) in the Beta distribution for the transparency attribute. These are treated as hyperparameters and detailed in Section A.9
Reconstruction objective: We use either the standard pixel-wise mean squared error (MSE) for simulated environments, or an APIPS-based L2 perceptual loss (Hoshen et al. 2019) for real-world datasets. When using LPIPS, the total reconstruction loss is the sum of the pixel-wise MSE and VGG-feature-ve -wise MSE, with the LPIPS feature loss weighted by y = 0.1. Formally, for simulated environments,
o 2 2112.
Trer
x - =1B + + 1b(x)) - (()) for real-world datasets,
where (b)) denotes VGG features.
A.5 POLICY LEARNING WITH LATENT PARTICLE WORLD MODELS
Pre-training Latent Particle World Model (LPWM) enables the extraction of rich latent dynamics from large-scale, actionless video datasets. Once paired video-action data becomes available, such pre-trained LPWMs can be leveraged for downstream policy learning. In this work, we demonstrate this ability in goal-conditioned settings, where the goal can be specified with an image language instruction.
The key idea (=m.))) Jm=o the ground-truth (GT) environment actions de- While the latent actions encode the transition from latent state z' to 2 2++1, LPWM produces a latent action per particle; thus, the mapping network must first aggregate information across particles to predict single global action.
the CONTEXT module (KCx)) to learn mapping from per-particle latent actions
To address this, we design the mapping network mi as an attention pooling network (Dosovitskiy al.) 2020; HHaramati et al. 2024), implemented as a compact two-layer transformer. This enables the model to adaptively pool the per-particle latent actions before regressing the global action.
Training procedure: Given a dataset of paired trajectories (Ho:T, 00.T-1), we encode the image sequence with frozen, pre-trained unconditional LPWM to obtain latent actions (=m")),M using the latent inverse dynamics head p (=² 2T These are projected to the transformer's 's inner dimension D. At each timestep. we concatenate learned action token (ACT) to the particle dimen- sion, forming input of shape [B, T, T.M 2, D], where B is the batch size. Here, the M particles correspond to the M foreground particles, the additional one represents the background particle, and the extra token is the learned action token.
The mapping network regresses the global action from the output corresponding to the action token: ml ([{=m
m. M
m= [ACT]{] M+1.
using the L1 loss:
= ||a - -.
Inference and planning: For deployment, given an execution horizon of i actions and a goal G. we unroll LPWM k + steps autoregressively, generating a trajectory of particles and their corresponding i latent actions. These latent actions are mapped to global actions using the trained mapping network, which are then executed sequentially in the target environment.
An important detail is that for each step, the next-state particles are generated by first sampling a latent action from the CONTEXT module's latent policy head p (2 12 ct = 2g). and then applying the DYNAMICS module. Notably, we empirically found that directly using the latent policy outputs for mapping degrades downstream performance; the mapping network performs best when evaluated on the outputs of the latent inverse module, as this matches the distribution seen during training. The difference may be due to distribution mismatch higher noise from the latent policy predictor-a question we leave for future investigation.
A high-level Py PyTorc--style code is provided in Figure 15
36

---

Published as a conference paper at ICLR 2026
Training loop
(1_seg, [_seg) dataset:
with sorch.no_gradien
Enoode sequence lateat actiors with Erozen LPWN Inverse dynamics
2_c_seq (PMM(__ shape: [B, D, Kfl, B_latent]
Project latent actiora and concatanate learned [ACT) boken
inputs concat ta_0_seq,
shaper [B, T, M-2, P]
10
Predict global action from [ACT] token output p_erdd neppiag_network (inpuutss [i, " 1, :1
11
12
13
# Computa Li less against ground truth actions
14
loss M (a_pred, [_seq)
15
bess.beckward optimlzer.step()
16
17
18
inference Goop
20
aba env. resere ()
goal get_goal image language
22
s_particles encoder (obs) Initial particle states
23
s_goal encode_goal (goal)
24
25 Szorage for predicted partiele states during unroll parsicla_trajeetoryy (*_particles]
27
Unzoll LPW (plan_herizon 1:11 steps
29 for in (plaap
sample lstent action
policy prior given cucrent particles etx. Let.eetetpooleyyy_perrer (t.pticccee z_goal)
31
32
Prodict partiale given pppatiiless dynts_particles, 8_o)
and latent action
34
36 Append predicted particles to trajectory
37
38
39 t Servers full particls sequence (batch) to latent actions in one call inverse_dyraat cs_batch paattilll_traaeeeeeeeeeeeeeeeeee 4 lateat actiors
antert_actions
outputa plan_horizon
42 Concatenata learned [ACT] token for mapping network input
aapping_inpan concat (latent_actlora, repear_ aplannhorria zon))
44
45 $ Predic: glabal action sequence from mapping network
action_sed sioou_seeenneee mappiiggnetwwork (napping_inputa)
47
predicted action sequence env. snv. (ecuaon_sequuncc)
49
Figure 15: PyTorch-style code for policy learning with Latent Particle World Models.
A.6 EXTENDED RELATED WORK
In this section, we provide a broad overview of related literature to situate Latent Particle World Models (LPWM) within the landscape of object-centric representation learning. LPWM is, the best of our knowledge, the first self-supervised object-centric model capable of being trained only from videos, optionally supporting multi-view training, and enabling various forms of conditioning- including action, language, and goal-image inputs. Since there are currently no other models with the same combination of capabilities, we review several adjacent and complementary lines of work to establish our contributions.
General video prediction and latent world models: Traditional video prediction models operate by encoding visual observations into a compact latent space, predicting future latents with a re- current (or, more recently, autoregressive) dynamics module, and decoding these latents back into pixel space. The notion of "world model" commonly refers to extensions of this pipeline that sup- port action conditioning and, some cases, reward modeling. Early methods (Finn et al.] 2016aa Ebert al. 2017b] Villegas et al. 2017 Lee al. 2018 Denton & Fergus 2018 Ha & Schmid- huber 2018 Hainer et al. 2020aa learned such representations using convolutional encoders and RNN-based dynamics capture dynamics in the latent space. More recent work improves
37

---

Published as a conference paper at ICLR 2026
long-horizon prediction robustness using discrete latent variables (Walker et al. 2021 Hafner al. 2020b 2023) or focuses on sealing up through hierarchical modeling and larger architectures (Vil-) legas et all 2019; Wu et al. 2021a Wang al. 2022). A core limitation of all these approaches is that they model seeee-level dynamics holistically, extracting representations that describe the entire frame at once without explicitly decomposing the scene into objects. This often results in blurry pre-
dictions or object disappearance during longer rollouts (Wu et al.) 2022b) Daniel & Tamar 2024).
To address these issues and improve sample quality, a number of recent works have incorporated
self-attention mechanisms for video dynamics (Nash et al., 2022 Yu et al. 2022 Yan et al. 2021
Zhang et al., 2023 Micheli et al.) 2023 2024, Dedieu et al. 2025), ) sometmes extending to richer forms of conditioning such as language instructions (Cen et al., 2025 Nematollahi et al.] 2025
Despite their advances in visual fidelity, these methods still lack explicit object-centric modeling. and are consistently outperformed by models with object-level inductive biases on tasks involving complex interactions (Wu et al. 2022b Zhang et al. (0255] Qi et al. (205). A further trend involves world models based on video diffusion (Alonso et al. 2024; Yang al. 2023 2024a Zhu et al. 2024 Yu et al. 2025). which achieve state-of-the-an generative fidelity but remain computationally intensive and, in their current form, forgo object-centric structure entirely in favor of scaling with model and dataset size.
Keypoint-based unsupervised video prediction: A distinct line of research aims to represent video dynamics through keypoint-based latent representations. Early works, such as Kim et al. (2019), combine unsupervised KeyNet (Jakab al. 2018) keypoint detection with class-guided video pre- diction using recurrent adversarial conditional VAE. Similarly, Minderer Similarly,Minderer et al. (2019) and Gao et al. (20TT) employ KeyNet for learning keypoints and use a variational RNN prior model stochastic dynamics, with the latter mapping predicted keypoints onto discrete grid for more robust long-term prediction. Although these methods successfully leverage keypoints for video structure, they do not explicitly capture object properties or interactions; visual features are extracted directly from feature maps rather than being represented as random latent variables. As a result, prediction quality often suffers from blurriness and object disappearance in long rollouts (Daniel & Tamar [022a 2024). More recent approaches take steps toward modeling interactions: V-CDN (Li et al. (0200 detects un- supervised keypoints using Transporter (Kulkarni al. 2019), constructs causal interaction graph, and predicts video outcomes using an Interaction Network (Battaglia et al., 2016). The original Deep Latent Particles (DLP) framework (Daniel & Tamar 2022a) introduced particle-based video predic- tion on real data but was limited by its simple graph neural network, which struggled with complex interactions. The subsequent DDLP (Daniel & Tamar 2024) advanced the framework to object- centric latent particles with richer attributes, enabling more nuanced video predictions. Compared to earlier keypoint- seypoint-based approaches, DDLP (Daniel & Tamar 2024) more closely resembles our model, as it captures both rich object attributes and intricate object interactions in video prediction. However, DDLP's reliance on particle tracking and its limitations in handling stochasticity motivate our advances; the present work addresses these challenges while extending the framework toward fully self-supervised world modeling and broader conditioning capabilities, as discussed next.
Unsupervised object-centric latent video prediction and world models: Unsupervised methods for object-centric video prediction build latent dynamics over decomposed scene elements, using one of three main paradigms: patch-based, slot-based, or particle- based representations.
Patch-based approaches (c.g.. RSQAIR (Stanié & Schmidhuber 2019), SPAIR (Crawford & Pineau 2019), SPACE (Lin et al.) (0200)) SCALOR (Jiang et al. 2019). G-SWM (Lin et al.] 2020a). STOVE (Kossen et al. (211g) represent objects with local "what", "where". "depth," " and "pres- ence" latent attributes, and typically model the joint latent dynamics by RNN-based modules. Later works such as SCALOR and G-SWM incorporated explicit interaction modules to capture object- object physics in prediction. Importantly, GATSBI (Min et al. 2021), an extension of the above with a separate keypoints module, stands out as patch- patch-based based model that can be considered a rudimen- tary conditiond world model, since it predicts scene evolution in response agent actions. However, unlike particle-base models, keypoints in GATSBI serve only to localize the "agent" in the scene and are not directly part of the object latent representation- -moot objects and background are discovered through separate modules, and the keypoint module merely distinguishes agent from non-agent entities. As a result, the full object representation remains patch-based rather than explicit keypoint- or particle-based. The patch-based typically require post-hoc or rule-based matching of object proposals across frames for temporal consistency. This reliance on frame-to to--tt--frr to-mre matching
38

---

Published as a conference paper at ICLR 2026
and the unordered nature of their object pose scalability challenge to complex or real-world video datasets.
Slot-based approaches (Burgess et al.
[019
Locatello et al.
et al. 2019: Engeleke ct al. 2021 Singh ct al. 2021| 2022a Veerapaneni et al.,
2020 Greff
2019
202T; Kipt et al. 2021 Singh et al.
2022b; Kabra et al.
2023
2020 typically represent scenes as a set of slots: commutation-invaiian latent vectors encoding spatial and appearance information for objects. These approaches generally adopt two-stage training strategy: a slot decomposition is first learned independently in a self-supervised manner, followed by a separate dynamics model
Sajjadi et a 2022 Weis al. 2021
trained on the inferred slots using recurrent models (Zoran et al., [02T] Nakano al.) 2023) or
Transformers (Wu et al. [2022b] Viilla- Corrales et al. |023]] Song et al. (2244). While recent exten-
sions incorporate conditioning on language (Villar-Corrales et al.) 2025 Wang et al., 2025a Jeong et al. 2025) or latent actions (Villar-Corral
& Behnke 2025), these models remain fundamentally limited by the quality and stability of the underlying slot decomposition. In practice, slot-based methods suffer from inconsistent decompositions, blurry predictions, and convergence issues, and recent research (Seitzer et al. 2023 Didolkar et al. [022]] Gong et al.) 2025 Kakogeorgiou et al.
2024 Jukié al. (0255) focuses stabilizing and scaling them, leaving open questions for robust term-term dynamics and world modeling.
Particle-based approaches, initiated by DLP (Daniel & Tamar 2022a) and advanced by DDLP (Daniel & Tamar] (024), provide compact, interpretable object representations using keypoint-based latent particles with extended attributes. DDLP jointly trains Transformer dynam- ies model and the particle representation, allowing stable object-centric decomposition and improved modeling of complex scenes. However, DDLP relies on particle tracking and sequential encoding, which restricts parallelization and stochasticity. Our proposed LPWM model is a direct extension to this lineage. LPWM eliminates the need for explicit tracking, enabling parallel encoding of all frames, trains end-to-end, and integrates latent action distribution for stochastic world modeling. This allows the model to capture transitions such as object occlusion, appearance, or random move- ments (e.g., agents or grippers), and supports comprehensive conditioning via actions, language, or particle-based modeling to the world model regime and addressing unsolved
goal aanags--avaancinn limitations of previous work.
Video prediction and world models with latent actions: To enable learning controllable or playable environments purely from videos, several works propose the use of latent actions-glon globbll latent variables that model the dynamic transition between consecutive frames. Models such as CADDY (Menapace al. 2021 2022) and Genie (Bruce et al. 2024 Savov et al.) 2025) learn discrete latent actions by quantizing the output of an inverse dynamics module. These latent ac- tions condition dynamics model to generate subsequent frames. Crucially, these approaches use a two-stage training process: first, the latent action module is trained, then the conditioned dy- namics module. During inference, users select latent actions from learned codebook to generate video sequences. AdaWorld (Gao et al., 2025) proposes a continuous analog without quantization, substituting quantization with strong KL regularization on the latent action distribution. This en- ables more flexible and smooth latent action representations. PlaySlot (Villar-Corrales & Behnke) 2025), the method most similar to ours in this category, augments slot-based object-centric pre- diction (OCVP (Villar-Corrales et al., 2023)) with a discrete global latent action module akin to CADDY, showcasing the benefits of object-centric decomposition for controllable video modeling. In contrast, our particle-based LPWM learns continuous, per-particle latent actions trained end-to- end jointly with the dynamics module. This design captures stochastic dynamics across multiple entities simultaneously. Furthermore, LPWM regularizes latent actions using a learned latent pol- icy, enabling stochastic sampling of latent actions at inference without external intervention, thereby supporting stochastic video generation. Additionally, unlike PlaySlot, LPWM's latent action module supports multiple conditioning modalities innssddincnnngnge coal-conditioning maaitiiooiiimmmminiggggnnn it readily appli- cable for post-hoe policy learning and control, as demonstrated in our experiments.
Decision-making with video inverse dynamics and latent actions: Recent works have increas- ingly focused on learning policies from videos by leveraging inverse dynamies modeling (IDM) or latent action representations. ILPO (Edwards et al., 2019) learns discrete latent actions via forward dynamics under the assumption of a known action set, then maps these latent actions to ground-truth (GT) actions for behavioral cloning (BC). Seer (Tian et al., 2024) jointly trains im- age prediction and GT action prediction via inverse modeling, without a latent action bottleneck, and effectively supports language-conditionn BC. LAPO (Schmidt & Jiang 2023) combines dis-
39

---

Published as a conference paper at ICLR 2026
crete latent actions learned via vector quantization (VQ) with policy learning; action decoder is jointly trained alongside an online reinforcement learning agent to map latent actions to GT ac- tions. LAPA (Ye et al. (225)) performs large-scale VQ-based latent action pretraining, which serves as the objective for vision-language model policy training, and fine-tunes for GT action mapping, with AMPLIFY (Collins et al. 2025) extending this by replacing latent actions with quantized key- point tracks, DreamGen (Jang ct al., 2025) further extends LAPA with diffusion-based objectives. Ada world (Gao et al. 2025) pre-trains large-scale autoregressive world models with continuous latent actions for downstream planning, while VideoWorld (Ren et al., 2025) trains an autoregres- sive discrete latent action model producing latent plans and frame- level IDM over decoded plans. Latent Diffusion Planning (Xie et al. 2025) and VILP (Xu et al., 2025) learn diffusion-basee plan- ners coupled with inverse dynamics modules. Similarly, Video Prediction Policy (Hu et al., (025) fine-tunes text-based large video diffusion model for plan generation, then learns a cnrffusion-baser policy via inverse dynamics. DynaMo (Cui et al. 2024) pre-trains image representations with paired inverse and forward dynamics self- sulf-vuper objectives for policy learning over these representa- tions. In contrast to these approaches, LPWM is an object-centric world model that integrates latent action learning directly with dynamics, producing per-entity latent actions that naturally accom- modate multiple interacting objects. Its latent policy further enables effective post-hoc mapping to GT actions and direct application to behavioral cloning and control tasks, distinguishing it from predominantly global or multi-stage latent action frameworks.
Decision-making with object-centric representations: As object-centric representations have matured, they have been increasingly incorporated into decision- massiinn-maiigg pipelines, demonstrat- ing strong performance on multi-object tasks that require complex reasoning and interaction. SMORL (Zadaianchuk et al. 2022) leverages patch-based object-centric representations in online RL, showing that structured perception improves sample efficiency and enables control in environ- ments with multiple entities. ECRL (Haramati et al. 2024) and EC-Diffuser (Qi et al. 2025) em- ploy DLP-based (particle-centric) representations, integrating them into online RL (ECRL) or imi- tation learning with diffuser- based policies (EC-Diffuser). These results provide clear evidence that object-centric models facilitate efficient policy learning and handle multi-object interaction chal- lenges. Complementary lines of work adapt slot-based world models for decision-making
FO-
CUS (Ferraro et al. 2025 2024), SOLD (Mosbach et al. (024)) and Dyn-O (Wang et af. (025b) augment online RL frameworks like Dreamer (Hafner al.] (020b) with slot-based object decom- position, yielding improvements in simulated env inviromments featuring a limited number of objects. OC-STORM (Zhang et al. 2025) extends STORM (Zhang et al., 2023) by combining a transformer- based dynamics module with object masks derived from supervised segmentation, thus relying on labeled inputs for decomposition. SegDAC (Brown Berseth 2025) and OCAAM (Rubinstein) et al. (025) similarly use deep RL over masked inputs, leveraging externlly supervised segmen- tation models. Some recent efforts bridge object-centric decomposition with latent action learning: Klepach et al. (2025) trains latent actions top of pre-trained slot-based representations, map- ping these actions to ground-truth policies via imitation learning. In contrast, LPWM is a fully self-supervised, object-centric world model: it builds directly on DLP, is trained end-to-end from pixels, and learns per-object latent actions as part of its joint dynamics training- maining-without requiring supervised segmentation or decoupled vision/policy phases, with object masks emerging a natural result of its reconstruction objective rather than any external supervision. Finally, LPWM supports post-hoc multi-object imitation learning and behavioral cloning in complex scenes.
Table summarrzes the various video prediction world modeling approaches across key dimensions.
A.7 DATASETS AND ENVIRONMENTS DETAILS
We provide detailed descriptions of all datasets used in this paper. Datasets are characterized by their properties: real-world or simulated origin, nature of menamics-ddterminnen physics, no external actions) or stochastic (external signals such as agent actions camera mo- tion)-and interaction density. Some datasets feature dense interactions. where object interactions frequent and most sequences include them, while others sparse, with less frequent delayed interactions, and some sequences may contain no interactions (Daniel & Tamar 2024).
(dominated by
OBJ3D: A simulated 3D dataset featuring dense interactions and deterministic dynamics, introduced by] Lin et al. (2020a). It consists of CLEVR-like objects (Johnson et al., 2017) in 100-frame videos of 128 x 128 resolution, where randomly colored ball rolls towards multiple objects in the scene
40

---

Published as a conference paper at ICLR 2026
Comparison of video prediction and world modeling approaches across key dimensions. Models are grouped by representation category: holistic. Table 4: http//obbeet-ceetie slot/object centric, and satficl//oojet--eentt AR: autoregressive: GNN: graph neural network. "Image Goal Cond." is image-gor goal conditioning support.
Dyn. Module
Transformer
Diffusion
Dillusion
Transfoemer
Transformes
Transformer
Transfommer
Transfrmes
Transfornee
Transfores
Transfommer Tranffrmer
RNN
RNN
RNN
RNN
RNN
RNN
GNN
RNN
RNN
Stochastic
Moltt- View
vadies
End- to- EEdd
Image- Inage--Gol Goal Cond.
Varies
Text Cond.
Action Cond,
7
Latent Actions
Dicertte
Diserdte
Cont. (per
Ohg Ceatric
Patch+ Keynt
Patch
Patch
(lype+Graph
Slots
Stots
Stots
Stots
Slots
Slots
Furticles
Particles
Partieles
2025
Dreamer DEDDS Hoanee & aL 201208
VE Vilte Grales of al. 2025
PlaySler (vlar Camale e & Hehhke
2003
CADDY Meminace dial 2012
DDLF P Damer& & Lamatt MAT
CONAISNAN AAISAAIVVEE Finn etal
VPdeoGPT Ton dfa Ar21
2019
STOVE Oesen aa 2019
STEDIE Nacan e OF 2022
OCVP Vilaee Mibrcoraic cornie di 3E
SOLD Musmeh aa 2029
Genie Baie 204
Diaman org 2023
SCALOR Jeme cla.
G-SWM Las DE
OATSBL Min eneer a. 2024
V-CDN OF 2020
PAKTS AYT an 2001
Stotfoemm WECKER
DLP Danie &
LPWM (UUiS)
Model
UniSam
41

---

Published as a conference paper at ICLR 2026
center, causing collisions. The dataset includes 2,920 training episodes, 200 validation, and 200 test episodes.
PHYRE: A simulated 2D dataset featuring sparse interactions and deterministic dynamics, designed for physical reasoning (Bakhtin ct al. 2019). We use the BALL-tier tasks in the wwwhii-tthhlllaae setting, where tasks are solved if a user-placed ball satisfies specific conditions (e.g., touching wall, floor, or object). Data consists of 128 x 128 frames generated from rollouts of all tasks except for tasks [12, 13, 16, 20, 21J, which contain substantial distractions. The dataset contains 2,574 training episodes, 312 validation, and 400 test episodes.
Mario: A simulated 2D dataset with stochastic dynamics and dense interactions, introduced by Smirnov et al. (202I). It consists of expert gameplay videos of Super Mario Bros downloaded from YouTube, featuring Mario traversing multiple levels. The videos include moving camera views with new objects and enemies appearing dynamically. The dataset contains 217 training trajectories and 25 test trajectories, each consisting of 100 frames with resolution 128 x 128. For FVD evaluation, we sample 100 trajectories for each video in the test set.
Sketchy: A real-world robotic dataset introduced by Cabi et al. (211), featuring a robotic gripper interacting with diverse objects. It has stochastic dynamics with sparse interactions. We focus on the ata wirk_greeeoon__eeeee
task, which includes 198 expert and 3,241 rollout trajectories. The dataset is split into 80% training, 10% validation, and 10% test. Each trajectory is truncated to the first 70 frames, resized to 128 x 128, and contains labeled actions enabling action-conditioned training.
BAIR: A real-world robotic dataset introduced by [bbrrt et al. (2017a), featuring robotic gripper manipulating diverse objects under a random play policy. The dataset exhibits stochastic dynamics and dense interactions, containing 43,264 training and 256 test trajectories at 64 x 64 resolution. For evaluation of FVD we follow the standard procedure of sampling 100 trajectories for each video the
in the test set (a total of 25,600 of generated videos). For 128 x 128 resolution training high-resolution version of the dataset introduced in Menapace et al. (202I). which contains 42,880
train trajectories, 1,152 for validation and 128 for test.
Bridge: A real-world robotic dataset introduced by Walke et al. (2023) featuring expert demon- strations of a WidowX robotic arm performing tasks guided by natural language instructions. exhibits stochastic dynamics and dense interactions. The dataset contains 25,460 training and 3,475 test trajectories, with episodes of varied lengths, all resized 128 128 resolution.
LanguageTable: A real-world tabletop robotic dataset introduced by Lynch et al. (2023), fea- turing language-guides
aetion-annotatee demonstrations of complex relational object arrangements based on shape, color, and relative position. The dataset exhibits stochastic dynamics with dense interactions and contains 179,976 episodes of variable length, resized to 128 x 128 resolution. We use an 80% training, 10% validation, and 10% test split.
PandaPush: A simulated 3D robotic environment introduced in ECRL (Haramati et al. 2024 and used by EC-Diffuser (Qi et al.) 2025) for goal-conditioned imitation learning. The task involves Isaac Gym-based (Makoviychuk et al. 2021) tabletop manipulation using a Franka Panda arm to push colored cubes goal configuration specified by image. We utilize the same offline two- view image dataset collected in EC-Diffuser, comprising approximately 9,000 episodes with 1-3 cubes each, and around 30 40 128 x 128 frames per episode from each view.
A simulated 3D environment and dataset from the offline goal-conditioned reinforcement learning benchmark OGBench (Park et al., 2025). Specifically, we use the Visual Scene environment and the associated "play" dataset, which features non-goal-directed interactions of a 6-DoF UR5e robot arm with various objects tabletop setting. The dataset includes 1,000 training and 100 validation trajectories, each containing 1,000 transitions, recorded at a resolution of 64 x 64.
OGBench-Sceee
A.8 BASELINE DETAILS
OCVP and PlaySlot: We use the official implementations Viilla-Corrals 2025) of OCVP and
PlaySlot and adapt the dynamic modules sizes to match LPWM, alongside modifying the CNN components for compatibility with 128 x 128 input resolution. Both models are trained in multiple stages, beginning with slot-based decomposition using SAVi (Elsayed al. 2022). Downstream
42

---

Published as a conference paper at ICLR 2026
video prediction performance is highly dependent on the quality of this initial decomposition. As noted in previous works (Daniel & Tamar 2024, Didolkar et al. 2024), SAVi can fail to assign distinct objects to separate slots and may require repeated runs with identical hyperparameters, to achieve satisfactory results. While we primarily adhere to recommended hyperparameters, slot as- signments sometimes ambiguous, with multiple objects per slot and occasional blurry recon- structions.
DVAE: We implement a non-object-centric, patch-based dynamics VAE (DVAE) world model adapted from LPWM. In DVAE, "particles" correspond to fixed-grid patch embeddings, where the number of patches matches LPWM's particle count, M. The baseline architecture preserves the same transformer backbone and parameter budget as LPWM, and supports identical conditioning modes (e.g., actions, language, image goals). However, DVAE does not model explicit object at- tributes, relying instead on spatially organized patch features. This approach is analogous to patch- based tokenization schemes commonly used in large-scale video generation (Yan et al.) 2021 Yang al. 2024b), but the patch embeddings are learned end-to-end without pretraining quantization, similar to LPWM's particle learning. To compensate for the lack of object-centric structure, we increase the latent dimension of each patch embedding. Patch extraction follows the standard proce- dure (Esser et al. 2021), whereby a CNN encoder downsamples input frames by a factor of f until M
the spatial dimensions resulting grid of patch features is used as the input "particle" set for downstream dynamics modeling and video prediction.
W. with H and W being the input height and width. The
T.
A.9 HYPERPARAMETER AND TRAINING DETAILS
Hyperparameters. We use the Adam Kingmma & Ba 2014 optimizer (91 0.9, 32 0.999, le - 6) with a constant learning rate of Se - 5. For all models we use an inner transfomer projection dimension of 512, and the latent actions dimension set to dasx = 7. The constant prior distribution parameters, reported in Table depend on the patch size used to extract the particles attributes and features. The complete set of the rest of the hyperparameters can be found in T
Warmup. In our training procedure, given a sequence of T frames, we typically apply the static ELBO loss to the first frame and the dynamic ELBO loss to the remaining T-1 frames. To facilitate robust learning of the initial particle decomposition, few iterations, usually corresponding to the first training epoch. In this stage, the static ELBO is applied to the first T frames, and only the final frame receives the dynamic ELBO. This warmup provides a strong initialization for the dynamics module and improves overall training stability and downstream performance.
introduce warmup stage during the first
Burn-in frames. Previous work (Wu et al. 2022b Villar-Corrales et al. 2023 Daniel & Tamar, 2024) introduces the concept of burn-in frames, where the first frames sequence (typically 4 n < 6) are provided as conditioning inputs to drive dynamics prediction under the assumption of deterministic dynamics. In DDLP, these initial frames are optimized using the static ELBO. In contrast, LPWM does not employ burn-in frames, as we assume stochastic dynamics and instead rely on latent actions drive predictions.
Stopping criteria. During training, we track several metrics calculated on the validation sets. Mainly, we save checkpoints for the best validation ELBO value and best validation LPIPS value, where the patent-action-condiite
generated video is compared to the GT video.
Resources. All experiments were conducted on various cloud- based computing platforms. For most datasets, training utilized single NVIDIA A100 or GH200 GPU. For larger-scale datasets such as LanguageTable Bridge, and Panda, training was performed on 8 GPUs, either A100s or H100s. The training duration varies by dataset size: small to medium datasets typically require few days to train, while large-scale datasets may take up to two weeks.
A. ADDITIONAL EXPERIMENTS AND RESULTS
This section presents additional experimental results and further insights complementing the main findings of this paper. In Section / we compare our modified DLPv3 model to its predecessors. Section A.10.2 provides further video prediction results, followed by an ablative analysis in Sec-
43

---

Published as a conference paper at ICLR 2026
across datasets. Base CNN channels count is 32. Ks refors to the Transformer based CONTEXT module. Fe refers to the Transformer- based DYNAMICS. FLOPs floating point operations per second (higher means more computational operations per second), FLOPs (Inference) corresponds to encoding and decoding of 16 frames, and FLOPs (Generation) correspond to one forward rollout of 15 frames conditioned on frame.
03Bench
64
30
MSE
0.02
0.05
0.02
8
0.25
[1,2,
(,,,2,2)
4
8
6
8
80
103M
4142G
34589G
x 128 (2 Views)
PandaPushi
25
64
10
MSE
0.04
0.1
n.D4
16
0.25
[1,4
[1, 1,1,1,4,5
8
84
112M
66416
40860G
LangIable
128 x
24
256
20
LPPS
0.08
0.2
n.0
12,4,8]
1,1,1,244
6
8
50
140M
32715G
185967G
128
1,2,4
Bridge
64
24
LPIPS
0.08
0.2
0.08
42, 8
147M
4667G
33R99G
128
11,11,2,44
BAIR
x
256
16
LPMS
0.08
0.2
0.08
0.325
12, 4,8
8
6
8
20
110M
66233
60731G
1,1,,,2,
1216420
Max
x
90
256
20
MSE
0.02
0.05
0.02
16
0.125
1,4,8
8
8
8
200
110M
82T2G
Sketeby
30
64
20
LPPSS
0.08
0.2
0.08
0.25
[1,4,8]
1,1,1,244
6
18
110M
31590
16036G
128
128
256
13
MSE
0.02
0.05
n.02
0.325
12, 4,8
1,
110M
8272G
121642G
128
2,2,1,244
33:33
X
64
20
LPPSS
0.08
0.2
0.08
[1,
8
6
16
110M
3153G
16036G
M OHf KP Proposals)
T' (Taining Horizon)
Reconstructon Loss
KP Proposal Parch
PG CNN Ch. Mult.
BG CNN Ch. Mult.
FLOP' (Iaference)
Hyperpaammttrr
L. (#f Purtclls)
Glimppe Rallo
# KC Layers
# Ki. Headss
# Fe Layens
# Fe Heads
Resolucion
# Ejpochs
Model Size
Table 5:
FLOPs
Bxa.
Bar
Figg
day
dog
44

---

Published as a conference paper at ICLR 2026
Attribute
Distribution
Parameters (glippse ratio 0.28)
Parameters (glimpse rabio 0.123)
Position Offset Zo
Normal, (N, o²)
0. 0.2
0. 0.1
Scale
Normal, N ((p, o9)
fr Sigmoid (0.25). 0.3
Sigmoid (0.125). 0.15
Depth
Normal, Wi,, o²)
0,
0,
Transparency S
Beta, Beta(a, ()
= 0.01, b 0.01
= 0.01, b == 0.01
Appearance Features y Abg
Normal, ((,,o))
0, o
0, a
Table 6: Prior distribution parameters for different glimpse (patch) ratios. Glimpses patches
taken around keypoints, where glimpse ratio glimpse size
image size
tion A. 10.3 Finally, Section A. 10.4 discusses and analyzes our imitation learning-based decision- making application.
1.10.1 COMPARISON OF DLPv3, DLPv2, AND DLPV1
We quantitatively evaluate our enhanced DLP variant, DLPv3 (see Section A.44, against the original DLP (Daniel & Tamar 2022a) and DLPv2 (Daniel & Tamar 2024) using publicly available im- plementations (Daniel & Tamar, 2022b Daniel, 2024). All models are trained in the single-image setting on the OBJ3D dataset, with identical particle counts and recommended hyperparameters. Training for each model is terminated when the validation LPIPS score ceases to improve. As pre- sented in Table 7 DLPv3 achieves substantially superior image reconstruction compared to prior versions. Notably, DLPvl lacks explicit modeling of object attributes and is therefore unable to generate bounding boxes and other attributes that contribute the performance.
PSNR
SSIM 1
I.PPS J
DLP
39.23 = 3.33
0.982 + 0.009
0.085
0.018
DLPv2
41,97 3.74
0.985 0.006
0.019 0.01
DLP/3
43.87 + 4.45
0.990 + 0.006
0.011 - 0.005
Table 7: DLPv3, DLPv2, and DLP image reconstruction performance comparison in the single- image setting, evaluated on the test set.
A.10.2 SELF-SUPERVISED ORJECT-CENTRES VIDEO PREDICTION AND GENERATION
that LPWM surpasses all baselines on LPIPS and FVD metrics across datasets with stochastic dynamics under all conditioning settings. Notably, Figure illlstrates that LPWM effectively preserves object permanence over the entire generation horizon, whereas competing methods often suffer from object blurring and deformation. Furthermore, LPWM accurately models complex object interactions which are better aligned with the language instructions, as evidenced by rollouts on various robotic datasets.
We also highlight LPWM's multi-modality sampling capability: by drawing multiple samples from the latent policy starting from the same initial frames and language prompts, LPWM produces di- verse and plausible rollouts. Several examples are presented in Figures I6 and in videos available on our project website. Results for datasets with deterministic dynamics are detailed in Table
Regarding our main object-centric slot-based baseline, PlaySlot, objects tend to drift rather than remain static. This likely stems from its use of global latent action vector that models tran- sitions for all entities collectively, unlike our approach that leverages per-pane per-particle latent actions. Slot-based models also suffer from limitations inherent their slot-decomposition modules, which produce blurry reconstructions and imperfect object decompositions, consistent with prior obser- vations. Moreover, slot models struggle on datasets containing many objects (e.g., Mario) due to memory-limited number of slots, whereas LPWM's low-dimensional latent particles effectively scale.
DVAE, our primary non-object-centric baseline, performs competitively synthetic datasets but falls short on real-world datasets, underscoring the advantages object-centric representations. In certain cases, DVAE patch latent actions. We note that some datasets involve sparse object interactions, and visual metrics tend to emphasize large entities, which favor DVAE's performance.
outperforms PlaySlot, likely because both DVAE and LPWM utilize per-
45

---

video prediction Published as a conference paper at ICLR 2026
Video rollout examples demonstrating these behaviors are available our project website: https: //taldatech. git. 10/ 1ttp//ppwwww
Representation Inductive Bias versus Model Scale: To further highlight the advantages of object- centric representations over simply scaling up model size using standard patch- based representa- tions, train an LPWM model with 100M parameters mark BAIR-64 and report its FVD in Table Despite its relatively small size, LPWM achieves performance comparable many larger video generation models. We attribute this to LPWM's inherent strength in modeling object interactions, which provides a significant advantage over large patch-base models that may generate crisp pixel-level details but struggle with physically plausible interactions (e.g.. gripper movements intersecting objects). This demonstrates that the inductive bi- ases encoded through object-centric representations can yield benefits that scale alone cannot easily achieve.
the standard video prediction bench-
Datasct
OBJ3D
PHYRE 15, c 10, P : 40
Setting
20, c 6, p 44
PSNRT
SSIMT
LPIPS
PSNRT
SSIMT
LPIPS
DVAE
5..44
0.923
0.085
0.94 2.061
0.047=0.04
G-SWM
31.7:16.2
0.924
0.118:00.0
2..6
0.93=0.05
0.078=0.06
SlotFormer/OCVP
31.2+:4.91
0.925+0.04
0.135
21.26:3.54
0.89=0.05
0.108=0.05
DDLP
31.29
0.923+10004
0.088=10.06
26.98
0.95=0.04
0.055=0.04
LPWM (Ours)
31.45=15.77
0.926#0.04
0.081
26.94
0.95=0.04
0.04
Table 8: Quantitative results on video prediction for datasets with deterministic dynamics. f is the training horizon, is the conditional frames inference and p is the predicted frames inference.
BAIR-66 (64 64)
FVDJ
LVT (Rakhimov et al.
2020
125.8
FVD-GANNFF Clark
2019
109.8
THVD- (uc ct al
20200
103.3
VideoGPT Yan et all 2021
103.3
CCVS LL Monng
2021
99.0
FitVid Bahacizadch
21121
93.6
MCVD Voleh et al. 2022
89.5
NÜWA Wu et al. 2022a
86.9
RaMViD (Hoppe
2022
84.2
MAGVIT-B Yu
76
RIVER (Daviyan et al
20233
73.5
CVP Shrivastava & Shrivastava
2024
70.1
VDM (Tlo et al. 2022
66.9
MAGVII-L Yu
2022
62
LPWM (Ours)
89.4
Table Video prediction results on BAIR-64 (64 x 64) conditioning on one past frame and pre- dicting 15 frames in the future. Table adapted from Shrivastava & Shrivastava (024)..
Language-conditiones video prediction: Table 10 provides additional visual metrics for language- conditioned settings. Specifically, we report the PSNR. SSIM and LPIPS when using the language- conditioned posterior latent-actions to reproduce the original trajectory. as opposed to the standard practice of sampling language-conditioned latent-actions from the latent prior where only FVD applicable.
A.1 ABLATION ANALYSIS
We perform ablation studies to evaluate the impact of key design decisions in LPWM, including latent action type (global vs. per-particle), latent action dimensionality, and positional embedding methods. Using the Sketchy dataset and evaluating after 10 training epochs (Table TT). we ob- serve that per-particle latent actions critical for strong centt-accioooonntiionn performance. However, for sampling diversity, global mean sraaninooing of latent actions yields im- proved FVD, suggesting benefits from global variables during generation. The model demonstrates
46

---

Published as a conference paper at ICLR 2026
Mario
SAIR
AsA
LmdM
Figure 16: Multi-modal future sampling by LPWM. Starting from the same initial frame, LPWM produces diverse possible future trajectories, illustrated on the Mario (left) and BAIR (right) datasets.
Dataset
Bridge-L
LenguageTables
PSNRT
SSIMI
LPIPS,
FVD.
PSNRT
SSIMT
LPIPS.
FVD.
0.75
0.177
146.85
36.014..44
L0.01
0.019:0001
26.78
LPWM (Ouns)
36.38114.
0.87.10.0
D.077 1.005
47.78
36.57.1..01
0.97 0.007
10.0060..06
15.96
Quantitative results on language-conditioned (L) video generation. PSNR, SSIM and LPIPS are calculated on iitentactionnconniiionn video prediction. FVD is reported for stochastic generation by sampling from the latent policy.
Table 10:
robustness latent action dimension as long as approximates the effective particle dimension (< 6 + dabj. i.e., 10 for Sketchy), balancing compression and information retention; our choice of dax 7 reflects this trade-off Finally, adaptive layer normalization (Adal.N) for embedding timestep and particle identity outperforms standard additive positional embeddings, as previously observed (Zhu et al. 2024). albeit with an increased parameter count.
Ablation Variant
Latent
Latent Actions
Positional
PSNR1
SSIM
LPIPS,
FVDJ
Actions das
Type
Embeddings
Original
Per.Particle
Learned Adal N
28.5543.30
0.91:0.05
0.072:40.03
120.32
da
Per Particle
Learned AdaLN
27.7113.50
0.8910.06
0.08110.03
177.64
da 3
Per-Particke
Learned AdaLN
29.08+3.15
0.92=00.55
0.070=0.00
117.46
da 10
10
Per: Particle
Learned Adol
28.97433.28
0.91
0.068
117.54
da = 14
14
Per PartPPrtice
Learned AdaLN
28.81:43.22
0.91:0.05
0.069:10.03
121.02
Global Latent Actions
Mean Pool
Learned AdaLN
27.24:13.7
0.89:1007
0.087:10.0M
100.75
Global Latent Actions
Token Atteation Pool
Learned AdaLN
21.54:+4.29
0. 0.80=0.11
0.1 1.67200009
142.64
Positional Embeddings
Per: Particle
Learned Additive
21.54:1..29
0.80::011
0.176:
142.64
Table 11: Ablation results: impact of latent action dimensions and type, and positional embeddings on LPWM performance. Results reported on the Sketchy dataset after 10 epochs of training. Results do not reflect final performance.
A.10.4 POLICY LEARNING WITH LATENT PARTICLE WORLD MODELS
This section provides additional details on our decision maasinnnmmking Iiriiaationlcarininn imitation policies from a pre-trained LPWM, as described Section 5.2
SeBench-Scenes designed to challenge agent's long-horizon sequential reasoning through manipulation of diverse objects including cubes, windows, drawers, and buttons. Pressing button toggles the lock status of associated objects, requiring complex, multi-step planning to arrange ob- jects into target configurations. The baselines are taken directly from the benchmark which includes
GCBC (Lynch et al. 2020), GCIQL (a goal-conditioned variant of IQL (Kostrikov et al., 2022
RRssults after 10 epochs do not reflect final performance; best performance across datasets achieved with dex 7
47

---

Published as a conference paper at ICLR 2026
Task
VQ-BeT
Diffuser
EIT+BC
EC Diffusion Policy
EC Diffuser
LPWM (Ours)
Cube
93
2.7
80=2
88.7 +
94.8 1.5
92.7 4.5
Cubes
5.2+1
13
14.6 12.5
38.8 + 10.6
91.7 + 3
74
3 Cubes
0.6
0.2
14 16.4
66.8 17
89.4 = 2.5
62.1 = 4.4
Table 12:
Performance results PandaPush, physics-hased tabletop benchmark in IsaacGym where Franka arm must arrange multiple cubes to match target goal images. Reported values are success rates over 500 trajectories across 5 seeds; results within one standard deviation of the best are shown in bold.
GCIVL (Park et al. 2023). QRL (Wang al. 2023 CRL (Eysenbach et al. 2022) and HIQI. (Park et al. 2023).
Our maage-goal-conditioned LPWM trained offline data with 30-frame horizon, where goals are sampled within a window spanning the last training frame to 70 steps into the future, typically encompassing 2-3 atomic tasks. Because trajectories stem from unstructured play data rather than task-specific demonstrations, the goal sampling window is limited to maintain informative transi- tions for goal conditioning. During inference, we sample 20 actions per step, execute them in the environment, and feed back new observations for subsequent action predictions. 'abll [3 sum- marizes results across tasks, while Figure 4 visualizes an example imagined trajectory alongside environment execution. Videos are available on project webpage.
Results discussion: OGBench datasets contain highly suboptimal, unstructured trajectories, posing challenges for behavioral cloning (BC), particularly on tasks requiring many atomic subtasks (e-g.. unlock drawer, open drawer, place cube). As reflected by GCBC's performance, straightforward BC struggles when the goal is distant from the initial state. Nonetheless, BC method achieves strong performance on tasks involving up to four atomic behaviors, including task1 and task3, outperforming all baselines multiple behavior modes and highlights its potential for integration with RL value functions to opti- mize goal-reaching policies.
these. We attribute this LPWM's expressiveness, which captures
PandaPush: designed to challenge complex, goal-conditioned multi-object manipulation. We use the same 1-3 cube manipulation dataset as EC-Diffuser (Qi et al. 2025), but unlike EC-Diffuser, LPWM and policy across all tasks, rather than separate policies for each task (c.g., one for cube, another for 2 cubes), which gives the baselines an advantage. Baselines, taken from Qi et al.] (2025), include VQ-BeT (Lee et al., (024). non- diffusion method using Transformer with flattened VQ-VAE image inputs; Diffuser (Janner al.) 2022), trained without guidance on flattened VQ-VAE inputs: EIT+BC, adaptation of the EIT policy (Haramati et al. 2024) to behavioral cloning using pre-trained DLP image representations; and EC Diffusion Policy, inspired by Chi et al. (2023) and modified for coal-condiiioning. learn- ing from pre-trained DLP representations. Table 12 summarizes results, and Figure [7 shows an example imagined trajectory alongside environment execution. Videos are available webpage.
we train single multi-view saaag--goal-condiiin
project
Results discussion: despite relatively simple policy compared complex diffusion-based methods, LPWM outperforms all baselines except EC Diffuser and matches EC Diffuser's 's performance on the l-cube task. While this work focuses on demonstrating the potential of adapting pre-trained LPWM for downstream decision-making, future work can explore combining LPWM with more advanced policies for multi-object reasoning. Additionally, we leverage the multi-view variant of LPWM in this experiment, modeling particle dynamics simultaneously from multiple views, demonstrating the framework's 's flexibility and enhancing the its ability to robustly handle occlusions (Haramati et al., 2024).
48

---

Published as a conference paper at ICLR 2026
to10
to20
tw40
Imagine-V1
Exccute-V1
Imagine-V2
Excaute-V2
Figure 17: LPWM generated goal-conditioned imagined trajectories (top) and actual environment executions (bottom) through learned mapping to actions PandaPush from views. LPWM generates dynamics in both views simultancously. handling occlusions by the gripper.
Task
GCBC
GCIVL
GCIQL
QRL
CRL
HIQL
LPWM (Ours)
taskl
59
84
56 14
44
52
80
100 10
task2
10
24 28
=1
2 22
1 =1
81 =7
6 1:
task3
+0
16
0 +u
40
=u
61 #11
89 49
task4
+1
0 40
3 #44
2 41
1 #1
20 =8
3 +5
task5
40
40
=00
40
20
32
+0
overall 12 #2
25
12 #2
10 +1
11
49 =4
40 +1
Table 13: Full results on Visual Scene with the
dataset. Results for baselines were taken from the original OGBench benchmark (Park et al. 2025) and represent success rates across 4 soeds. Results within standard deviation are highlighted in bold.
49


Reviews:
[{"review_id": "4vvZrjkcqD", "reviewer": "Reviewer_NJH6", "summary": "This paper introduces Latent Particle World Model (LPWM), an extension of Deep Latent Particles (DLP) for self-supervised object-centric world modeling in videos. LPWM decomposes scenes into latent particles (keypoints with attributes like position, scale, depth, transparency, and features) and models stochastic dynamics using a novel CONTEXT module that infers per-particle latent actions via inverse dynamics and a learned policy prior. The model is trained end-to-end as a temporal VAE, supporting conditioning on actions, language, images, and multi-view inputs. Evaluations focus on video prediction/generation across synthetic and real-world datasets (e.g., OBJ3D, BAIR, Mario), where it outperforms baselines in metrics like LPIPS and FVD. Additionally, LPWM is applied to goal-conditioned imitation learning on PandaPush and OGBench-Scene, showing competitive results.", "questions": "1. The paper extends DDLP via the CONTEXT module for additional conditioning (actions, language, images). Could you explain how this theoretically or practically surpasses limitations of prior object-centric video prediction methods? \n\n2. Experiments focus on video prediction, yet world models aim to support decision-making (e.g., policy training). Could you compare LPWM to the Dreamer series or other world models on sample or learning efficiency? Alternatively, justify why the current evaluation suffices for decision-making potential. \n\n3. Table 5 shows varied hyperparameters across tasks, potentially limiting generality. Could you explain their selection and provide ablation results on sensitivity (e.g., to latent action dimension or learning rate)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The idea of incorporating object-centric methods into world model is well-motivated. The proposed method of flexible and supports diverse conditioning (actions, language, goals, multi-view), which is practical. \n\n2. The authors provided extensive experimental results to show the effectiveness of the proposed LPWM. Results on real-world datasets (e.g., BAIR, Bridge) highlight its robustness beyond simulated environments.\n3. LPWM achieves SOTA on stochastic video prediction (e.g., FVD of 85.45 on Sketchy vs. baselines) and competitive imitation learning performance (e.g., outperforming HIQL on OGBench task3). Ablations validate key design choices (per-particle vs. global actions, latent dimensions). The compact model size and efficiency (e.g., matching larger diffusion models on BAIR-64) underscore the benefits of inductive biases over pure scaling.\n\n4. Particle decompositions (keypoints, masks, bounding boxes) provide inherent explainability, aligning with neuroscience inspirations (e.g., \"what-where\" pathways), which could appeal to applications in robotics or microscopy.\n\n5. The supplementary material of this work is sufficient and comprehensive.", "weaknesses": "1. The contributions are somewhat incremental, as the authors extend an existing video prediction method (DDLP) by introducing a context module for additional conditioning. However, this should not warrant rejection, given the extensive experiments demonstrating the effectiveness of these improvements.\n\n2. The experiments could be strengthened. While the evaluations focus primarily on video prediction, the world model is intended for policy training. Comparisons with other world models (e.g., the Dreamer series and variants) in terms of sample or learning efficiency would be valuable.\n\n3. There are also concerns regarding the hyperparameters. The model uses diverse hyperparameters for different tasks, as detailed in Table 5. This may undermine the method's generality and raise questions about hyperparameter selection. Additional ablation studies on hyperparameter sensitivity would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VFi83mj4Py", "reviewer": "Reviewer_yZTp", "summary": "The paper tackles the problem of conditional world modelling with latent action discovery. The main contribution of the work is the extension of the DDLP approach to handle per-particle action latents. This gives the model more flexibility in modelling complex scenes and shows SoTA video modelling results.", "questions": "What would it take to scale this method to work on any dataset zero-shot?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Introducing per-particle action latent, rather than considering a single global action latent (as done in past works) is very sensible, especially for complex datasets, as it introduces an additional degree of freedom and increases the representational power of the model.\n* The idea of the context module that combines external conditioning with implicit actions is sound and effective. This provides a universal way to implement action conditioning.\n* I find the interplay between the inverse dynamics and the latent policy modules very elegant. The learned dynamics learns the latent action space, while the policy learns the prior, and at the same time regularizes the training of the former module.\n* SoTA results on video modelling", "weaknesses": "* The novelty is somewhat limited - the authors extend the previous work by intruding per-particle action latent. Nevertheless, it has shown improved results.\n* I feel like the choice to keep all M particles (and not to perform filtering to avoid tracking) sacrifices the ability to separate real objects from “empty” slots, and thus sacrificing interpretability and explicit object modelling - which in my opinion is a nice property of the original particle models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4CtqMd8mmo", "reviewer": "Reviewer_gkk7", "summary": "The paper proposes a novel particle based architecture for learning world models, action-conditioned video generation models. In addition to previous Particle based video models, the authors propose to combine these with per-particle latent actions, adding additional flexibility to the model.\n\nReviewer positionality: while I have co-authored related work, I do not normally review vision-centric papers, but RL papers. This will bias my review and I invite the authors to correct me on field-specific standards.", "questions": "How are causal actions handled in the framework. I am not quite able to tell whether the dynamics are conditioned on all particle actions or not. If they aren't how are causal relations (a robot moves a block) modelled, and if they are, how is causal confusion resolved? Is the assumption that enough data will solve this issue? \n\nIs the number of particles architecturally fixed? If yes, what is the principle advantage of particles over slots?\n\nCan known action information (e.g. robotics datasets where action information is readily available) be used to improve and ground latent action learning. In the imitation experiments an action mapping is learned post-hoc, but for applied decision making researchers (robotics, RL, etc.) feeding available action information in at training time would greatly improve the applicability of the method.\n\nCan pre-trained segmentation or feature learning models such as SAM, Dino, or V-JEPA be used to further improve the method or replace some components, to reduce the load of training on large scale video datasets.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors thoroughly motivate their work and propose a reasonable addition to particle-based generative dynamics models.\n\nThe authors thoroughly ablate the base model changes as well as the proposed per-latent particle, which makes the evaluation of the method admirably robust. I have some minor questions on some of the scores, which I believe are due to my unfamiliarity with the datasets.\n\nWhile the writing could be improved (see below) the overall method is straightforward and clear, and the", "weaknesses": "The gains on the vision datasets seem very marginal, and not significant at a reasonable confidence interval. Provided confidence intervals seem to overlap, that makes the results very hard to judge.\n\nNo information is provided on how confidence intervals (the +/- numbers in the tables) were computed. This makes the previous issue more problematic to assess.\n\nThe robotics experiments do not seem to compare planning with other latent action methods with the proposed method. That limits the clarity and scientific impact of the comparison. Can the authors provide some insights into the performance of e.g. DDLP on this task?\n\nThe robotics experiments are limited in scope and not performed on benchmarks where object interaction is particularly crucial, or where objects have complex dynamics. As one of the main motivation for learning latent actions is decision making and imitation, I think expanding the experimental section here and adding more details on the setup would greatly increase the relevance of this paper to the community. For example, while the method performs well on tasks 1 and 3, it completely fails on task 2, but what this signifies is not discussed. Furthermore, results presented in the main paper are cherry picked without justification, which I consider very bad practice.\n\nPlease specify the exact tasks chosen in OG-Bench for ease of comparison with other works.\n\nTable 7 compares the changes to DLP introduced in this paper. The proposed method (evaluated in Table 8 on the same dataset) seems to strongly underperform this baseline, yet this is not mentioned in the ablations. I assume this is because one is tested as an image model and one as a video model, but without this context the results are confusing.\n\nAnother issue with the paper is that it only formally fulfills the criteria of being 9 pages long. The method is de-facto impossible to fully comprehend and re-implement with the main body of the paper, and so the appendix becomes essentially a part of the main paper. I would strongly advise the authors to revise and tighten the writing in the main paper to move important architectural details to there. For example, the introduction is quite lengthy, with a slightly superfluous aside on the human visual system, and dominated by a massive image, and some details on the datasets could easily be moved to the appendix. This is not a reason for me to recommend rejection, but it would probably improve the paper to tighten the writing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)