DDLP: Unsupervised Object-centric Video Prediction with Deep Dynamic Latent Particles
Authors: Tal Daniel
Aviv Tamar
Venue: Accepted by TMLR ()

Abstract: We propose a new object-centric video prediction algorithm based on the deep latent particle (DLP) representation of Daniel and Tamar (2022). In comparison to existing slot- or patch-based representations, DLPs model the scene using a set of keypoints with learned parameters for properties such as position and size, and are both efficient and interpretable. Our method, \textit{deep dynamic latent particles} (DDLP), yields state-of-the-art object-centric video prediction results on several challenging datasets. The interpretable nature of DDLP allows us to perform ``what-if'' generation -- predict the consequence of changing properties of objects in the initial frames, and DLP's compact structure enables efficient diffusion-based unconditional video generation. Videos, code and pre-trained models are available: https://taldatech.github.io/ddlp-web

Published in Transactions on Machine Learning Research (02/2024)
DDLP: Unsupervised Object-centric Video Prediction with Deep Dynamic Latent Particles
Tal Daniel
taldanielm haapmiicci@@ommmppssseeenn....co
Electrical and Computer Engineering Technion Isruel Institute of Technology
Aviv Tamar
aviut Brichmiieciio..ccc
Electrical and Computer Engineering Technion Israel Institute of Technology
Reviewed on OpenReview:
net/ net/forum? id= Wgn8zärthg
Abstract
We propose object-centric video prediction algorithm based the deep latent particle (DLP) representation of Daniel & Tamar (2022a). In comparison to existing slot- or patch-based representations, DLPs model the scene using a set of keypoints with learned parameters for properties such as position and size, and are both efficient and interpretable. Our method, deep dynamic latent porticles (DDLP), yields video prediction results on several challenging datasets. The interpretable nature of DDL.P allows perform "what-if" generation predict the consequence of changing propertics of objects in the initial frames, and DLP's compact structure enables efficient diffussinn based unconditional video generation. Videos, code and pre-trained models available: https://taldatees github. 10
the---rftth--ttttt object-centric
catiuuub.o//dddiiii /dlp-web//b
10
t=
t=20
t-
t0
Ground Truth
DDLP Prediction
"Whot If the green ball started at the left?"
Figure Video prediction on OBJ3D. Top: ground-truth vidco overlaid with DDLP's inferred (posterior) trajectory for particle on green ball (blue). Middle: DDLP's generated video conditioned the first 4 frames, and predicted trajectory for particle green ball (blue). Bottom: modification in the latent particle space (red arrow) according to the "what if..?" question, and the resulting DDLP video. Note the different trajectory resulting from different collisions due to the modification.

---

Published in Transactions on Machine Learning Research (02/2024)
1 Introduction
Object-centric models [Burgess et al.
2019 Lin et al. 2020c Daniel & Tamar 2022a leverage unique objects with physical attributes like size, appearance, and opacity. Applied in diverse contexts such as continuous- dynamical systems (Battaglia et al. 2016), single images (Lin et al. 2020c Daniel & Tamar 2022a), and recently, videos (Wu et al.) 2022
inductive bias to represent scene as a collection of
Lin et al. 2020b), these models consistently yield more accurate representations compared to general mothods modeling pixel space directly. Here, we focus on object-centric video prediction a challenging domain with potential applications in reinforcement learning (Yoon al.) (023) and robotics (Zadaianchuk et al.) 2022). The object-centric bias entails tradeoff between expressivity and accuracy. On the one hand, it enforces limitations on the possible videos that can be represented, which for example must be composed of individual objects, with relatively mild scene changes. On the other hand, it has the potential to accurately predict complex physical interactions between several objects, such as the collision depicted in Figure [ difficult
task for more general models, even when they are very large (Yan et al. 2021 Yu et al. 2022
Object-centric video prediction requires learning of two components object representations and dynamics prediction. Recent work used off-the-shelf object representations based on either patches or slots, alongside recurrent neural networks or transformer architectures for dynamics prediction (Lin et al. 2020b Wu et al., 2022). However, such representations have complexities that limit their performance. Patch-bases methods process objects from all patches at training, leading to increased complexity and difficultics in scaling up (Wu et al. 22021b 2022), ; while at inference they require a threshold-basse filtering process to filter out proposals (Jiang et al. 2019), which requires subtle tuning; slot-based models struggle with real-word datasots (Seitzer et al. 2023 and their time and memory requirements rise rapidly with the number of objects in the scene.
We hypothesise that stronger object representations can significantly advance object-centric video prediction. Our candidate is the recently discovered deep latent particles (DLP, Daniel & Tamar (2022a)) representation. DLP employs set of learned keypoints (particles), identifying areas-of-interes from which object properties
are inferred. DLPs therefore strike a balance between a lightweight representation for all patches and a heavy representation for a small number of slots, potentially allowing to handle scenes with more objects than blotd methods, and utilize powerful transformer bastdd dynamics, which are difficult to scale with patch-based models (Wu al.) 2021b). Further, the DLP representation is interpretable, due to the explicit physical propertics of each particle, and enables to control generation by modifying latent space properties of particles.
While Daniel & Tamar (022a)) demonstrated preliminary video prediction with DLPs, by simply extracting particles for each frame individually and learning their dynamics, we find that this approach is not competitive on nontrivial benchmark tasks such as in Figure [ Our main technical contribution is an extension of DLP to handle dynamic scenes, which we term deep dynamic latent particles (DDLP). Intuitively, instead of extracting particles for each frame independently, DDLP learns consistent representation across the whole video. The key challenge, however, is that this requires tracking particles in time, and we propose novel differentiable tracking module so that the representation can be learned end-to-end. To model particle dynamics, we develop a novel Particle Interaction Transformer that captures both the spatial and temporal aspects of object interactions. In addition, we propose several technical improvements to DLPs, which further improve generation quality.
In our experiments, we find that DDLP is competitive with or outperforms the state of the art on several challenging object-centric video prediction benchmarks, including scenes with up 20 objects (difficult for slot-based models), and long videos spanning up to 100 frames (difficult for patch-based models to remain consistent). Notably, DDLP achieves these results while also boasting lower training memory requirement.
In addition, the finer control that the DDLP representation allows over the image generation enables modifying physical parameters of the initial scene, and answering "what-if" questions based on the generated video, as demonstrated in Figure the model imagines what would have happened if one of the objects was placod in a different initial position. To the best of our knowledge, this task cannot be solved with existing object-centric models, for which physical properties cannot be extracted and modified easily. Finally, taking advantage of
2

---

Published in Transactions on Machine Learning Research (02/2024)
the compact DDLP representation, show that DDLP can be combined with diffusion models (Ho et al. 2020) to quickly generate long video scenes unconditionally. To summarize, propose DDLP, versatile object-centric video prediction model that is both SOTA on object-centric benchmarks, and enables several new capabilities that may be important for future downstream tasks.
2 Related Work
We discuss different unsupervised object-centric latent video prediction studies based on their representation for objects. An extended literature review appears in Appendix
Unsupervisod object-centric latent video prediction methods largely rely on representations CX-
tracted by match-based methods (Stanić & Schmidhuber 2019 Crawford & Pineau 2019
Lin Greff al.
eet al. 2020ch slot-based methods (Burgess et. al.] 2019 Locatello et al.] 2020
2019
Engelcke et al.] 2019:
2021 Kipf et al.) 2021 Singh et al. 2022b; Kabra et al. 2021
Singh et al. 2021 [202aa] Anonymous 2023 Sajjadi et al. 2022 Weis et al. 2021 Veerapaneni
et al., 2020) that do not explicitly model objects properties such as position, scale, or depth.
STOVE Kossen et al. 2019
SCALOR (Jiang et al. 2019) and G-SWM (Lin et al.] 2220bb belong to the patch-based family of object- centric representations, and use the 'what', 'where', 'depth', and 'presence' latent attributes to represent the objects, and an RNN to model the latent dynamics, where SCALOR and G-SWM add an interaction module to model the interactions between objects. OCVT (Wu et al.) 2021b) uses a similar representation. and Transformer VVaswani et al. 2017
Method
Driving
Dynamics Training Module
Representation
Scheme
SCALOR (Jang ot al.
2019
Patch-latent
RNN
End-to-Rm End End-to-End
STOVE (Kosson et &l
2019
Patch-lateut
RNN
G-SWM Lan
20120)
Patch-latent
RNN
End-to- End
OCVT (Wu et al 20216
Patch-latent
Transformer
2-Stage
PARTS Zoran al. 2021
Slot-latent
RNN
End-to- End
STEDIE Nakano of al.
X123
Slot-latent
RNN
End-to-End
SlotFormer (Wu et at.
2022
Slot-lateut
Transformer
2-Stage
DLP DDaniel & DDLI (OL]s(Uu)]
2022
Particlo-latent
GNN
2-Stage
Particlo-latent
Transformer
End-to End-to-End
Table 1: Unsupervised object-centric video prediction. We categorize models by representation families: patch-latent use all patches of the image to represent objects, slot-latent decompose the image to assigned slots to represent objects and particle-latent use keypoints to represent objects and decompose the image.
Radford et al. 2018) is used to model
the dynamics in a 2-stage training scheme. As the patch- based approaches produce many unordered object proposals, a matching algorithm used match between objects in consccutive frames before the input to the Transformer, making OCVT hard to scale for real-world datasets. In contrast, our method is trained end-to-end and is based on keypoints, which both significantly reduces the complexity of object decomposition and does not require matching algorithm to track objects. PARTS (Zoran et al. 2021 | and STEDIE (Nakano et al. 2023) use blot- representations and employ an RNN-based dynamics model of the slots, while OCVP (Villa-Coorales et al., 2023) and Slot Former (Wu et al. 2022) use a deterministic Transformer to model the dynamics of the slots in a 2-stage training strategy, utilizing the attention mechanism to model the interaction between slots. Differently from Former,, our model uses the more compact latent particles, providing an explicit keypoint- based representation for the objects' location and appearance, which allows for video editing in the latent space, and uses the Transformer a stochastic prior for latent particles that is trained jointly with the particle representation, removing the need to sequentially train two separate models. As we show in our experiments, thanks to the compact representation, our model can fit scenes with many objects and can be utilized for efficient unconditional video generation. Lastly, while the original DLP work (Danicl & Tamar 2022ah demonstrated results on vidco prediction, it has limitations. Specifically, particles are extracted independently from each frame, and a simple order-invarianl graph neural network is used, which is incapable of modeling complex interactions due to the lack of a tracking module, discussed in Daniel & Tamar (022aa). Table summarizes the object-centric vidco prediction approaches.
3 Background
Variational Autoencoders (VAEs): VAE Kingma & Welling 2014) learned likelihood- based latent variable model of data po( that maximizes the evidence lower bound (ELBO) using an approximate posterior
3

---

Published in Transactions on Machine Learning Research (02/2024)
distribution q(z|z): logpo(x) By(s(a) [log po(22]]] - (X(((2()))l(())) = ELBO(#), where the Kullback- Leibler (KL) divergence is [L((())))ll(())) Be(s)s) [log a(ala)
p(=) The approxinate posterior go(=|zr is also known the encoder, while pe (z]22) is termed the decoder. The approximate posterior q0(2|z), likelihood po(2|1r) and prior p(z) are typically modeled as Gaussian distributions. The ELBO can be optimized using the reporameterization trick. Following, the term reconstruction error and log (o]p) are used interchangeably.
Deep Latent Particles (DLP): DLP (Daniel & Tamar 2022a is a VAE-based unsupervised object-centric model for images. The key idea in DLP is that the latent space of the VAE is structured as a set of K particles 2 == [2 2p] E (x(x2)))) where 2f kxd is a latent feature vector that encodes the visual appearance of each particle, and 2p E RRxx encodes the position of each particle as (z.y) coordinates in Euclidean space. Thus, the VAE in DLP is disentangled by construction into discrete set of interest points in the image. We briefly explain the prior, the encoder, and decoder in DLP. For complete treatment we refer to Daniel & Tamar (002a)).
The VAE prior p(z|z) in DLP is conditioned on the image x, and has a different structure for 2y and 2p. For the appearance featurcs 2f, a standard zero-mcan Gaussian prior is used. For the positions, however, the prior is structured as Gaussians centered on --y keypoint proposals, with a constant variance. The keypoint proposals are produced by CNN applied to each patch of the image, followed by a spatial-softaaz (SSM, Jakab et al. 2018 Finn et al. 2016b) layer to produce set of z- coordinates, one for each patch. The VAE encoder for 2p is another CNN that maps from the image to means and oogvariancees of 2p. As the posterior keypoints S1 and the prior keypoint, proposals S2 are unordeeed sels of Gaussian distributions, the KI, term for the position latents is replaced with the Chamfer-Ki KL: dcH-KL dCH- = min,,es, ((ppll)))) +Sates, mins, K Lzp bp)). For the appearance features 2,, the encoder uses Spatial Transformer Network (STN, Jaderberg et al. 2015) to apply small CNN to region around the position, 2p, for every particle. This is termed as glimpse. Finally, for the d another glimpse is applied to each particle to create an RGBA image patch around the particle. The patches are combined to create the final image. All components of the DLP model are learned in an unsupervised fashion, by maximizing the ELBO (i.c., minimizing the reconstruction loss and the KL-divergence between the posterior and prior distributions).
4 Method
In this section, we give a detailed description of our model, which extends the original DLP (Daniel & Tamar 2022a) to video, termed deep dynamic latent particles (DDLP). We begin with an improvement of the DLP model for single images, which we term DLPv2 (Section 4.1 The DLPv2 model extends the DLP particle representation with more attributes, namely, scale, depth and transparency, and modifies the encoder architocture. We emphasize that DLPv2 can be seen as a standalone replacement for DLP the single-image
setting, as illustrated in Figure g
We continue with DDLP- video prediction model based on DLPv2 (Sections 4.2 4.4 In our generative model, a sequence of frames I2 is generated from a sequence of sets of latent particles 20.... 2t. A main idea in DDLP is that a Tramsformer-based dynamics module, which predicts the particles in the next frame, is used only as a voriational prior during training of the model. The model is trained to reconstruct sequences of T frames, in a VAE fashion, where for reconstructing frame It; an encoder has access to the frames ......, Fe, and outputs a postcrior distribution of the particles 2, that compose frame 2,, while the prior only has access to frames 1t-,,, and explicitly models the evolution of the particles' distribution p(z41122). By the standard VAE loss function, which tries to minimize the KL-divergence between the posterior and prior, the prior is effectively trained to accurately predict the particle evolution. At inference time, we can directly use the Transformer to predict the particles of the next frame, and roll out a complete sequence of frames. To implement this idea successfully, however, we had to solve several technical challenges. The main difficulty is how to align the particles in the posteriors of different frames: treating each frame individually, as in [Daneel & Tamar (002aa). does not necessarily yield aligned particles. Our idea is a new particle tracking posterior, which uses ideas from the video tracking literature to align the keypoint posteriors in different frames. Other challenges include ro-designing the attention matrix in the Transformer to allow
Daniel Tamar 2022al proposed several decoder architectures. We focus here on their Object-Baseer model.
4

---

Published in Transactions on Machine Learning Research (02/2024)
particles to attend one another, and finding suitable positional embedding for the particles in the Transformer. In the following, we detail the components of our method. A high-level specification of our model is illustrated in Figure
8g
8 7
Foster ir Pirtiless
[z][
Porticle Decoder
Portiele Decoder
Porticle Decoder
060
(000
2o
2
2y-1
O0O
O
[ -
ticlee
Particle Encoder
Particle Encoder
Particle Encoder
(x) T-
Eateard
Addi & Nom Mill-lwad Attation
Dynonics Prior
Xn
Xy-1
Figure 2: DDLP model illustration. Input are frames (1x ( For each 0 particle encoder produces the latent particle representation given and 2t-1; this encoder tracks the particles to induce consistency between frames. Each individual latent 2, is fed through the particle decoder to produce the reconstruction of frame dt. In addition, a Transformer based dynamics module models the prior distribution parameters { given T=2 and a KT. loss term minimizes the distance between the prior { and posterior -
4.1 DLPv2 A Modified Latent Particle
We expand upon the original DLP's definition of a latent particle, described in Section 3 by incorporating additional attributes. This refinement follows recent works in object-centric vidco modeling [Li et al. 2020b Jiang et al. 2019) that demonstrated the importance of explicitly modeling the scale and depth of objects in videos with objects of variable sizes or videos of 3D scenes. Tn DLPv2, A foreground latent particle z = Zep Zes Zd, 2t, 24] E 5-+m is a disentangled latent variable composed of the following learned stochastic latent attributes: position 2p (Mpp o.)) E RR, scale 2g N ( RR,, depth 2a N of) E IR transparency 2, Beta(a,, bt) and visual features 2, ~ NMp,, of) Fm where m is the dimension of assign a single abstract particle for the background that is always located in the center of the image and described only by mbg latent background visual features, 2bg ~ "(Hbg, ) E R Training of DLPv2 is similar to standard DLP, but with modifications of the encoding and decoding that take into account the finer control over inforence and gencration due to the additional attributes. The main modifications include an additional CNN to infer the latent attributes from particle glimpses, incorporating the scale altribute in the Iransformation matrix of the STN and factoring the depth attribute when stitching the decoded image. We provide extended details of the modifications in Appendix D.I The training objective for DLPv2 follows the original DLP. Given an input image H
learned visual features. We illustrate the role of each attribute in Figure 3 Moreover,
we optimize:
CDLP (ree(x, 8) + 3KL (Chamfer
+ ( + By ( (1)
where & the reconstructed image, Lreo F) the reconstruction error, 2p are the posterior keypoints, Py (zplar) denotes the patch-wise spatial-softmax prior of the original DLP, Ral/ all the ceciibuutsssccllle
5

---

Published in Transactions on Machine Learning Research (02/2024)
transparency and depth, except for the appearance features zy, and SKL and By coefficients to balance the reconstruction loss and the KI.-divergence terms.
As the focus of this work is video prediction, we detail the prior, encoder, and decoder of DLPv2 in Appendix D.1 In the following, when referring to individual attributes particle in a particular time step, we use a double subscript. E.g., Ep,t denotes the position component of the particle time
Figure 3: Modified latent parti- cle. DLPv2 introduces extended particle attributes: scale, depth and transparency.
Position Ep
Scale 2, E R² Depth 2 E
Transparency x E Visual Features 2y
Scale Heligtt
Posiban
Deptt
Scale Width
Visual Features
4.2 Particle Tracking Posterior
Our goal is to predict the future dynamics of a given video using the latent particle representation. The postcrior (encoder) in DLP uses a CNN applied to the image propose keypoints. However, simply treating each frame individually would not work, the order of the particles is not aligned between frames and there is no guarantee that objects in consecutive frames will be assigned the same particle, as illustrated in Appendix D.5 Previous attempts to model the dynamics of unordered set of objects resorted to using computationally expensive matching algorithms such as the Hungarian algorithm (Wu al. 2021b). We propose here tracking approach to achieve particle-alignment between frames, by predicting the keypoints of frame 2; offsets from the keypoints of frame 1t-1- This idea is inspired by neural tracking algorithms (Held et al. 2016 Harley et al., 2022), which solve a similar problem for object tracking in videos, but here we apply it within an end-to-end training of video prediction model.
Cross-Correlation
Xt
X
(pt)))
Neas (Pt)
For each particle, our tracking posterior takes as input Pt-1, the particle position in the previous frame It-1, and searches for Pt, the particle position in frame I; in patch centered the previous position (see Figure 4. Since we want. to learn this tracking behavior, we implement this idea using differentiable components as follows.
Figure 4: Cross-correlation for tracking. Given keypoint po- sition Pt for frame Zt, we cross- correlate the patch at Pt, 2<(p:) and the patch from the consecu- tive frame at the same location It (p:) get a heatmap corre- sponding to the object displace- ment.
Given a sequence of frames {=e) we perform an iterative encoding where in each step (1) the posterior keypoints from the previous step serve as anchors, from which an offsct will be learned to the current step illustrated in Figure 2 and (2) to indicate the region in which the consecutive keypoint may be, the RGB patch input to the encoder is concatenated with an additional single-channel score-map generated by cross-correlating consecutive patches in time as described next.
Our underlying assumption is that the displacement of objects between consecutive frame small, so the encoder can scarch for the object
small region around its previous location (Held et al. 2016). To that
end, we simply use 2 the anchor keypoints for the next frame, i.e., Ea,t = 2p,-11 where for & =0

---

Published in Transactions on Machine Learning Research (02/2024)
the anchors are keypoint proposals from the prior of singlo-image DLP. In addition, to minimize false detections in cases where objects parts of objects enter the region of the object-of-interest, we compute the normalized tross-correlation (Harley et al.) 2022) between the patches, or glimpses, from Ft-1 and Ia around Ep,t-1, which we denote (t-1(P(-1) and z; (Pt-1). This is implemented efficiently with group-convolution. This single-channel generated score-map can help direct the search for the object the consecutive frame as visualized in Figure The score-map concatenated channel-wise to the input of the encoder component that outputs the particle attributes from each glimpse. We provide more details on the tracking process in Appendix D.5
4.3 Image Decoder
The decoder, closely following the original DLP, reconstructs the scene from the latent particle representation, and is composed of object decoder and background decoder, where the object components and background are stitched together to generate the final reconstructed image. In DDLP, the decoder operates on the latent particles of each frame independently. In Appendix D.4 we detail the decoder's architecture and provide a more technical description of the stitching process.
4.4 Generative Predictive Dynamics with Particle Interaction Transformer
f
f
CO
F2
X
b:]
10
M
bg
AgT
12
68
b/
6
80
80
ACT
45
Ay
Ag
80
82
46
IC
M4
42
A6
Figure 6: Illustration of the relative posi- tional encoding bf. Particles in the same timo-step get the same temporal relative embedding by, while across time, particle i get unique relative positional embedding w.r.t the other particles b¹.
Attention in PINT: particles attend to each other. A causal mask is applied to attend to past and current particles.
Figure 5:
Figure 7: Relative posi- tional bias matrix PINT. by 6 +bb, where denotes the relative time and j the relative particle position.
The dynamics module models the prior distribution Pv (2)+1 of the particles' temporal evolution. Tn this work, inspired by the success of Transformers for long-term prediction (Wu et al. 2022 Raffel et al. 2020), we propose stochastic Transformer-based prior dynamics module that is trained in conjunction with the posterior, and utilizes the attention mechanism capture the interaction between particles over time in parallel.
Our proposed dynamics module, termed Particle Interaction Trunsformer (PINT), is a causal Transformer decoder based on the design of Generative Pre-trained Transformer (GPT, (Radford et al. 2018 Given the set of particles Z = {( ((., KK 2bg)< whore T is the prodiction horizon and 2bg is a special particle representing the background (see Appendix D.I. PINT outputs the one-step future prediction 2 {(Eo, -1, T- First, the input Z is projected to G(Z) E R(K+I)xTXD, to match the Transformer inner dimension D. Differently from previous approaches, we do not add positional embeddings before the attention layers. Instead, building recent improvements in the literature, add learned per-head relatine positional bias (hha et al. 2018 Raffel et al.] 2020) which is directly added to the attention matrix before the softmax operation. Empirically, we found this approach more effective than the standard learned positional embeddings that are added prior to the al iention layers.
We base implementation on the minGPT open-source implementation
Karpathy 2021

---

Published in Transactions on Machine Learning Research (02/2024)
However, ignoring the structure of the imput and naively adding positional embedding to each particle may break permutation equivariance among entities, as shown in slot-based approaches (Wu et al. 2022 To that end, we decompose the added relative positional bias to a temporal relative positional bias and a spatial relative bias. The temporal embedding matrix Brime E RTXT assures that all particles in the same time-step gel the same relative temporal encoding, while the spatial embedding matrix Bpos E R((+1))((+1) assures that cach particle is assigned a unique relative embedding through time. The relative positional bias is illustrated in Figure 6 Beime and Bpos are repeated (K + 1) and T times respectively, and summed to create the relative positional bias matrix = Bume + pos E (1(K*)))))(((++111)) B(i. j) = by = b, + b, where denotes the relative time and j the relative particle, as illustrated in Figure 7
QKK + B V. The relative positional bias is added directly to the query-key matrix, i..., Attention = Softmax VD"
To model particle interactions, the attention operation is modified such that particles attend to each other in the same timo-step and across time, i..,, the attention matrix A E we use a causal mask to ensure the attention only considers past inputs, as illustrated in Figure 3 Finally, the attention layers' output is projected with a FC network to the parameters of the prior distribution for the different particle properties. To speed-up training, we use teacher forcing (Williams & Zipser 1989) at training time, while at inference time, predictions are generated autorogressively. Unless stated otherwise, in all experiments we use a Transformer with 6 layers and 8 heads per layer. A detailed description of PINT's architecture can be found in Appendix D.6
and similarly to GPT
Burn-in Frames with the Single Image Prior To accurately model the dynamics of moving entities and their interactions in video, single frame is not enough to extract meaningful dynamics information, such velocity and accelcration. In DDLP, define timo-step threshold 7 where for t 2 7 the output prior parameters by PINT are used to calculate the KL-divergence term between the posterior and prior, while for t <<> T, also termed burn-in frames (Wu et al., 2022), we use the standard constant prior parameters as in the single-imaaee DLP, detailed in Appendix D.2 to calculate the KL-divergence term (similarly to Wu et al. 2022
we set T = 4 in all our experiments).
Training and Implementation Details Similarly to the training of other VAF-based video prediction models (Denton & Fergus, 2018 Lin et al.) (000bb), DDLP is trained maximize the sum of the temporal ELBO components Σ ELBO(xt), i.e. minimize the sum of reconstruction errors and KL-divergences over time:
LDDLP =
(z)) = Σ CDLP, +
Laynats
1=0
1=
where CDLP, is defined in Equation and = Lrve Ze) + we use the same 0KL coefficient as before for all temporal attributes. The KL-divergence terms for the distributions we employ have closed-form formulation, detailed in Appendix error, we use either the standard pixel-wise MSE or VGG-based percept.ual loss (Hoshen et al. 2019), depending on the domain. The reconstruction and KL terms are balanced with a coefficient term & Higgins et al. 2017). DDLP is trained end-to-end, effectively regularizing the posterior particles to be predictable by the learned prior particles, with the Adam (Kingma & Ba 2014) optimizer and initial lcarning rate of 2e 4 which is gradually decreased with a step scheduler. Our method is implemented in PyTorch (Paszke et al.) 2017) and we use 1-4 consumer GPUs to train our models. We report the full set of syper-parameters in Appendix E and provide complexity analysis Appendix F. Code and pre-trained models are available at: https://ggitub. cww com/talaatecchhadpp
where here
E and for the reconstruction
5 Experiments
Our experiments: (1) compare DLPv2 to the original DLP in the single-image setting (Appendix [.11] (2) benchmark DDLP on video prediction against the state-of G-SWM and SlotFormer; (3) demonstrate the capabilities of DDLP in answering 'what if... questions by modifying the initial scenes in the latent
space; (4) evaluate our design choices through ablation study; and (5) showcase an application for efficient. unconditional video generation by learning a diffusion process in DDLP's latent space.
8

---

Published in Transactions on Machine Learning Research (02/2024)
GT
Rec
O8
OBJ3D
66
Traffic
17.80
Figure 8: DLPv2 object-centric decomposition. We demonstrate the performance of our proposed improved single-image DLP. Please see Appendix for more details.
5.1 Datasets and Experimental Setup
We evaluate our model on datasets with different dynamics and visual properties 3 Following brief
overview the datasets; an extended description in Appendix G
Balls-Interaction: (Jiang al., 2019) A 2D dataset of 3 random-colored balls bouncing and colliding. Videos 100-frames, 64 64 pixels. DDLP, similarly to G-SWM, is trained on sequences of 20 frames. At inference, both models are conditioned on 10 frames and predict the next 40. For this dataset, DDLP uses 10 latent particles and the MSE roconstruction loss. As this synthetic dataset includes GT ball positions, we can precisely measure the long-bord horizon prediction accuracy difficult task for video prediction models. OBJ3D: [Lin et al. 2020b) 3D dataset containing CLEVR-like objects (Johnson et al.) 2017]. In each 100-frame video 128 128 resolution, & random-colored ball is rolled towards random objects of different types positioned in the center of the scene. This dataset has become a standard benchmark for physical understanding it includes complex object interactions. We train DDLP with 12 particles on sequences of 10 frames with perceptual loss; We train SlotFormer with 6 slots sequences of 16 frames and G-SWM sequences of 20 frames we found it crucial for accurate collision prediction. At inference, all models are conditioned on frames and predict the next 44.
CLEVRER: Yi et al.) 2019) A 3D dataset containing CLEVR-like objects. Each vidco consists of objects entering the scene from various locations, potentially colliding. As new objects may enter the scene at future timesteps, we trim every video to maximize the visibility of objects the initial scene. We resize the frames 128 X 128 and use sequences of 20 frames for learning, where DDLP is trained with 12 particles and perceptual loss. At inference, challenging than OBJ3D as there are more variations in the initial conditions.
condition
10 frames and predict the next 90. This dataset is more
PHYRE: (Bakhtin et al.] 2019) A 2D dataset of physical puzzles. We collect simulated data in the form of 128 x 128 frames from the BALL-tier tasks in the ball-within-tempplae setting. We train the models on sequences of 20 frames, where DDLP is trained with 25 particles and the MSE reconstruction loss. At inference time, we condition on frames and predict the next 90 frames.
Traffic: (Daniel & Tamar 2022al Real-world videos of varying number (up to 20) of cars of different sizes and shapes driving along opposing lanes, captured by traffic camera. We separate the video to chunks of 50 frames of size 128 x 128. For this dataset train DDLP with 25 particles sequences of 10 frames with perceptual loss, while SlotFormer is trained with 10 slots on sequences of 16 frame and G-SWM is trained on sequences of 20 frames which was essential to reduce cars from vanishing in the prediction. At inference, all models are conditioned on 6 frames and predict the remaining 44 frames.
WW did not compare with Flom CLEVRER and PHYRE since Wu al. 2022 reported that special pre-proccssing is required for Slot Former work these datasets. Further details found Appendix H 4Please see Appendix regarding required compute train SlatFormer.

---

Published in Transactions on Machine Learning Research (02/2024)
5.2 Unsupervised Video Prediction Results
We evaluate DDLP on long-term video prediction as described above, while training on short sequences of frames. Our SOTA baselines include the patch-latent based model G-SWM (Lin et al., 2020b), and the bot-based model SlotFormer (Wu et al. 2022). using their publicly available code and pre-trained models (Lin)
et al.
2020a
Wu et al. 2023)
Evaluation metrics: for all datasets, we report the standard visual metric PSNR, SSIM (Wang et al, 2004) and LPIPS (Zhang et al. 2018), to quantify the quality of the generated sequence compared to the ground-truth (GT) sequence. In addition, for Balls-Interaction we calculate the mean Euclidean error (MED10, Lin al. 2020b) summed over the first 10 prodiction steps as we have GT ball positions.
Results: We present quantitative results in Tables 2 and 3 with additional results and rollouts available in Appendix ! An extended comparison with SlotFormer can be found in Appendix H Please visit https::/taalee beepaattchhgghuub... 1o/ddlp web/ to see video rollouts. In the following, we summarize our observations on DDLP's performance in comparison to the baselines.
Observation 1: Particle tracking is crucial, and the RPE improves performance, as evidenced by the ablation study (Table P bottom). More ablations are reported in Appendix L8
Observation 2: DDLP is beller al long-lerm consistency. In Balls-Interaction, where many collisions affect the balls' trajectories, DDLP outperforms G-SWM on all metrics (Table 2). No- tably, the accumulated error of the balls' positions is substantially lower, indicating that combining a keypoint representation with a Transformer-basee While transformers have been shown to improve patch-based models, their scalability to more complex datasets is limited (Wu et al. 2021b); DLPs are demonstrated to overcome this limitation. Similarly, DDLP outperforms the baselines on all other datasets (with marginal improvement on CLEVRER), as shown in Table 3 We provide example rollouts for the Traffic dataset in Figure 10 and for OBJ3D in row 2 of Figure [ demonstrating the superiority of DDLP in long-term prediction. In contrast, objects in FlotFormer's and
dynamics module yields more accurate predictions.
G-SWM's predictions get blurry, vanish or deform over long horizons. We further demonstrate object deformation in Figure g
DOLP
Observation DDLP performs beller in scenes with a large and earying number of objects. This particularly evident in Traffic, as shown in Figure IQ Note that both DDLP and G-SWM are able to correctly capture the objects, thus, attribute this to the limited number of slots available in SlotFormer (we used the maximal number of slots available under our compute resources). It is worth noting that DDLP outperforms G-SWM in long-term consistency, as discussed in the previous observation.
G-SWH
Figure 9: Long-term pre- diction results on OBJ3D. While G-SWM tends to deform objects over time (deformed yellow cylinder), DDLP demonstrates bet- ter long-term consistency in shape and position.
Observation 4: DDLP produces videos with better perceptual quality such as more realistic shadows and reflections. Quantitatively, in terms of perceptual quality (LPIPS). DDLP significantly improves upon the baselines (15 35% Traffic, 30% on PHYRE, 25% - 35% on OBJ3D, and 8% on CLEVRER). This comes with a marginal cost in the distortion measures SSIM and PSNR (2% on Traffic, better performance on PHYRE, 1.5% on OBJ3D, and 3% on CLEVRER)), demonstrating a clear improvement in the perception-distorionn tradeoff (Blau & Michaeli 2018). We attribute these results to the use of the perceptual loss in our model. While it possible to apply the same loss to G-SWM, for SlotFormer this would require even higher memory requirements. Notably, even when trained with pixel-wise MSE, DDLP outperforms G-SWM in terms of perceptual similarity (LPIPS) for the Balls-Interaction and PHYRE datasets, as shown in Tables Observation 5: DDLP errs when new objects appear during the contert window. Upon analyzing the videos for which DDLP obtains lower LPIPS scores than the baselines, we observed that these are typically scenes where new objects emerge during the context window, which DDLP not designed to capture. On CLEVRER
2 and 3
We the open-source PIQA library Rozel 2022
10

---

Published in Transactions on Machine Learning Research (02/2024)
b-10
15
tw30
1-40
Figure 10; Rollout from the Traffic dataset. Models are conditioned on the first 10 frames and predict the next 40. DDLP exhibits sharper predictions compared to G- SWM and FlotFormer. The limited number of slots SlotFormer results in missed objects. Zoom-in for better observation.
6-5WM
and Traffic, for instance, objects can enter the scene at different time-steps, providing G-SWM an advantage because of its object discovery module, as depicted in Figure
We further discuss DDLP's limitations
Section 0
Balle-Interaction
MED10 J.
LPIPS
G-SWM
0.256
0.23:0.15
DDLP (Ours)
0.149
0.18+:0115
-No RPE
0.157
0.194
-DLP (No Tracking)
2.736
0.393
Table 2: Iatle-Interaction dataset comparison. MED10 the mean euclidean distance error of pre- dicted balls positions, summed over the first steps of generation. Bottom right: Ablation study on the Balls-Interaction dataset. In each row, we remove one component from the full model. No RPE standard positional embedding is used instead of relative positional encoding. DLP (No Tracking) standard single-image DLP encoding for all frames instead of tracking.
Traffic
PSNR 1
SSIM 7
LPIPS I
PSNR 1
SSIM
LFIPS L
G-SWM
31.716.2
0.924.1000
0.118:10007
24.8811.007
0.5810.07
0.235:10.04
FlomFrrmer
31.2+4.91
0.925=0.04
0.135:0.05
24.93:0.85
0.60+10.04
0.18::0.03
DDLP (Ours)
3..29
0.923
0.08810.06
24.441102
0.61
0.15:10.02
Dataset
CLEVRER
PHERE
G-SWM
30. 10.1226.69
0.93=0.01
0.146200007
21.61:66.22
0.93:00.05
0.07840006
DDLP (Ours)
29.21=6.16
0.92=0.04
0.134:20008
26.98:55.3
0.95
0.055:10.04
Table Quantitative results on video prediction.
5.3
What If...? Video Prediction
As DDLP provides with an interpretable latent representation, can directly modify scenes in the latent space by changing objects' learned attributes position, scale, depth or shape. Equipped with a pre-trained DDLP on OBJ3D from the previous section, we take an initial sequence of frames, encode them to latent. particles and manually locate and modify specific particles corresponding to objects-of-interst according specific scenario, c.g., "what the green ball moved the left?". Then, we unroll the latent sequence with the dynamics module and decode to produce video of the consequences of our interventions. Row
in practice apply the same modification (e.g., move particle by Ar +0.2) for the first. Sburn-i frames" (r= 4) assume small displacements between frames.
11

---

Published in Transactions on Machine Learning Research (02/2024)
3 in Figure I demonstrating repositioning of objects, and additional examples in Appendix 1.7 controlling scale and visual features, demonstrate that PINT generalizes well to our interventions. We emphasize that "what if" generation exploits DDLP's unique structure, and to our knowledge cannot be obtained with alternative representations such as slot-based models. See https://taldateet// hwwpalddddaa....hh......... github 1o wo/dddp--edbbb for video rollouts.
5.4 Unconditional Video Generation with DiffuseDDLP
We propose an application of DDLP to unconditional object-centric video generation by learning diffusion model over the latent particles, termed DiffuseDDLP. Our main observation that training the diffusion model over particles instead of over pizels is more efficient, and exploit the object-centric structure of the problem to produce high quality videos.
We base our model on Denoising Diffusion Probabilistic Model (DDPM, Ho et al. 2020, see Appendix B for an extended description). In DiffuseDDLP we train a DDPM to generate the latent representation of the first т = 4 frames of video, & E RTxXxF, where is the number of particles, and is the total dimension of the particles' features, using data from pre-trained DDLP model (cf. Section 5). Then, these generated particles are fed PINT to predict particles in T subsequent frames, yielding a sequence of particles 2y TxF Finally, the DDLP decoder produces sequence of images RR Tx3x HxW from the generated sequence of particles.
A key technical finding that we discovered to be important is the structure of the denoising neural network in the diffusion model. Typically, a UNet model is used for DDLP (Ho et al. 2020 However, we find that using DDLP's Translormer-based dynamics module architecture to serve as the DDPM's denoiser yields significantly better results. our DDPM operates on the compact latent representation of DDLP, training takes less than day on a consumer-grade GPU, and for inference are able generate 100-frame-long videos in a few seconds on standard CPU.
We train DDI. on 3 datasets: Balls- Inils-Interaction, OBJ3D and Traffic using the pre-trained DDLPs from Section 5 Figure [1 shows frames from generated videos, demonstrating faithful interactions between the objects and high visual quality. Please visit In Figure 14 in the appendix we plot the first frames that were generated by the DDPM diffusion model (decoded to images for visualization purposes), and demonstrate high temporal consistency between the generated particles. Extended details and more visualizations are available in Appendix B
github. 1 detp/ddllwb/////// to view videos.
0BJ3D
Traffic
Balle
Figure 11: Video generation with DiffuseDDLP. A DDPM diffusion model first generates the first t = frames represented as latent particles, which are then unrolled the future with a pretrained PINT dynamics module, and then decoded to pixels with a pretrained DDLP decoder.
12

---

Published in Transactions on Machine Learning Research (02/2024)
6 Conclusion and Limitations
DDLP is new object-centric vidco prodiction model that uses DLPs as an underlying representation, thereby allowing to scale in the dynamics prediction complexity (compared to patches) and number of objects in the scene (compared to slots). We have demonstrated DDLP's SOTA performance in video prediction on various datasets and utilized the explicit latent particles representation to modify videos in the latent space. We see great promise in modcling dynamics with the particles representation as tool understand the consequence of interventions, potentially in reinforcement learning where actions also involved in the physical dynamics.
We conclude with several limitations of DDLP, which may inspire future investigation. Failure cases are further discussed in Appendix J The assumption that consecutive displacements are small is not valid when new objects enter the scene; we hypothesize that a birth/death process for new particles such as in G-SWM can help mitigale this problem. Our approach does not account for camera movement, i.e., scenes where the background constantly changes and objects appear and disappear between frames. Finally, for videos with objects of various different sizes, the model architecture should be modified to handle multi-scale patches, which we leave for future work.
t-10
Figure 12: Limitation: DDLP fails to capture new objects that appear during the context window.
DOLP
G-SWM
Acknowledgments
This work received funding from the European Union (ERC, Bayes-RL, Project Number 101041250). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Ageney. Neither the European Union nor the granting authority can be held responsible for them.
13

---

Published in Transactions on Machine Learning Research (02/2024)
References
Anonymous, Structured world representations via block-slot attention. In Submitted to The Eleventh International Conference Learning Representations, 2023. URL cect::://ppemeevieeenne/////// 2PHE4fht19t under review.
Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. Phyre: A new benchmark for physical reasoning. Advances in Neural Information Processing Systems, 32, 2019.
Peter Battaglia, Razvan Pascamu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems, 29, 2016.
Yochai Blau and Tomer Michacli. The perception-distortion tradcoff. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 6228 6237, 2018.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Trina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. orXiv preprint arXiv:1901.11390 2019.
Eric Crawford and Joelle Pincau. Spatially invariant unsupervised object detection with convolutional neural networks. In AAAI, 2019.
Tal Daniel and Aviv Tamar. Unsupervised image representation learning with deep latent particles. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 4644 4665. PMLR, 17 23 Jul 2022a.
Tal Daniel and Aviv Tamar.
2022b.
Emily Denton and Rob Fergus. Stochastic video generation with learned prior. In International conference on machiin learning, pp. 1174 1183. PMLR, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Froderik Ebert, Chelsca Finn, Alex X Lee, and Sergey Levine. Sclf-supervised visual planning with temporal skip connections. In CoRL, pp. 344 356, 2017.
Gamaleldin Fathy Elsayed, Aravindh Mahendran, Sjoerd van Steenkisle, Klaus Greff, Michael Curtis Mover, and Thomas Kipf. SAVi++: Towards end-to-end object-centric learning from real-world videos. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Newral Information Processing Systems, 2022. URL /tttps:/ http:///pppprrooeeeeaeeaaorrrrrp 2tss/foormmmdd=ff1555122222
Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. arXiv preprint 1rX10:1907.13052, 2019.
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object representa- tions without iterative refinement. ar preprint NXi::2104.09952 2021.
Chelsca Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. Advances in neural information processing systems, 29, 2016a.
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEFE International Conference on Roboties and Automation (ICRA), pp. 512 519. IEEE, 2016b.
Xiaojic Gao, Yueming Jin, Qi Dou, Chi-Wing Fu, and Pheng-Ann Heng. Accurate grid keypoint learning for efficient video prediction. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (TROS), pp. 5908 5915. IEFF, 2021.
14

---

Published in Transactions on Machine Learning Research (02/2024)
Klaus Greff, Raphaäl Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, pp. 2424 2433. PMLR, 2019.
Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle videos revisited: Tracking through occlusions using point trajectories. arXiv preprint arXiv:2204.0411 04,13, 2022.
David Held, Sebastian Thrun, and Silvio Savarese. Learning to track at 100 fps with deep regression networks. In European conference computer vision, pp. 749 765. Springer, 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE. Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Yedid Hoshen, Ke Li, and Jitendra Malik. Non-adversarial image synthesis with generative latent nearest. neighbors. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5811-5819, 2019.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances in newral information processing systems, 2022,, 2015.
Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of object landmarks through conditional image generation. In Proceedings of the 32nd International Conference on Newral Information Processing Systems, pp. 4020-4031, 2018.
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. orXiv preprinl arXiv:1910.0253 2019.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901 2910, 2017.
Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matt Botvinick, Alexander Lerchner, and Chris Burgess. Simone: View-invariant, temporally-abstraceed object representations via unsupervised video decomposition. Advances in Neural Information Processing Systems, 34:20146 201599, 2021.
Andrej Karpathy.
2021.
Yunji Kim, Seonghyeon Nam, In Cho, and Scon Joo Kim. Unsupervised keypoint learning for guiding class-conditional video prediction. Advances in neural information processing systems, 32, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 12 2014.
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), ICLR, 2014.
Thomas Kipf, Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexcy Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. arXio preprint Xrxi::2111.12 12594, 2021.
Jannik Kossen, Karl Stelzner, Marcel Hussing, Claas Voeleker, and Kristian Kersting, Structured object-aware physics prediction for video modeling and planning. Xixie preprint arXin:19100 1910.02225.. 2019.
15

---

Published in Transactions on Machine Learning Research (02/2024)
Tejas Kulkarni, Ankush Gupta, Catalin lonescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised learning of object keypoints for perception and control. Xr preprinl arXiv:1906.1188 2019.
Alex X Lee, Richard Zhang, Frederik Ebert, Picter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprinl Xrxi::180, 010.2, 2018.
Yunzhu Li, Antonio Torralba, Animashree Anandkumar, Dieter Fox, and Animesh Garg. Causal discovery in physical systems from videos. arXiv preprint arXiv:2007.00691, 2020.
Zhixuan Lin, Yi-Fu Wu, Skand Peri, Bofeng Fu, Jindong Jiang, and Sungjin Ahn. https://github.ce zhixuan- Shixuan-lin/G-Be 1in/G-SWM 2020a.
Zhixuan Lin, Yi-Fu Wu, Skand Peri, Bofeng Fu, Jindong Jiang, and Sungjin Ahn. Improving generative imagination in object-centric world models. In International Conference on Machine Learning, pp. 6140- 6149. PMLR, 2020b.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. orXiv preprint urXiv:2001.02407 2020c.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. ar preprint arXiv:2006. 15055, 2020.
Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin P Murphy, and Honglak Lec. Unsuper- vised learning of object structure and dynamics from videos. Advances in Neural Informalion Processing Systems, 32, 2019.
Akihiro Nakano, Masahiro Suzuki, and Yutaka Matsuo. Interaction-based disentanglement of entities for object-centric world models. In The Eleventh International Conference on Leurning Representations, 2023. URL https:://ppnnrevie:
Charlie Nash, João Carreira, Jacob Walker, Jain Barr, Andrew Jacgle, Mateusz Malinowski, and Pe- ter Battaglia. Transframer: Arbitrary frame prediction with generative models. arXiv preprint arXiv:2203. 09494, 2022.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Alec Radford, Karthik Narasimhan, Tim Salimans, Πya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Colin Raffel, Noam Shazcer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter Liu, et al. Exploring the limits of transfer learning with a unified lext-to-lext transformer. J. Mach. Learn. Res., 21(140):1-61 67, 2020.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recogriitio,, pp- 779-788, 2016.
François Rozet. https://githhnn http://hhhhh..co//aaaooooooo//// 2022.
Mchdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjocrd van Stcenkiste, Filip Pavetié, Mario Lučié, Leonidas Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. arXiv preprinl ar Xiv:2206. 06922, 2022.
16

---

Published in Transactions on Machine Learning Research (02/2024)
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon- Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, and Francesco Locatello. Bridging the gap to real-world object-centric learning. In The Bleventh International Conference on Learning Representations, 2023. URL hattps://ppenrevier http:://ppparrooeeeeee/ooorrrra wttp/foooumiddd9ttk---EE
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. arXiv preprint arXie:1803.00 02155, 2018.
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. ar preprint arX 2021.
Gautam Singh, Yeongbin Kim, and Sungjin Ahn. Neural block-slot representations. arXi::221..01177, 2022a..
arXiv preprint
Gautam Singh, Yi-Fu Wu, and Sungjin Ahn Simple unsupervised object-centric learning for complex and naturalistic videos. arXiv preprint arXiv:2205. 14065, 2022b.
Dmitriy Smirnov, Michael Gharbi, Matthew Fisher, Vitor Guizilini, Alexei A. Efros, and Justin Solomon. MarioNette: Self-supervised sprite learning. Conference on Neural Information Processing Systems, 2021.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In NIPS, 2015.
Aleksandar Stanić and Jürgen Schmidhuber. R-sqair: relational sequential attend, infer, repeat. arXiv preprint Xix::11910. 052311 2019.
Jennifer J Sun, Serim Ryou, Roni Goldshmid, Brandon Weissbourd, John O Dabiri, David J Anderson, Ann Kennedy, Yisong Yue, and Pietro Perona. Self-supervised keypoint discovery in behavioral videos. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2171-2180, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Rishi Veerapaneni, John Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual bosel-based reinforcement learning. In Conference on Robot Learning, pp. 1439-1455 PMTR, 2020.
Angel Villar-Corrales Ismail Wahdan, and Sven Behnke. Object-centric video prediction via decoupling of object dynamics and interactions. In Internation Conference on Image Processing (ICIP), 2023.
Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee. Learning to generate long-term future via hicrarchical prediction. international conference on machine learning, pp. 3560. PMIRR 2017.
Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc V Le, and Honglak Lee. High fidelity vidco prediction with large stochastic recurrent neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Jacob Walker, Ali Razavi, and Aäron van den Oord. Predicting video with vqvae. ar preprinl ar arXiv:2103.01900 2021.
Phil Wang. http://hht...ooooooc//aaaa/aaaaa
2022.
Yunbo Wang, Haixu Wu, Jianjin Zhang, Zhifeng Gao, Jianmin Wang, Philip Yu, and Mingsheng Long. Predrnn: recurrent neural network for spatiotemporal predictive learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
17

---

Published in Transactions on Machine Learning Research (02/2024)
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEFE tronsactions on image processing, 13(4):600-611 2004.
Marissa A Weis, Kashyap Chitta, Yash Sharma, Wieland Brendel, Matthias Bethge, Andreas Geiger, and Alexander S Ecker. Benchmarking unsupervised object representations for video sequences. J. Mach. Learn.
Res., 22:183-1, 2021.
Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Comput., 1:270-280, jun 1989.
Bohan Wu, Suraj Nair, Roberto Martin-Martin, Li Fei-Fei, and Chelsea Finn. Greedy hierarchical variational autoencoders for large-scale video prediction. In Proceedings of the IEEE/CV Conference on Compuler Vision and Pattern Recognition, pp. 2318 2328, 2021a.
Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. Generative video transformer: Can objects be the words? In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11307 11318. PMLR, 18 24 Jul 2021b.
Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsupervised visual dynamics simulation with object-centric models. orXiv preprinl ar arXiv:2210008801 2022.
Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg. https://giituub.ccmmaairtt SlotFormer] 2023.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpi: Video generation using vq-vae and transformers. arXiv preprint arXte:2104.10157, 2021.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv: 2019.
Jaesik Yoon, Yi-Fu Wu, Heechul Bae, and Sungjin Ahn. An investigation into pre-training object-centric representations for reinforcement learning. arXiv preprint or arXiv:2302.00 0000:2492.4499 2023.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. arXiv preprint rxXf::212.005999 2022.
Andrii Zadaianchuk, Georg Martius, and Fanny Yang. Self-supervised reinforcement learning with indepen- dently controllable subgoals. In Conference on Robot Learning, pp. 384-394. PMI.R, 2022.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 586-595, 2018.
Daniel Zoran, Rishabh Kabra, Alexander Lerchner, and Danilo J. Rezende. Parts: Unsupervised segmentation with slots, attention and independence maximization. In Proceedinggs of the IEEE/CVF International Conference on Compuler Vision (ICCV), pp- 10447, October 2021.
18

---

Published in Transactions on Machine Learning Research (02/2024)
Appendix
Contents
Introduction
2
Related Work
3
Background
3
Method
4
DLPv2 Moditied Latent Particle
5
2
Particle Tracking Posterior
6
4.3
Image Decoder
7
4.4
Generative Predictive Dynamics with Particle Interaction Transformer
7
Experiments
8
5.1
Datasets and Experimental Setup
9
5.2
Unsupervised Video Prediction Results
5.3
What If...? Video Prediction
11
5.4
Unconditional Video Generation with Diffuse 1
12
6
Conclusion and Limitations
13
A Broader Impact Statement
21
B DiIPuseDDLP: Unconditional Video Generation with Latent Particle Denoising Diffusion
21
B.1 Background: Denoising Diffusion Probabilistic Models (DDPMs)
21
B.2 DDI
22
Unconditional Video Generation with Diffuse 1
23
C Extended Related Work
23
D Method Extended Details
25
D.1 DLPv2 A Modified Latent Particle
25
................................
D.2 DLPv2
wise Conditional Prior
26
................................
D.3 DLPv2 Encoder
27
D.4 DLPv2 Image Decoder
29
D.5 DDLP Particle Tracking Posterior for Videos
30
D.6 DDLP Generative Predictive Dynamics with Particle Interaction Transtormer
31
E Additional Training and Optimization Details
32
19

---

Published in Transactions on Machine Learning Research (02/2024)
F Complexity Analysis
34
G Datasets
35
H Extended Comparison with SlotFormer
35
Additional Results
37
L1
DLPv2 Unsupervised Object-centric Scene Decomposition
37
[.111
Comparison of DLPv2 vs. DLPvl
37
Video Prediction: Balls- Interaction
................................
3
Video Prediction: Trallic
41
Video Prediction: OBJ3D
41
L5
Video Prediction: PHYRE
42
Video Prediction: CLEVRER
42
What If..?? Video Prediction
.8
Ablation Study
43
Failure Cases
43
20

---

Published in Transactions on Machine Learning Research (02/2024)
A Broader Impact Statement
DDLP is a SOTA object-centric video prediction model that offers an efficient and interpretable solution, outperforming previous methods. The implications of this research may touch upon various societal sectors.
Al Reseorch: The proposed model can improve the interpretability of unsupervised scene decomposition and understanding, leading to more transparent Al systems. The public availability of our code and pre-trained models could foster reproducibility, aiding other researchers in building upon this work.
Autonomous Vehicles and Robotics: Accurate object-centric video prediction may be important for safe and effective operation of robots. DDLP might aid in faster, interpretable decision-making and risk assessment.
Healtheave: DDLP may also find applications medicine, for example analyzing medical videos which can be decomposed to interacting objects, assisting in downstream tasks of diagnosis and treatment.
Like other generative algorithms, this model could be exploited for misuse. The interpretability of DDLP could help in this regard, allowing for detection and regulation mechanisms to be developed.
B DiffuseDDLP: Unconditional Video Generation with Latent Particle Denoising Diffusion
In this section, propose to extend DDLP to unconditional object-centric video generation by learning diffusion model over the latent particles. We base our model on Denoising Diffusion Probabilistic Model (DDPM, Ho et al. (2020)) and the publicly available code base (Wang, 2022). Our DDPM employs a pre-trained DDLP model to generate the latent representation of the first few frames of a video, which are then used as input to the conditional dynamics module DDLP that unrolls the latent representation of the video into the future, and decodes them back pixel-space with DDLP's decoder. Crucially to the success of our approach is the design of the denoiser we adapt DDLP's Transformer-based dynamics module architecture to serve as the DDPM's denoiser instead of the typical UNet for 1D and 2D data, As our DDPM operates on the compact latent representation of DDLP, training converges in less than day and we are able to generate 100-frame-long videos in a few seconds on a standard CPU.
B.1 Background: Denoising Diffusion Probabilistic Models (DDPMs)
Denoising Diffusion Probabilistic Models (DDPMs) are probabilistic models that aim to capture the density of a given dataset by assuming that it arises from a diffusion process. The diffusion process is modeled as a sequence of invertible diffusion steps, with each step being parameterized by a time-dependent noise level.
Forward Diffusion Process: The forward diffusion process begins with data point sampled from a real data distribution, æu g(x). In this process, Gaussian noise, with variance schedule 3, (0, (0,1),=1, is added to the sample in T steps. This produces sequence of noisy samples, IT. Mathematically, the forward diffusion process is represented as:
T ((11T(=)) II (()1(1111)
(( = N(2;; 11-,,,0-11,80 BrI).
As 3, + 1, Fr equivalent to an isotropic Gaussian distribution. The forward process allows æ, to be sampled at any arbitrary time step using the closed-form solution: (20, N((z; varzo, (1 aa).), where āz= [1 Bi).
Reverse Diffusion Process: Given a sample from a Gaussian noise, .T NO, 7), the goal of the reverse diffusion process is to model ((x,) to reach a true sample zp. However, this is intractable, and therefore, the reverse process in DDPM approximates these conditional probabilitics with a learned model, pe, which is defined as:
T
pe = p(Tr)
II
pa (
21

---

Published in Transactions on Machine Learning Research (02/2024)
po( (1 Eg(zr,)),
Loss Function: During training, the Gaussian noise term is reparameterized, and a neural network is trained to predict the added noise Er from the input 2, at time step t. The loss term Le minimizes the difference from f 1_ vac (xt
(1) (t). Mathematically, the loss function represented as:
= [A]ls colVārz0 VI Cef t)
where A (1-a,) However, in practice, was found that DDPMs work better without the A term, simplifying the loss function and removing the need to model Eg.
B.2 DiPFuseDDLP
We propose diffusion-basse approach for unconditional object-centric video generation termed DiffuseDDLP. Our approach employs a DDPM over the latent space of pre-trained DDLP as follows: given sequence of imput frames Tu, ...,..... -, we encode sequence of latent particles H= E RT*KxF with DDLP's frozen encoder, where K is the number of particles, including a background particle, and is the total dimension of the concatenated attributes (position, scale, depth, transparency and visual features). Then, we train DDPM model to approximate the distribution of the latent particles sequence of length т. Equipped with a trained DDPM, we sample sequences of T particles and use them as input to the pre-trained DDLP dynamics module, PINT, unroll the sequence to horizon T, where T >>> (we use = 4 for all datasets). Finally, we generate video by decoding the sequence of generated particles back to pixel-space with DDLP's pre-trained image decoder. The overall process is depicted in Figure 13
{E E TXCXHXW
GEN:O
{ ~N(1
No0 ( ) 1=0 IXXXX
Particle Decoder
Add & Norm
Zo
Z1
Z2
Add & Norm
Feed
Feed Forward
Zo
Z1
ZT
Forward
Norm
Add & Norm
Multi-Head
Multi-Head Attention
Attention
N Denoising Steps
PINT Denoiser
DDLP's PINT
0,0 0,0 T-1 {2x t=
T-1
E TxxxD
E R TXXXD {Z 1t=0
Figure 13: DiffuseDDLP model illustration. We learn DDPM generate the first 7 frames represented where K is the number of particles, and F is the total dimension of the
latent particles, 2, B KxF Rx particle's features. Then, unroll the particles evolution over T steps using DDLP's PINT, 2T E X and decode them to back to pixel-space with DDLP's image decoder to generate a video E xTx3*HxW
Denoiser Architecture: The typical denoiser architecture based on a 1D or 2D UNet. However, due to the unique structure of DDLP's latent space, a denoiser architecture that considers the evolution of
22

---

Published in Transactions on Machine Learning Research (02/2024)
set of particles over time is required. In initial experiments, both the 1D and 2D UNet introduced unsatisfying visual artifacts, with the 1D UNet performing slightly better. However, we find that, the same transformer-based architecture that was introduced in DDLP, PINT, performs well in predicting the dynamics of the particles. In Diff we repurpose PINT to perform denoising instead of prodicting the next state of the particles. We keep the same architecture and positional bias that were introduced in DDLP, ensuring that the denoiser is optimized for DDLP's unique structure.
B.3 Unconditional Video Generation with DiffuseDDLP
We train DDI. on 3 datasets: Balls-Interaction, OBJ3D and Traffic using the pre-trained DDLPs from Section 5 In all our experiments, we generate the first 7 = 4 frames represented as latent particles, 2, R with the Icarned DDPM, unroll their cvolution over T 100 steps using DDLP's PINT, 27 RTxKxF and decode them to back to pixel-space with DDLP's image decoder to generate video $ E RTx3xHxW We trained DDPM for 100, 000 iterations and used N 1000 denoising steps in DDPM both during training and inference. As DDLP's latent space is compact, training DDPM is fast and takes less than a day on single RTX A4000 GPU. For example, for OBJ3D, each frame is represented as 12 foreground particles and background particle, i.e. K = 13, and each particle has a total F = 14, 13 14 = 728, which much smaller than a single CIFAR1O image,
resulting in r x K x F C H x W = 32 x 32 3072.
Results:
The results in Figure 14 demonstrate that the DDPM in DiffuseDDLP generates temporally consistent latent particles (decoded to images for visualization purposes), which are then unrolled to the fulure with DDLP's PINT, resulting in faithful interaction dynamics depicted in Figure [1 Moreover, the generated videos showease the high quality of the generated content. Please visit https://taldatee github 1o/ ddlp-webb to view the vidco generations.
t=
t= 2
t = 4
Balls
OBJ3D
Traffic
Figure 14: Generation results of the first. r = frames represented as latent particles by the DDPM of DiffuseDDLP, decoded to pixel-space with DDLP's image decoder for visualization purposes.
C
Extended Related Work
We discuss different video prediction studies based on their representation for objects: keypoints, object-centric models, and general latent vector-based representations.
23

---

Published in Transactions on Machine Learning Research (02/2024)
General unsupervised latent video prediction traditionally, latent-based video prediction is composed of encoding visual observations into a compact latent space, predicting the next latents using a recurrent. dynamics module, and decoding them back into images. Early models (Finn et al. 2016a Ebert et al.) 2017: Villegas et al.) 2017 Lee et al.) 2018 Denton & Fergus 2018) learn latent representations with a convolutional-basee architecture and an RNN- -aadeaded dynamics module to model the latent dynamics, while recent attempts to improve long-horizon prediction use discrete latent space (Walker et al. 2021) or focus ucaling-up via hierarchical modeling or larger architectures (Villegas et al., 2019 Wu et al. 2021a] Wang et al, 2022). All aforementioned methods model the dynamics based on representation extracted by considering the whole scene at once, without explicit decomposition to objects, which leads to blurriness and objects
disappearing in the long-term prediction (Wu et al. 2022). Several recent works (Nash et al. 2022 Yu et al.
2022 Yan et al. 20211) model the dynamics using self attention (Vaswani et al.) |017]] Dosovitskiy et al.,
2020 While these methods improved the visual quality of the generated videos, they do not explicitly model
objects and are outperformed by object-centric models on data involving complex physical interactions (Wul
et al.
2022).
Keypoint-based unsupervised video prediction: the idea of utilizing keypoint- beypeint-based based latent represen- tations has been explored in several works. Kim et al. (2019) Iearn a class-guided vidco prediction model by learning a KeyNet (Jakab et al. -based keypoint detector and then training a motion generator implemented as a recurrent adversarial conditional VAF. Similarly, (Minderer et al. 2019) and (Gao et al.] 2021) use KeyNet to learn keypoints and propose a variational RNN as a prior to model stochastic dynamics, where the latter further maps the keypoints onto a discrete binary grid for improved performance. While in essence we strive to achieve similar goal, all previous KP-based approaches do not explicitly model object properties and interactions and the visual features are not represented as random latent variables, but derived from the source frame sequence feature maps. This naturally leads to blurriness or disappearing objects in long-term predictions (Daniel & Tamar 2022a). On the other hand, V-CDN (Li et al. (020) first detect unsupervised keypoints with Transporter (Kulkann et al.) 2019), from which a causal graph is built and then performs video prediction of physical interaction with an Interaction Network (Battaglia et al. 2016). Lastly, the original DLP (Daniel & Tamar 2022a) has shown promising results for video prediction from latent particles on a real-world dataset; however, a simple graph neural network is used that is incapable of modeling complex interactions.
Unsupervised object-centric latent video prediction: these methods largely rely on object-centric
representations extracted by patch-based methods (Stanné & Schmidhuber 2019
Crawford & Pineau
2019
Lin et al.) 2020c) or slot-based methods (Burgess et al. 2019;
Locatello ct al.,
2020 Greff et al. 2019
Engeleke el al. 2019: 202T; Kipl et al. 2021 [Sngh et al. 2022b Kabra et al. 2021
Singh el al. [2021] 2022a
Anonymous 2023 Sajjadi et al. 2022 Weis et al., 2021 Vecrapaneni et al. 2020 STOVE (Kossen et al,
2019), SCALOR (Jiang et al.] 2019) and G-SWM (Lin et al. 2020b) belong to the patch-based family of object-centric representations, and use the 'what', 'where', 'depth', and 'presence' latent attributes to represent the objects, and an RNN to model the latent dynamics, where SCALOR and G-SWM add an interaction module to model the interactions between objects. OCVT (Wu et al. 2021bb uses a similar representation, and a Transformer (Vaswani et al., 2017 Radford et al., 2018) is used to model the dynamics in a 2-stage training scheme. As the patch- based approaches produce many unordered object proposals, a matching algorithm is used to match between objects in consocutive frames before the input to the Transformer, making OCVT hard to scale for real-world datasets. In contrast, our method is trained end-to-end and is based on keypoints, which both significantly reduces the complexity of object decomposition and does not require a matching algorithm to track objects. PARTS (Zoran et al. 2021) and STEDIE (Nakano let al. 2023) use slot-based representations and employ an RNN-based dynamics model of the slots, while OCVP [Villa-Corralle et al. 2023) and Flomer (Wu et al. 2022) use a deterministic Transformer to model the dynamics of the slots in a 2-stage training strategy, utilizing the attention mechanism to model the interaction between slots. Differently from SlotFormer, our model uses the more compact latent particles, providing an explicit keypoint- based representation for the objects' location and appearance, which allows for video editing in the latent space, and uses the Transformer as stochastic prior for latent particles that is trained jointly with the particles representation, removing the need to sequentially train two separate models.
We use the term "latent video prediction' to refer models where the video prediction is based latent encoding of the original frames.
24

---

Published in Transactions on Machine Learning Research (02/2024)
As we show in experiments, thanks to the compact representation, our model can fit scenes with many objects. Table 1 summarizes the object-centric video prediction approaches.
D Method Extended Details
In this section, we go into the finer details of our model with accompanying visualizations and illustrations. We begin with our modified definition of latent particle, followed by an overview of the structured keypoints prior used in DLP and how to encode latent particles from a single image. Next, we extend the encoding scheme to a video, L.e, & sequence of image frames, to allow tracking particles over time. Then, we detail how to reconstruct (decode) images from particles. Finally, we construct Transformer- based dynamics module that serves a stochastic dynamics prior for the particles' evolution over time.
D.1 DLPv2 A Modified Latent Particle
For a more accurate prediction and to improve the autoencoding performance, we extend the definition of a latent particle from the original DLP as described in Section 3 with additional attributes, following recent works in object-centric video modeling (Lin et al. 2020b Jiang et al.) 2019). y A foreground latent particle = [žp; Zxy Zd, 2t, zy] E R6-m is a disentangled latent. variable composed of the following learned stochastic latent attributes: position 2p [Mp,, op) E RR, scale 2, ~ N o²) I depth zd ~ NN (()aa))) E R, transparency ze Beta(a,, b.) R and visual features 2f NNu,, of) Rm where m the dimension of learned visual features. Moreover, we assign single abstract particle for the background that is always located in the center of the image and described only by mbg latent background visual features,
Fbg ~ WV o%. E mhs.
The motivation for the additional attributes follows previous works (Lin et al. 2020b Jiang et al. 2019) that demonstrated the importance of explicitly modcling the scale and depth of objects in vidoos with objects of variable sizes or videos of 3D scenes. Next, we motivate the role of each attribute as illustrated in Figure B
Position &p E =R: describes the spatial location of the particle, i.e., its x- coordinates in [-1, 1j. Similarly to other object-centric models (Lin et al. 2020b Jiang et al. 2019), 2p is a modeled as a Gaussian distribution N(Hp, oz). As in DLP, the prior for the position is based on SSM from patches, which constrains Zp to have a spatial meaning.
Scale 2, E R2: describes the height and width of the bounding box around the particle. Similarly to other cenecc-centric models, 2, is modeled as a Gaussian distribution ((u,, of), and activated with a Sigmoid function to constrain the values to be in 10.
Depth E R: describes the depth of the particle. Its main role is to define the order in which the decoded objects are stitched together when reconstructing the image. Similarly to G-SWM (Lin et al.. 2020b), zd is modeled as a Gaussian distribution ((Ha, oz))
Transparency 2, E R: describes the degree to which a particle's features are part of the scene, a value in [0,1]. 1]. When 2 = 0, the object is completely transparent and not part of the scene, when 2, 1, the object completely visible, and when 2 (0, 1) the object is partially visible. Unlike previous works which model 2 a "presence" variable sampled from Bernoulli distribution, in DLPv2 model 2 as Beta distribution, namely Beta(a, b.). There are two main benefits of doing so: (1) unlike Bernoulli distribution which is discrete and requires relaxations or gradient estimations to be differentiable, Beta distribution is continuous and supports differentiable sampling via reparameterization, making the optimization process simpler, and (2) allowing continuous values for the transparency in (0, 1) can naturally model partially visible objects in the scene. Fortunately, similarly to Gaussian and Bernoulli distributions, Beta distribution has a closed-form KL-divergence solution.
Features 2 E _T Abg E R describes the particle's visual appearance, i.e, features of its surrounding region. Similarly to other object-centric models, 24 and 2bg are modeled as Gaussian distributions (( and ((bby respectively.
25

---

Published in Transactions on Machine Learning Research (02/2024)
D.2 DLPv2 Patch-wise Conditional Prior
Similarly to the original DLP, to disentangle position from appearance of the particles, we consider conditional VAE formulation (Sohn et al. 2015), where prior for the keypoints is explicitly learned given image x, (=(x non-position particle attributes. The prior distribution parameters, means and covariances for Gaussian distributions, and a, in the Beta distribution, for the rest of the attributes za are not learned, i.e, they are hyper- paprr-parameters.
Py (=pla)) X (Z ()), where = [=a, Rdy 2t, æy] denotes the latent variable for all
To generate a set of prior keypoints, termed keypoint proposals we follow a patch- patch-based based approach, which was found to work well in prior work (Jiang et al.) 2019 Lin al. 2020b Daniel & Tamar 2022a). The
imput E "xxx3 x3 is split into Kp patches of size D x D (usually D 18, 16)), and for each patch, small convolutional neural network (CNN) followed by spatial-softmax (SSM) layer outputs single keypoint proposal. A similar process of unsupervised keypoint detection via SSM has been proposed in Iakaab et al. (2018); however, in DLP there are two main differences: (1) the process is applicd over patches and not the whole image, producing a single keypoint per-par per-patch using a single heatmap as input to SSM and (2) the output only serves as prior and not directly used to reconstruct the image.
In practice, the set of proposals can grow large with the number of patches, we learn to filter the keypoints to produce subset of L prior keypoints, where L is parammerr Differently from the original DLP, where the I. chosen keypoints were the top-I. distant keypoints from the center of their respective center, in DLPv2 SSM variance & = o +o +0 (Sun et al.
the top-L keypoints with the smallest sum of empirically-caleus 2022], are chosen. The motivation for using the variance is that the SSM operation detects areas-of if-terrst based on the activation heatmap, and when an object, or part of it, is found, the activation is tightly centered around it, i.c., sharp softmax values, as described next. Empirically, using the SSM variance resulted in better detection of small objects than the original distance heuristic of Daniel & Tamar (2022a). The architecture of the proposed prior is illusterated in Figure 16
The process is formally described as follows. Given a heatmap H E RRx which is the output of a CNN applied to an image patch, the softmax function is applied spatially over the H and W dimensions to produce probabilities for each coordinate. Using these values, we calculate the expected coordinate values for each axis, and their covariance following (Sun et al., 2022). Given a softmax-activated normalized heatmap E IR xW where each value hij = (i. indicates the probability of coordinate (i.j) to be keypoint, we are interested in extracting the expected coordinate, the mean, (Mz, Myy and the coordinate covariance values of og and Day as follows:
P5 = Σ Fi Σ H(i,))
By = Σ Vi Σ H(ij))
0 Σ (1: pz) )² HH(i))
o Σ (1) mb) Hi,
Gry = Σ (x= - Hb) - Hy)
Note that E, Hi and Σ. Hi.j) are the marginalized distributions per-axis. The SSM operation is illustrated in Figure [5] Intuitively, in regions where the activation is sharp, we expect the softmax to produce a distribution with a small covariance values, which may indicate a non-background arca-of-interest,, c.g., objects, lines or corners, making the low-covariance suitable criterion for detection and filtering. Note that the SSM operation is differentiable, i.e., the heatmap values are learned according to reconstruction
26

---

Published in Transactions on Machine Learning Research (02/2024)
objective of the images, thus leading the SSM operation to locate regions of the image that crucial for appropriate reconotruction of the scene.
Architecture details: the architecture of the prior encoder, following the original DLP, is a small CNN that is fed with the patches, structured has a 3 layers, ReLU activated with 4-group Group Normalization, with [16, 32, 64] or [32, 32, 64] channels in each layer, ending with 1 convolution mapping a single channel output per patch, where SSM applied over the 2D feature map output.
H
(Hx (H)(H), (y)
(2)
Oxy(H)
ox (H)
Figure 15: Spatial-softmax. Given heatmap H "xw which is the output of CNN, the softmax function is applied spatially over the H and W dimensions produce probabilities for each coordinate. Using these values, we calculate the expected coordinate values for cach axis, and their covariance.
po ug, Mby N M.
Patchlfy
CONV
SSM
NN
Filter I Propoeals
Patchas
Patch-uise Conditional Prior
Patch-wise conditional prior architecture. The input x E #xxx3 split into patches, and for each patch, small CNN followed by an SSM layer outputs single kcypoint proposal. We filter top-L prior keypoint proposals with the smallest sum of empiricalll-celeuullee SSM variance.
Figure 16:
D.3 DLPv2 Encoder
The role of the encoder is to produce the posterior latent particles for input image. To that end, it models the approximate posterior, (2(2)²) (z.l2) x gø(2os Rey Ra, 21|2, 2a) (o(2y/z, Zps 2a), where 2 anchor keypoints and %o are offsets from these anchors that together comprise 2p 2a + Z.. The motivation for this modular approach originates in anchor-based object delection (Redmon et. al. 2016 showing that using anchors produces more accurate bounding-boste boxes by relaxing the optimization objective to only predict the offsct from the anchor instcad of the exact position. This idca later propagated unsupervisod object detection (Lin al.] 2020cc Smirnov al. 2021)). significantly improving the detection performance. The encoding process is hierarchical and composed of steps: (1) anchor encoding learned offset of K keypoint.
27

---

Published in Transactions on Machine Learning Research (02/2024)
anchors from the L keypoint proposals; (2) attribute encoding extraction of position offset, scale and transparency attributes for each particle and (3) appearance encoding encoding the particles visual features. In the following, we explain the design of each step.
We begin with encoding K particle positions. DLP of Daniel & Tamar (222a) uses a CNN followed by a fully-connected (FC) network to encode K posterior keypoints from the whole image. We found this approach results in inaccurate object positions and misses small objects; in DLPv2, we depart from this approach to a fine-grained resolution approach where we utilize patches, or glimpses, 2p E R XXXXXXX3 where S' is hyperparameter for the patch size. To that end, we model the anchor encoder g0(2z (z) as follows: first, similarly to the keypoint proposals selection process described in Sec. D.2 we take the top-K lowest SSM-variance keypoints out of the L proposals to get a set of anchors pk- Then, patches are extracted by using the prior keypoint proposals PL E R as imput to a Spatial Transformer Network (STN, Jaderberg ct al. 2015). Next, a small CNN outputs features (rp) from each patch which are flattened and fed through a feedforward network (FFN) to output the offset o(p) of cach anchor which is added to the original K selected anchors to get a set of modified anchor keypoints za pK + ok R This set of keypoint. anchors approximates objects location, and we proceed to the attribute encoding step where we lock a more accurate position and extract the rest of the spatial attributes.
Next, to extract the position ofsset 2a, scale 2x, depth za and transparency 2 attributes for each partiele, we extract glimpses from the anchor keypoints 2. via STN as before, which are input to the attribute encoder go(2o; Ræs Rd, ze|æ;, za) modeled as small CNN followed by a FC network that outputs the parameters of the distributions described in Section D.1 for each particle. Note that the network parameters are shared between all particles. In practice, the prior network is re-used the anchor encoder (=(=a(2c) and the attribute encoder do produces the offset from the anchors.
With all particle attributes in hand, we follow the original DLP and extract the particle visual features with the appearance encoder 4s (o(zyl,, az, Zp. 2.). Similarly to the previous steps, STN is used to extract glimpses of size S, with the one difference of using the scale 2g as input to the STN in addition to 2p = Zz + ag. The scale parameter important when the regions around objects are smaller or larger than S x S. Note that since we used STN in all steps, all parameters are differentiable end-to-end. The appearance encoder is designed similarly to the attribute encoder and outputs the Gaussian distribution paramcters of the visual features surrounding the particle.
Finally, we assign an abstract particle for the background that is always located in the center of the image and its visual features bbg are encoded with a background encoder (r, 2p) which is a CNN followed by a FC network, similarly to the original DLP. The input to the background encoder is a masked image- the posterior keypoints 2p are used to generate K masks of size S x S that mask out the corresponding regions of the original image x. The entire posterior encoding process is illustrated in Figure 17
Architecture details: the anchor encoder first uses STN to extract patches from K keypoints, PK Kx2 that were filtered based on the keypoints-covariance out of the L keypoint proposals. The patches are fed through 3-ayer, ReLU activated with 4-group Group Normalization, CNN with [16, 32, 64] channels, where the final output of this CNN is flattened and fed through a 3-layer, ReLU activated, MLP with layer sizes [256, 128, 2] to output anchor offsets ок E Kx2 which are activated with Tanh activation. Finally, the anchor encoder outputs the modified anchors 2, = PK + ок E xx* Next, the attribute encoder operates similarly to the anchor encoder patches are extracted from the modified anchors 2, using STN and fed through similar CNN and MLP as before, with the only difference of linear output heads for each attribute, i.c., projection layers for the offsct, scale, transparency and depth. Each head outputs the parameters of 0² the approximate posterior distributions, that is, μ and log for the Gaussian attributes (offset, scale and depth) and (log a, log b) for the Beta-disttrbuted transparency attribute. We use log for numerical stability when calculating the KL-divergence during the optimization process. Finally, for the appearance encoder, we use again the same exact process with the same CNN-MLP architecture to encode the visual features of each particle. Differently from the previous encoding steps, this time we utilize the scale attribute as well as input to the STN to lock-on the object, and the dimension output head of the final MLP is the feature dimension m, which is hyper-parameter Typical values for m are between [3, 12). The background encoder is a 4-ayer CNN with [32, 64, 128, 256] or [32, 32, 64, 64, 128] channels followed by a 3-layer MLP that models
28

---

Published in Transactions on Machine Learning Research (02/2024)
the Gaussian distribution parameters of the background latent with layer sizes [256, 256, 2mbgl where mog is the background latent feature dimension, which for simplicity, we set mbgg = in all our experiments.
Prlor
Anchor Encoder
STN Spaticl Transforner
Filter
STN
Attribute Encoder
Zp
7
7
Zt
Fareground Particlee
(2)
STN
Appearance Encader
zf
KP Moake
Baskgraund Particls
Bockground
Encoder
Zog
Posterior Porticte Enceder
Figure 17: Posterior encoder architecture. We perform hicrarchical encoding process where first approximate object locations by modifying the keypoint proposals that serve anchors. Then, these anchors are used to extract precise attribute per particle, and finally, we encode the appearance features of each particle. The keypoints are used to generate masks to hide parts of the original image, which is then encoded with the background encoder.
D.4 DLPv2
Image Decoder
Recall that the decoder in DLP models the likelihood, pe(x|z) po(x 2zs 2d, Zt, Abg) and composed of an object decoder and background decoder. The object decoder follows the original DLP fully-connected layer followed by a small upsampling CNN that takes in a single latent particle features z/ and decodes an RGBA
(6)
patch of E Sx55x a reconstruction of each object's appearance. Then, the depth 2, and transparency 2; are used factor the alpha channels of the object, which essentially determines the order of stitching and the visibility of the particle, and 2p and 2, used to position and scale the decoded patches in the full H W canvas, Lig, using STN. The transparency and depth factorization process is detailed in PyTorch-style code in Figure 18
The background decoder is similar in design to standard VAEs, taking in 2bg which is upsampled via a FC layer followed by CNN generate bgg- Finally, the foreground components and the background are stitched together by a x fig + (1 a) x Abg. where the masks are the last channel of the RGBA glimpse output of the decoder. The decoder architecture is depicted in Figure 19
Architecture details: the architecture of the object decoder mirrors the architecture of the appearance encoder: a 2-layer MLP takes in 2f E m with 256 hidden units in cach layer and outputs feature vector which is reshaped to a 2D feature map that is then up-sampled by a CNN with similar reversed-order channels to the appearance encoder. The final CNN layer output a 4-channel 2D feature map activated with a Sigmoid activation, where the first channel the decoded alpha map of the particle, and the rest are RGB channels. The background decoder mirrors the background encoder, but in reverse, and outputs the RGB channels of the background component. of the reconstructed image.
29

---

Published in Transactions on Machine Learning Research (02/2024)
def factor_ factor_alphy Baahha_mppaapphaaaa. abj, rgh_ob) z_t, 2_d):
# alpha_ [ipha_obj: obj:[batch [batch_size, size, num par_ppcaelles 1, h, w] glimpses alpha
rgb_obj
[batch_size bbbbobjjbbbh_siia
num_particles, 3, h, w], glimpses RGB
t: batch size; num particles, 1], transparency attribute
# d: batch_ size, num _at___aaatcles,, 1], depth attribute
# factorize transparency alpha obj = alpha obj z # batch size, num particles, 1, h, w]
10
# factorize rgb mask out irrelevant pixels rgba_ obj = alpha_obb
11
rgb_obg
12
# [batch_sszee num pamt_paricles, 3, h, w]
18
# depth-based importance map determine the stitching order importance_map alpha_oby * sigmoid(-z_d) # [batch_size,
14
15
particles. 1, h, w] # normalize importance map between [0, 1]
importance_ map importance map (sum (importaance map, dim=1) le-5) # re-order and stitch the rgba maps
(rgba_ath abj importance_map) (mm(dim=1)) # bbathh_szze 3. h, w]
(dim=1)
21
# create the mask for the background bg mask (alpha obj importance map) sum sum (dim=l)
22
23
# [batch_ size, 1, h, w]
24
return _ojeets__anvvs_, bg mask
26
Figure 18: PyTorch-style implementation of the transparency and depth factorization process in the stitching process.
Bockground Particle
BG Decoder
Zog
Foreground Particles
Object Decoder
2 k-O
STN
Spatial Traneformer
Image Decoder
Figure 19: Image decoder architecture. The object decoder outputs an RGBA patch reconstruction of each object's appearance. The background decoder reconstructs the background component and finally, the foreground components and the background are stitched together to produce the final image reconstruction.
Up till now we have described DLPv2 for single images. We next continue to the details of the vidco prediction model DDLP.
D.5 DDLP Particle Tracking Posterior for Videos
Recall that for accurate dynamics modeling we must track the particles over time. Treating each frame individually would not work, the order of the particles is not aligned between frames and there is
30

---

Published in Transactions on Machine Learning Research (02/2024)
guarantee that objects in consecutive frames will be assigned the same particle. The particle-aligmment problem is illustrated in Figure 20
Particles Aligned
X Particles Not Alignedd
x
X
X
x++1
2r+1
20
21+1
Tracking
No Tracking
Figure 20: Illustration of the particlo-alignment problem. Without tracking, i.c., treating each frame individually, there guarantee that the order of the particles is preserved between frames, e-g-, particle #0 in frame (green ball) and particle #0 in frame 2++1 (red cylinder) describe different objects.
To that end, we made an assumption that the displacement of objects between consecutive frames is small, such that the attribute encoder searches for the object a small region around its previous location. We take steps implement particle tracking: (1) We use p,,-1 as the anchor keypoints for the next frame, i.e. Z 2p,t-1, where for t 0 the anchors are extracted similarly to single-image setting (cf. Section 4.1 and (2) we compute the normalized cross-correlation between the extracted glimpses from x²~1 and x' around 2,,t-1, implemented efficiently with group-convolution. In Figure 21 we provide PyTorch-style code implementing the normalized cross-correlatio. Finally, The single-channel generated score-map is concatenated thannel-wise to the input of the attribute encoder.
def correlate keruel):
# [atch size ch, h1, w1] # kernel: [batch_size size, ch, h2, w2] batch_size ch, hl, wl x. shape x = x. reshape (1, batch size size ch, h1, w1) groups batch size # for group convolution output = cxnv2d(x, kernel,
groups=groups, padding=0, stride=l, bias=None) # correlation identity to calculate the norm
corr id = conv2d(x as 2, ones_ (iike(ken) kernel)), groups=groups, padding=0, stride=l, blas=None) norm = sqrt (smm(eerne) (kennel ** 2) corr_id) return output / (norm + 1e-5)
11
13
Figure 21: Efficient PyTorch implementation of normalized cross-correlation with group-convolution
D.6 DDLP Generative Predictive Dynamics with Particle Interaction Transformer
The dynamics module models the prior distribution (=(2+1/2)) of the particles temporal evolution. A good dynamics model should accurately capture the role of the particles' attributes and the interaction between the particles. For end-to-end training of the posterior and prior, previous bas-d methods (Minderer et al. 2019 Lin al. 2020b) used RNNN-baso dynamic modules as stochastic prior, optionally combined with an interaction module (Battaglia al. 2016). Alternatively, several approaches (Wu al.) 2022 2021b deterministic 2-step training scheme where in the first stage a representation of the visual states is
31

---

Published in Transactions on Machine Learning Research (02/2024)
learned, and in the second stage Transformer-basse dynamic module models the sequence of latent variables where the attention mechanism is used to capture the relationship wetween entities.
Recall that our proposed dynamics module, Particle Interaction Transformer (PINT), is a causal Transformer decoder based on the design of Generative Pre-trained Transformer (GPT, Radford et al. 2018 Differently from previous approaches, we do not add positional embeddings before the atfention layers and use a learned per-head relative positional bias (Shaw et al. 2018 Raffel et al., 2020) which is directly added to the attention matrix before the softmax operation. We decompose the added relative positional bias to a temporal relative positional bias and a spatial relative bias. The relative positional bias is illustrated visually in Figure 23 Btime and B Pos are repeated (K + 1) and N times respectively, and summed to create the relative positional bias mairix B Beime Bpos E (R((K)))))((+++)) {B)b b/ = b, + W/ where i denotes the relative time and j the relative particle, illustrated in Figure 22b The relative positional bias is added directly to the query-key matrix, i.e., Attention = Softmax vD
QK + BB V. To model particle interactions, the attention
operation is modified such that particles attend to each other in the same time-step and across time, i.. the attention matrix A E ((K++))))((++11)) and similarly GPT we causal mask to ensure the attention only considers past inputs, as illustrated in Figure 22a
Architecture details: our particle interaction Transformer (PINT) models the stochastic prior for the particles and is constructed as follows: a 3-layer projection MLP with 256 hidden units takes in the latent particles' concatenated learned attributes {=)0- and projects them to dimension D = 256. Next, a particle Trunsformer, based on the open-source minGPT (Karpathy) (021) with 6 layers and 8 heads per layer for all datasets, performs attention between particles and across time as described before, where we directly add the relative positional bias to the attention matrix before softmax. Finally, 3-layer decoder MLP outputs the prior distribution parameters with similar architecture to the projection MLP. In practice, we model the residual change A = PINT((2 10-22 of the particle attributes, i.e., [2] = {z)0-2 2 + A, for all attributes except for the transparency attribute which is directly modeled as PINT's output. To speed-up training, we use teacher forcing (Williams & Zipser 1989) at training time (i.c., instead of foeding the dynamics module its previous prediction, the ground-truth value is used the posterior particles in our case), while at inference time, predictions are generated autoregressively. We use GELU activations for all MLPs and the standard Layer Normalization in the MLPs and attention layers, where use the pre-normalization architecture. We use Dropout regularization layers with p 0.1 in the particle Transformer attention layers. We follow minGPT's initialization for the ML.Ps and attention layers where linear layers are initialized from a Gaussian distribution (1, 0.02*).
E
Additional Training and Optimization Details
Optimization objective: the objective for DLPv2 follows the original DLP. Given an input image x E "xxx33, we optimize:
CDLP = freela, 1) + 9KL (ChamterrK((qaaaalllllllaaa//// + KL(oo(tar///)llltaarr// + + (
(2)
where ã is the reconstructed image, (xe Z) is the reconstruction loss, Zp are the posterior keypoints, Py (xpla) denotes the patch-wise spatial-softmax prior of the original DLP, žalf are all the attributes scale, transparency and depth, except for the appearance features 2f, and ßKI. and 3y are coefficients to balance the reconstruction loss and the KL-divergence terms. Note that the effective coefficient for the visual features is 3KL- -B,, and in all our experiments we set By 0.001 following the original DLP (Daniel & Tamar 2022a).
For DDLP, we optimize the sum of ELBOs over time, LDILP = - -TT- ELBO(x.) = Ei LDLP, + ΣT-1 faym,t, where LILP, is defined in Equation and
Layn,t = Cree Z.) + (L(g
where here we use the same 0KL coefficient as before for all temporal attributes. As the prior parameters for the KT-divergence are the output of PINT, and not hyper-parameter constants, we found that there was no need, in terms of performance, to explicitly introduce separate coefficients for the temporal attributes, though further investigation may be done to validate this.
32

---

Published in Transactions on Machine Learning Research (02/2024)
0
1
2
C
4
L2
b
6
b
b
b
b
b2
00
to
bg
bg
bj
b4
bj
b2
bg
bj
bg
2
b6
b
bi
b2
2
bg
b
b
b²
bb
bg
bj
bj
bf
1
1
b=
b
b2
bg
b
b
b
bj
2
b-1
b=
64
by
b -
by
b9
0
b
b2
b2
bg
b
t2
tz
b_
b
b
b
b
b"_
bg"
bg
bg
2
bー
b-
b
b
b
bo
ba²
ba"
bg
(a) Attention in PINT. Modified to allow particles to attend to each other. causal mask applied to attend past and current particles.
(b) Relative positional bias matrix in PINT. b/
by + by, where denotes the relative time and j the relative particle position.
Figure 22: Particle Interaction Transformer (PINT): attention and positional bias.
b
X
bü
b2
Do
69
6
Figure 23: Mustration the relative positional encoding. Particles in the same time-step t get the same temporal relative embedding bn, while across time, particle get unique relative positional embedding w.r.t the other particles b². Together, each particle is assigned space-time relative positional encoding bf.
Closed-form formula for Beta distribution: recall that the transparency attributes modeled Beta distributed random variables, which requires us to calculate the KL-divergence between the posterior and prior parameters. Fortunately, similar to the Gaussian distribution, the KL-divergence for the Beta distribution has a closed-form formula. Let X, ~ Beta(a, and X₂ ~ Beta(a', B'), then:
DxI(Xx||X,)) = in (B(a'b)) (a a + (b (b) + (a' - a + b- b')ol( b), B(a,b)
where B(x,y) = [())(() T(z+0) and I is the Gamma function, and & - In (Txx)) = [()) is the logarithmic TØ)
derivative of the Gamma function ('digamma"). In PyTorch, the closod-form formula can be implemented using:
logarithm the gamma function
33

---

Published in Transactions on Machine Learning Research (02/2024)
log gamma torch Igamma(x)
# logarithmic derivative of the gamma function psi torch forch.diganm digamma (x)
Separable Chamfer-KL: in practice, we found that for DLPv2, keypoint positions lock more accurately on the center of objects we model the anchors 2u as deterministic variables instoad of random variables, and only the offset from the anchor 2o is a random Gaussian variable, making 3p - 2n + &o a random Gaussian variable as well. Implementation-viss we optimize the deterministic Chamfer distance between the posterior anchor keypoints and the proposal prior keypoints, where for the offsee we optimize the standard Gaussian KL-divergence. In DDLP, the above is relevant only for the first < frames, as they are optimized w.r.t the constant prior parameters as described in the main text. For 2 7, we use the standard Gaussian KL-divergence between the posterior keypoints and PINT's output prior parameters.
Optimization: following is a description of finer training details; first, similarly to the original DLP, we perform warm-up stage where the background encoder and decoder are not trained to give an advantage the object modeling components. We reserve the first epoch for the warm-up, except for the Traffic dataset, which due its smaller size required warm-up epochs. After the warm-up step, reserve one epoch for noisy step where the learned alpha channels are artificially added small Gaussian noise sampled from [00, 0.12) to learn sharper masks. In addition, gradually anneal the weight of Layn from 0 to 1 at linear rate over the first 10,000 iterations for a more stabilized training of the Transformer.
Hyper-parameters: all CNNs are initialized from a small Gaussian distribution V(0,0.013) and use roplication-padding We use the Adam (Kingma & Ba 2014) optimizer (31 = 0.9, B2 0.999, c le 4) with initial learning rale of 2e - 4 and step scheduler that multiplies the learning rate by 0.95 at the end of each epoch. The constant prior distribution parameters are the same for all datasets and reported in Table 5 The complete set of the rest of the
can be found in Table 4
Dataset
Balls-Interns Interaction OBJ3D / CLEVRER
Traffic
PHYRE
Input Frames (Train/ Inference) T
20/10
10/6
10/6
15/10
Posterior KP K
12
25
25
Prior KP Proposals L
64
64
Reconstruction Loss
MSE
Perceptual
Perceptual
MSE
8xL
0.1
40
40
0.1
Prior Patch Size
16
16
Glimpse Size S
16
32
32
16
Feature Dim
3
8
8
Epochs
20
30
25
Table 4: Detailed hyperparameters used for the various experiments in the paper.
Attribute
Distribution
Parameters
Position Offsct 2o
Normal, N(u, Normal, NH,,
μ 0,
Scale 2,
μ Sigmoid
ggiiggpee size
), or
image size
Depth zd
Normal, NN,, Bela, Beta(u, b) Normal, O2,
a = 0,
Transparency Zt
a 0.1, 0.1
Appearance Features 2
P 0, =
Table 5: Prior Distribution Parameters
F Complexity Analysis
Our experiments with DDLP and G-SWM were conducted primarily on machines equipped with NVIDIA RTX 2080 11GB GPUs or NVIDIA A4000 16GB GPU. We trained SlotFormer on 4 NVIDIA A100 80GB GPUs. For the evaluated models, provide summary of several complexity parameters in Table G
We
34

---

Published in Transactions on Machine Learning Research (02/2024)
used the OBJ3D dataset at 128 X 128 resolution for evaluation setting. The inference time calculated based on Former's evaluation setting, which involves 6 conditional frames and unrolling to 50 frames. The reported values are for the models used in our paper's results, with G-SWM and SlotFormer trained using recommended hyperparameters. We report the inference time on a standard CPU and inference on an A4000 GPU. Note that DDLP requires significantly less memory compared to SlotFormer, which has twice the number of slots than our particles, even though both use transformer-based dynamics modules. Typical training of DDLP takes up 3 days on 4 NVIDIA RTX 2080 11GB GPUs for 128 x 128 resolution datasets, and up to 1.5 days similar hardware for 64 64 resolution datasets.
Model DDLP
Model Size
GPU Memory (per GPU)
Inference Time [ss] CPU: 1234 25.7, GPU: CPU: 6923 + 289.3, GPU: 222 + 12.3 CPU: 314 + 17.6, GPU: 213 + 11.7
10M 40MB)
9GB
SlotFormer K 6
4M ( 15MB)
Stage 1: 10GB, Stage 2: 22 GB (+PP16)
G-SWM
13M 50MB)
13GB
Table 6: Model size, GPU memory usage and inference time for different models. K is the number of particles/slots.
G Datasets
We provide extended details on the datasets used throughout this paper. The datasots can be roughly calegorized to (1) dense interactions, where interactions between objects occur frequently and most of the sampled sequences include interactions, and (2) sparse interactions, where interactions are less frequent occur in the distant horizon, and sampled sequences may contain no interactions at all.
Balls-Interaction a dense interaction 2D dataset introduced in Jiang et al. (019 consists of videos of candom-colored balls bouncing and colliding with each other. Each video, or episode, is 100-frame long, where each frame is of size 64 x 64. We follow Lin et al. 2020b) generation process and use 10,000 episodes for training, 200 for validation and 200 for test.
OBJ3D dense interaction 3D dataset containing CLEVR-like objects (Johnson et al. 2017) introduced in Lin et al. 20200b) In each 100-frame video of 128 x 128 resolution, randon-colored ball is rolled towards a random number of objects of different types positioned in the center of the scene, colliding with them. The dataset contains 2,920 episodes for training, 200 for validation and 200 for test.
CLEVRER a sparse interaction 3D dataset containing CLEVR-like objects introduced in Yi et al. (2019). Each video consists of objects entering the scene from various locations, potentially colliding. As new objects may enter the scene future times-teps, we trim every video to maximize the visibility of objects in the initial scene and we resize the frames to 128 128. We use the the first 5,000 videos for training, 1,000 for validation and 1,000 for lest.
PHYRE a sparse interaction 2D dataset of physical puzzles introduced in Bakhtin et al. (2019). designed for physical reasoning. We collect simulated data in the form of 128 x 128 frames from the BALL-ticr tasks in the ball-wiihhin-tenpplaee setting, where lasks are considered solved if the user-placed ball meets a certain condition, e.g., touch the wall/floor/ /anoorr object. We generate rollouts from all tasks, except for tasks [12, 13, 16, 20,21] which include a lot of distractions. We use 2,574 episodes for training, 312 for validation and 400 for test.
Traffic real-world dataset introduced in Daniel & Tamar (022a) containing videos of cars of different sizes and shapes driving along opposing lanes, captured by traffic camera. We separate the video to episodes of 50 frames, each frame of size 128 x 128. The dataset contains 133,000 frames, where we take 80% for training, 10% for validation and 10% for test.
Extended Comparison with SlotFormer
In this soction, we present an extended comparison between our proposed DDLP model and Slot Former, a recent slot-based video prediction model that achieves state-of-the-art results (Wu (002).. To conduct our experiments, we used the available official code and pre-trained models (Wu et al.] 2023). It is important.
35

---

Published in Transactions on Machine Learning Research (02/2024)
to note that the results reported in [Wu et al. 2022 are the validation set the same dataset that used for tuning hyperparameters, and are for the 64 64 resolution version of OBJ3D. To ensure consistency with our paper, we used their pre-trained model to evaluate the reported metrics on the validation set and also computed the same metrics on a test set (which was not used for training nor hyperparameter tuning). Our paper only reports results for the test sels. For a visual comparison between DDLP and Slot Former, please refer to the videos available our webpage:
hww.seatteehggiihhu.ee git.ub.thub.... 10 wot.dp/pdwbwbbbb
compared DDLP and SlotFormer the OBJ3D dataset in two experiments. First, trained 64 x 64 version of DDLP to compare with SlotFormer's pre-trained model in their evaluation setting. Second, we trained a 128 x 128 version of SlotFormer and compared it to our model. We evaluated both experiments in SlotFormer evaluation setting, which involves 6 conditional frames and a rollout horizon of 50 for a fair comparison. Based on the LPIPS metric, which is better aligned with human perception (Wul ltt al., 2022), our model outperforms FlotFormer, as shown Table Figure 24 and the video rollouts on the webpage demonstrate that SlotFormer struggles when objects rotate, resulting in object deformation.
OBJ3D comparison:
0BJ30 64 x 64
08J33 128 x 128
PSNR
SSIM 7
LPIPS $ 0.08 0.06
PSNR 1
SSIM
LPIPS
Former
32.16 + 5.43
0.90 0.07
31.2 4.9
0.92 + 0.04
0.135 + 0.05
DDLP (Ours)
32.01 # 5.42
0.90 + 0.07
0.07 + 0.06
30.1 + 4.86
0.91 - 0.04
0.088 = 0.06
Table 7: DDLP vs. SlotFormer video prediction results on the OBJ3D dataset. For 64 x 64 resolution, we evaluate the publicly available pre-trained model. Results reported the test-set.
GT
DDLP
SFormer
Figure 24: DDLP SlotFormer video prediction comparison the 0BJ3D dataset.
Traffic comparison: for the Traffic dataset, as it includes many moving cars, we assumed that a large number of slots is required. SlotFormer used a maximum number of slots in their experiments. We tried training it with 10 slots, but faced GPU memory issues on the same hardware required to train a 128 x 128 Traffic model; however, we were able to train Foomer with slots on stronger cloud hardware. We attempted several runs until we got it to work, see our note regarding instability below. Since the Traffic dataset has scenes with more than 10 cars, underlying assumption is that each slot can model more than car. This assumption is reasonable, same direction per lane, and the latent dimension of cach slot is high (128), much larger than the dimension of each particle in DDLP. As shown in Figure 10 SlotFormer's provide more visual results SlotFormer on the Traffic dataset for 64 x 64 and 128 128 resolutions on the webpage. Our visualization of per-slot rollouts, available on the webpage due to their visual size, indoed validates that slots sometimes model more than single object, affecting the predictions when rolling out the dynamics model. Clearly, DDLP outperforms Slot Former in this domain.
the dynamics of the dataset are not complex, with moving in the
morizoo predictions become blurry. We
36

---

Published in Transactions on Machine Learning Research (02/2024)
To further explain these results, recent studies into slot-based models attempt to shed light on why they struggle with real-world datasets. As Seizerr et al. (2023) concluded, image reconstruction is not enough for slot-based decomposition, and feature-wise loss from a pre-trained self-supervised method can mitigate the limitations of slot attention. Specifically, [Seitzer Seettte ct al. [0033 shows that the slot-based models STEVE (Singh et al. 2022b) and SAVi++ {Elsayed et al.) 2022). which are used in Flomrr are mostly limited to synthetic datascts. Replacing these methods with the one proposed in Seitzer et al. (2023) might improve SlotFormer, although it would increase the training complexity.
Furthermore, while SlotFormer evaluated the CLEVRER and PHYRE datasets, we found it challenging to compare DDLP with SlotFormer as Wu et al. (2022) reported special pre-processing steps roquired for Flom to work on these datasets. For PHYRE, the base slot-attention was reported to fail to detect objects in light colors, resulting in poor segmentation results that were addressed by manually adjusting specific pixel values. While this was not necessary for DDLP to work on this dataset, we used a color-inverted version of PHYRE for visual clarity, without any manual pixel adjustments. For CLEVRER, Wu al. (2022) trained 64 64 resolution images, subsampled each episode by factor of 2. In contrast, we trained DDLP on 128 x 128 resolution images without subsampling. Note that both DDLP and SlotFormer cannot handles new objects that appear in the scene. To account for this, Wu et al. 20022 manually filtered out vidco clips where new objects appeared during the rollout period. We did not manually filter clips. To handle object appearance, we noticed that trimming a fixed number of frames from the beginning and end of the clips yielded clips with much less objects that enter the scene.
We encountered several challenges when training SlotFormer. Firstly, the hardware requirements for Flom were higher than those for DDLP. We found that the machines used to train DDLP (4xRTX2080) were not sufficient to train Flomrr which is in line with the reported requirements in the official repository (Wul ct al., 2023). Additionally, as SlotFormer is trained in two stages, each with different memory requirements, we had to resort to using cloud-compute (4 NVIDIA A100 80GB GPUs) to run the experiments, making it less accessible than our model. For a complexity analysis, please see Appendix F and for a comparison of complexity between DDLP and SlotFormer, please see Table G Secondly, we encountered instability during the first stage of SlotFormer's training, which involves training the base SAVi++ slot-attention model and required several attempts to make it work the Traffic dataset. This aligns with the official code repository which recommends rumning 3 different runs with the same syper-parameters and picking the best one. In our experience, DDLP was more stable to train and search for working hyper-parameters.
Additional Results
This section presents additional visual results from the datasets we experimented on. The complete evaluation metrics for vidco prodiction on all datasets are reported in Table g
1.1 DLPv2 Unsupervised Object-centric Scene Decomposition
Here we detail our improved DLPv2 scene decomposition capabilities. In Figure g we demonstrate how a single image is decomposed to particles by plotting the keypoints, objects' bounding boxes with confidence- based scores, objects' learned masks and the learned background. The bounding-ber box scores are calculated based on the the per-particle uncertainty, similarly to the original DLP:
V(ux) = Σ log(o?)))
where ok, is the standard deviation in the 8th axis (i.c., the x and y coordinates) of ug. In addition, when integrating the tracking posterior, we are able to track the particles through time, demonstrated in Figure 25
1.1.1 Comparison of DLPv2 vs. DLPv1
In this section, we quantitatively and qualitatively compare our proposed method, DLPv2, with the original DLP using the publicly available implementation (Daniel & Tamar 2022bb. We train both models in the
37

---

Published in Transactions on Machine Learning Research (02/2024)
[=]
L=10
1=20
(=40
GT
25.74
37
i
BB
24.
25.78
23.61
9.24
OB
BG
Figure 25: Tracking particles through time. We plot the learned keypoints, bounding boxes and objects. Notice how the order of the particles is preserved over time.
single-image setting on the Traffic and OBJ3D datasets, using the same number of particles and recommended syper-parameters. We stop training once the validation LPIPS metric stops improving. As shown in Table R our proposed DLPv2 significantly outperforms the original DLP, indicating that our introduced modifications capture objects more accurately, resulting in improved image reconstructions. Qualitative image decomposition
38

---

Published in Transactions on Machine Learning Research (02/2024)
comparisons between the two models are presented in Figures 26 and 27 Notably, DLPv2 generates more accurate particle positions, which are located at the center of objects, resulting in more disentangled representation. Furthermore, it should be noted that DLPv1 does not model object attributes and, as a result, is incapable generating bounding boxes.
PSAR
SSIM
LPIPS 1
PSNR
SSIM
LPIPS
DLP DLPv2 (Ours)
39.23 + 3.33
0.982 = 0.009
0.085 + 0.018
24.21 I 0.9
0.67
0.05
0.13 = 0.02
Table 8: DLP vs. DLPv2 image reconstruction comparison in the single-image setting. Results are reported on the test-set.
CB
O.Pv2
OLP
Figure 26: DLP vs. DLPv2 single-image decomposition on the Traffic dataset. It be seen that DLPv2 produces more accurate particle positions that located in the centers of the cars.
CBI
O.P+2
DLP
Figure 27: DLP vs. DLPv2 imgle-image decomposition on the OBJ3D dataset. It can be seen that DLPv2 produces more accurate particle positions that located in the center of the objects, leading to better sagground-backgronnn segmentation.
1.2 Video Prediction: Balls-Interaction
Figures 28 and 29 present rollouts from the Balls-Inee Innle-Inttrattoon dataset, which contains a lot of collisions. The models are conditioned on the first frames and predict frames into the future. Note how the accumulated error of the balls position of G-SWM is larger than DDLP's predictions also demonstrated quantitatively in Table 10
39

---

Published in Transactions on Machine Learning Research (02/2024)
Dataset
08J30
Traffic
PSNR 1
SSIM T
LPIPS |
PSNR 1
SSIM
LPIPS
G-SWM
31.716.2
0.92410.05
24.8841.077
0.235-10.04
SlotFormer
31.2+4.91
0.925=0.04
0.135=0.00
24.93+0.86
0.60+4004
0.18+0.03
DDLP (Ours)
31.29=5.22
0.923=0.01
0.08840.06
21/111022
0.61++0.06
0.15#0.02
Dafaset
CLEVRER
G-SWM
30.12-6.69
0.93=0.01
0.14620.00
21.6146.22
0,93+0003
0.07840.06
DDLP (Ours)
29.21=6.16
0.92=0.04
0.134=0.08
26.9845.3
0.9540.04
0.053=0.04
Table 9: Quantitative results video prediction.
Balls-Interaction MED10
PSNR
SSIM f
LPIPS $
G-SWM
0.256
13.86
0.753+0.12
0.232:0.15
DDLP (Ours)
0.149
16.74 16.741883
0.78910.12
0.18910.15
Table 10: Balls-Interaction dataset comparison. MEDIO is the mean euclidean distance error of predicted balls positions, summed over the first 10 steps of generation.
DDLP
G-SWM
Figure 28: Rollout from the Balls-Interaction dataset.
DDLP
6-SWM
Figure 29: Rollout from the Interatittootion dataset.
40

---

Published in Transactions on Machine Learning Research (02/2024)
1.3
Video Prediction: Traffic
Figures 30 and 31 present more rollouts from G-SWM and DDLP trained on the Traffic dataset. The models conditioned the first. 10 frames and predict 40 frames into the future. Note how cars in G'SWM's rollouts get blurry or vanish in the long term.
t-10
GT
DDLP
Figure 30: Rollout from the Traffic dataset.
DDLP
G-SWM
Figure 31: Rollout from the Traffic dataset.
1.4 Video Prediction: OBJ3D
Figures and 3R are conditioned on the first 10 frames and predict 90 frames into the future. Note how objects in G-SWM's rollouts get blurry in the long term.
present more rollouts from G-SWM and DDLP trained on the 0BJ3D datasct. The models
41

---

Published in Transactions on Machine Learning Research (02/2024)
DDLP
G-SWM
Figure 32: Rollout from the DBJ3D dataset.
Figure 33: Rollout from the DBJ3D dataset.
1.5
Video Prediction: PHYRE
Figures
35 and 36 present rollouts from G-SWM and DDLP trained on the PHYRE dataset. The models
are conditioned on the first 10 frames and predict frames into the future. Note how objects in G-SWM's rollouts get blurry or vanish in the long term.
1.6 Video Prediction: CLEVRER
Figures 37 and 38 present rollouts from G-SWM and DDLP trained on the CLEVRER dataset. The models are conditioned on the first 10 frames and predict 90 frames into the future.
1.7
What If...? Video Prediction
In Figures
39
40
and
visualize examples of latent particle modification of videos from OBJ3D.
42

---

Published in Transactions on Machine Learning Research (02/2024)
DDLP
G-SWM
Figure 34: Rollout from the PHYRE dataset.
:
:
:
:
Figure 35: Rollout from the PHYRE dataset.
1.8 Ablation Study
We evaluate the contribution the design choices of DLPv2 and DDLP. In Table II we report the performance the Balls-Interaction datasct by removing the corresponding component out of: using the standard learned positional embeddings in PINT instead of the relational positional encoding and using the standard DLP encoder for all frames instead of tracking. Observe that the full model achieves the best performance, and that tracking the particles is crucial for accurate dynamics modeling We perform further ablation of changing the filtering heuristic of the prior proposals to the distance-heuristie used in the original DLP, instead of filtering based on the SSM covariance.
J Failure Cases
Our method is not without limitations sometimes objects are missed, get deformed vanish, as we demonstrate in Figures 42 and 43 However, these phenomena less frequent than in other methods,
43

---

Published in Transactions on Machine Learning Research (02/2024)
:
DDLP
G-SWM
Figure 36: Rollout from the PHYRE dataset.
t=10
t-15
2-
t-
t-40
DDLP
6-SWM
Figure 37: Rollout from the CLEVRER dataset.
MEDI0
PSNR
LPIPS
Full
0.149
16.74
0.789
0.189
.oo RPE
0.157
16.73
0.787
0.194
-No Tracking
2,736
11.68
0.393
-No NCC 0.168
16.52
0.783
0.196
Distanco-filtering
Table 11: Ablation study the Balle-Interaction dataset. In each row, remove component from the full model. No RPE use standard positional embedding instead of relative positional encoding. No Tracking use the standard single-image DI.P encoding for all frames instead of tracking. Distance-filtering using the cettance-heuristie instead the SSM covariance to filter keypoint proposals. No NCC- no normalized cross-corrclation in the tracking process.
evident by the quantitative results. We believe that scaling-up the model size, e.g., larger Transformer, may help alleviate these, and perhaps additional tuning of the model hyper-parameters.
44

---

Published in Transactions on Machine Learning Research (02/2024)
DDLP
G-SWM
Figure 38: Rollout from the CLEVRER dataset.
45

---

Published in Transactions on Machine Learning Research (02/2024)
t=10
t=15
t=20
t=30
t=40
Ground Truth
Prediction
"whot If the yellow ball vos lorger? ?
"Whot the yellow ball started at the left?"
"Whot If the greon ball started at the !oft?"
Figure 39: DDLP video prediction on OBJ3D. The first row shows the ground-truth frame sequence with the predicted posterior keypoint trajectory (blue). In the second row, we present DDLP's generated particles' trajectory (blue) when conditioned on the first 4 frames, and the decoder's reconstructed sequence from the particles. Row 3-5 present modifications performed in the latent particle space (red arrow) according wwhat if...?' questions, and the resulting rollout by the dynamics module. Note the different object dynamics resulted from each modification.
46

---

Published in Transactions on Machine Learning Research (02/2024)
t=20
t=40
Ground Truth
DDLP Prediction
"Whot ball started the left?"
"whot there was yellou ball?"
Figure 40: DDLP vidco prediction on OBJ3D. The first row shows the ground-truth frame sequence with the predicted posterior keypoint trajectory (blue). In the second row, we present DDLP's generated particles' trajectory (blue) when conditioned on the first 4 frames, and the decoder's reconstructed sequence from the particles. Row 3-4 present modifications performed in the latent particle space (red arrow) according to wwhat if...?' questions, and the resulting rollout by the dynamics module. Note the different object dynamics resulted from each modification.
47

---

Published in Transactions on Machine Learning Research (02/2024)
t=15
t=20
t=30
t=40
Ground Truth
DOLP Prediction
"Whot If yellow ball pink cube?"
"Uhat If yellou boll started little to the left?"
Figure 41: DDLP video prediction on OBJ3D. The first row shows the ground-truth frame sequence with the predicted posterior keypoint trajectory (blue). In the second row, we present DDLP's generated particles' trajectory (blue) when conditioned on the first 4 frames, and the decoder's reconstructed sequence from the particles. Row 3-4 present modifications performed in the latent particle space (red arrow) according wwhat if...?' questions, and the resulting rollout by the dynamics module. Note the different object dynamics resulted from each modification.
48

---

Published in Transactions on Machine Learning Research (02/2024)
t-10
t-
t=20
1-30
tm40
DDLP
G-SWM
Figure 42: Failure case on CLEVRER. DDLP misses an object, leading to wrong dynamics modeling.
t-20
30
tw
DDLP
Figure 43: Failure case on PHYRE. DDLP deforms the objects, leading to wrong dynamics modeling.
49


Reviews:
[{"review_id": "OP4stYPdAY", "reviewer": "Reviewer_uzkC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. Lightweight video prediction that can work with many objects\n2. Can perform ‘what-if’ generation by playing around with the latent representations which are interpretable (wrt position, shape, depth etc)  \n3. Can also perform video generation with a diffusion model learned on the latent particles.\n\nWeakness:\n1. Ablation experiments -   \n   a. DLPv2 is a major contribution for improved performance, however we do not see the comparison of method with DLPv2 in main \n       results in table 2. It should work as a strong baseline and should be present in the main table.  \n   b. Ablation with important components in a standard way. I feel the 3 major components are DLPv2, particle tracking in posterior \n       (encoder), particle interaction transformer as presented in the paper. The ablations should have results with these three explicitly. I \n       see in table 2 ablation there is -DLP (No Tracking) which I assume should be DLPv2 ? Apart from that other components ablation is \n       really important for the main paper.  \n\n2. Main table results - should also include PSNR, SSIM scores for proper evaluation. Although these numbers are in appendix, it also makes sense to put them in the main table. \n\n3. Comparison with SlotFormer on CLEVRER, PHYRE and Balls-Interaction dataset. Also I may misunderstand something here, but from the published slotformer work, the LPIPS score for obj3d dataset is 0.080 while the reported number in the manuscript is 0.13+-0.05. They also present numbers for CLEVRER dataset and achieve 0.11 LPIPS score, and the proposed work DDLP achieves 0.13+-8.  \n\n4. To show error in prediction, authors presented MED10 metric which computes mean euclidean distance error of predicted balls positions for first 10 steps of generation. Would also be nice to see scores and comparisons for more steps.  \n\n5. Discussion on why object centric representation based method is needed. There are a range of work for video prediction without object centric representation learning e.g. RIVER[1], VDM[2] etc. Authors should also motive the need for object centric representation learning for video prediction. \n\n6. Ablation with positional embeddings - \nThe choice of choosing positional bias as $b_i + b_j$, why not $b_i \\times b_j$, or some other formulation ? Comparison with just using $b_i$ or just using $b_j$, to understand importance of spatial and temporal positional embeddings separately. \n\n[1] Efficient Video Prediction via Sparsely Conditioned Flow Matching, ICCV 23  \n[2] Video diffusion models. arXiv preprint arXiv:2204.03458, 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work proposes a new framework for object-centric video prediction based on Deep Latent Particles (DLP). The contributions include improvement of DLP for single images by including additional attributes such as scale, depth and transparency. It proposes inclusion of particle tracking posterior to efficiently track particles among different frames. Further it also proposes Particle interaction Transformer which enriches the prior distribution of particles with temporal information. Overall, the work improves over prior works based on LPIPS metric score.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Suggested changes in writing:  \n    a. Introduction lacks motivation of why DDLP is required.  \n    b. I feel DLPv2 details should be in the main paper and not in appendix as it is also the contribution. (D1-4)  \n    c. Also the optimisation strategy should be in the main paper: section E, Optimisation objective (page 31). This section helps the reader \n        to understand the final optimization loss for reconstruction.  \n    d. Figure 8, caption: GSWN tends to “deform object” over time, maybe a more elaborative caption would help. I can observe a slightly \n         poor reconstruction with red circle, but a proper caption with more details as what is deformed here e.g. poor edges etc would help.  \n    e. Authors mention SAVi++ is limited to synthetic datasets [page 36], which seems to be incorrect as SAVi++ works with real world \n        scenes on Waymo Open dataset. \n\n2. Experimental changes:  \n   a. Figure 7 shows only decomposition of DLPv2 but no comparison with DLP. Would be nice to see the qualitative comparison with \n       DLPv2.", "broader_impact_concerns": "All concerns are already present in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SNmHXJ2L7k", "reviewer": "Reviewer_YXT5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1, The overall writing of this manuscript is clear and easy to follow.\n\n2, Explicitly modeling the attributes of the particles is a good idea for video prediction, as it provide us more interpretable intermediate representation.\n\n3, The proposed DDLP achieves better at lone-term consistency generation, which aligns well with the intuition behind the design.\n\n4, Extending the latent particles to unconditional generation with diffusion model is interesting and may find more audience in the community.\n\nWeaknesses\n\n1, I would suggest making the main part of this manuscript more self-contained instead of frequently referring to the supplemental materials. For example, on page 7, it is said that “The relative positional bias is illustrated in Figure 22.”. One needs to jump almost to the end of the manuscript to find the illustration and then jump back. It would be better to just have a simple illustration in the main body and then refer the details to the supplemental materials if readers are interested in the details.\n\n2, Some parts of the technological design are unclear. I found it a little hard to fully understand the details. Please refer to the “Requested Changes” part.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this manuscript, the authors study the problem of Object-centric video prediction. The core contributions of this work are threefold.\n\nFirst, propose DLPv2, which is an improved version of DLP, enhanced with more explicit attributes of the particles like scale, depth, and transparency. These attributes can be used to control the generation process in a more granular manner. \n\nSecond, they re-design the architecture of the transformer, especially the attention matrix with modified position encoding, to make the overall framework more suitable to model the interaction among time and across particles.\n\nThird, they introduce tracking into the framework to align the keypoints across different frames to make the particles more stable, leading to more consistent predictions.\n\nThe experiments indeed show the advances of the proposed DDLP, as well as its new ability to generate “what if …” prediction.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some points that are unclear.\n\n1, On page 6, it is said that “(1) the posterior keypoints from the previous step serve as anchors, from which an offset will be learned to the current step as illustrated in Figure 2”. However, in Fig. 2 there is no illustration of the offset.\n\n2, In the \"what if\" experiments, we can indeed change the latent position code to force the ball to the left-right. But I’m not sure how to deal with the Burn-in frames, as we still need some real images of where the ball is located on the left side (correct me if is there anything I misunderstanding).\n\n3, For the explicit modeling of attributes of the particles, do we need the ground truth attributions? It is unclear to me how to train these attributes to bind their behaviors to the decoder. That is to say: how do we ensure that changing the attributes of the particles can lead to a corresponding change of the generated image?", "broader_impact_concerns": "As the core contributions are the design of the new framework, I do not see Boarder Impact Concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JdAinfuLl2", "reviewer": "Reviewer_8tEt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ As a extended paper for its conference version, the authors have clearly introduced the improvement brought by this paper and the motivation behind, as well as the perquisite knowledge to understand the topic.\n+ The autoregressive transformer in Sec.4.4 is well designed, and spatial-temporal discriminativeness is considered and handled as spatial-temporal positional bias.\n+ The efficiency and the performance of the proposed method are relatively better significantly than existing works.\n+ The appendix compensate some finer details such as the model structures and the model training details. \n\nWeakness:\n- The overall organization of the paper could be improved. Some details are in the appendix instead of the main paper, e.g. the detailed network structure and the detailed training losses. Instead, I would suggest to move some of the content in the main paper to the appendix, to ensure that the reader can fully understand \"what\" is the model and \"how\" it works, \"why\" some components are designed as it is can be optional in appendix. For example, the authors can introduce the added attributes more briefly.\n- The spatial-temporal bias design looks inconsistent to it motivation to me. The authors using \"+\" to fuse the temporal and spatial (order) biases, which is not a one-to-one mapping, i.e. the same added bias may represent to different temporal and order index. Instead of adding, multiplication makes more sense to me since it ensure a one-to-one mapping. \n- More realistic data such as the BAIR Robotic Pushing can be added to verify the method in real world.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper extends the Deep Latent Particle (DLP) to Dynamic DLP for future video frame prediction. To this end, the authors introduced several improvements including: a) adding scale, depth, and transparency attributes to the DLP attribute set, b) associating particles between frames using correlation, c) an autoregressive transformer predictor that predicts multiple particles in a single inference, and d) an unconditional particle attributes diffusion process to generate a scene with realistic physical interactions. To demonstrate the effectiveness of the proposed method, the authors conducted experiments on five different datasets, and on tasks of video prediction and unconditional video generation, both of which have achieved superior performance over previous methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In general, I do not see significant weakness on the method itself except for the question about the temporal-spatial bias fuse.\nThe authors can focus on improving the organization of the paper, e.g.\n- Reprioritize the content in the appendix and the main paper.\n- Make the method sections more clear with better illustration, currently the figures are confusing while the text are less well organized, e.g. the Figure3 is confusing to add MLPs right to the other attributes, which can confuse readers about whether the visual features encoding need to take other attributes as input (I assume not).", "broader_impact_concerns": "The Broader Impact has been sufficiently discussed in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper was reviewed by three reviewers. After the rebuttal period, all three reviewers recommended "leaning accept''. 

One remaining concern is that the main text should be made more self-contained by moving some contents from the supplementary materials to the main text. 

The reviewers were positive about the proposed method in general, including the design of the model architecture, the what-if generation, and the diffusion-based generation. There were initially some concerns about experiments, including ablation studies and additional results, but these concerns were all successfully addressed in the rebuttal period. 

Overall, I think this paper makes a useful contribution to video prediction, and I recommend accept with minor revision.
Decision: Accept (poster)