Learning 3D Particle-based Simulators from RGB-D Videos
Authors: William F Whitney
Tatiana Lopez-Guevara
Tobias Pfaff
Yulia Rubanova
Thomas Kipf
Kim Stachenfeld
Kelsey R Allen
Venue: ICLR 2024 poster (2024)
Keywords: simulation
dynamics
nerf
particle dynamics
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Learned dynamics models that combine 3D representations and ray-based rendering.

Abstract: Realistic simulation is critical for applications ranging from robotics to animation. Traditional analytic simulators sometimes struggle to capture sufficiently realistic simulation which can lead to problems including the well known "sim-to-real" gap in robotics. Learned simulators have emerged as an alternative for better capturing real-world physical dynamics, but require access to privileged ground truth physics information such as precise object geometry or particle tracks. Here we propose a method for learning simulators directly from observations. Visual Particle Dynamics (VPD) jointly learns a latent particle-based representation of 3D scenes, a neural simulator of the latent particle dynamics, and a renderer that can produce images of the scene from arbitrary views. VPD learns end to end from posed RGB-D videos and does not require access to privileged information. Unlike existing 2D video prediction models, we show that VPD's 3D structure enables scene editing and long-term predictions. These results pave the way for downstream applications ranging from video editing to robotic planning.

Published as a conference paper at ICLR 2024
LEARNING 3D PARTICLE-BASED SIMULATORS FROM RGB-D VIDEOS
William F. Whitney; Tatiana LopezaGuevara; Tobias Pfaff, Yulia Rubanova, Thomas Kipf, Kimberly Stachenfeld, Kelsey R. Allen Google DeepMind
ABSTRACT
Realistic simulation is critical for applications ranging from robotics animation. Traditional analytic simulators sometimes struggle to capture sufficiently realistic simulation which can lead problems including the well known "sim-to-real" gap in robotics. Learned simulators have emerged as an alternative for better capturing real-world physical dynamies, but require access to privileged ground truth physics information such as precise object geometry or particle tracks. Here we propose a method for learning simulators directly from observations. Visual Particle Dynamics (VPD) jointly learns latent particle-basel representation of 3D scenes, a neural simulator of the latent particle dynamics, and a renderer that can produce images of the scene from arbitrary views. VPD learns end to end from posed RGB-D videos and does not require access to privileged information. Unlike existing 2D video prediction models, we show that VPD's 3D structure enables scene editing and long-term predictions. These results pave the way for downstream applications ranging from video editing robotic planning.
INTRODUCTION
Physical simulation underpins a diverse set of fields including robotics, mechanical engineering, game development, and animation. In each of these fields realistic simulation is critical and substantial effort goes into developing simulation engines that are specialized for the unique requirements and types of physical dynamics of that field (Todorov al., 2012; OpenCFD, 2021; Coumans & Bai, 2016). Obtaining sufficiently realistic simulators for particular scene requires additional work (Shao et al., 2021), including the tuning of simulation assets like object models and textures, as well as the physical properties that will lead to realistic dynamics (Qiao et al., 2022). Even with significant effort and tuning. it is often impossible to perfectly capture the physics of any particular real-world scene, as sub-scale variations in surfaces and textures can have significant impacts on how objects behave.
Over the last few years, learned simulators have emerged as an alternative to carefully hand-crafted analytic simulators. They can be trained to correct the outputs of analytic solvers (Kloss et al., 2022; Ajay et al., 2018), or trained to mimic analytic physics directly, but at significantly faster speeds (Pfaff et al., 2021; Kochkov et al., 2021). Learned simulators can capture many different dynamics, ranging from liquids and soft materials to articulated and rigid body dynamics (Li et al., 2019; Allen al., 2023). When researchers can provide state-based information for real-world dynamic scene (the positions and poses of objects over time), learned simulators can mimic real-world physics better than careful tuning with analytic simulators (Allen al., 2022).
However, learned simulators still require access to privileged "ground truth" physics information to be trained. The vast majority require near-perfect state estimation the poses, exact 3D shapes, and positions- - of all objects. at all time points, in a scene. Recent attempts to relax these requirements still require access other forms of supervised information such as object segmentation masks (Driess et al., 2022; Riochet et al., 2020; Shi et al., 2022).
While there exist many video prediction models that do not require such privileged information (Finn et al., 2016; Clark et al., 2019; Ho et al., 2022), these models are not simulators. Unlike simulators, 2D video models do not operate in 3D, do not generally support rendering a dynamic scene from a
"Equal contribution. Correspondence wwwhitney zepol ttatttt |googl com.

---

Published as a conference paper at ICLR 2024
Visual Particle Dynamics
Encoder
Dynamics
Renderer
World
Ingas Ioages
Excadel Paists
Prodicted Paints
Randraa Iragge
Figure 1: diagram of our model. The encoder maps multiple RGB-D cameras into a latent point cloud representation using convolutional encoder. The dynamics model, hierarchical graph neural network, learns to transform this point cloud to new point cloud on the next time-step. The 3D latent point cloud is decoded into image, supporting an image-based loss that be used train the entire pipeline end-to-end.
new viewpoint, and do not allow 3D editing of scenes to produce new simulation outcomes. Recent works such as NeRF-dy Li al. (2021a) which represent scenes as single latent vectors can support rendering from new viewpoints, but cannot support 3D editing. Similarly, recent advances in 3D neural representations for dynamic scenes (Li et al., 2023; Du et al., 2021c), can provide beautiful reconstructions of single videos without access to privileged physics information, but these models are not simulators either. Once trained, they represent a recording of a specific scene, and cannot be applied to new scene without re-training.
Here we ask whether we can learn simulators from multi-view RGB-D observations alone. Our model, Visual Particle Dynamics (VPD), jointly learns latent patticl-bbsse 3D scene representation, a predictive model represented as hicrarchical graph neural network on that representation, and a conditional renderer that can produce new images from arbitrary views. It is trained end end multi-view RGB-D data without object masks or 2D segmentation models, and has the following desirable properties which we demonstrate:
VPD supports 3D state editing. explicit 3D representation can be edited, simulated, and re-rendered from novel views.
VPD supports multi-material simulation. We demonstrate results for multi-body rigid dynamics and soft-body interactions.
VPD outperforms 2D video models in data efficiency. With as few as 16 trajectories, VPD can learn simulator for simple dynamic scene.
To our knowledge, Visual Particle Dynamics is the first fully learned simulator that supports these crucical properties, and which does not require access to any privileged supervised information for training inference.
RELATED WORK
Given the importance of realistic simulation across a wide variety subject areas (robotics, graphics, engineering), there are a variety of techniques for learning predictive models with different capabilities, each with different assumptions about input data requirements. We outline the general classes of learned predictive models below, and summarize the capabilities of key related works in Table 1.
Learned simulators Learned simulators aim at replacing analytic simulators with a learned function approximator. This means they are generally trained on state information from a ground truth solver, and often cannot be used directly visual input data. Depending the application, state can be represented as point clouds (Li et al., 2019; Sanchez-Gonzalez et al., 2020; Mrowca et al., 2018), meshes (Pfaff et al., 2021; Allen et al., 2023), or as SDFs (Le Cleac'h 'h et al., 2023). Learned function approximators such as graph neural networks (GNNs) (Battaglia et al., 2018), continuous convolutional kerness (Ummenhofer et al.. 2019), MLPs (Li et al., 202 la), then be used to model the evolution of the state through time. Our work goes beyond these approaches by never requiring access the states directly, instead learning from RGB-D videos.
2

---

Published as a conference paper at ICLR 2024
Method
Supervision
Sensors
Capabilities
No states
No masks
Type
Editable Rendering 3D Prediction
NeRF-dy (Li et al., 2021a)
/
multi RGB
x
3D-IntPhys (Xue et al., 2024)
X
X
multi RGB
/
/
RoboCraft (Shi et al., 2022)
/
X
multi RGB-D RGB
X
/
SlotFormer (Wu et al., 2023)
/
/
X
Driess et al. (2022)
x
multi RGB
Dynibar (Li et al., 2023)
V
multi RGB multi RGB-D
Ours
Table I: Comparison of input data requirements and capabilities of prior methods. VPD learns full dynamics prediction model with 3D bias, editability, and free-camera rendering, without requiring state information or segmentation masks, but does require multiple RGB-D sensors.
Bridging perception and simulators Several approaches rely on these learned simulators as a dynamics backbone, but attempt learn perceptual front-ends that provide point clouds mesh information from video. Some attempts rely on pre-trained dynamics models which were trained with ground truth state information (Guan et al., 2022: Wu al., 2017), and learn mapping from perception to 3D state. Learning this mapping for a general setting is a very hard task, and hence most methods only work for simple scenes, and/or require object segmentation masks seperate the scene into components (Janner et al., 2019; Driess et al., 2022; Shi et al., 2022: Xue et al., 2024). In contrast, VPD removes the requirement of either having pre-trained simulator or object segmentation masks and trains end to end with pixel supervision. NeRF-dy (Li et al., 2021a) reverses this formula, first learning vector representation of scenes, then learning dynamies model on this representation. We describe NeRF-dy in Section 4.1 and compare to its choices in Section 5,
Analytic simulators and system identification Another strategy for constructing a simulation of real-world scene involves modeling the scene in analytic simulator, then fitting the physics coefficients of this simulator to match the real data as well as possible. Differentiable simulators (Hu al., 2020b; Freeman al., 2021; Du et al., 2021a; Howell et al., 2022: Macklin, 2022) enable the use of gradients in system identification; however, this typically requires ground-truth object positions. Recent work (Qiao al., 2022; Le Cleac'h h al., 2023) combines implicit 3D object representations with differentiable simulator to support capturing object models from the real world. Methods for inferring articulated models from video (Heiden et al., 2022) similarly depend on pre-trained instance segmentation models and have only been applied to simple settings. Overall these methods can give good results, but are limited by the types of physics representable in the analytic simulator and the complexities of state estimation.
Unsupervised video models Video models do not require object segmentation masks or pre-trained dynamics models. Some methods directly model dynamics in pixel-space (Finn et al., 2016; Clark et al., 2019; Ho et al., 2022), but often fail to model dynamics reliably. Other models attempt to discover object representations (Du et al., 2021b), more recently, simultaneously learn object representations and dynamics (Watters et al., 2017: Qi et al., 2020; Wu et al., 2023). Given the breadth of existing video data, large-scale learning with transformers and diffusion models has shown promise for enabling morihhoizzon planning in robotics (Hafner et al., 2023; Du et al., 2023). However, these learned models are not simulators. Those that learn dynamics do not generally operate 3D and they do not support compositional editing. VPD possesses both of these qualities while still learning from fully unsupervised data.
Video radiance fields A class of methods extend NeRF (Mildenhall et al., 2020) with temporal axis, to encode individual videos in neural network weights (Li et al., 2021b; Du et al., 2021c; Li et al., 2023). Unlike the methods above however, these approaches encode a single video trajectory and do not support predicting videos, or generalizing across different videos. In contrast, VPD supports both.
3 VISUAL PARTICLE DYNAMICS
Visual Particle Dynamics (VPD) model consists of three components: (1) an encoder which maps set of one more RGB-I images into 3D latent particle representation; (2) dynamics model
3

---

Published as a conference paper at ICLR 2024
(a) Encode
(b) Process
(c) Decode
Figure 2: Hierarchical message passing. (a) The particle representing the past timestep 1 (blue), and particles representing the present state & (green) send their features and relative positions to the abstract nodes (red). (b) The abstract nodes perform multi-step message passing amongst themselves. (c) The abstract nodes send updates back to the present-time particle nodes.
which predicts the evolution of these particles through time; and (3) renderer which decodes the particles into an image. See Figure I for an overview of our method. All of these components are trained together and end to end via supervision on multi-step pixel-wise loss.
3.1
ENCODER
The encoder converts set of RGB-D images, corresponding short video captured from more views, into a set of latent particles for each time-step. First, we unproject each pixel into 3D) space using each camera's transformation matrix and per-pixel depth value to obtain the 3DD location Xij. Next, we process the input RGB image using a UNet (Ronneberger et al., 2015) to produce per-pixel latent features Zej. Latent features and 3D locations for each pixel (i,j) combined to form a latent particle (Xi;: Zsz) as shown in Figure 7.
This procedure produces set of particles P for each image. If more than camera is present, can simply merge all the particle sets from each time-step. The UNet shares its weights between all cameras and does receive camera or depth inputs. In our experiments, we filter the set of particles to those that lie within a designated area containing the objects of interest, which we refer to as the workspace (see Table 3). We then subsample the particles uniformly at random to limit the memory footprint of the network. See Appendix D for results with varying numbers of particles.
3.2 DYNAMICS MODEL
The dynamics model predicts the set of particles at time t + given the current and previous particles using GNN, i.c. pi+1 = D(Pl-1,p') and is applied recursively to make multi-step particle rollouts. Prior work on GNN-based dynamics models generally assumes ground truth correspondences between points at different time-steps Saanche-GGozzlle et al., 2020), and use those compute e.g. velocities. However, correspondences are not available when working with video, and can be unreliable or unspecified (e.g. objects moving out of frame). We therefore decide to entirely side-step finding correspondences. Instead, we connect latent points of different time-steps in graph based on spatial proximity and rely on the network to learn to correctly propagate information.
Graph construction The most straightforward approach to construct such a graph would be to connect each point in Pt to all points in pt within certain spatial distance. However, working with videos requires a large number of points (~ 214) to accurately represent the textures and geometry of complex scene. Directly connecting these point clouds would generate excessive number of graph edges. Instead, we employ a hierarchical 2-layer GNN (Cangea et al., 2018; Ying et al., 2018; Lee al., 2019) sparsify connections. Inspired by Hu et al. (2020a); Fortunato al. (2022), construct sparse layer of "abstract nodes" A, whose locations are chosen uniformly at random from Pt. thereby roughly matching its point density. Each point in pr pt connects up to two nearest abstract nodes within a small spatial radius r., enabling the abstract nodes to convey information over time. The abstract nodes are connected with bidirectional edges to other abstract nodes within a larger radius r as well reciprocal edges to each connected node in Pr (Figure 2). Communicating through this sparse set of abstract nodes significantly reduces the number of necessary edges. Details on graph construction and parameters can be found in Appendix A.
Message passing To compute the dynamics update, we use multigraph message passing with an Mhcode-Process-DDe architecture (Pfaff al., 2021). The multigraph G is defined by the particle
4

---

Published as a conference paper at ICLR 2024
nodes Vpi-1, Vp,, abstract nodes VA, and edges Ap EA-AA, EA-pe between these node sets. We encode latent features from the image encoder into the nodes Vp, and the distance vector between node positions into the edges. Nodes VAA do not receive any initial features.
Next, we perform one round of message passing on all nodes Vp to abstract nodes VA. followed by K 10 message passing steps between abstract nodes, and one message passing step back from VA to Vp.. Finally, the features Vp are decoded to the difference vectors (Aæ, A in position, and latent state. These vectors are added to the particle state Pr to form the estimate p used for rendering- Since the renderer operates on relative locations, the predicted Áær directly corresponds to objects moving in the predicted video, while changes in A allows for predicting appearance changes caused by motion, such as a shadow moving beneath a flying object.
Algorithm I in Appendix A details the entire message passing algorithm. Following Pfaff al. (2021), we use MLPs for encoding, decoding and message passing, with separate sets of weights for different edge and node types. See Appendix A for all architectural details.
3.3 RENDERER
We use a neural renderer similar to (Xu et al., 2022; Guan et al., 2022) to render particle set into an image from desired camera. Following the volumetric rendering formulation from Mildenhall et al. (2020), rendering a given ray consists of querying neural network at set of locations along the ray to get an RGB color and density, then compositing these values along the ray to get the color of single pixel. The renderer R predicts a color è given a particle set and the origin and direction (o, d) of this ray: query location are computed from a set of lc approximate nearest neighbor particles (see Appendix I).
R(P, (o, d.). ) For VPD, and similar to Xu et (2022), the input features at each
Let Ps be the location of a particle and Pz be its latent features, with N(x)) the set of particles close to location We compute the features at location æ using a kernel k to be (xr) EpeN(a) k(x,, pr)) pzg. For VPD, we use a set of concentric annular kernels defined by radius r and bandwidth
ky 6r, x²) = exp {
(1|=²'
x 2l2 r)
ba
This kernel weights points more highly when they are at a distance from the query location. We concatenate the computed features using a set of m kernels k ..... km giving the final feature vector
f(x) fa, (x) ..... @ fum (x)
where @ denotes concatenation. The network then predicts the color è and density â at location and a viewing direction u as (2, a) = ML)(((()) @ ((u)). where y is NeRF positional encoding.
3.4 TRAINING
Given multi-view RGB-D video, train VPD by encoding the first two timesteps into particle sets (P1, p2). We then recursively apply the dynamics model pi+1 D(p'-1, pt) to generate particle sets (P3 , p2+T) for some prediction length The complete loss the expected squared pixel reconstruction error timesteps 3... T and ray indices across all cameras:
LVPD
1 T
Σ 1=3
E
(
pt, (0)
(3)
In our training we use 256 sampled rays per timestep to evaluate this expectation.
4 EXPERIMENTAL. SETUP
We test VPD on three datasets which stress different simulator capabilities. The MuJoCo block dataset (Todorov al., 2012) is visually simple but tests model's ability to accurately represent crisp rigid contact (Allen et al., 2022). The Kubric datasets (Greff et al., 2022) encompass a range of visual complexities, from Platonic solids to densely-textured scans of real objects and backgrounds,
5

---

Published as a conference paper at ICLR 2024
Input frames
Prediction
17
24
31
MuJoCo
Block
Deformable
Collision
Kubric
MOVi-C
Camera angle
30
60 90
120
150
180
Deformable Collision
t-15
Figure 3: Top: Example rollouts from VPD different datasets. Bottom: Viewpoint generalization; VPD is unrolled for time steps, and the latent state is rendered from novel camera positions. See video site and Appendix F, Appendix G for more rollouts.
and tests a model's ability to represent multi-ohject interactions varied visual environments. The deformable dataset evaluates model's ability to represent the dynamics of non-rigid objects with large number of degrees of freedom. In all cases, the models are provided with RGB-D views from multiple cameras. For evaluation, 16 trajectories and we report each model's PSNR (with SSIM in Appendix E) (Wang et al., 2004). The PSNR (Peak Signal Noise Ratio) correlates with the mean squared error between the ground truth and predicted images, and therefore captures both deviations in the dynamics and effects of blurring.
chosen random and held out from each dataset,
MuJoCo block contains 256 trajectories, each with 64 time-steps, of single block being tossed onto a plane with randomized initial height, velocity, and angular velocity. RGB-D images are generated from 16 128 128 pixel resolution cameras arranged half sphere.
Kubric Movi-A/B/C (Greff et al., 2022) includes trajectories of objects (from a set of 3, 10, and 1033 different possible shapes, respectively) being thrown randomized initial velocities and sizes. We re- re-generate the dataset to include camera information. RGB-D images are rendered from 9 cameras of 128 128 pixel resolution arranged in half dome. 350 trajectories are used for training, each with 96 time-steps.
grey/coloured/eextreel floor with
Deformables is a dataset of deformable objects, simulated and rendered using Blender (Blender, 2018) softbody physics. In Deformable Block cube is dropped onto the floor with randomized initial position and orientation. Deformable Multi, one object out of set of five (Cube, Cylinder, Donut, Icosahedron, Rubber duck) is dropped onto the floor with randomized initial position and orientation. Deformable Collision two objects are dropped and collide onto the floor. All datasets consist of 256 training trajectories with 80 time steps. The scene is rendered using four cameras of 128 x 128 pixel resolution arranged in a half dome.
4.1 BASELINES
We provide two major baseline comparisons to highlight the importance of VPD's compositional 3D architecture: SlotFormer and NeRF-dy. SlotFormer is compositional 2D object-basse based video model (Wu et al., 2023) and NeRF-dy is non-compositionall 3D prediction model (Li al., 2021a).
SlotFormer is 2D video prediction model that operates over a slot-based latent representation (Wu al., 2023). It uses slot-based image encoder, SAVI (Kipf al., 2021), and models how these slots

---

Published as a conference paper at ICLR 2024
MuJoCo Block
Deformable Collision
Kubric MOVi-C
28
35-
PSNR
PSNR
30-
PSNR
22
30-
25-
20-
25
10
20
30
10
20
30
18
10
20
30
Rollout step
Rollout step
Rollout step
Footoomer
Global
Sequential
VPD
Figure 4: Video image quality as function of rollout duration. VPD captures sharp detail, leading to high PSNR the first steps of prediction. Low PSNR at the first steps indicates blurry reconstructions.
change over time with a transformer. In contrast to VPD, SlotFormer operates on single-view RGB camera data. To provide SlotFormer with the same data that VPD has access to, we treat each view in the multi-view dataset as a separate trajectory for training. Matching prior work, we condition SlotFormer 6 history time-steps rather than 2 used for our model. We adapt the open-source code to run on the Kubric, MuJoCo and Deformable datasets.
NeRF-dy (Li et al., 2021a) is the most similar work to ours, with its ability to use multi-view video data and perform novel view synthesis. Since neither the code nor the datasets from that work are available, we perform ablations to VPD that roughly capture the key architectural differences between VPD and NeRF-dy as highlighted below:
1. NeRF-dy uses a global latent vector per scene. Instead of 3D latent representation, NeRF- dy uses a single latent vector to represent the scene. This affects both the dynamics model (implemented as an MLP on z instead of GNN) and the renderer (globally conditioned 2). We implement this as an ablation, "Global", which uses the image and camera orientation encoder from Li et al. (2021a), and models the dynamics using an MLP rather than a GNN. As in VPD, the encoder and the dynamics are trained jointly.
2. NeRF-dy uses sequential training of the encoder/decoder followed by dynamics. NeRF- dy pretrains the encoder and decoder then holds their weights fixed and trains the dynamics model with supervision in the latent space. We use this training scheme in conjunction with the VPD particle-based architecture to investigate the significance of end-to-end training. This ablation is referred to as "Sequential".
More implementation details for these baselines are available in Appendix B and Appendix C.
RESULTS
Across the MuJoCo, Kubric, and Deformables datasets, we find that VPD produces detailed, phys- ically consistent long-horizon rollouts (Section 5.1), supports compositional 3D editing and re- rendering of novel scenes (Section 5.2), and can learn simple dynamics with as few as 16 trajectories and 1-2 RGB-D cameras (Section 5.3). All metrics and visuals are from the held-out set. Videos of rollouts for each model, as well as generalization experiments, can be found on the project website.
5.1 VIDEO PREDICTION QUALITY
We first evaluate each learned simulator's ability to generate long horizon video predictions. After training, each model is rolled out to generate predicted videos for 32 timesteps into the future, which are compared to the ground truth future frames. VPD performs consistently well across all three datasets (Table 2) which is to be expected given the high quality qualitative rollouts (Figure 3). The Global representation struggles to capture multi-object scenes, which can be seen in its lower scores on the Kubric datasets and Deformable Collision. SlotFormer performs well on the solid-colored rigid bodies it was designed for (MOVi-A/B), but struggles represent detailed textures (MOVi-Ci C) or deformable objects when they come into contact with the floor. VPD is able to successfully represent both multi-object collision dynamics and object shape changes. Kubric's scenes pose challenge for
7

---

Published as a conference paper at ICLR 2024
Initial Trajectory
Deleted Sphere
Deleted
Cylinder
Deleted
Floor
16
30
50
Figure 5: A demonstration of deleting various elements from our model's 3D scene representation on the Deformable Collision dataset before running the learned simulator forward for 50 time steps.
VPD because they have a huge diameter (100 meters); we find VPD predictions can be improved slightly by bounding the input scene (Appendix H).
VPD's prediction quality also degrades gracefully as a function of rollout step (Figure 4). VPD is able to represent significantly more detail than SlotFormer. as evinced by its high PSNR scores in the first few rollout steps. Meanwhile VPD's predictions at 32 steps are often better than the Global ablation at any step, and it remains much more accurate than the Sequential ablation at 32 steps.
These results highlight the ability of VPD's latent 3D particle representation to precisely capture a scene, and the locality bias of the VPD dynamics to preserve scene coherence over time.
5.2 EDITABILITY AND NOVEL VIEW GENERALIZATION
3D point cloud editing When using learned simulator with a 3D scene representation, it is possible to directly interrogate and even edit the scene before simulating. This capability could support many applications such as adding 3D assets to a 3D scene estimated from a set of sensors, tweaking object shapes, deleting objects, moving them around.
We demonstrate 3D editing and simulation with VPD on a held-out test trajectory from the de- formables collision dataset. Figure 5 visualizes the latent particles by applying the VPD renderer to each particle location to get a color, then plotting the colored points 3D. In the original trajectory, a deformable pink ball and red cylinder fall onto floor, squishing together into a pink and red lump. In order to edit the points corresponding to each object (ball, cylinder, floor) we roughly cluster the particles by position on the first frame. If we delete the pink ball from the inferred 3D scene, the cylinder still falls onto the ground. However, the top of the cylinder is no longer squished, since the pink ball did not come into contact with it. If we delete the cylinder, the pink ball falls all the way to the floor without deforming. If we delete the floor, the two objects both fall at a steady rate and neither one deforms since they both fall at the same acceleration. In all cases, this is roughly the behavior we would expect from the true underlying physics.
Method
MuJoCo|
Deformable
Kubric
Block
Block
Multi
Collision
MOVi-A
MOVi-B
MOVi-C
SlotFormer
32.541
28.673
27.545
23.145
30.934
28.810
22.367
Global
29.601
26.969
26.636
22.394
27.604
26.201
20.068
Sequential
25.858
30.854
31.256
27.911
28.613
25.921
21.854
VPD (Ours)
33.076
31,520
31.221
28.725
29.014
27.194
23.142
Table 2: PSNR scores averaged over 32 rollouts steps (higher is better).
8

---

Published as a conference paper at ICLR 2024
PSNR
324
38
Input
8
38-
30-
PSNR
85-
34-
30-
16
32
94
120
236
10
20
30
10
90
Dataset size
Rollout step
Rollout step
StatFormer
Global VPD
Predictdd depth Thue depth
(a) Dataset sizes
(b) Input views
Estimated depth
Figure 6: Video prediction results with various changes the MuJoCo dataset that ablate different aspects of the input data. (a) VPD works effectively across a range of dataset sizes, whereas baselines require more data achieve good results. (b) VPD learn well conditioned on one RGB-D camera as on 9 RGB-D cameras. (c) VPD is able to sustain reasonably high prediction quality even with an imprecise learned depth model (blue line).
Novel viewpoint generalization Operating on 3D latent point clouds also allows us render high-quality images from novel viewpoints, even when training very few camera poses. In Figure 3, we show that VPD trained on the "Deformable Collision" dataset with only fixed camera angles has no trouble generalizing to novel camera positions in ring around the object. This sparse setting challenging for c.g. NeRF-based methods which have weaker spatial biases, and impossible for 2D predictive models like SlotFormer. See Appendix Figure 9 for more examples.
5.3 DATA SPARSITY
In many applications, it might not be possible to obtain large numbers of RGB-D images of particular scene at each timestep. We therefore investigate how well VPD performs when it has access very small number of trajectories input views, and when it must depth information predicted by a monocular depth estimation network.
Different numbers of trajectories We generate datasets for MuJoCo Block of sizes 16, 32, 64, 128 and 256 trajectories, and evaluate each model's video prediction quality on the same set of 16 find that
held-out trajectories (Figure 6a). Surprisingly, trajectories, VPD achieves a PSNR of 31 31.22 which is only 1.85 points less than VPD trained with 256 distinct trajectories, and comparable in performance to baseline models trained with 4-8 as much data. This suggests that VPD could be suitable for applications in robotics where collecting many trajectories can be expensive.
with few 16 distinct training
Different numbers of views We train models on MuJoCo Block with between and 9 input views. With fewer views, VPD will have significant uncertainty about the object's shape (for example, it cannot see the back of the block). Despite this challenge, even with a single RGB-D camera VPD 9 RGB-D cameras. While we are unsure how VPD performs
performs comparably to when so well with single RGB-D camera, one hypothesis is that since VPD's dynamics are trained end to end with the encoder, the dynamics could compensate for the particles that are missing due to self-occlusion. Future work will investigate whether these results apply to complex datasets, and whether separately training the dynamics and the encoder erases this effect.
VPD with ground-truth depth information throughout experi- ments, we provide preliminary experiment using estimated depth instead. We extend the encoder's UNet with additional output feature plane and interpret its value pixel-wise metric depth. This depth prediction is supervised with an MSE loss on the training images and receives gradients via end end training. On the MuJoCo Block dataset, VPD makes predictions of reasonable quality even with imprecise depth estimates (Figure 6c).
Predicted depth While
6 DISCUSSION
Visual Particle Dynamics (VPD) represents first step towards simulators that can be learned from videos alone. By jointly learning a 3D latent point cloud representation, dynamics model for evolving that point cloud through time, and renderer for mapping back to image space, VPD does require

---

Published as a conference paper at ICLR 2024
state information in the form of object masks, object geometry, object translations and rotations. Since VPD encodes particles in 3D, it also supports novel view generation and video editing through direct point cloud interactions. To knowledge, this combination of 3D interpretability, editability, and simulation is unique among predictive models learned without physical supervision.
However, VPD also has limitations. As evidenced in videos and figures, VPD struggles with blurring over very long rollouts, a common challenge in video prediction. also requires access RGB-D videos, which are not always available. While we showed proof-of-concept for VPD working with predicted depth instead of ground-truth depth, this still requires a depth signal for training which may not always be available. Future work will need to investigate the potential application of pre-trained depth models with fine-tuning to particular scenario.
We believe that VPD's ability to learn simulators directly from sensors has several important implica- tions. For robotics, VPD could be used to learn simulator directly from real data without requiring a separate state estimation step. This could then support much more effective sim-to-real transfer by learning simulator that better reflects the robot's s environment. For graphics, VPD could support new applications such as realistic video editing for systems where is difficult to hand-code simulators. Overall, VPD opens new directions for how simulators can be learned and deployed when no analytic simulator available.
REFERENCES
A. Ajay.J, J. Wu, N. Fazeli, M. Bauza, L. P. Kaclbling, J. B. Tenenbaum, and A. Rodriguez. Augmenting physical simulators with stochastic neural networks: Case study of planar pushing and bouncing. In International Conference on Intelligent Robots and Systems (IROS), 2018.
Kelsey R Allen, Tatiana LopezaGuevara, Yulia Rubanova, Kim Stachenfeld, Alvaro Sanchez- Gonzalez, Peter Battaglia, and Tobias Pfaff. Graph network simulators can learn discontinuous, rigid contact dynamics. In 6th Annual Conference on Robot Learning, 2022.
Kelsey R. Allen, Yulia Rubanova, Tatiana Lopez-Guevara, William Whitney, Alvaro Sanchez- Gonzalez, Peter Battaglia, and Tobias Pfaff. Learning rigid dynamics with face interaction graph networks. In International Conference on Learning Representations, 2023.
Peter W Battaglia, Jessica B Hamrick, Vietor Bapst, Alvaro Sanchez-Gonzalez Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 18X6v 01261,, 2018.
Blender. Blender a 3D modelling and rendering package. Blender Foundation, 2018. URL
http:/ /ttp:://wwww wwwnder........... org.
Cätälina Cangea, Petar Velickovic, Nikola Jovanovic, Thomas Kipf, and Pietro Lio'. Towards sparse hierarchical graph classifiers. ArXiv, abs/1811.01287, 2018. URL https:/ //api.//ppii/ 53321991..
Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets.
arXiv preprint arXiv: 2019.
Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://pybullet org, 2016.
Danny Driess, Zhiao Huang, Yunzhu Li, Russ Tedrake, and Marc Toussaint. Learning multi-object dynamics with compositional neural radiance fields. arXiv preprini arXiv:2202.1 1855, 2022.
Tao Du, Kui Wu, Pingchuan Ma, Sebastien Wah, Andrew Everett Spielberg, Daniela Rus, and Wojeiech Matusik. Diffpd: Differentiable projective dynamics with contact. ArXiv, abs/2101.0500 059117, 2021a. URL https://///........ /appi...... semant cewholccchhhlooooooco wrgg/oopussaaaaeeooo
627650.
Yilun Du, Kevin Smith, Tomer Ulman, Joshua Tenenbaum, and Jiajun Wu. Unsupervised discovery of 3d physical objects from video. Neural Information Processing Systems, 202
10

---

Published as a conference paper at ICLR 2024
Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. In 2021 IEEE/CVE International Conference on Computer Vision (ICCV), pp. 44304-14344 IEEE Computer Society, 2021c
Yilun Du, Mengjio Yang, Bo Dai, Hamjun Dai, Ofir Nachum, Joshua B Tenenbaum, Dale Schu- urmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. URL https://arxiv. org/abx/2302.00//1 2023.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. Advances in neural information processing systems, 29, 2016.
Meire Fortunato, Tobias Pfaff, Peter Wimsserger, Alexander Pritzel, and Peter Battaglia. Multiscale meshgraphnets, 2022.
C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax a differentiable physics engine for large scale rigid body simulation. ArXiv, 0bs/2200..1288 2021. URL https: /ppi. seaatt icscholar wwtpCooppss//:cc22.......
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J. Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsuch-Ti Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang. Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: A scalable dataset generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3749-3761, June 2022.
Shanyan Guan, Huayu Deng, Yunbo Wang, and Xiaokang Yang. Neurofluid: Fluid dynamics grounding with particle-driven neural radiance fields, 2022.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04 2023.
Eric Heiden, Ziang Liu, Vibhav Vineet, Erwin Coumans, and Gaurav S. Sukhatme. Inferring articu- lated rigid body dynamics from rgbd video. 2022 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 8383-8390, 2022. URL httph:tt////e/tt /tas:////piiisee semant commmnntccchoo..o iescholsala ar. org/Corpus 24129447549 7594294.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:22 2022.
Taylor A. Howell, Simon Le Cleac'h, h, Zico Kolter, Mac Schwager, and Zachary Manchester. Dojo: A differentiable simulator for robotics. ArXiv, 0abs/2203.00806. 2022. URL https: // //api. aaapi.semantie www.nttisccoool..o...
Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic segmentation of large-scale point clouds, 2020a.
Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, and Frédo Durand. Difftaichi: Differentiable programming for physical simulation. ICLR, 2020b.
Michael Janner, Sergey Levine, William T. Freeman, Joshua B. Tenenbaum, Chelsea Finn, and Jiajun Wu. Reasoning about physical interactions with object-oriented prediction and planning, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. CoRR, abs/1412.6980,, 2014. URL https:: /ppi. www.nticscholii....o wtw.hooaa..orgooouus........ 6628106.
Thomas Kipf, Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. arXiv preprint arXiv:2111.12594 2021.

---

Published as a conference paper at ICLR 2024
Alina Kloss, Stefan Schaal, and Jeannette Bobg. Combining learned and analytical models for predicting action effects from sensory data. The International Journal of Robotics Research, 41(8): 778-797, 2022.
Dmitrii Kochkov, Jamie A. Smith, Ayya Alieva, Qing Wang, Michael Brenner, and Stephan Hoyer. Machine learning-acee accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences, 118(21), 2021.
Simon Le Cleac'h, h, Hong-Xing Yu, Michelle Guo, Taylor Howell, Ruohan Gao, Jiajun Wu, Zachary Manchester, and Mac Schwager. Differentiable physics simulation of dynamics augmented neural objects. IEEE Robotics and Automation Letters, 8(5):2780-27877 2023.
augamecs-augmemtted
Junhyun Lee, Inyeop Lee, and Jaewoo Kang. Self-attention graph pooling. ArXiv, abs/1994.08082, 2019. URL https:://// : ara.pissaaa....c.cll..a......... org/Corpus
ott.aaacciccccolla.ooooooooooooo : 119314157.
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenenbaum, and Antonio Torralba. Learning
particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In International Conference Learning Representations, 2019.
Yunzhu Li, Shuang Li, Vincent Sitzmann, Pulkit Agrawal, and Antonio Torralba. 3d neural scene representations for visuomotor control. arXiv preprint arXiv:2107.04000
2021a.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021b.
Zhengqi Li, Qianqian Wang. Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image- based rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 4273-4284, 2023.
Miles Macklin. Warp: A high-performance python framework for gpu simulation and graphics. https: /ttt::///tttbb com com/nvidia/wwrrgg..occcii.aaaaaa March 2022. NVIDIA GPU Technology Confer- ence (GTC).
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li F Fei-Fei, Josh Tenenbaum, and Daniel L Yamins. Flexible neural representation for physics prediction. Advances in neural information processing systems, 31, 2018.
OpenCFD. OpenFOAM CFD solver. www. www.foaam. org, 2021.
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter Battaglia. Learning mesh-based simulation with graph networks. International Conference on Learning Representations, 2021.
Haozhi Qi, Xiaolong Wang, Deepak Pathak, Yi Ma, and Jitendra Malik. Learning long-term visual dynamics with region proposal interaction networks. arXiv preprint arXiv:2008. 00ti:22008.02665,, 2020.
Yi-Ling Qiao, Alexander Gao, and Ming C. Lin. Neuphysics: Editable neural geometry and physics from monocular videos. In Conference Neural Information Processing Systems (NeurIPS), 2022.
Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, and Emmanuel Dupoux. Intphys: A framework and benchmark for visual intuitive physics reasoning, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation, 2015.
Alvaro Sanchez-Gonzan Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec. and Peter Battaglia. Learning to simulate complex physics with graph networks. International Conference Machine Learning, pp. 8459-8468 PMLR, 2020.
12

---

Published as a conference paper at ICLR 2024
Han Shao, Tassilo Kugelstadt, Torsten Hädrich, Wojtek Palubicki, Jan Bender, Sören Pirk, and Dominik L Michels. Accurately solving rod dynamics with graph learning. Advances in Neural Information Processing Systems, 34:4829-488 4842, 2021.
Haochen Shi, Huazhe Xu, Zhiao Huang, Yunzhu Li, and Jiajun Wu. Robocraft: Learning see. simulate, and shape elasto-plastic objects with graph networks, 2022.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. IEEE, 2012.
Benjamin Ummenhofer, Lukas Pranti, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid simulation with continuous convolutions. In International Conference on Learning Representations, 2019.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
Nicholas Watters, Daniel Zoran, Theophane Weber, Peter Battaglia, Razvan Pascanu, and Andrea Tacchetti. Visual interaction networks: Learning physics simulator from video. Advances in neural information processing systems, 30, 2017.
Jiajun Wu, Erika Lu, Pushmeet Kohli, Bill Freeman, and Josh Tenenbaum. Learning to see physics via visual de-animation. Advances in Neural Information Processing Systems, 30, 2017.
Ziyi Wu, Nikita Dvomik, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsupervised visual dynamics simulation with object-centric models, 2023.
Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: based neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5438-5448, 2022.
Haotian Xue, Antonio Torralba, Josh Tenenbaum, Dan Yamins, Yunzhu Li, and Hsiao- Yu Tung. 3d-intphys: Towards more generalized 3d-grounded visual intuitive physics under challenging scenes. Advances in Neural Information Processing Systems, 36, 2024.
Rex Ying, Jiaxuan You, Christopher Morris, Xiang Ren, William L. Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. ArXiv, abs/180 806. 08800, 2018. URL https: api semant wwt.chccchascccccccoocc org/ Corpus :4:2:00::33
Yanshu Zhang*, Shichong Peng*, Alireza Moazeni, and Ke Li. Papr: Proximity attention point rendering. Advances in Neural Information Processing Systems, 2023.
13

---

Published as a conference paper at ICLR 2024
Appendix
A VPD IMPLEMENTATION DETAILS
This section describes details of our VPD implementation. Dataset-specific hyperparameters be found in Table 3.
A.1 ARCHITECTURES
Encoder Figure 7 shows schematic of the encoder network. The network uses UNet architecture (Ronneberger et al., 2015) with the following numbers of channels: (64, 128, 256, 512, 512, 256, 128, 64. 16). The network outputs 16-dimensional feature vector Zij for each input pixel, which we associate with the corresponding particle. When using the UNet for depth prediction, we add another output channel (17 total) and take this channel be pixel-wise depth prediction; the predicted depth is not included in the particle features Z. There is a ReLU activation between each layer and batch normalization. The UNet has final softplus activation. The pixels from each image are unprojected together into the world coordinate frame, cropped to only include locations in the workspace (Table 3). They are then subsampled uniformly at random down to a set of 2 particles representing a single time-step.
Dynamics model The dynamics model is hierarchical message passing graph neural network using MLPs for the node and edge encoders, updaters, and decoders. There separate set of parameters for each type of {node, edge} (encoder, updater, decoder). We use 10 steps of message passing amongst the abstract nodes with un-tied weights at each message passing step. Each encoder or updater MLP has shape (128, 128), the location decoder MLP has shape (128, 128, 128, 3), and the feature decoder MLP has shape (128, 128, 128, 16), where the location delta and 16 the particle feature delta. The updater MLPs use output layer norm and output residual connection.
The outputs of this GNN are one 3-d location delta vector and one 16-d feature delta vector for each current- tintenttttmmteppppeo particle. The location delta goes into tanh function and then is scaled by the "location scale" from Table 3, which helps early in training by preventing particles from moving unrealistically fast. The particles the next timestep produced by applying these deltas.
The renderer is based the jaxnerf codebase with accelerated Pallas CUDA kernels for nearest-neighhor finding. We use 16 approximate nearest neighbors for computing the features for input the NeRF MIPs, and the kernel parameters are in Table 3. We use two tiers of sampling for NeRF rendering, a coarse layer with 128 sampling locations and no view directions and a fine layer with 128 (coarse) + 256 (fine) sampling locations. The coarse and fine networks have 8 layers of width 256 before concatenating view directions (if using), followed by 1 layer of width 128. Skip connections are added to every 4th layer and ReLU activations are used throughout. We use fourth-degree positional encodings on the view directions in the fine MI.P
Renderer
A.2 TRAINING
During training, we roll out the model for T time steps. For each unrolled time step, render 256 rays, and supervise on the corresponding ground truth pixel value as in Equation 3. We apply a small amount of Gaussian noise centered and with sigma given by "location noise" in Table to the particle locations during training rollouts to improve robustness. We use a batch size of 16 split across 16 GPUs. Optimization uses the Adam optimizer (Kingma & Ba, 2014) with learning rate that begins at 3e 4, then decays by factor of 3 at 100K and 300K updates. Models are trained for 400K updates.
A.3 HIERARCHICAL MESSAGE PASSING
The hierarchical message algorithm is detailed Algorithm 1. The initial features for the particle nodes are the features predicted by the UNet encoder, and the initial feature for the edges is the 3D
14

---

Published as a conference paper at ICLR 2024
Parameter
MuJoCo
Deformable
Kubric
0.1
1.0 3.0
1.0
0.3
Input views
4
9
Kernel radii
(0.0, 0.05, 0.1, 0.5)
(0.0, 0.5, 1.0, 4.0)
(0.0, 0.1, 0.4, 2.0)
Kernel bandwidths
(0.05, 0.05, 0.05, 0.5)
(0.5, 0.5, 1.0, 4.0)
(0.1, 0.2, 0.4,
Location scale
0.1
0.5
0.5
Location noise
le-5 5
1e-3
1e-4
Near plane
0.3
9.0
0.5
Far plane
5.0
30
15 -10,-10,110)
Workspace min
[-1,-1,-1]
[-8, -8, -2] [8, 8]
Workspace max
[1, 1,
[10, 10, 10]
Table 3: Hyperparameters for VPD by dataset
relative position between the start and end nodes. Abstract nodes start with no features, identified in Algorithm Ø.
Algorithm 1: Hierarchical message passing
input:Iyped graph 9 = (V,E)
/* Encode particle nodes and edges. */
Vp + EncodeNode((/p)
E + EncodeEdge(Cp-A))
/* Update particle + abstract edges. */
E UpdateFddge(rp--: Vp
Update abstract nodes.
V + UpdateNode(
/* Perform message passing among abstract nodes. +/
k I.... do
EA-A* OodatteEdge(EA--A VA. VA)
VA + UpdateNode (A-IA)
end
/* Update features of abstract + current particle nodes. x/
EA-PP Updatefdgee(A--pp VA, Vpi)
/* Update current-time particle nodes.
Vp + UpdateNode((ppr,
/* Decode predicted locations and features. */
AX f- Decodel_ocation (Pp:)
AZ DecodeFeature() p..)
B SLOTFORMER TRAINING
We experimented with both the OBJ3D and CLEVRER hyperparameter configurations for all datasets, and found the results from the OBJ3D configuration to be superior. These hyperparameters are in Table
Kubric datasets use 8 slots while the Mujoco and Deformable datasets 4 slots (8 slots caused instabilities in training). Kubric was also downsampled to 64 x resolution for training due to training stability challenges. MuloCo and Deformables datasets were trained 128 x 128 resolution.
After training, the slots learned by SAVi are extracted for the training and validation datasets. SlotFormer is then trained these slot representations, using the hyperparameters in Table 5.
15

---

Published as a conference paper at ICLR 2024
VPD Encoder
M
RGB
features
UNet
Deprik 4
Jijj
Rig
umproject
Figure 7: diagram of the VPD Encoder. The encoder maps an RGB-D image into a latent point cloud representation. Each pixel observed from given camera is unprojected into 3D using the depth information. The RGB image fed into a UNet to produce per-pixel latent features.
Hyperparameter
Setting
epochs
40
learning rate
le 4
gradient clip
0.05
history length
frame offset
batch size slot mlp size num iterations encoder channels
64
256
(3, 64, 64, 64, 64)
encoder ks encoder out channels decoder channels decoder resolution decoder ks predictor type pred norm first pred rnn pred_num.layers. pred.num.heads pred.ffn.dim use_post.recconloss kidmethod
128
(128, 64, 64, 64, 64)
(8, 8)
transformer
True True
128 4
True
none
post recon loss
kld loss w
1e 4
Table 4: SAVi training hyperparameters
C ABLATION IMPLEMENTATION DETAILS
C. SEQUENTIAL
Supervising the dynamics in the particle-based latent space is more complex than it is with the NeRF-dy architecture, since particles have no correspondences across timesteps. To avoid the correspondence issue, randomly sample 2 3D locations in the scene's workspace, then compute the renderer input features at those points under (1) the particle cloud at the next timestep predicted by the dynamics model and (2) the particle cloud from the encoder obtained from the set of RGB-
16

---

Published as a conference paper at ICLR 2024
Hyperparameter
Setting
epochs learning rate history length sample frames
Kubric: 15, Deformables: 80, MuJoCo: 80
2e 4
6
16 (10 rollout)
frame offset
batch size
64
slot size
128
t-pe
sin 128
model
num layers
num heads
ffn_dim norm-first
128 x 4
True (128, 64, 64, 64, 64)
decoder channels
decoder resolution
(8, 8)
decoder ks
use img recon loss
True 10
rollout length
slot.reconnosss.w
img.recon.loss.ww
Table 5: SlotFormer training hyperparameters
images at the next timestep. The loss then the mean squared error between the features at corresponding locations. This ensures that the dynamics model's predictions lead to renderer input features which are correct at all locations in the scene. In the limit of perfect predictions, this would translate correct reconstruction.
We find sequential training to be less robust than end-to-end training and needed to use lower learning rates: 1e 4 to start for both pre-training and dynamics training, decaying by factor of 10 followed by an additional factor of 3. We perform 100K steps of auto-encoder pre-training and 400K steps of dynamics model training.
D SCALING WITH NUMBER OF POINTS
We find that VPD's video quality increases as more particles are used to represent the scene. greater number of particles enables the model to represent details of geometry and textures.
33.0
Average PSNR
32.5
32.0
31.5
1024
2048
4096
8192-
18984
Number of particles
Figure 8: Evaluating the performance of VPD with varying numbers of particles, measured as average
PSNR over 32-step rollout. More particles result in consistently better performance.
17

---

Published as a conference paper at ICLR 2024
E
SSIM RESULTS
SSIM is an alternative metric to PSNR for evaluating image similarity. It compares statistics of windowed pixels and permits more blurring than the pixel-wise enror of PSNR. Results are broadly similar to scoring with PSNR, but with less discriminitive power due to SSIM's insensitivity to detail.
Method
MuJoCo
Deformable
Kubric
Block
Block
Multi Collision
MOVi-A
MOVi-B
MOVi-C
SlotFormer
0.954
0.957
0.956
0.913
0.880
0.867
0.547
Global
0.954
0.936
0.932
0.885
0.848
0.846
0.529
Sequential
0,699
0.953
0.953
0.933
0.850
0.845
0,606
VPD (Ours)
0.960
0.957
0.953
0.939
0.857
0.859
0.703
Table SSIM scores averaged over 32 rollouts steps (higher is better). Ties awarded within 0.01.
F RENDERING FROM NOVEL VIEWS
Since VPD contains 3D representation of the scene and learns ray-based renderer, we can render novel views from camera positions not seen in training. Figure shows an example of this for the Deformable Collisions dataset on held-out trajectories.
Figure 9: Generalization over camera pose. A VPD model trained on Deformable Collision is rolled on the test set for 15 frames, and the final latent state rendered from multiple camera positions oriented in ring around the objects. A video example of free camera movement can be found [here].
18

---

Published as a conference paper at ICLR 2024
G ROLLOUT COMPARISONS BETWEEN VPD AND BASELINES
In each of these diagrams, the first two frames are ground-truth inputs to the models. SlotFormer is conditioned on six frames, not two; the first four frames of SlotFormer conditioning are not shown. All videos are from the held-out evaluation set.
Input t=0
Input t=1
t=3
[=9
[=17
1=24
[=31
Ground touth
Stoffrmerr
Sequeroo
VPD
Figure 10: Rollout comparison on MuloCo Block dataset
Input t=
Input t=1
t=3
t=
t=17
t=24
t=31
Groand buth
aendal
Figure 11: Rollout comparison Deformable Block dataset
19

---

Published as a conference paper at ICLR 2024
Input t-0
Input t-]
t-3
t-9
t-17
t-24
t-31
trath
Sequnnail
VIO
Input tm0
Irput t=1
t=3
t=9
t=17
t=24
t-31
butb
Sequerbal
WP
Figure 12: Rollout comparison on the Deformable Multi dataset.
20

---

Published as a conference paper at ICLR 2024
Input two
Input t=1
tw3
[m9
t-17
1=24
t-31
Ground mobs
Securtcll
VPD
Input
Sequaniial
Figure 13: Rollout comparison on Deformable Collision dataset.
21

---

Published as a conference paper at ICLR 2024
Input
Input t=]
tm3
[-9
t-17
t-24
t-31
touth
0
Steforme
Sequiettal
t-9
t-31
Groandd
Suquamaa
Figure 14: Rollout comparison on Deformable Collision dataset,
22

---

Published as a conference paper at ICLR 2024
Input
Input
WD
Figure 15: Rollout comparison on Kubric MOVi-A dataset.
Input
Input t=]
twg
t-17
tw31
Sequencal
Figure 16: Rollout comparison on Kubric MOVi-B dataset.
23

---

Published as a conference paper at ICLR 2024
Input t=0
Input t=1
t=3
t=
t=17
t=24
t=31
trats
Global
Seeuunicial
VP2
Figure 17: Rollout comparison Kubric MOVi-C dataset.
24

---

Published as a conference paper at ICLR 2024
H IMPROVING VPD RECONSTRUCTION QUALITY ON KUBRIC.
In the main text we use the Kubric dataset with the conventional camera setup, where the cameras point sideways and cover the walls surrounding the scene. However, scenes with far-away objects generally challenging for NeRFs, as they require sampling points at lower resolution along the ray, resulting in blurry reconstructions.
We generate version of Kubric where the cameras are positioned higher and point downwards, creating bounded scene that is more suitable for NeRFs. Figure 18 demonstrates that the quality of VPD reconstructions considerably improves the updated camera setup. The objects are more crisp and track the ground-truth object trajectory more accurately.
Input t=0
Input t=1
t=3
t=9
t=17
1=24
t=31
VPD
(a) MOVi-A with high cameras
Input t-0
Inpuc t-1
t-3
t-9
t-17
t-24
t-31
VPb
(b) MOVi-C with high cameras
Figure 18: VPD recontructions Kubric with the modified camera setup.
25

---

Published as a conference paper at ICLR 2024
I FEATURE ENCODING EXPERIMENTS
The partiele features from VPD resemble those used in PointNerf (Xu et al., 2022), although there some differences. PointNerf includes an extra feature MLP F' x - p²)). This MLP processes nearby points (within a certain range k) to predict the feature pattern at a specific æ. We experimented with modified version of VPD that uses similar additional feature encoder F, but had reduce the number of prediction steps to T 2 to fit in memory. We did not observe any significant benefits when compared to the full VPD model, particularly when considering longer rollouts (Figure 19a).
Another variation of feature encoding is the normalization feature described in PAPR (Zhang* et al., 2023). We explored variation of VPD that normalizes the feature (Equation (1)) contributions with respect to the relative distances between the query point æ and a given nearby particle pz. Similarly, we did not observe any notable advantages compared to the complete VPD model (Figure 19b).
38-
36
36-
PSNR
PSNR
34-
34-
32-
32-
30-
30-
28
28
10
20
30
0
10
20
30
Rollout step
Rollout step
Feat-VPD (T=2)
VPD
VPD (T=2)
Norm-VPD
VPD
(a) Feature encoding strategies on MulaCo
(b) Feature normalization strategies on MuJeCo Block.
Block.
26


Reviews:
[{"review_id": "9kIUS6MABr", "reviewer": "Reviewer_cZVt", "summary": "This paper proposes a method to learn a particle-based representation, a neural simulator, and a neural renderer from posed RGB-D videos. \n\nFirst, this method projects pixels back to 3D point clouds with known depth values and camera poses. Second, a UNet is applied to the RGB channels to extract per-pixel features. Then, a neural simulator implemented using MeshGraphNet is applied to infer the dynamics of the scene. In the end, a NeRF-style neural renderer is used to synthesize the image, where the feature of a query point is computed by filtering over neighboring points. \n\nWith ground truth RGB-D video sequences, this pipeline can learn the feature extractor, neural simulator, and neural renderer end-to-end. Experiments show that it can handle different dynamics (rigid, soft, contact, etc.). Ablation studies suggest that the designed components are necessary for the entire system.", "questions": "What's the memory and time consumption to train this pipeline? The neural simulator might need to take many samples to converge while the NeRF rendering is usually slow (although there are faster implementations, I'm not sure if it's used here).\n\nIn Figure 8, while the PSNR stays the same between 1024->2048 and 4096->8192 while takes big jumps between 2048->4096 and 8192->16384.\n\nThis method seems to treat the same points in different views as different points. \"If more than one camera is present, we can simply merge all the particle sets from each time-step\"\nIs this strategy appropriate, with unbalanced camera distribution? It would make more sense to me if additional steps could be taken to find and resolve the correspondence among views. And how is this method tacking the points across timesteps?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The motivation and building blocks of this paper are clear and reasonable. The choices of neural simulator and renderer are up-to-date. With a general, unstructured, and differentiable simulator (GNN) and render (NeRF), it is easier to simultaneously train this system end-to-end.\n\nMoreover, training the system end-to-end can indeed improve performance. Without carefully handcrafted dynamics and rendering models, this pipeline can reconstruct complex 3D dynamic information from 2.5D input.\n\nThe learned dynamics can be used to perform editing. And it is also impressive to see this method can also render the dynamic shadow correctly (as in the video demo).", "weaknesses": "The method is actually only capturing the surface points. There are no inner points reconstructed with only the depth values. We can also see this effect in the demo videos, where objects seem to be hollow.\n\nThe algorithm might rely on high-quality and multi-view RGBD videos. In the experiments, the background is clean and the objects are relatively simple. There are not so many tests on real-world data where the depth values are noisy and the view angles are sparse.\n\nMore like a systematic integration, this paper has relatively limited technical contribution. The feature extraction, neural renderer, and simulator use existing building blocks. It might be better if there more explorations on how to improve its interpretability, robustness, generalizability, and performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YUFOaSRD8T", "reviewer": "Reviewer_CLWA", "summary": "The paper introduces VPD, a method to learn particle simulators given multiview RGBD videos. The setup is similar to NeRF-dy (Li et al.). The major difference is an explicit point-based neural simulation and rendering pipeline that avoids learning mappings between images and latent states. \n\nVPD is evaluated on dastsets with deformable and rigid objects, showing better performance than SlotFormer (a video prediction method) and re-implemented NeRF-dy (a ) that roughly matches the paper.", "questions": "N.A.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Presentation**\n- The paper is generally easy to follow, and the video results on the website are helpful.\n\n**Method**\n- The point-based representations enable 3D editing capability during/before simulation. \n- The use of PointNeRF is a nice design choice that couples particle-based simulation and differentiable rendering.", "weaknesses": "**Writing**\n- The idea of learning dynamics/physics simulators from videos is not particularly new (e.g., NeRF-dy, [A-B]), but the intro and related work are positioned in a way that appears those works are not relevant. To give the readers sufficient context, I would recommend putting more effort into discussing the existing \"video->simulator\" works, their limitations, and the key differences in this work. \n\n\n**Efficiency**\n- One advantage of using a latent representation for simulation learning (e.g., NeRF-dy) is its efficiency, especially when the number of particles is large. Given enough training data, one would expect a latent representation to be more performant.\n\n**Motivation**\n- To obtain better physics simulators (of system dynamics), what do we gain by learning from videos? The motivation and evidence could be made more concrete.\n  - Specifically, the problems mentioned in the intro seem can be solved by system identification with a differentiable simulator, e.g., Taichi, Warp, Brax, dojo, which can find physical parameters (e.g., friction coefficient) from input videos without re-learn how to simulate. One example is Le Cleac’h et al., 2023.\n  - If the goal is to speed up the simulation, it seems distilling physics-based simulators into a neural architecture is a strong competitor.\n- Overall, it would be great if the paper could give a compelling example where VPD outperforms (or has the hope to outperform) the physics-based simulator. One aspect might be the generality. Another case is when there are complex disturbances in the environment that cannot be modeled by the physics simulator.  \n\n**Rigid objects**\n- It seems the results on rigid objects (Kubric Movi-a) contain undesirable deformations. This seems to suggest that the learned GNN is not able to enforce rigidity constraints for high-stiffness materials. \n- It would be nice to show how the method works on more challenging cases, such as the collision between rigid objects (or a rigid object and the ground). Those cases typically require a small timestep for physics-based simulation. \n\n\n[A] Qiao, Yi-Ling, Alexander Gao, and Ming Lin. \"Neuphysics: Editable neural geometry and physics from monocular videos.\" Advances in Neural Information Processing Systems 35 (2022): 12841-12854.\n\n[B] Heiden, Eric, et al. \"Inferring articulated rigid body dynamics from rgbd video.\" 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wtg52GO0ZV", "reviewer": "Reviewer_HnBD", "summary": "In this paper, the authors proposed a neural-network-based 3D simulator where a graph neural network is used to estimate the 3D dynamics. The simulator transforms the multi-view RGBD images into a latent point cloud and uses a differentiable simulator to transform the simulated result into images, such that an image-based loss is used to train the simulator end-to-end. The proposed simulator supports 3D state editing, novel view re-rendering and multi-material simulation.", "questions": "Can the proposed simulator support different physics properties, such as density, friction?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper uses a explicit point cloud representation, such that it supports scene editing, novel-view re-rendering. The experimental results have shown the effectienss of the method in such applications.", "weaknesses": "The claim that existing learned world models are not simulators is not accurate. The function of simulator is to estimate the transition model of the system. \nThe main weakness of the proposed simulator is that it does not support the simulation of the environment response to different actions. Therefore, it can only predict the future state of a passive system.\nSecondly, the author does not validate the extrapolation ability, which is important for a simulator. Experiments on out-of-distribution initial state can be added.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vlmz0UE08v", "reviewer": "Reviewer_8vFC", "summary": "The proposed paper introduces a novel approach for learning simulations from videos. The approach relies on RGB-D images as input and uses a combination of neural networks (UNet, GraphNetwork, Renderer) to first generate a latent particle representation that is then rendered into output images.", "questions": "- Section 3.1.: It is not clear how the per-pixel latent features are generated. Ronneberger et al. uses down convolutions to produce a compressed latent space, which would not result in a per-pixel latent. How does the UNet architecture used in this paper differ from the original one?\n\n- Section 3.1. This section is difficult to follow as some term are not defined or introduced properly. E.g. what is a 'conditioning set'? What is a \"work space\" and how is it defined?\n\n- While the paper mentions 'blurring over very long rollouts', it does not provide a discussion on what this actually means. After how many time steps do outputs become blurry? Can this be quantified? What about baseline methods?\n\n- The authors may also want to discuss the following paper: H. Shao, T. Kugelstadt, T. Hädrich, W. Pałubicki, J. Bender, S. Pirk, D. L. Michels, Accurately Solving Rod Dynamics with Graph Learning, Conference on Neural Information Processing Systems (NeurIPS), 2021", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses an important research direction -- learning physics directly from videos is a challenging and open problem. \n- To my knowledge, the proposed model and training procedure (with RGB-D images) is novel.\n- The proposed method seems technically sound and the results are convincing. \n- The paper is mostly well-written and easy to follow.", "weaknesses": "- Some sections of the text could provide more details (more details below). \n- Only synthetic data was used to test the method. \n- Not all experiments are convincing (more details below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)