3D-IntPhys: Learning 3D Visual Intuitive Physics for Fluids, Rigid Bodies, and Granular Materials
Authors: Haotian Xue
Antonio Torralba
Daniel LK Yamins
Joshua B. Tenenbaum
Yunzhu Li
Hsiao-Yu Tung
Venue: Submitted to ICLR 2023 (2023)
Keywords: Visual Intuitive Physics
Neural Implicit Representations
Graph Neural Networks
Learning-Based Dynamics Modeling
Particle-Based Dynamics

Abstract: Given a visual scene, humans have strong intuitions about how a scene can evolve over time under given actions. The intuition, often termed visual intuitive physics, is a critical ability that allows us to make effective plans to manipulate the scene to achieve desired outcomes without relying on extensive trial and error. In this paper, we present a framework capable of learning 3D-grounded visual intuitive physics models purely from unlabeled images. Our method is composed of a conditional Neural Radiance Field (NeRF)-style visual frontend and a 3D point-based dynamics prediction backend, in which we impose strong relational and structural inductive bias to capture the structure of the underlying environment. Unlike existing intuitive point-based dynamics works that rely on the supervision of dense point trajectory from simulators, we relax the requirements and only assume access to multi-view RGB images and (imperfect) instance masks. This enables the proposed model to handle scenarios where accurate point estimation and tracking are hard or impossible. We evaluate the models on three challenging scenarios involving fluid, granular materials, and rigid objects, where standard detection and tracking methods are not applicable. We show our model can make long-horizon future predictions by learning from raw images and significantly outperforms models that do not employ an explicit 3D representation space. We also show that, once trained, our model can achieve strong generalization in complex scenarios under extrapolate settings.

Under review as a conference paper at ICLR 2023
3D-INTPHYS: LEARNING 3D VISUAL INTUITIVE PHYSICS FOR FLUIDS, RIGID BODIES, AND GRANU- LAR MATERIALS
Anonymous authors Paper under double-blind review
ABSTRACT
Given visual scene, humans have strong intuitions about how scene evolve over time under given actions. The intuition, often termed visual intuitive physics, is critical ability that allows to achieve desired outcomes without relying on extensive trial and error. In this paper, we present framework capable of learning 3D-grounded visual intuitive physics models purely from unlabeled images. Our method is composed of a conditional Neural Radiance Field (NeRF)-style visual frontend and a 3D point- based dynamics prediction backend, in which we impose strong relational and structural inductive bias to capture the structure of the underlying environment. Unlike existing intuitive point-based dynamics works that rely on the supervision of dense point trajectory from simulators, we relax the requirements and only assume access to multi-view RGB images and (imperfect) instance masks. This enables the proposed model to handle scenarios where accurate point estimation and tracking are hard impossible. We evaluate the models on three challenging scenarios involving fluid, granular materials, and rigid objects, where standard detection and tracking methods are not applicable. We show our model can make long-horizon future predictions by learning from raw images and significantly outperforms models that do not employ an explicit 3D representation space. We also show that, once trained, our model can achieve strong generalization in complex scenarios under extrapolate settings.
make effective plans manipulate the scene
INTRODUCTION
Humans can achieve strong intuitive understanding of the 3D physical world around us simply from visual perception (Baillargeon et al., 1985; Battaglia et al., 2013; Spelke, 1990; Smith et al., 2019; Sanborn et al., 2013; Carey & Xu, 2001). As we constantly make physical interactions with the environment, the intuitive physical understanding applies to objects of wide variety of materials (Bates et al., 2018; Ullman et al., 2019). For example, after watching videos of water pouring and doing the task ourselves, we can develop mental model of the interaction process and predict how the water will move when we apply actions like tilting or shaking the cup (Figure 1). The ability to predict the future evolution of the physical environment is extremely useful for humans to plan our behavior and perform everyday manipulation tasks. It is thus desirable to develop computational tools that learn 3D-grounded models of the world purely from visual observations that can generalize to objects with complicated physical properties like fluid and granular materials.
There has been series of works on learning intuitive physics models of the environment from data. However, most existing work either focuses on 2D environments (Watter et al., 2015; Agrawal et al., 2016; Fragkiadaki et al., 2016; Xu et al., 2019; Finn Levine, 2017: Babaeizadeh et al., 2021; Qi
et al., 2021; Kipf et al., 2019; Ye et al., 20; Hafner et al., 2019b;a; Schrittwieser et al., 2020; Li
et al., 2016; Finn et al., 2016; Lerer et al., 2016; Veerapaneni et al., 2019; Girdhar et al., 2020; Chang
et al., 2016; Xue et al., 2016) or has make strong assumptions about the accessible information of the underlying environment (Li et al., 2019a; 2020;
et al., 2020; Pfaff et al., 2021; Zhang al., 2016; Tacchetti et al.. 2018; Sanchez-Gonzalez al., 2018: Battaglia al., 2018; Ajay et al., 2019; Janner et al., 2019a) (c.g., full-state information of the fluids represented as points). The limitations prevent their use in tasks requiring explicit 3D understanding of the environments and

---

Under review as a conference paper at ICLR 2023
Old Scene
3D-Aware Intuitive Physics
Old Scene
Old Scene
New Scene
Figure 1: Visual Intuitive Physics Grounded in 3D Space. Humans have strong intuitive understanding of the physical environment. We can predict how the environment would evolve when applying specific actions. This ability roots in our understanding of 3D and applics to objects of diverse materials, which is essential when planning our behavior to achieve specific goals. In this work, we leverage combination of implicit neural representation and particle representation to build 3D-grounded visual intuitive physics models of the world that applies objects with complicated physical properties, such fluids, rigid objects, and granular materials.
make it hard to extend to more complicated real-world environments where only visual observations are available. There are works aiming to address this issue by learning -D-grrunddd representation of the environment and modeling the dynamics in a latent vector space (Li et al., 202 However, these models typically encode the entire scene into one single vector. Such design does not capture the structure of the underlying systems, limiting its generalization compositional systems systems of different sizes (e.g., unseen container shapes or different numbers of floating ice cubes).
In this work, propose 3D Visual Intuitive Physics (3D-IntPhys), framework that learns intuitive physics models of the environment with explicit 3D and compositional structures, purely from visual observations. Specifically, the model consists of (1) a perception module based conditional Neural Radiance Fields (NeRF) (Mildenhall et al., 2020; Yu et al., 2021) that transforms the input images and instance masks into 3D point representations and (2) a dynamics module instantiated as graph neural networks to model the interactions between the points and predict their evolutions over time. Despite advances in graph- based dynamics networks (Sanchez-Gonae
et al., 2020; et al., 201 20),,), existing methods require strong supervision provided by 3D GT point trajectories, which are hard to obtain in most real setups. To tackle the problem, we train the dynamics model using (1) a distribution- based loss function measuring the difference between the predicted point sets and the actual point distributions at the future timesteps and (2) a spacing loss to avoid degenerated point set predictions. Our perception module learns ceatial-qquivarian representations of the environment grounded in the 3D space, which then transforms into points as a flexible representation to describe the system's is state. Our dynamics module regards the point set graph and exploits the compositional structure of the point systems. The structures allow the model to capture the compositionality of the underlying environment, handle systems involving objects with complicated physical properties (e.g., fluid and granular materials), and perform extrapolated generalization, which we show via experiments greatly outperform various baselines without a structured 3D representation space.
2 RELATED WORK
Visual dynamics learning. Existing works learn predict object motions from pixels using frame- centric features (Agrawal et al., 2016; Finn & Levine, 2017: Babaeizadeh et al., 2021; Hafner et al., 2019b;a; Suh & Tedrake, 2020; Lee et al., 2018: Vondrick et al., 2015; Zhang et al., 2018; Burda et al., 2019; Hafner et al., 2019c; Wu et al., 2021) object-centric features (Fragkiadaki et al., 2016; Watters et al., 2017: Kipf et al., 2019; Qi et al., 2021; Janner et al., 2019b; Veerapaneni et al., 2019; Ding et al., 2020; Girdhar et al., 2020; Riochet et al., 2020; Ye et al., 2019a), yet, most works only demonstrate the learning in 2D scenes with objects moving only on a 2D plane. We argue that one reason that makes it hard for these existing methods to be applied to general 3D visual scenes is because they often operate on view- niew-dependent features that can change dramatically due to changes in the camera viewpoint, which shouldn't i have any effect the actual motion of the objects. Recent
2

---

Under review as a conference paper at ICLR 2023
Input
Viewpaints
Dynamics Prediction Particle Space
X6
x6
x6
3D DD-Aware Perceptiio
Encodde
Encoder
Encoder
Griph-Based Dynamics Predictor
(e, 0) = f(x, d, W(a(xr))
xN
Rendering
Loss
Objoct PointCloud
Sparse Sampling
Instance Segmentation
Instance PointCleud
Merge Loss
Chamfer Loss
Long Time Rolling (Test Time)
Figure 2: Overview of 3D Visual Intuitive Physies (3D-IntPhys). Our model consists of two major compo- nents: Left: The pereeption module maps the visual observations into implicit neural representations of the environment. We then subsample from the reconstructed implicit volume obtain a particle representation of environment. Right: The dynamics module, instantiated as graph neural networks, models the interaction within and between the objects and predicts the evolution of the particle set.
works by Bear et al. (2021) have shown that only methods that use view-invariant representations pave the way toward human-level physics dynamics prediction in diverse scenarios.
Researchers have attempted to learn object motion in 3D (Tung et al., 2020; Xu al., 2020; Manuelli et al., 2020; Li al., 2021b). Tung et al. (2020) and Xu et al. (2020) use object-centric volumetric representations inferred from RGB- D to predict object motion, yet, these volumetric approaches have much higher computation costs than 2D methods due the 4D representation bottleneck, which hinders them from scaling up to more complex scenes. Manuelli et al. (2020) use self-supervised 3D implicit representations model multi-object dynamics
keypoints and Driess et al. (2022a:b) but cannot handle objects with high degrees of freedom like fluid and granular materials. Li et al. (2021b) use neural implicit representation reduce the potential computational cost, yet the works have not shown how the approach can generalize to unseen scenarios. Our works aim to solve the tasks of learning generalizable object dynamics 3D by combining the generalization strength of input-feature -ofaatuuurecncitonnneee implicit representation and point-based dynamics models.
Point-baseds dynamics models. Existing works in point- and mesh-based dynamics models (Li et al., 2019a; Mrowca et al., 2018; Ummenhofer et al., 2019;
Gonzallz et al., 2020; Pfaff al., 2021) have shown impressive results predicting the dynamics of rigid objects, fluid (Li et al., 2019a; Mrowca et al., 2018; Ummenhofer et al., 2019; Sanchez-Gonzalez
al., 2020; Allen et al., 2022; de Avila Belbute-Peres et al., 2020), deformable objects (Li et al., 2019a; Mrowca et al., 2018; Manchez-Gonzate et al., 2020), and clothes (Pfaff et al., 2021; Lin al., 2021). Most works require access to full 3D states of the points during training and testing, yet, such information is usually not accessible in real-world setup. Li et al. (2020) learn visual frontend to infer 3D point states from images, but still require 3D point states and trajectories during training time. Shi et al. (2022) propose to learn point dynamics directly from vision, but they only consider elasto-plastic objects consisting of homogeneous materials. How to learn about 3D point states and their motion from raw pixels remain a question. Our paper tries build the link from pixels to points using recent advances in unsupervised 3D inference from images using NeRF (Mildenhall et al., 2020; Yu et al., 2021).
3 METHODS
We present 3D Visual Intuitive Physies (3D-IntPhys), a model that learns to simulate physical events from unlabeled images (Figure 2). 3D-IntPhys contains perception module that transforms visual observations into 3D point cloud that captures the object geometries (Section 3. and point-based simulator that learns to simulate the rollout trajectories of the points (Section 3.2). The design choice of learning physics simulation in a 3D 3D-point representation space enables stronger simulation performance and generalization ability. The performance gain mainly comes from the fact that describing/learning objects' motion and interactions in 3D are easier compared to doing so in 2D since objects live and move persistently in the 3D space. ID-Intss also supports better
3

---

Under review as a conference paper at ICLR 2023
generalization ability since its neural architecture explicitly models how local geometries of two objects/parts interact, and these local geometries and interactions can be shared across different and novel object combinations.
Although 3D-IntPhys learns to simulate in a 3D representation space, we show it can learn without any 3D supervision such as dense point trajectories as in previous work (Sanchez-Gonzal 2020; Li et al., 2019a). Dense point trajectories are hard and sometimes impossible to obtain in the real world, e.g., capturing the trajectories of each water point. 3D-IntPhys does not require such 3D supervision and can simply learn by observing videos of the scene evolution.
et al.,
3.1 2D-To-3D PERCEPTION MODULE
Given static scene, the perception module learns to transform or few posed RGB images, {(I,, mi E (1,2...... ............ N. taken from N, different views, into a 3D point cloud representation of the scene, X. We train the model in an unsupervised manner through view reconstruction, using dataset consisting of N, videos, where each video has Ny frames, and each frame contains images taken from Ny viewpoints.
Neural Radiance Field (NeRF). NeRF (Mildenhall et al., 2020) learns to reconstruct a volumetric radiance field of scene from unlabeled multi-view images. After training. the model learns predict the RGB color с and the corresponding density a of a query 3D point x E IR' from the viewing direction R33 with a function (c. f(x, d). We can formulate a camera ray as r(t) o + td, where o E R is the origin of the ray. The volumetric radiance field can then be rendered into a 2D image via C(r)= f ((t) where T(t) exp(- Sr (s)ds) handles occlusion. The rendering range is controlled by the depths of the near and far plane (i.e., tn and ty). We can train NeRF through view prediction by:
C Σ (ER(P)
C((r) - C(r)
where R(p) is the set of camera rays sampled from target camera pose p.
Image-conditioned NeRF To infer the NeRF function from an image, previous work proposed to encode the imput image into vector, with a CNN encoder, function (Li et al., 2021b). We found this type of architecture in general hard to train and does not generalize well. Instead, we adopt pixelNeRF (Yu et al., 2021), which conditions NeRF rendering with local features, as opposed to global features. Given an image / in a scene, pixelNeRF first extracts a feature volume using CNN encoder W E(I). For a point x in the world coordinate, we retrieve its feature vector by projecting it onto the image plane, so that we can get the feature vector W(T(x)). PixeINeRF combines the feature vector together with the 3D position of that point and predict the RGB color and density information:
conditioning input to the target NeRF
V(x) = (c. = f(x, d. W(x(x))).
PixeINeRF can also incorporate multiple views to improve predictions when more input views are available; this will help decrease ambiguity caused by occlusion and will greatly help us get better visual perceptions of the target scene.
3D point representation from pixelNeRF. From a few posed RGB images, I, of scene s, we infer a set of points for O₂ target object (such as fluid, cube) in the scene. We achieve this by first sampling set of points according to the predicted occupancy measure, then clustering the points into objects using object segmentations. We found that sampling with low resolution will hurt the quality of the rendered point cloud to generate objects with inaccurate shapes, while sampling with high resolution will increase the computation for training the dynamics model since the input size increases. To speed up training while maintaining the quality of the reconstructed point cloud, we first infer the points with higher resolution and do sparse sampling of each point cloud using FPS (Farthest Point Sampling) (Eldar et al., 1997). Next, we cluster the inferred points into objects according to object segmentation masks. Since solving object segmentation in general is not the main focus of this paper, we resort to using the color information to obtain the masks.
3.2 POINT-BASED DYNAMICS SIMULATOR
Given the point representation at the current time step, X,, the dynamics simulator predicts the points' evolution T steps in the future,
... X.+r), using graph- based networks (Sanchez- Gonzalez al., 2020; Li et al., 2019a). 9a). We first form graph (V, E)) based on the distance between
4

---

Under review as a conference paper at ICLR 2023
points. If the distance between two points is smaller than threshold 6, we include edge between these two points. Each vertex 8, = (#a, af') E V contains the velocity of the point, zy. and point attributes, a,, to indicate the point's 's type. For each relation, (i.j) E, we have its associated relation attribute d..- indicating the types of relation and the relative distance between the connected points.
Spatial message passing and propagation. At time step f, can do message passing update the points, v/ E V, and relation representations, (i,j) E E, in the graph:
Iij. = Qe(U,,, (peraa))
(i,j) E E
(3)
hi, = QU( EREfjRLNEEE Ia.r)
0, E
(4)
where Qu and Qe are encoders for vertices and relations respectively. Please refer to Battaglia et al. (2018) for more details. Though this kind of message passing can help with updating representation, can only share one-hop information in each step, limiting its performance on instantaneous passing of forces. To improve long-range instantaneous effect propagation, we use multi-step step message propagation as in Li al. (2019b:a). Starting from propagation step 0 with an initialization step where h², = hig for all vertices and Fiy Dij. for all edges, we do L steps of propagation by
Step I E {1,2, .... L}}
Ln Pe 1-1 hip Ma7a)
(i,)) E
(5)
h1, = P. Eneulisee
(R,)) VEV
(6)
where Pe, P. are propagation functions of nodes and edges, respectively, and Jnz is the effect of relation (i, propagation step I. k4, the hidden states for each point in the propagation process. Finally, we have the predicted states of points at time step + after L steps of propagation:
0,,c+1 fpeed (h(,))
Environments. We assume that the surrounding environment (e.g.. the table) known and the robot/tool/comtainer are of known shape and fully actuated. where the model has access to their complete 3D state information. We convert the full 3D states into points through sampling on the meshes and include these points in the prediction of the graph-based dynamics.
Fluids, rigid bodies, and granular materials. We distinguish different materials by using different point attributes a.. We also set different relation attributes af in Equation to distinguish different interaction (e.g., Rigid-Fluids, Fluids-Fludds, Granular-Pusher For rigid objects, to ensure the object shapes remain consistent throughout the rollout predictions, add differentiable rigid constraint in the prediction head following Li et al. (2019a).
Training dynamics model without point-level correspondence. Since our perception model parses each RGB image into object-centrie point clouds independently, there does not exist an explicit one-to-one correspondence for points across frames. To handle this, measure the Chamfer distance between the prediction X, (Ve, E.) from the dynamics network and the inferred point state X, = (V,, E,) from the perception module and treat it as the objective function. The Chamfer distance between two point cloud V and V is defined
(V V)
Σ
min | -lll=+11111 NEV
min ||x Σ
v12
IV
zEV VEV
FEV
We found that training the model with Chamfer distance dense scenes with granular materials will often lead to predictions with unevenly distributed points where some points stick too close to each other. To alleviate this issue, we further introduce a spacing loss L,, which penalizes the gated distance (gated by dmin) of nearest neighbor of each point ensure enough space between points:
= -(V)=E
Der (ReLU(dmma
min ||el - =
v'E(WW
The one-step prediction loss Lay for training the dynamics model is L. V) + (VV where o reweights the second loss. To improve long-term rollout accuracy, we train the model with two-step predictions using the first predicted state as input and feed it back into the model to generate the second predicted state. With the two-step loss, the model becomes more robust to errors generated from its own prediction. Finally, the Lay losses for all rolling steps are summed up to get the final loss for this trajectory. More implementation details are included in the supplementary material.
5

---

Under review as a conference paper at ICLR 2023
Scene Trajectories
FluidPour
FluidCubeShake
GranularPush
Interpolate pate
Camera
Camera
Camera
Extrapolate
Camera
Camera
Diff. container & more fluid
More fluid & cubes
More granular media
Figure 3: Data Collection and Evaluation Setups. Left: We collect multi- videos of the environment from six cameras. Right: We consider a diverse set of evaluating environments involving fluids, rigid objects, granular materials, and their interactions with the fully-actuated container and the environment. We evaluate the learned visual intuitive physics model on both the interpolated settings (i.c., seen enviromment hut with different action sequences) and extrapolated settings (i.e., unseen environment with different amounts fluids, cubes. granular pieces, and containers of different sizes).
EXPERIMENTS
The experiment section aims answer the following three questions. (1) How well the visual inference module capture the content of the environment (i.c., can we use the learned representations reconstruct the scene)? (2) How well does the proposed framework perform in scenes with objects of complicated physical properties (e.g., fluids, rigid and granular objects) compared to baselines without explicit 3D representations? (3) How well do the models generalize extrapolate scenarios?
Datasets. We generated three simulated datasets using the physics simulator Nvidia FleX (Macklin al., 2014). Each of the datasets represents one specific kind of manipulation scenario, where robot arm interacts with rigid, fluid, and granular objects (Figure 3). For each of the three scenarios, we apply randomized input actions and change some properties of objects the scene, e.g., the shape of the container, the amount of water, and the color/number of cubes, to make it diverse. To test the generalization capability of the trained model, we design extrapolated datasets where the data generated from extrapolated set of parameters outside the training distribution.
a) FluidPour. This scenario contains a fully- selly-acttated cup pouring fluid into a container. We design the extrapolate dataset to have a larger container, more quantity of fluid, and different pouring actions.
b) FluidCueeShake. This scenario contains a fully- fully-actuated container that moves on top table. Inside the container fluids and cubes with diverse colors. We design the extrapolate dataset to have different container shapes, number of cubes, cube colors, and different shaking actions.
GranularPush. This environment contains fully-actuated board pushing pile of granular pieces. We design the extrapolate dataset to have a larger quantity of granular objects in the scene than the model has ever during training-
Baselines. We compare our method with two baselines, NeRF-dy (Li et al., 2021b) and autoencoder (AE) (similar to GQN (Eslami et al., 2018) augmented with a latent-space dynamics model). NeRF-dy is a 3D-aware framework that also learns intuitive physics from multi-view videos. Yet, instead of learning the object dynamics with explicit and compositional 3D representations, the model learns dynamics models with implicit 3D representations in the form of single latent vector. We also compare our method with an autoencoder- based reconstruction model (AE) (Eslami et al., 2018) that can perform novel-view synthesis but is worse at handling 3D transformations than neural implicit representations. AE first learns scene representations through per-frame image reconstruction, and then it learns dynamics model top of the learned latent representations. All methods take RGB images and camera parameters as inputs. To incorporate object-level information, we perform color-based segmentation obtain object masks as additional inputs to the baselines.
Implementation details. We train and test the perception module with 6 camera views. To obtain set of points from the learned perception module, sample points a 40 x 40 x 40 grid from an area 55cm x 55cm x 35em at the center of the table for FluidPour, 63cm x 63cm x 63cm with 40 40 40 grids for FluidCubeShake, and on 70 70 70 grid area for GranularPush.

---

Under review as a conference paper at ICLR 2023
FluidPour
FluidCuheShake
GranularPush
Metrics
Model
InD
OoD
InD
OoD
InD
OoD
AE
451.03
542.86
869.3
1727.55
562.06
1537.2
MSE(L)
NeRF-dy
202.95
317.27
527.46
1585.97
481.95
1020.0
Ours
111.66
124.33
66.52
81.38
147.97
646.85
AE 0.86
0.84
0.71
0.86
0.81
0.62
SSIM(t))
NcRF-dy 0.89
0.86
0.73
0.65
0.81
0.61
Ours 0.90
0.89
0.94
0.93
0.89
0.69
Table 1: Quantitative Results of the Perception Module. We compare method with autoencoder (AE) and NeRF-dy (Li al.. 2021b) with additional instance masks based color. We measure the quality rendered images by computing the Mean Squared Error (MSE) and Structural Similarity Index Measure (SSIM) compared to the ground truth. InD stands for in-distribution tests, and OoD stands for out-of-distribttion tests.
Interpolate
Extrapolate
Interpolate
Extrapolate
Interpolate
Extrapolate
Ground Truth
Autoencoder
NeRF-dy
Ours
(a) FluidPour
(b) FluidCubeShake
(c) GranularPush
Figure 4: Qualitative Reconstruction Results Perception Module. The images generated by our method contain more visual details and are much better aligned with the ground truth. Our model is much better handling large scene variations than NeRF-dy, especially in extrapolate settings.
We evaluate and include points with density (measured by the occupancy in the predicted neural radiance fields) larger than 0.99. We subsample the inferred points with FPS with ratio of around 5% for FluidPour and 10% for FluidCubeShake and GranularPush. We currently determine the sampling rate by manually picking the minimum rate that yields a reasonable point cloud describing the object shapes. The threshold dmin is set 0.08 and the weighting a is set to 10. We select the distance so that each point will have on average 20 to 30 neighbors. Details for data generation parameters, model architecture, training schema, and more data samples are included
supplementary material.
4.1 IMAGE RECONSTRUCTION FROM LEARNED SCENE REPRESENTATIONS
We test how well the perception modules capture scene information by evaluating the visual front-end of models on their ability to reconstruct the observed scene from the inferred representations. We measure the difference between the reconstructed and ground truth images with Mean Squared Error (MSE) and Structural Similarity (SSIM) pixel level (TTble 1). Our perception module outperforms all baselines in all three environments. The performance gap is exaggerated in extrapolate settings, especially in scenarios that involve complex interactions between rigid and deformable materials (see Figure for qualitative comparisons).

---

Under review as a conference paper at ICLR 2023
Pour
Shake
Push
metted
cor mith
Interpolate
2.33
032
2:13
1.43
0.00
0.3
1:4
122
refdy
rethad
1:4
Extrapolate
2:33
032
0.0
2.32
2.59
2.00
242
10
Figure 5: Quantitative Results of the Dynamics Module. This figure compares our method and NeRF-dy (Li
al., 2021b) their long- horizon open-loop future prediction performance. The loss is measured the
Chamfer distance between the predicted particle evolution and the actual future. Our methad outperforms the baseline in both interpolate and extrapolate settings, showing the benefits of explicit 3D modeling.
4.2 LEARNED VISUAL DYNAMICS ON IN- IN-DISTRIBUTION HELD-OUT SCENES
Next, compare long-term rollouts in the 3D space. We evaluate the models using the Chamfer distance between the predicted point cloud and the ground truth. For NeRF-dy, we decode the predicted rollouts latent vectors into the point cloud with the learned NeRF decoder. We exclude the comparison with AE since is unclear how to decode the learned representations into point clouds. We show quantitative comparison in Figure and qualitative results in Figure 6. 3D-IntPhys can learn reasonable scene dynamics in all scenarios and significantly outperforms NeRF-dy. While NeRF-dy can learn relatively reasonable movements of fluids, it fails to learn complex dynamics such as the floating cube and the morphing of the granular materials. The results suggest that the proposed explicit 3D point-based representations are critical learning complex multi-material dynamics.
4.3 GENERALIZATION ON OUT-OF-DISTRIBUTIO SCENES
To test the generalization ability of the models, we introduce extrapolate settings of all of the three scenarios. See "Extrapolate" results in Table 1, Figure and 6. The proposed 3D-IntPhys gener- alizes well to extrapolate settings both at the visual perception stage and the dynamics prediction stage, whereas NeRF-dy and autoencoder both fail at generalizing under extrapolate settings. For example, in FluidShake, both baselines cannot capture the number and the color of the rigid cubes (Figure 4). And in GranularPush, both baselines fail to capture the distributions of the granular materials. NeRF-dy performs much worse extrapolation scenes compared to in-distribution scenes, suggesting that incorporating 3D information in an explicit way, as opposed to implicit, is much better at capturing the structure of the underlying environment, thus leading to better generalization. We further test our model on completely unseen changes to the environment - in the GranularPush environment, we extend the width of the pusher by factor of and 5. Though the stretched pusher has never shown in the training data, our model can make reasonable pushing predictions (see Fig 7).
CONCLUSIONS
In this work, we propose 3D-aware and compositional framework, 3D-IntPhys, to learn intuitive physics from unlabeled visual inputs. Our framework can work on complex scenes involving fluid, rigid objects, and granular materials, and generalize to unseen scenes with containers of different sizes, more objects, or larger quantities of fluids and granular pieces. We show the proposed model outperforms baselines by a large margin, highlighting the importance of learning dynamics models in an explicit 3D representations space. One major limitation of our work is the assumption of the access to object masks. Although the masks do not have to be perfect and there has been significant progress on (self- Jsupervised object segmentation in recent years, it still be hard to obtain these masks in more complicated real-world settings. An exciting future direction is to learn the segmentation and material properties jointly with the dynamics from the observation data.
8

---

Under review as a conference paper at ICLR 2023
Interpolate
Extrapolate
NeRF-dy Ground Truth
Ours
GT
Ours NoRF-dy
NeRF-dy Ground Truth
Ours
Figure 6: Qualitative Results of the Dynamics Module on Future Prediction. Here we visualize our model's 's predicted future evolution of the particle set compared with the NeRF-dy (Li et al., 2021b) baseline in both interpolate and extrapolate settings. Our method correctly identifies the shape/distrihttion of the fluids, rigid objects, granular pieces with much better accuracy than NeRF-dy. future evolution predicted by our method also matches the ground truth much hetter and produces reasonable results even in extrapolate settings.
Particle-View
Top-View
Width 2
Width 5
Figure 7: Strong Generalization Ability of the Dynamics Module to Wider Pushers. We evaluate dynamics model on unseen width of pushers GranularPush environment. The left part shows point space where red indicates granular materials, green shows table and pusher, and the arrow shows how the pusher is about move. The right part shows from top view of rendering results.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Pulkit Agrawal, Ashvin V Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. Advances in neural information processing systems, 29, 2016.
Anurag Ajay, Maria Bauzá, Jiajun Wu, Nima Fazeli, Joshua B. Tenenbaum, Alberto Rodriguez, and Leslie Pack Kaelbling. Combining physical simulators and object-based networks for control. CoRR, abs/ Abs//994.00680, 2019. URL httt:://aaxiiiii org/ org/abs/ 1904. 06580.
Kelsey R. Allen, Tatiana Lopez-Guevane Kimberly L. Stachenfeld, Alvaro
Peter W. Battaglia, Jessica B. Hamrick, and Tobias Pfaff. Physical design using differentiable learned simulators. CoRR, abs/2202.00728,, 2022. URL h 00728.
http://awwwwccgggaaa//// wwtpvv..rr/aaaaaaaaeee 2202..
Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan. Fitvid: Overfitting in pixel-level video prediction. CoRR, abs/2106.13199 2021.
URL http:::t//s/eett /ttp///avvvvv//v////a//eeeeeawv/ org/ abs/2106 2206. 13195.
Renée Baillargeon, Elizabeth S. Spelke, and Stanley Wasserman. Object permanence in five-month- old infants. Cognition, 20:191-200 208, 1985.
Christopher Bates, Ilker Yildirim, Joshua B. Tenenbaum, and Peter W. Battaglia. Modeling human intuitions about liquid flow with particle-basee simulation. CoRR, abs/1 0111109.0011 809.01524, 2018. URL http::/arxet org/ abs 1809.8800 01524.
Peter W. Battaglia, Jessica B. Hamrick, and Joshua B. Tenenbaum. Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences, 110:18327 18332, 2013.
Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro
Vinícius Flores Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Çaglar Gülçehre, H. Francis Song, Andrew J. Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey R. Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matthew M. Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. CoRR, abs/l 888.00261, 2018. URL http: /ttt:://axivve arxiv. org/ awwaaaasssss 01261.
Daniel M Bear, Blias Wang, Damian Mrowca, Felix Binder, Hsiau-Yi Yu Fish Tung, RT Pramod, Cameron Holdaway. Sirui Tao, Kevin Smith, Li Fei-Fei, et al. Physion: Evaluating physical prediction from vision humans and machines. arXiv preprint arXiv:2106.08261 2021.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A. Efros. Large-scale study auriosity-drivent learning. In ICLR, 2019.
Susan Carey and Fei Xu. Infants' knowledge of objects: beyond object files and object tracking. Cognition, 1791)119-2113 2001. ISSN 0010-0277. doi: https://doi.rr/// 10166800100227700000155442 URL attps://wwe http://www.cie.eedi....e sciencedirect. www..ccoo/sciiiie Objects and Attention.
Michael B. Chang. Tomer D. Ullman, Antonio Torralba, and Joshua Tenenbaum. compositional object-based approach to learning physical dynamics. CoRR, abs/1612.00341, 2016. URL
http:://rxixl org abs/ 1612.661110 11111.0111....... 00341.
Filipe de Avila Belbute-Pere
Thomas D. Economon, and J. Zico Kolter. Combining differentiable PDE solvers and graph neural networks for fluid flow prediction. CoRR, abs/2007 0bbb/2007.0449,, 2020. URL wwtpthttp///// /ttp//axiii// org/ /bgg/bs///0 abs/2201 04439.
David Ding, Felix Hill, Adam Santoro, and Matt M. Botvinick. Object-based attention for spatio- temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architectures. CoRR, 0as/2012.08500 2020. URL https: /ttess///reeeeee /arxivvvv.vvv www.viooo//vv...ee...eoooooo abs /2 08508.
10

---

Under review as a conference paper at ICLR 2023
Danny Driess, Jung-Su Su Ha, Marc Toussaint, and Russ Tedrake. Learning models as functionals of signed-distance fields for manipulation planning. In Conference on Robot Learning, pp. 245-255. PMLR. 2022a.
Danny Driess, Zhiao Huang. Yunzhu Li, Russ Tedrake, and Mare Toussaint. Learning multi-object dynamics with compositional neural radiance fields. arXiv preprint urXiv:2202. /8855, 2022b.
Y. Eldar, M. Lindenbaum, M. Porat, and Y.Y. Zeevi. The farthest point strategy for progressive image sampling. IEEE Transactions on Image Processing, 6(9):1305-135,, 1997. doi: 10. 109/83. 623193.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 360(6394)
2018.
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp- 2786 2793. IEEE, 2017.
Chelsea Finn, lan Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. CoRR, abs/1605.071177, 2016. URL http://arxiv...amm///////// org/ abs// 1605. 07157.
Katerina Fragkiadaki, Pulkit Agrawal, Sergey Levine, and Jitendra Malik. Learning visual predictive models of physics for playing billiards. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://///// aaaaaaiivvvv.....aaaaaaaeaaa. org. 1111. 07404.
Rohit Girdhar, Laura Gustafson, Aaron Adcock, and Laurens van der Maaten. Forward prediction for physical reasoning. CoRR, abs/2006.10734, 2020. URL httpt: battps://arxiv.crttt 10734.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01660 2019a.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pp. 2555-2565. PMLR, 2019b.
Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream control: Learning behaviors by latent imagination. CoRR, abs/1912.01603, 2019c. URL //axxiv.. org/ /bs//bb/// 1912.1 01 603.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine, When to trust your model: Model-based policy optimization. CoRR, 08253.
2019a. URL http::/ttttttttteee http///axxvvvv/aa//////eeeee.aav org brg//bb/ /1906.
Michael Janner, Sergey Levine, William T. Freeman, Joshua B. Tenenbaum, Chelsea Finn, and Jiajun Wu. Reasoning about physical interactions with object-oriented prediction and planning. In International Conference on Learning Representations, 2019b.
Thomas Kipf, Elise der Pol, and Max Welling. Contrastive learning of structured world models.
arXiv preprint 1911v:1911.122477 2019.
Alex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. CoRR, abs/1804.01523. 2018. URL http:://aaxti org/abs a100 01523.
Adam Lerer, Sam Gross, and Rob Fergus. Learning physical intuition block towers by example. CoRR, abs/ 1b0//0033,,,12,, 2016. URL http::/arxiv. wtgg/bb//////e /6003, 01312.
Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, and Zhaoyang Lv. Neural video synthesis. arXiv preprint 203 2021a.
11

---

Under review as a conference paper at ICLR 2023
Wenbin Li, Seyedmajid Azimi, Ales Leonardis, and Mario Fritz. To fall not to fall: A visual approach to physical stability prediction. CoRR, abs/1604.00066, 2016. URL http:://axixi org/abs abs//bb//a 600 00066.
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In ICLR, 2019a.
Yunzhu Li, Jiajun Wu, Jun Yan Zhu, Joshua B Tenenbaum, Antonio Torralba, and Russ Tedrake. Propagation networks for model- control under partial observation. In 2019 International Conference on Robotics and Automation (ICRA), pp. 1205-1211, IEFF, 2019h.
Yunzhu Li, Toru Lin, Kexin Yi, Daniel Bear, Daniel L.K. Yamins, Jiajun Wu, Joshua B. Tenenbaum, and Antonio Torralba. Visual grounding of learned physical models. In International Conference Machine Learning, 2020.
Yunzhu Li, Shuang Li, Vincent Sitzmann, Pulkit Agrawal, and Antonio Torralba. 3d neural scene representations for visuomotor control. arXiv preprint (it::2107.04004 2021b.
Xingyu Lin, Yufei Wang, Zixuan Huang, and David Held. Learning visible connectivity dynamics for cloth smoothing. In Conference on Robot Learning, 2021.
Miles Macklin, Matthias Müller, Nuttapong Chentanez, and Tae- Tae-Yong Kim. Unified particle physics for real-time applications. ACM Transactions on Graphics (TOG), 33(4):1-12, 2014.
Lucas Manuelli, Yunzhu Li, Pete Florence, and Russ Tedrake. Keypoints into the future: Self-supervised correspondence in model-based reinforcement learning. arXiv preprint arXiv:2009.05085, 2020.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pp. 405 421. Springer, 2020.
Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li Fei-Fei, Joshua B. Tenenbaum, and Daniel L. K. Yamins. Flexible neural representation for physics prediction. CoRR, 0aw/1806.08047 2018. URL
org/ /1806. C8047.
Tobias Pfaff, Meire Fortunato, Alvaro Gonzalz and Peter W. Battaglia. Learning mesh- based simulation with graph networks. In International Conference on Learning Representations, 2021.
Haozhi Qi, Xiaolong Wang, Deepak Pathak, Yi Ma, and Jitendra Malik. Learning long-term visual dynamics with region proposal interaction networks. In ICLR, 2021.
Ronan Riochet, Josef Sivic, Ivan Laptev, and Emmanuel Dupoux. Occlusion resistant learning of intuitive physics from videos. CoRR, abs/2005.00069, 2020. URL https: //tpxx/xxxvv.rr////// abs/2005. 00069.
Adam N. Sanborn, Vikash K. Mansinghka, and Thomas L. Griffiths. Reconciling intuitive physics and newtonian mechanics for colliding objects. Psychological review, 120 2:41 1--7, 2013.
Alvaro Sanzhez-Gonzalez, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin A. Ried- miller, Raia Hadsell, and Peter W. Battaglia. Graph networks as learnable physics engines for inference and control. CoRR, abs/1806.01242, 2018. URL h htt.tp:://axxi..or///abb.ts/ 01242.
Alvaro Lanchez-Gonzalee Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec. and Peter Battaglia. Learning simulate complex physics with graph networks. In International Conference on Machine Learning, pp. 8459- 8468. PMLR, 2020.
Julian Schrittwieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with leamed model. Nature, 588(7839):604-600) 2020.
12

---

Under review as a conference paper at ICLR 2023
Haochen Shi, Huazhe Xu, Zhiao Huang, Yunzhu Li, and Jiajun Wu. Robocraft: Learning see. simulate, and shape Ulasto-plastic objects with graph networks. arXiv preprint arXiv:2005.02909, 2022.
Kevin Smith, Lingjie Mei, Shunyu Yao, Jiajun Wu, Elizabeth Spelke, Josh Tenenbaum, and Tomer Ullman. Modeling expectation violation in intuitive physics with coarse probabilistic object representations. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buu Bucc E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. WRLhhttps://prooeeee #88f243bf ded 1
neurips. httpsssccppppee/119////// File// pdf.
Elizabeth Spelke. Principles of object perception. Cognitive Science, 14(1):29-56, 1990. ISSN 0364-0213. doi: http://cc..o////.000000666401110
URL https: /www.
http:/etttccccceccnnenaerrttaeae http:///ccccceeeeeaaaaa///////// sciencedi
HJ Suh and Russ Tedrake. The surprising effectiveness of linear models for visual foresight in object pile manipulation. In International Workshop on the Algorithmic Foundations of Robotics, pp. 347-363. Springer, 2020.
Andrea Tacchetti, H. Francis Song, Pedro A. M. Mediano, Vinícius Flores Zambaldi, Neil C. Ra- binowitz, Thore Graepel, Matthew M. Botvinick, and Peter W. Battaglia. Relational forward models for multi-agent learning. CoRR, abs/1809. 1004, 2018. URL httt:://axxv...or abs/1809 1 11044.
Hsiao-Yo Yu Fish Tung, Zhou Xian, Mihir Prabhudesai, Shamit Lal, and Katerina Fragkiadaki. 3d-oes: Viewpoint-invarian abject-factoriza. environment simulators. arXiv preprint S.Kiv:0011.064666 2020.
Tomer Ullman, Eliza Kosoy, Ilker Yildirim, Amir Arsalan Soltani, Max H Siegel, Josh Tenenbaum, and Elizabeth Spelke. Draping an elephant: Uncovering children's ss reasoning about cloth-covered objects. In Proceedings of the 41st Annual Conference of the Cognitive Science Society, pp. 2019.
Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid simulation with continuous convolutions. International Conference on Learning Representutions, 2019.
Rishi Veerapaneni, John D. Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine, Entity abstraction in visual model-based reinforcement learning. CoRR, abs/1910..1227., 2019. URL http: / aaw.v.iiiivv../v./ org/ abs / 1 1910. 11110.118800 12827.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating the future by watching unlabeled video. CoRR, abs/1504.08823,, 2015. URL http: /utt:://arxiv... www.:iiii.i..iiooooooooooooooooe orq/abs/550. 1504. 08023.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. Advances neural information processing systems, 28, 2015.
Nicholas Watters, Daniel Zoran, Theophane Weber, Peter Battaglia, Razvan Pascanu, and Andrea Tacchetti. Visual interaction networks: Learning a physics simulator from video. Advances in neural information processing systems, 30, 2017.
Bohan Wu, Suraj Nair, Roberto Martín-Martín, Li Fei-Fei, and Chelsea Finn. Greedy hierarchical variational autoencoders for large-scale video prediction. CoRR, abs/2103.04174, 2021. URL
https::/arxias org/ abs 221..11111.000 04174.
Zhenjia Xu, Jiajun Wu, Andy Zeng, Joshua Tenenbaum, and Shuran Song. Densephysnet: Learning dense physical object representations via multi-step dynamic interactions. In Robotics: Science and Systems (RS5), 2019.
Zhenjia Xu, Zhanpeng He, Jiajun Wu, and Shuran Song. Learning 3d dynamic scene representations for robot manipulation. In Conference Robotic Learning (CoRI.), 2020.
13

---

Under review as a conference paper at ICLR 2023
Tianfan Xue, Jiajun Wu, Katherine L Bouman, and William T Freeman. Visual dynamics: Probabilis- tic future frame synthesis via cross convolutional networks. In Advances In Neural Information Processing Systems, 2016.
Yufei Ye, Dhiraj Gandhi, Abhinav Gupta, and Shubham Tulsiani. Object-centric forward modeling for model predictive control. In CoRL, 2019a.
Yufei Ye, Maneesh Singh, Abhinav Gupta, and Shubham Tulsiani. Compositional video prediction. In International Conference on Computer Vision (ICCV), 2019b.
Alex Yu, Vickie Yc, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one few images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 47787457,, 2021.
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew J. Johnson, and Sergey Levine. SOLAR: deep structured latent representations for model-based reinforcement learning. CoRR, abs/1808.0911 2018. URL http: / arxiv. org/ abs /1808. 09105.
Renqiao Zhang, Jiajun Wu, Chengkai Zhang, William T. Freeman, and Joshua B. Tenenbaum. A comparative evaluation of approximate probabilistic simulation and deep neural networks as accounts of human physical scene understanding. CoRR, abs/1605.011 138, 2016. URL http: 01138.
//arxiv.
14

---

Under review as a conference paper at ICLR 2023
A ADDITIONAL RESULTS
To better understand the performance of our framework visually, we prepare test time rollouts of our framework well those of various baselines in the supplementary video.
A.1 ABLATION STUDY
We find that training the model with Chamfer distance in dense scenes with granular materials will often lead predictions with unevenly distributed points where some points stick too close each other. To alleviate the issue, we introduce the spacing loss to penalize the distance between these points. We set the threshold of penalty dmin be 0.08 and the loss weight be 10. We find that spacing loss can help improve the performance of the dynamics learner especially under extrapolate settings, as shown in Figure 8. We provide qualitative results in the supplementary video.
GranularPush Extrapolate
GranularPush Interpolate
0.6
w spacing loss
0.5
wro spacing loss
0.6
Chamier Less
0.4
0.4
0.3
0.2
0.1
0.1
0.0
10
15
20
30
Rolling Step
# Rolling Step
Figure 8: Ablation Study on the Spacing Loss. Training dynamics models in the GranularPush scenario with spacing loss results in better rolling prediction. The performance gap even more substantial in the extrapolate setting.
B IMPLEMENTATION DETAILS
B. DATASET GENERATION
Our datasets are generated by the NVIDIA Flex simulator (Macklin et al., 2014). Each of the three scenarios (Pour, Shake and Push) has 500 videos of trajectories taken from 6 views, with each trajectory consisting of 300 frames. We manually select the 6 views with reasonable coverage of the tabletop space to minimize the occlusion. We have included the camera parameters in OpenGL format in the supplementary zip file (camera. viewmatrix.tx)). The 500 trials generated from five different sets of environmental parameters, detailed in Table 3. We take one set of parameters that are outside the training distribution the extrapolate dataset for evaluating model generalization. For the rest of the four settings, we randomly split them into train and test sets with a ratio 0.8.
Next, provide more details for each scenario:
In the FluidPour environment, we randomly initialize the position of the upper container and then generate random back-and mack-and-forth actions by tilting the container. The action space then the position and tilting angle of the upper container.
FluidCubeShake, we also randomly initialize the position of the container and the cubes inside the container. We then generate random but smooth action sequences moving the container in the 2D plane. The action space is then the x-y location of the container.
GranularPush, we randomly initialize the position of the granular pile. Then, for each push, we randomly generate the starting and ending positions of the pusher and move the pusher along the straight line with an angle perpendicular the pushing direction. The action space four-number tuple stating the starting and ending position on the 2D plane.
The following table shows the moving range of the robot arms the FluidPour and FluidCubeShake environments after normalizing the robot into size that is the same in the real world (unit:
15

---

Under review as a conference paper at ICLR 2023
X- X-Range
Y-Range
Z-Range
FluidPour
-29.11,-12.60
(42.00, 60.00]
[-7.78, 7.78]
FluidCueSShake
[-3.25, 42.25]
[19.25, 19.25]
[-24.50, 24.00]
Table 2: Robot Action Space(centimeeters:: we show the range the robot arms can move in the
FluidPour and FluidCubeShake environments.
Y2
X2
Z2
Z1
Figure 9: Illustration of the Environment Settings. the FluidPour scenario, robot arm holds container and tries to pour some fluid into another container. In the FluidShake scenario, a robot moves a container with some fluid and cubes. We show the parameters for the container shape referred in Table 3.
SceneName
Params
Env1
Env2
Env3
Env4
Extrapolate
X2
0.53
0.53
0.81
0.81
0.81
Y2
0.53
0.81
0.53
0.81
0.81
Z2
1.24
1.24
1.24
1.24
1.24
FluidPour
XI
1.35
1.35
1.35
1.35
1.35
Y1
1.35
1.35
1.35
1.35
1.35
Z1
0.74
0.74
0.74
0.74
0.74
AaountofWater
5125
5125
6125
5375
7625
XI
0.88
0.88
1.32
1.32
1.32
Y1
0.88
1.32
0.88
1.32
1.32
FluidCubeShake
CubeNumber Water 2173
1
2
3322
3322
4858
4983
GranularPush
GranularNumber 2197
4032
5832
9261
12167
Table 3: Scene Parameters for Generating the Interpolate and Extrapolate Datasets. We generate the datasets by varying the shape of container, amount of water, number of cubes, and quantity of the granular material. Zr, X;, Y, are the height, width, and depth for container i. Please refer to Figure 9 for more details.
centimeters). For GranularPush, the pusher is moving over the entire table; we ignore the specific number in this environment as we do not have robot arms reference.
Additional dataset samples. We show samples from the FluidPour, FluidCubeShake and Granu- larPush dataset in Figure 10, 11 and 12, respectively. Note that all trajectories for the extrapolate settings are used only for testing and will not show up during the training process. We include more samples from the dataset in the video format in the supplementary video.
B.2 MODEL ARCHITECTURE
Image-conditional NeRF. We follow the architectural design by Yu al. (2021). For the feature encoder, we employ a ResNet-34 backbone to extract features. We use the output layers prior to the first four pooling layers, upsampling them using bilinear interpolation the same size, and
16

---

Under review as a conference paper at ICLR 2023
Interpolate
Extrapolate
Figure 10: Samples from FluidPour Dataset. We show sequences of frames over time with an interval of 20 frames. The sequences above the dashed line are for interpolate data, and the bottom images illustrate the extrapolate data.
Interpolate
Extrapolatc
Figure 11: Samples from FluidCubeShake Dataset. We show sequences of frames over time with an interval 20 frames. The sequences above the dashed line are for interpolate data, and the bottom images illustrate the extrapolate data.
then concatenating these four feature maps. We initialize the weight of the feature extractor of the scene using ImageNet pre-trained weight. For the NeRF function f. We use fully-connected ResNet architecture with 5 ResNet blocks with width of 512.
Dynamics predictor. For the edge and vertice encoders, Qe and Qu. we use 3-layer fully-connected networks activated by the ReLU function with 150 hidden units. For the propagators, Pe and P., use a 1- llaayer culle-coneetted network followed by ReLU activation. The output dimension of the linear layer is 150.
17

---

Under review as a conference paper at ICLR 2023
Interpolate
Extrapolate
Figure 12: Samples from GranularPush Dataset. We show sequences of frames over time with an interval 20 frames. The sequences above the dashed line are for interpolate data, and the bottom images illustrate the extrapolate data.
Sampling 3D points from the trained visual perception module. We sample points on a 40 x 40 40 grid from area of 55cm x 55em x 55cm and 63cm x 63cm x 63cm at the center of the table FluidPour and FluidCubeShake respectively, and on a 70 x 70 6cm x 6cm X 6cm for GranularPush. We evaluate and include points with density (measured by the occupancy in the predicted neural radiance fields) larger than 0.99. To reduce the total number of points, subsample the inferred points with FPS with ratio of 5% for FluidPour and 10% for FluidCubeShake and GranularPush.
grid from an area of
Graph building. We set the neighbour distance threshold ō to be 0.2, 0.15, 0.15 for FluidPour, FluidCubeShake and GranularPust respectively. We select the threshold so that each point will sample the points with lower density
have average 20 30 neighbors. Since, in FluidPour, 20000poinssmm we use a larger threshold for this scenario. For FluidShape and GranularPush, since the density is around 3000 points/m². cut down the number by 25%.
We found that if the threshold is too small, the performance will degrade significantly since each particle will only receive messages from a few neighbors (and miss out the larger context). On the other hand, setting the threshold too large will cause the training time to increase since the graph will have more edges. We found that setting the threshold around the right scale generally leads more effective training of reasonable dynamics network.
B.3 TRAINING DETAILS
The models are implemented in PyTorch. We train the perception module using Adam optimizer with learning rate of 1c-4, and we reduce the learning rate by 80% when the performance on the validation set has stopped improving for 3 epochs. To compute the rendering loss when training the perception module, we sample points through each ray the scene and set the ray-batch size of the NeRF query function ff to be 1024 x 32. Training the perception module single scenario takes around hours on one RTX-3090.
We train the dynamics simulator using Adam optimizer with learning rate of 1e-4, and we reduce the learning rate by 80% when the performance on the validation set has stopped improving for epochs. The batch size is set to 4. We train the model for 20, 30, and 40 epochs for FluidPour, FluidCubeShake, and GranularPush. respectively. It takes around 10 15 hours to train the dynamics single RTX-3090.
model in environment
18

---

Under review as a conference paper at ICLR 2023
B.4 GRAPH-BASE DYNAMICS MODEL WITHOUT PARTICLE-LEVER CORRESPONDENCE
The velocity of an object provides critical information on how the object will move in the future, yet, do not have access such information when tracking the object is impossible. As described in Section 3.2, the attributes a of vertex V in the built graph consists of (1) velocity of this point in the past frames and (2) attributes of the point (rigid, fluid, granular). To get the velocity we should have the history position of this vertex. However, since the point clouds are inferred from each frame independently, we do not know how each point moves over time since we do not have point correspondence between frames.
vertex v,
To address the problem, we leverage the fact that some objects in the scene are easier track, and we try to use the motion of these trackable objects to infer motion for the untrackable units. We assume that we know the dense-labeled states of some known fully-actuated shapes like desks and cups connected to the robot arms. Here will list specific scenario where cup of water is poured into another cup. In this case, we have two different types of points: points for fluid and points for cups, we name the states of them in time step t V = {upp)} J and V {( respectively. For the particle encoder Qv. if the particle belongs to the cups, then the input of particle encoder
contains n, history states before to {VS (n.)oao n.)tt. If the particle belongs to the water, then we have no history states, so the input of Qu is all-zero.
By adding the relative position between receiver and sender points, can pass the momentum of Vp to Vg. Compared with human intuition, we can get an intuitive prediction of the movement of water by simply knowing the past movement of the cup without knowing the past movement of water.
Following Sanchez-Gonzalez Gonzalez et al. (2020), we use the velocity of points and their relative position as inputs to the dynamics module instead of using the absolute positions of the points. This ensures the model vaanslaton-iivariana so the learned dynamics model can be shared across different spatial locations.
B.5 INFERENCE SPEED OF OUR MODEL
The prediction speed of the dynamics module depends on the number of input particles, and it takes around 0.1s for graphs with around 300 nodes in FluidShake and FluidPour, and around 0.2s for scenes with 700+ nodes in GranularPush.
For our visual module, the main time consumption comes from NeRF sampling, it takes 0.2s to sample from grid space introduced in Section 4 of our paper, this was run in blocks, with block- si0e made up 4G V100 GPU. And it can be even faster with larger blocks. The sub-sumplle uub-sampling process (FPS, segmentation) is fast since they are all written in parallel versions, which takes less than 5ms.
C POTENTIAL SOCIETY IMPACT
Our work shows the possibility of learning dynamics models from raw sensory inputs, opening up opportunities automate the design of differentiable physics engines through data-driven learning algorithms. The resulting system can potentially benefit many downstream tasks, including general scene understanding, roboties manipulation, the construction of 3D generative models, and inverse tasks like planning/control and inverse design. Furthermore, predictions from our model are highly interpretable, which makes it straightforward to explain model behaviors and re-purpose the outputs for other downstream applications.
Though data-driven approaches are potentially more scalable with enough data, concerns still exist that it might be hard to ensure the robustness of the model under sensor noise and adversarial attacks. It also becomes less clear how to fully mitigate data biases. Therefore, bringing in advanced techniques from ML robustness will be one critical future avenue to pursue.
19


Reviews:
[{"review_id": "lAln6iVXZw", "reviewer": "Reviewer_E7oV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a pipeline for predicting the future steps of the physical simulation starting from the video input from multiple cameras. The model builds a Nerf model from several input frames, samples the particles from Nerf and runs the particle-based simulator using the extracted particle representation.\n", "summary_of_the_review": "This paper is one of the few approaches that performs the simulation from the video inputs, which is novel.However, this approach is severely limited in the environments that it can model (namely, simulated environments with colored fluids and differently-coloured objects), assumes to know that full mesh and state of the robot/container and relies on the velocities of the known objects to infer the velocities of the fluid. The authors need to be clear about the limitations of their work and the environments where the method is applicable: because fluid particles do not have velocities or past history, it would not be able to produce the correct fluid dynamics. I suggest rejecting the paper due to severe limitations of the approach that are not clearly stated.\n", "strength_and_weaknesses": "\n## Strengths\n\nThis paper is one of the few approaches that are able to perform the dynamics from the video inputs. Unlike the previous approach Nerf-DY, this paper uses a more structured representation of the simulation. That allow the model to generalise to new sizes of containers and different amount of fluid, which is consistent with the findings in the previous papers. The paper connects the two approaches that are known to work well from the previous works: PixelNerf (for a high-res 3D model) and particle-based simulators (for fluid modelling and generalisation). The past works have already shown the strong generalisation to new scenes due to locality of the particle interactions, so it is not surprising to see the similar results in this paper too.\n\n## Weaknesses\n\nThere are a few weaknesses in the method that limit the applicability of the work to different systems.\n\nThe method requires a major knowledge about the underlying simulation states, which defeats the purpose of learning from video. Specifically:\n- Knowledge of the 3D meshes of the robot/container/pusher and their positions/velocities at every time step. This is a strong assumption, because these meshes constitute a large part of the state. The meshes are also notoriously hard to extract from the video in the real-world scenarios.\n- Knowledge of fluid density to set the minimum distance for the spacing loss.\n- The model relies on the fact that the fluid is coloured and is easy to distinguish from other objects only based on the blue color. Similarly, the objects need to be of different colors, always visible and are easy to track across time.\n\nAnother major drawback is that the approach still requires to have the particle simulation as the ground-truth. The mesh/particles and the initial state are notoriously hard to obtain, preventing the previous simulators to train on the real-world data. Therefore, learning directly from videos is desirable. However, the proposed model unable to learn entirely from the video data. Moreover, it requires a corresponding ground-truth particle simulation for every step. It defeats the whole point of learning simulation from videos. \n\n### Representing the correct fluid dynamics.\n\n*Supplementary Section B.4*: “If the particle belongs to the water, then we have no history states, so the input of Qv is all-zero. …. Compared with human intuition, we can get an intuitive prediction of the movement of water by simply knowing the past movement of the cup without knowing the past movement of water”\n\nI don’t think this is true. Consider the case when the we start moving the cup back and forth, and we start recording the simulation when the cup is already moving. The fluid particles already have the prior velocity that is not the same as the velocity of the cup. It is not possible to infer the correct fluid dynamics without the history of fluid particles or velocities.\n\nAnother example from the paper is the fluid falling from one container to another.  Velocities of the falling particles depend on gravity and the previous particle velocity. Here we might be able to infer the fluid velocities **only** if we assume that fluid velocities is zero in the beginning of the simulation and we have access to the **entire** history of states. In all other cases, it is not possible to represent the correct dynamics of the fluid.\n\nThe fact that the videos look pretty good without the velocity information may indicate that the model exploits some biases from the data that allow to roughly imitate the dynamics under the point-cloud Chamfer loss. Specifically, I suspect that the model misuses the heuristic that the fluid particles have roughly the same as the velocities as the pusher/constrainer. It is visible in the attached videos of the ablations with and without the spacing loss: the particles that do not interact with the pusher are moving with the same speed as the pusher itself. The spacing loss also introduces a strong inductive bias that the fluid density remains constant. \n\nIt would be helpful to add the comparison of the dynamics component to the ground truth, starting from the known particle state and evaluated under the Chamfer distance, to compare the correctness of the predicted fluid dynamics. This should be possible to do, as the paper assumes having the ground truth particle simulation.\n\nAuthors should be honest about the very limited range of applicability of their model: the fluid velocity can be inferred from the velocities of the surrounding objects, which is a rare case. Even a simple case of the falling block of fluid cannot be represented by the current model. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hYAK6Pbmmo", "reviewer": "Reviewer_1Jtf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method for learning visual intuitive physics models solely from unlabeled multi-view images. Specifically, the method first leverages a conditional NeRF to learn the 3D geometry for the scene, which are then segmented and clustered to produce the 3D point cloud representation for the target fluid, rigid bodies and granular materials. A graph neural network is then applied to learn the dynamics of 3D points. The experiments demonstrate that the methods can produce good visual dynamics and generalize to simple unseen scenes.", "summary_of_the_review": "I think the assumption of known segmentation is too strong in order to combine the two existing modules. It's unknown how the methods would perform without such an unreasonable assumption. I think more experiments (learned segmentation) and more analysis (robustness to worse segmentation) should be added. I think the paper is below the bar of ICLR.\n", "strength_and_weaknesses": "Strengths:\n- The paper provides a possible direction for learning visual dynamics by advocating explicitly utilizing the geometry information.\n- The experiments are clear. The proposed method outperforms existing methods.\n\nWeaknesses:\n- The proposed two modules are simple combination of existing works. The Perception Module uses the existing PixelNeRF to learn underlying geometry from sparse views. The dynamics simulator applies standard graph-based backbone. There are no much new in the network design.\n\n- The proposed methods use the color information to obtain the masks, which is a very strong assumption and not reasonable. Although the author claims it could also be solved with object segmentation, I think the problem of segmenting fluid, rigid bodies and granular materials is non-trivial (such as labeling, generalization etc) while the proposed method has a very strong reliance on it. This is even more severe when the author claims the methods can achieve \"strong generalization\". It would be better if the author can show the robustness of the proposed methods with different learned segmentation results. Otherwise I don't think the paper has enough contribution to the problem of learning visual dynamics. It's more like combining two existing methods with a strong and unreasonable assumption to solve a problem.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "L-x3w6s7BG", "reviewer": "Reviewer_TwJK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about dynamics prediction. Given multiple video streams taken from different view points of the same scene, for instance cubes floating in a liquid, the task is to predict future states of the scene. This is a very interesting problem. The proposed solution has a bit of an engineering approach but is reasonable. I generally enjoyed reading this paper and have only a few questions or suggestions how to improve the paper.", "summary_of_the_review": "The proposed pipeline is reasonable and the problem addressed by the authors is very important and interesting. I am not very familiar with the topic itself, though, so my confidence is not high.", "strength_and_weaknesses": "This paper presents a consistent pipeline how to infer future states of physical systems given images only. The pipeline to rely on a NeRF model and uplift instance segmentations into 3D space is a neat idea.\n\nHow to infer dynamics from raw pixels is a problem not addressed by prior works.\n\nI have 2 questions remaining: In Sec 3.2, how do you obtain estimates for the velocities since there does not exist explicit correspondences between 3D points across frames? Secondly, what are the point attributes a_i^v? Are these the class labels from the instance segmentation? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5KwjL2YHIm", "reviewer": "Reviewer_z9jL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to learn a visual dynamics prediction model in the 3D space. The 3D point cloud representation is learned only from a few images from multiple views (with known camera pose). After that, they use instance segmentation to parse individual objects from the point cloud. Then it trains a graph-based prediction model on the 3D representations, without relying on the ground-truth correspondence, using the chamfer and spacing loss. The proposed method is evaluated using the prediction loss on a few simulation tasks including Pour, Shake, and Push.", "summary_of_the_review": "In summary, I think some aspects of this paper is worth reading to the community. There are still some discussion and experiments missing as I pointed out in the weakness section and it may still need some work. I’m also happy to discuss and increase my score if the above concerns are addressed.", "strength_and_weaknesses": "Strength:\n1) The idea of combining NeRF and graph-based dynamics prediction is interesting. Results are also visually good.\n2) The writing is pretty clear and complete. It’s easy to read and understand the main message of the paper. The figures are also nice and well-designed.\n3) Code is provided. This could be a good asset to the community.\n\nWeakness:\n1) Most of components are adopted from previous works, including image-conditioned nerf and training dynamics prediction using chamfer distance. Getting instance point cloud from segmentation is new but this is relatively easy (color segmentation) and this is not as “imperfect” as the authors claimed in the abstract given the relatively simple scene. It remains unclear if this pipeline can be used in other complex scenarios or even in the real-world.\n2) The evaluation is purely based on visual quality while previous work (Li et al. 2021b) also evaluate on control accuracy.\n3) What is the merge loss in Figure 2? Does that mean the spacing loss?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject