3D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenes
Authors: Haotian Xue
Antonio Torralba
Joshua B. Tenenbaum
Daniel LK Yamins
Yunzhu Li
Hsiao-Yu Tung
Venue: NeurIPS 2023 poster (2023)
Keywords: Intuitive Physics
Computer Vision

Abstract: Given a visual scene, humans have strong intuitions about how a scene can evolve over time under given actions. The intuition, often termed visual intuitive physics, is a critical ability that allows us to make effective plans to manipulate the scene to achieve desired outcomes without relying on extensive trial and error. In this paper, we present a framework capable of learning 3D-grounded visual intuitive physics models from videos of complex scenes with fluids. Our method is composed of a conditional Neural Radiance Field (NeRF)-style visual frontend and a 3D point-based dynamics prediction backend, using which we can impose strong relational and structural inductive bias to capture the structure of the underlying environment. Unlike existing intuitive point-based dynamics works that rely on the supervision of dense point trajectory from simulators, we relax the requirements and only assume access to multi-view RGB images and (imperfect) instance masks acquired using color prior. This enables the proposed model to handle scenarios where accurate point estimation and tracking are hard or impossible. We generate datasets including three challenging scenarios involving fluid, granular materials, and rigid objects in the simulation. The datasets do not include any dense particle information so most previous 3D-based intuitive physics pipelines can barely deal with that. We show our model can make long-horizon future predictions by learning from raw images and significantly outperforms models that do not employ an explicit 3D representation space. We also show that once trained, our model can achieve strong generalization in complex scenarios under extrapolate settings.

3D-IntPhys:
Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenes
Haotian Xue
Antonio Torralba 2
Joshua Tenenbaum?
Daniel Yamins²
Yunzhu Li 3,4x
Hsiao- Yu Tung
Georgia Tech
MIT
Stanford Univeristy
UIUC
Abstract
Given visual scene, humans have strong intuitions about how scene evolve over time under given actions. The intuition, often termed visual intuitive physics, is critical ability that allows to make effective plans to manipulate the scene achieve desired outcomes without relying on extensive trial and error. In this paper, we present framework capable of learning 3D-grounded visual intuitive physics models from videos of complex scenes. Our method is composed of a conditional Neural Radiance Field (NeRF)-style visual frontend and 3D point-based dynamics prediction backend, using which we can impose strong relational and structural inductive bias to capture the structure of the underlying environment. Unlike existing intuitive point-based dynamics works that rely on the supervision of dense point trajectory from simulators, we relax the requirements and only assume access to multi-view RGB images and (imperfect) instance masks acquired using color prior. This enables the proposed model to handle scenarios where accurate point estimation and tracking are hard or impossible. We generate datasets including three challenging scenarios involving fluid, granular materials, and rigid objects in the simulation. The datasets do not include any dense particle information most previous 3D-based intuitive physics pipelines can barely deal with that. We show our model can make long-horizon future predictions by learning from images and significantly outperforms models that do not employ an explicit 3D representation space. We also show that once trained, our model can achieve strong generalization in complex scenarios under extrapolate settings. The code is released in https: :/pp:::///h //gi thub. www.xaaiaarrttD-DIAI... IntPhys.
1 Introduction
Humans achieve strong intuitive understanding of the 3D physical world around us simply from visual perception [5, 8, 52, 51, 46, 11]. As we constantly make physical interactions with the environment, the intuitive physical understanding applies objects of wide variety of materials [6, 56). For example, after watching videos of water pouring and doing the task ourselves, we can develop mental model of the interaction process and predict how the water will move when apply actions like tilting or shaking the cup (Figure 1). The ability to predict the future evolution of the physical environment extremely useful for humans to plan behavior and perform everyday manipulation tasks. It is thus desirable to develop computational tools that learn 3D-grounded models of the world purely from visual observations that can generalize objects with complicated physical properties like fluid and granular materials.
"This work was done when Haotian Xue was a visiting student advised by Joshua Tenenbaum in MIT CSAIL and he was directly mentored by Yunzhu Li and Hsiao-Ya Yu Tung.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Old Scene
3D-Aware Intuitive Physics
Old Scene
Old Scene
New Scene
Figure I: Visual Intuitive Physics Grounded in 3D Space. Humans have strong intuitive understanding of the physical environment. We can predict how the enviromment would evolve when applying specific actions. This ability roots in understanding of 3D and applies objects of diverse materials, which is essential when planning our behavior to achieve specific goals. this work, we are the first to leverage combination of implicit neural representation and explicit 3D particle representation to build 3D 3D-grounded visual intuitive physics models of the challenging scenes that applies to objects with complicated physical properties, such as fluids, rigid objects, and granular materials.
There has been series of works on learning intuitive physics models of the environment from data. However, most existing work either focuses on 2D environments [60, 1, 21, 64, 20, 4, 44, 28, 67, 24, 23, 49, 33, 19, 31, 58, 22, 12, 65] or has to make strong assumptions about the accessible information of the underlying environment [36, 35, 47, 43, 70, 54, 48, 7, 2, 26] (e.g.. full-state information of the fluids represented points). The limitations prevent their use in tasks requiring an explicit 3D understanding of the environments and make it hard to extend to more complicated real-world environments where only visual observations are available. There works aiming to address this issue by learning 3D-grrunded representation of the environment and modeling the dynamics in latent vector space [34, 32]. However, these models typically encode the entire scene into one single vector. Such design does not capture the structure of the underlying systems, limiting its generalization to compositional systems or systems of different sizes (c.g., unseen container shapes or different numbers of floating ice cubes).
In this work, propose 3D Visual Intuitive Physics (3D-Inthhys), framework that learns intuitive physics models of the environment with explicit 3D and compositional structures with visual inputs.
Specifically, the model consists of (1) perception module based on conditional Neural Radiance Fields (NeRF) [41, 68] that transforms the input images and instance masks into 3D point represen- tations and (2) dynamics module instantiated as graph neural networks to model the interactions between the points and predict their evolutions over time. Despite advances in graph-based dynamics networks [47, 36], existing methods require strong supervision provided by 3D GT point trajectories, which are hard to obtain most real setups. To tackle the problem, we train the dynamics model using (1) distribution-baa
loss function measuring the difference between the predicted point sets and the actual point distributions at the future timesteps and (2) a spacing loss to avoid degenerated point set predictions. Our perception module learns spatial-equivariant representations of the envi- ronment grounded in the 3D space, which then transforms into points as a flexible representation to deseribe the system's state. Our dynamics module regards the point set as a graph and exploits the compositional structure of the point systems.
The structures allow the model to capture the compositionality of the underlying environment, handle systems involving objects with complicated physical properties, and perform extrapolated generalization, which we show via experiments greatly outperform various baselines without structured 3D representation space.
2 Related Work
Visual dynamics learning. Existing works learn to predict object motions from pixels using frame- centric features [1, 20, 4, 24, 23, 53, 30, 59, 69, 10, 25, 62) or object-centric features [21, 61, 28, 44, 27, 58, 14, 22, 45, 66], yet, most works only demonstrate the learning in 2D scenes with objects

---

Input Video Camera Viewpoints
Dynamics Prediction in Partiele Space
x6
X6
3D- Atcare Percepttio
Encoder
Encoder
Encoder
Graphi-usee Dynamics Predictor
(c. 0) = f(x d, ((r(z))
Pred
XN
Rendering
Loss
Objject PointCloud
Sparse Sampling
Instance Segmentation
Instamen PointCloud
Marge Loss
Chamfer Loss
Long Time Rolling (Test Time)
Figure 2: Overview of 3D Visual Intuitive Physics (3D-IntPhys). Our model consists of two major compo- nents: Left: The perception module maps the visual observations into implicit neural representations of the environment. We then subsample from the reconstructed implicit volume obtain particle representation of the environment. Right: The dynamies module, instantiated as graph neural networks, models the interaction within between ohjects and predicts evolution particle set.
moving only on a 2D plane. We argue that one reason that makes it hard for these existing methods to be applied to general 3D visual scenes is because they often operate on view-dependent features that can change dramatically due to changes in the camera viewpoint, which shouldn' 't have any effect on the actual motion of the objects. Recent works by [9] have shown that only methods that 3D view-invariant representations can pave the way toward human-level physics dynamics prediction in diverse scenarios.
Researchers have attempted to learn object motion in 3D [55, 63, 40, 34), [55] and [63] use object- centric volumetric representations inferred from RGB-D to predict object motion, yet, these volumet- ric approaches have much higher computation costs than 2D methods due to the 4D representation bottleneck, which hinders them from scaling up to more complex scenes. [40] use self-supervised 3D keypoints and [15, 16] use implicit representations to model multi-object dynamics but cannot handle objects with high degrees of freedom like fluid and granular materials. [34] use neural implicit representation to reduce the potential computational cost, yet the works have not shown how the ap- proach can generalize to unseen scenarios. Our works aim to solve the tasks of learning generalizable object dynamics in 3D by combining the generalization strength input- input-feature- centure-conditioned implicit representation and point-based dynamics models.
Point-based dynamics models. Existing works in point- and mesh-based dynamics models [36, 42, 57, 47, 43] have shown impressive results in predicting the dynamics of rigid objects, fluid [36, 42, 57, 47, 13), deformable objects [36, 42, 47], and clothes (83. Most works require access to full 3D states of the points during training and testing, yet, such information is usually not accessible in real-world setup. [35] learn a visual frontend to infer 3D point states from images, still require 3D point states and trajectories during training time. [50] propose to learn point dynamics directly from vision, but they only consider clasto-plastic objects consisting of homogeneous materials. How to learn about 3D point states and their motion from raw pixels remain question. Our paper tries to build the link from pixels to points using recent advances unsupervised 3D inference from images using NeRF [41, 68).
3 Methods
We present 3D Visual Intuitive Physies (3D-IntPhys), a model that learns simulate physical events from unlabeled images (Figure 2). 3D-IntPhys contains perception module that transforms visual observations into 3D point cloud that captures the object geometries (Section 3.1) and point-based simulator that learns to simulate the rollout trajectories of the points (Section 3.2). The design choice of learning physics simulation a 3D-point representation space enables stronger simulation performance and generalization ability. The performance gain mainly comes from the fact that describing/learning objects' motion and interactions in 3D are easier compared to doing so in 2D since objects live and move persistently in the 3D space. 3D-IntPhys also supports better

---

generalization ability since its neural architecture explicitly models how local geometries of two objects/parts interact, and these local geometries and interactions can be shared across different and novel object combinations.
Although 3D-IntPhys learns simulate in 3D representation space, show it can learn without any 3D supervision such as dense point trajectories as in previous work [47,36] Dense point trajectories hard and sometimes impossible to obtain in the real world, e.g. capturing the trajectories of each water point. 3D-IntPhys does not require such 3D supervision and can simply learn by observing videos of the scene evolution.
3.1 2D-to-3D Perception Module
Given static scene, the perception module learns to transform one or few posed RGB images, {(4,, m; E {1, 2, 2.. NN }., taken from N, different views, into a 3D point cloud representation of the scene, X. We train the model in an unsupervised manner through view reconstruction, using dataset consisting of N, videos, where each video has Ny frames, and each frame contains images taken from No viewpoints.
Neural Radiance Field (NeRF). NeRF [41] learns to reconstruct a volumetric radiance field of a scene from unlabeled multi-view images. After training, the model learns to predict the RGB color e and the corresponding density a of a query 3D point x E IR3 from the viewing direction d E R with function (c, o) f(x, d). We can formulate camera ray r(t) + id, where is the origin of the ray. The volumetric radiance field can then be rendered into a 2D image via where T(t) = exp(- - J.- (s)ds) handles occlusion. The rendering
R3
C(r) = J ((t))a(t)(())) range is controlled by the depths of the near and far plane (i.e., in and tt). We can train NeRF through view prediction by:
L = Σ EPERP) |(((r) - C(r)
where Rpp) is the set of camera rays sampled from target camera pose p.
image-conditioned NeRF. To infer the NeRF function from an image, previous work proposed to encode the input image into a vector, with a CNN encoder, as conditioning input to the target NeRF function [34]. We found this type of architecture is in general hard to train and does not generalize well. Instead, we adopt piXelNeRF [68], which conditions NeRF rendering with local features, as opposed to global features. Given image I in scene, pixeINeRF first extracts feature volume using a CNN encoder W = E().. For a point x in the world coordinate, we retrieve its feature vector by projecting it onto the image plane, so that we can get the feature vector W(T (x)). PixeINeRF combines the feature vector together with the 3D position of that point and predict the RGB color and density information:
V(x) = (c = fxx d. W(x(x))).
In the experiment section, we will show that it is surprisingly effective to train only one general model to learn a conditional Neural Radiance Field that can apply all videos in one type of scene (e.g. FluidPour) with five different settings (e.g. extrapolate setting), which provides a better 3D representation for the scene and greatly facilitates the learning of 3D Intuitive Physics.
Explicit 3D representation from pixeINeRF. From a few posed RGB images I, of scene we infer a set of points for O, target object (such as fluid, cube) in the scene. We achieve this by first sampling set of points according to the predicted occupancy measure, then clustering the points into objects using object segmentations. We found that sampling with low resolution will hurt the quality of the rendered point cloud to generate objects with inaccurate shapes, while sampling with high resolution will increase the computation for training the dynamics model since the input size increases. To speed up training while maintaining the quality of the reconstrueted point cloud, first infer the points with higher resolution and do sparse sampling of each point cloud using FPS (Farthest Point Sampling) [17]. Next, cluster the inferred points into objects according object segmentation masks. Since solving object segmentation in general is not the main focus of this paper, we resort using the color information obtain the masks.
3.2 Point-Based Dynamics Learner
Given the point representation at the current time step, X,, the dynamics simulator predicts the points' evolution I' steps in the future, (X++1, X ... x using graph- networks [47, 36].

---

We first form a graph (V, E) based on the distance between points. If the distance between two points smaller than a threshold 6, we include an edge between these two points. Each vertex (0s, a.) V contains the velocity of the point, di, and point attributes, aj, to indicate the point's type. For each relation, () E F., we have associated relation attribute agz indicating the types of relation and the relative distance between the connected points.
Spatial message passing and propagation. At time step t, can do message passing to update the points, 2 E V, and relation representations, (i,j) E E, in the graph:
Dej.t = Qe(vi, (ge (1,5)
(3)
b4. Qu(Us, E ExEOIISEEY 95R,1) 2 EV
where Qu and Q- are encoders for vertices and relations respectively. Please refer to [7] for more details. Though this kind of message passing can help with updating representation, can only share one-hop information each step, limiting its performance on instantaneous passing of forces. To improve long-range instantancous effect propagation, we use multi-step message propagation as in [37, 36]. The propagation step is shown in Alg I:
Algorithm 1: Point-based Dynamics Predictor
Data: Current timestep t, point cloud Vi, vertex encoder Qu,edge encoder Qe, vertex propagator Pr., edge propagator Pe, state predictor fs
Result: V++1 Form graph Ge = (V,, Ee)
//message passing
Iiy. = Qe Uy,, (i.5) E E,
h4 Qu(ur, EXEORONEE lik.e) U E V.
//mesagee propagation
h,, h4,1, = J1.t:
for / {1.2,3,... L} do
h (i.,)) Er 1-1 M (h ExEDIOneEy Pire) ,0 E.
I'y.1 Po 1-1 1-1
end
//sate prediction
01,41 (b,,)
V++1 {vi,+1}
where P., P. are propagation functions of nodes and edges. respectively, and dn is the effect of relation (i, in propagation step l. h/_ the hidden states for each point in the propagation process. Finally, we have the predicted states of points at time step / + after L steps of propagation:
(5) Environments. We assume that the surrounding environment (e.g., the table) is known and the robot/rool/comtainer are of known shape and fully actuated, where the model has access to their complete 3D state information. We convert the full 3D states into points through sampling the 3D meshes and include these points in the prediction of the graph-based dynamics.
0 (h
Fluids, rigid bodies, and granular materials. We distinguish different materials by using different point attributes ay. We also set different relation attributes af, in Equation interaction (e.g.. FluidsF Fluids-Fluids, Granular-Pusheri For rigid objects, to ensure the object shapes remain consistent throughout the rollout predictions, we add a differentiable rigid constraint in the prediction head following [36].
distinguish different
Training dynamics model without point-level correspondence. Since our perception model parses each RGB image into object-centte point clouds independently, there does not exist an explicit one-to-one correspondence for points across frames. To handle this, we measure the Chamfer distance between the prediction X, = (Vi, f.,) from the dynamics network and the inferred point state X, (V,, Ex) from the perception module and treat it as the objective function. The Chamfer
5

---

Scene Trajectories
FluidPour
FluidCubeShake
GrunularPush
Interpolate
Camera
Camera
Extrapolate
Camera
Camera
Camera
Diff. containcr & more fluid
More fluid & cubes
More granular modia
Figure 3: Data Collection and Evaluation Sctups. Left: We collect multi-view videos of the environment from six cameras. Right: We consider diverse set of evaluating environments involving fluids, rigid objects, granular materials, and their interactions with the fully- fully-actuated container the environment. We evaluate the learned visual intuitive physics model on both the interpolated settings (i.e., seen environment but with different action sequences) and extrapolated settings (i.e., unseen environment with different amounts of fluids, cubes. granular pieces, and containers of different sizes).
distance between two point cloud V and V is defined as:
Lo
Σ min | v112
min || v122.
(6)
vEV
IV
VEV
We found that training the model with Chamfer distance dense scenes with granular materials will often lead to predictions with unevenly distributed points where some points stick too close to each other. To alleviate this issue, we further introduce a spacing loss L., which penalizes the gated distance (gated by dmin) of nearest neighbor of each point ensure enough space between points:
= L.(V)=
(ReLU(dmin)
min v'E(VVo)
1 - =112))2
(7)
The one-step prediction loss Lay for training the dynamics model is L L.(V, VV + LLVV where o reweights the second loss. To improve long-term rollout accuracy, we train the model with two-step predictions using the first predicted state as input and feed it back into the model to generate the second predicted state. With the two-step loss, the model becomes more robust to errors generated from its own prediction. Finally, the Lay losses for all rolling steps are summed up to get the final loss for this trajectory. More implementation details are included in the supplementary material.
Experiments
The experiment section aims to answer the following three questions. (1) How well can the visual inference module capture the content of the environment (i.c., can we use the learned representations reconstruct the scene)? (2) How well does the proposed framework perform in scenes with objects of complicated physical properties (e.g., fluids, rigid and granular objects) compared to baselines without explicit 3D representations? (3) How well the models generalize extrapolate scenarios?
We generated three simulated datasets using the physics simulator Nvidia FleX [39]. Each of the datasets represents one specific kind of manipulation scenario, where robot arm interacts with rigid, fluid, and granular objects (Figure 3). For each of the three scenarios, we apply randomized input actions and change some properties of objects in the scene, e.g., the shape of the container, the amount of water, and the color/number of cubes, to make it diverse. To test the generalization capability of the trained model, we design extrapolated datasets where the data is generated from an extrapolated set of parameters outside the training distribution.
Datasets.
a) FluidPour. This scenario contains futla-attuated cup pouring fluid into container. We design the extrapolate dataset to have a larger container, more quantity of fluid, and different pouring actions.
b) FluidCubeShale This scenario contains actuaeee container that moves top of table. Inside the container are fluids and cubes with diverse colors. We design the extrapolate dataset to have different container shapes, number of cubes, cube colors, and different shaking actions.

---

Interpolate
Extrapolate
Ground Truth
NeRF-dy
Ours
GT
Ours NeRF-dy
NcRF-dy Ground Truth
Ours
Figure 4: Qualitative Results of the Dynamies Module Future Prediction. Here visualize our model's predicted future evolution of the particle set as compared with NeRF-dy [34] baseline both interpolate and extrapolate settings. Our method correctly identifies the shape/distribution of the fluids, rigid objects, and granular pieces with much better accuracy than NeRF-dy. dy. The future evolution predicted by our method also matches the ground truth much better and produces reasonable results in extrapolate settings.
c) GranularPush. This environment contains fully- aettaee board pushing pile of granular pieces. We design the extrapolate dataset have larger quantity of granular objects in the scene than the model has ever seen during training.
Baselines. We compare method with two baselines, NeRF-dy [34] and autoencoder (AE) (similar to GQN [18] augmented with a latent-space dynamics model). NeRF-dy a 3D-aware framework that also learns intuitive physics from multi-view videos. Yet, instead of learning the object dynamics with explicit and compositional 3D representations, the model learns dynamics models with implicit 3D representations in the form of single latent vector. We also compare our method with an autoencoder-bant based reconstruction model (AE) [18] that can perform novel-view synthesis but is worse handling 3D transformations than neural implicit representations. AE first learns scene representations through fer-frame image reconstruction, and then it learns dynamics model on top of the learned latent representations. All methods take RGB images and camera parameters as inputs. To incorporate object-level information, we perform bolor-based segmentation to obtain object masks as additional inputs to the baselines. The implementation details and parameter settings of our method can be found in the supplementary materials.
7

---

FluidPour
FluidCubeShake
GranularPush
Metrics
Model
mnl)
OoD
InD
OoD
InD
OoD
AE
451.03
542.86
869.3
1727.55
562.06
1537.2
M
NeRF-dy
202.95
317.27
527.46
1585.97
481.95
1020.0
Ours
111.66
124.33
66.52
81.38
147.97
646.85
AE 0.86
0.84
0.71
0.86
0.81
0.62
SSIM(t)
NeRF-dy 0.89
0.86
0.73
0.65
0.81
0.61
Ours 0.90
0.89
0.94
0.93
0.89
0.69
Table 1: Quantitative Results of the Perception Module. We compare method with autoencoder (AE) NeRF-dy [34] with additional instance masks based color. We measure the quality rendered images by
computing the Mean Squared Error (MSE) and Structural Similarity Index Measure (SSIM) compared to the ground truth. InD stands for in-distribution tests, and OoD stands for out-of-distributee tests.
Interpolate
Extrapolate
Interpolate
Extrapolate
Interpolate
Extrapolate
Ground Truth
Autocncoder
NeRF-dy dy
Ours
(a) FluidPour
(b) FluidCubeShake
(c) GranularPush
Figure 5: Qualitative Reconstruction Results of the Perception Module. The images generated by method contain more visual details and much better aligned with the ground truth. Our model is much better handling large seene variations than NcRF-dy, especially in extrapolate settings.
4.1 Image Reconstruction From Learned Scene Representations
We test how well the perception modules capture scene information by evaluating the visual front-end of all models their ability to reconstruct the observed scene from the inferred representations. We measure the difference between the reconstructed and ground truth images with Mean Squared Error (MSE) and Structural Similarity (SSIM) in pixel level (Table 1). Our perception module outperforms all baselines all three environments. The performance gap is exaggerated in extrapolate settings, especially in scenarios that involve complex interactions between rigid and deformable materials (Figure qualitative comparisons).
4.2 Learned Visual Dynamics On In-Distribution Held-Out Scenes
Next, we compare long-term rollouts in the 3D space. We evaluate the models using the Chamfer distance between predicted point cloud and the ground truth. For NeRF-dy, we decode predicted rollouts latent vectors into the point cloud with the learned NeRF decoder. We exclude the comparison with AE since is unclear how to decode the learned representations into point clouds. We show quantitative comparison in Figure and qualitative results in Figure 4. 3D-Inthys can

---

Pour
Shake
Push
0.33
nett
0.10
2.0
0.12
1.5
1.9
6.00
33
0.33
0.15
14
cardy
Extrapolate
612
0.09
04
Figure 6: Quantitative Results of the Dynamics Module. This figure compares our method and NeRF-dy [34] their long-hou hhrizon open-loop futtrr prediction loss. The loss measured the Chamfer distance between the predicted particle set evolution and actual future. Our method outperforms baseline both interpolate extrapolate settings, showing benefits of explicit 3D modeling.
Particle-View
View
Width x
Width 5
Figure 7: Strong Generalization Ability of the Dynamics Module to Wider Pushers. We evaluate our dynamics model on unseen width of pushers in GranularPush environment. The left part shows in 3D space where red indicates granular materials, green shows the table and pusher, and the arrow shows how the pusher about move. The right part shows from the top view of the rendering results.
learn reasonable scene dynamics in all scenarios and significantly outperforms NeRF-dy, While NeRF-dy can learn relatively reasonable movements fluids, it fails to learn complex dynamics such the floating cube and the morphing of the granular materials. The results suggest that the proposed dynamics.
explicit 3D point-based representations are critical to learning complex
4.3 Generalization on Dut-of-Distribution Scenes
To test the generalization ability of the models, we introduce extrapolate settings all of the three scenarios. See "Extrapolate" results in Table 1, Figure 5, 6, and 4. The proposed 3D- 3D-IntPhys gener- alizes well extrapolate settings both at the visual perception stage and the dynamics prediction stage, whereas NeRF-dy and autoencoder both fail at generalizing under extrapolate settings. For example, in FluidShake, both baselines cannot capture the number (Figure 5). And in GranularPush, both baselines fail to capture the distributions of the granular materials. NeRF-dy performs much worse on extrapolation scenes compared to in-distribution scenes, suggesting that incorporating 3D information in explicit way, opposed to implicit, is much better at capturing the structure of the underlying environment, thus leading to better generalization. We further test our model completely unseen changes to the environment in the GranularPush environment, we extend the width of the pusher by a factor and 5. Though the stretched pusher has never shown in the training data, our model can make reasonable pushing predictions (see Fig 7).
the color of the rigid cubes
5 Conclusions
In this work, we propose 3D-aware and compositional framework, 3D-Inthhys, to learn intuitive physics from unlabeled visual inputs. Our framework can work on complex scenes involving fluid,

---

rigid objects, and granular materials, and generalize to unseen scenes with containers of different sizes, more objects, or larger quantities of fluids and granular pieces. We show the proposed model outperforms baselines by a large margin, highlighting the importance of learning dynamics models in an explicit representations space. The major limitation of our work is the assumption of access to object masks. However, with the progress be possible to get such kinds of masks in real-world 3D environments. Our work serves as a pioneer in visual intuitive physics learning of complex scenes, and it is an exciting future direction to learn more complex intuitive physics from real-world data with the help of these large models.
segmentation in the wild [71, 29], we believe that it will
10

---

References
[1] P. Agrawal, A. V. Nair, P. Abbeel, J. Malik, and Levine. Learning to poke by poking: Experiential learning of intuitive physics. Advances in neural information processing systems, 29, 2016.
[2] A. Ajay, M. Bauzá, Wu, N. Fazeli, J. B. Tenenbaum, A. Rodriguez, and L. P. Kaelbling. Combining physical simulators and object-based networks for control. CoRR, abs/ 0bss/194.0058)) 2019.
[3] K. R. Allen, T. Lopez-Guevara, K. L. Stachenfeld, A. Sanchez-Gonzalee P. W. Battaglia, J. B. Hamrick, and T. Pfaff. Physical design using differentiable learned simulators. CoRR, abs/2202.00728, 2022.
[4] M. Babaeizadeh, M. T. Saffar, S. Nair, S. Levine, C. Finn, and D. Erhan. Fitvid: Overfitting in pixel-level video prediction. CoRR, abs/2100..3195, 2021.
[5] R. Baillargeon, E. S. Spelke, and S. Wasserman. Object permanence in five-month-old infants. Cognition, 20: 20:100022200 191-208, 1985.
[6] C. Bates, I. Yildirim, J. B. Tenenbaum, and P. W. Battaglia. Modeling human intuitions about liquid flow with particle-hased simulation. CoRR, abs/1 has/18099015244 2018.
[7] P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Gonzalz,,, V. F. Zambaldi, M. Malinowski,
A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, Ç. Gülçehre, H. F. Song, A. J. Ballard, J. Gilmer, G. E. Dahl, A. Vaswani, K. R. Allen, C. Nash, V. Langston, C. Dyer, N. Heess, D. Wierstra, P. Kohli, M. M. Botvinick, O. Vinyals, Y. Li, and R. Pascanu. Relational inductive biases, deep learning, and graph networks. CoRR, abs/1806.0126// 2018.
[8] W. Battaglia, B. Hamrick, and J. B. Tenenbaum. Simulation an engine of physical scene understanding. Proceedings of the National Academy of Sciences, 110:18327 18332, 2013.
[9] D. M. Bear, E. Wang, D. Mrowca, F. J. Binder, H.-Y. F. Tung. R. Pramod, C. Holdaway, S. Tao, K. Smith, L. Fei-Fei, et al. Physion: Evaluating physical prediction from vision in humans and machines. arXiv preprint arXiv: 1xXiv:2106.08260 08261, 2021.
[10] Y. Burda, H. Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros. Large-scale study of curiosity-driven learning. ICLR, 2019.
[11] S. Carey and Xu. Infants' knowledge of objects: beyond object files and object tracking. Cognition, 80(1):179-215 2001. Objects and Attention.
[12] M. B. Chang, T. D. Ullman, A. Torralba, and J. B. Tenenbaum. A compositional object-bases approach learning physical dynamics. CORR, abs/1612.00344 2016.
[13] de Avila Belbute-Peres, T. Economon, and Z. Kolter. Combining differentiable PDE solvers and graph neural networks for fluid flow prediction. CoRR, abs/2007. 0is//0007.0439, 2020.
[14] D. Ding, F. Hill, A. Santoro, and M. M. Botvinick. Object- attention for apatio-temporg reasoning: Outperforming neuro-symbolic models with flexible distributed architectures. CoRR, abs/2012.08508, 2020.
[15] D. Driess, J.-S. Ha, M. Toussaint, and R. Tedrake. Learning models as functionals of signed- distance fields for manipulation planning. In Conference on Robol Learning, pages 245-255. PMLR, 2022.
[16] D. Driess, Z. Huang, Y. Li, R. Tedrake, and M. Toussaint. Learning multi-object dynamics with compositional neural radiance fields. arXiv preprint arXiv:2202.11855, 2022.
[17] Y. Eldar, M. Lindenbaum, M. Porat, and Y. Zeevi. The farthest point strategy for progressive image sampling. IEEE Transactions Image Processing, 6(9):1305-13155 1997.
[18] S. A. Eslami, D. Jimenez Rezende, F. Besse, F. Viola, A. S. Morcos, M. Garnelo, A. Ruderman, A. A. Rusu, I. Danihelka, K. Gregor, et al. Neural scene representation and rendering. Science, (1006394):1204-12211 2018.
11

---

[19] C. Finn, J. Goodfellow, and S. Levine. Unsupervised learning for physical interaction through video prediction. CoRR, abs//660507117, 2016.
[20] C. Finn and S. Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages
IEEE, 2017.
[21] K. Fragkiadaki, P. Agrawal, S. Levine, and J. Malik. Learning visual predictive models of physics for playing billiards. In Y. Bengio and Y. LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Truck Proceedings, 2016.
[22] R. Girdhar, L. Gustafson, A. Adcock, and L. van der Maaten. Forward prediction for physical reasoning. CoRR, abss2006.10034,, 2020.
[23] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent 2019.
imagination. arXiv preprint
[24] D. Hafner, T. Lillicrap, Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International conference machine learning, pages 2555 2565. PMLR, 2019.
[25] D. Hafner, T. P. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by 2019.
latent imagination. CoRR,
[26] M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model- policy optimization. CoRR, 00s/1996.00223,, 2019.
[27] M. Janner, S. Levine, W. Freeman, J. B. Tenenbaum, C. Finn, and J. Wu. Reasoning about physical interactions with oriented prediction and planning. In International Conference on Learning Representations, 2019.
[28] T. Kipf, E. van der Pol, and M. Welling. Contrastive learning of structured world models. arXiv preprint 11X1v:1911.12244 2019.
[29] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao. S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. arXiv preprint arXiv:2304.02644, 2023.
[30] A. X. Lee, R. Zhang, F. Ebert, P. Abbeel, C. Finn, and S. Levine. Stochastic adversarial video prediction. CoRR, abs/ 804.01123, 2018.
[31] A. Lerer, S. Gross, and R. Fergus. Learning physical intuition of block towers by example. CoRR, abs/1603.01312, 2016.
[32] T. Li. M. Slavcheva, M. Zollhoefer, S. Green, C. Lassner, C. Kim, T. Schmidt, S. Lovegrove, M. Goesele, and Z. Lv. Neural 3d video synthesis. arXiv preprint arXiv:2103.0m 025570 2021.
[33] W. Li, S. Azimi, A. Leonardis, and M. Fritz. To fall or not to fall: A visual approach to physical stability prediction. CoRR, abs/ abs//600.. 600.00066, 2016.
[34] Y. Li, S. Li, V. Sitzmann, P. Agrawal, and A. Torralba. 3d neural scene representations for visuomotor control. arXiv preprint arXiv:2107.04004 2021.
[35] Y. Li, T. Lin, K. Yi, D. Bear, D. L. Yamins, J. Wu, J. B. Tenenbaum, and A. Torralba. Visual grounding of learned physical models. In International Conference on Machine Learning, 2020.
[36] Y. Li, Wu, R. Tedrake, J. Tenenbaum, and A. Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In ICLR, 2019.
[37] Y. J. Wu, J.-Y. Zhu, J. B. Tenenbaum, A. Torralba, and R. Tedrake. Propagation networks for model-based control under partial observation. In 2019 International Conference on Robotics and Automation (ICRA), pages 1211-1211. IEEE, 2019.
[38] X. Lin, Y. Wang, Z. Huang, and D. Held. Learning visible connectivity dynamics for cloth smoothing. In Conference Robot Learning, 2021.
12

---

[39] M. Macklin, M. Müller, N. Chentanez, and T.-Y. Kim. Unified particle physies for real-time applications. ACM Transactions on Graphics (TOG), 33(4):1-12, 2014.
[40] L. Manuelli, Y. Li, Florence, and R. Tedrake. Keypoints into the future: Self-supervised correspondence in model-based reinforcement learning. arXiv preprint arXiv:2009.050855 2020.
[41] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pages 405-421. Springer, 2020.
[42] D. Mrowca, C. Zhuang, E. Wang, N. Haber, L. Fei-Fei, J. B. Tenenbaum, and D. L. K. Yamins. Flexible neural representation for physies prediction. CoRR, 0as/180060804477 2018.
[43] Pfaff, M. Fortunato, A. Sanchez-Gonzalee and W. Battaglia. Learning mesh-based simulation with graph networks. In International Conference on Learning Representations, 2021.
[44] H. Qi, X. Wang. D. Pathak, Y. Ma, and J. Malik. Learning long-term visual dynamics with region proposal interaction networks. In ICLR, 2021. 2021.
[45] R. Riochet, J. Sivic, Laptev, and E. Dupoux. Occlusion resistant learning of intuitive physics from videos. CoRR, abs/2005.00069, 2020.
[46] A. N. Sanborn, V. K. Mansinghka, and T. L. Griffiths. Reconciling intuitive physics and newtonian mechanics for colliding objects. Psychological review, 120 2:411-37, 2013.
[47] A. Sanchez-Gonzalez, J. Godwin, T. Pfaff, R. Ying, J. Leskovec, and P. Battaglia. Learning to simulate complex physics with graph networks. In International Conference Machine Learning, pages 8459 8468. PMLR, 2020.
[48] A. Goncaez-Gonzale,, N. Heess, J. T. Springenberg. J. Merel, M. A. Riedmiller, R. Hadsell, and P. W. Battaglia. Graph networks as learnable physics engines for inference and control. CORR, abs/1806.00 lbss/800.01222, 2018.
[49] J. Schritwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifree, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, T. Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(788)))6600-600)) 2020.
[50] H. Shi, H. Xu, Z. Huang, Y. Li, and J. Wu. Robocraft: Learning to see, simulate, and shape clasto plasto-plastic objects with graph networks. arXiv preprint (xXi:2205.00200 2022.
[51] K. Smith, L. Mei, S. Yao, J. Wu, E. Spelke, J. Tenenbaum, and T. Ullman. Modeling expectation violation in intuitive physies with coarse probabilistic object representations. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[52] E. S. Spelke. Principles of object perception. Cognitive Science, 14(1):29 56, 1990.
[53] H. Suh and R. Tedrake. The surprising effectiveness of linear models for visual foresight in object pile manipulation. In International Workshop on the Algorithmic Foundations of Robotics, pages 347-363. Springer, 2020.
[54] A. Tacchetti, H. F. Song, P. A. M. Mediano, V. F. Zambaldi, N. C. Rabinowitz, T. Graepel, M. M. Botvinick, and P. W. Battaglia. Relational forward models for multi-agent learning. CoRR, 0bs/1809.11 1044, 2018.
[55] H.-Y.F. Tung, Z. Xian. M. Prabhudesai, S. Lal, and K. Fragkiadaki. 3d-oes: Viewpoint inviriaan obcect-factorieed environment simulators. arXiv preprint arXiv:2011.0646. 2020.
[56] Ullman, Kosoy, Yildirim, A. A. Soltani, M. H. Siegel, J. Tenenbaum, and E. S. Spelke. Draping an elephant: Uncovering children's ss reasoning about cloth-covered objects. In Pro- ceedings of the s1st Annual Conference of the Cognitive Science Society, pages 3008-3014, 2019.
13

---

[57] B. Ummenhofer, L. Prantl, N. Thuerey, and V. Koltun. Lagrangian fluid simulation with continuous convolutions. In International Conference on Learning Representations, 2019.
[58] R. Veerapaneni, J. D. Co-Reyes, M. Chang, M. Janner, C. Finn, Wu, J. B. Tenenbaum, and S. Levine. Entity abstraction in visual model-based reinforcement learning. CoRR, abs/1910.12888 2019.
[59] C. Vondrick, H. Pirsiavash, and A. Torralba. Anticipating the future by watching unlabeled video. CoRR, 100.00534 2015.
[60] M. Watter, Springenberg, Boedecker, and M. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. Advances in neural information processing systems, 28, 2015.
[61] N. Watters, Zoran, Weher, Battaglia, Pascanu, and A. Tacchetti. Visual interaction networks: Learning physics simulator from video. Advances in neural information processing systems, 30, 2017.
[62] B. Wu, S. Nair, R. Martin-Martin, L. Fei-Fei, and C. Finn. Greedy hierarchical variational autoencoders for large-scale video prediction. CoRR, abs/2103.04174, 2021.
[63] Z. Xu, Z. He, Wu, and S. Song. Learning 3d dynamic scene representations for robot manipulation. In Conference on Robotic Learning (CoRL), 2020.
[64] Z. Xu, J. Wu, A. Zeng, J. B. Tenenbaum, and S. Song. Densephysnet: Learning dense physical object representations via multi-step dynamic interactions. In Robotics: Science and Systems (RSS), 2019.
[65] T. Xue, J. Wu, K. L.. Bouman, and W. T. Freeman. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks. In Advances In Neural Information Processing Systems, 2016.
[66] Y. Ye, D. Gandhi, A. Gupta, and S. Tulsiani. Object-centric forward modeling for model predictive control. In CoRL,, 2019.
[67] Y. Ye, M. Singh, A. Gupta, and S. Tulsiani. Compositional video prediction. In International Conference Computer Vision (ICCV), 2019.
[68] A. Yu, Ye, M. Taneik, and A. Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578-4587, 2021.
[69] M. Zhang, S. Vikram, L. Smith, Abbeel, M. Johnson, and Levine, SOLAR: deep struc- tured latent representations for model-based reinforcement learning. CoRR, abs/1 108 09,00095 2018.
[70] R. Zhang, J. Wu, C. Zhang, W. T. Freeman, and J. B. Tenenbaum. A comparative evaluation of approximate probabilistic simulation and deep neural networks accounts of human physical scene understanding. CoRR, abs/1605.00 1605.01 138, 2016.
[71] X. Zou, J. Yang, H. Zhang, F. Li, L. Li, J. Gao, and Y. J. Lec. Segment everything everywhere all once, arXiv preprint arXiv:2304.067/ 2023.
14

---

A
Additional Results
To better understand the performance of framework visually, prepare test time rollouts of our framework as well as those various baselines in the supplementary video. The video is published anonymously and can be accessed in
A.1 Ablation Study
We find that training the model with Chamfer distance in dense scenes with granular materials will often lead to predictions with unevenly distributed points where some points stick too close to each other. To alleviate the issue, we introduce the spacing loss to penalize the distance between these points. We set the threshold of penalty dmin to be 0.08 and the loss weight to be 10. We find that spacing loss can help improve the performance of the dynamics learner especially under extrapolate settings, as shown in Figure 8, We provide qualitative results in the supplementary video.
GrunularPush Extrapolate
GranularPush Interpolate
0.6
0.7
w spacing loss
0.5
w/o spacing
Chamfer Loss
0.5
0.4
0.4
0.3
0.3
0.7
0.2
0.0
10
15
20
30
0.0
10
20
25
30
Rolling Step
Rolling Step
Figure 8: Ablation Study on the Spacing Loss. Training dynamies models in the GranularPush scenario with spacing loss results in better rolling prediction. The performance gap is even more substantial in the extrapolate setting.
B
Implementation Details
B.1 Dataset Generation
Our datasets generated by the NVIDIA Flex simulator. Each of the three scenarios (Pour, Shake and Push) has 500 videos of trajectories taken from views, with each trajectory consisting of 300 frames. We manually select the views with reasonable coverage of the tabletop space minimize the occlusion. The 500 trials are generated from five different sets environmental parameters, detailed in Table 3. We take one set of parameters that outside the training distribution as the extrapolate dataset for evaluating model generalization. For the rest of the four settings, we randomly split them into train and test sets with ratio of 0.8.
Next, we provide more details for each scenario:
In the FluidPour environment, we randomly initialize the position of the upper container and then generate random bock-and-fortte actions by tilting the container. The action space is then the position and tilting angle of the upper container.
In FluidCubeShake, we also randomly initialize the position of the container and the cubes inside the container. We then generate random but smooth action sequences moving the container in the 2D plane. The action space is then the x-y location the container.
In GranularPush, we randomly initialize the position of the granular pile. Then, for each push, we randomly generate the starting and ending positions of the pusher and move the pusher along the straight line with an angle perpendicular to the pushing direction. The action space is a four-number tuple stating the starting and ending position the 2D plane.
The following table shows the moving range of the robot arms the FluidPour and FluidCubeShake environments after normalizing the robot into size that is the same in the real world (unit:
15

---

X-Range
Y-Range
Z-Range
FluidPour FluidCubeSSake
[-29.11, -12.66]
[42.00, 60.00] [19.25, 19.25]
[-7.78, 7.78] [-24.50, 24.00]
[-3.25, 42.25]
Table Robot Action Space(centimeterss:: we show the range the robot arms can move in the FluidPour and FluidCubeShake environments.
Y2
X2
Z2
Y1
X1
Figure 9: Illustration of the Environment Settings. In the FluidPour scenario, robot arm holds a container and tries to pour some fluid into another container. In the FluidShake scenario, a robot moves container with some fluid and cubes. We show the parameters for the container shape referred Table 3.
SceneName
Params
Env1
Env2
Env3
Env4
Extrapolate
X2
0.53
0.53
0.81
0.81
0.81
Y2
0.53
0.81
0.53
0.81
0.81
Z2
1.24
1.24
1.24
1.24
1.24
FluidPour
XI
1.35
1.35
1.35
1.35
1.35
Y1
1.35
1.35
1.35
1.35
1.35
ZI
0.74
0.74
0.74
0.74
0.74
Water
5125
5125
6125
5375
7625
XI 0.88
0.88
1.32
1.32
1.32
YI
0.88
1.32
0.88
1.32
1.32
FluidCubeShake
CubeNumber Water 2173
1
2
2
3322
3322
4858
4983
GranularPush
GranularNumber 2197
4032
5832
9261
12167
Table 3: Scene Parameters for Generating the Interpolate and Extrapolate Datasets. We generate the datasets by varying the shape of container, amount of water, number of cubes, and quantity of the granular material. Z,, X;, Y, are the height, width, and depth for container i. Please refer to Figure for more details.
centimeters). For GranularPush, the pusher is moving over the entire table; we ignore the specific number in this environment as do not have robot arms reference.
Additional dataset samples. We show samples from the FluidPour, FluidCubeShake and Granu- larPush dataset in Figure 10, 11 and 12, respectively. Note that all trajectories for the extrapolate settings are used only for testing and will not show up during the training process. We include more samples from the dataset in the video format in the supplementary video.
16

---

Interpolate
Extrapolate
Figure 10: Samples from FluidPour Dataset. We show sequences of frames over time with an interval of 20 frames. The sequences above the dashed line are for interpolate data, and the bottom images illustrate the extrapolate data.
Interpolate
Extrapolate
Figure 11:
Samples from FluidCubeShake Dataset. We show sequences of frames over time with interval of 20 frames. The sequences above the dashed line are for interpolate data, and the bottom images illustrate the extrapolate data.
B.2 Model Architecture
Image-conditional NeRF. We follow the architectural design by [68]. For the feature encoder, we employ a ResNet-34 backbone to extract features. We use the output layers prior to the first four pooling layers, upsampling them using bilinear interpolation to the same size, and then concatenating these four feature maps. We initialize the weight of the feature extractor of the scene using ImageNet pre-trained weight. For the NeRF function f, We use fully-connected ResNet architecture with ResNet blocks with width of 5 512.
17

---

Interpo late
Extrapolate
Figure 12: Samples from GranularPush Dataset. We show sequences of frames over time with an interval of 20 frames. The sequences above the dashed line are for interpolate data, and the bottom images illustrate the extrapolate data.
Dynamics predictor. For the edge and vertice encoders, Qe and Qu. we use 3-layer fully-connected networks activated by the ReLU function with 150 hidden units. For the propagators, Pe and Pa, we use I-layer fully-connected network followed by ReLU activation. The output dimension of the linear layer is 150.
Sampling 3D points from the trained visual perception module. We sample points on 40 x x0 grid from an area of 55cm x 55cm x 55cm and 63cm x 63cm x 63cm at the center of the table FluidPour and FluidCubeShake respectively, and on a 70 6em x 6cm X 6cm for GranularPush. We evaluate and include points with a density (measured by the occupancy in the predicted neural radiance fields) larger than 0.99. To reduce the total number of points, we subsample the inferred points with FPS with ratio of 5% for FluidPour and 10% for FluidCubeShake and GranularPush.
grid from an area of
Graph building. We set the neighbour distance threshold d to be 0.2, 0.15, 0.15 for FluidPour, FluidCubeShake and GranularPush respectively. We select the threshold so that each point will have on average 20 30 neighbors. Since. in FluidPour, we sample the points with lower density 2000points/mo we use larger threshold for this scenario. For FluidShape and GranularPush. since the density is around 3000 points/m². we cut down the number by 25%.
We found that if the threshold is too small, the performance will degrade significantly since each particle will only receive messages from a few neighbors (and miss out on the larger context). On the other hand, setting the threshold too large will cause the training time increase since the graph will have more edges. We found that setting the threshold around the right scale generally leads to more effective training of reasonable dynamics network.
B.3 Training Details
The models are implemented in PyTorch. We train the perception module using Adam optimizer with learning rate of 1e-4, and we reduce the learning rate by 80% when the performance on the validation set has stopped improving for 3 epochs. To compute the rendering loss when training the perception module, sample 64 points through each ray in the scene and set the ray-batch size of the NeRF query function ff to be 1024 x 32. Training the perception module on a single scenario takes around hours
RTX-3090.
We train the dynamies simulator using Adam optimizer with a learning rate of 1e-4, and we reduce the learning rate by 80% when the performance the validation set has stopped improving for 3
18

---

epochs. The batch size is set to 4. We train the model for 20, 30, and 40 epochs for FluidPour, FluidCubeShake, and GranularPush, respectively. takes around 10 15 hours train the dynamics model in one environment on one single RTX-3090.
B.4 Graph-Based Dynamics Model without Particle-level Correspondence
The velocity an object provides critical information on how the object will move in the future, yet, we do not have access to such information when tracking the object is impossible. As described in Section 3.2, the attributes a vertex v; the built graph consists (1) velocity of this point in the past frames and (2) attributes of the point (rigid, fluid, granular). To get the velocity of a vertex we should have the history position of this vertex. However, since the point clouds are inferred from each frame independently, do not know how each point moves over time since do not have point correspondence between frames.
To address the problem, we leverage the fact that some objects in the scene are easier to track, and we try to use the motion of these trackable objects infer motion for the untrackable units. We assume that know the dense-labeled states of some known fully-actuated shapes like desks and cups connected to the robot arms. Here we will list one specific scenario where a cup of water poured into another cup. In this case, we have two different types of points: points for fluid and points for cups, we name the states of them in time step as V2 = {urr and V = (o's.)) respectively. For the particle encoder Qu if the particle belongs to the cups, then the input of particle encoder contains n history states before to {Vs (v }. the particle belongs to the water, then we have no history states, so the input of Q. is all-zero.
By adding the relative position between receiver and sender points, can pass the momentum of Vp to Vs. Compared with human intuition, we can get an intuitive prediction of the movement of water by simply knowing the past movement the cup without knowing the past movement of water.
Following [47], we use the velocity of points and their relative position as inputs to the dynamics module instead of using the absolute positions of the points. This ensures the model is translation- invariant so the learned dynamics model can be shared across different spatial locations.
B.5 Inference Speed of Our Model
The prediction speed of the dynamics module depends the number of input particles, and it takes around 0.1s for graphs with around 300 nodes in FluidShake and FluidPour, and around 0.2s for scenes with 700+ nodes in GranularPush.
For our visual module, the main time consumption comes from NeRF sampling, it takes 0.2s to sample from grid space introduced in the experiment section of our paper, this run in blocks, with block-size=1000, made up 4G of a V100 GPU. And it can be even faster with larger blocks. The sub-samplin samplimmligg process (FPS, segmentation) is fast since they all written in parallel versions, which takes less than 5ms.
C Potential Society Impact
Our work shows the possibility of learning dynamics models from raw sensory inputs, opening up opportunities to automate the design of differentiable physics engines through data-driven Icarning algorithms. The resulting system can potentially benefit many downstream tasks, including general scene understanding, robotics manipulation, the construction 3D generative models, and inverse tasks like planning/control and inverse design. Furthermore, predictions from model are highly interpretable, which makes it straightforward to explain model behaviors and re-purpose the outputs for other downstream applications.
Though data-driven approaches are potentially more scalable with enough data, concerns still exist that it might be hard to ensure the robustness of the model under sensor noise and adversarial attacks. It also becomes less clear how to fully mitigate data biases. Therefore, bringing in advanced techniques from ML robustness will be one critical future avenue to pursue.
19

---

Figure 13: SAM Working on FluidCube Shake: Recent large segmentation models can well generate masks for different objects in the scene.
D Important Clarifications and Discussions
Q: What is the input of 3D-Inthhys?
Video inputs and object instance masks based on color assumption.
Q: What is the difference of our work compared with [34, 35, 16]?
1. Compared with nerf-dy [34]: our method use explicit 3D representation instead of implicit representation, which we show in our paper that our method can generalize better than [34].
2. Compared with VPGL [35]: (1)[35] requires GT 3D particle visual frontend, while our 3D-IntPhys does not need particle-level supervision (2) In [35], the 3D representation is particle set, which is ordered list. As result, it only generate fixed number of points. In contrast, our method produces dense representations that are not limited ordered sets, making it varying sizes. (3) [35] assumes that we have a learned dynamics model from the simulation as dynamics prior, while learn the dynamics model from the data.
supervision to train the
flexible and adaptable to systems of
3. Compared with Comp nerfdyn [16]: [16]only works rigid objects and rope that exhibits only slight deformation, most the objects do not have topological changes, and they are all constrained to move in 2D plane. So while the object-centric dynamics model used in [16] can solve the tasks in their paper, the object-centric representation is not suitable for learning the complex dynamics of fluids granular materials, in our paper. Our settings contain much more diverse 3D dynamics of challenging materials, which can not be solved using [16].
Q: Is the color segmentation of the fluid objects a reasonable assumption?
It should be noted that the color-based segmentation will not degrade the challenging problem of learning 3D Intuitive Physics, since the task focuses more on learning complex visual dynamics from images.
We want to emphasize that the work focuses more on learning complex visual dynamics from images, as opposed to solving object segmentation in general. Learning fluids dynamics from videos is challenging task, and there are only a few existing works. NeRF-dy the closest to us, yet the model's generalization ability is limited. We have shown in the proposed work that we can significantly improve the generalization ability by operating with a hybrid of implicit and explicit, as opposed to pure implicit, 3D representations, We agree object segmentation is critical visual understanding
20

---

problem, and solving it is an important next step to getting a more general visual dynamics learning framework.
With recent advancements such as SAM [29] and SEER [71], which focus on segmentation in real- world scenarios, the possibility of video segmentation without the need for annotations has emerged (as is shown in Figure 13). This development paves the way for leveraging existing large-scale models enhance the segmentation pipeline, offering great promise for future applications.
Q: Since the fluid has zero velocitys, how to predict the infuitive dynamics?
The intuition is that can infer the water movement from the container's 's movement. We also assume that the imitial velocity water nearly zero, which is also used [50], so the momentum can be gradually passed from the container to the water.
We propose this assumption so that the intuitive physics model can be learned from (1) particles sampled from the neural radiance field, which is not stable (2) point clouds without one-to-one correspondence. The results show that we can learn reasonable dynamics (water poured out from cup. water falling in the container, cubes moving in water, and granular materials pushed away by pusher). It also shows potential of distribution- bbased loss learning visual dynamics.
21


Reviews:
[{"review_id": "FGXZ4A4p9t", "reviewer": "Reviewer_cs73", "summary": "The paper introduces a 3D point-based dynamic model for visual intuitive physics reasoning. Specifically, The model adopts a conditional NeRF as the perception module to extract 3D point clouds from multi-view images. It then employs a graph neural network that operates on the extracted points for forward dynamic predictions.\n\nThe proposed approach achieves state-of-the-art performance across three simulation tasks (FluidPour, FluidCubeShake, GranularPush) i, with evident improvement in both visual reconstruction quality and dynamic prediction accuracy.", "questions": "Most of the questions I have are listed in the Weakness section, and I am willing to adjust my scores accordingly based the authors' responses and other reviewers' comments.\n\nSome additional minor comments:\n- Figure 6.: could be improved with x- and y-labels. I assume x is the horizon, and y is the chamfer loss.\n\nPost-rebuttal update: The rebuttal answered most of my questions and concerns, and also clarified the issues raised by other reviewers. So I stand by my original assessment -- weak accept --- for this submission. Overall the idea and execution are very neat, and I would love to see more follow up work on this one.", "limitations": "The paper includes potential limitations and societal impact in the conclusions and the supplementary.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper improves upon prior work in the following aspects:\n1. The explicit 3D point-based representation gives a good inductive bias for reasoning about local states.\n2. The use of conditional NeRF enables better generalizability to out-of-distribution scenarios compared to the prior art.\n3. The proposed method predicts correct particle arrangement, achieving state-of-the-art results on the benchmark.\n\nCompared to the image-based encoding that suffers from geometry inconsistency, the proposed 2D-to-3D lifting extracts and operates directly on explicit 3D representation, circumventing the inconsistency problem. The direct reasoning on consistent and local geometric structures enables the method to achieve better accuracy on dynamic predictions.\n\nOverall, the paper is well-written, the idea is sound, and the results back up the design choices.", "weaknesses": "While the work has several merits, it also has some limitations/issues:\n- Reliance on instance-level masks. While SAM can provide instance-level masks (as in Supp. 148-152), how the mask quality can impact the performance\n- The acquisition of point attributes a^v and relation a^e is not described. If these features have to be defined manually, then it should also be listed as one of the limitations.\n- An additional stage of training for the conditional NeRF is required. Can the proposed approach work on point clouds extracted from Structure from Motions? If a^v and a^e are not associated with perceptual features, then the PixelNeRF can be replaced with something simpler.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThanks for the careful review and valuable comments. Here we will address the questions raised up by the reviewer:\n\n\n\n> Reliance on instance-level masks. While SAM can provide instance-level masks (as in Supp. 148-152), how the mask quality can impact the performance\n\n\n\nThe instance-level masks in the paper are acquired solely from color information, which are not acquired from ground-truth masks. In the **additional pdf file** in General Response, we demonstrate that the masks we use still have unmarginal difference from ground-truth masks, possibly due to points with ambiguous colors. However, despite this challenge, our pipeline can still effectively learn reasonable dynamics using the same input. Meanwhile, to make it a fair comparison, we also provide the color segmentation as input in baseline methods like [34].\n\nFurther we did experiments on mask quality to show that our method is robust to the segmentation quality: we perturb the segmentation by 1% and 5% on the Env-1 settings of FluidPour:\n\n|    | Loss@T= 5 |Loss@T=20 | Loss@T=35|\n| ---- | ---- | ----- | ----- |\n| Perturb 0%   |  0.020  | 0.030   | 0.020   |\n| Perturb 1% | 0.020   | 0.040   |0.030   |\n| Perturb 5% | 0.020   | 0.045   |0.028   |\n\n\n\n\n\nAs for SAM, we compare the SAM segmentation as a strong visual prior on FluidPour and find that it can effectively segment the water out, which means that we can use  SAM on our scenes as an alternative. We sample 10 frames and 3 views from each settings and compute the IoU (%) of SAM and our color-based segmentation of water:\n\n|    | Env-1|Env-2|Env-3|Env-4|Env-5|\n| ---- | ---- | ----- | ----- | ----- | ----- |\n| View-1   |  100  | 100   |100   |100   | 98 |\n| View-3 | 99   | 99   |98   | 99   | 96 |\n| View-5 | 100   |100   |100   |100   | 98 |\n\n\n\n> The acquisition of point attributes a^v and relation a^e is not described. If these features have to be defined manually, then it should also be listed as one of the limitations.\n\n\n\nThe point attributes $a^v$ and $a^e$ are one-hot features which are used to define the physical properties of points (e.g. fluid or rigid). These attributes are defined after the segmentation, which are widely used in point-based dynamics learners. While  [35] can learn physical properties automatically, it needs a trained dynamics prior on GT 3D particle sets and have stronger assumption about the 3D representation as ordered particle sets.\n\nYet, we want to emphasize that the work focuses more on learning complex visual dynamics from images, as opposed to solving object discovery and feature discovery in general.  Learning fluids dynamics from videos is a challenging task, and there are only a few existing works. NeRF-dy is the closest to us, yet the model's generalization ability is limited. We have shown in the proposed work that we can significantly improve the generalization ability by operating with a hybrid of implicit and explicit, as opposed to pure implicit, 3D representations. \tThe future investigation may include using LLM to provide dynamics prior knowledge giving the physical properties of the objects (e.g. fluids or rigid body).\n\n\n\n\n\n> An additional stage of training for the conditional NeRF is required. Can the proposed approach work on point clouds extracted from Structure from Motions? If a^v and a^e are not associated with perceptual features, then the PixelNeRF can be replaced with something simpler.\n\n\nWe need an additional stage of training conditional NeRF, which is actually effective: we only trained on a small subset of videos ($\\sim$  50), and the conditional NeRF can generalize to the rest of the scenes ($\\sim$ 450) in the dataset, which shows that conditional NeRF is effective and can generalize well to extrapolated settings.\n\nOn the other hand, Structure from Motion (SfM) primarily produces the surface representation of point clouds using multi-view inputs, which may not be ideal for learning fluid dynamics. Conditional Neural Radiance Fields (NeRF) are trained on a bunch of videos including thousands of different frames, enabling them to learn valuable biases and effectively reconstruct the inner points of fluids.\n\nFurthermore, it is important to note that SfM offers a sparse representation of the scene, whereas NeRF provides dense reconstruction. This characteristic makes NeRF more versatile, allowing it to handle sparse sampling with greater flexibility.\n\n\n\n> Figure 6.: could be improved with x- and y-labels. I assume x is the horizon, and y is the chamfer loss.\n\nYes, x is horizon and y is the chamfer loss of predicted states. We will add the detailed x, y labels for better understanding.\n\n\n"}, {"review_id": "mBUQehetMF", "reviewer": "Reviewer_493i", "summary": "Authors propose an approach to learn physics based model of real world objects from multi-view video. Authors argue that 3D representation of the scene is important to reason about the physics and use per-frame Nerf to extract the object point cloud from every frame of the multi-view video. They build a graph of the object points and use spatial message passing to learn the dynamics.\nAuthors use Nvidia FleX to generate simulated data for testing and compare their method with NeRF-dy (Li et al. CoRL'21).", "questions": "I'm not an expert in this area. I like the ideas presented here as they are simple and make intuitive sense. However I'm concerned about the novel contributions of the work. Please see pts. 1,2 above. I'd be happy to update my score after the rebuttal.", "limitations": "Authors briefly mention one of the limitations along with the conclusions but a more in-depth analysis would be appreciated. This could go in the supp. mat. and authors can at least leave a pointer in the main paper.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The proposed method is simple and intuitive.\n+ The idea of leveraging Nerf to build a 3D model of the world and using it to model physics is interesting.\n+ The method compares favourably to baseline.", "weaknesses": "[Technical]\n1. Sec 3: Proposed method consists of two steps, extracting the scene point cloud using Nerf [41] and computing dynamics using message passing [7,36,37].\nIsn’t this similar to [35] where they predict 3D particle locations using NN and then use message passing + some other learning to predict the dynamics? If Nerf produces better points why can’t we replace the MLP based points from [35] with Nerf and use rest of the method? They also don’t make assumptions about physical properties of objects (also see pt. 2).\nIs the key idea here that [35] requires GT 3D supervision whereas the proposed method does not? But doesn't having multi-view images with known cameras (as used in the method) sort of provide this supervision?\n\n2. L154: what is captured in point attribute a_i^v? How is it obtained? Is it known beforehand? Adding dimensionality to variables would also help. \nUnlike proposed method, prior work [35] does not require prior knowledge about the physical properties of the object or the scene. Isn’t the proposed method restrictive? Please correct me if I misunderstood this.\n\n3. L141: How is object segmentation obtained? Is it provided as input? All inputs and outputs must be clearly explained.\nA good idea would be to mention these in the intro to method section. Similarly all assumptions about the problem setting such as in L168-171 should also be clarified early on.\n\n[Minor]\nL106-109: Authors write they learn 3D-IntPhysics without any 3D supervision which is not entirely true. They use multi-view images with known camera for Nerf based representation which is also not entirely \"not 3D\".\nAuthors also say that prior work uses GT trajectories which are hard to obtain in real setting, but getting multi-view images with GT camera parameters is also not trivial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for reviewing our work and providing valuable comments. Below, we try to address the questions you raised:\n\n\n\n\n\n> Proposed method consists of two steps, extracting the scene point cloud using Nerf [41] and computing dynamics using message passing [7,36,37]. Isn’t this similar to [35] where they predict 3D particle locations using NN and then use message passing + some other learning to predict the dynamics? If Nerf produces better points why can’t we replace the MLP based points from [35] with Nerf and use rest of the method? They also don’t make assumptions about physical properties of objects (also see pt. 2). Is the key idea here that [35] requires GT 3D supervision whereas the proposed method does not? But doesn't having multi-view images with known cameras (as used in the method) sort of provide this supervision?\n\nThe settings of VPGL [35] differs a lot from our paper: (1) [35] requires GT 3D particle set as supervision to train the visual frontend, while our 3D-IntPhys does not need particle-level supervisions (2) In [35], the 3D representation is a particle set, which is an ordered list. As a result, it can only generate a fixed number of points. In contrast, our method produces dense representations that are not limited to ordered sets, making it more flexible and adaptable to systems of varying sizes. (3) [35] assumes that we have a learned dynamics model from the simulation as a dynamics prior, while we learn the dynamics model from the data.\n\nSimply replacing MLP in VGPL with NeRF will not work. First, the particle set  is  an ordered list, so that we can infer the physical properties using MLP directly. Also, [35] still needs to train a dynamics prior on dense 3D GT obtained from a simulator, which is not applicable in our settings, since we aim to directly learn from visual observations. Though VGPL do not make assumptions about the physical properties, it  makes more assumptions about the ordered particle sets and the dynamics prior to do inference about the physical properties in the scene.\n\n\n\n\n> L154: what is captured in point attribute a_i^v? How is it obtained? Is it known beforehand? Adding dimensionality to variables would also help. Unlike proposed method, prior work [35] does not require prior knowledge about the physical properties of the object or the scene. Isn’t the proposed method restrictive? Please correct me if I misunderstood this.\n\n\nThe point/edge attributes $a^v$ and $a^e$ are one-hot features which are used to define the physical properties of points (e.g.  fluids are [0,0,1], rigid objects are [0, 1, 0], containers are [0, 0, 1]). These attributes are defined after the segmentation, which are widely used in point-based dynamics learners.\n\nAs we mentioned in the last question, though VGPL do not make assumptions about the physical properties, it needs more assumptions about the 3D representation (ordered particle set) and the pre-trained dynamics prior on 3D GT point-clouds to do inference about the physical properties in the scene.\n\n\n\n\n\n\n\n\n> L141: How is object segmentation obtained? Is it provided as input? All inputs and outputs must be clearly explained. A good idea would be to mention these in the intro to method section. Similarly all assumptions about the problem setting such as in L168-171 should also be clarified early on.\n\n\nWe thank the reviewers for pointing this out. The object segmentation in our paper is acquired using color informations, where the input is actually some prior information in forms of (color, object) pairs, e.g. (blue, water), (yellow, sand), (green, box). By leveraging this paired information, we effectively segment the objects from the scene.\n\n\nYet, we want to emphasize that the work focuses more on learning complex visual dynamics from multi-view images, which itself is a challenging tasks, and there are only a few existing works. NeRF-dy is the closest to us, yet the model's generalization ability is limited. We have shown in the proposed work that we can significantly improve the generalization ability by operating with a hybrid of implicit and explicit, as opposed to pure implicit, 3D representations. Additionally, visual foundation models like SAM have demonstrated impressive performance in segmentation, achieving results nearly identical to our method, as discussed in our **general response**. Therefore, SAM could be considered an alternative to color-based segmentation.\n\n\n\n\n> [Minor] L106-109: Authors write they learn 3D-IntPhysics without any 3D supervision which is not entirely true. They use multi-view images with known camera for Nerf based representation which is also not entirely \"not 3D\". Authors also say that prior work uses GT trajectories which are hard to obtain in real setting, but getting multi-view images with GT camera parameters is also not trivial.\n\n\nIn the table-top settings we considered in this paper, obtaining the camera parameters through calibration is not particularly hard to implement, and dataset of as much as eight cameras have be created [3].\n\nYet, from 2D inputs combined with camera parameters to 3D representations is still a 2D-to-3D lift [1]. Also, some work has demonstrates the potential to release the camera poses from learning NeRF   [2].\n\n\n\n[1] Visual Reinforcement Learning with Self-Supervised 3D Representations.\n\n[2] MonoNeRF: Learning Generalizable NeRFs from Monocular Videos without Camera Poses.\n\n[3] DexYCB:: A Benchmark for Capturing Hand Grasping of Objects\n\n\n\n\n\n> Authors briefly mention one of the limitations along with the conclusions but a more in-depth analysis would be appreciated. This could go in the supp. mat. and authors can at least leave a pointer in the main paper.\n\n\nThanks for your advice, we will include a more detailed discussion about the limitation of our work and add a pointer in the main paper.\n"}, {"review_id": "BhAB0PbCGH", "reviewer": "Reviewer_R6LT", "summary": "This paper proposes a method that learns to approximate physical interactions of primitive scenes from a set of videos through the use of of a NeRF and a dynamics prediction model. A static NeRF (based on PixelNeRF) is used to reconstruct a per time-step point cloud which is then used to supervise the dynamics prediction module via a Chamfer loss. A regularization is introduced in Eq. 7 of the paper to prevent the dynamic module to settle for a degenerate solution. ", "questions": "How sensitive is the dynamics learning training to the parameter $\\delta$ (line 153)? An ablation of a few values will aid in understanding the robustness of the system \n\n", "limitations": "Yes.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) Qualitatively the proposed method outperforms prior art.\n\n2) The paper is well written and motivated", "weaknesses": "1) No videos of interpolated or extrapolated dynamics are provided. MSE itself is not a good measure of the good a simulation is.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n\nThank you for reviewing our work and providing valuable comments. Below, we try to address the questions you raised:\n\n\n\n> No videos of interpolated or extrapolated dynamics are provided. MSE itself is not a good measure of the good a simulation is.\n\n\n\nWe kindly request the reviewers to watch the supplementary videos showcasing our results, which can be accessed through the anonymous link provided in Line:3 of supplement material of our submission. Your time and attention to these videos will provide valuable insights into the effectiveness and performance of our approach.\n\n\n\n> How sensitive is the dynamics learning training to the parameter $\\delta$ (line 153)? An ablation of a few values will aid in understanding the robustness of the system\n\n\nThat is a good question. In Line:58 of the supplementary material, we explain the choice of $\\delta$: we select the threshold so that each point will have on average 20$\\sim$30 neighbors. Through extensive experiments, we found that 20$\\sim$30 neighbors is a sweet point: when $\\delta$ is too small, powers from surrounding nodes are weakly propagated, when $\\delta$ is too large, the interactions between points becomes redundant and is too slow to run.\n\nFollowing the advice from the reviewers, we do experiments on ablation on different $\\delta$ value, and more results are put in the **additional pdf** of the **General Response** section.\n\n\n\n\n\n\n"}, {"review_id": "XPRLvkF4zy", "reviewer": "Reviewer_kxU1", "summary": "The paper proposes to learn intuitive physics from visual input by learning to infer the 3D scene structure and dynamics. The claim is that this explicit 3D reasoning generalizes better than existing implicit representations, and can be learned from videos without any explicit 3d supervision. To this end, the method uses PixelNeRF to learn the 3D representation of the scene and uses a graph-based neural network to predict the dynamical evolution of the scene. Evaluation is done in three simulated dataset and shows better performance compared to NeRF-dy and autoencoder models.", "questions": "How does the method perform on real-world videos?", "limitations": "The main limitations is it's unclear how the method will generalize to real-world scenarios.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "3D geometry and compositionality are key components for inferring physical dynamics, so the paper takes the right approach in enforcing 3D and compositionality in the model.", "weaknesses": "The paper does not show any results on real data. If the model is able to learn only from multiview videos the claims would have been stronger with real data.\nComparison with [16] is missing in the paper.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The main limitations is it's unclear how the method will generalize to real-world scenarios.\n\nThanks for your advice about real-world experiments. Our paper is among the first to embed explicit 3D information and strong inductive bias into 3D intuitive physics learning.\n\n\nHere we  want to emphasize that the work focuses more on learning complex visual dynamics from multi-view images, which itself is a challenging tasks, and there are only a few existing works. NeRF-dy is the closest to us, yet the model's generalization ability is limited. We have shown in the proposed work that we can significantly improve the generalization ability by operating with a hybrid of implicit and explicit in simulation datasets, as opposed to pure implicit, 3D representations.\n\nWith stronger vision prior knowledge (object discovery, segmentation, e.g. SAM and SEER) in the future, we can try more exciting experiments in real world, making it possible to learn from bunch of videos around us.\n\n\n\n> If the model is able to learn only from multiview videos the claims would have been stronger with real data. Comparison with [16] is missing in the paper.\n\nCompared with Compnerf [16]: [16] only work on rigid objects and weakly-deformable ropes, most of the objects do not have topological changes, and they are all placed on the table. So  [16] uses object-centric dynamics model, which is not suitable for learning complex dynamics of fluids or granular materials. Our settings contain much more diverse 3D dynamics of challenging materials, which can not be solved using [16].\n"}, {"review_id": "fH5HAQQd10", "reviewer": "Reviewer_SEVF", "summary": "This paper proposed a combination of a NeRF-style perception module and a 3D point-based dynamics module to model challenging physical phenomena involving fluid, granular materials, and rigid objects. The perception module learns spatial-equivariant representations and transforms the environment into points. The dynamics module then exploits the compositional structure of the point systems with GNNs. \n\nThe proposed method significantly outperformed existing models that do not employ explicit 3D representations. The method also shows strong long-horizon prediction and generalization to complex scenarios.", "questions": "- The training of NeRF only promises the quality of object surfaces and does not impose any constraint on the interior of an object. As a result, the object occupancy field may only represent the surfaces of each object. I would assume that such surfaces are not good for modeling physical interactions, especially fluid-related ones. Why can your model reconstruct and predict the dynamics accurately?\n- In Figure 2, does the Merge Loss refer to the spacing loss in Eq. 7?\n- In section 4, how is the Structural Similarity Index Measure (SSIM) defined? What does it measure?\n- The differences shown in Table 1 are impressive. But what are the units for MSE and SSIM?\n- In Table 1, the GranularPush OoD results are significantly worse than the other cases. What are the reasons and implications for the difference?", "limitations": "The proposed method relies on point colors for segmentation, which implies that there must be a one-to-one mapping between color and instance, and the colors must be distinguishable. In addition, the fluid must be colored and non-transparent, which further limits the application of the proposed method. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The proposed method can model complex dynamics without strong supervision by GT 3D point trajectories, which are only available in simulated environments. It also generalizes well to unseen environments with extrapolated parameters due to its explicit point-based representation. ", "weaknesses": "- This paper has omitted some details of the proposed method, including how the dataset is constructed, how each module is designed, how evaluation metrics are computed, and how particle velocities are computed. For contents that can be found in the supplementary material, a pointer in the main text would be better appreciated. For others, see Questions.\n- In addition, since this paper claims that it can learn complex physical dynamics from purely visual supervision, it would be more interesting if there was a real-world demonstration of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n\nThank a lot for identifying our work and providing valuable comments. Below, we try to address the raised questions:\n\n\n\n> The training of NeRF only promises the quality of object surfaces and does not impose any constraint on the interior of an object. As a result, the object occupancy field may only represent the surfaces of each object. I would assume that such surfaces are not good for modeling physical interactions, especially fluid-related ones. Why can your model reconstruct and predict the dynamics accurately?\n\n\nThat is a good question. Actually, we are excited to find that, with the strong color prior imposed by conditional NeRF, the interior of fluids can be filled (refer to **additional pdf** in **General Responce** for more visualization results). \n\nThe reason may be: if the NeRF model is only trained on one frame of pouring water, with high probability only the surface will have density. However, if we train it on bunch of videos of water pouring, the NeRF function f(color, position) will be strongly conditioned on color, which can predict the point as water if the point is colored blue from many views. We put more visualizations of the NeRF in the **additional pdf**.\n\n\n> In Figure 2, does the Merge Loss refer to the spacing loss in Eq. 7?\n\nYes. We will rephrase it as spacing loss for better expression.\n\n\n\n\n\n\n\n\n\n> In section 4, how is the Structural Similarity Index Measure (SSIM) defined? What does it measure?\n\nThe Structural Similarity Index Measure (SSIM) is a widely used image quality assessment metric. It is designed to quantify the similarity between two images, specifically evaluating the perceptual quality and structural information present in the images. SSIM was introduced as an improvement over traditional metrics like Mean Squared Error (MSE) or Peak Signal-to-Noise Ratio (PSNR), as it takes into account the human visual perception characteristics.\n\nThe detailed definition can be found in [1].\n\n\n[1]: Zhou Wang and A. C. Bovik, \"A universal image quality index,\" in IEEE Signal Processing Letters, vol. 9, no. 3, pp. 81-84, March 2002, doi: 10.1109/97.995823.\n\n\n\n\n\n\n> The differences shown in Table 1 are impressive. But what are the units for MSE and SSIM?\n\nThe unit for MSE is pixel value squared, and the unit for SSIM is a fraction from 0 to 1.\n\n\n\n> In Table 1, the GranularPush OoD results are significantly worse than the other cases. What are the reasons and implications for the difference?\n\nFirst, Granular Push is a more challenging task because: (1) the diversity of fluids shapes is much larger than FluidPour and FluidCubeShake (at most time the fluids are restricted in the container), (2) the number of points is larger, making it easier to spread in the simulator.\n\nAmong GranularPush, Granular OoD cases contain 12167 points, which is more than any other settings. The loss will be larger, but the results are still reasonable.\n"}]

Author rebuttal:
# General Response

We thank the reviewers for their valuable comments on our paper. We are excited to see that reviewers (Reviewer cs73, 493i, R6LT, kxU1, SEVF) identified the novelty of our technical contribution, acknowledged the better performance of our methods over baselines (Reviewer cs73, 493i, R6LT, kxU), and found the paper well-written and motivated (Reviewer cs73, R6LT).



Here we try to address some commonly asked questions:



> Comparison with existing SOTA of 3D intuitive physics [34], [35] and [16] (Reviewer 493i)

- Compared with nerf-dy [34]: our method use explicit 3D representation instead of implicit representation, which we show in our paper that our method can generalize better than [34].
---
- Compared with VPGL [35]: (1) [35] requires GT 3D particle set as supervision to train the visual frontend, while our 3D-IntPhys does not need particle-level supervision (2) In [35], the 3D representation is a particle set, which is an ordered list. As a result, it can only generate a fixed number of points. In contrast, our method produces dense representations that are not limited to ordered sets, making it more flexible and adaptable to systems of varying sizes. (3) [35] assumes that we have a learned dynamics model from the simulation as a dynamics prior, while we learn the dynamics model from the data.
---

- Compared with Comp_nerfdyn [16]: [16] only works on rigid objects and a rope that exhibits only slight deformation, most of the objects do not have topological changes, and they are all constrained to move in a 2D plane. So while the object-centric dynamics model used in [16] can solve the tasks in their paper, the object-centric representation is not suitable for learning the complex dynamics of fluids or granular materials, as in our paper. Our settings contain much more diverse 3D dynamics of challenging materials, which can not be solved using [16].



>  How to get the point attributes $a^v, a^e$ (Reviewer cs73, 493i)

The point attributes $a^v$ and $a^e$ are one-hot features which are used to define the physical properties of points (e.g. fluid or rigid). These attributes are defined after the segmentation, which are widely used in point-based dynamics learners.



> About color segmentation. (Reviewer R6LT, 493i)


We want to emphasize that the work focuses more on learning complex visual dynamics from images, as opposed to solving object discovery and segmentation in general. Learning fluid dynamics from videos is a challenging task, and there are only a few existing works. NeRF-dy is the closest to us, yet the model's generalization ability is limited. We have shown in the proposed work that we can significantly improve the generalization ability by operating with a hybrid of implicit and explicit, as opposed to pure implicit, 3D representations. We agree object segmentation is a critical visual understanding problem, and solving it is an important next step to getting a more general visual dynamics learning framework.

Also, we noticed that stronger visual foundation models like SAM and SEER are proposed as excellent segmentation tools. We further do experiments to show that it is possible to use SAM as an alternative, specifically, we compare  SAM on FluidPour environments and find that it can effectively segment the water out. We sample 10 frames and 3 views from each setting and compute the IoU (%) of SAM and our color-based segmentation of water:

|    | Env-1|Env-2|Env-3|Env-4|Env-5|
| ---- | ---- | ----- | ----- | ----- | ----- |
| View-1   |  100  | 100   |100   |100   | 98 |
| View-3 | 99   | 99   |98   | 99   | 96 |
| View-5 | 100   |100   |100   |100   | 98 |



> About the video visualization (Reviewer R6LT)
 
We kindly refer the reviewer to the video submitted as a link in the supplementary material (Line 3). The video was uploaded anonymously before the submission deadline.




Meta-review: This paper appears was unanimous accept by the reviewers. They all appreciated the novelty and improvement over baselines, found the paper well-written and motivated, and felt the paper is a good contribution to the field.  The author rebuttal was reviewed and commented on by all with healthy discussion between the authors and the reviewers and the AC.  

Congratulations!

In revising your paper for the camera-ready, please include and comments and clarifications that were made or proposed during the rebuttal and discussion phase.
Decision: Accept (poster)