SlowGuess/NeurIPS_2023_OCR huggingface dataset:
c) GranularPush. This environment contains a fully-actuated board pushing a pile of granular pieces. We design the extrapolate dataset to have a larger quantity of granular objects in the scene than the model has ever seen during training.
Baselines. We compare our method with two baselines, NeRF-dy [34] and autoencoder (AE) (similar to GQN [18] augmented with a latent-space dynamics model). NeRF-dy is a 3D-aware framework that also learns intuitive physics from multi-view videos. Yet, instead of learning the object dynamics with explicit and compositional 3D representations, the model learns dynamics models with implicit 3D representations in the form of a single latent vector. We also compare our method with an autoencoder-based reconstruction model (AE) [18] that can perform novel-view synthesis but is worse at handling 3D transformations than neural implicit representations. AE first learns scene representations through per-frame image reconstruction, and then it learns a dynamics model on top of the learned latent representations. All methods take RGB images and camera parameters as inputs. To incorporate object-level information, we perform color-based segmentation to obtain object masks as additional inputs to the baselines. The implementation details and parameter settings of our method can be found in the supplementary materials.
FluidPour	FluidCubeShake	GranularPush
Metrics	Model	InD	OoD	InD	OoD	InD	OoD
MSE(↓)	AE	451.03	542.86	869.3	1727.55	562.06	1537.2
NeRF-dy	202.95	317.27	527.46	1585.97	481.95	1020.0
Ours	111.66	124.33	66.52	81.38	147.97	646.85
SSIM(↑)	AE	0.86	0.84	0.71	0.86	0.81	0.62
NeRF-dy	0.89	0.86	0.73	0.65	0.81	0.61
Ours	0.90	0.89	0.94	0.93	0.89	0.69
Table 1: Quantitative Results of the Perception Module. We compare our method with autoencoder (AE) and NeRF-dy [34] with additional instance masks based on color. We measure the quality of rendered images by computing the Mean Squared Error (MSE) and Structural Similarity Index Measure (SSIM) compared to the ground truth. InD stands for in-distribution tests, and OoD stands for out-of-distribution tests.
Figure 5: Qualitative Reconstruction Results of the Perception Module. The images generated by our method contain more visual details and are much better aligned with the ground truth. Our model is much better at handling large scene variations than NeRF-dy, especially in extrapolate settings.
4.1 Image Reconstruction From Learned Scene Representations
We test how well the perception modules capture scene information by evaluating the visual front-end of all models on their ability to reconstruct the observed scene from the inferred representations. We measure the difference between the reconstructed and ground truth images with Mean Squared Error (MSE) and Structural Similarity (SSIM) in pixel level (Table 1). Our perception module outperforms all baselines in all three environments. The performance gap is exaggerated in extrapolate settings, especially in scenarios that involve complex interactions between rigid and deformable materials (Figure 5 qualitative comparisons).
4.2 Learned Visual Dynamics On In-Distribution Held-Out Scenes
Next, we compare long-term rollouts in the 3D space. We evaluate the models using the Chamfer distance between the predicted point cloud and the ground truth. For NeRF-dy, we decode the predicted rollouts latent vectors into the point cloud with the learned NeRF decoder. We exclude the comparison with AE since it is unclear how to decode the learned representations into point clouds. We show quantitative comparison in Figure 6 and qualitative results in Figure 4. 3D-IntPhys can
Figure 6: Quantitative Results of the Dynamics Module. This figure compares our method and NeRF-dy [34] on their long-horizon open-loop future prediction loss. The loss is measured as the Chamfer distance between the predicted particle set evolution and the actual future. Our method outperforms the baseline in both interpolate and extrapolate settings, showing the benefits of explicit 3D modeling.
Figure 7: Strong Generalization Ability of the Dynamics Module to Wider Pushers. We evaluate our dynamics model on unseen width of pushers in GranularPush environment. The left part shows in 3D space where red indicates granular materials, green shows the table and pusher, and the arrow shows how the pusher is about to move. The right part shows from the top view of the rendering results.
learn reasonable scene dynamics in all scenarios and significantly outperforms NeRF-dy. While NeRF-dy can learn relatively reasonable movements of fluids, it fails to learn complex dynamics such as the floating cube and the morphing of the granular materials. The results suggest that the proposed explicit 3D point-based representations are critical to learning complex multi-material dynamics.
4.3 Generalization on Out-of-Distribution Scenes
To test the generalization ability of the models, we introduce extrapolate settings of all of the three scenarios. See "Extrapolate" results in Table 1, Figure 5, 6, and 4. The proposed 3D-IntPhys generalizes well to extrapolate settings both at the visual perception stage and the dynamics prediction stage, whereas NeRF-dy and autoencoder both fail at generalizing under extrapolate settings. For example, in FluidShake, both baselines cannot capture the number and the color of the rigid cubes (Figure 5). And in GranularPush, both baselines fail to capture the distributions of the granular materials. NeRF-dy performs much worse on extrapolation scenes compared to in-distribution scenes, suggesting that incorporating 3D information in an explicit way, as opposed to implicit, is much better at capturing the structure of the underlying environment, thus leading to better generalization. We further test our model on completely unseen changes to the environment – in the GranularPush environment, we extend the width of the pusher by a factor of 2 and 5. Though the stretched pusher has never shown in the training data, our model can make reasonable pushing predictions (see Fig 7).
5 Conclusions
In this work, we propose a 3D-aware and compositional framework, 3D-IntPhys, to learn intuitive physics from unlabeled visual inputs. Our framework can work on complex scenes involving fluid,
rigid objects, and granular materials, and generalize to unseen scenes with containers of different sizes, more objects, or larger quantities of fluids and granular pieces. We show the proposed model outperforms baselines by a large margin, highlighting the importance of learning dynamics models in an explicit 3D representations space. The major limitation of our work is the assumption of access to object masks. However, with the progress on segmentation in the wild [71, 29], we believe that it will be possible to get such kinds of masks in real-world 3D environments. Our work serves as a pioneer in visual intuitive physics learning of complex scenes, and it is an exciting future direction to learn more complex intuitive physics from real-world data with the help of these large models.
[57] B. Ummenhofer, L. Prantl, N. Thuerey, and V. Koltun. Lagrangian fluid simulation with continuous convolutions. In International Conference on Learning Representations, 2019.
[58] R. Veerapaneni, J. D. Co-Reyes, M. Chang, M. Janner, C. Finn, J. Wu, J. B. Tenenbaum, and S. Levine. Entity abstraction in visual model-based reinforcement learning. CoRR, abs/1910.12827, 2019.
[59] C. Vondrick, H. Pirsiavash, and A. Torralba. Anticipating the future by watching unlabeled video. CoRR, abs/1504.08023, 2015.
[60] M. Watter, J. Springenberg, J. Boedecker, and M. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. Advances in neural information processing systems, 28, 2015.
[61] N. Watters, D. Zoran, T. Weber, P. Battaglia, R. Pascanu, and A. Tacchetti. Visual interaction networks: Learning a physics simulator from video. Advances in neural information processing systems, 30, 2017.
[62] B. Wu, S. Nair, R. Martin-Martin, L. Fei-Fei, and C. Finn. Greedy hierarchical variational autoencoders for large-scale video prediction. CoRR, abs/2103.04174, 2021.
[63] Z. Xu, Z. He, J. Wu, and S. Song. Learning 3d dynamic scene representations for robot manipulation. In Conference on Robotic Learning (CoRL), 2020.
[64] Z. Xu, J. Wu, A. Zeng, J. B. Tenenbaum, and S. Song. Densephysnet: Learning dense physical object representations via multi-step dynamic interactions. In Robotics: Science and Systems (RSS), 2019.
[65] T. Xue, J. Wu, K. L. Bouman, and W. T. Freeman. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks. In Advances In Neural Information Processing Systems, 2016.
[66] Y. Ye, D. Gandhi, A. Gupta, and S. Tulsiani. Object-centric forward modeling for model predictive control. In CoRL, 2019.
[67] Y. Ye, M. Singh, A. Gupta, and S. Tulsiani. Compositional video prediction. In International Conference on Computer Vision (ICCV), 2019.
[68] A. Yu, V. Ye, M. Tancik, and A. Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578-4587, 2021.
[69] M. Zhang, S. Vikram, L. Smith, P. Abbeel, M. J. Johnson, and S. Levine. SOLAR: deep structured latent representations for model-based reinforcement learning. CoRR, abs/1808.09105, 2018.
[70] R. Zhang, J. Wu, C. Zhang, W. T. Freeman, and J. B. Tenenbaum. A comparative evaluation of approximate probabilistic simulation and deep neural networks as accounts of human physical scene understanding. CoRR, abs/1605.01138, 2016.
[71] X. Zou, J. Yang, H. Zhang, F. Li, L. Li, J. Gao, and Y. J. Lee. Segment everything everywhere all at once. arXiv preprint arXiv:2304.06718, 2023.
A Additional Results
To better understand the performance of our framework visually, we prepare test time rollouts of our framework as well as those of various baselines in the supplementary video. The video is published anonymously and can be accessed in https://sites.google.com/view/3d-intphys
A.1 Ablation Study
We find that training the model with Chamfer distance in dense scenes with granular materials will often lead to predictions with unevenly distributed points where some points stick too close to each other. To alleviate the issue, we introduce the spacing loss to penalize the distance between these points. We set the threshold of penalty  $d_{min}$  to be 0.08 and the loss weight  $\sigma$  to be 10. We find that spacing loss can help improve the performance of the dynamics learner especially under extrapolate settings, as shown in Figure 8. We provide qualitative results in the supplementary video.
Figure 8: Ablation Study on the Spacing Loss. Training dynamics models in the GranularPush scenario with spacing loss results in better rolling prediction. The performance gap is even more substantial in the extrapolate setting.
B Implementation Details
B.1 Dataset Generation
Our datasets are generated by the NVIDIA Flex simulator. Each of the three scenarios (Pour, Shake and Push) has 500 videos of trajectories taken from 6 views, with each trajectory consisting of 300 frames. We manually select the 6 views with reasonable coverage of the tabletop space to minimize the occlusion. The 500 trials are generated from five different sets of environmental parameters, detailed in Table 3. We take one set of parameters that are outside the training distribution as the extrapolate dataset for evaluating model generalization. For the rest of the four settings, we randomly split them into train and test sets with a ratio of 0.8.
Next, we provide more details for each scenario:
- In the FluidPour environment, we randomly initialize the position of the upper container and then generate random back-and-forth actions by tilting the container. The action space is then the position and tilting angle of the upper container.
- In FluidCubeShake, we also randomly initialize the position of the container and the cubes inside the container. We then generate random but smooth action sequences moving the container in the 2D plane. The action space is then the x-y location of the container.
- In GranularPush, we randomly initialize the position of the granular pile. Then, for each push, we randomly generate the starting and ending positions of the pusher and move the pusher along the straight line with an angle perpendicular to the pushing direction. The action space is a four-number tuple stating the starting and ending position on the 2D plane.
The following table shows the moving range of the robot arms in the FluidPour and FluidCubeShake environments after normalizing the robot into a size that is the same as in the real world (unit:
X-Range	Y-Range	Z-Range
FluidPour	[-29.11, -12.66]	[42.00, 60.00]	[-7.78, 7.78]
FluidCubeShake	[-3.25, 42.25]	[19.25, 19.25]	[-24.50, 24.00]
Figure 9: Illustration of the Environment Settings. In the FluidPour scenario, a robot arm holds a container and tries to pour some fluid into another container. In the FluidShake scenario, a robot moves a container with some fluid and cubes. We show the parameters for the container shape referred in Table 3.
Table 2: Robot Action Space(centimeters): we show the range the robot arms can move in the FluidPour and FluidCubeShake environments.
SceneName	Params	Env1	Env2	Env3	Env4	Extrapolate
FluidPour	X2	0.53	0.53	0.81	0.81	0.81
Y2	0.53	0.81	0.53	0.81	0.81
Z2	1.24	1.24	1.24	1.24	1.24
X1	1.35	1.35	1.35	1.35	1.35
Y1	1.35	1.35	1.35	1.35	1.35
Z1	0.74	0.74	0.74	0.74	0.74
AmountofWater	5125	5125	6125	5375	7625
FluidCubeShake	X1	0.88	0.88	1.32	1.32	1.32
Y1	0.88	1.32	0.88	1.32	1.32
CubeNumber	1	1	2	2	3
Water	2173	3322	3322	4858	4983
GranularPush	GranularNumber	2197	4032	5832	9261	12167
Table 3: Scene Parameters for Generating the Interpolate and Extrapolate Datasets. We generate the datasets by varying the shape of container, amount of water, number of cubes, and quantity of the granular material.  $Z_{i}, X_{i}, Y_{i}$  are the height, width, and depth for a container  $i$ . Please refer to Figure 9 for more details.
centimeters). For GranularPush, the pusher is moving over the entire table; we ignore the specific number in this environment as we do not have robot arms as a reference.
Additional dataset samples. We show samples from the FluidPour, FluidCubeShake and GranularPush dataset in Figure 10, 11 and 12, respectively. Note that all trajectories for the extrapolate settings are used only for testing and will not show up during the training process. We include more samples from the dataset in the video format in the supplementary video.
Figure 10: Samples from FluidPour Dataset. We show sequences of frames over time with an interval of 20 frames. The sequences above the dashed line are for interpolate data, and the bottom images illustrate the extrapolate data.
Figure 11: Samples from FluidCubeShake Dataset. We show sequences of frames over time with an interval of 20 frames. The sequences above the dashed line are for interpolate data, and the bottom images illustrate the extrapolate data.
B.2 Model Architecture
Image-conditional NeRF. We follow the architectural design by [68]. For the feature encoder, we employ a ResNet-34 backbone to extract features. We use the output layers prior to the first four pooling layers, upsampling them using bilinear interpolation to the same size, and then concatenating these four feature maps. We initialize the weight of the feature extractor of the scene using ImageNet pre-trained weight. For the NeRF function  $f$ , We use fully-connected ResNet architecture with 5 ResNet blocks with a width of 512.
Figure 12: Samples from GranularPush Dataset. We show sequences of frames over time with an interval of 20 frames. The sequences above the dashed line are for interpolate data, and the bottom images illustrate the extrapolate data.
Dynamics predictor. For the edge and vertex encoders,  $Q_{e}$  and  $Q_{v}$ , we use 3-layer fully-connected networks activated by the ReLU function with 150 hidden units. For the propagators,  $P_{e}$  and  $P_{v}$ , we use a 1-layer fully-connected network followed by ReLU activation. The output dimension of the linear layer is 150.
Sampling 3D points from the trained visual perception module. We sample points on a  $40 \times 40 \times 40$  grid from an area of  $55\mathrm{cm} \times 55\mathrm{cm} \times 55\mathrm{cm}$  and  $63\mathrm{cm} \times 63\mathrm{cm} \times 63\mathrm{cm}$  at the center of the table for FluidPour and FluidCubeShake respectively, and on a  $70 \times 70 \times 70$  grid from an area of  $6\mathrm{cm} \times 6\mathrm{cm} \times 6\mathrm{cm}$  for GranularPush. We evaluate and include points with a density (measured by the occupancy in the predicted neural radiance fields) larger than 0.99. To reduce the total number of points, we subsample the inferred points with FPS with a ratio of  $5\%$  for FluidPour and  $10\%$  for FluidCubeShake and GranularPush.
Graph building. We set the neighbour distance threshold  $\delta$  to be 0.2, 0.15, 0.15 for FluidPour, FluidCubeShake and GranularPush respectively. We select the threshold so that each point will have on average 20 30 neighbors. Since, in FluidPour, we sample the points with lower density 2000points/ $m^2$ , we use a larger threshold for this scenario. For FluidShape and GranularPush, since the density is around 3000 points/ $m^2$ , we cut down the number by  $25\%$ .
We found that if the threshold is too small, the performance will degrade significantly since each particle will only receive messages from a few neighbors (and miss out on the larger context). On the other hand, setting the threshold too large will cause the training time to increase since the graph will have more edges. We found that setting the threshold around the right scale generally leads to more effective training of a reasonable dynamics network.
B.3 Training Details
The models are implemented in PyTorch. We train the perception module using Adam optimizer with a learning rate of  $1e - 4$ , and we reduce the learning rate by  $80\%$  when the performance on the validation set has stopped improving for 3 epochs. To compute the rendering loss when training the perception module, we sample 64 points through each ray in the scene and set the ray-batch size of the NeRF query function  $f$  to be  $1024\times 32$ . Training the perception module on a single scenario takes around 5 hours on one RTX-3090.
We train the dynamics simulator using Adam optimizer with a learning rate of  $1e - 4$ , and we reduce the learning rate by  $80\%$  when the performance on the validation set has stopped improving for 3
epochs. The batch size is set to 4. We train the model for 20, 30, and 40 epochs for FluidPour, FluidCubeShake, and GranularPush, respectively. It takes around  $10 \sim 15$  hours to train the dynamics model in one environment on one single RTX-3090.
B.4 Graph-Based Dynamics Model without Particle-level Correspondence
The velocity of an object provides critical information on how the object will move in the future, yet, we do not have access to such information when tracking the object is impossible. As described in Section 3.2, the attributes  $a_i^v$  of a vertex  $v_i$  in the built graph consists of (1) velocity of this point in the past frames and (2) attributes of the point (rigid, fluid, granular). To get the velocity of a vertex  $v$ , we should have the history position of this vertex. However, since the point clouds are inferred from each frame independently, we do not know how each point moves over time since we do not have point correspondence between frames.
To address the problem, we leverage the fact that some objects in the scene are easier to track, and we try to use the motion of these trackable objects to infer motion for the untrackable units. We assume that we know the dense-labeled states of some known fully-actuated shapes like desks and cups connected to the robot arms. Here we will list one specific scenario where a cup of water is poured into another cup. In this case, we have two different types of points: points for fluid and points for cups, we name the states of them in time step  $t$  as  $V_{P}^{t} = \{v_{P,i}^{t}\}$  and  $V_{S}^{t} = \{v_{S,i}^{t}\}$  respectively. For the particle encoder  $Q_{v}$ , if the particle belongs to the cups, then the input of particle encoder contains  $n_{s}$  history states before  $t_0: \{V_S^{(t_0 - n_s):t_0}\}$ . If the particle belongs to the water, then we have no history states, so the input of  $Q_{v}$  is all-zero.
By adding the relative position between receiver and sender points, we can pass the momentum of  $V_{P}$  to  $V_{S}$ . Compared with human intuition, we can get an intuitive prediction of the movement of water by simply knowing the past movement of the cup without knowing the past movement of water.
Following [47], we use the velocity of points and their relative position as inputs to the dynamics module instead of using the absolute positions of the points. This ensures the model is translation-invariant so the learned dynamics model can be shared across different spatial locations.
B.5 Inference Speed of Our Model
The prediction speed of the dynamics module depends on the number of input particles, and it takes around 0.1s for graphs with around 300 nodes in FluidShake and FluidPour, and around 0.2s for scenes with  $700+$  nodes in GranularPush.
For our visual module, the main time consumption comes from NeRF sampling, it takes 0.2s to sample from a grid space introduced in the experiment section of our paper, this was run in blocks, with block-size=1000, made up 4G of a V100 GPU. And it can be even faster with larger blocks. The sub-sampling process (FPS, segmentation) is fast since they are all written in parallel versions, which takes less than 5ms.
C Potential Society Impact
Our work shows the possibility of learning dynamics models from raw sensory inputs, opening up opportunities to automate the design of differentiable physics engines through data-driven learning algorithms. The resulting system can potentially benefit many downstream tasks, including general scene understanding, robotics manipulation, the construction of 3D generative models, and inverse tasks like planning/control and inverse design. Furthermore, predictions from our model are highly interpretable, which makes it straightforward to explain model behaviors and re-purpose the outputs for other downstream applications.
Though data-driven approaches are potentially more scalable with enough data, concerns still exist that it might be hard to ensure the robustness of the model under sensor noise and adversarial attacks. It also becomes less clear how to fully mitigate data biases. Therefore, bringing in advanced techniques from ML robustness will be one critical future avenue to pursue.
Figure 13: SAM Working on FluidCube Shake: Recent large segmentation models can well generate masks for different objects in the scene.
D Important Clarifications and Discussions
Q: What is the input of 3D-IntPhys?
Video inputs and object instance masks based on color assumption.
Q: What is the difference of our work compared with [34, 35, 16]?