SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haotian Xue
Antonio Torralba
Joshua Tenenbaum
Daniel Yamins<sup>3</sup>
Yunzhu Li 3,4*
Hsiao-Yu Tung*
<sup>1</sup> Georgia Tech
$^{2}$  MIT
3 Stanford Univeristy
4 UIUC
Given a visual scene, humans have strong intuitions about how a scene can evolve over time under given actions. The intuition, often termed visual intuitive physics, is a critical ability that allows us to make effective plans to manipulate the scene to achieve desired outcomes without relying on extensive trial and error. In this paper, we present a framework capable of learning 3D-grounded visual intuitive physics models from videos of complex scenes. Our method is composed of a conditional Neural Radiance Field (NeRF)-style visual frontend and a 3D point-based dynamics prediction backend, using which we can impose strong relational and structural inductive bias to capture the structure of the underlying environment. Unlike existing intuitive point-based dynamics works that rely on the supervision of dense point trajectory from simulators, we relax the requirements and only assume access to multi-view RGB images and (imperfect) instance masks acquired using color prior. This enables the proposed model to handle scenarios where accurate point estimation and tracking are hard or impossible. We generate datasets including three challenging scenarios involving fluid, granular materials, and rigid objects in the simulation. The datasets do not include any dense particle information so most previous 3D-based intuitive physics pipelines can barely deal with that. We show our model can make long-horizon future predictions by learning from raw images and significantly outperforms models that do not employ an explicit 3D representation space. We also show that once trained, our model can achieve strong generalization in complex scenarios under extrapolate settings. The code is released in https://github.com/xavihart/3D-IntPhys.
Humans can achieve a strong intuitive understanding of the 3D physical world around us simply from visual perception [5, 8, 52, 51, 46, 11]. As we constantly make physical interactions with the environment, the intuitive physical understanding applies to objects of a wide variety of materials [6, 56]. For example, after watching videos of water pouring and doing the task ourselves, we can develop a mental model of the interaction process and predict how the water will move when we apply actions like tilting or shaking the cup (Figure 1). The ability to predict the future evolution of the physical environment is extremely useful for humans to plan our behavior and perform everyday manipulation tasks. It is thus desirable to develop computational tools that learn 3D-grounded models of the world purely from visual observations that can generalize to objects with complicated physical properties like fluid and granular materials.
*This work was done when Haotian Xue was a visiting student advised by Joshua Tenenbaum in MIT CSAIL and he was directly mentored by Yunzhu Li and Hsiao-Yu Tung.
Figure 1: Visual Intuitive Physics Grounded in 3D Space. Humans have a strong intuitive understanding of the physical environment. We can predict how the environment would evolve when applying specific actions. This ability roots in our understanding of 3D and applies to objects of diverse materials, which is essential when planning our behavior to achieve specific goals. In this work, we are the first to leverage a combination of implicit neural representation and explicit 3D particle representation to build 3D-grounded visual intuitive physics models of the challenging scenes that applies to objects with complicated physical properties, such as fluids, rigid objects, and granular materials.
There has been a series of works on learning intuitive physics models of the environment from data. However, most existing work either focuses on 2D environments [60, 1, 21, 64, 20, 4, 44, 28, 67, 24, 23, 49, 33, 19, 31, 58, 22, 12, 65] or has to make strong assumptions about the accessible information of the underlying environment [36, 35, 47, 43, 70, 54, 48, 7, 2, 26] (e.g., full-state information of the fluids represented as points). The limitations prevent their use in tasks requiring an explicit 3D understanding of the environments and make it hard to extend to more complicated real-world environments where only visual observations are available. There are works aiming to address this issue by learning 3D-grounded representation of the environment and modeling the dynamics in a latent vector space [34, 32]. However, these models typically encode the entire scene into one single vector. Such design does not capture the structure of the underlying systems, limiting its generalization to compositional systems or systems of different sizes (e.g., unseen container shapes or different numbers of floating ice cubes).
In this work, we propose 3D Visual Intuitive Physics (3D-IntPhys), a framework that learns intuitive physics models of the environment with explicit 3D and compositional structures with visual inputs.
Specifically, the model consists of (1) a perception module based on conditional Neural Radiance Fields (NeRF) [41, 68] that transforms the input images and instance masks into 3D point representations and (2) a dynamics module instantiated as graph neural networks to model the interactions between the points and predict their evolutions over time. Despite advances in graph-based dynamics networks [47, 36], existing methods require strong supervision provided by 3D GT point trajectories, which are hard to obtain in most real setups. To tackle the problem, we train the dynamics model using (1) a distribution-based loss function measuring the difference between the predicted point sets and the actual point distributions at the future timesteps and (2) a spacing loss to avoid degenerated point set predictions. Our perception module learns spatial-equivariant representations of the environment grounded in the 3D space, which then transforms into points as a flexible representation to describe the system's state. Our dynamics module regards the point set as a graph and exploits the compositional structure of the point systems.
The structures allow the model to capture the compositionality of the underlying environment, handle systems involving objects with complicated physical properties, and perform extrapolated generalization, which we show via experiments greatly outperform various baselines without a structured 3D representation space.
Visual dynamics learning. Existing works learn to predict object motions from pixels using frame-centric features [1, 20, 4, 24, 23, 53, 30, 59, 69, 10, 25, 62] or object-centric features [21, 61, 28, 44, 27, 58, 14, 22, 45, 66], yet, most works only demonstrate the learning in 2D scenes with objects
Figure 2: Overview of 3D Visual Intuitive Physics (3D-IntPhys). Our model consists of two major components: Left: The perception module maps the visual observations into implicit neural representations of the environment. We then subsample from the reconstructed implicit volume to obtain a particle representation of the environment. Right: The dynamics module, instantiated as graph neural networks, models the interaction within and between the objects and predicts the evolution of the particle set.
moving only on a 2D plane. We argue that one reason that makes it hard for these existing methods to be applied to general 3D visual scenes is because they often operate on view-dependent features that can change dramatically due to changes in the camera viewpoint, which shouldn't have any effect on the actual motion of the objects. Recent works by [9] have shown that only methods that use 3D view-invariant representations can pave the way toward human-level physics dynamics prediction in diverse scenarios.
Researchers have attempted to learn object motion in 3D [55, 63, 40, 34], [55] and [63] use object-centric volumetric representations inferred from RGB-D to predict object motion, yet, these volumetric approaches have much higher computation costs than 2D methods due to the 4D representation bottleneck, which hinders them from scaling up to more complex scenes. [40] use self-supervised 3D keypoints and [15, 16] use implicit representations to model multi-object dynamics but cannot handle objects with high degrees of freedom like fluid and granular materials. [34] use neural implicit representation to reduce the potential computational cost, yet the works have not shown how the approach can generalize to unseen scenarios. Our works aim to solve the tasks of learning generalizable object dynamics in 3D by combining the generalization strength of input-feature-conditioned implicit representation and point-based dynamics models.
Point-based dynamics models. Existing works in point- and mesh-based dynamics models [36, 42, 57, 47, 43] have shown impressive results in predicting the dynamics of rigid objects, fluid [36, 42, 57, 47, 3, 13], deformable objects [36, 42, 47], and clothes [43, 38]. Most works require access to full 3D states of the points during training and testing, yet, such information is usually not accessible in a real-world setup. [35] learn a visual frontend to infer 3D point states from images, but still require 3D point states and trajectories during training time. [50] propose to learn point dynamics directly from vision, but they only consider elasto-plastic objects consisting of homogeneous materials. How to learn about 3D point states and their motion from raw pixels remain a question. Our paper tries to build the link from pixels to points using recent advances in unsupervised 3D inference from images using NeRF [41, 68].
We present 3D Visual Intuitive Physics (3D-IntPhys), a model that learns to simulate physical events from unlabeled images (Figure 2). 3D-IntPhys contains a perception module that transforms visual observations into a 3D point cloud that captures the object geometries (Section 3.1) and a point-based simulator that learns to simulate the rollout trajectories of the points (Section 3.2). The design choice of learning physics simulation in a 3D-point representation space enables stronger simulation performance and generalization ability. The performance gain mainly comes from the fact that describing/learning objects' motion and interactions in 3D are easier compared to doing so in 2D since objects live and move persistently in the 3D space. 3D-IntPhys also supports better
generalization ability since its neural architecture explicitly models how local geometries of two objects/parts interact, and these local geometries and interactions can be shared across different and novel object combinations.
Although 3D-IntPhys learns to simulate in a 3D representation space, we show it can learn without any 3D supervision such as dense point trajectories as in previous work [47, 36]. Dense point trajectories are hard and sometimes impossible to obtain in the real world, e.g., capturing the trajectories of each water point. 3D-IntPhys does not require such 3D supervision and can simply learn by observing videos of the scene evolution.
3.1 2D-to-3D Perception Module
Given a static scene, the perception module learns to transform one or a few posed RGB images,  $\mathbf{I} = \{(I_i,\pi_i)|i\in \{1,2,\dots ,N_v\} \}$ , taken from  $N_{v}$  different views, into a 3D point cloud representation of the scene,  $\mathbf{X}$ . We train the model in an unsupervised manner through view reconstruction, using a dataset consisting of  $N_{t}$  videos, where each video has  $N_{f}$  frames, and each frame contains images taken from  $N_{v}$  viewpoints.
Neural Radiance Field (NeRF). NeRF [41] learns to reconstruct a volumetric radiance field of a scene from unlabeled multi-view images. After training, the model learns to predict the RGB color  $\mathbf{c}$  and the corresponding density  $\sigma$  of a query 3D point  $\mathbf{x} \in \mathbb{R}^3$  from the viewing direction  $\mathbf{d} \in \mathbb{R}^3$  with a function  $(\mathbf{c}, \sigma) = f(\mathbf{x}, \mathbf{d})$ . We can formulate a camera ray as  $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$ , where  $\mathbf{o} \in \mathbb{R}^3$  is the origin of the ray. The volumetric radiance field can then be rendered into a 2D image via  $\hat{\mathbf{C}}(\mathbf{r}) = \int_{t_n}^{t_f} T(t)\sigma(t)\mathbf{c}(t)dt$ , where  $T(t) = \exp(-\int_{t_n}^{t}\sigma(s)ds)$  handles occlusion. The rendering range is controlled by the depths of the near and far plane (i.e.,  $t_n$  and  $t_f$ ). We can train NeRF through view prediction by:
$$
\mathcal {L} = \sum_ {\mathbf {r} \in \mathcal {R} (\mathbf {P})} \| \hat {\mathbf {C}} (\mathbf {r}) - \mathbf {C} (\mathbf {r}) \|, \tag {1}
$$
where  $\mathcal{R}(\mathbf{p})$  is the set of camera rays sampled from target camera pose  $\mathbf{p}$ .
Image-conditioned NeRF. To infer the NeRF function from an image, previous work proposed to encode the input image into a vector, with a CNN encoder, as a conditioning input to the target NeRF function [34]. We found this type of architecture is in general hard to train and does not generalize well. Instead, we adopt pixelNeRF [68], which conditions NeRF rendering with local features, as opposed to global features. Given an image  $I$  in a scene, pixelNeRF first extracts a feature volume using a CNN encoder  $\mathbf{W} = E(I)$ . For a point  $\mathbf{x}$  in the world coordinate, we retrieve its feature vector by projecting it onto the image plane, so that we can get the feature vector  $\mathbf{W}(\pi (\mathbf{x}))$ . PixelNeRF combines the feature vector together with the 3D position of that point and predict the RGB color and density information:
$$
\mathbf {V} (\mathbf {x}) = (\mathbf {c}, \sigma) = f (\mathbf {x}, \mathbf {d}, \mathbf {W} (\pi (\mathbf {x}))). \tag {2}
$$
In the experiment section, we will show that it is surprisingly effective to train only one general model to learn a conditional Neural Radiance Field that can apply to all videos in one type of scene (e.g. FluidPour) with five different settings (e.g. extrapolate setting), which provides a better 3D representation for the scene and greatly facilitates the learning of 3D Intuitive Physics.
Explicit 3D representation from pixelNeRF. From a few posed RGB images  $\mathbf{I}$ , of a scene  $s$ , we infer a set of points for  $O_{s}$  target object (such as fluid, cube) in the scene. We achieve this by first sampling a set of points according to the predicted occupancy measure, then clustering the points into objects using object segmentations. We found that sampling with low resolution will hurt the quality of the rendered point cloud to generate objects with inaccurate shapes, while sampling with high resolution will increase the computation for training the dynamics model since the input size increases. To speed up training while maintaining the quality of the reconstructed point cloud, we first infer the points with higher resolution and do sparse sampling of each point cloud using FPS (Farthest Point Sampling) [17]. Next, we cluster the inferred points into objects according to object segmentation masks. Since solving object segmentation in general is not the main focus of this paper, we resort to using the color information to obtain the masks.
3.2 Point-Based Dynamics Learner
Given the point representation at the current time step,  $\mathbf{X}_t$ , the dynamics simulator predicts the points' evolution  $T$  steps in the future,  $\{\mathbf{X}_{t + 1},\mathbf{X}_{t + 2},\dots \mathbf{X}_{t + T}\}$ , using graph-based networks [47, 36].
We first form a graph  $(V,E)$  based on the distance between points. If the distance between two points is smaller than a threshold  $\delta$ , we include an edge between these two points. Each vertex  $v_{i} = (\dot{x}_{i},a_{i}^{v})\in V$  contains the velocity of the point,  $\dot{x}_i$ , and point attributes,  $a_i^v$ , to indicate the point's type. For each relation,  $(i,j)\in E$ , we have its associated relation attribute  $a_{ij}^{e}$ , indicating the types of relation and the relative distance between the connected points.
Spatial message passing and propagation. At time step  $t$ , we can do message passing to update the points,  $v_{i} \in V$ , and relation representations,  $(i,j) \in E$ , in the graph:
$$
g _ {i j, t} = Q _ {e} \left(v _ {i, t}, v _ {j, t}, a _ {i j} ^ {e}\right), (i, j) \in E \tag {3}
$$
$$
h _ {i, t} = Q _ {v} \left(v _ {i, t}, \sum_ {k \in \{j | (i, j) \in E \}} g _ {i k, t}\right), v _ {i} \in V \tag {4}
$$
where  $Q_v$  and  $Q_e$  are encoders for vertices and relations respectively. Please refer to [7] for more details. Though this kind of message passing can help with updating representation, it can only share one-hop information in each step, limiting its performance on instantaneous passing of forces. To improve long-range instantaneous effect propagation, we use multi-step message propagation as in [37, 36]. The propagation step is shown in Alg 1:
Algorithm 1: Point-based Dynamics Predictor
Data: Current timestep  $t$ , point cloud  $V_{t}$ , vertex encoder  $Q_{v}$ , edge encoder  $Q_{e}$ , vertex propagator  $P_{v}$ , edge propagator  $P_{e}$ , state predictor  $f_{s}$
Result:  $V_{t + 1}$
Form graph  $G_{t} = (V_{t},E_{t})$
//message passing
$$
g _ {i j, t} = Q _ {e} \left(v _ {i, t}, v _ {j, t}, a _ {i j} ^ {e}\right), (i, j) \in E _ {t}
$$
$$
h _ {i, t} = Q _ {v} \left(v _ {i, t}, \sum_ {k \in \{j | (i, j) \in E \}} g _ {i k, t}\right), v _ {i} \in V _ {t}
$$
//message propagation
$$
h _ {i, t} ^ {0} = h _ {i, t}, g _ {i, t} ^ {0} = g _ {i, t},
$$
for  $l\in \{1,2,3,\dots,L\}$  do
$$
\left| \begin{array}{l l} & g _ {i j, t} ^ {l} = P _ {e} (g _ {i j, t} ^ {l - 1}, h _ {i, t} ^ {l - 1}, h _ {j, t} ^ {l - 1}) \quad , (i, j) \in E _ {t} \\ & h _ {i, t} ^ {l} = P _ {v} (h _ {i, t} ^ {l - 1}, \sum_ {k \in \{j | (i, j) \in E \}} g _ {i k, t} ^ {l}) \quad , v _ {i} \in V _ {t} \end{array} \right.
$$
//state prediction
$$
v _ {i, t + 1} = f _ {s} \left(h _ {i, t} ^ {L}\right)
$$
$$
V _ {t + 1} = \{v _ {i, t + 1} \}
$$
where  $P_{e}, P_{v}$  are propagation functions of nodes and edges, respectively, and  $g_{ij,t}^{l}$  is the effect of relation  $(i,j)$  in propagation step  $l$ .  $h_{i,t}^{l}$  is the hidden states for each point in the propagation process. Finally, we have the predicted states of points at time step  $t + 1$  after  $L$  steps of propagation:
$$
\hat {v} _ {i, t + 1} = f _ {s} \left(h _ {i, t} ^ {L}\right). \tag {5}
$$
**Environments.** We assume that the surrounding environment (e.g., the table) is known and the robot/tool/container are of known shape and fully actuated, where the model has access to their complete 3D state information. We convert the full 3D states into points through sampling on the 3D meshes and include these points in the prediction of the graph-based dynamics.
Fluids, rigid bodies, and granular materials. We distinguish different materials by using different point attributes  $a_{i}^{v}$ . We also set different relation attributes  $a_{ij}^{e}$  in Equation 3 to distinguish different interaction (e.g., Rigid-Fluids, Fluids-Fluids, Granular-Pusher). For rigid objects, to ensure the object shapes remain consistent throughout the rollout predictions, we add a differentiable rigid constraint in the prediction head following [36].
Training dynamics model without point-level correspondence. Since our perception model parses each RGB image into object-centric point clouds independently, there does not exist an explicit one-to-one correspondence for points across frames. To handle this, we measure the Chamfer distance between the prediction  $\hat{\mathbf{X}}_t = (\hat{V}_t,\hat{E}_t)$  from the dynamics network and the inferred point state  $\mathbf{X}_t = (V_t,E_t)$  from the perception module and treat it as the objective function. The Chamfer
Figure 3: Data Collection and Evaluation Setups. Left: We collect multi-view videos of the environment from six cameras. Right: We consider a diverse set of evaluating environments involving fluids, rigid objects, granular materials, and their interactions with the fully-actuated container and the environment. We evaluate the learned visual intuitive physics model on both the interpolated settings (i.e., seen environment but with different action sequences) and extrapolated settings (i.e., unseen environment with different amounts of fluids, cubes, granular pieces, and containers of different sizes).
distance between two point cloud  $\hat{V}$  and  $V$  is defined as:
$$
L _ {c} (\hat {V}, V) = \frac {1}{\| \hat {V} \|} \sum_ {x \in \hat {V}} \min  _ {y \in V} \| x - y \| _ {2} ^ {2} + \frac {1}{\| V \|} \sum_ {x \in V} \min  _ {y \in \hat {V}} \| x - y \| _ {2} ^ {2}. \tag {6}
$$
We found that training the model with Chamfer distance in dense scenes with granular materials will often lead to predictions with unevenly distributed points where some points stick too close to each other. To alleviate this issue, we further introduce a spacing loss  $L_{s}$ , which penalizes the gated distance (gated by  $d_{\mathrm{min}}$ ) of nearest neighbor of each point to ensure enough space between points:
$$
L _ {s} (\hat {V}) = \sum_ {v \in \hat {V}} \left(\operatorname {R e L U} \left(d _ {\min } - \min  _ {v ^ {\prime} \in \{\hat {V} \backslash v \}} \| v ^ {\prime} - v \| _ {2} ^ {2}\right)\right) ^ {2}. \tag {7}
$$
The one-step prediction loss  $L_{dy}$  for training the dynamics model is  $L_{c}(\hat{V}, V) + \sigma L_{s}(\hat{V})$  where  $\sigma$  reweights the second loss. To improve long-term rollout accuracy, we train the model with two-step predictions using the first predicted state as input and feed it back into the model to generate the second predicted state. With the two-step loss, the model becomes more robust to errors generated from its own prediction. Finally, the  $L_{dy}$  losses for all rolling steps are summed up to get the final loss for this trajectory. More implementation details are included in the supplementary material.
The experiment section aims to answer the following three questions. (1) How well can the visual inference module capture the content of the environment (i.e., can we use the learned representations to reconstruct the scene)? (2) How well does the proposed framework perform in scenes with objects of complicated physical properties (e.g., fluids, rigid and granular objects) compared to baselines without explicit 3D representations? (3) How well do the models generalize in extrapolate scenarios?
Datasets. We generated three simulated datasets using the physics simulator Nvidia FleX [39]. Each of the datasets represents one specific kind of manipulation scenario, where a robot arm interacts with rigid, fluid, and granular objects (Figure 3). For each of the three scenarios, we apply randomized input actions and change some properties of objects in the scene, e.g., the shape of the container, the amount of water, and the color/number of cubes, to make it diverse. To test the generalization capability of the trained model, we design extrapolated datasets where the data is generated from an extrapolated set of parameters outside the training distribution.
a) FluidPour. This scenario contains a fully-actuated cup pouring fluid into a container. We design the extrapolate dataset to have a larger container, more quantity of fluid, and different pouring actions.
b) FluidCubeShake. This scenario contains a fully-actuated container that moves on top of a table. Inside the container are fluids and cubes with diverse colors. We design the extrapolate dataset to have different container shapes, number of cubes, cube colors, and different shaking actions.
Figure 4: Qualitative Results of the Dynamics Module on Future Prediction. Here we visualize our model's predicted future evolution of the particle set as compared with the NeRF-dy [34] baseline in both interpolate and extrapolate settings. Our method correctly identifies the shape/distribution of the fluids, rigid objects, and granular pieces with much better accuracy than NeRF-dy. The future evolution predicted by our method also matches the ground truth much better and produces reasonable results even in extrapolate settings.