Learning Multi-Object Dynamics with Compositional Neural Radiance Fields
Authors: Danny Driess
Zhiao Huang
Yunzhu Li
Russ Tedrake
Marc Toussaint
Venue: CoRL 2022 Poster (2022)
Keywords: Neural Radiance Fields
Dynamics Model Learning
Neural Implicit Representations
Visual Prediction
Graph Neural Networks
Representation Learning
Robotic Manipulation

Abstract: We present a method to learn compositional multi-object dynamics models from image observations based on implicit object encoders, Neural Radiance Fields (NeRFs), and graph neural networks. NeRFs have become a popular choice for representing scenes due to their strong 3D prior. However, most NeRF approaches are trained on a single scene, representing the whole scene with a global model, making generalization to novel scenes, containing different numbers of objects, challenging. Instead, we present a compositional, object-centric auto-encoder framework that maps multiple views of the scene to a set of latent vectors representing each object separately. The latent vectors parameterize individual NeRFs from which the scene can be reconstructed. Based on those latent vectors, we train a graph neural network dynamics model in the latent space to achieve compositionality for dynamics prediction. A key feature of our approach is that the latent vectors are forced to encode 3D information through the NeRF decoder, which enables us to incorporate structural priors in learning the dynamics models, making long-term predictions more stable compared to several baselines. Simulated and real world experiments show that our method can model and learn the dynamics of compositional scenes including rigid and deformable objects.
Video: https://dannydriess.github.io/compnerfdyn/

Learning Multi-Object Dynamics with Compositional Neural Radiance Fields
Danny Driess TU Berlin
Zhiao Huang UC San Diego
Yunzhu Li MIT
Russ Tedrake MIT
Marc Toussaint TU Berlin
Abstract: We present a method to learn compositional multi-object dynamics models from image observations based on implicit object encoders, Neural Radi- ance Fields (NeRFs), and graph neural networks. NeRFs have become popular choice for representing scenes due to their strong 3D prior. However, most NeRF approaches are trained on a single scene, representing the whole scene with a global model, making generalization to novel scenes, containing different num- bers of objects, challenging. Instead, we present compositional, object-centric auto-encoder framework that maps multiple views of the scene to set of latent vectors representing each object separately. The latent vectors parameterize indi- vidual NeRFs from which the scene can be reconstructed. Based on those latent vectors, we train a graph neural network dynamics model the latent space achieve compositionality for dynamics prediction. A key feature of our approach is that the latent vectors are forced encode 3D information through the NeRF decoder, which enables us to incorporate structural priors in learning the dynamics models, making long-term predictions more stable compared several baselines. Simulated and real world experiments show that our method can model and learn the dynamics of compositional scenes including rigid and deformable objects. Video: https : //tp:///andnneeeeeee iess. wewprriiiiss..u.b....e.... io/ compnerfdyn/
Keywords: Neural Radiance Fields, Dynamics Models, Graph Neural Networks
1 Introduction
Learning models from observations that predict the future state of scene is fundamental con- cept for enabling an agent to reason about actions to achieve desired goal. A major challenge in learning predictive models is that raw observations such as images are usually high- nigh-dimensional. Therefore, common approach to map the observation space into a lower-dimensional latent rep- resentation of the scene via an auto-encoder structure. Based on those latent vectors, dynamics model can be learned that predicts the next latent state, conditioned on actions an agent takes. An intuition for this is that if a latent vector is sufficient to reconstruct the observations, then it contains enough information about the scene to learn a dynamics model on top of it. While an auto-encoder structure combined with a latent dynamics model is general approach that is applicable for large variety of tasks, it raises multiple challenges. First, scenes in our world are composed of multiple ob- jects. Therefore, a fixed-size latent vector has difficulties in generalizing over different and changing numbers of objects in the scene than during training. both due the limited capacity of fixed-size vectors and lack of diversity in the training distribution. Second, image observations are 2D, but the 3D structure of our world is essential for many tasks reason about the underlying physical processes governing the dynamics the model should predict. Dealing with occlusions, object perma- nence, and ambiguities in 2D views is challenging for 2D image representations. Importantly, many forward predictive models in visual observation spaces suffer from instabilities in making long-term predictions, often manifested in blurry image predictions [1].
One way to address these issues is to incorporate inductive biases and structural priors in the model architectures. Li et al. [2] proposed to use Neural Radiance Fields (NeRFs) [3] as decoder within an auto-encoder to learn dynamics models in latent spaces. NeRFs exhibit strong structural priors about the 3D world, leading to increased performance over 2D baselines. However, the approach of [2] represents the whole scene as a single latent vector, which we found insufficient for scenes composed of multiple, different numbers of objects, in terms of representation and dynamics prediction.
In the present work, we aim to overcome these challenges by incorporating inductive biases on the compositional nature and underlying 3D structure of our world both in learning the latent representa-
6th Conference on Robot Learning (CoRL 2022), Auckland, New Zealand.

---

(a) Bottom row renderings of forward predictions with dynamic model, top row ground truth (b) novel view Figure I: Visual forward predictions with our model. (a) left: initial scene, (a) right: after 15 prediction steps into the future. (b) renderings of the model after 15 prediction steps from novel views. Future predictions are based on the initial observation of the scene at = 0 and then rendered from the prediction of the latent vectors. Bluc pusher is articulated by the robot. Despite multiple objects interacting, predictions are sharp and accurate.
tions themselves and the dynamics model. We propose compositional, object-centric auto-encoder framework whose latent vectors are used to learn a compositional forward dynamics model in that learned latent space based graph neural networks (GNN). More specifically, we learn an implicit object encoder that maps image observations vectors that each represent object in the scene separately. These latent object encodings then parameterize individual NeRFs for each object. We apply compositional rendering techniques to synthesize images from multiple viewpoints, which forces the object-centric NeRF functions and the corresponding latent vectors to learn precise 3D configurations of the constituting objects. This 3D inductive bias both in the encoder and the compositional NeRF decoder enables to incorporate priors from the models' own predictions about objects interactions via an estimated adjacency ma- trix into learning the GNN dynamics model, making long-term dynamics predictions more stable. This long term-stability allows us utilize a planning method based on RRT' the latent space.
the scene from multiple views to a set of latent
In our evaluations, we show through comparisons that non-compositional auto-encoder frameworks and non-compositional dynamics models struggle with tasks containing multiple objects, while our framework generalizes well over different numbers of objects than during training and is capable of generating sharp and stable long-term predictions. Relative to more traditional multibody system identification [4], these models learn the geometry of unknown objects in addition to (implicitly) learning the inertial and contact parameters. We demonstrate the performance of the approach in terms of image reconstruction error, dynamics prediction error, and planning, generalizing over different numbers of objects than during training. Our experiments include rigid and deformable objects in simulation and with a real robot. To summarize, our main contributions are
A compositional scene encoding framework that uses implicit object encoders and NeRF decoders for each object, forcing the iiea-invariant latent representation to learn about the 3D structure of the problem in composable way.
A factored dynamics model in the latent space graph neural network (GNN), exploit- ing the compositional nature of the scene representation and an adaptive adjacency matrix estimated from the model itself to yield stable long-term predictions.
Related Work
Learning Dynamics Models for Compositional Systems. Graph neural networks (GNNs) have shown success in introducing relational inductive biases [5]. enabling them to model the dynamics
of compositional systems consisting of interactions between multiple objects [6, 8, 9, 10, 11], large-scale dynamical systems represented with particles and meshes [12, 13. 14, 15, 16. 17]. or from visual observations [18. 19, 20, 21, 22, 23, 24]. Our method differs from prior work by learn- ing compositional scene representations grounded in 3D space from visual observations. Our novel combination of implicit object encoders and graph-based based neural dynamics models reflects the struc- ture of the underlying scene, which endows our agent with better generalization ability in handling complicated compositional dynamic environments.
NeRF for Compositional and Dynamic Scenes. Recent advances on neural implicit repre- sentations [25] have demonstrated widespread success in image synthesis 3D reconstruc- tion [26. 27, 28, 29, 30J. Notably, Neural Radiance Fields (NeRF) show impressive results on novel- view synthesis [3]. Initial NeRF approaches were trained on single scene without generalization. Prior work [31, 32, 33, 34, 35, 36] have since proposed to modify neural scene representations to make them compositional for static scenes without considering dynamics of object interactions. Re-

---

DASRT
DASEr
DNERF
encoder
reconstruction for arbitrary views ar any predicted time-step with compositional NeRF decoder
ztim
latent vectors
GNN
GNN
observations the initial scene from multiple views
estimate adjacency matrix A from NeRF
estimate adjacency matrix from NeRF
Figure 2:
Overview of the dynamics prediction framework. The initial scene observations are encoded with f into set of latent vectors 21:m. each representing the objects individually. The GNN dynamics model predicts the evolution of the latent vectors. At each step, the predicted latent vectors can he rendered into an arbitrary view with the compositional NeRF decoder. Refer the appendix for visualizations of Ω and GNN.
cent research has also extended NeRF to enable view synthesis from sparse set of views [37], as well as modeling dynamic scenes by learning implicitly represented flow fields or time-variant latent codes [38, 39, 40, 41, 42, 43, 44, 45, 46, 33, 47]. However, these approaches for dynamic environ- ments typically interpolate over single time sequence and are not able handle scenes of different initial configurations or different action sequences, limiting their use in downstream planning and control tasks. Li et [2] addressed this issue by combining with modeling the dynamics in a latent space. Yet, they employed a single latent vector as the whole scene representation, which we will show is insufficient modeling compositional systems. In contrast, our method considers a graph-based scene representation to capture the structure of the underlying scene and achieves significantly better generalization performance than [2].
NeRF auto-encoding framework
Implicit Models in Robotics. Implicit models in roboties have been explored, e.g., for grasping [48, 49, 50, 51], deformable objects [52], object descriptors [53], or general manipulation constraints [54, 55]. Analytic signed distance functions (SDFs) [56, 57, 54] or learned NeRFs [58] are used for trajectory planning. One assumption in [54, 55] is that SDF values are available during training. Our work, in contrast, directly operates on RGB images without requiring explicit 3D shape supervision.
3 Overview- - Compositional Visual Dynamics Learning
Our dynamics learning framework (Fig. 1) consists of three parts, an object encoder 12 turning ob- servations into a set of latent vectors 21:m. a compositional NeRF-basee decoder DNERF that renders the latent vectors back into images of the scene train the encoder, and graph neural network dynamics model FONN predicting the evolution of the scene in the latent space. This section gives high-level overview, while Sec. 4, Sec. 5 as well the appendix Sec. C, Sec. provide details.
Assume that a scene is observed by RGB images E 233xyXXF, = 1,..., V from V many camera views and that the scene contains objects 1,..., m. We further assume have access the camera projection matrices K E R3x4 for each view and binary masks Mj E {0, ( X of each object j in view i. Given those posed images and masks, the goal is to learn an encoder Ω that fuses the information of the objects observed from the multiple views into a of latent vectors Z1am by querying Ω on the individual masks M1.V such that
2j = S2 (I KKVV MM.V) R
(I)
represents the object j separately. Q is trained end-to-end with a NeRF decoder DNERF reconstructing
I -= DNeeF K)
for arbitrary views specified by the camera matrix K from the set of latent object representations 21am- The initial observation of the scene encoded with into the initial latent vectors 2i.m- The GNN dynamics model 2 = FONN (zf.m) then generates long-term predictions of future latent states of that can also be decoded with DNORF to yield visual predictions from arbitrary views.
3

---

4 Encoding Scenes with Compositional Image-Conditioned NeRFs
4.1 Implicit Object Encoder
Instead of learning Q defined in (1) as direct mapping from images, camera matrices and masks to the latent vectors, we first encode each object in the scene as a feature-valued function functio over 3D space, conditioned on the image observations. This allows us to incorporate multiple views of the objects in geometrically consistent way, as well as to apply 3D affine transformations to the objects, which will be important for the dynamies model (Sec. 5). This function is then turned into latent vector by evaluating it on a workspace set followed by a 3D convolutional network.
All object feature functions are based on the same feature encoder E(T', Kx())) E <R that outputs an Mo-dimensional feature vector from the image of view any 3D world coordinate x E TR3. This is realized by first projecting into camera coordinates K" (z) K²(x)
(x)) (' d' (z)) T E R3
where u' (ar), o² (a) are pixel coordinates in the image plane and d' d((x) E IR is the depth of x from the camera origin. Hence, E is a function of the camera coordinates only and not of absolute world coordinates. Using bilinear interpolation, the encoder E(I', K' determines the RGB values of I' at (w' v (x)) which are passed through a dense neural network (MLP). Parallel to this, a dense MLP encoding of K K((x)) is computed. The concatenated outputs of both MLPs define the encoding feature vector E(I', K* (ax))). Intuitively, ((I',K((') (())) is a feature vector computed from what can be seen of the world at x: in the image I from viewpoint i, taking into account its location relative to the camera origin of the view i, which important not only to enable the model to reason about the 3D geometry, but also to enable us to obtain functional representation of specific object j. Namely, we define the feature function for object j by summing over the individual views
E (M))EM)
B(T', (x))) (R()) ERD with p(x)=
E E: (.()EEM)
1.
(3)
p(x)
Importantly, for a specific x, this sum only takes those views into account where the object j can be seen, i.e., where the camera coordinates (x)x)) of x are within the object's mask Mj. We
define yy (z) 0 E RT if p(x) O, meaning if an object not observed from any view at æ, the corresponding feature vector is zero. An advantage of this formulation is that it naturally handles occlusions in different views and fuses the observations from different views consistently.
Given the implicit object descriptor function y/ (-) of object j, we turn into latent vector 2j E RR representing object with a 3D convolutional network t as follows. Formally, 2j { is a function of the object function. As discussed in [54], learning a function of a function can be realized with neural networks by evaluating yj on a workspace set. We assume that the interactions in the scene happen within a workspace set X C R that is large enough to contain all objects. This workspace set is discretized as the voxel grid Xr, Rdxhxw, The object descriptor functions are then evaluated on Xr, which produces an object feature voxel grid that is processed with a 3D convolutional neural network leading the latent vector zj E R i.e.
2j (1,) = CN,)),
Note that the same workspace set Xr is used for all objects. The appendix (Sec. C) contains visual- izations of the architectures of F, y and & (Fig. 8, 9, 10).
In summary, the object encoder 21mm Ω (TL:V KI.V MI.. Xx) maps images from multiple views, object masks and the set Xr to latent vectors. The resulting 2j 's contain not only the appear- ance of the objects, but also their spatial configurations in the scene relative to other objects.
4.2 Decoder Compositional, Conditional NeRF Model
The general idea of NeRF [3] is to learn function ff that predicts at a 3D world coordinate x E g the RGB color value c(x) E 3 and volume density (x)) E Rzo. Based on (c()), = f()) images from arbitrary views and camera configurations can be rendered by determining the color of the pixels along corresponding camera rays through volumetric rendering. For details, see Sec. C.
Compared to this standard NeRF formulation where one single model used to represent the whole scene, we associate separate NeRFs with each object, meaning that the NeRF for object j
(a,(x), fz(x) f(x, 2z)
(5)
4

---

is conditioned on 2j for j = l,..., m. oj is the density and oj the color prediction for object j, respectively. To turn those fi.m back into a global NeRF model that can be rendered an image, we sum the individual predicted object densities (z²) = -i= m Ty (z) and obtain the colors as their
density weighted combination c(x) नं. -=1 (z)))
These composition formulas have been proposed multiple times in the literature, e.g. [59, 32]. This composition forces the individual NeRFs to learn the 3D configuration of each object individually and therefore ensures that each fz only predicts the object where it is located in the 3D space.
To summarize, the compositional NeRF-decoder DNERF takes the set of latent vectors Z1:m for objects j 1,..., m and the camera matrix K for desired view as input render / DNcRF(Zl.m: K). Since we only represent the objects and not the background as NeRFs, render- ing the composed NeRF will yield an image with the background subtracted. In the experiments,
we investigate the importance of the decoder being both compositional and a NeRF
4.3 Training
The auto-encoder framework is trained end-to-end on an L2 image reconstruction loss for view i
L
Σ
(I Moa DNERF (12 (T&., KL.V MLV Xn). K*) ue 1
(6)
Since solely the objects are represented as NeRFs and not the background, we compute the union of the masks of the individual objects Mir = Vi= Mî and define the target image as I M with Min being a slightly enlarged union mask. Please refer to the appendix Sec. C for more details.
5 Latent Dynamics Model with Graph Neural Networks Having trained the auto-encoder framework, we learn a graph neural network dynamics model
++1 FONN (zim::
2 1am
in the latent space, where A' E {0. mm is the adjacency matrix at time f. Following [8], we use multi-step message passing to deal with cases where multiple objects interact within one prediction step. Refer to the appendix Sec. D and Algo. 1 for more details about our GNN dynamics model.
Adjacency Matrix from Learned Model. The adjacency matrix A in the GNN dynamics model (7) plays an important role in indicating which objects interact. While a dense adjacency matrix, i.e. graph where each object interacts with all other objects, would in principle work as the GNN could figure out from the latent vectors which objects interact, we found that the long-horizon prediction performance is greatly increased if A is more selective in reflecting which objects actually interact.
We propose utilize the NeRF decoder density prediction oy for each object to determine the adjacency matrix from the models' own predictions during training and planning. In order to do so, we define the entries of the adjacency matrix between objects and j based on the collision integral
Aoj = {o
fxlo(s, x)) > alfa(a,, 25) > x] dx > 0
(8)
else
over the density predictions of the learned NeRF model for threshold 2 0. Estimating A this way takes the actual 3D geometry of the objects in the scene into account and thereby informs the GNN dynamics model, leading to more stable predictions. Please refer to the appendix Sec. D for more details about A and how it is used the forward prediction Algo.
Actions. So far, we have formulated the GNN dynamics model without a notion of actions. We interpret action as an intervention to latent vector and train the GNN to predict the latent vectors at the next time step as a result to this modification. This allows us to not explicitly distinguish between controlled and uncontrolled/passiee objects. In order realize these interventions and hence to incorporate actions in the first place, we utilize the fact that our object encoder is built from an implicit representation. Assume that an action is a rigid transformation g E R applied on object j. As described in Sec. C we can modify the object's 's latent vector 2/ into the transformed E/ 2 1 [g)][2y] representing the rigidly transformed object j. The model FANN then predicts how the other objects in the scene react to this rigid transformation of the articulated object.
5

---

dense
2D) auto encader
CNN ancoder
dynamics
0.04
Image Uirer
0.3
Relative Image
0.2
0.02
0.1
20 40
20
40
0
20
Prediction Steps
Prediction Steps
Prediction Steps
Prediction Steps
(a) 2 objects
(b) objects
(c) 8 objects
(d) Relative error (4 objects)
Figure 3: Image prediction error comparison between the reconstructed image from the predicted latent vectors over the number of time steps into the future and ground truth image observations, for test datasct of scenarios containing 4, and 8 objects (plus the pusher). 2 and 8 objects is generalization over number of objects, 4 is during training. One step corresponds to 2 em movement, i.e. for 50 steps the pusher has moved m.
6 Experiments
We demonstrate our framework on pushing tasks both in simulation and in the real world. The scenarios are challenging as they are composed of multiple, interacting objects, sparse rewards, and complex dynamics [60, 61, 62, 63]. Please refer to the video https: htt////dannnndsrseeeeeee .gtthu.. io/ co/compnerdyy/ as well as the appendix for more details and further experiments.
6.1 Visual Reconstruction and Prediction Performance - Comparison to Baselines
We compare framework to con-coopoiiiinnal scene representations, non-compositional dynam- ics models, 2D CNN baselines (visual foresight) without NeRF as decoder, and the importance of estimating the adjacency matrix from the model itself.
Reconstruction and Prediction Performance for Generalization over Numbers of Objects. Fig. 4a shows predictions of the model forward unrolled in time for an action sequence of the red pusher, i.e. applying Algo. (appendix) an initial scene observation and rendering the predicted latent vectors with the NeRF decoder. Despite the movements in this scene leading to multiple ob- ject interactions, even after 38 time steps, the rendered predictions from the model still sharp and reflect the underlying dynamics. By utilizing the estimated adjacency matrix, there is little drift in the objects, leading to long-term prediction stability. Due its compositional nature, our model generalizes to scenes that contain more or less objects than in the training set, as shown in Fig. where eight objects plus the pusher observed and reconstructed with high quality from novel views, although during training the model has seen only and exactly 4 objects.
Comparison to Von-Compositional Scene Representation Baselines. We compare to two non- compositional baselines where the scene is represented globally with one single latent vector per time-step. The dynamics model for these baselines is an MLP z=+1 action g as an additional input. The first baseline (Global NeRF) is the approach from [2],
F2 g) that takes the
i.e. we use their CNN encoder to produce one latent vector that conditions global NeRF which reconstructs the whole scene (Fig. 14c). The second baseline (Global 2D CNN auto encoder) uses both 2D CNN encoder and 2D CNN decoder as well as single latent vector representing the whole scene (Fig. 4c). Such frameworks have been used many times in the literature, e.g. [1. 64, 65, 66. 67] and are known as visual foresight. Fig. 3 shows that both global basclines are significantly inferior in our scenarios our proposed compositional framework, especially for long horizons.
Comparison to 2D Baselines Importance of NeRF as Decoder. In this section, we replace the NeRF decoder with 2D CNN decoder to investigate the importance of NeRFs. This decoder takes as input one single latent vector and the camera matrix, i.c. 1 = DeNN (z, K). order to make it compositional,
aggregate the set of latent vectors 21:m from Ω with mean operation and then pass the aggregated feature through an MLP to produce the single 2 for DCNN- The rest of the architecture, i.e. implicit object encoder and GNN, stays the same. Since there is clear way to estimate the adjacency matrix from DeNN we use a dense adjacency matrix for the GNN. As one can see in Fig. 3, the long-term prediction performance of the CNN decoder is significantly worse than with a compositional NeRF model as the decoder, especially when asking for numbers of objects that differ from the training distribution. Qualitatively, one reconstruction is much less sharp compared to the NeRF-based models, but especially also that even after only few sime-steps, the predictions with the CNN decoder are of little use.
in Fig. 4c that not only the initial

---

38
(a) Predictions with our method
(a) Ground truth
(b) Predictions with dense adjacency matrix bascline
(b) Reconstruction
Predictions with CNN decoder baseline (no NeRF)
Figure 5: Generalization to twice as many objects as during training.
Visual forward predictions. With our proposed method (a), the predictions are very sharp, even after 38 steps, while with dense adjacency matrix (b) leads to drifting objects until the predictions are not useful anymore. The CNN decoder baseline is even worse, such that after only a few steps the predictions are of little use. Multiple object interactions happen in this scene.
Figure 4:
Comparison to CNN Encoder. Exchanging the implicit object encoder with a 2D CNN composi- tional encoder leads to an auto-encoder framework similar to [32]. As seen in Fig. 3, the performance better compared to the other baselines, but still clearly worse than with the proposed method.
Importance of Estimating the Adjacency Matrix. In Sec. 5, we propose how the adjacency matrix of the GNN can be estimated from the learned NeRFs to increase the long-term stability of the predictions. Here we compare to a dense adjacency matrix, i.e. where the network has to figure out from the latent vectors themselves which objects interact. As a dense has significantly worse long-horizon prediction performance compared to our proposed way of estimating A through the learned NeRF model. In the and 8 object case (generalization over numbers of objects), the predictions with the dense are useless after only a few time-steps.
in Fig. and Fig. 4b,
Non- Non-Compositionall Dynamics. Replacing the GNN with fully connected MLP 2+11 MM (=1.m) leads to worse performance than with a GNN with dense adjacency matrix. This model cannot generalize to different numbers of objects due to its fixed input size.
Summary Performance Comparisons Our method outperforms all baselines both in terms of pure reconstruction error (as can be seen in Fig. by the error after O prediction steps) and its ability to perform long-term predictions forward unrolled on the model's own predictions. Estimating the adjacency matrix from the model itself is important for long-term stability as it prevents objects from drifting away. Too large drift makes future predictions for pushing tasks meaningless. Since the reconstruction error of our proposed method without dynamics is better than the baselines, the question arises if the increased performance an artifact of the lower reconstruction error. We show in Fig. 3d the error in the image space between renderings when having access to the observations each step and the renderings from the predicted latent vectors into future after observing the scene only at the beginning. This shows the increase in error relative to the reconstruction process. The results indicate that not solely the reconstruction itself is the reason for the better performance, but that the structural choices of our framework also enable to learn the dynamics more precisely.
6.2 Planning and Execution Results on Object Sorting Task
To demonstrate the effectiveness of the learned model, we utilize it to solve a box sorting task, where the pusher needs to push colored boxes into their corresponding goal regions as shown in Fig. This task is inspired by [68] and involves multiple challenges: As multiple objects interact, greedy strategy of pushing objects straight to the goal region fails. Movements, i.e. actions, of the pusher often not immediately lead to change the cost function, since contact with the object from suitable side has to be established [54, 63]. In the appendix Sec. E we propose a latent space RRT that uses our framework for planning. Refer the appendix and the video for more details about our proposed planning algorithm and comparisons to baselines.
6.3 Real World Experiments & Deformable Objects
Fig. Ia shows the rendered forward predictions of our model for a real world scenario where a robot pushes shoe and giraffe-shaped toy. Fig. 1b renderings from novel view points. We further show in Fig. 7 that our method is also applicable to deformable objects. The appendix Sec. G contains more details regarding this experiment, including quantitative analysis.
7

---

(a) Initial scene
(b) Final goal achieved
(c) Initial scene
(d) Final goal achieved
Figure 6: Two planning scenarios with leamned visual dynamics model and latent space RKT. The goal is the blue and yellow boxes into their respective shaded areas. (a), (c) initial states, (b), (d) shows the achieved goal the end the planning/cxecution loop.
19
ground truth
forward predictions
Figure 7: Forward predictions for deformable object scenario. The predicted reconstructions based the latent vector from the initial observation (t 0) that is then forward predicted with the dynamics model.
7 Discussion & Limitations
The appendix Sec. A contains an extended discussion regarding limitations of this work.
Object Masks. The compositional scene encoding framework requires object masks to achieve compositionality. Many mature methods for instance segmentation have been developed such that we believe having masks as input is a reasonable assumption and we have also shown the appli- cability of our method in real world scenarios where no precise masks are available. We further investigate in Sec. F.4 and Sec. G.2 the robustness of our method with respect to mask perturbations. Methods for unsupervised discovery objects [32, 69] could also be integrated into our pipeline.
Latent Representations. We have shown the great benefits of a compositional latent representation as it not only provides generalization over different numbers of objects in the scene, but also leads to increased reconstruction and dynamics prediction performance compared to non-compositional baselines. Furthermore, latent representations compress observations, enabling efficient dynamics prediction. As each object in the scene represented as a latent vector of finite size, one could argue that latent models are capable of mainly representing objects with shapes similar to the training dis- tribution. We currently use single neural network to represent all objects. Therefore, all variations controlled by this latent vector (of size 64 in our experiments). Despite
in object appearance this being a rather small latent space, our framework already exhibits interpolation capabilities for object shapes and poses. The variety of the scenes considered in this work, including the deformable object, can sufficiently be represented with this latent vector. Therefore, we are confident that by exposing the method to more diverse set of objects and potentially increasing the size of the la- tent space, one could see generalization capabilities beyond the training object category distribution, which is interesting area for future research.
Long-Tere Term Prediction Stability. Our dynamics model framework exhibits significantly better long-term prediction stability compared to baselines. Our experiments indicate that this is due the structural biases enabled through (compositional) NeRFs. This stability allowed for planning scenarios requiring long-horizons, which none of the baseline methods could support. However, believe that there is still room for improvement regarding the prediction stability.
the model
8 Conclusion
Visual dynamics models of high interest to the computer vision and robotics community, they avoid explicit shape model assumptions and imply end-to-end perception. However, to support ma- nipulation planning and reasoning need models that generalize strongly over objects and provide stable long-term predictions. In this paper we proposed a system that introduces 3D structural and compositional priors at various levels, namely compositional NeRFs, 3D implicit object encoders, and GNNs dynamics with an adaptive adjacency matrix. Together our system exhibits significantly stronger long-tern term prediction performance compared to multiple basclines without these priors or without compositionality. and supports using a latent space RRT planner. We have shown general- ization over different numbers of objects, notably up to two times more than during training.

---

Acknowledgments
This research has been supported by the Deutsche Forschungsgemmiinschaff (DFG, German Research Foundation) under Germany's 's Excellence Strategy EXC 2002/1 "Seience of Intelligence" project number 390523 135 and Amazon. com Services, LLC PO# #2D-063 00236. Danny Driess thanks the International Max-Planck Research School for Intelligent Systems (IMPRS-IS) for the support. The authors thank Valentin Hartmann for discussions regarding RRTs.
References
[1] Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, and S. Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv: 1rXiv:8812.00566. 2018.
[2] Y. Li, S. Li, V. Sitzmann, Agrawal, and A. Torralba. 3d neural scene representations for visuomotor control. In Conference Robot Learning, pages 112-123. PMLR, 2022.
[3] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pages 405-421, 2020.
[4] R. Tedrake. Underactuated Robotics: Algorithms for Walking. Running, Swimming, Flying, and Manipulation (Course Notes for MIT 6.832). 2022. URL http:/ http:://undrractuated. mit.edu.
[5] P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanzhez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: (.0.:0/800.0266,,, 2018.
[6] P. W. Battaglia, R. Pascanu, M. Lai, D. Rezende, and K. Kavukcuoglu. Interaction networks for learning about objects, relations and physics. arXiv preprint arXiv:1612.00222, 2016.
[7] M. B. Chang, T. Ullman, A. Torralba, and B. Tenenbaum. A compositional object-based approach to learning physical dynamics. arXiv preprint arXiv:1612.00344 2016.
[8] Y. Li, J.-Y. Zhu,J J. B. Tenenbaum, A. Torralba, and R. Tedrake. Propagation networks for model-based control under partial observation. In 2019 International Conference on Robotics and Automation (ICRA), pages 1205-1211. IEEE, 2019.
J.-Y.
[9] A. Sanchez-Gonzalez, N. Heess, J. T. Springenberg. J. Merel, M. Riedmiller, R. Hadsell, and Battaglia. Graph networks as learnable physics engines for inference and control. In Inter- national Conference on Machine Learning, pages 4470-4479. PMLR, 2018.
[10] N. Funk, G. Chalvatzaki, B. Belousov, and J. Peters. Leanbassemble with structured represen- tations and search for robotic architectural construction. In Proceedings of the Sth Conference on Robot Learning, 2022.
[11] T. Silver, R. Chitnis, A. Curtis, J. B. Tenenbaum, T. Lozano-Peree and L. P. Kaelbling. Plan- ning with learned object importance in large problem instances using graph neural networks. In Proceedings of the AAAI conference artificial intelligence, 2021.
[12] D. Mrowca, C. Zhuang, E. Wang, N. Haber, L. Fei-Fei, J. B. Tenenbaum, and D. L. Yamins. Flexible neural representation for physics prediction. arXiv preprint I 08047, 2018.
[13] Y. Li, J. Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint arXiv:1810.015: 2018.
[14] Y. Li, Lin, K. Yi, D. Bear, D. Yamins, J. Wu, J. Tenenbaum, and A. Torralba. Visual grounding of learned physical models. In International conference machine learning, pages 5927-5936. PMLR, 2020.
[15] B. Ummenhofer, L. Prantl, N. Thuerey, and V. Koltun. Lagrangian fluid simulation with con- tinuous convolutions. In International Conference Learning Representations, 2019.

---

[16] A. Sanchez-Gonzalez, J. Godwin, T. Pfaff, R. Ying, J. Leskovec, and P. Battaglia. Learning simulate complex physies with graph networks. In International Conference on Machine Learning, pages 8459-8440 8468. PMLR, 2020.
[17] T. Pfaff, M. Fortunato, A. Sanchez-Gonzaler and P. W. Battaglia. Learning mesh- based sim- ulation with graph networks. arXiv preprint arXiv:2010.03409, 2020.
[18] Y. Ye, M. Singh, A. Gupta, and S. Tulsiani. Compositional video prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10353-10362, 2019.
[19] J.-T. Hsich, B. Liu, D.-A. Huang. L. Fei-Fei, and J. C. Niebles. Learning to decompose and disentangle representations for video prediction. arXiv preprint 1rXivv1806.04166, 2018.
[20] N. Watters, D. Zoran, T. Weher, P. Battaglia, R. Pascanu, and A. Tacchetti. Visual interaction networks: Learning a physics simulator from video. Advances in neural information process- ing systems, 30:4539 4547, 2017.
[21] K. Yi, C. Gan, Y. Li, P. Kohli, J. Wu, A. Torralba, and J. B. Tenenbaum. Clevrer: Collision events for video representation and reasoning, arXiv preprint arXiv:1910.01422, 2019.
[22] H. Qi, X. Wang, D. Pathak, Y. Ma, and J. Malik. Learning long-temm term visual dynamics with region proposal interaction networks. arXiv preprint arXiv:2008.0222 2020.
[23] H.-Y. F. Tung, Z. Xian, M. Prabhudesai, S. Lal, and K. Fragkiadaki. 3d-oes: Viewpoint- invariant fbcect-factorized environment simulators. arXiv preprini 1.06644,, 2020.
[24] G. Zhu, Z. Huang, and C. Zhang. Object-oriented dynamics predictor. In Advances in Neural Information Processing Systems, 2018.
(25) Y. Xie, T. Takikawa, S. Saito, O. Litany, S. Yan, N. Khan, F. Tombari, J. Tompkin, V. Sitz- mann, and S. Sridhar. Neural fields visual computing and beyond. arXiv preprint arXiv:2111.114266 2021.
[26] L.. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14600-4700 2019.
[27] S. Peng, M. Niemeyer, L. Mescheder, M. Pollefeys, and A. Geiger. Convolutional occupancy networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 523-540. Springer, 2020.
[28] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning con- tinuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165-174, 2019.
[29] V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene representation networks: Continuous 3d-structure-aware neural scene representations. arXiv preprint UrXiv:/906.0/66 01618, 2019.
[30] S. Saito, Z. Huang, R. Natsume, S. Morishima, A. Kanazawa, and H. Li. Pifu: Pixel- aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2304-2314, 2019.
[31] M. Guo, A. Fathi, J. Wu, and T. Funkhouser. Object-centric neural scene rendering. arXiv preprint arXiv:2012.08803, 2020.
[32] K. Stelzner, K. Kersting, and A. R. Kosiorek. Decomposing 3d scenes into objects via unsu- pervised volume segmentation. arXiv preprint arXiv:2104.01148, 2021.
[33] Ost, F. Mannan, N. Thuerey, J. Knodt, and Heide. Neural scene graphs for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2856-28666 2021.
[34] H.-X Yu, J. Guibas, and J. Wu. Unsupervised discovery of object radiance fields. arXiv preprint arXiv:2107.07905, 2021.
10

---

[35] C. Elich, M. R. Oswald, M. Pollefeys, and J. Stueckler. Weakly supervised learning of multi- object 3d scene decompositions using deep shape priors. arXiv preprint A.Ki::2010004000, 2020.
[36] A. Kundu, K. Genova, X. Yin, A. Fathi, C. Pantofaru, L. J. Guibas, A. Tagliasacchi, F. Del- laert, and T. Funkhouser. Panoptic neural fields: A semantic object-aware neural scene rep- resentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12871-12881 2022.
[37] A. Yu, V. Ye, M. Tancik, and A. Kanazawa. pixelnerf: Neural radiance fields from or few images. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages
2021.
(38) A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318-10327, 2021.
[39] K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R. Martin- Bartal. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865-5874, 2021.
[40] Y. Du, Y. Zhang. H.-X. Yu, J. B. Tenenbaum, and J. Wu. Neural radiance flow for 4d view synthesis and video processing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14324-14334, 2021.
[41] H. Xu, T. Alldieck, and C. Sminchisescu. H-nerf: Neural radiance fields for rendering and temporal reconstruction of humans in motion. Advances in Neural Information Processing Systems, 34, 2021.
[42] K. Park, U. Sinha, P. Hedman, J. Barron, S. Bouaziz, D. B. Goldman, R. Martin-Brualla, and S. M. Seitz. Hypernerf: A nigher-dimensional representation for topologically varying neural radiance fields. arXiv preprint 2tt.::20....122222 2021.
[43] M. Niemeyer, Mescheder, M. Oechsle, and Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5379-5389, 2019.
[44] E. Tretschk, A. Tewari, V. Golyanik, M. Zollhöfer, C. Lassner, and C. Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12959-12970, 2021.
[45] Z. Li, S. Niklaus, N. Snavely, and O. Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 6498-6508, 2021.
[46] W. Xian, J.-B. Huang. J. Kopf. and C. Kim. Space-time neural irradiance fields for free- viewpoint video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
[47] Li, M. Slavcheva, M. Zollhoefer, S. Green, C. Lassner, Kim, T. Schmidt, Lovegrove, M. Goesele, and Z. Lv. Neural 3d video synthesis. arXiv preprint orXiv:2103.02597 2021.
[48] M. Breyer, J. J. Chung, L. Ott, S. Roland, and N. Juan. Volumetric grasping network: Real-tim
6 dof grasp detection clutter. In Conference on Robot Learning. 2020.
[49] Z. Jiang, Y. Zhu, M. Svetlik, K. Fang, and Y. Zhu. Synergies between affordance and geometry: 6-dof grasp detection via implicit representations. arXiv preprint arXiv:2 104.01142, 2021.
[50] M. Van der Merwe, Q. Lu, B. Sundaralingam, M. Matak, and T. Hermans. Learning continuous 3d reconstructions for geometrically aware grasping. In Int. Conf. Robotics and Automation (ICRA), 2020.
[51] Ichnowski, Y. Avigal, J. Kerr, and K. Goldberg. Dex-nerf: Using a neural radiance field to grasp transparent objects. arXiv preprint arXiv:21 10,17,7,,022
11

---

[52] B. Shen, Z. Jiang, C. Choy, L. J. Guibas, S. Savarese, A. Anandkumar, and Y. Zhu. Acid: Action-conditional implicit visual dynamics for deformable object manipulation. arXiv preprint WrXiv:2203.06880 2022.
[53] L. Yen-Chen, P. Florence, J. T. Barron, T.YY Lin, A. Rodriguez, and P. Isola. Nerf- supervision: Learning dense object descriptors from neural radiance fields. arXiv preprint vrXiv:2203.0911: 2022.
[54] D. Driess, J.-S. Ha, M. Toussaint, and R. Tedrake. Learning models as functionals of signed- distance fields for manipulation planning. In Conference on Robot Learning (CoRL), 2021.
[55] J.-S. Ha, D. Driess, and M. Toussaint. Learning neural implicit functions as object representa- tions for robotic manipulation. arXiv preprint 1rX1v:21 2021.
[56) K. Hauser. Semi-infinite programming for trajectory optimization with non-convex obstacles. The International Journal of Robotics Research, 2018.
[57] S. Pfrommer, M. Halm, and M. Posa. Contactnets: Learning of discontinuous contact dynamics with smooth, implicit representations. Conference on Robot Learning, 2020.
[58] M. Adamkiewicz, T. Chen, A. Caccavale, R. Gardner, P. Culbertson, J. Bohg, and M. Schwa- ger. Vision-only robot navigation in a neural radiance world. IEEE Robotics and Automation Letters, 2022. doi: 10. 1099
[59] M. Niemeyer and A. Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
[60] F. R. Hogan and A. Rodriguez. Feedback control of the pusher-slider system: A story of hybrid and underactuated contact dynamics. arXiv preprint arXiv:1611.08266 2016.
[61] Zhou, Y. Hou, and M. T. Mason. Pushing revisited: Differential flatness, trajectory plan- ning, and stabilization. The International Journal of Robotics Research, 38(12-13):1477-4443 2019.
[62] D. Driess, J.-S. Ha, and M. Toussaint. Learning to solve sequential physical reasoning prob- lems from scene image. The International Journal of Robotics Research (IJRR), 2021. doi: 10. 10.1177/0278364921 1556667.
[63] I. Schubert, D. Driess, O. S. Oguz, and M. Toussaint. Learning to execute: Efficient learning of universal plan- conditindi policies in robotics. In NeurlPS 2021 Neeral Information Processing Systems, 2021.
[64] M. Watter, J. T. Springenberg, J. Boedecker, and M. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. arXiv preprint 150.:::5506003666 2015.
[65] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pages 2555 2565. PMLR, 2019.
[66] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:9912.0103,, 2019.
[67] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, T. Graepel, al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839) 2020.
[68] P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mor- datch, and Tompson. Implicit behavioral cloning. In Proceedings of the Sth Conference on Robot Learning. PMLR, 2022.
[69] M. S. Sajjadi, D. Duckworth, A. Mahendran, S. van Steenkiste, F. Pavetić, M. Lučić, L. J. Guibas, K. Greff, and T.. Kipf. Object scene representation transformer. arXiv preprint arXiv:2206.0663 2022.
12

---

[70] C. Reiser, S. Peng. Y. Liao, and A. Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mips. In International Conference on Computer Vision (ICCV), 2021.
[71] C. P. Burgess, L. Matthey, N. Watters, R. Kabra, I. Higgins, M. Botvinick, and A. Ler- chner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:1901.1 11390, 2019.
[72] K. Greff, R. L. Kaufman, R. Kabra, N. Watters, C. Burgess, D. Zoran, L. Matthey, M. Botvinick, and A. Lerehner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, pages 2424-2433. PMLR, 2019.
[73] N. Li, C. Eastwood, and R. Fisher. Learning object-centric representations of multi-object scenes from multiple views. Advances Neural Information Processing Systems, 33:5656- 5666, 2020.
[74] R. Veerapaneni, J. D. Co-Reyes, M. Chang, M. Janner, C. Finn, J. Wu, J. Tenenbaum, and Levine. Entity abstraction in visual model-based reinforcement learning. In Conference on Robot Learning, pages 1439-144000 PMLR, 2020.
[75] Z. Lin, Y.-. Wu, S. Peri, B. Fu, J. Jiang, and S. Ahn. Improving generative imagination in object-centric world models. In International Conference on Machine Learning, pages 6140- 6149. PMLR, 2020.
[76] T. Kipf, E. van der Pol, and M. Welling. Contrastive learning of structured world models. arXiv preprint arXiv: 1911. 12247, 2019.
[77] D. Driess, J.-S. Ha, and M. Toussaint. Deep visual reasoning: Learning to predict ac- tion sequences for task and motion planning from an initial scene image. arXiv preprint urXiv:2006.05398 2020.
[78] M. Toussaint, J.-S. Ha, and D. Driess. Describing physics for physical reasoning: Force-based sequential manipulation planning. IEEE Robotics and Automation Letters, 2020.
[79] V. N. Hartmann, A. Orthey, D. Driess, O. Oguz, and M. Toussaint, Long-horizon multi-robot rearrangement planning for construction assembly. arXiv preprini arXiv:2 arXiv:2106.022443 106.02489, 2021.
[80] M. Bauza, F. R. Hogan, and A. Rodriguez. data-efficient approach to precise and controlled pushing. In Conference on Robot Learning, pages 336-345, PMLR, 2018
[81] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pages 1101-1112. PMIR, 2020.
[82] C. Finn, Goodfellow, and S. Levine. Unsupervised learning for physical interaction through video prediction. arXiv preprint 105.0777
[83] F. Ebert, C. Finn, A. X. Lee, and S. Levine. Self-supervised visual planning with temporal skip connections. In CoRL, pages 344-356, 2017.
[84] C. Schenck and D. Fox. Perceiving and reasoning about liquids using fully convolutional networks. The International Journal of Robotics Research, 47(4-5):422-711 2018.
[85] H. Suh and R. Tedrake. The surprising effectiveness of linear models for visual foresight in object pile manipulation. arXiv preprint arXiv:2002.0909) 2020.
[86] D. Driess, J.-S. Ha, R. Tedrake, and M. Toussaint. Learning geometric reasoning and control for long-horizon tasks from visual input. In Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA), 2021.
[87] Z. Xu, Z. He, J. Wu, and S. Song. Learning 3d dynamic scene representations for robot manipulation. arXiv preprini w:Xiv:2011.01968 2020.
[88] M. Strecke and J. Stueckler. DiftSDFSim: Differentiable rigid-body dynamics with implicit shapes. In International Conference on 3D Vision (3DV), Dec. 2021.
[89] Y. Wi, P. Florence, A. Zeng, and N. Fazeli. Virdo: Visio-tactile implicit representations of deformable objects. arXiv preprint arXiv:2202.0008 2022.
13

---

[90] D. Kulkarni, A. Gupta, C. Ionescu, S. Borgeaud, M. Reynolds, A. Zisserman, and V. Mnih. Unsupervised learning object keypoints for perception and control. Advances neural information processing systems, 32: 12:10724-10734, 2019.
[91] L. Manuelli, Y. Li, P. Florence, and R. Tedrake. Keypoints into the future: Self-supervised cor- respondence model-based reinforcement learning. arXiv preprint (rXiv:2009.05088 2020.
[92] Y. Li, A. Torralba, A. Anandkumar, D. Fox, and Garg. Causal discovery in physical systems from videos. Advances in Neural Information Processing Systems, 33, 2020.
[93] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 1134-1141. IEEE, 2018.
[94] P. Agrawal, A. Nair, P. Abbeel, J. Malik, and S. Levine. Learning poke by poking: Experi- ential learning of intuitive physics. arXiv preprint arXiv: 07449, 2016.
[95] D. P. Kingma and J. Ba. Adam: method for stochastic optimization. arXiv preprint a7X1v:1412.6988, 2014.
[96] B. Ichter and M. Pavone. Robot motion planning in learned latent spaces. IEEE Robotics and Automation Letters, 4(3):2407-2413 2019.
[97] E. F. Camacho and C. B. Alba. Model predictive control. Springer science & business media, 2013.
[98] Johnson, B. Hariharan, L. Van Der Maaten, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference computer vision and pallern recognition, pages 2901- 2910, 2017.
14


Reviews:
[{"review_id": "kvW578INHK1", "reviewer": "Reviewer_ZKto", "summary": "The authors propose to compositionally encoder scene RGB observations into implicit neural representations. The dynamics is represented as a factored graph. The authors demonstrated the validity of their approach with various experiments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n* The paper is very well written. I had a great time going over the paper with all the interesting details. Things are presented in a clear manner, and a lot of content is packed into the short pages.\n* The method is quite reasonable. It captures the \"3D-ness\" of objects, as well as their more lower dimensional dynamics relationship.\n* A lot of the relevant works are provided, showcasing the authors' rich knowledge on the field. This is also helpful for the readers.\n\nWeakness:\n* I think some quite related recent works along the direction is not cited. For example, using implicit neural representation to represent deformable object dynamics: ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation Bokui Shen, Zhenyu Jiang, Christopher Choy, Leonidas J. Guibas, Silvio Savarese, Anima Anandkumar, Yuke Zhu, RSS, 2022.       NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields. Lin Yen-Chen, Pete Florence, Jonathan T. Barron, Tsung-Yi Lin, Alberto Rodriguez, Phillip Isola. In ICRA 2022.\n* I would love to hear more about the authors' thoughts on generalization. I think this might be a limitation of the approach that's worth discussing. When working with an open set of objects, will such approach still work? It will be really hard to keep a large library of INRs, and the GNN will have a difficult time reasoning if its library of latent codes grow more and more.\n* Supplementary material file doesn't have a bibliography. Seems like a bug.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is an acceptance in my opinion. However, I would love to hear more from the authors about how they see such approach generalizing and being used on more objects that are unseen. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "* buggy supplementary bibliography", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "dzjL9QuE3U", "reviewer": "Reviewer_35gm", "summary": "This paper studies the problem of multi-object video/motion prediction problem in robotics. Specifically, the framework consists of three modules: the encoder, GNN dynamics model, and NeRF decoder. First, the encoder takes in initial observations of the scene from multiple views and computes feature volume and latent variables for each individual object. Second, the NeRF decoder renders the view for reconstruction. Third, the GNN dynamics model learns to predict future latent variables using the adjacency matrix inferred from the NeRF density between each of the two objects. In practice, the encoder and decoder models are first trained end-to-end using the L2 image reconstruction loss, and GNN is further learned using the inferred latent variables. Experiments are mainly conducted on the non-public synthetic dataset and a few real objects (up to 4).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n* [S1] This paper studies an important task in robotic manipulation by leveraging vision signals (e.g., appearance, motion, and shape) for representation learning. \n* [S2] The paper provides sufficient technical details (especially in the Supp) for replication.\n\nWeaknesses\n\n* [W1] The proposed framework assumes that accurate object masks are given as input to the model, which is not realistic. Although this has been discussed in section 7, the reviewer would like to know how robust the proposed model is given inaccurate masks provided using off-the-shelf instance segmentation & tracking models. Ablation studies on this are highly recommended for the next version of the paper.\n* [W1.1] With inaccurate object masks, the reviewer suspects the proposed method may suffer from sudden lighting changes or ID switches (e.g., all objects with the same appearance). It would be great to clarify this.\n\n* [W2] The motion dynamics shown in the paper and demo are easily predictable using a simple regression model. For example, if the objects of interest are articulated intelligent agents (e.g., toys) which can move by themselves, motion prediction involves a higher level of uncertainty and multi-modality. The reviewer fails to see how does GNN model handle this in the current setting. Please discuss this more explicitly as the limitation of the proposed model.\n\n* [W3] This paper conducts experiments on non-public datasets, which makes it a bit challenging to understand the significance of the work. In addition, experimental evaluations only include ablations of different components of their proposed method rather than existing SoTA implementations. The reviewer highly recommends adding results on public benchmarks (e.g., Multi-Object Datasets and Kubrik [NewRef1-2]) even without the dynamic GNN.\n\n* [W4] Besides prior work [31-35], there is more recent work on modeling dynamic scenes using compositional NeRFs (e.g. [NewRef3]). It would be good to discuss these existing works and consider them as potential baselines in the next version of the paper.\n\nReferences\n* [NewRef1] Multi-Object Datasets. [GitHub](https://github.com/deepmind/multi_object_datasets).\n* [NewRef2] Kubric: A scalable dataset generator, In CVPR 2022.\n* [NewRef3] Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation, In CVPR 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This is a borderline paper. On the positive side, it studies an important problem with meaningful extensions to the previous work. However, the reviewer holds reservations due to the strong assumption on object mask, over-simplified motion setting, and significance of the work. The reviewer would also like to exchange ideas with peer reviewers after the rebuttal.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Mentioned in [W1-W4].", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "GBlGDEKx3X-", "reviewer": "Reviewer_d3gL", "summary": "The authors propose a method for learning dynamics from multi-view series data of an environment consisting of multiple objects. The proposed method has NeRF to include 3D inductive bias and GNN to extract the compositional representation to enable long-term prediction. Experiments on both simulators and real-world situations confirm the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well organized and easy to read.\n- The proposed method is valid and novel compared to existing NeRFs.\n- Experiments confirm its effectiveness on complex datasets of multiple objects. In particular, it is very significant that it is demonstrated to work properly on deformable objects and objects in real-world environments.\n\nWeaknesses:\n- In this paper, the authors aim to obtain object-specific representations from multi-view images and to predict future observations. However, the authors do not mention many related works on similar tasks (although some are cited, such as [32]). For example, many studies on acquiring object-specific representations from images or videos for segmentation and prediction have been studied as object-centric representation learning (or object-centric world model) (1,2,3,4,5). In particular, in (5), object-specific representations are acquired from videos using graph neural networks. There is also work on learning object-centric representations in multi-view (6). I think this proposed method has novelty compared to these in that it acquires object representations in multi-view and time series, but the authors should cite these studies and explicitly discuss the differences.\n(1) MONet: Unsupervised Scene Decomposition and Representation\n(2) Multi-Object Representation Learning with Iterative Variational Inference\n(3) Entity Abstraction in Visual Model-Based Reinforcement Learning\n(4) Improving Generative Imagination in Object-Centric World Models\n(5) Contrastive Learning of Structured World Models\n(6) Learning Object-Centric Representations of Multi-Object Scenes from Multiple Views\n- The authors state that \"we believe having masks is a reasonable assumption,\" but many of the above object-centric methods do not assume that masks are given, and acquire object-specific representations only from images and videos. Therefore, it is not obvious that this assumption is reasonable in my opinion. The authors would like more explanation as to what perspective this assumption is valid compared to related studies.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The method is novel and its ability to make long-term predictions in the real world is a significant contribution; however, as mentioned above, I still have concerns regarding the lack of citation of previous studies and the appropriateness of the problem setting. Therefore, I consider it a weak rejection.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "I would like you to work on the concerns I have listed in \"weaknesses\". If you respond appropriately to these and revise your paper to reflect them, I will consider raising the score.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "3Yri2IO74yI", "reviewer": "Reviewer_eGyX", "summary": "The paper leverages NeRF for multi-object scene representation from image observation in a compositional manner. NeRF based autoencoders are deployed to encode each object into latent vectors, which were used compositionally to learn a compositional forward dynamic model using GNN.  The learnt model generalized across objects and provides stable long-term prediction that helps with manipulation planning and reasoning. RRT based methods can then be used to plan in the latent space for downstream manipulation tasks.\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The paper shows that compositional NeRF and GNN can produce a latent space where long horizon prediction is stable. Examples are shown for both simulation and real environment. And thanks to the nature of NeRF, the latent space is agnostic to view points. \n- The method can be generalized to the number of objects and handles the interaction between objects well, even when the object is deformable.\n- Comprehensive ablation studies are performed to support the design choices of the model components.\nWeaknesses:\n- For experiment 6.1, using image error only to measure the long horizon prediction error is not so convincing. NeRF is known to be a much better decoder to represent static 3d scenes compared to CNN. Even without forward dynamic prediction, CNN is not expected to have better reconstruction quality. But it doesn’t necessarily mean the latent space learnt with a CNN decoder is not suitable for long horizon dynamic prediction. From the supplementary video, CNN decoder can also capture the pose of the objects reasonably well. It is worth reporting the pose error of objects as well. Also there are a few questions regarding figure 3. \n  - MLP dynamic model is shown on fig 3(b) only.\n  - Why does the prediction error go down after 40 steps?\n  - The y-axis scale of fig 3c is different from the rest.\n  - It would be great to have more visualizations when using Global NeRF as decoder in Fig 4. It is interesting to see how the compositionality improves the decoder.\n- For experiment 6.2, only qualitative results are shown. More statistics would be helpful to determining the contribution of each module compared to the baseline. For example success rate, avg. time/step taken to complete the task, etc. Without them, the experiment is not robust enough to support the claim that the method produces better latent space for planning and reasoning for robotic tasks.\n- Similarly, only qualitative results are shown for the real world experiments. According to the video, the image prediction is no longer sharp when the objects are textured. This makes the visual reconstruction quality questionable as the proposed method might not be able to scale to more complex objects (in terms of shape and textures).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper shows that compositional NeRF and GNN can produce a latent space where long horizon prediction is stable. But the supporting experiments are weak. See above for more details.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See above", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)