3D Neural Scene Representations for Visuomotor Control
Authors: Yunzhu Li
Shuang Li
Vincent Sitzmann
Pulkit Agrawal
Antonio Torralba
Venue: CoRL2021 Oral (2021)
Keywords: learning-based dynamics modeling
3d-aware representation learning
neural radiance field
robotic manipulation

Abstract: Humans have a strong intuitive understanding of the 3D environment around us. The mental model of the physics in our brain applies to objects of different materials and enables us to perform a wide range of manipulation tasks that are far beyond the reach of current robots. In this work, we desire to learn models for dynamic 3D scenes purely from 2D visual observations. Our model combines Neural Radiance Fields (NeRF) and time contrastive learning with an autoencoding framework, which learns viewpoint-invariant 3D-aware scene representations. We show that a dynamics model, constructed over the learned representation space, enables visuomotor control for challenging manipulation tasks involving both rigid bodies and fluids, where the target is specified in a viewpoint different from what the robot operates on. When coupled with an auto-decoding framework, it can even support goal specification from camera viewpoints that are outside the training distribution. We further demonstrate the richness of the learned 3D dynamics model by performing future prediction and novel view synthesis. Finally, we provide detailed ablation studies regarding different system designs and qualitative analysis of the learned representations.

3D Neural Scene Representations for Visuomotor Control
Yunzhu Li" MIT CSAIL
Shuang Li* MIT CSAIL 1 lishuang@mit.ede
Vincent Sitzmann MIT CSAIL si sitzmann@mit. edu
liyunzhuGmit.edu
Pulkit Agrawal MIT CSAIL
Antonio Torralba MIT CSAIL
gulkitagdmit.ees edu
torralbadmit.een edu
A Abstract:
Humans have strong intuitive understanding of the 3D environment around us. The mental model of the physics in our brain applies to objects of different perform wide range of manipulation tasks that are far beyond the reach of current robots. In this work, we desire to learn models for dynamic 3D scenes purely from 21) visual observations. Our model combines Neural Radiance Fields (NeRF) and time contrastive learning with an autoencoding framework, which learns viewpoint-invariint 3D-aware scene representations. We show that a dynamics model, constructed over the learned representation space, enables visuomotor control for challenging manipulation tasks involving both rigid bodies and fluids, where the target is specified in viewpoint different from what the robot operates on. When coupled with an auto-decoding framework, it can even support goal specification from camera viewpoints that are outside the training distribution. We further demonstrate the richness of the learned 3D dynamics model by performing future prediction and novel view synthesis. Finally, we provide detailed ablation studies regarding different system designs and qualitative analysis of the learned representations.
materials and enables
Introduction
Existing state-of-the-an model-based systems operating from vision typically treat images as grids of pixels [1, 2, 3]. The world, however, is dinee-dimensional. Modeling the environment from 3D enables amodal completion and allows the agents to operate from different views. Therefore, is desirable to obtain good 3D-aware representations of the environment from 2D observations to achieve better task performance when an accurate inference of 3D information is essential, which can further make it easier to specify tasks, learn from third-person videos, etc.
One of the core questions of model learning in robotic manipulation is how determine the state representation for learning the dynamics model. The desired representation should make it easy to capture the environment dynamics, exhibit good 3D understanding of the objects in the scene, and be applicable to diverse object sets such as rigid or deformable objects and fluids. One line of prior work learns the dynamies model directly in the image pixel space [4, 5, 6, 7]. However, modeling dynamics in such a high-dimensional space is challenging, and these methods typically generate blurry images when performing the long-horizon future predictions. Another line of work focused on only predicting task-relevant features identified as keypoints [8. 9, 10. 11, 12]. Such models perform well in terms of category- level generalization, i.e., the same set of keypoints can represent different instances within the same category, but not sufficient model objects with large variations like fluids and granular materials. Other methods learn dynamics in the latent space [13, 14, 15, 2, 3]. However, the majority of these methods learn dynamics models using 2D convolutional neural networks and reconstruction loss - which has the same problem as predicting dynamics in the image
"qqul contribution. Project Page: htgp://gg//eeeepppreetattaaaaaaa
github. 1o/nerf-dy/
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Control resalt of 2D 2D-hased approach
Control result of our AD-aware approach
Goal image to achieve (camera view outside the training distribution)
Control results from the robot observation view
Control results from the goal image view
Control results from the robot observation view
Control results from the goal image view
Comparison of the control results between 2D-based baseline and 3D-aware approach. The task here is to achieve the configuration shown on left, observed from a viewpoint that is outside the training distribution. The agent only takes a single-view visual observation as input (images with blue frames) from a viewpoint that is vastly different from the goal. Our method generalizes well in this scenario and outperforms the 2D-based bascline, demonstrating the benefits of the learned 3D-aware scene representations.
Figure 1:
space, i.e., their learned representations lack equivariance to 3D transformations. Time contrastive networks [16], the other hand, aim learn viewpoint-invariant representations from multi-view inputs, but do not require detailed modeling of 3D contents. As a result, previously unseen scene configurations and camera poses are out-of-distribution samples for the state estimator. As we will see, this leads to wrong state estimates and results in faulty control trajectories.
Meanwhile, recent work in computer vision has made impressive progress in the learning of 3D- structured neural scene representations. These approaches allow inference of 3D structure and appearance, trained only given 2D observations, either by overfitting by generalizing across scenes [20, 21, 22]. Through their 3D inductive bias, the scene representations inferred by these models encode the scene contents with better accuracy and are invariant to changes in camera perspectives. It is desirable to push these ideas further to obtain a deeper understanding of how these methods, which directly reason over 3D, can bring in new characteristics and how they can be beneficial for dynamics modeling and complicated control tasks.
single scene [17, 18, 19] or
In this work, we aim to leverage recently proposed 3D-structure-aware implicit neural scene repre- sentations for visuomotor control tasks. We thus propose to embed neural radiance fields [19] in an auto-encoder framework, enabling tractable inference of the 3D 3D-structure- aware scene state for dynamic environments. By additionally enforcing a time contrastive loss on the estimated states, we ensure that the learned state representations that predicts the evolution of the state space conditioned on the input action, enabling us to perform control in the learned state space. Though the representation itself is grounded in the 3D implicit field, the convolutional encoder is not. At test time, we overcome this limitation by performing [23, 20], enabling accurate state estimation even for out-of-distribution camera poses and, therefore, control of tasks where the goal view is specified in an entirely unseen camera perspective. These contributions enable us perform model-based visuomotor control of complex scenes, modeling 3D dynamics of both rigid objects and fluids. Through comparison with various baselines, the learned representation from our model is more precise describing the contents of 3D scenes, which allows it to accomplish control tasks involving rigid objects and fluids with significantly better accuracy (Figure 1). Please see our supplementary video for better visualization.
viewpoint-invariant We then train a dynamics model
diferrence-vi--pptiiizee
We summarize our contributions as follows: (i) We extend an autoencoding framework with a neural radiance field rendering module and time contrastive learning that allows us to learn 3D- aware scene representations for dynamics modeling and control purely from visual observations. (ii) By incorporating the auto-decoder mechanism at test time, our framework can adjust the learned representation and accomplish the control tasks with the goal specified from camera viewpoints outside the training distribution. (iii) We are the first to augment neural radiance fields using a time-invarian dynamics model, supporting future prediction and novel view synthesis across wide range of environments with different types of objects.
2 Related Work
3D Scene Representation Learning. Prior work leverages the latent spaces of autoencoder-like models as learned representations of the underlying 3D scene to enable novel view synthesis from

---

Encoder
Decoder
Dynamies
Image at t'
Multi- view input
(4+
[Le]
images time
Rey
Ray
Encoder
Action a,
Dynamics Prediction
Sayz
Imsage Encoder
Image Incoder
Encoder
Image feature
(1u1)
Encoder
2, y,
Radiance and Valume Density RRe C2,, B., oz)
Negative
Postlive
State Encoder
View direction 9, +
Volume Rendering
924-1
8s4-1
Contrastive Lass
State representation: 8t
on Pinels
1.2 Loss
(4) Learning 3D-Aware State Space
(b) Learning Predictive Models
Figure 2: Overview of the training procedure. Left: encoder that maps the input images into latent scene representation. The images are first sent to an image encoder generate image feature representations v. Then we combine the image features from the same time step using a state encoder to obtain the state representation &;. A time contrastive loss is applied to enable our model to be invariant to camera viewpoints. Middle: decoder that takes the scene representation as input and generates the visual observation conditioned on a given viewpoint. We use an 1.2 loss ensure the reconstructed image he similar the ground truth current
image. Right: a dynamies model that predicts the future scene representations $141 by taking representation 8, and action as. We use an L2 loss to enforce the predicted latent representation to be similar to the scene representation 814-1 extracted from the true visual observation 44+1.
single image [24, 25]. Eslami al. [26] embed this approach probabilistic framework. To
endow models with 3D structure, voxelgrids can be leveraged as neural scene representations [27.
28, 17. 29, 30], while others have tried to predict particle sets from images [31] or embed an explicit 3D representation to enable inference from never-hefore-see
viewpoints [32]. Sitzmann et al. [20]
propose to learn neural implicit representations of 3D shape and appearance supervised with posed 2D images via differentiable renderer. Generalizing across neural implicit representations can also be realized by local conditioning on CNN features [33, 34, 35], but this does not learn a global representation of the scene state. Alternatively, gradient-based meta-learning has been proposed faster inference of implicit neural representations [36]. Deformable scenes can be modeled by transporting input coordinates to neural implicit representations with implicitly represented flow field or time-variam latent code [37, 38, 39, 40, 41, 42, 43, 44, 45]; however, they typically fit one trajectory and cannot handle different initial conditions and external action inputs, limiting their use in control.
Model-Based RL in Robotic Manipulation. We can categorize model-based RL methods by whether they use physics- or data-driven models, and whether they assume full state access or only visual observation. Methods that rely on physics-based models typically assume full-state information of the environment [46, 47] and require the knowledge of the object models, making them hard to generalize to novel objects or partially observable scenarios. For data-driven models, people have attempted to learn a dynamics model for closed-loop loop planar pushing [48] or dexterous manipulation [49]. Schenck and Fox [50, 51] tackle similar fluid pouring task via closed- simulation. Although they have achieved impressive results, they rely on state estimators customized for specific tasks, limiting their applicability more general and diversified manipulation tasks.
Various model-based RL methods have been proposed to learn state representations from visual observations, such as amage-space dynamics [4, 5, 1, 7], keypoint representation [52, 9, 12], and low-dimensional latent space [13, 15, 2, 3]. Some works learn a meaningful representation space using reconstruction loss [15, 2]. Others jointly train the forward and inverse dynamics models [14], or use time contrastive loss to regularize the latent embedding [16]. We differ from the previous methods by explicitly incorporating a 3D volumetric rendering process during training.
3 3D-Aware Representation Learning for Dynamics Modeling
Inspired by Neural Radiance Fields (NeRF) [19], we propose framework that learns viewpoint- invariant model for dynamic environments. As shown in Figure 2, our framework has three parts: (1) an encoder that maps the input images into a latent state representation, (2) a decoder that generates
3

---

(fi)
Encoder 8
Viewpoinl
Decoder
- (1uaa)
Viewpoint
(4)
Encoder
Decoder
(1+)
Dynamics
fas
Dynamics Prediction
Säya
Gradieni
Prediction
Action de
Action dty1
Loss for test time oplimization
(a) Forward Prodiction and Novel-View Symthesis
(b) Auto-Decoding for Extrapolate Viewpoint Generalization
Figure 3: Forward prediction and viewpoint extrapolation. (a) We first feed the input image(s) at time / to the encoder to derive the scene representation 8z. The dynamics model then takes 8; and the corresponding action sequence as input iteratively predict the future. The decoder synthesizes the visual observation conditioned on the predicted state representation and an input viewpoint. (b) We propose an auto-decoding inference- www--vvii-onneeeee iiatomiiiiztionn framework to enable extrapolated viewpoint generalization. Given an input image I taken from a viewpoint outside the training distribution, the encoder first predicts the scene representation 8t- Then the decoder reconstructs the observation I, from 8, and the camera viewpoint from I. We calculate the L2 distance between I, and f, and backpropagate the gradient through the decoder to update st. The updating process is repeated for K iterations, resulting in more accurate 8, the underlying 3D scene.
an observation image under a certain viewpoint based on the state representation, and (3) a dynamics model that predicts the future state representations based on the current state and the input action.
3.1 3D-Aware Scene Representation Learning
Neural Radiance Field. Given a 3D point æ E RR3 in scene and a viewing direction unit vector d R from a camera, NeRF learns to predict volumetric radiance field. This is represented using a differentiable rendering function fNORF that predicts the corresponding density a and RGB color e using (NeRr(2;, d) c). To render the color of an image pixel, NeRF integrates the information along the camera ray using Ö(r) = -he (h))a(
where rh) = o + hd is the camera ray with its origin E R33 and unit direction vector d E R², and T(h) = exp( f (s)ds) is the accumulated transparency between the pre-defined near depth hnear and far depth lafar along that camera ray. The mean squared error between the reconstructed color C' and the ground truth C' is:
Lree Σ1(e(r) - C(r)
(I)
Neural Radiance Field for Dynamic Scenes. One key limitation of NeRF is that it assumes the scene is static. For a dynamic scene, it must learn a separate radiance field fNERF for each time step. This severely limits the ability of NeRF to be used in planning and control, as is unable to handle dynamic scenes with different initial configurations or input action sequences. While other models have shown generalization across scenes [20, 53], it's unclear how they can be used visuomotor control. To enable [NERF to model dynamic scenes, we learn encoding function sene that maps the visual observations to a feature representation 8 for each time step and learn the volumetric radiance field decoding function based on Let {4, denotes the set of 2D images that capture 3D scene at time t from one or more camera viewpoints. The image taken from the 7th viewpoint is represented as Hf. We use ResNet-188 18 [54] to extract feature vector for each image. We take the output of ResNet-18 before the pooling layer and send to a fully-connecter layer, resulting in 256 dimension image feature vü. This image feature is concatenated with the corresponding camera viewpoint information (a 16-D vector obtained by flattening the camera view matrix) and processed using small multilayer perceptron (MLP) to generate the final image feature. The scene representation 8, at time is generated by first averaging the image features across multiple camera viewpoints, then being encoded using another small MLP and normalized to have a unit 1.2 norm.
Given a 3D point x, a viewing direction unit vector d, and a scene representation 8t, we learn a function face d, 8,) (a,, C.) to predict the radiance field represented by the density o, and RGB color c, Similar to NeRF, we use the integrated information along the camera ray to render the color of image pixels from input viewpoint and then compute the image reconstruction loss using Equation 1. During each training iteration, we render two images from different viewpoints to calculate more accurate gradient updates. faee depends on the scene representation 8t, forcing it to encode the 3D contents of the scene to support rendering from different camera poses.
Time Contrastive Learning. To enable the image encoder to be viewpoint invariant, we regularize the feature representation of each image of using multi view time contrastive loss (TCN) [16] (see Figure 2a). The TCN loss encourages features of images from different viewpoints at the same time

---

step to be similar, while repulsing features of images from different time steps to be dissimilar, More specifically, given a time step we randomly select one image I as the anchor and extract its image feature oy using the image encoder. Then we randomly select one positive image from the same time step but different camera viewpoint 1i and one negative image from a different time step but the same viewpoint Tg. We use the same image encoder extract their image features vy and vz. Similar to [16], we minimize the following time contrastive loss:
Li max (Iloy vž l12 |oo 1 1 + a, O)
(2)
where a is hyper-paramett denoting the margin between the positive and negative pairs.
3.2 Learning the Predictive Model
After we have obtained the latent state representation forward dynamics model, $1+1 faya(s,, az). Given 8; and sequence of actions {a,, ac+1..... }. we predict H steps in the future by iteratively feeding in actions into the one-step forward model. We implement fayn as an MLP network which is trained by optimizing the following loss function:
use supervised learning estimate the
Layn Σ
BhLth 1. where beth = faya a(4h-1), &: =
(3)
We define the final loss combination of the image reconstruction loss, the time contrastive loss, and the dynamics prediction loss: C = Leee + Le + Ldyn- We first train the encoder fene and decoder face together using stochastic gradient descent (SGD) by minimizing Lee and Lee which makes sure that the learned scene representation 8 encodes the 3D contents and is viewpoint-invariant We then fix the encoder parameters, and train the dynamics model fayn by minimizing Layn using SGD, Please see the supplementary materials for the network architecture and training details.
4
Visuomotor Control
4.1 Online Planning for Closed-Loop Control
When given the goal image | and its associated camera pose, we can feed them through the encoder fene to get the state representation for the goal configuration g8oal We use the same method to compute the state representation for the current scene configuration $1. The goal of the online planning problem
find an action sequence aj..... a 1 that minimizes the distance between the predicted future representation and the goal representation at time T. As shown in Figure 3a, given sequence of actions, our model can iteratively predict a sequence of latent state representations. The latent-space dynamics model can then be used for downstream closed- loop control tasks via online planning with modec-predictive control (MPC). We formally define the online planning problem as follows:
min a1.....7--
1T122
S.t. 81 = 81, Sc+1 = fdyn(Se, az).
(4)
Many existing off-the-shelf model-based RL methods can be used to solve the MPC problem [5, 49, 15, 4, 9, 55, 56]. We experimented with random shooting, gradient-based trajectory optimization, cross-entropy method, and model-predicctvv path integral (MPPI) planners [57] and found that MPPI performed the best. In our experiments, we specify the action space as the position and orientation of the arm's end-effector. Then, the joint angle of the arm calculated via inverse kinematics. Please check our supplementary materials for more details on how we solve the planning task.
4.2
Auto-Decoder for Viewpoint Extrapolation
End-to-end visuomotor agents can undergo significant performance drop when the test-time visual observations are captured from camera poses outside the training distribution. The convolutional image encoder suffers from the same problem as it is not equivariant to changes in the camera pose, meaning it has hard time generalizing to out-of-distribttion camera views. As shown in Figure 3b, when encountered an image from a viewpoint outside training distribution, with a pixel distribution vastly different from what the model is trained on, passing it through the encoder fene will give an amortized estimation of the scene representation s, It has a high chance that the decoded image is different from the ground truth as the viewpoint has never been encountered during training.
We fix this problem at test time by applying the inference-by-optiizzaaie (also named as an auto- decoding) framework that backpropagates through the volumetric renderer and the neural implicit
5

---

Result goal image f
Triil
Trial
Thial iaa
Target vaining (inerpolation)
begst (extrpplaaion)
seen during
tuining
Control process and images observed agent
(b) Control results
(c) Goal configuration
Figure 4: Qualitative control results of our method three types of testing scenarios. The image the right shows the target configuration we aim achieve. The left three columns show the control processes, which are also the imput images to the agent. goal image. Trial #1 specifies the goal using a different viewpoint from the agent's but has been encountered during training. Trial #2 uses a goal view that an interpolation of training viewpoints. Trial #3 uses an extrapolated viewpoint that is outside the training distribution. Our method performs well in all settings.
fourth column
control results from the same viewpoint as
Goal image
Control result from the goal view
Goal image
Control result from the goal view
(outside train distr.)
Ours
Ours wio AD
TC+AE
(outside train distr.) Ours
Ours wio AD
TC+AE
Agent view
Agent
Mismatch
Mismatch
FluidPour
FluidShake
Figure 5: Qualitative comparisons between our method and baseline approaches on the control tasks. We show the closed- loop control results the FluidPour and FluidShake environments. The goal image viewpoint (top-left image of each block) is outside training distribution and is different from viewpoint observed by the agent (bottom-left image each block). Our final control results are much better than a variant that does not perform auto-dccoding test-time optimization (Ours w/o AD) and the best-performing baseline (TC+AE), both of which fail to accomplish the task and their control results (blue points) exhibit apparent deviation from the target configuration (red points) when measured in the 3D points space of the fluids and floating cube.
representation into the state estimate [23, 20). This is inspired by the fact that the rendering function, (ar d, s.) (at, ct) is viewpoint equivariant, where the output only depends on the state representation 8t, the location æ, and the ray direction d, meaning that the output is invariant to the camera position along the camera ray, i.e., even move the camera closer farther away along the camera ray, face still tends to generate the same results. We leverage this property and calculate the L2 distance between the input image and the reconstructed image Lus = I, - f. and then update the scene representation s, using stochastic gradient descent. We repeat this updating process K times to derive the state representation of the underlying 3D scene. Note that this update only changes the scene representation while keeping the parameters in the decoder fixed. The resulting representation used as goal Equation to solve the online planning problem.
Experiments
Environments. We consider the following four environments involving both fluid and rigid objects to evaluate the proposed model and baseline approaches. The environments are simulated using NVIDIA FleX [58]. (1) FluidPour (Figure 7a): This environment contains fully-actuated cup that
pours fluids into container at the bottom. (2) FluidShake (Figure 7b): A fully-actuated container

---

Target scon during tnining
Torget view imsile tmining (interpobtion)
Target ostside trining distr. (cstmpebriso))
6.12
0.2
0.05
0.2
0.5
0.04
0.0
0.0
6.00
0.0
0.0
0.0
Postion
Angle Euer
Eroor
(6 Quantisative contrul resulls on FluidPour
(b) Quantitative control rosulss on FluidStake
Figure 6: Quantitative comptiiitiv between our method and baselines on the FluidPour and FluidShake. In each environment, we compare the results using three different evaluation metrics under three settings, i.c., (1) the target image view seen during training. (2) the target image view is inside the training distribution hut not seen during training (interpolation), and (3) the target image view is outside the training distribution (extrapolation). The height of the bars indicates the mean, and the error bar denotes the standard error. Our model significantly outperforms all baselines under all testing settings.
moves on a 2D plane, Inside the container are fluids and a rigid cube floating on the surface. (3) RigidStack (Figure 7c): Three rigid cubes form a vertical stack and are released from certain height but in different horizontal positions. They fall down and collide with each other and the ground. (4) RroidDrop (Figure 7d): A cube falls down from a certain height. There is a container fixed at a random position on the ground. The cube either falls into the container or bounces out. We use 20 camera views for all environments and generated 1, 000 trajectories of 300 time steps for both FluidPour and FluidShake as an offline dataset to train the model, 800 trajectories of S0 time steps for RigidStack, and 1, 000 trajectories of 50 time steps for RigidDrop.
Evaluation Metrics. We use the first two environments, i.e., FluidPour and FluidShake, to measure the control performance, where we specify the target configuration of the control task using images from (1) one of the viewpoints encountered during training, (2) an interpolated viewpoint between training viewpoints, and (3) an extrapolated viewpoint outside the training distribution (Figure 4).
We provide quantitative evaluations on the control performance in FluidPour and FluidShake by extracting the particle set from the simulator, and measuring the Chamfer distance between the result and the goal, which we denote "Chamfer Dist.". In FluidPour, we provide additional measurements on the L2 distance of the position/orientation of the cup towards the goal, denoting as "Position Error" and "Angle Error" respectively. In FluidShake, we calculate the L2 distance of the container and cube's 's position towards the goal and denote them as "Container Error" and "Cube Error" respectively.
5.1 Baseline Methods
For comparison, we consider the following three baselines: TC: Similar to TCN [16], only uses time contrastive loss for learning the image feature without having to reconstruct the scene. We learn a dynamics model directly on the image features for control. TC+AE: Instead of using Neural Radiance Fields to reconstruct the image, this method uses the standard convolutional decoder to reconstruct the target image when given a new viewpoint. This would then be similar to GQN [26] augmented with time contrastive loss. NeRF: This method is direct adaptation from the original NeRF paper [19] and is the same as ours except that it does not include the time contrastive loss during training and the auto-decoding test-time optimization. We use the same dynamics model shown in Figure 2b and train the model for each baseline respectively for dynamic prediction. We use the same feedback control method, i.e., MPPI [57], for our model and all the baselines.
5.2 Control Results
Goal Specification from Novel Viewpoints. Figure 4c shows the goal configuration, and we ask the learned model to perform three control trials where the goal is specified from different types of viewpoints. The left three columns show the MPC control process from the agent's 's viewpoint. The fourth column visualizes the final configuration the agent achieves from the same viewpoint as the goal image. Trial #1 specifies the goal using a different viewpoint from the agent's but has been encountered during training. Trial #2 uses a goal view that an interpolation of training viewpoints. Our agent can achieve the target configuration with decent accuracy. For trial #3, we specify the goal view by moving the camera closer, higher, and more downwards with respect to the container. Note this goal image view is outside the distribution of training viewpoints. With the help of test-time
7

---

Novel view symthesis
Novel view symthosi
GT
(+) PhiidPour
(b) FluidShake
Novel view synthesis
Novel view syathesis
GT
Ours
(6) RigiSSock
(d) RigidDrop
Figure 7: Forward prediction and novel view synthesis four environments. Given representation an input action sequence, our dynamics model predicts the subsequent latent scene representation, which used as the input of our decoder model to reconstruct the corresponding visual observation based on different viewpoints. In each block, we render images based on the open-loop future dynamic prediction from left to right. Images in the dotted hox compare our model's novel view synthesis results in the last time step with the ground truth from three different viewpoints.
auto-decoding optimization introduced Section 4.2, our method can successfully achieve the target configuration, shown in the figure.
Baseline Comparisons. We benchmark our model with the baselines by assessing their performance on the downstream control tasks. Figure 5 shows the qualitative comparison between our model (Ours), a variant of our model that does not perform the auto-decoding test-time optimization (Ours w/o AD), and the best-performing baseline (TC+AE) introduced in Section 5.1. We find that when the target view outside the training distribution and vastly different from the agent view, our full method shows much better performance in achieving the target configuration. The variant without auto-decoding optimization and TC+AE fail to accomplish the task and exhibit an apparent deviation from the ground truth in the 3D points space of the fluids and floating cube. We also provide quantitative comparisons on the control results. Figure 6 shows the performance in the FluidPour and FluidShake environments. We find our full model significantly outperforms the baseline approaches in both environments under all scenarios and evaluation metrics. The results effectively demonstrate the advantages of the learned 3D-aware of the contents in the 3D environments and are capable of extrapolation viewpoint generalization.
representations, which contain more precise encoding
5.3 Dynamic Prediction and Novel View Synthesis
Conditioned on a scene representation and an input action sequence, our dynamics model fayn can iteratively predict the evolution of the scene representations. Our decoder can then take the predicted state representation and reconstruct the corresponding visual observation from a query viewpoint. Figure 7 shows that our model can accurately predict the future and perform novel view synthesis on four environments involving both fluid and rigid objects, suggesting its usefulness in trajectory optimization. Please check our video results in the supplementary material for better visualization.
Conclusion
In this paper, we proposed learn viewpoint- innraiiinnninaiaanttt 3D-aware scene representations from visual observations using an autoencoding framework augmented with a neural radiance field rendering module and time contrastive learning. We show that the learned 3D representations perform well on the prodel-based visuomotor control tasks. When coupled with an auto-decoding decoding test-time optimization mechanism, method allows goal specification from viewpoint outside the training distribution. We demonstrate the applicability of the proposed framework in a range of complicated physics environments involving rigid objects and fluids, which we hope can facilitate future works visuomotor control for complex and dynamic 3D manipulation tasks.

---

References
[1] Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, and S. Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.00568, 2018.
[2] D. Hafner, T. Lillicrap, Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603. 2019.
[3] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, T. Graepel, al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):604-099, 2020.
[4] C. Finn, I. Goodfellow, and S. Levine. Unsupervised learning for physical interaction through video prediction. arXiv preprint arXiv: 2016.
[5] F. Ebert, C. Finn, A. X. Lee, and S. Levine. Self-supervised visual planning with temporal skip connections. In CoRL, pages 344-356, 2017.
[6] L. Yen-Chen, M. Bauza, and Isola. anperience-embeddee visual foresight. In Conference on Robot Learning, pages 1015 1024. PMLR, 2020.
[7] H. Suh and R. Tedrake. The surprising effectiveness of linear models for visual foresight in object pile manipulation. arXiv preprint (XXi:22022.00093, 2020.
[8] L. Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for category- level robotic manipulation. arXiv preprint arXiv: 06684, 2019.
[9] L. Manuelli, Y. Li, Florence, and R. Tedrake. Keypoints into the future: Self-supervised correspondence in model-base reinforcement learning. In Conference on Robot Learning (CoRL), 2020.
[10] C. Wang, R. Martin,Martin, D. Xu, Lv, C. Lu, L. Fei-Fei, S. Savarese, and Zhu. 6-pack: Category-level 6d pose tracker with anchor-based keypoints. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 10059-10066. IEEE, 2020.
[11] W. Gao and R. Tedrake. kpam 2.0: Feedback control for category-level robotic manipulation. arXiv preprint arXi::2102.06279, 2021.
[12] Y. Li, A. Torralba, A. Anandkumar, D. Fox, and A. Garg. Causal discovery in physical systems from videos. Advances in Neural Information Processing Systems, 33, 2020.
[13] M. Watter, J. T. Springenberg, J. Boedecker, and M. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. arXiv preprint 1506005506..355, 2015.
[14] P. Agrawal, A. Nair, P. Abbeel, J. Malik, and S. Levine. Learning to poke by poking: Experiential learning of intuitive physics. arXiv preprint arXiv: 1606. 00449, 2016.
[15] D. Hafner, T. Lillicrap, Fischer, R. Villegas, Ha, Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pages 2555-2565. PMLR, 2019.
[16] Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain. cime-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1134-1141 141. IEEE, 2018
[17] V. Sitzmann, Thies, F. Heide, M. Nießner, G. Wetzstein, and M. Zollhöfer. Deepvoxels: Learning persistent 3d feature embeddings. In Proc. CVPR, 2019.
[18] S. Lombardi, T. Simon, J. Saragih, G. Schwartz, A. Lehrmann, and Y. Sheikh. Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint arXiv: /10.//9060777.. 2019.
[19] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, pages 405-421. Springer, 2020.

---

[20] V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene representation networks: Continuous 3d-structure-awar neural scene representations. arXiv preprint arXiv: 0/618, 2019.
[21] H.-Y. F. Tung, R. Cheng, and K. Fragkiadaki. Learning spatial common sense with geometry- aware recurrent networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2019.
[22] H.-Y. F. Tung, Z. Xian, M. Prabhudesai, S. Lal, and K. Fragkiadaki. 3d-oes: Viewpoint-invariant object-factorized environment simulators. arXiv preprint (00i::2011006464: 2020.
[23] J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 163-174, 2019.
[24] D. E. Worrall, J. Garbin, D. Turmukhambetov, and G. J. Brostow. Interpretable transforma- tions with encoder-decoder networks. In Proc. ICCV, volume 4, 2017.
[25] M. Tatarchenko, A. Dosovitskiy, and T. Brox. Single-view to multi-view: Reconstructing unseen views with convolutional network. CoRR abs/15/ 0aas/5//1.67000 1.06702, 1(2)22, 2015.
[26] S. A. Eslami, D. J. Rezende, F. Besse, F. Viola, A. S. Morcos, M. Garnelo, A. Ruderman, A. A. Rusu, Danihelka, K. Gregor, et al. Neural scene representation and rendering. Science, 360 (6394):1204-12100 2018
[27] D. J. Rezende, S. Eslami, Mohamed, P. Battaglia, M. Jaderberg, and N. Heess. Unsupervised learning of 3d structure from images. arXiv preprint arXiv:1607.00662, 2016.
[28] T. Nguyen-Phuoc, C. Li, S. Balaban, and Y.-L. Yang. Rendernet: A deep convolutional network for differentiable rendering from 3d shapes. arXiv preprint arXiv: 06570, 2018.
[29] H.-Y. F. Tung, R. Cheng, and K. Fragkiadaki. Learning spatial common sense with geometry- aware recurrent networks. Proc. CVPR, 2019.
[30] J.-Y. Zhu, Z. Zhang, C. Zhang, J. Wu, A. Torralba, J. Tenenbaum, and Freeman. Visual object networks: image generation with disentangled 3d representations. In Proc. NIPS, pages 118-129, 2018.
[31] Y. Li, T. Lin, K. Yi, D. Bear, D. Yamins, J. Wu, J. Tenenbaum, and A. Torralba. Visual grounding of learned physical models. In International conference on machine learning, pages 5927-5936. PMLR, 2020.
[32] G. Saponaro, L. Jamone, A. Bernardino, and G. Salvi. Beyond the self: Using grounded affordances to interpret and describe others' actions. IEEE Transactions on Cognitive and Developmental Systems, 12(2):209-224) 2019.
[33] S. Saito, Z. Huang, R. Natsume, S. Morishima, A. Kanazawa, and H. Li. Pifu: Pixel-aligned im- plicit function for bigh-resolution clothed human digitization. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 2304-2314 2019.
[34] A. Trevithick and B. Yang. Grf: Learning a general radiance field for 3d scene representation and rendering. arXiv preprini arXiv:2010.04595 2020.
[35] A. Yu, V. Ye, M. Tancik, and A. Kanazawa. pixelnerf: Neural radiance fields from one or few images. arXiv preprint arXiv:2012.021900 2020.
[36] V. Sitzmann, E. R. Chan, R. Tucker, N. Snavely, and G. Wetzstein. Metasdf: Meta-learning signed distance functions. arXiv preprint arXiv:2006.0966 2020.
[37] M. Niemeyer, L. Mescheder, M. Occhsle, and A. Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 5379-5389, 2019.
[38] K. Park, U. Sinha, T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R.-M. Brualla. Deformable neural radiance fields. arXiv preprint arXiv:2011.12948, 2020.
10

---

[39] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. arXiv preprint arXiv:2011.13961 2020.
[40] E. Tretschk, A. Tewari, V. Golyanik, M. Zollhöfer, C. Lassner, and C. Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of deforming scene from monocular video. arXiv preprini urXiv:2012. 122477 2020.
[41] Z. Li, S. Niklaus, N. Snavely, and O. Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes, arXiv preprint arXiv:2011. 2020.
[42] Y. Du, Y. Zhang, H.-X. Yu, J. B. Tenenbaum, and J. Wu. Neural radiance flow for 4d view synthesis and video processing, arXiv e-prints, pages arXiv-2012, 2020.
[43] W. Xian, J.-B. Huang. J. Kopf, and C. Kim. Space-time neural irradiance fields for free- viewpoint video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9421-9431, 2021. 2021.
[44] J. Ost, F. Mannan, N. Thuerey, J. Knodt, and F. Heide. Neural scene graphs for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pullern Recognition, pages 2856-2865, 2021.
[45] T. Li, M. Slavcheva, M. Zollhoefer, S. Green, C. Lassner, C. Kim, T. Schmidt, S. Lovegrove, M. Goesele, and Z. Lv. Neural 3d video synthesis. arXiv preprint wxXiv:2103.02597 2021.
[46] R. Hogan and A. Rodriguez. Feedback control of the pusher-slider system: story of hybrid and underactuated contact dynamics. arXiv preprint arXiv:161 Sedi:1661.002666 2016.
[47] J. Zhou, Y. Hou, and M. T. Mason. Pushing revisited: Differential flatness, trajectory planning. and stabilization. The International Journal of Robotics Research, 38(12-13) 2019.
[48] M. Bauza, R. Hogan, and Rodriguez. data-efficient approach to precise and controlled pushing. In Conference on Robot Learning, pages 336-345. PMLR, 2018.
[49] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pages 1101-1112. PMLR, 2020.
[50] C. Schenck and D. Fox. Reasoning about liquids via closed-loop simulation. arXiv preprint arXiv:103.001656. 2017.
[51] C. Schenck and D. Fox. Perceiving and reasoning about liquids using fully convolutional networks. The International Journal of Robotics Research, 37(4-5):452-771, 2018
[52] T. D. Kulkarni, A. Gupta, C. Ionescu, S. Borgeaud, M. Reynolds, A. Zisserman, and V. Mnih. Unsupervised learning of object keypoints for perception and control. Advances in neural information processing systems, 32:10724-1073 2019.
[53] M. Niemeyer, L.. Mescheder, M. Oechsle, and A. Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3504-3515, 2020.
[54] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). June 2016.
[55] Y. Li, J. Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In ICLR, 2019.
[56] Y. Li, Wu, J.-Y. Zhu, J. Tenenbaum, A. Torralba, and R. Tedrake. Propagation networks for model-based control under partial observation. In ICRA, 2019.
[57] G. Williams, A. Aldrich, and E. Theodorou. Model predictive path integral control using covariance variable importance sampling. arXiv preprint 000.:: 149, 2015.
[58] M. Macklin, M. Müller, N. Chentane7, and T.-Y. Kim. Unified particle physics for real-time applications. ACM Transactions Graphics (TOG), 33(4):1-12, 2014.

---

[59] E. Olson. Apriltag: A robust and flexible visual fiducial system. In 2011 IEEE International Conference on Robotics and Automation, pages 3400-3407. IEEE, 2011.
[60] G. F. Franklin, J. D. Powell, A. Emami-Naeini, and J. D. Powell. Feedback control of dynamic systems, volume 4. Prentice hall Upper Saddle River, NJ, 2002.
12


Reviews:
[{"review_id": "vGLj1zRMlKL", "reviewer": "Reviewer_i7GY", "summary": "This paper presents a method for viewpoint-equivariant representation learning for visuomotor control. The approach involves learning implicit 3D representations of tabletop scenes with NeRFs and using time-contrastive loss to enforce time consistency. This implicit dynamic scene representation is used as a forward model for model predictive control to achieve specific state-configurations. The viewpoint-equivariant representations allow for specifying target configuration with goal images from arbitrary viewpoints. Experimental results show that the approach is able to achieve desired state-configurations with fluids and rigid-body objects in simulated setups.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ This paper proposes an exciting direction for learning scene-dynamics with 3D awareness – a powerful prior for embodied robotics. Most end-to-end learning methods ignore the underlying 3D environment and try to directly learn from pixels. Enforcing an implicit 3D structure could help robots learn reusable skills that aren’t tethered to a specific viewpoint.\n+ The subfield of neural scene representation learning has been gaining a lot of traction in the vision and graphics community. This work could potentially lead to interesting directions for learning scene and object representations for robotics applications.\n+ Generally, the paper is well written, the methods are easy to understand, and the figures are informative. The supplementary video is also well made and effectively showcases the capabilities of the approach.\n\n\nWeaknesses: \n- One concern is that the evaluation setup is a bit contrived and doesn’t really showcase the full-potential of the 3D dynamic scene representation in a realistic robotics use-case. In the fluid-pouring task, the goal is to achieve the exact fluid state specified in the goal image and the quantitative evaluations measure the Chamfer distance between the fluid particles and the pose error of the cup. In a realistic use-case it’s hard to provide an exact goal-image of the fluid state at a specific time step, and even so it’s unclear if this sort of specificity is necessary in the first place. Alternatively, the evaluation could measure the robot’s ability to complete end-tasks like pouring the liquid upto a certain level – say “20 ml”, or for the fluid-shaking task – make 4 loops around the container with the cube touching each of the four sides. These end-task evaluations could provide better insights into how important 3D-dynamics are for precise model predictive control. \n- Another concern is that using NeRF restricts scene representations to mostly tabletop scenes. It’s unclear how neural implicit representations can be scaled to embodied settings like household kitchens or rooms with partial observability. Further, if we are restricted to a tabletop setting, I am not sure why the camera pose would dramatically change in a real-world setup between the input and goal viewpoints other than minor perturbations to the physical setup. One could imagine a user taking a photo of the intended goal from a third-person perspective, but again capturing the state of a fluid at a certain timestep is a very unnatural way of specifying goals. In summary, the paper currently lacks a strong evaluation use-case where “3D awareness” is absolutely necessary for visuomotor control. Showcasing results on tasks that specifically require 3D-reasoning would substantially strengthen the contributions of the paper. \n- How can NeRFs handle stochasticity? Motions of granular media and fluids are inherently stochastic in nature. Directly training a predictive model leads to blurry predictions, as evident in the supplementary figures and videos. This limits the approach to short-horizon tasks where the forward model can only be rolled-out for a few steps into the future.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposes an interesting direction for learning implicit dynamic models with 3D awareness. However, the experimental evaluation is missing a strong robotics use-case that highlights the importance of learning 3D-aware implicit dynamics models for model-predictive control.\n\n**Post Rebuttal**  \nThank you for the detailed response and new results. I'm still a bit skeptical about learning visual-prediction models for stochastic tasks that involve fluids (even with VAEs). The blurriness of the real-world results partly reaffirms this. But nonetheless, I think the paper poses an interesting direction for using NeRFs in robotics, which could be valuable to future follow-up works. I am keeping my original score of Weak Accept. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "Major:\nSee comments above on better evaluation tasks.\n\nMinor:\n- L232: “similar to [25]” -> “similar to XX approach [25]”\n- Related work: there has been some recent progress in learning dynamic/video NeRFs (https://arxiv.org/abs/2103.02597). Although these-=- methods do not condition NeRFs on actions to learn a dynamics model, it might be still good to cite them.\n- How many training images are needed per scene? How long does it take to train each scene? \n- Can you provide any quantitative evaluations for Section 5.3? Perhaps L2 distances on nearest neighbor search and novel-view synthesis?\n"}}, {"review_id": "RqZqOTO8Rhf", "reviewer": "Reviewer_BFGN", "summary": "This paper claims it is possible to learn a representation of a scene, that can be used both to predict new views and future views. It demonstrates this claim on several simple scenes, including some involving particle systems.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThis is an exciting claim.\n\nThe paper shows it can be done on some simple examples, which will inspire future work.\n\nWeaknesses.\n\nThese are not real images/videos or a real robot. It is claimed Covid made this difficult, although very simple and cheap ($2K) robots could be installed at home to do the evaluation with.\n\n\"Great claims require great evidence\". It is worth looking skeptically at the evaluation tasks.\n\n1) All of them involved simple clean images without clutter (much less other things moving in the scenes). It appears that the system knew \nto\nonly look at the blue particle systems (based on Figure 5 lower row) or the cubes, and not the containers or the robot or table. \nReal pouring of transparent liquids are hard to capture.\nSo I think we have to be skeptical about the claims about learning vision.\n\n2) It appears that the state vector for dynamics for these systems is pretty simple (positions and velocities of the robot tip or block release\npoints. The space that needs to be searched is relatively simple. To what extent is the system just matching the\nmotion of the robot?\n\n3) It appears the time intervals are short.\n\n4) It does not appear the physics for the \"liquid\" are realistic.\nThere is something physically wrong with the pours in Figures 4\nand 5: The arc of the pour is too smooth and goes out too far. There is\nlittle to provide the large horizontal velocity needed\nto the liquid (or particles). The pour should go almost straight down,\ngiven the container is not moving horizontally at a high rate during\nthe time interval.\nI  suspect the amount\nof pour and sloshing is not done well either.\n\n5) Figure 5 shows that the major thing going on is overlapping in 3D the\nblue stuff by translating the containers. It is probably the same for the cubes. If the width of the\npour is small, the system will probably be happy with no pour. There is\nno notion of liquid levels changing over time forcing pouring to occur.\n\nIn FluidShake is the position of the yellow cube being controlled?\n\nUsually systems like these are flooded with irrelevant\nfeatures, and the type of learning used can't figure out what is relevant or irrelevant with a reasonable amount of training data.\n\nI am skeptical that this system can handle color/texture map variations on the objects, or changing the shape or size of the containers\nor blocks, or even movement of the table.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper makes a claim and some progress on behavior capture for liquid/granular materials, and, although imperfect, should be accepted.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See the weaknesses above to make the paper more convincing.\n", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "O-qvDdyYGLc", "reviewer": "Reviewer_1uVz", "summary": "The authors propose learning a (multiview) latent space dynamics model where the latent state is (self)supervised via both a time contrastive loss and a NERF-based autoencoder. By parameterizing the radiance field with both the latent state and viewpoint, they allow a single NERF to model the dynamic scene as it changes across time. At inference time, the state vector (for the goal state only) is estimated by backpropagating through the NERF decoder (keeping weights frozen) and updating only the state parameters (i.e. an auto-decoding framework similar to DeepSDF). The authors show that view equivariance endemic to NERFs allows for minimal performance decrease using an MPPI planner with novel (out of distribution) goal viewpoints.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The overall idea of the paper is laudable. It builds on recent advances in 3D scene representations and (factorized) visuomotor control and proposes an elegant and intuitive solution for introducing additional inductive bias in the form of a 3D spatial representation. The experiments suggest that this sort of approach is very well suited to reducing the amount of data required for visuomotor generalization by reducing the need to train across many goal viewpoints.  \n\nThe biggest weakness of this paper is it's failure to address the practicality of the method in terms of data efficiency, computation, and robustness to real-world scene variation.\n\nOn the data efficiency and computation fronts, the paper would be vastly strengthened by more details on the number of training examples required and the compute time. NERFs are notorious for being expensive to fit; how many views (and from what distribution) are used to fit a dynamic scene during training and how long does phase 1 (the portion learning the state representation and NERF) take to train?\n\nExperimentally, this paper's impact is somewhat lessened by its lack of real-world data. Because the simulation environments are consistent, things like lighting and shadows will be exactly the same between training and inference. The current process fits a NERF (and state representation) to a very specific scene setup and it's not clear how this approach will work in the physical world when things like lighting shift and other sources of variation may occur between training and inference. The authors also do not use the auto-decoder procedure on the robot's ego views, only on the goal viewpoint. It would be good to discuss this choice, naively it seems like having ego-view invariance would also be pretty valuable (imagine just setting a camera down anywhere in the robot's workspace instead of keeping it fixed to a single location).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While this paper does have shortcomings (I would have liked to see additional experiments in less carefully controlled environments and a discussion of some of the practical aspects of the method) it presents an original and useful contribution to the field. The results presented here are promising and should encourage others in the community to build on this sort of spatially supervised approach.\n\nUpdate (post Author Response): In light of the thorough responses from the authors and additional information and experiments provided, I'm updating my \"Clarity Of Presentation\" score from \"Good\" to \"Very Good\". I think that this paper is fundamentally novel, interesting, and valuable to the field and should be accepted.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "Please see above comment about including more experimental details such as the amount of data used to train for each task and the training time required. The paper would be stronger with a more fully fleshed out appendix addressing these details.\n\nA few other questions:\n- I'd also like the authors to clarify exactly what rendered viewpoints were used to train the NERF representation. Is the NERF portion trained with viewpoints similar to the \"out of distribution\" evaluation views?\n\n- Additionally, why not use the same approach (auto-decoder though the NERF instead of using an encoder) for estimating the current state during control, is it just computationally too expensive to do this at every timestep? This would allow the policy to generalize to different ego perspectives.\n\n- What is the nominal trajectory used with MPPI, is it random is does it come from somewhere (e.g. a demonstration)?\n\nAs a side (minor) comment, this related work may also be relevant: the authors embedded an explicit 3D representation into a language-grounding classification task to enable inference from never-before-seen viewpoints). Grounding language attributes to objects using bayesian eigenobjects (Cohen et al. 2019)"}}, {"review_id": "E-vlPtPwHQV", "reviewer": "Reviewer_5NEJ", "summary": "This paper demonstrates an approach for modelling dynamics of a scene using an implicit neural representation similar to NeRF. The model is based on an autoencoder framework, where an encoder maps first-person images to a latent state representation that encodes the system state, and a decoder takes the state representation and a viewpoint as input and renders an image. A dynamics model in the latent state space enables predicting future states, and applying the decoder to render future observations from arbitrary viewpoints.\n\nAt test-time, the system takes a viewpoint and a goal image as input, and uses an optimization process to construct the corresponding scene representation for the goal. This optimization process works better on out-of-training-distribution viewpoints than directly encoding the goal image with the encoder.\n\nThe latent state dynamics are learned with a time-contrastive loss that pulls state representations from different viewpoints at the same time-step closer, while pushing representations from the same viewpoint but different time-steps apart. As a result, the latent state vector focuses on the state of the dynamic system in a viewpoint-invariant way, while the decoder is responsible for modelling viewpoint-variant rendering. This makes the state representations amenable for planning and control.\n\nThe paper applies MPC to reach a desired goal state represented by an image from an arbitrary viewpoint. The control experiments include two scenes: fluid pouring and moving a water-filled box to create waves. Additionally, the paper shows that the system is capable of novel view synthesis of a collapsing tower of blocks, and a falling cube.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Technically strong and well thought out approach.\n- Demonstrates an exciting new result modelling scene dynamics over a latent space on which an implicit volumetric scene representation is conditioned.\n- Able to model dynamics of rigid bodies, fluids, and soft bodies.\n- Convincing experimental results.\n\nWeaknesses:\n- Does not yet enable generalization of dynamics understanding to new scenes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Implicit representations of geometry and rendering have recently seen explosive interest in computer vision and graphics research, due to enabling previously unimaginable results, such as neural rendering from arbitrary viewpoints, scene composition, and 3D reconstructions from still images. Such reasoning about geometry and dynamics could greatly benefit robotics applications as well. Traditionally, model-based reasoning (e.g. MPC, task and motion planning) in 3D has required either hand-specified state representations and dynamics models, running simulators in the loop, or learned models that reason in 2D from fixed viewpoints. If we could instead learn such dynamics models and scene representations in 3D just from observations, that could pave the way for new planning and learning algorithms and breakthroughs. This paper takes a step in that direction and should appear at the conference. The main weakness is the significant amount of learning required for each scene individually and the lack of generalization to new scenes, however this is not unlike most RL research, and also inline with the state of the art in implicit scene representations in vision.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "The main paper would benefit from at least a few sentences commenting on how many viewpoint images, from how many timesteps were needed during training, and how long did the training take per scene. The NeRF approach required a lot of viewpoints and a very expensive optimization. I wonder how that is affected by adding the temporal dimension and the time-contrastive loss. "}}]

Author rebuttal:

Meta-review: **Update after rebuttal**
The authors have addressed most issues raised by the reviewers, and overall I recommend accept.
Two more comments:
* I agree with the reviewer that raised the concern of integrating the results presented in the rebuttal into the paper. Please try to do so as best as you can. 
* I still would like to see a more "balanced" discussion on what the limitations are

**Initial meta-review**:
**Summary**:
This work presents an approach towards learning viewpoint-equivariant visual latent representations that can be used to used for visual planning/control (via a learned dynamics model in the latent space). The proposed framework/algorithm allows to specify targets/goals from viewpoints different from the robots viewpoint, and can support goal specification from camera viewpoints that are outside the training distribution.

**Strengths**:
The manuscript addresses a very important and timely problem within the robotics community. Progress on learning visual latent representations and dynamics models that generalize across viewpoints can have major impact.

The manuscript is well written, the work will motivated, and a technically strong contribution is made.

**Weaknesses**:
The reviewers raise some questions that should be addressed by the authors

* Overall, several reviewers raise the issue of evaluation on simulated data only, where the data is very “clean/simple”. It is unclear how well this approach would perform on real world data? The authors should consider performing a subset of the evaluation on real world data (maybe not the downstream task evaluation, but only the learning representations part). Alternatively, there may be be way to introduce some noise?
* it is unclear how much training data from how many viewpoints is required to train the latent representations and predictive model, and how much time it took to train those representations.
* The manuscript is missing a clear discussion of the limitations in terms of generalization and what has been learned. While it is useful to use the big vision/claims for motivation, it is equally important to clearly state what has been achieved with this work, and where the limitations lie. For instance it seems unlikely that the system for the pouring task learns anything about the water levels (eg half full) for various sized containers. A thorough discussion would better enable future research on this topic.
Decision: Accept (Oral)