Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper claims it is possible to learn a representation of a scene, that can be used both to predict new views and future views. It demonstrates this claim on several simple scenes, including some involving particle systems.

strengths_and_weaknesses: Strengths

This is an exciting claim.

The paper shows it can be done on some simple examples, which will inspire future work.

Weaknesses.

These are not real images/videos or a real robot. It is claimed Covid made this difficult, although very simple and cheap ($2K) robots could be installed at home to do the evaluation with.

"Great claims require great evidence". It is worth looking skeptically at the evaluation tasks.

1) All of them involved simple clean images without clutter (much less other things moving in the scenes). It appears that the system knew 
to
only look at the blue particle systems (based on Figure 5 lower row) or the cubes, and not the containers or the robot or table. 
Real pouring of transparent liquids are hard to capture.
So I think we have to be skeptical about the claims about learning vision.

2) It appears that the state vector for dynamics for these systems is pretty simple (positions and velocities of the robot tip or block release
points. The space that needs to be searched is relatively simple. To what extent is the system just matching the
motion of the robot?

3) It appears the time intervals are short.

4) It does not appear the physics for the "liquid" are realistic.
There is something physically wrong with the pours in Figures 4
and 5: The arc of the pour is too smooth and goes out too far. There is
little to provide the large horizontal velocity needed
to the liquid (or particles). The pour should go almost straight down,
given the container is not moving horizontally at a high rate during
the time interval.
I  suspect the amount
of pour and sloshing is not done well either.

5) Figure 5 shows that the major thing going on is overlapping in 3D the
blue stuff by translating the containers. It is probably the same for the cubes. If the width of the
pour is small, the system will probably be happy with no pour. There is
no notion of liquid levels changing over time forcing pouring to occur.

In FluidShake is the position of the yellow cube being controlled?

Usually systems like these are flooded with irrelevant
features, and the type of learning used can't figure out what is relevant or irrelevant with a reasonable amount of training data.

I am skeptical that this system can handle color/texture map variations on the objects, or changing the shape or size of the containers
or blocks, or even movement of the table.


summary_of_recommendation: This paper makes a claim and some progress on behavior capture for liquid/granular materials, and, although imperfect, should be accepted.

issues: See the weaknesses above to make the paper more convincing.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper demonstrates an approach for modelling dynamics of a scene using an implicit neural representation similar to NeRF. The model is based on an autoencoder framework, where an encoder maps first-person images to a latent state representation that encodes the system state, and a decoder takes the state representation and a viewpoint as input and renders an image. A dynamics model in the latent state space enables predicting future states, and applying the decoder to render future observations from arbitrary viewpoints.

At test-time, the system takes a viewpoint and a goal image as input, and uses an optimization process to construct the corresponding scene representation for the goal. This optimization process works better on out-of-training-distribution viewpoints than directly encoding the goal image with the encoder.

The latent state dynamics are learned with a time-contrastive loss that pulls state representations from different viewpoints at the same time-step closer, while pushing representations from the same viewpoint but different time-steps apart. As a result, the latent state vector focuses on the state of the dynamic system in a viewpoint-invariant way, while the decoder is responsible for modelling viewpoint-variant rendering. This makes the state representations amenable for planning and control.

The paper applies MPC to reach a desired goal state represented by an image from an arbitrary viewpoint. The control experiments include two scenes: fluid pouring and moving a water-filled box to create waves. Additionally, the paper shows that the system is capable of novel view synthesis of a collapsing tower of blocks, and a falling cube.

strengths_and_weaknesses: Strengths:
- Technically strong and well thought out approach.
- Demonstrates an exciting new result modelling scene dynamics over a latent space on which an implicit volumetric scene representation is conditioned.
- Able to model dynamics of rigid bodies, fluids, and soft bodies.
- Convincing experimental results.

Weaknesses:
- Does not yet enable generalization of dynamics understanding to new scenes.
summary_of_recommendation: Implicit representations of geometry and rendering have recently seen explosive interest in computer vision and graphics research, due to enabling previously unimaginable results, such as neural rendering from arbitrary viewpoints, scene composition, and 3D reconstructions from still images. Such reasoning about geometry and dynamics could greatly benefit robotics applications as well. Traditionally, model-based reasoning (e.g. MPC, task and motion planning) in 3D has required either hand-specified state representations and dynamics models, running simulators in the loop, or learned models that reason in 2D from fixed viewpoints. If we could instead learn such dynamics models and scene representations in 3D just from observations, that could pave the way for new planning and learning algorithms and breakthroughs. This paper takes a step in that direction and should appear at the conference. The main weakness is the significant amount of learning required for each scene individually and the lack of generalization to new scenes, however this is not unlike most RL research, and also inline with the state of the art in implicit scene representations in vision.
issues: The main paper would benefit from at least a few sentences commenting on how many viewpoint images, from how many timesteps were needed during training, and how long did the training take per scene. The NeRF approach required a lot of viewpoints and a very expensive optimization. I wonder how that is affected by adding the temporal dimension and the time-contrastive loss.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a method for viewpoint-equivariant representation learning for visuomotor control. The approach involves learning implicit 3D representations of tabletop scenes with NeRFs and using time-contrastive loss to enforce time consistency. This implicit dynamic scene representation is used as a forward model for model predictive control to achieve specific state-configurations. The viewpoint-equivariant representations allow for specifying target configuration with goal images from arbitrary viewpoints. Experimental results show that the approach is able to achieve desired state-configurations with fluids and rigid-body objects in simulated setups.
strengths_and_weaknesses: Strengths:
+ This paper proposes an exciting direction for learning scene-dynamics with 3D awareness – a powerful prior for embodied robotics. Most end-to-end learning methods ignore the underlying 3D environment and try to directly learn from pixels. Enforcing an implicit 3D structure could help robots learn reusable skills that aren’t tethered to a specific viewpoint.
+ The subfield of neural scene representation learning has been gaining a lot of traction in the vision and graphics community. This work could potentially lead to interesting directions for learning scene and object representations for robotics applications.
+ Generally, the paper is well written, the methods are easy to understand, and the figures are informative. The supplementary video is also well made and effectively showcases the capabilities of the approach.


Weaknesses: 
- One concern is that the evaluation setup is a bit contrived and doesn’t really showcase the full-potential of the 3D dynamic scene representation in a realistic robotics use-case. In the fluid-pouring task, the goal is to achieve the exact fluid state specified in the goal image and the quantitative evaluations measure the Chamfer distance between the fluid particles and the pose error of the cup. In a realistic use-case it’s hard to provide an exact goal-image of the fluid state at a specific time step, and even so it’s unclear if this sort of specificity is necessary in the first place. Alternatively, the evaluation could measure the robot’s ability to complete end-tasks like pouring the liquid upto a certain level – say “20 ml”, or for the fluid-shaking task – make 4 loops around the container with the cube touching each of the four sides. These end-task evaluations could provide better insights into how important 3D-dynamics are for precise model predictive control. 
- Another concern is that using NeRF restricts scene representations to mostly tabletop scenes. It’s unclear how neural implicit representations can be scaled to embodied settings like household kitchens or rooms with partial observability. Further, if we are restricted to a tabletop setting, I am not sure why the camera pose would dramatically change in a real-world setup between the input and goal viewpoints other than minor perturbations to the physical setup. One could imagine a user taking a photo of the intended goal from a third-person perspective, but again capturing the state of a fluid at a certain timestep is a very unnatural way of specifying goals. In summary, the paper currently lacks a strong evaluation use-case where “3D awareness” is absolutely necessary for visuomotor control. Showcasing results on tasks that specifically require 3D-reasoning would substantially strengthen the contributions of the paper. 
- How can NeRFs handle stochasticity? Motions of granular media and fluids are inherently stochastic in nature. Directly training a predictive model leads to blurry predictions, as evident in the supplementary figures and videos. This limits the approach to short-horizon tasks where the forward model can only be rolled-out for a few steps into the future.

summary_of_recommendation: This paper proposes an interesting direction for learning implicit dynamic models with 3D awareness. However, the experimental evaluation is missing a strong robotics use-case that highlights the importance of learning 3D-aware implicit dynamics models for model-predictive control.

**Post Rebuttal**  
Thank you for the detailed response and new results. I'm still a bit skeptical about learning visual-prediction models for stochastic tasks that involve fluids (even with VAEs). The blurriness of the real-world results partly reaffirms this. But nonetheless, I think the paper poses an interesting direction for using NeRFs in robotics, which could be valuable to future follow-up works. I am keeping my original score of Weak Accept. 
issues: Major:
See comments above on better evaluation tasks.

Minor:
- L232: “similar to [25]” -> “similar to XX approach [25]”
- Related work: there has been some recent progress in learning dynamic/video NeRFs (https://arxiv.org/abs/2103.02597). Although these-=- methods do not condition NeRFs on actions to learn a dynamics model, it might be still good to cite them.
- How many training images are needed per scene? How long does it take to train each scene? 
- Can you provide any quantitative evaluations for Section 5.3? Perhaps L2 distances on nearest neighbor search and novel-view synthesis?
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The authors propose learning a (multiview) latent space dynamics model where the latent state is (self)supervised via both a time contrastive loss and a NERF-based autoencoder. By parameterizing the radiance field with both the latent state and viewpoint, they allow a single NERF to model the dynamic scene as it changes across time. At inference time, the state vector (for the goal state only) is estimated by backpropagating through the NERF decoder (keeping weights frozen) and updating only the state parameters (i.e. an auto-decoding framework similar to DeepSDF). The authors show that view equivariance endemic to NERFs allows for minimal performance decrease using an MPPI planner with novel (out of distribution) goal viewpoints.
strengths_and_weaknesses: The overall idea of the paper is laudable. It builds on recent advances in 3D scene representations and (factorized) visuomotor control and proposes an elegant and intuitive solution for introducing additional inductive bias in the form of a 3D spatial representation. The experiments suggest that this sort of approach is very well suited to reducing the amount of data required for visuomotor generalization by reducing the need to train across many goal viewpoints.  

The biggest weakness of this paper is it's failure to address the practicality of the method in terms of data efficiency, computation, and robustness to real-world scene variation.

On the data efficiency and computation fronts, the paper would be vastly strengthened by more details on the number of training examples required and the compute time. NERFs are notorious for being expensive to fit; how many views (and from what distribution) are used to fit a dynamic scene during training and how long does phase 1 (the portion learning the state representation and NERF) take to train?

Experimentally, this paper's impact is somewhat lessened by its lack of real-world data. Because the simulation environments are consistent, things like lighting and shadows will be exactly the same between training and inference. The current process fits a NERF (and state representation) to a very specific scene setup and it's not clear how this approach will work in the physical world when things like lighting shift and other sources of variation may occur between training and inference. The authors also do not use the auto-decoder procedure on the robot's ego views, only on the goal viewpoint. It would be good to discuss this choice, naively it seems like having ego-view invariance would also be pretty valuable (imagine just setting a camera down anywhere in the robot's workspace instead of keeping it fixed to a single location).

summary_of_recommendation: While this paper does have shortcomings (I would have liked to see additional experiments in less carefully controlled environments and a discussion of some of the practical aspects of the method) it presents an original and useful contribution to the field. The results presented here are promising and should encourage others in the community to build on this sort of spatially supervised approach.

Update (post Author Response): In light of the thorough responses from the authors and additional information and experiments provided, I'm updating my "Clarity Of Presentation" score from "Good" to "Very Good". I think that this paper is fundamentally novel, interesting, and valuable to the field and should be accepted.
issues: Please see above comment about including more experimental details such as the amount of data used to train for each task and the training time required. The paper would be stronger with a more fully fleshed out appendix addressing these details.

A few other questions:
- I'd also like the authors to clarify exactly what rendered viewpoints were used to train the NERF representation. Is the NERF portion trained with viewpoints similar to the "out of distribution" evaluation views?

- Additionally, why not use the same approach (auto-decoder though the NERF instead of using an encoder) for estimating the current state during control, is it just computationally too expensive to do this at every timestep? This would allow the policy to generalize to different ego perspectives.

- What is the nominal trajectory used with MPPI, is it random is does it come from somewhere (e.g. a demonstration)?

As a side (minor) comment, this related work may also be relevant: the authors embedded an explicit 3D representation into a language-grounding classification task to enable inference from never-before-seen viewpoints). Grounding language attributes to objects using bayesian eigenobjects (Cohen et al. 2019)
review_title: null

Meta-review: **Update after rebuttal**
The authors have addressed most issues raised by the reviewers, and overall I recommend accept.
Two more comments:
* I agree with the reviewer that raised the concern of integrating the results presented in the rebuttal into the paper. Please try to do so as best as you can. 
* I still would like to see a more "balanced" discussion on what the limitations are

**Initial meta-review**:
**Summary**:
This work presents an approach towards learning viewpoint-equivariant visual latent representations that can be used to used for visual planning/control (via a learned dynamics model in the latent space). The proposed framework/algorithm allows to specify targets/goals from viewpoints different from the robots viewpoint, and can support goal specification from camera viewpoints that are outside the training distribution.

**Strengths**:
The manuscript addresses a very important and timely problem within the robotics community. Progress on learning visual latent representations and dynamics models that generalize across viewpoints can have major impact.

The manuscript is well written, the work will motivated, and a technically strong contribution is made.

**Weaknesses**:
The reviewers raise some questions that should be addressed by the authors

* Overall, several reviewers raise the issue of evaluation on simulated data only, where the data is very “clean/simple”. It is unclear how well this approach would perform on real world data? The authors should consider performing a subset of the evaluation on real world data (maybe not the downstream task evaluation, but only the learning representations part). Alternatively, there may be be way to introduce some noise?
* it is unclear how much training data from how many viewpoints is required to train the latent representations and predictive model, and how much time it took to train those representations.
* The manuscript is missing a clear discussion of the limitations in terms of generalization and what has been learned. While it is useful to use the big vision/claims for motivation, it is equally important to clearly state what has been achieved with this work, and where the limitations lie. For instance it seems unlikely that the system for the pouring task learns anything about the water levels (eg half full) for various sized containers. A thorough discussion would better enable future research on this topic.
Decision: Accept (Oral)