Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors propose a method for learning dynamics from multi-view series data of an environment consisting of multiple objects. The proposed method has NeRF to include 3D inductive bias and GNN to extract the compositional representation to enable long-term prediction. Experiments on both simulators and real-world situations confirm the effectiveness of the proposed method.
strengths_and_weaknesses: Strengths:
- The paper is well organized and easy to read.
- The proposed method is valid and novel compared to existing NeRFs.
- Experiments confirm its effectiveness on complex datasets of multiple objects. In particular, it is very significant that it is demonstrated to work properly on deformable objects and objects in real-world environments.

Weaknesses:
- In this paper, the authors aim to obtain object-specific representations from multi-view images and to predict future observations. However, the authors do not mention many related works on similar tasks (although some are cited, such as [32]). For example, many studies on acquiring object-specific representations from images or videos for segmentation and prediction have been studied as object-centric representation learning (or object-centric world model) (1,2,3,4,5). In particular, in (5), object-specific representations are acquired from videos using graph neural networks. There is also work on learning object-centric representations in multi-view (6). I think this proposed method has novelty compared to these in that it acquires object representations in multi-view and time series, but the authors should cite these studies and explicitly discuss the differences.
(1) MONet: Unsupervised Scene Decomposition and Representation
(2) Multi-Object Representation Learning with Iterative Variational Inference
(3) Entity Abstraction in Visual Model-Based Reinforcement Learning
(4) Improving Generative Imagination in Object-Centric World Models
(5) Contrastive Learning of Structured World Models
(6) Learning Object-Centric Representations of Multi-Object Scenes from Multiple Views
- The authors state that "we believe having masks is a reasonable assumption," but many of the above object-centric methods do not assume that masks are given, and acquire object-specific representations only from images and videos. Therefore, it is not obvious that this assumption is reasonable in my opinion. The authors would like more explanation as to what perspective this assumption is valid compared to related studies.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The method is novel and its ability to make long-term predictions in the real world is a significant contribution; however, as mentioned above, I still have concerns regarding the lack of citation of previous studies and the appropriateness of the problem setting. Therefore, I consider it a weak rejection.
issues: I would like you to work on the concerns I have listed in "weaknesses". If you respond appropriately to these and revise your paper to reflect them, I will consider raising the score.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper studies the problem of multi-object video/motion prediction problem in robotics. Specifically, the framework consists of three modules: the encoder, GNN dynamics model, and NeRF decoder. First, the encoder takes in initial observations of the scene from multiple views and computes feature volume and latent variables for each individual object. Second, the NeRF decoder renders the view for reconstruction. Third, the GNN dynamics model learns to predict future latent variables using the adjacency matrix inferred from the NeRF density between each of the two objects. In practice, the encoder and decoder models are first trained end-to-end using the L2 image reconstruction loss, and GNN is further learned using the inferred latent variables. Experiments are mainly conducted on the non-public synthetic dataset and a few real objects (up to 4).
strengths_and_weaknesses: Strengths

* [S1] This paper studies an important task in robotic manipulation by leveraging vision signals (e.g., appearance, motion, and shape) for representation learning. 
* [S2] The paper provides sufficient technical details (especially in the Supp) for replication.

Weaknesses

* [W1] The proposed framework assumes that accurate object masks are given as input to the model, which is not realistic. Although this has been discussed in section 7, the reviewer would like to know how robust the proposed model is given inaccurate masks provided using off-the-shelf instance segmentation & tracking models. Ablation studies on this are highly recommended for the next version of the paper.
* [W1.1] With inaccurate object masks, the reviewer suspects the proposed method may suffer from sudden lighting changes or ID switches (e.g., all objects with the same appearance). It would be great to clarify this.

* [W2] The motion dynamics shown in the paper and demo are easily predictable using a simple regression model. For example, if the objects of interest are articulated intelligent agents (e.g., toys) which can move by themselves, motion prediction involves a higher level of uncertainty and multi-modality. The reviewer fails to see how does GNN model handle this in the current setting. Please discuss this more explicitly as the limitation of the proposed model.

* [W3] This paper conducts experiments on non-public datasets, which makes it a bit challenging to understand the significance of the work. In addition, experimental evaluations only include ablations of different components of their proposed method rather than existing SoTA implementations. The reviewer highly recommends adding results on public benchmarks (e.g., Multi-Object Datasets and Kubrik [NewRef1-2]) even without the dynamic GNN.

* [W4] Besides prior work [31-35], there is more recent work on modeling dynamic scenes using compositional NeRFs (e.g. [NewRef3]). It would be good to discuss these existing works and consider them as potential baselines in the next version of the paper.

References
* [NewRef1] Multi-Object Datasets. [GitHub](https://github.com/deepmind/multi_object_datasets).
* [NewRef2] Kubric: A scalable dataset generator, In CVPR 2022.
* [NewRef3] Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation, In CVPR 2022.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This is a borderline paper. On the positive side, it studies an important problem with meaningful extensions to the previous work. However, the reviewer holds reservations due to the strong assumption on object mask, over-simplified motion setting, and significance of the work. The reviewer would also like to exchange ideas with peer reviewers after the rebuttal.
issues: Mentioned in [W1-W4].
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors propose to compositionally encoder scene RGB observations into implicit neural representations. The dynamics is represented as a factored graph. The authors demonstrated the validity of their approach with various experiments.
strengths_and_weaknesses: Strength:
* The paper is very well written. I had a great time going over the paper with all the interesting details. Things are presented in a clear manner, and a lot of content is packed into the short pages.
* The method is quite reasonable. It captures the "3D-ness" of objects, as well as their more lower dimensional dynamics relationship.
* A lot of the relevant works are provided, showcasing the authors' rich knowledge on the field. This is also helpful for the readers.

Weakness:
* I think some quite related recent works along the direction is not cited. For example, using implicit neural representation to represent deformable object dynamics: ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation Bokui Shen, Zhenyu Jiang, Christopher Choy, Leonidas J. Guibas, Silvio Savarese, Anima Anandkumar, Yuke Zhu, RSS, 2022.       NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields. Lin Yen-Chen, Pete Florence, Jonathan T. Barron, Tsung-Yi Lin, Alberto Rodriguez, Phillip Isola. In ICRA 2022.
* I would love to hear more about the authors' thoughts on generalization. I think this might be a limitation of the approach that's worth discussing. When working with an open set of objects, will such approach still work? It will be really hard to keep a large library of INRs, and the GNN will have a difficult time reasoning if its library of latent codes grow more and more.
* Supplementary material file doesn't have a bibliography. Seems like a bug.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper is an acceptance in my opinion. However, I would love to hear more from the authors about how they see such approach generalizing and being used on more objects that are unseen. 
issues: * buggy supplementary bibliography
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper leverages NeRF for multi-object scene representation from image observation in a compositional manner. NeRF based autoencoders are deployed to encode each object into latent vectors, which were used compositionally to learn a compositional forward dynamic model using GNN.  The learnt model generalized across objects and provides stable long-term prediction that helps with manipulation planning and reasoning. RRT based methods can then be used to plan in the latent space for downstream manipulation tasks.

strengths_and_weaknesses: Strength:
- The paper shows that compositional NeRF and GNN can produce a latent space where long horizon prediction is stable. Examples are shown for both simulation and real environment. And thanks to the nature of NeRF, the latent space is agnostic to view points. 
- The method can be generalized to the number of objects and handles the interaction between objects well, even when the object is deformable.
- Comprehensive ablation studies are performed to support the design choices of the model components.
Weaknesses:
- For experiment 6.1, using image error only to measure the long horizon prediction error is not so convincing. NeRF is known to be a much better decoder to represent static 3d scenes compared to CNN. Even without forward dynamic prediction, CNN is not expected to have better reconstruction quality. But it doesn’t necessarily mean the latent space learnt with a CNN decoder is not suitable for long horizon dynamic prediction. From the supplementary video, CNN decoder can also capture the pose of the objects reasonably well. It is worth reporting the pose error of objects as well. Also there are a few questions regarding figure 3. 
  - MLP dynamic model is shown on fig 3(b) only.
  - Why does the prediction error go down after 40 steps?
  - The y-axis scale of fig 3c is different from the rest.
  - It would be great to have more visualizations when using Global NeRF as decoder in Fig 4. It is interesting to see how the compositionality improves the decoder.
- For experiment 6.2, only qualitative results are shown. More statistics would be helpful to determining the contribution of each module compared to the baseline. For example success rate, avg. time/step taken to complete the task, etc. Without them, the experiment is not robust enough to support the claim that the method produces better latent space for planning and reasoning for robotic tasks.
- Similarly, only qualitative results are shown for the real world experiments. According to the video, the image prediction is no longer sharp when the objects are textured. This makes the visual reconstruction quality questionable as the proposed method might not be able to scale to more complex objects (in terms of shape and textures).

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper shows that compositional NeRF and GNN can produce a latent space where long horizon prediction is stable. But the supporting experiments are weak. See above for more details.
issues: See above
review_title: null

Meta-review: metareview: This paper introduces a dynamics model for multi objects that combines a GNN and Neural radiance field. The paper shows that the improved rendering this allows for longer horizon dynamics prediction as it results in an improved latent space, and that compositional dynamics is important for dynamics prediction. Results are impressive, in both simulation and real-world scenes, with MPC used to demonstrate control using the learned vision-based dynamics model (although it would be interesting to see this scale to scenes with more complex backgrounds). The original review process highlighted that:

* The paper is well-organised and easy to read
* The work is novel, and results show that it works on non-trivial deformable objects
* The NeRF aids in the learning of 3D inductive biases, and important property lacking in more free-form approaches

but reviewers expressed:
* Some concerns about generality, scaling to open set objects
* Concerns about a requirement that objects masks are already known
* and queried some missing related work

The authors provided a strong rebuttal to these reviews (some scalability already present, additional experiments around the effects of object mask uncertainty, additional discussion of related work). In light of this, I recommend acceptance.


confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)