Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The method enables the robot to apply 6-DOF pick-up and place actions given RGB-only image of the environment with objects. Instead of directly learning 6 DOF pose, the system learns to predict the best-projected plane of the scene that enables an orthogonal approach of the end-effector to this projected plane for successful pick-up and place. NeRFs enable rendering of the scene with orthogonal projection from any arbitrary camera pose, given a number of scenes from different camera poses. NeRFs is used to generate a number of projections from sampled camera poses, and the problem is reduced to learning affordances and 3 DOF (x,y,yaw) parameters of pick-up and place actions.

The method is verified in the simulator in different environments, and in the real robot in an insertion task. The performance is shown to surpass the performance of a number of SOTA methods in the simulator. The method does not require depth information, yet achieves good performance in 6 DOF action execution.
strengths_and_weaknesses: The strenghts of the paper:
- Instead of RGB-D data, the method works with RGB only data. This enables the system to work with reflective and this structured objects.
- Object detection is not required. The affordances are learned in pixel level.
- Orthogonal projection enables better affordance prediction compared to perspective projection.
- The method works well in out-of-distribution validation cases compared to other methods.
- The paper is very well written.
- The method is verified with experiments that require generalization capabilities.

The weak points in the paper:
- Many stated contributions can be attributed to the existing NeRF method. As such, the used scene representation created by NeRF is not novel. The orthogonal projection is not challenging with NeRFs. Assigning affordances to pixels of scenes is also not novel.
- The paper combines the following: Generating a number of projected scenes using NeRF and assigning affordances to the pixels in these scenes. This approach is smart, but it corresponds to an engineering effort of combining existing SOTA methods, rather than proposing a highly original/novel method with significant impact to the field.
- The method is compared with baselines such MLP, Form2Fit, Transporter-SE(2/3). This paper out-performed the other methods probably because of the underlying NeRFs method, however NeRF is not a contribution of this paper. In order to better identify/show the contribution, the paper could have been compared with other existing control methods that use NeRF such as [54] or [60].
- There is no object detection step, yet, the method was not verified in cluttered settings with more complex actions (such as grasping with a gripper).

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper utilizes the NeRF in a smart way in order to create potential projections of an arbitrary environment and reduces 6DOF end-effector pick-up/place pose search into a search of projections + orthogonal approach with 3DOF for pick-up/place. Although this is clever and shown to be effective compared to the existing methods, this corresponds to a combination of existing methods rather than a proposal of an original method with high impact, theoretically. Practically, as the contribution is on the control part, the proposed system needed to be compared with other NeRF-based methods.
issues: - Is is possible to compare with other NeRF based control methods? [54] makes 6 DOF control. Can [60] be extended to 6DOF control?
- Is taking 30 snapshots from different camera poses feasible and scalable in the real world?
- How well does the method perform in cluttered environments?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper uses a state-of-the-art neural representation, i.e., neural radiance fields (nerfs), to plan robot manipulation actions. Using a set of 2D images, acquired from a diverse range of angles/poses, a 3D volume of the scene is optimized, from which an orthographic view is rendered and fed into a policy that predicts a pick-and-place action value. In essence, the diverse set of initially acquired views allows synthesizing new views that might result in selecting a better manipulation position. 

strengths_and_weaknesses: strengths:
- a good use-case of nerfs for robotic manipulation
- a good range of evaluations, including real robot experiments

weaknesses:
- no methodological novelty
- too many views initially required to build nerf
- too long optimization/computation time prior to action prediction + very long time to gather the initial views -> infeasible for realistic manipulation tasks

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper investigates a straight-forward application of neural representations for robot manipulation. In particular, neural radiance fields (nerfs) are optimized from multiple views that allows constructing the 3D volume of the scene, which in turn enable better manipulation action planning. The evaluations and the experiments cover a good range and provides convincing evidence on the usability of the approach. The major drawbacks stand as the computation time and the initial time cost of acquiring 30 images from different views, both of which diminishes the value proposition of the work significantly.

issues: - an in-depth analysis on the effect of number of initial images.
- related work: Nerf for Robotics subsection needs to provide a context as to the difference of the proposed approach to the mentioned prior work.
- related work: how does this work compare to [R1,2] 
- where do expert demonstrations come from?
- evaluation of and discussion on how/whether the same approach is applicable to 2/3-finger based grasping/picking actions. 

[R1] D. Driess et al., "Learning Multi-Object Dynamics with Compositional Neural Radiance Fields", arXiv preprint, 2022.
[R2] D. Driess et al., "Reinforcement Learning with Neural Radiance Fields", arXiv preprint, 2022.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents MIRA, a supervised pick-and-place manipulation framework that uses a NeRF-based scene representation to efficiently learn 6-DoF policies. The framework first builds an NeRF of a tabletop scene with 30 images captured from various viewpoints. A number of sampled viewpoints are rendered to generate novel perspectives of the scene. Then, actions are predicted by selecting the best pixel location to pick and place (separately) over all the rendered perspectives. This approach is able to learn robust 6-DoF policies without any depth data, which makes it applicable to objects with reflective, glossy, and translucent surfaces that are typically hard to model with commodity depth sensors. Experimental results include comparisons against TransporterNets, Form2Fit, and ground-truth state MLPs. The results show that MIRA significantly outperforms these baselines, specifically in scenes with out-of-distribution object poses, all without using explicit depth information.
strengths_and_weaknesses: **Strengths** 

+ Overall, the idea of using NeRFs with action-centric perception 6-DoF manipulation is quite novel and an interesting direction for the field. Naive image-to-action agents often lack the 3D prior for efficiently learning 6-DoF actions. Even with sufficient data, image-based models struggle to be robust to scene changes. Further, NeRFs also circumvent common issues with depth-cameras like modeling specular, glossy, translucent surfaces, and small objects. While there are some limitations to MIRA's approach, the formulation here has a lot of potential to be extended to general problems in vision-based manipulation. It’s exciting to see how future works build on this approach. 

+ The experiments cover a good set of baselines for action-centric perception. These evaluations test settings with both in-distribution and out-of-distribution object poses. The results also include ablations that investigate specific contributions like orthographic projection and multi-view negative sampling for EBM training. The simulated experiments are set in an open-source simulator (Ravens), and should be easily reproducible, especially if the authors release their code and setup. The real-world results are quite compelling, particularly the results with shiny and translucent objects.  

+ The writing and presentation of methods is mostly clear. Figure 1 provides a good overview of the framework. Section 3 does a good job of providing preliminaries on NeRFs and orthographic ray-casting. The supplementary video also provides a good overview of the approach and highlights its benefits. 

**Weaknesses** 

- One key limitation (acknowledged in the limitations section) is that MIRA requires 30 multi-view RGB images, which takes 1 minute to capture with a robot arm. Optimizing the NeRF requires another 10 seconds. While this might be feasible for simple pick-and-place tasks, it’s not easily applicable to dynamic tasks that require real-time visuomotor control. For instance, while opening a cabinet or pouring water into a mug, it’s hard to capture 30 images with the same manipulator that is interacting with the object. However, given the rapid-pace of advancements with neural radiance fields, it might be possible to build 3D scene representations with limited viewpoints like in pixelNeRF (Yu et al. 2020). Nonetheless, the contribution here to quasi-static 6-DoF pick-and-place tasks itself is sufficient and interesting to the robot-learning community.  

- Section 3.2 and 3.3 could benefit from additional details on the approach. More specifically, are all the multi-view images batched and processed with the ConvNet in a single-pass? Isn’t this expensive? I also assume it’s one ConvNet with shared weights for all the images, or is each image processed with independent ConvNets and kept spatially consistent across scenes? What is the ratio of positive to negative samples for the EBM training? And how dense are these samples – can you use every pixel in the image as training samples, like in TransporterNets? 

- Similarly, Section 4.2 (or an appendix section) could provide additional details on the image-capture process. How are the 30 multi-view images captured for the real-robot experiments? Is this a predefined path executed with IK or a motion-planner? This information might be useful for practitioners to understand how to get a good coverage of the tabletop. Also, were all 121 virtual images crucial for precise 6-DoF actions? Would it be possible to use less virtual images? Perhaps this could be investigated with simulated experiments by varying the number of virtual images and studying how it affects performance in Ravens tasks. 

References: 

Yu et al. 2020 – https://arxiv.org/abs/2012.02190

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: MIRA presents a novel and interesting approach for using NeRFs with action-centric perception for  pick-and-place manipulation in 6-DoF settings. The ideas and experiments from the paper are very relevant to the CoRL community. 
issues: Mostly additional details regarding the approach. See the Weaknesses section above for details.
review_title: null

Meta-review: metareview: The paper presents a supervised pick-and-place manipulation framework that uses a NeRF-based scene representation to efficiently learn 6-DoF policies. The paper is well written and guides the reader well. It has a good technical quality. The paper is more towards solving an important application using multitude of appropriately combined tools.  The reviewer concerns were adequately addressed. With the current experiments, the paper makes sufficient contributions to the community. . 

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)