Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This manuscript introduces a multi-view transformer for 3D object manipulation.  The transformer contains an attention mechanism to gather information from multiple view images (re-rendering from virtual views) and output the target end-effort pose. The low-level controller or motion planner generates actions to achieve these target poses. The authors applied the proposed pipeline in the RLBench to demonstrate its effectiveness and also tested it in real-robot experiments.
review: Strengths:
1. The paper is well-organized and easy to follow.
2. The proposed Robotic View Transformer framework achieves significant improvements compared to other baselines
3. The authors applied the proposed pipeline in the real robot experiments

Weakness:
1.  Can these multi-view images be directly achieved using multiple cameras, since the pipeline requires first reconstructing a point cloud of the scene? If real multiple-camera images are fed into the pipeline, will it improve the performance? 
2. How accurate are the re-rendered images? Does the quality of these re-rendered images affect the performance of the action prediction?
3. What would happen if the predicted key frame(gripper poses) is not achievable due to collision or joint limit?


robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The authors proposed a Robotic View Transformer framework for 3D object manipulation achieving significant improvements compared to other baselines. However, this pipeline requires first reconstructing a point cloud. How the quality of the reconstructed point cloud affects the performance is not clear to the reviewer. If multiple images are needed to reconstruct the point cloud, what are the advantages of re-rendering and how the quality of the rendered images affects the pipeline performance is not clear.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This work proposes a new framework (RVT) for 3D object manipulation. Instead of using voxel-based or NeRF-based methods, the authors propose to synthesize virtual 2D images from a point cloud of the scene and feed the virtual images into a multi-view transformer. The multi-view transformer predicts a 2D heatmap for each view, which is then back-projected to the 3D space to determine the next best end-effector position. The global features aggregated from the outputs of the transformer are used to predict the end-effector rotation, gripper actions, and collision indicator.

The authors performed thorough simulation experiments on RLBench, where results show that RVT not only outperforms PerAct in 16 of the 18 RLBench tasks but also achieves much faster training time and inference speed compared to PerAct. The authors also conducted a set of real-world experiments, showing that their proposed method could reliably perform a range of 5 manipulation tasks on a real robot platform.
review: Strengths:

The paper presents an interesting method to improve the accuracy and speed of 3D object manipulation. The writing is clear and easy to read. The author also conducted a thorough literature review of related works in the field of visual object manipulation and multi-task policy learning.

Issues:

One issue in the real robot experiments in the paper is that there seems to be no quantitative comparison between the proposed method and the baseline. It would be great if the authors could consider running PerAct on real robot environments to confirm that their method still outperforms the best baseline in the sim experiments.

The paper also seems to lack discussion of the relationship between their work and other recent works that make use of transformer-based or diffusion-based architectures for performing 3D object manipulation [1, 2] (note that those works went public on Arxiv at least a month before the CoRL deadline).

Some clarification questions:
- In Section 3 Line 160, the authors mention that the heatmaps of multiple virtual views are back-projected to 3D. Could the authors elaborate on how this process is done? Is there a mapping between the 2D pixels in the virtual views and the 3D points in the point cloud? What does the algorithm do to the parts of the image where there are no corresponding points in the 3D space? Do the heatmap values from multiple views get averaged after they are back-projected onto a set of points?
- PerAct extracts a set of keyframe actions from the demo by capturing bottleneck end-effector poses in the action sequence that has (1) near-zero joint velocities or (2) an unchanged gripper open state. It seems from line 292 that the keyframe actions are specified in a different way in RVT. Could the authors clarify the difference between the two ways of collecting demos?
- In the ablation experiments, the authors argued in lines 271-274 that rearranging sensor cameras to match re-rendering views in order to bypass re-rendering would "void the gains from using orthographic projections, 3D augmentation, and adding correspondences", but did not seem to actually run such an experiment. Could the authors add such an experiment in Table 2 left? Also, it would be interesting to see ablation experiments where only the sensor camera locations change on top of the original experiment (ie. Table 2 left first row).
- In the real robot experiments, only one camera view is used. How carefully does this view have to be selected so that the method performs well? Would the performance of the method improve if more cameras are placed in the workspace?

[1] Zhao et al. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. RSS 2023.

[2] Chi et al. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. RSS 2023.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper introduces a novel method for 3D object manipulation that could inspire future efforts in this area. The contributions are clear and the experiments are well-executed. The paper could improve with more real robot quantitative results and some minor writing fixes. If the authors could address those concerns, I would be happy to recommend the manuscript for publication.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper introduces the Robotic View Transformer (RVT) for manipulating objects in 3D. RVT leverages Behavioral Cloning to learn from a small number of demonstrations, enabling it to execute various manipulation tasks using a single policy. By utilizing a language description of the task and a set of RGB-D images as input, RVT predicts the target end-effector pose and gripper open/close state at each time step to generate actions. A notable architectural innovation of RVT is the incorporation of "multi views," where virtual cameras are strategically positioned in the scene and aggregated through an attention mechanism to make predictions in a 3D space. Comparisons with Perceiver-Actor (PerAct) from CoRL 2022 show that RVT gets similar performance but trains *much* faster and has much faster inference speed.

review: Strengths of the paper and RVT:

1. Perceiver-Actor (PerAct), a model proposed at CoRL 2022, is quickly getting a lot of traction in the robot learning community (with lots of citations for a recent paper). I view RVT as the next version of PerAct, and it has 36X faster training time and 2.6X faster inference time, for the same performance. This is a very compelling result because PerAct's training time and requirements are a huge limitation to its usage in the research community.

2. RVT can use a small number of demonstrations to train a single multi-task policy for a wide variety of language- and image-conditioned 3D manipulation tasks, where the policy has to predict full 6D robot end-effector poses, as well as a gripper opening/closing choice.

3. There are a lot of simulation experiments using RLBench and compelling real world results.

4. There are lots of ablations and analysis of the design choices for RVT.

Minor comments and questions:

1. For Figure 1, why does the performance of RVT stagnate after 60%? Would it possibly decrease if it was trained longer, due to overfitting?

2. In line 272: can the paper briefly clarify why "this will void the gains from using orthographic projections, 3D augmentation, and adding correspondences"?

There are a few limitations of the paper as mentioned in the conclusion. In addition to these, I would first suggest adding that performance is still only 60% (from Figure 1) so there's still more to go. Second, some might view this paper as "only" an engineering contribution built on top of PerAct. However, I think the usage of multiple views here results in sufficient technical novelty.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, this paper makes a strong contribution to the research community. PerAct was already one of my favorite papers due to its multi-task and manipulation performance, and I'm happy to see research that makes training and running it much faster. (I am not an author on PerAct nor do I collaborate with the authors.)
review_title: null

Meta-review: The paper presents an extension of PerAct using a continuous representation introducing a multi-view transformer architecture that achieves significant improvements over the prior method, and the paper also provides real world evaluation of the method. Overall, they idea of keyframe prediction of end-effector poses for complex manipulation tasks is a method that  gains a lot of traction.

The reviewers have evaluated mostly positive the work, and the authors have done a good job in the rebuttal period, particularly providing additional results on the real world comparison of RVT and PerAct, showcasing the superior performance of PerAct. The addition of clarifications on the experimental process and the the additional real-world comparison are essential for the paper and I expect to see them added in the camera-ready version of the paper.
Decision: Accept (Oral)