Super Robot View Transformer

Authors: Xiaohan Lei, Min Wang, Wengang Zhou, Houqiang Li
Primary area: applications to robotics, autonomy, planning
Keywords: robotic manipulation, multi-task learning, robot view transformer

Abstract:
Learning a single model for multiple robotic manipulation tasks, particularly high-precision tasks, has been a long-standing challenge in robotics research due to uncertainties inherent in both the model and the data. These uncertainties, namely epistemic uncertainty arising from model limitations and aleatoric uncertainty stemming from data variability, hinder precise control.
While the Robot View Transformer (RVT) improves performance by re-rendering point clouds from fixed viewpoints and processing structured 2D virtual images, it still suffers from occlusion artifacts in rendering and limited action precision due to resolution constraints.
To address these limitations, we propose the Super Robot View Transformer (S-RVT) framework, which integrates three novel components: the Super Point Renderer (S-PR), the Super-resolution Multi-View Transformer (S-MVT), and the Hierarchical Sampling Policy (HSP). The S-PR enhances the rendering process to mitigate occlusion artifacts, while the S-MVT integrates super-resolution to the output heatmaps, enabling finer-grained manipulation. The HSP efficiently samples multi-view heatmaps in 3D space to obtain accurate 3D poses.
These innovations collaboratively mitigate the challenges of occlusion and precision in manipulation tasks. Our experimental results demonstrate that S-RVT achieves a success rate of 87.8 \% across 18 manipulation tasks, surpassing the state-of-the-art of 81.4 \%. Notably, for high-precision manipulation tasks, S-RVT exhibits nearly a two-fold improvement over existing methods, underscoring its effectiveness in precise control scenarios. Our code and trained models will be released to support further research.

Decision: reject
Reviewer scores: 6, 5, 5, 3, 6

Reviews and discussion:
<thread>
creation date:
1730479771964

thread_type:
official_review

<thread_object>
creator: reviewer_cpa1

<content>
summary:
Building on prior work, RVT and RVT-2, this study introduces S-RVT to address limitations like occlusion issues and resolution constraints. Specifically, it presents S-PR (Super Point Render) to enhance rendering and reduce occlusion artifacts, S-MVT (Super-resolution Multi-View Transformer) to integrate super-resolution to output heatmaps, and HSP (Hierarchical Sampling Policy) for accurate 3D pose estimation through a coarse-to-fine sampling approach. Experimental results show that S-RVT outperforms previous methods.

strengths:
* The approach of addressing limitations in prior work, such as handling occlusion issues in the rendering process and overcoming resolution constraints in pose prediction, is valuable.
* The authors perform extensive experiments, including comparisons with baseline models and ablation studies, to demonstrate the effectiveness of the proposed components.
* They also conduct several real-world experiments and provide the video evidence.

weaknesses:
* It would be beneficial to clearly specify the differences between the proposed method and prior work, indicating which contributions are adopted from previous studies and which are newly introduced in this paper. For example, in Section 3.2, RVT-2 appears to have already implemented z-ordering and screen-space splatting techniques, yet this is not clarified here. Additionally, in Section 3.3, lines 255 to 263 (nearly half the paragraph) contain content similar to RVT and RVT-2, it would be helpful to focus more on the novel methods introduced in this work. Clearly distinguishing between techniques inherited from previous work and unique innovations would improve understanding and highlight the contributions of this study.
* In Section 3.3, additional details on the upsampling process would be helpful for clarity. Could the authors expand on how the upsampling is implemented?
* The authors are encouraged to provide further analysis of the experimental results:
  - In Table 1, S-RVT performs worse than RVT in tasks such as 'put in safe' and 'sweep to dustpan,' and S-RVT2 performs worse than RVT-2 in 'slide block' and 'sweep to dustpan.' Although these lower scores are acceptable for specific tasks, more in-depth analysis of why the proposed method underperforms in these cases would strengthen the findings.
  - In the ablation study, the impact of each component varies between S-RVT and S-RVT2. For instance, SPR is more critical for S-RVT2, whereas HSP has a greater influence on S-RVT. Additional analysis on why these components affect the models differently would offer valuable insights.
* The authors are also encouraged to discuss failure cases of the proposed method.

questions:
* In line 257, the statement *"S-MVT generates heatmaps with $sr$ times higher resolution"* is unclear, as $sr$ is not introduced in the preceding paragraphs.
* In line 260, should this result in $16^2$ patches instead of $16$ patches?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730565506984

thread_type:
official_review

<thread_object>
creator: reviewer_huzo

<content>
summary:
This paper proposed a framework for multi-task imitation learning with key-state-based methods for robotic manipulation learning, especially for high-precision tasks. The author claims their model addresses the epistemic uncertainty of the proposed framework. The framework, SRVT, consists of three modules: the Super Point Renderer (S-PR), the Super-resolution Multi-View Transformer (S-MVT), and the Hierarchical Sampling Policy (HSP). This paper shows both simulation and real-world experiment results to evaluate the framework.

strengths:
1. This paper conducts sufficient experiments to evaluate the proposed framework on various robotic manipulation tasks with different baseline methods.
2. The pictures in this paper are presented clearly.
3. Related works are comprehensively reviewed.

weaknesses:
1. The overall writing and flow of the paper need considerable improvement. The abstract, introduction, and related works sections are repetitive and convey similar concepts. Moreover, a paper should streamline these sections to provide a progressive understanding.

2. This paper's primary claim that the proposed MVT module can advance epistemic uncertainty is not validated. To address such uncertainty, the paper must provide theoretical proofs, uncertainty analysis, and ablation studies. Some common methods, like conditional variable at risk, can be used for this.

questions:
1. In the paper: "The 3D points are projected onto the 2D image plane by converting them into image coordinates using GPU accelerated matrix operations." Could the author give more details on the implementation and visualization of this?

1. Why the MVT module can address the epistemic uncertainty? Please provide detailed information.

2. In the HSP section, can the author provide some details about how HSP can solve the GPU memory overflow problem?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730646274080

thread_type:
official_review

<thread_object>
creator: reviewer_fpi1

<content>
summary:
This paper presents Super Robot View Transformer (S-RVT) -- a series of techniques to improve Robot View Transformer.  It consists of 3 modules: the Super Point Renderer that mitigates occlusion artifacts, the Super-resolution Multi-View Transformer that performs superresolution to the output heatmaps, and the Hierarchical Sampling Policy that efficiently samples multi-view heatmaps in 3D space. The experiments suggest S-RVT obtains a consistent performance boost against RVT on the RLBench benchmark.

strengths:
- This paper is well-written and easy to follow.
- The proposed modifications on RVT are intuitive, and the RLbench experiments verify the effectiveness of S-MVT in simulator settings.

weaknesses:
- I doubt the fundamental rationality of the Super Point Renderer (S-PR) and Super-resolution Multi-View Transformer (S-MVT) modules.
  - According to my understanding, S-PR renders the objects occluded by the robot. It would definitely be effective in the simulator, but how would that be made possible in the real world?
  - Meanwhile, S-MVT aims to perform super-resolution to the multi-view images. However, why not just enhance the resolution of RGB-D images in the beginning? D515 could capture depth photos in a resolution of up to 1024x768, but the RGB-D images used in the paper only have a resolution of 128x128.
- While S-MVT is compared with MVT in the simulator setting, it is not compared with MVT in the real-world setting. I am particularly confused about how S-PR is implemented in real-world scenarios.

questions:
- How is S-PR implemented in real-world scenarios?
- What will happen if the resolution of input images is raised?
- What's the performance of MVT in the real-world experiments?

I will consider raising scores if my concerns are addressed in the rebuttal period.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730672110150

thread_type:
official_review

<thread_object>
creator: reviewer_4vqv

<content>
summary:
The paper addresses the limitations in previous virtual view-based methods, focusing on the occlusion problems and the resolution constraints. To resolve these problems, it proposes the Super Robot View Transformer (S-RVT) comprising of three modules: the Super Point Renderer (S-PR) that enhances the rendering process to mitigate occlusion artifacts, the Super-resolution Multi-View Transformer (S-MVT) that integrates superresolution
to the output heatmaps, and the Hierarchical Sampling Policy (HSP) that samples multi-view heatmaps in 3D space to obtain accurate 3D poses. Experiments show that the proposed framework improves the performances of RVT and RVT2 in various setups.

strengths:
- The authors have conducted abundant experiments and ablation studies in both sim and real.
- The experimental results look good. The proposed framework brings consistent improvements to RVT and RVT2 across different scenarios.
- The paper is well-written. The concepts and intuitions are explained together with concrete examples, making it very easy to understand.

weaknesses:
- To my understanding, the paper is mainly addressing the uncertainty problems (in RVT or RVT-like robot learning frameworks): the aleatoric uncertainty is addressed by the virtual-view pointcloud rendering, and the epistemic uncertainty is addressed by the feature map superresolution. On one side, I like the intuitions discussed in the paper, on the other side, simply looking at the framework, the ways of resolving these problems look very straightforward, with a sequence of concrete engineering efforts. It would be good to have more concrete discussions, based on the method, on how RVT didn't address these uncertainties well and how the framework resolves these issues -- this can show better linkage between the high-level intuitions of the paper and the concrete steps in the method.
- A concrete question following the previous question is about the pointcloud rendering: It is simply done by a 2D projection, but what is the quality of the projected virtual views? Does it have any requirements on the placement of the (real) camera? Specifically, in the ablation study, it shows that going from 4 virtual views to 5 views decreases performance. I think this implies that the virtual views are not all of good quality which helps the algorithm to figure out better policies.

questions:
- It would be nice if the authors could have some discussions on the data efficiency of the proposed framework. As discussed in the Related Work section, both transformer deployment and imitation learning with high precision require substantial training data. In this paper, the sim experiments use 100 demonstrations per task, and the real experiments use 15-20 demonstrations per task. How does it compare to other methods?
- Another very relevant question: For the real experiments, the paper states that "the number of demonstrations for each task is determined by its complexity and the degree of variability in task configurations." How much will this affect the performances? Specifically, for the two tasks "stack blocks" and "plug charger", how would the model perform if there are only 15 demonstrations, as in the other two tasks?
- I am curious, in Table 1 task Sweep to Dustpan, why is S-RVT success rate lower than RVT? Are there any specific features of this task that make it different from the others, or is it just a normal fluctuation in the measurement? (This is really just my curiosity. The overall experimental results look good to me, and a 10% success rate drop out of 25 tests here is acceptable.)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717635802

thread_type:
official_review

<thread_object>
creator: reviewer_jbkg

<content>
summary:
This paper proposes an improvement over the RVT/RVT2 method by introducing enhancements such as S-PR, S-MVT, and HSP. The updates effectively address the problem of view occlusion in RVT from certain angles, particularly the top view. Extensive experiments demonstrate impressive performance gains with these adjustments, validating their effectiveness.

strengths:
1. **Performance Improvement**: The proposed changes significantly enhance performance by mitigating occlusions from specific viewpoints, improving view flexibility.
2. **Robust Experimentation**: The paper includes multiple experiments to verify the efficacy of each introduced method, which supports the validity of the approach.

weaknesses:
1. **Lack of Visual Illustration**: The concept of the down view is not entirely clear, especially concerning why it lacks color. A visual illustration showing the virtual camera position could enhance understanding.
2. **S-PR Explanation**: It’s challenging to grasp exactly how S-PR contributes to generation. A comparative figure demonstrating results before and after applying S-PR would clarify this aspect.
3. **Lack Real World Baseline**: The baseline experiment in real world is missing.
4. **Unclear Contribution of Different Designs**: Most of the performance in Table 2 (S-RVT2) are within 1 point, which makes it unclear whether many of them are still useful.

questions:
1. **Down View Clarification**: Could the authors clarify the colorless nature of the down view and its virtual camera position? Including a figure illustrating the virtual camera position could make this aspect clearer. Also can you explain why the down view looks colorless compare with top view?
2. **Effect of S-PR on Generation**: The design of S-PR is unclear. Could the authors include a figure comparing the rendered images before and after applying S-PR? This would help illustrate S-PR’s impact on generation.
3. **Focused Experimentation in Table 2**:  The improvements mainly impact tasks needing precise top-down alignment (e.g., Insert Peg, Sort Shape). Given that most differences in Table 2 are within 1%, an experiment disabling both S-PR and down view could clarify other designs’ contributions. Additionally, statistical tests would help determine if small differences are meaningful. It will also be great if you can discuss the speed-accuracy trade-off applying these designs.
4. **Baseline and Failure Analysis**: Real-world experiments would benefit from a baseline comparison. A typical baseline would be an RVT/RVT2 without designs introduced in the paper. It would also help if the authors could conduct a failure analysis to highlight potential improvement areas. If challenges arose in implementing real-world baselines, an explanation would be valuable.
5. **Left View Performance Drop**: The results in Table 2 indicate a counterintuitive performance drop when incorporating a left view. A straightforward experiment replacing the right view with the left view (without adding extra views) could help isolate the cause. Based on this experiment, the authors can explore whether this drop is due to:
   - the presence of both left and right views introducing redundancy or conflicting information,
   - the left view alone, as opposed to the right view, negatively impacting performance,
   - or simply having an excess of views, which may complicate heatmap prediction?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732501631733

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Request for Further Discussion on Review Comments

comment:
Dear Reviewers,

Thank you for your previous feedback and time spent reviewing our submission. We truly appreciate the effort and insights provided. As we continue refining the work based on your comments, we believe further discussion or clarification on certain points would be beneficial. 

Your input would greatly help in enhancing the quality and clarity of the work. We look forward to hearing from you and would be grateful for your further guidance.  

Thank you once again for your time and assistance.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes S-RVT, an extension of RVT-1/2 for 3D object manipulation. It aims to address limitations in RVT, such as occlusion artifacts and limited action resolution, by introducing components like the Super Point Renderer and the Hierarchical Sampling Policy. Experiments are conducted in both simulation and real-world settings (as discussed in the rebuttal).

Strengths:

- Significant empirical improvements over RVT on the RLBench benchmark.
- Multiple experiments and ablations in sim and real.

Weaknesses:

- As Reviewer jBkg pointed out, it is unclear whether all the new components are necessary, given the small percentage improvement in performance. Consequently, it is uncertain if all components contribute to a statistically significant improvement, especially since the authors mention "normal performance fluctuation" on the benchmark in the rebuttal.
- The explanation regarding S-RVT reducing uncertainty appears confusing and lacks theoretical justification. According to the AC, the paper would benefit from emphasizing its empirical strengths rather than making auxiliary claims about uncertainty without substantial evidence.

Overall:
While the paper demonstrates clear empirical gains in addressing the problem of 3D object manipulation, the necessity of all the introduced components remains unclear. Additionally, the focus on claims about uncertainty lacks clarity and theoretical grounding. Therefore, I recommend rejection of the current version.