Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper presents Semantic-Geometric Representation (SGR), a perception module for robots that integrates rich semantic information from pre-trained 2D models with 3D spatial reasoning. The experiments demonstrate SGR's effectiveness in enabling robots to perform diverse simulated and real-world manipulation tasks, outperforming state-of-the-art methods in single-task and multi-task scenarios. The paper also highlights SGR's unique capability to generalize to novel semantic attributes.
review: Overall, the paper presented an interesting approach to integrate semantic and geometric representations, and claims that it stands as the only approach capable of generalizing to previously unseen semantic attributes. The paper is well written and the flow makes the paper easy to understand. However, there are a few points which should be addressed:

* The paper mentions that SGR achieves significantly higher performance when exposed to unseen colors or shapes. What happens in scenarios where a different shaped object requires a slightly different manipulation strategy? Does the method show signs of being able to adjust to different shapes in terms of the manipulation strategy?
* How is success measured? The authors mention that the real-robot experiments involve success out of 10 trials. However, the average success rate of SGR as compared to per act is only 11% higher, which is equivalent to one additional success. Because real-robot experiments are difficult to reproduce in general, and is prone to noise, such an increase isn't convincing.
* What is the runtime / computation resources needed by the method?
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper presents a compelling approach to integrating semantic and geometric representations, showing promising results in generalizing to previously unseen semantic attributes. The paper is well-written, and the flow enhances its readability. However, there are some aspects that need addressing, as mentioned earlier in the review section. Considering the interesting approach presented in the paper and its potential contributions to the field, I recommend a weak accept. However, addressing the above-mentioned points in a revised version would significantly strengthen the paper's overall quality.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper presents a robot perception framework that processes 2D images and point cloud information in two separate branches. The 2D sensor images are processed in a semantic branch equipped with pre-trained visual encoders so that the model could extract relevant semantic information in the scene. The raw point cloud is processed in the geometric branch with a point-based network so that the encoded representation in this branch captures the 3D structure of the scene. The outputs of the two branches are integrated into a fusion network to produce a global feature, which is then used to predict the next robot action.

The paper presents results on using the proposed framework in behavior cloning in 8 simulation environments and 8 real robot tasks. The results show that the proposed method outperforms various baselines that use different 2D representations or 3D representations.
review: **Strengths.**

* The paper tackles an important problem of constructing a universal visual representation for robot manipulation. The proposed method is well-motivated and clearly presented.
* The idea of making use of both 2D and 3D representation learning methods for the purpose of robot manipulation seems to be an under-explored area.
* The authors conducted extensive experiments and showed quantitative results including comparisons with baselines in both sim and real.

**Clarification questions with the proposed method.**

* What is the benefit of processing 2D images and 3D point clouds in two branches when one could design a framework that makes use of pre-trained vision models (eg. CLIP) while processing both 2D images and 3D point clouds at the same time? For example, one could take a PerAct-like approach to use a transformer-based architecture, process the 2D images with CLIP-based visual encoders and feed the encoded representations into the transformer as additional inputs.
* In the semantic branch, it seems that only RGB images from multiple views are used as inputs. Could the model benefit from processing the depth images from multiple views too?
* In the geometric branch, why is only the raw point cloud loaded, when the RGB colors of the points could be easily added for each point?

**Issues with experiments.** Although the authors performed a decent range of experiments on both simulation and real robot platforms, several problems exist in the details of those experiments and need to be addressed.

* The authors performed sim experiments on 8 RLBench tasks, but it seems that those tasks do not overlap the 18 RLBench tasks PerAct evaluated on. Could the authors pick a few tasks evaluated in PerAct and show at least some single-task quantitative results on them?
* In the "generalizing to novel attributes" experiments presented in Figure 4, the PerAct baseline is missing. Is there a particular reason results on this baseline are not presented? It would be nice if the authors could run and present results on PerAct in this table.
* It seems that the quantitative results of the real robot experiments are not very good. There indeed seems to be a gap between the performance of SGR and PerAct. However, out of the 8 tasks evaluated, SGR only outperforms PerAct in 4 of them.
* The paper lacks ablation experiments that study the extent to which parts of the method contribute to the overall performance. Some experiments that the authors could consider performing include (1) deleting one of the two branches in the network architecture while keeping other components of the framework; (2) switching out the pre-trained encoder with one that is trained from scratch.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, this is a nice paper with an interesting idea. However, because of the existing issues in the experiments section, my internal rating of this paper is borderline. I would be happy to adjust my rating if the authors make their experimental results more convincing.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This work presents Semantic-Geometric Representation (SGR), a universal perception module for robotics that leverages the rich semantic information of large-scale pre-trained 2D models and inherits the merits of 3D spatial reasoning. To do so, a network is designed to fuse the information from semantic and geometric features from 3D an 2D modalities respectively.

review: Strengths:
- Despite some clarifications missing as mentioned below, the paper overall is well-written and easy to follow

- The idea of fusing semantic and geometric information is reasonable. The pipeline is appropriately designed.

- The experiments in both simulation and real world have been conducted. The results are encouraging and outperforms the baselines by a large margin.


Weakness and questions:
- In experiments, the proposed method leverages 4 cameras for observation. However, for comparison methods (both 2D and 3D), it is not clearly mentioned whether the same multi-view observation has been provided for fair comparison. Also some of the methods were not pretrained using multi-view images and thus cannot take full advantage of the multi-view. To solely validate the proposed fusion technique, single-view setup comparison would be helpful.

- In intro, the statement of “SGR stands as the only approach capable of generalizing to previously unseen semantic attributes, including colors and shapes” seems over-claimed. In fact, multiple prior works [A-D] have tackled much more challenging category-level manipulation tasks in the real world. Discussion or comparison to these line of works is encouraged to be added into the paper.

- It is mentioned “robot proprioceptive information is incorporated”. However, this is not reflected in fig2. It is suggested to ensure the consistency.

- It would be more interesting to show side-by-side comparison to the baselines in the qualitative results or in the video demos.

Overall, the work is potentially suitable for publication but needs some more clarifications given the current state. Addressing the above concerns is highly encouraged.

References:
    A) Manuelli, L., Gao, W., Florence, P. and Tedrake, R., 2019, October. kpam: Keypoint affordances for category-level robotic manipulation. In The International Symposium of Robotics Research (pp. 132-157). Cham: Springer International Publishing.
    B) Wen, B., Lian, W., Bekris, K. and Schaal, S., 2022. You only demonstrate once: Category-level manipulation from single visual demonstration. RSS 2022
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, the work is potentially suitable for publication but needs some more clarifications given the current state. Addressing the above concerns is highly encouraged.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper proposed a way to fuse semantic information with RGB data and spatial information with depth data to generate better representation for robot manipulation.
review: The paper is very clear and follows a logical flow making it a pleasure to read. The idea of combining RGB information with depth is natural and the way this paper adopt is innovative.

The literature review is comprehensive. The reviewer especially likes the style of splitting into different category and at end of each highlighting the difference between this and aforementioned works.

The architecture image in Figure 2 is informative and clearly shows the framework.

The new representation (SGR) is then used with behavior cloning (BC) and compared with multiple SOTA and shows significant improvement in most of the tasks.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall I found the paper of very good quality, the idea is innovative and supported with both simulation & real robotics experiments, thusI recommend accepting the paper.
review_title: null

Meta-review: The paper introduces a new method on learning semantic-geometric representation for robot manipulation.

After the rebuttal, reviewers still have diverge opinions on the paper. The remaining concerns of the reviewers are
- Experimental evaluation of the proposed method.
- Some claim of the paper is too strong.

Overall, the paper introduces a novel method which improves over previous state-of-the-art approaches.
Therefore, I recommend acceptance for the paper.
Due to the remaining concerns of the reviewers, I recommend the paper as a poster presentation.
Decision: Accept (Poster)