Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper studies the topic of VLM feature distillation for robotics grasping. Specifically, it proposed a method to distill high-resolution dense 2D features to 3D grids using NeRF. The distilled 3D feature grids is further used in grasping pose matching, with an emphasis on  generalization to open-world objects. Experiments have been conducted on grasping and language-guided object manipulation in the real-world setting. Results demonstrate the effectiveness of the proposed distillation method and strong generalization of the features in grasp pose matching.
review: This paper is well-structured with clear descriptions of the method, experimental results, and limitations. The reviewer finds the supplementary document and the video helpful.

For example, the findings summarized in Table 1 are very reasonable, as CLIP feature contains robust semantic information and DINO feature contains more geometric information. The differences are well supported by the experiment results on the cluttered scene.

In section 3.2, the local geometry transformation is a very effective design for tasks including robotic grasping and manipulation. It might be good to mention the prior work that shares a similar design [NewRef1-2].

References
- Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics, RSS 2017.
- Learning 6-DOF Grasping Interaction via Deep Geometry-aware 3D Representations, ICRA 2018.


robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The reviewer does not find obvious caveats of the paper, besides the limitations (dynamic scenes and the latency) discussed. It is a great work that involves VLM feature, NeRF, and robotic learning all together. I highly recommend accepting the paper at the conference.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper focuses on enhancing robotic manipulation tasks by bridging the gap between 2D image features and the detailed understanding of 3D geometry required for such tasks. The authors leverage NeRF representations to get a dense 3D scene representation. CLIP features are distilled into the NeRF model, allowing it to render features in 3D. The grasping and manipulation demonstrations are connected to 3D feature distributions by observing the pose of the demonstrated grasp and storing the nearby CLIP features and their relative 3D locations. With features from the vision-language model CLIP, the authors demonstrate the capability to designate novel objects for manipulation via free-text natural language. The results show that the model can generalize to novel object instances effectively.
review: **Strengths:**
- The paper introduces a novel approach to derive dense 3D features using a pretrained vision-language model CLIP with NeRF-like representation.
- The paper presents a novel algorithm for integrating both language features as well as stored visual features toward grasping and placing novel objects.
- The paper is well-written and includes supplementary material with details on the distillation procedure and extraction of dense features from CLIP along with the pseudo-code. 
- Real-world experiments and results are shown to substantiate the effectiveness of the proposed method.

**Weakness:**
- It is unclear what the paper defines as novel objects. Are these the objects from the same category shown in the demonstration? Or is the work generalizable to a new category of objects? To what degree can the language labels for the objects vary? 
- The paper clearly states that the computation is a limitation. It will be beneficial to see the actual time it takes during the inference time upon the query.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I believe the work would be valuable to the community. 

**Post-rebuttal update:** The authors have answered my questions. I appreciate the improvement in the quality of the manuscript. My recommendation to accept this paper remains.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper employs Distilled Feature Fields (DFF) for robotic manipulation. It utilizes DFF as a robust, generalizable, and semantic-aware scene representation. The authors demonstrate this representation's efficacy in tasks such as few-shot manipulation and language-conditioned manipulation.
review: Strengths:
1. The concept of utilizing DFF as a visual representation within robotics appears logical. It's both 3D and semantic-aware, a feat current representations can achieve separately, but not in conjunction.
2. The task design is intelligent, maximizing the benefits of DFFs. The action space is a 6D transformation; with the proposed action reparameterization, the action is also mapped into the 3D feature space, creating homogeneity between the representations of both observation and action.
3. The experimental results successfully demonstrate the proposed method's efficacy.
4. The exposition is exceptionally clear and easy to follow.

Weaknesses:
1. The task design, while inventive, is also somewhat limited. How could this representation be generalized to accommodate closed-loop manipulations?
2. Have alternatives to action reparameterization been considered? For instance, could the query points be set as the midpoints between the two grippers post-transformation?
3. On line 186, could you provide examples of objects with high feature similarities to the demonstrations? Under what circumstances would CLIP outperform DINO?
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Although the idea is interesting and the experiments are solid, this paper is more about applying CV techniques to robotic tasks with some modification (to make it faster and more available etc.). So I vote for weak accept.

After rebuttal:
I still think the main method is still built on top of *Neural Feature Fusion Fields*, showing interesting applications to robotics. So I keep my vote to weak accept.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes to obtain high-resolution feature fields via InstantNGP and other engineering tricks to improve training time, and then use such feature field to construct 6-DOF poses with iterative optimisation by retrieving the closest poses presented in the demonstrations. The proposed framework is evaluated through real-world grasping tasks, showing good generalisation in novel objects. 
review: The paper is clearly in a good quality, with well-motivated and well-designed framework. However, I found some explanations in the method section a bit confusing, and some other improvements could be done for the experiments.

1.	I found no problem understanding the framework design up till L112, then it starts to get a bit confusing. I understand the framework starts with constructing a feature neural field with CLIP features, and adding some engineering tricks such as removing the global token and using InstantNGP to improve the training speed and construct high-resolution local information. But later in Section 3.2, I failed to understand i) why and what are the query points; ii) are we learning per-feature 6DOF pose via a learnable map? iii) what and why we need the encoding of a task, and why average the features over the demos represents the encoding for a task? I thought the features should represent task-agnostic semantic information, so we may do object-level grasping?
2.	As using a similar strategy in Neural Descriptor Fields, do we have a prior knowledge knowing that the object we want to grasp has a similar object in the query demo? If so, I believe we need do a proper ablation on how the performance would be changed based on different number/variations of the query demos.
3.	Generalisation over text prompts. Though the paper presents a good generalisation on object categories. I am wondering how the generalisation is on language prompts? Or simply the language prompts her are simply object category names? 
4.	Other design choices. As since the proposed network is mainly based on the pre-trained vision-language features. A much simpler and cleaner framework would be just to learn how the transformation from the learned feature map to grasping point based on a text prompt, without using demonstrations as queries, which I believe is a strong baseline. 
 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Good method, but needs some extra clarification. Good results on robotics and video.
review_title: null

Meta-review: The paper introduces a novel method for robot manipulation by distilling image features from a vision-language model to 3D scenes. The method enables 6D grasping and language-guided grasping of unseen objects using few-shot demonstrations.

The reviewers achieve a consensus on accepting the paper with high scores. Therefore, I recommend to accept the paper as an oral presentation.
Decision: Accept (Oral)