Neural Attention Field: Emerging Point Relevance in 3D Scenes for One-Shot Dexterous Grasping

Authors: qianxu  wang
congyue  deng
tyler  lum
yuanpei  chen
yaodong  yang
jeannette  bohg
yixin  zhu
leonidas  guibas
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-05-20 05:35:54

Abstract: One-shot transfer of dexterous grasps to novel scenes with object and context variations has been a challenging problem. While distilled feature fields from large vision models have enabled semantic correspondences across 3D scenes, their features are point-based and restricted to object surfaces, limiting their capability of modeling complex semantic feature distributions for hand-object interactions. In this work, we propose the *neural attention field* for representing semantic-aware dense feature fields in the 3D space by modeling inter-point relevance instead of individual point features. Core to it is a transformer decoder that computes the cross-attention between any 3D query point with all the scene points, and provides the query point feature with an attention-based aggregation. We further propose a self-supervised framework for training the transformer decoder from only a few 3D pointclouds without hand demonstrations. Post-training, the attention field can be applied to novel scenes for semantics-aware dexterous grasping from one-shot demonstration. Experiments show that our method provides better optimization landscapes by encouraging the end-effector to focus on task-relevant scene regions, resulting in significant improvements in success rates on real robots compared with the feature-field-based methods.

Main paper: 
 introduction: 
 Generalization has been a long-studied problem in robot learning from demonstrations, with oneshot generalization being one of the most challenging settings. Great progress has been made with recent advances in large vision models which provide visual features with cross-scene semantic correspondences [1, 2] . Given a pre-trained vision model on 2D images, a common strategy is to distill its features into 3D and match the end-effector position in the target scene to the demonstration by minimizing their feature differences [3, 4, 5, 6] .However, the distilled feature fields can only model point correspondences on the object surfaces. While this is sufficient for simple manipulations with a parallel gripper whose interactions with objects are primarily based on individual contact points, this is insufficient for more complex endeffectors. For example, the hand-object interactions required for dexterous grasping are no longer point-based or limited to the object surface, but instead spatially distributed or even possibly off the object surface. Prior dexterous grasping method with feature fields [6] computes query features in the free space as a sum of closeby scene-point features weighted by distances (Fig. 1 top). While it achieves reliable results in single-object scenes, it is prone to failure when there are distractors in the vicinity of the target object due to its naive feature aggregation based on spatial proximity.To design a better feature representation, we start with the intuition that, when humans interact with a scene, an intention is formed even before their hands reach the target object. Driven by this intention, they have their attention focused on certain scene regions with higher task relevance instead of looking into the details of every part of the scene. Following this intuition, we propose the neural attention field for representing semantic-aware dense feature fields in the 3D space by modeling inter-point relevance instead of individual point features (Fig. 1 ). More concretely, we replace the distance-weighted feature interpolation in [6] with a transformer decoder that computes the cross-attention between any 3D query point with all the scene points, and aggregates the scene features to the query point based on the attention rather than spatial proximity (Fig. 1 We further propose a self-supervised framework for training the transformer decoder from only a few 3D scenes without hand demonstrations (for example, the 4 pointclouds in Fig. 2 left ). Here we take the scene points as queries and learn a self-attention on the scene pointcloud. Each scene point itself carries a raw feature, which is updated through the transformer aggregation within each scene. Our key intuition is that, point correspondences can be computed across scenes via feature similarities, and they should stay identical for both features before and after applying the transformer. This leads to a contrastive feature loss for optimizing the transformer decoder parameters.Post-training, the transformer decoder can be directly applied to novel scenes for querying the feature of any 3D point in the near-object space. We can then use it to transfer one-shot demonstration of dexterous grasps from a source scene to a target scene following the same strategy as neural feature fields [7, 6] , where we sample query points on the hand surface and minimize their feature differences between the two scenes.Experimentally, we show that our neural attention field induces better optimization landscapes by encouraging the end-effector to focus on task-related scene regions and omitting distractions, enabling robust one-shot dexterous grasping in various challenging scenarios such as grasping objects in complex scene contexts, grasp transfer between objects of varying shapes but similar semantics, or functional grasping of tools. Real-robot results demonstrate our significant improvements in success rates compared with the feature-field-based methods.To summarize, our key contributions are: (i) We propose the neural attention field for representing semantic-aware dense feature fields in the 3D space by modeling inter-point relevance instead of individual point features. (ii) We propose a self-supervised framework for training the attention field with a few scenes. (iii) Our attention field enables semantics-aware dexterous grasping from one-shot demonstration and encourages the end-effector to focus on task-related scene regions. (iv) Real-robot experiments show our strong generalization under different scene variations. 
related work: 
 Neural field for manipulation Point correspondences induced by features facilitate the transfer of manipulation policies across diverse objects. In contrast to methods based on key points [8, 9, 10] , recent efforts focus on developing dense feature fields around 3D scenes through implicit representations [7, 11, 12, 13, 14, 15, 16, 17, 18, 19] . The integration of large vision models have propelled research towards leveraging distilled features fields for enabling few-shot or one-shot learning of 6-DOF grasps [3] , sequential actions [4] , and language-guided manipulations [5] . These efforts predominantly focus on simple manipulations using parallel grippers. Closest to us is [6] which uses distilled feature fields to transfer dexterous grasps across objects. However, as mentioned, it computes query features in the free space as a sum of closeby scene-point features weighted by distances, making it unstable to distractors near the target object.Distilled feature field With the recent advances of large vision models on 2D images, an effective way of acquiring semantics-aware feature fields for 3D scenes is to first obtain 2D image features and then distill them to 3D. [20, 21, 22] lifts of semantic information from 2D segmentation networks to 3D, enabling 3D segmentations with language embeddings. [23, 24] delve into integrating pixelaligned image features from models like LSeg or DINO [1] into 3D NeRFs, highlighting their impact on manipulating 3D geometry. Further, [25] and [26] explore distilling non-pixel-aligned image features, such as those from CLIP [27] , into 3D scenes without the need for fine-tuning.Dexterous grasping Dexterous manipulation, central to advanced robotic applications, necessitates nuanced understanding and control, akin to human-like grasping capabilities [28, 29, 30, 31, 32, 33, 34] . Analytical approaches [35, 29, 30] focus on direct modeling of hand and object dynamics. Recent learning-based methods have introduced state [36, 37, 38, 39, 40] and vision-based strategies [41, 42, 43, 44, 45, 46] , targeting realistic scene comprehensions. However, most of these methods depend on large demonstration datasets for training. Notably, [47] proposes a grasp synthesis algorithm focusing on geometric and physical constraints for functional grasping that generalizes within similar object shapes using minimal demonstrations, 3 Method 
preliminaries: 
 Demonstration transfer with feature field Given a 3D scene pointcloud X, a feature field around the scene is a function f (•, X) : R 3 → R C that maps every query point q ∈ R 3 in the 3D space to a C-dimension feature f ∈ R C . For an end-effector placed in the feature field with pose parameters β, one can sample a set of query points Q|β ∈ R Q×3 on the end-effector surface conditioned on its parameters and obtain their features f (Q|β, X) ∈ R Q×C , which can be viewed as the feature for the entire end-effector w.r.t.β. Given a demonstration with a source pointcloud X and end-effector parameters β, one can transfer it to a target scene X by minimizing an energy function induced by the two feature fieldsEQUATIONIntuitively, this minimizes the feature differences of the end-effector positions in the scene, indicating that the end-effector should be applied to the locations with similar semantic meanings. More elaborations can be found in [7, 6] .Distilled feature field An effective approach for acquiring a semantic-aware 3D feature field is to first extract semantic features on 2D images with large vision models [1, 2] and then distill them to 3D. A line of works [26, 5, 19] directly reconstruct continuous implicit feature fields alongside NeRF representations. If given a 3D pointcloud and its point correspondences to the 2D image pixels derived from RGBD sensor inputs, one can also directly back project and aggregate the image features on each pixel to the 3D points. The discrete pointcloud features can be further propagated into the surrounding space via spatial interpolation [6] . For a pointcloud X = {x i } with per-point features F = {f i }, the feature z at a query point q ∈ R 3 is interpolated from the pointcloud features with inverse distance weighting:EQUATION)Figure 2 : Self-supervised training for the transformer decoder. Left: Given a few scenes, we first select the corresponding keypoints computing cyclic mutual nearest neighbors (MNN) based on their feature similarities. Right: We take the selected keypoints as queries for each scene and enforce the features before and after the D θ -aggregation to induce the same keypoint correspondences across scenes. Specifically, we apply an InfoNCE loss to preserve the orders of the keypoints given the permutation equivariance of transformers.While this spatial interpolation propagates the features from the discrete surface points to the nearsurface space smoothly, this operation lacks semantic awareness and blurs the inter-scene feature correspondences as the query point goes off the object surfaces, making it unstable to transfer endeffector poses from the source scene to the target. For example, when transferring a grasp from a source scene containing only a toy animal to a target scene in which the toy animal is surrounded by several irrelevant objects, we aim to ensure that the features focus primarily on the toy animal itself and are less affected by the other objects despite their spatial adjacency (Fig. 1 top).To better represent dense and semantic-aware features in the near-object space, we introduce the neural attention field (Fig. 1 ) that predicts a cross-attention between the query point q and the scene points X to replace the inverse distance weighting w i in Eq. 2. More concretely, it is realized with a lightweight transformer decoder network D θ (key, query, value) with its keys, query, and value being the scene points X, the query point q, and the scene-point features F respectively:EQUATIONwhich provides an aggregated feature f D ∈ R C for any query point q ∈ R 3 according to the learned attention maps. As the 3D point coordinates X and q carry limited information, we also append the per-point features F to X and the spatially aggregated feature f in Eq. 2 to q, resulting in keys [X, F] and query [q, f ]. Detailed architecture of the transformer decoder can be found in the appendix.As there is no straightforward way to annotate ground-truth features or attention maps, we propose a self-supervised framework for training the transformer decoder D θ in Eq. 3 on a few 3D scenes related to the task. In most of our experiments, we use 4 scene pointclouds to train D θ and then apply it to a set of tasks on transferring dexterous hand poses. We also experiment with different training and inference setups including different numbers of training scenes and different content similarities between the training and inference scenes (Sec. 4.2).We consider directly using the scene points x i ∈ X as queries and compute their D θ -aggregated features via a self-attention:EQUATIONWe denote F D = {f D,i } i the collections of aggregated features for all points in X. Conceptually, this can also be viewed as placing a virtual query at each point in the scene pointclouds.Given a few different scene pointclouds X (1) , • • • , X (I) with raw point features F (1) , • • • , F (I) , we can compute their aggregated point features F(1)EQUATIOND respectively. Both F and F D induce point correspondences across scenes via nearest neighbor retrieval w.r.t.feature similarities. Based on this, we introduce a self-supervised loss for training D θ by enforcing that the induced correspondences from F and F D should be identical between any pair of scenes (Fig. 2 ).More concretely, we first compute a set of keypoints K (1) , • • • , K (I) ∈ R K×3 for each scene based on the raw point features F (i) , with each keypoint indicating a correspondence across all scenes (e.g.the point on the monkey's left hand in all scenes). We then update the features of these keypoints via Eq. 4, acquiring their D θ -aggregated features F (i) D . Finally, as transformer networks are permutation equivariant, we enforce the keypoint orders to stay consistent before and after applying D θ with a contrastive loss onF (i) D .Cyclic MNN keypoint selection (Fig. 2 left ) We compute the keypoints K (1) , • • • , K (I) such that each K (i) is a subset of the points in X (i) and any pair of K (i) , K (j) are corresponding points across the two scenes according to the similarity of their raw features F (i) , F (j) . We introduce a cyclic mutual nearest neighbor (MNN) strategy for this corresponding keypoints selection, as illustrated in Fig. 2 left. Specifically, we randomly assign an order 1, • • • , I to the scenes. For each adjacent scene pair X (i) and X (i+1) (or X (I) and X (0) ), any keypoint in K (i) must fall into the k-nearest neighbor of the corresponding keypoint in K (i+1) according to their feature differences. We reject any point that violates this constraint for any i.Self-consistency feature loss (Fig. 2 right) The keypoint features are updated with D θ via:EQUATIONWe then enforce that, after this D θ update, the feature-induced keypoint correspondences stay unchanged. Specifically, we employ an InfoNCE [48] loss as illustrated in Fig. 2 right:L = i,j∈[I] L ij , L ij = K k=1-log exp(sim((fEQUATIONwheref (i) D,k is the k-th entry of F (i)D , namely the output feature of D θ of the k-th keypoint in scene i, and τ is a temperature parameter. This loss minimizes the updated feature differences of matched keypoints and maximizes that of unmatched keypoints. We optimize for the network parameters θ in D θ w.r.t. this loss.Following [6] , we consider the dexterous hand parameters β of the hand pose and joint rotations. Given a demonstration of hand parameters β in the source scene X (with F), we optimize for β in the target scene X (with F) using the neural attention field. We start by sampling Q points on the hand surfaces in both the source and target hands, generating query point sets Q| β, Q|β ∈ R Q×3 conditioned on the hand parameters. The sampling is done in the canonical hand pose and is identical between the source and target hand surfaces. We follow the sampling strategy in [6] which has a higher sampling density on the fingers than on the palm.We then obtain the features for the end-effector query points by passing them into the trained transformer decoder D θ and acquire their features D θ (Q| β, X, F), D θ (Q|β, X, F) ∈ R Q×C . With these features, we optimize for the end-effector pose β in the target scene while keeping all network parameters in D θ frozen. The objective is to minimize the feature differences between the demonstration and target hand poses via an l1 loss, formulated as an energy function E(β) w.r.t. β: We also integrate the physical viability functions from [49, 6] , including the inter-penetration and self-penetration energy functions:EQUATIONEQUATIONwhere δ is a threshold parameter, and a pose constraint E pose (β) that penalizes out-of-limit hand pose as in [49] . The overall optimization combines these terms:EQUATIONThe weighting between feature energy loss and the physical constraints are set to be λ pen = 10 -1 , λ spen = 10 -2 , λ pose = 10 -2 , which is the same as [6] . In all our experiments, we randomly initialize the hand position and joint parameters and optimize for via gradient descent. 
experiments: 
 Training and inference In all our experiments, we pre-train the transformer decoder network D θ with a few scene pointclouds (2-4 scenes without demonstrations). We then provide a 1-shot demonstration with a dexterous hand pose in a 3D scene and directly transfer it to different scenes. This demonstration scene pointcloud doesn't have to be included in the pre-training scenes. We experiment with different pre-training and demonstration scene setups as listed in the tables. Each evaluation is done 10 times with scene variations in the object poses (for both single-object and multi-object scenes).We conduct all our experiments in the real world with a Shadow Dexterous Hand of 24 DoF. We restrict the 2 DoF at the wrist, focusing the optimization on the remaining 22 DoF. The hand is mounted on a UR10e arm, introducing 6 additional DoF. Precautions are taken to prevent the dexterous hand from contacting the table surface directly, and the range of motion is limited by the arm's elbow. For RGBD scans, we use four pre-calibrated Femto Bolt sensors at each corner of the table. Post-processing is applied to the captured scans to remove background elements: In single-object experiments, the object's pointcloud is segmented using SAM [50] . For multi-object scenarios, we apply physical constraints to limit the experimental area to the tabletop and employ RANSAC [51] to exclude the table surface from the analysis.Baselines We mainly compare our method to the feature-field-based methods, including a vanilla DFF with the feature acquisition strategy in [25] , and SparseDFF [6] which introduces an additional multiview feature refinement mechanism for better feature qualities. ; the featureinduced energy fields for our method and SparseDFF [6] . We only visualize the 2D sections of the 3D energy fields and yellow indicates lower energy values and thus higher feature similarities. Our method shows more concentrated low-energy regions around the target grasping positions.Figure 5 : Real-robot results. From left to right, we show our results of grasping objects in scene contexts with distraction, grasp transfer between objects with similar semantics but shape variations, and functional grasping of tools.Object in scene contexts A major advantage of our neural attention field is that the cross-attention mechanism between end-effector queries and the scene points encourages the focus on the scene points of higher relevance to the task indicated by the demonstration. This is particularly beneficial when there are task-irrelevant objects in the scene and we want to accurately locate specific regions on the target object and avoid distractions. Tab. 1 shows our results for grasping the target objects from multi-object scene contexts with only a one-shot grasping demonstration on a single object. Our method has consistently higher success rates than the previous feature-field-based methods. Real-robot executions are shown in Fig. 5 left. Fig. 4 visualizes the energy fields of our method and a vanilla distilled feature field method SparseDFF [6] . We can see that our attention field induces energy functions with more concentrated minimas centered around the target grasping positions (such as the monkey arm in the top left example), while the distilled feature field induces more irregular energy functions in the space, resulting worse optimization landscapes.Cross-object grasp transfer Tab. 2 shows our results on dexterous grasp transfer between objects of varying shapes but similar semantics. In addition to filtering out the irrelevant content in multiobject scenes, our attention field also brings benefits even in single-object scenes by encouraging the transformer queries to focus on task-related object regions. For example, the necks of the horse and the giraffe are of very different shapes and are close to the back where the grasp is applied, but semantically they are irrelevant to the task. Simply computing the features according to spatial relations as in the feature fields can be misleading, while our attention field overcomes this issue. Real-robot executions are shown in Fig. 5 middle .  
conclusions: 
 To better model hand-object interactions, we propose the neural attention field, which represents semantic features in the 3D space via modeling inter-point relevance instead of individual point features. It enables robust and semantic-aware transfer of dexterous grasps across scenes by encouraging the end-effector to focus on scene regions with higher task relevance.Limitations and future work While we focus on dexterous grasps in this work, the neural attention field and its subsequent feature-based end-effector optimization can be applied to other dexterous manipulations beyond grasping. Extension to other manipulations with action sequences would be an interesting and useful future direction.scenes in all the ablation settings are identical. In Fig. 6 , we evaluate the average distance between the hand-surface query points and the target region (monkey arm) in the scene, as a rough metric indicating how close the hand reaches the target post-optimization. The target region is manually annotated with MeshLab.Quantitative results Fig. 9 shows the qualitative results on grasping the monkey arm with the same demonstration but different pre-training setups for the transformer decoder. The hand optimization trajectories are shown in green.  
 
 Appendix: 8th Conference on Robot Learning (CoRL 2024), Munich, Germany. 


Decision: The paper introduces a new method for one-shot grasp transfer for dexterous grasping. A key component is the introduced neural attention field to capture dependencies between 3D points by using an attention mechanism. Grasps are optimized using features from the  neural attention field.

Strengths

- The reviewers agree on that the neural attention field for one-shot grasping is new.
- The paper is well written and easy to read.

Weaknesses

The reviewers have raised several concerns about the paper.

- Limited novelty of the paper regarding the difference between the proposed method and [6].
- Limited experimental evaluations. No comparison with other state-of-the-art baselines. Can object segmentation be used with SparseDFF?
- Clarification of certain experimental setup.

Post-Rebuttal
- The authors provided their answers to address the concerns from the reviewers. However, the reviewers did not participant in the discussion with several messages from the AC.
- The AC checked the rebuttal and agrees with the authors. The proposed method has enough novelty to address the one-shot grasping problem for dexterous hands. It shows improvement over SparseDFF [6].

Reviews: review: title: title: Significant Improvement Needed in Experimental Quality and Contributions
paper_summary: paper_summary: This work introduces a neural attention field using a transformer decoder to model inter-point relevance, enhancing feature representation and focusing on task-relevant regions. They uses a self-supervised framework to train the model on a few 3D scenes and applied to new scenes for grasping purposes from one-shot demonstrations.
main_review: main_review: Strength:
- Success utilization of a transformer to model inter-point relevance.
- Successfully demonstrated a self-supervised training framework for the transformer decoder that requires only a few 3D scenes and reducing dependency on large annotated datasets.
- Demonstrates real-robot grasping with one-shot demonstration.

Weakness:
- The paper’s contribution is relatively limited, primarily adding an extra layer to an existing method. The reviewer expected more comprehensive application results beyond simple grasping experiments on just a few objects.
- The lack of state-of-the-art baselines for comparison makes the current experiment resemble an ablation test, merely showing the improvement of adding the neural attention field on top of DFF. It fails to compare with other widely used object-centric or semantic-aware frameworks for visual manipulation.
- The training and evaluation scope is too narrow to draw meaningful conclusions about the effectiveness and generalizability of the proposed method.
questions: questions: From an applied robotics perspective, I am curious how the authors envision this method affecting downstream applications. Specifically, in a problem setting where the model is trained on grasping one or a few objects and then applied to grasp the same or similar objects in more complex environments. Although fast adaptation is beneficial, the applicability seems limited because new objects with new semantics would still require new demonstrations for re-training.
limitations: limitations: 1
review_summary: review_summary: Recommend a rejection unless followup draft demonstrates a significant improvement in experimental quality and contributions
title: title: Concerns about the Real-World Experiments and differences to SparseDFF
paper_summary: paper_summary: This paper presents a neural attention field method for one-shot transfer of dexterous grasps. The work builds on previous work [6] by shifting the focus to task-relevant scene points instead of spatial proximity. The work is evaluated in real-world scenarios using a shadow hand.
main_review: main_review: The paper is very well written, and the figures nicely illustrate the method.

My major concern with this work is that it is very similar to [6]. It would be good if the authors could highlight the differences more. The authors write in the introduction that SparseDFF “is prone to failure when there are distractors.” Couldn’t this be addressed by adding a segmentation step method and removing the distractors in the scene? It would be good to see some experiments on this.

Other than that, it seems like the objects in the real-world experiment “Monkey to horse context” are placed much more apart than for the baselines. This does not really support the claim that SparseDFF “is prone to failure when there are distractors” in the scene. I would like to see some plots about the spatial distribution of the objects. SparseDDF also shows some experiments when grasping the Monkey in clutter and reports 100% success rate. Why the drop if grasping is restricted to the arm? Furthermore, when grasping the mallet, it seems that in the experiments it was placed in a similar manner but rotated for the baselines. Why does SparseDFF perform worse here if there are not that many distractors?

## Minor:

-	Line 18: Desterous Grasping
-	Line 162: missing )
-	Equation 6: additional (
-	The rotation animation in the video at 1:58 is unnecessary and confusing
-	It would be good to have labels for the energy field in Fig. 3 and Fig. 4 to help the reader to understand what the colors mean. The video narration mentions this, but it would be good to have a label.
questions: questions: 1.) Why not simply add a segmentation step to [6]? This would filter out the distractors.

2.) The clutter in the real-world experiment seems to be biased towards this method. It seems like objects are placed more apart in the “Monkey to horse context.” E.g., the power drill in the video at 2:45 is. Can you comment on that?

3.) Actually, can you redo the “Monkey to horse context” experiment and place the objects close together as you did for SparseDFF?

4.) Similarly, the rubber mallet in 2:45 seems to be rotated less than for the baselines. Why does SparseDFF only perform 50% if there are few distractors?

5.) Can you elaborate more on the task-relevance of this feature?
limitations: limitations: 2
review_summary: review_summary: Dexterous grasping is challenging, but I have some concerns about the experiment setup, as objects are placed much further apart, questioning the claim that this method is more robust to distractors. I am looking forward to the rebuttal.
title: title: Neural Attention Field: Emerging Point Relevance in 3D Scenes for One-Shot Dexterous Grasping
paper_summary: paper_summary: The paper introduces a neural attention field based on an attention decoder to create semantic-aware features for one-shot dexterous grasping transfer. This approach models inter-point relevance instead of individual point features, enhancing the ability to generalize grasps across different 3D scenes. The authors propose a self-supervised framework for training the attention decoder, eliminating the need for extensive hand-labeled datasets. Experimental results demonstrate that this method significantly improves performance over baseline feature-field-based methods, showing better optimization landscapes and higher success rates in grasping tasks with real robots.
main_review: main_review: Strengths
- Semantic-Aware Features: The neural attention field enables free space pose optimization for hand grasping by creating semantic-aware features. This allows the model to focus on task-relevant regions of the scene, improving the accuracy and robustness of grasping tasks.
- Self-Supervised Learning: The proposed self-supervised learning framework facilitates efficient learning from a few samples, reducing the need for extensive labeled datasets. This approach enhances the practicality and scalability of the method.
- Improved Performance: Experimental results demonstrate that the method significantly outperforms baseline feature-field-based methods. The neural attention field provides better optimization landscapes, leading to higher success rates in dexterous grasping tasks.
- Generalization: The method shows strong generalization capabilities, successfully transferring grasps to new objects and scenes with varying contexts. This robustness is critical for real-world applications where the environment can be highly dynamic and unpredictable.


Weaknesses
- Lack of Detail on Design Choices: The paper lacks detailed explanations on certain design choices and implementation aspects, such as the method for pairing key points across scenes (lines 151-152) and the definition of Q in equation 8 .
- Potentially Costly Optimization Runtime: The optimization process, especially with the transformer-based approach, could be computationally expensive, leading to longer runtimes and potentially limiting its practicality for real-time applications.
- Expensive Data Labeling Efforts: Despite the self-supervised learning framework, significant data labeling efforts may still be required, particularly for soft or similar objects, which could limit scalability.
- Clarity and Readability: The paper could improve in clarity, especially in explaining the calculation of scene features (Appendix Figure A) and the impact of query point initialization. Enhancing these explanations would aid reader understanding.
- Performance in Occluded Scenes: The paper does not thoroughly evaluate the method's performance in scenes with occlusions, which could significantly impact the effectiveness of the proposed approach. Further experiments are needed to assess robustness in such scenarios.
questions: questions: Questions for Rebuttal
- How does this method transfer to scenes with unseen objects as distractors?
- How is the pairing of key points across scenes achieved (lines 151-152)? Does this method require a curated dataset with cross-scene point pairs?
- What is $\bar{\mathbb{Q}}$ in equation 8 $E_{pen}$?
- Does the initialization of query points (Q) affect the quality of the final grasp pose?
- How are the scene features calculated, as shown in Appendix Figure A?
- How does the method perform in scenes with occlusions?
limitations: limitations: 3
review_summary: review_summary: The paper introduces a neural attention field for one-shot dexterous grasping, using a transformer decoder to enhance semantic understanding. The self-supervised learning approach enables efficient training from few samples and shows significant performance improvements over baselines. However, the paper lacks detail on key design choices and does discuss potential costly runtime and data collection process. Despite these issues, the contributions and results justify acceptance.