Neural Attention Field: Emerging Point Relevance in 3D Scenes for One-Shot Dexterous Grasping
Authors: Qianxu Wang
Congyue Deng
Tyler Ga Wei Lum
Yuanpei Chen
Yaodong Yang
Jeannette Bohg
Yixin Zhu
Leonidas Guibas
Venue: CoRL 2024 (2024)
Keywords: Desterous Grasping
One-Shot Manipulation
Distilled Feature Field
Neural Implicit Field
Self-Supervised Learning

Abstract: One-shot transfer of dexterous grasps to novel scenes with object and context variations has been a challenging problem. While distilled feature fields from large vision models have enabled semantic correspondences across 3D scenes, their features are point-based and restricted to object surfaces, limiting their capability of modeling complex semantic feature distributions for hand-object interactions. In this work, we propose the *neural attention field* for representing semantic-aware dense feature fields in the 3D space by modeling inter-point relevance instead of individual point features. Core to it is a transformer decoder that computes the cross-attention between any 3D query point with all the scene points, and provides the query point feature with an attention-based aggregation. We further propose a self-supervised framework for training the transformer decoder from only a few 3D pointclouds without hand demonstrations. Post-training, the attention field can be applied to novel scenes for semantics-aware dexterous grasping from one-shot demonstration. Experiments show that our method provides better optimization landscapes by encouraging the end-effector to focus on task-relevant scene regions, resulting in significant improvements in success rates on real robots compared with the feature-field-based methods.

Neural Attention Field: Emerging Point Relevance in 3D Scenes for One-Shot Dexterous Grasping
Qianxu Wang Peking University
Congyue Deng Stanford University
Tyler Lum Stanford University
Yuanpei Chen Peking University
Yaodong Yang Peking University
Jeannette Bohg Stanford University
Yixin Zhu Peking University
Leonidas Guibas Stanford University
L2 dist.
Feature fiold
What should focus on?
festure
scene points
Attention field (ours)
transformer decoder Us
query
featuro
point
One One-shot demonistration
Genaralizat shapas and scene contexts
points
Figure 1: Given a one-shot demonstration of dexterous grasp, we want to generalize to novel scene variations with relevant semanties. To better model the complex semantic feature distributions in hand-object interactions, we propose the neural attention field, which represents a semantic- aware dense feature field by modeling inter-point relevance instead of individual point features. encourages the end-effector to focus on scene regions with higher task relevance instead of spatial proximity, resulting in robust and semantic-aware transfer of dexterous grasps across scenes.
Abstract: One-shot transfer of dexterous grasps to novel scenes with object and context variations has been challenging problem. While distilled feature fields from large vision models have enabled semantic correspondences across 3D scenes, their features are point-based and restricted to object surfaces, limit- ing their capability of modeling complex semantic feature distributions for hand- object interactions. In this work, we propose the neural attention field for rep- resenting semantic-aware dense feature fields in the 3D space by modeling inter- point relevance instead of individual point features. Core it is a transformer decoder that computes the cross-ates coss-aatention between any 3D query point with all the scene points, and provides the query point feature with an attention- based aggrega- tion. We further propose a self-supervised framework for training the transformer decoder from only a few 3D pointelouds without hand demonstrations. Post- training, the attention field can be applied to novel scenes for semantics-awarc dexterous grasping from one-shot demonstration. Experiments show that our method provides better optimization landscapes by encouraging the end-effector to focus on task-relevant scene regions, resulting in significant improvements in success rates on real robots compared with the feature-field-bat based methods.
Keywords: Dexterous Grasping, One-Shot Manipulation, Distilled Feature Field, Neural Implicit Field, Self-Supervised Learning
1 Introduction
Generalization has been long-studies problem robot learning from demonstrations, with one- shot generalization being one of the most challenging settings. Great progress has been made with recent advances large vision models which provide visual features with cross-scene semantic correspondences [1, 2]. Given a pre-trained vision model on 2D images, common strategy is to
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

distill its features into 3D and match the end-effector position in the target scene to the demonstration by minimizing their feature differences [3, 4, 5, 6].
However, the distilled feature fields can only model point correspondences on the object surfaces. While this is sufficient for simple manipulations with parallel gripper whose interactions with objects are primarily based on individual contact points, this is insufficient for more complex end- effectors. For example, the hand-object interactions required for dexterous grasping are no longer point-based or limited to the object surface, but instead spatially distributed or even possibly off the object surface. Prior dexterous grasping method with feature fields [6] computes query features in the free space as a sum of closeby scene-point features weighted by distances (Fig. top). While it achieves reliable results in single-object scenes, is prone failure when there are distractors in the vicinity of the target object due its naive feature aggregation based spatial proximity.
To design better feature representation, we start with the intuition that, when humans interact with a scene, an intention is formed even before their hands reach the target object. Driven by this intention, they have their attention focused on certain scene regions with higher task relevance instead of looking into the details of every part of the scene. Following this intuition, we propose the neural attention field for representing semantic-aware dense feature fields in the 3D space by modeling inter-point relevance instead of individual point features (Fig. 1). More concretely, feature interpolation in [6] with a transformer decoder that computes
replace the distance-weighte the cross cross-attentin attention between any 3D query point with all the scene points, and aggregates the scene features to the query point based on the attention rather than spatial proximity (Fig. bottom).
We further propose a self-supervised framework for training the transformer decoder from only a few 3D scenes without hand demonstrations (for example, the pointclouds in Fig. 2 left). Here we take the scene points as queries and learn self-attention on the scene pointcloud. Each scene point itself carries raw feature, which is updated through the transformer aggregation within each scene. Our key intuition is that, point correspondences can be computed across scenes via feature similarities, and they should stay identical for both features before and after applying the transformer. This leads to a contrastive feature loss for optimizing the transformer decoder parameters.
Post-training, the transformer decoder can be directly applied to novel scenes for querying the fea- ture of any 3D point in the near-object space. We can then use it to transfer one-shot demonstration of dexterous grasps from source scene to a target scene following the same strategy as neural feature fields [7, 6]. where we sample query points on the hand surface and minimize their feature differences between the two scenes.
Experimentally, we show that our neural attention field induces better optimization landscapes by encouraging the end-effector to focus task-related scene regions and omitting distractions, abling robust one-shot dexterous grasping in various challenging scenarios such as grasping objects in complex scene contexts, grasp transfer between objects of varying shapes but similar semantics, or functional grasping of tools. Real-robot results demonstrate our significant improvements in success rates compared with the feature-fiel field- based methods.
To summarize, our key contributions are: (i) We propose the neural attention field for representing semantic-aware dense feature fields in the 3D space by modeling inter-point relevance instead of individual point features. (ii) We propose a self-supervised framework for training the attention field with few scenes. (iii) Our attention field enables semantics dexterous grasping from one-shot demonstration and encourages the end-effector to focus on task-related scene regions. (iv) Real-robot experiments show our strong generalization under different scene variations.
2
Related Work
Neural field for manipulation Point correspondences induced by features facilitate the transfer of manipulation policies across diverse objects. In contrast to methods based on key points [8, 9, 10], recent efforts focus on developing dense feature fields around 3D scenes through implicit represen- tations [7, 11, 12, 13, 14, 15, 16, 17, 18, 19]. The integration of large vision models have propelled

---

research towards leveraging distilled features fields for enabling few-shot or one-shot learning of 6-DOF grasps [3], sequential actions [4], and language-guided manipulations [5]. These efforts predominantly focus on simple manipulations using parallel grippers. Closest to us is [6] which uses distilled feature fields to transfer dexterous grasps across objects. However, as mentioned, computes query features in the free space as sum of closeby scene-point features weighted by distances, making it unstable distractors near the target object.
Distilled feature field With the recent advances of large vision models on 2D images, an effective way of acquiring semantics-awant feature fields for 3D scenes is to first obtain 2D image features and then distill them to 3D. [20, 21, 22] lifts of semantic information from 2D segmentation networks to 3D, enabling 3D segmentations with language embeddings. [23, 24] delve into integrating pixel- aligned image features from models like LSeg or DINO [1] into 3D NeRFs, highlighting their impact on manipulating 3D geometry. Further, [25] and [26] explore distilling non-pixel-aligne image features, such those from CLIP [27], into 3D scenes without the need for fine-tuning.
Dexterous grasping Dexterous manipulation, central to advanced robotic applications, necessi- tates nuanced understanding and control, akin to human-like grasping capabilities [28, 29, 30, 31, 32, 33, 34). Analytical approaches [35, 29, 30] focus on direct modeling of hand and object dy- namics. Recent learning-based methods have introduced state [36, 37, 38, 39, 40] and vision-based strategies [41, 42, 43, 44, 45, 46), targeting realistic scene comprehensions. However, most of these methods depend on large demonstration datasets for training. Notably, [47] proposes grasp synthe- sis algorithm focusing on geometric and physical constraints for functional grasping that generalizes within similar object shapes using minimal demonstrations,
3 Method
3.1 Preliminaries
Demonstration transfer with feature field Given a 3D scene pointcloud X, a feature field around the scene is a function f(- X) : R23 + RC that maps every query point q E R in the 3D space to a C-dimension feature f E RCC For an end-effector placed in the feature field with pose parameters 3, one can sample set of query points Q E R on the end-effector surface conditioned on its parameters and obtain their features f(Q]3, X) E R which can be viewed as the feature for the entire end-effector w.r.t.B. Given a demonstration with a source pointcloud X and end-effector parameters 3, one can transfer it to a target scene X by minimizing an energy function induced by the two feature fields
arg min E(3) arg min [[ 3, x) f(Q X)I11.
(1)
3
Intuitively, this minimizes the feature differences of the end-effector positions in the scene, indicat- ing that the end-effector should be applied to the locations with similar semantic meanings. More elaborations can be found in [7, 6].
Distilled feature field An effective approach for acquiring semantic-aware 3D feature field is to first extract semantic features on 2D images with large vision models [1, 2] and then distill them to 3D. A line of works [26, 5, 19] directly reconstruct continuous implicit feature fields alongside NeRF representations. If given a 3D pointcloud and its point correspondences to the 2D image pixels derived from RGBD sensor inputs, one can also directly back project and aggregate the image features on each pixel to the 3D points. The discrete pointcloud features can be further propagated into the surrounding space via spatial interpolation [6]. For pointcloud X {x.} with per- per-point features F = {f.). the feature z at query point R3 is interpolated from the pointcloud features with inverse distance weighting:
N
1/ l x
wife. where
(2)
Σラ=1 1/llq x, 112
3

---

Cyclic MNN keypoint selection
keypoints (query)
Self-consistency loss
transformer decoder Dg
scene
keypoints (query)
transformer decoder Dy
InfoNCE loss
scene points
preserve keypoint correspondences
Figure 2: Self-supervised training for the transformer decoder. Left: Given few scenes, first select the corresponding keypoints computing cyclic mutual nearest neighbors (MNN) based on their feature similarities. Right: We take the selected keypoints as queries for each scene and enforce the features before and after the Dø-aggregation to induce the same keypoint correspondences across scenes. Specifically, we apply an InfoNCE loss to preserve the orders of the keypoints given the permutation equivariance of transformers.
While this spatial interpolation propagates the features from the discrete surface points to the near- surface space smoothly, this operation lacks semantic awareness and blurs the inter-scene feature correspondences as the query point goes off the object surfaces, making it unstable to transfer end- effector poses from the source scene to the target. For example, when transferring a grasp from a source scene containing only a toy animal to a target scene which the toy animal is surrounded by several irrelevant objects, we aim ensure that the features focus primarily on the toy animal itself and are less affected by the other objects despite their spatial adjacency (Fig. 1 top).
3.2 Neural Attention Field
To better represent dense and semantic-aware features in the near-object space, we introduce the neural attention field (Fig. 1) that predicts a cross- attssttttnnnoon between the query point and the scene points to replace the inverse distance weighting w, in Eq. 2. More concretely, it realized with a lightweight transformer decoder network Do query, value) with its keys, query, and value being the scene points X, the query point q. and the scene-point features F respectively:
fp Dg(X,q.
(3)
which provides an aggregated feature fn R for any query point q E R according to the learned attention maps. As the 3D point coordinates X and q carry limited information, we also append the per-point features F to X and the spatially aggregated feature f in Eq. to q. resulting in keys [X, F) and query (q. f]. Detailed architecture of the transformer decoder be found in the appendix.
3.3 Few-Shot Self-Supervised Training
As there is no straightforward way to annotate ground-trun truth features or attention maps, we propose self-supervised framework for training the transformer decoder D, in Eq. 3 on few 3D scenes related the task. In most of experiments, apply
4 scene pointclouds train D, and then of tasks transferring dexterous hand poses. We also experiment with different training and inference setups including different numbers of training scenes and different content similarities between the training and inference scenes (Sec. 4.2).
We consider directly using the scene points x, E X as queries and compute their ag-agggegated features via a self-attention:
fD, D,,XX,,,FF
(4)

---

We denote FD = {fD.sh the collections of aggregated features for all points in X. Conceptually, this can also be viewed as placing a virtual query at each point in the scene pointclouds.
Given a few different scene pointclouds x ,.... X with raw point features F(I).... ,F(,, we can compute their aggregated point features F
D ,.... FDD respectively. Both F and FD induce point correspondences across scenes via nearest neighbor retrieval w. feature similarities. Based on this, we introduce a self-supervised loss for training Da by enforcing that the induced correspon- dences from F and F should be identical between any pair of scenes (Fig. 2).
More concretely, we first compute a set of keypoints K K()) E kx for each scene based on the raw point features F(1), with each keypoint indicating correspondence across all scenes (e.g.the point on the monkey's is left hand in all scenes). We then update the features of these key- points via Eq. 4, acquiring their Dø-aggregated features F Finally, transformer networks are permutation equivariant, we enforce the keypoint orders to stay consistent before and after applying Dg with contrastive loss on F(:)
Cyclic MNN keypoint selection (Fig. 2 leftt) We compute the keypoints K(),... (K()) such that each (+) is subset of the points in X and any pair of ((), K(i) are corresponding points across the two scenes according to the similarity of their raw features F(1), F(5), We introduce a cyclic mutual nearest neighbor (MNN) strategy for this corresponding keypoints selection, as illustrated in Fig. 2 left. Specifically, we randomly assign an order I,... ,I to the scenes. For each adjacent scene pair (() and ( (or (!) and X ()),, any keypoint in K must fall into the k-nearest neighbor of the corresponding keypoint in K ((++) according to their feature differences. We reject any point that violates this constraint for any i.
Self-consistency feature loss (Fig. 2 right)
The keypoint features are updated with Da via:
Da
F(1), i= 1,,..,,
We then enforce that, after this Do update, the feature-induced keypoint correspondences stay un- changed. Specifically, we employ an InfoNCE [48] loss as illustrated in Fig. 2 right:
K
cxp(sim((f)) D. ((1)) (/20) (22)/T))
L= Σ Cij. (EE
log En exp(sim (1) +0) D. D/T)
(6)
where D. is the k-th entry of F namely the output feature of Da of the k-th keypoint in scene and is temperature parameter. This loss minimizes the updated feature differences of matched keypoints and maximizes that of unmatched keypoints. We optimize for the network parameters in Do w.r.t this loss.
3.4 End-Effector Optimization
Following [6], we consider the dexterous hand parameters 3 of the hand pose and joint rotations. Given a demonstration of hand parameters in the source scene X (with F), we optimize for / in the target scene X (with F) using the neural attention field. We start by sampling Q points on the hand surfaces in both the source and target hands, generating query point sets Q 3 Q Rxx conditioned on the hand parameters. The sampling done in the canonical hand pose and is identical between the source and target hand surfaces. We follow the sampling strategy in [6] which has higher sampling density on the fingers than on the palm.
We then obtain the features for the end-effector query points by passing them into the trained trans- former decoder Da and acquire their features D2 3, X, F), Da 1, X, F) R With these features, optimize for the end-effector pose / in the target scene while keeping all net- work parameters in Do frozen. The objective is to minimize the feature differences between the demonstration and target hand poses via an 11 loss, formulated as an energy function E(3) w.r.t. B:
E(B) I. Q 3 X. F) Do 8, 11
5

---

feature difference
end-affector optim.
transformer decoder Dg
demonestatioo hand pose
source pointcloud
argmin
transformer decoder Dg
barget pointctoud
Figure 3: End-effector optimization in the neural attention field. Left: We sample query points both the demonstration hand and the target hand to be optimized and obtain their features through the transformer decoder Da- Middle: The feature differences induce an energy field. Yellow indicates lower energy values and thus higher feature similarities. Right: Minimizing the energy function w.r.tthe hand parameters gives the final grasping pose. Both hand positions and joint parameters optimized. The optimization trajectory is shown green and a few hand poses sampled along the trajectory shown in blue (optimization steps indicated with colors from shallow dark).
We also integrate the physical viability functions from [49, 6], including the inter-penetration and self-penetration energy functions:
Σ
(ixeoddist(xx, 8Q); Espen(B)
Σ
1 1pgq max(6 ||p ql12,0),
(8)
xEX
P.qEQ)
where is a threshold parameter, and a pose constraint (p)s that penalizes out-of-limit hand pose in [49]. The overall optimization combines these terms: + Xpose (3).
E(B) Epen + >spen
(9)
The weighting between feature energy loss and the physical constraints are set to be Apen 10-1, Aspen = 10-2, Apusee = 10-2, which is the same as [6]. In all our experiments, we randomly initialize the hand position and joint parameters and optimize for via gradient descent.
Experiments
Training and inference In all experiments, pre-train the transformer decoder network Dg with a few scene pointclouds (2-4 scenes without demonstrations). We then provide 1-shot demonstration with a dexterous hand pose in a 3D scene and directly transfer it to different scenes. This demonstration scene pointeloud doesn't 't have to be included in the pre-training scenes. We experiment with different pre-training and demonstration scene setups as listed in the tables. Each evaluation is done 10 times with scene variations in the object poses (for both single-object and multi-object scenes).
Real-robot setup We conduct all our experiments in the real world with a Shadow Dexterous Hand of 24 DoF. We restrict the 2 DoF at the wrist, focusing the optimization on the remaining 22 DoF. The hand is mounted on a UR10e arm, introducing additional DoF. Precautions are taken to prevent the dexterous hand from contacting the table surface directly, and the range of motion limited by the arm's elbow. For RGBD scans, we four pre-calibrated Femto Bolt sensors each corner of the table. Post-processing is applied to the captured scans to remove background elements: In single-object experiments, the object's pointcloud is segmented using SAM [50]. For multi-object scenarios, we apply physical constraints to limit the experimental area to the tabletop and employ RANSAC [51] to exclude the table surface from the analysis.
Baselines We mainly compare our method to the feature-field-basee methods, including a vanilla DFF with the feature acquisition strategy in [25], and SparseDFF [6] which introduces additional multiview feature refinement mechanism for better feature qualities.

---

source grasp (demo.)
target grasp (ours)
energy field (ours)
energy field (FFarseDFF)
source graep (demo.)
target grasp (ours)
energy field (ours)
energy field (SparseDFF)
Figure 4: Visualizations of the energy fields and end-effector optimization. In each group from left to right are: the source scene with demonstration (hand shown blue); the target scene and our results (optimization trajectory shown in green and final resulting hand shown blue); the feature- induced energy fields for our method and SparseDFF [6]. We only visualize the 2D sections of the 3D energy fields and yellow indicates lower energy values and thus higher feature similarities. Our method shows more concentrated low-energy regions around the target grasping positions.
scene contexts with distractions
novel objects of different shapes
functional grasping
Figure 5: Real-robot results. From left to right, we show results of grasping objects in scene contexts with distraction, grasp transfer between objects with similar semantics but shape variations, and functional grasping of tools.
4.1 Real-Robot Results
Object in scene contexts mechanism between end-effector queries and the scene points encourages the focus on the scene points of higher relevance to the task indicated by the demonstration. This is particularly beneficial when there are task-irrelevant objects in the scene and we want to accurately locate specific regions on the target object and avoid distractions. Tab. shows our results for grasping the target objects from multi-object scene contexts with only a one-shot grasping demonstration on a single object. Our method has consistently higher success rates than the previous feature-field-bass methods. Real-robot executions are shown Fig. left.
major advantage of our neural attention field that the cross-attention
Fig. 4 visualizes the energy fields of our method and vanilla distilled feature field method SFarseDFF [6]. We can see that our attention field induces energy functions with more concen- trated minimas centered around the target grasping positions (such as the monkey arm in the top left example), while the distilled feature field induces more irregular energy functions in the space, resulting worse optimization landscapes.
Cross-object grasp transfer Tab. 2 shows our results on dexterous grasp transfer between objects of varying shapes but similar semantics. In addition to filtering out the irrelevant content in multi- object scenes, our attention field also brings benefits even in single-object scenes by encouraging the transformer queries to focus on task-related object regions. For example, the necks the horse and the giraffe are of very different shapes and are close to the back where the grasp is applied, but semantically they are irrelevant to the task. Simply computing the features according to spatial relations as in the feature fields can be misleading, while attention field overcomes this issue. Real-robot executions shown in Fig. middle.
7

---

Table 1: Object scene contexts.
Da pre-train
Demo scene
Test scene
Ours
SparseDFF
DFF
4 monkey
monkey right arm
monkey in context
10/10
6/10
4/10
4 monkey
monkey back
monkey in context
10/10
8/10
5/10
4 monkey
monkey back
horse in context
9/10
3/10
0/10
Table 2: Cross-object grasp transfer.
Dg pre-train
Demo scene
Test scene
Ours
SparseDFF
DFF
1 horse, 1 rhino
horse back
giraffe
9/10
5/10
0/10
1 horse, 1 giraffe
giraffe back
rhino
10/10
2/10
0/10
1 rhino, 1 giraffe
rhino back
horse
10/10
7/10
0/10
I rhino, dinosaur
horse back
giraffe
7/10
5/10
0/10
4 monkey
horse back
giraffe
9/10
5/10
0/10
Table Functional grasping.
Do pre-train
Demo scene
Test scene
Ours
SparseDFF
DFF
nail hmmr., rubber mallet Thor hmmr. Thor hmmr.., rubber mallet
nail hmmr. Thor hmmr.
Thor hmmr.
9/10
8/10
8/10
nail hmmr.,
rubber mallet
8/10
4/10
0/10
rubber mallet
nail hmmr.
9/10
5/10
0/10
Functional grasping The semantic awareness in neural attention also facilitates functional grasping. Tab. 3 shows our results for functional grasping of tools where the demonstrations and executions are on tools with varying shapes but similar functionalities. Real-robot executions are shown in Fig. 5 right.
4.2 Ablation Studies
Now conduct ablation studies for the De pre-training setups on the number of pre- training scenes and whether the demonstration scene is included. All ablation experiments are done with the monkey toy. Fig. 6 plots the results of our method under different num- bers of Da pre-training scenes. Our method shows consistent performances over these dif-
Hand distance target region
2.000
Include Demo.
Exclude Demo.
SparselFFF
8435
81.50
ferent pre-training setups. Qualitative results
can be found in the appendix.
Number of pre craining scenes
Figure 6: Dg pre-training with different setups.
Conclusions
To better model hand-object interactions, we propose the neural attention field, which represents semantic features in the 3D space via modeling inter-point relevance instead of individual point features. enables robust and semantic-aware transfer dexterous grasps across scenes by encour- aging the end-effector to focus scene regions with higher task relevance.
Limitations and future work While we focus on dexterous grasps this work, the neural atten- tion field and its subsequent feature-based end-effector optimization can be applied to other dexter- manipulations beyond grasping. Extension to other manipulations with action sequences would be an interesting and useful future direction.

---

Acknowledgments
This work was supported in part by the Toyota Research Institute, the National Science Foundation under Grant Number 2342246, and by the Natural Sciences and Engineering Research Council of Canada (NSERC) under Award Number 526541680.
References
[1] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties self-super vision transformers. In International Conference on Computer Vision (ICCV), 2021.
[2] M. Oquab, T. Darcet, Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, Fernandez, D. Haz- iza, F. Massa, A. El-Nouby, al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.077 2023.
[3] W. Shen, G. Yang, A. Yu, J. Wong, L. P. Kaelbling, and Isola. Distilled feature fields enable few-shot manipulation. In Conference on Robot Learning (CoRL), 2023.
[4] X. Lin, J. So, S. Mahalingam, F. Liu, and P. Abbeel. Spawnnet: Learning generalizable visuo- motor skills from pre-trained networks. arXiv preprint arXivv2307.03356 2023.
[5] A. Rashid, S. Sharma, C. M. Kim, J. Kerr, L. Y. Chen, A. Kanazawa, and K. Goldberg. Lan- guage embedded radiance fields for zero-shot task-oriented grasping. In 7th Annual Conference on Robot Learning, 2023.
[6] Q Wang, Zhang, C. Deng, You, Dong, Y. Zhu, and Guibas. Sparsedff: Sparse-view feature distillation for one-shot dexterous manipulation. arXiv preprint arXiv:2310.16838, 2023.
[7] A. Simeonov, Y. Du, A. Tagliasacchi, B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitz- mann. Neural descriptor fields: Se (3)-cquivariant object representations for manipulation. In 2022 International Conference on Robotics and Automation (ICRA), pages 1394-6400 IEEE, 2022.
[8] L. Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for category- level robotic manipulation. The International Symposium of Robotics Research, 2019.
[9] Z. Xue, Yuan, J. Wang, X. Wang. Y. Gao, and H. Xu. Useek: Unsupervised se (3)- equivariant 3d keypoints for generalizable manipulation. In International Conference on Robotics and Automation (ICRA), 2023.
[10] P. R. Florence, I.. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. In Conference Robot Learning, pages 373-385. PMLR, 2018.
[11] R. Wu, C. Ning, and H. Dong. Learning foresightful dense visual affordance for deformable object manipulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10947-10956, 2023.
[12] H. Ryu, H.-i. Lee, J.-H. Lee, and J. Choi. Equivariant descriptor fields: Se (3)-equivariant energy-based models for end-to-end visual robotie manipulation learning. The Eleventh International Conference on Learning Representations, 2022.
[13] Q. Dai, Y. Zhu, Geng, C. Ruan, Zhang, and Wang. Graspnerf: multiview-baat 6-dof grasp detection for transparent and specular objects using generalizable nerf. In International Conference Robotics and Automation (ICRA), 2023.

---

[14] A. Simeonov, Y. Du, Y.-C. Lin, A. R. Garcia, L. P. Kaelbling, T. Lozano--éérr and P. Agrawal. Se (3)-equivariant relational rearrangement with neural descriptor fields. In Conference on Robot Learning (CoRL), 2023.
[15] T. Weng, D. Held, F. Meier, and M. Mukadam. Neural grasp distance fields for robot manipu- lation. In International Conference on Robotics and Automation (ICRA), 2023.
[16] J. Urain, N. Funk, J. Peters, and G. Chalvatzaki. Se (3)-diffusionfields: Learning smooth cost functions for joint grasp and motion optimization through diffusion. In International Conference Robotics and Automation (ICRA), 2023.
[17] Y. Zhao, R. Wu, Z. Chen, Y. Zhang, Q. Fan, K. Mo, and H. Dong. Dualafford: Learning collaborative visual affordance for dual-gripper manipulation. arXiv e-prints, pages arXiv- 2207, 2022.
[18] R. Wu and Y. Zhao. Vat-mart: Learning visual action trajectory proposals for manipulating 3d articulated objects. In International Conference on Learning Representations (ICLR), 2022, 2022.
[19] Y. Ze, G. Yan, Y.-H. Wu, A. Macaluso, Ge, J. Ye, N. Hansen, L. E. Li, and X. Wang- Gnfactor: Multi-task real robot learning with generalizable neural feature fields. In 7th Annual Conference Robot Learning, 2023.
[20] S. Zhi, T. Laidlow, S. Leutenegger, and A. J. Davison. In-place scene labelling and under- standing with implicit scene representation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[21] Y. Siddiqui, L. Porzi, S. R. Bulò, N. Müller, M. Nießner, A. Dai, and P. Kontschieder. Panoptic lifting for 3d scene understanding with neural fields. Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[22] Z. Ren, A. Agarwala, B. Russell, A. G. Schwing, and O. Wang. Neural volumetric object selection. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[23] S. Kobayashi, E. Matsumoto, and V. Sitzmann. Decomposing nerf for editing via feature field distillation. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[24] V. Tschernezki, I. Laina, D. Larlus, and A. Vedaldi. Neural feature fusion fields: 3d distillation of self-supervised 2d image representations. In International Conference 3D Vision (3DV), 2022.
[25] S. Peng, Genova, C. Jiang, A. Tagliasacchi, M. Pollefeys, T. Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[26] Kerr, M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik. Lerf: Language embedded radiance fields. arXiv preprint arXiv:2303. 09555, 2023.
[27] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, Clark, al. Learning transferable visual models from natural language supervi- sion. In International Conference on Machine Learning (ICML), 2021.
[28] J. K. Salisbury and J. Craig. Articulated hands: Force control and kinematic issues. Interna- tional Journal of Robotics Research (IJRR,, 1(1):4-17, 1982.
[29] M. R. Dogar and S. S. Srinivasa. Push-grasping with dexterous hands: Mechanics and method. In International Conference on Intelligent Robots and Systems (IROS), 2010.
[30] S. Andrews and P. G. Kry. Goal directed multi-fing finger manipulation: Control policies and analysis. Computers & Graphics, 37(7) 2013.
10

---

[31] N. C. Dafle, A. Rodriguez, R. Paolini, B. Tang, S. S. Srinivasa, M. Erdmann, M. T. Mason, I. Lundberg, H. Staab, and T. Fuhlbrigge. Extrinsic dexterity: In-hand manipulation with external forces. In International Conference on Robotics and Automation (ICRA), 2014.
[32] V. Kumar, E. Todorov, and S. Levine. Optimal control with learned local models: Application dexterous manipulation. In International Conference on Robotics and Automation (ICRA), 2016.
[33] H. Qi, A. Kumar, R. Calandra, Y. Ma, and J. Malik. In-hand object rotation via rapid motor adaptation. In Conference on Robot Learning (CoRL), 2023.
[34] Y. Liu, Y. Liu, C. Jiang, K. Lyu, W. Wan, H. Shen, B. Liang, Z. Fu, H. Wang, and L. Yi. Hoi4d: 4d egocentric dataset for category- human-object interaction. Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[35] Y. Bai and C. K. Liu. Dexterous manipulation using both palm and fingers. In International Conference on Robotics and Automation (ICRA), 2014.
[36] T. Chen, J. Xu, and P. Agrawal. A system for general in-hand object re-orientation. In Confer- ence on Robot Learning (CoRL), 2022.
[37] S. Christen, M. Kocabas, Aksan, Hwangbo, Song, and O. Hilliges. D-grasp: Physically plausible dynamic grasp synthesis for hand-object interactions. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[38] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. International Journal of Robotics Research (IJR)), 39(1)33-0,, 2020.
B. McGrew, J. Pachocki,
[39] Q. She, R. Hu, J. Xu, M. Liu, Xu, and H. Huang. Learning high-dof reaching-and andgganpinngg via dynamic representation of gripper-object interaction. ACM Transactions on Graphics (TOG), 41(4):1-14, 2022.
[40] Y.HH. Wa, J. Wang, and X. Wang. Learning generalizable dexterous manipulation from human grasp affordance. In Conference on Robot Learning (CoRL), 2023.
[41] Mandikal and K. Grauman. Learning dexterous grasping with object-centric visual affor- dances. International Conference on Robotics and Automation (ICRA), 2021.
[42] Mandikal and K. Grauman. Dexvip: Learning dexterous grasping with human hand pose priors from video. In Conference Robot Learning (CoRL), 2022.
[43] P. Li, T. Liu, Y. Li, Y. Zhu, Y. Yang, and S. Huang. Gendexgrasp: Generalizable dexterous grasping. In International Conference on Robotics and Automation (ICRA), 2023.
[44] Y. Qin, B. Huang, Z.-H. Yin, H. Su, and X. Wang. Dexpoint: Generalizable point cloud rein- forcement learning for sim-to-real dexterous manipulation. In Conference on Robot Learning (CoRI.), 2023.
[45] Y. Xu, W. Wan, J. Zhang, H. Liu, Z. Shan, H. Shen, R. Wang, H. Geng, Y. Weng, J. Chen, et al. Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal genera- tion and goal-conditioned policy. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[46] W. Wan, H. Geng, Y. Liu, Z. Shan, Y. Yang, L. Yi, and H. Wang. Unidexgrasp++: Improving dexterous grasping policy learning via geometry-aware curriculum and iterative generalist- specialist learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3891-3902, 2023.

---

[47] W. Wei, P. Wang, and S. Wang. Generalized anthropomorphic functional grasping with mini- mal demonstrations. arXiv preprint A.Nv::333.//8000 2023.
[48] T. Chen, S. Korbllith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International Conference Machine Learning (ICML), 2020.
[49] R. Wang, Zhang. Chen, Y. Xu, P. Li, T. Liu, and H. Wang. Dexgraspnet: large-scale robotic dexterous grasp dataset for general objects based on simulation. In International Con- ference on Robolics and Automation (ICRA), 2023.
[50] A. Kirillov, E. Mintun, N. Ravi, Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, Dollár, and R. Girshick. Segment anything. arXiv:230.002643, 2023.
[51] M. A. Fischler and R. C. Bolles. Random sample consensus: paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24 (6):381-395, 1981.
12

---

A Network Details
Transformer decoder architecture Fig. illustrates the network architecture of our transformer decoder. We compute the pairwise distances between the hand points and scene points and apply them as weights to the keys and queries for more stable convergence during training.
pairwise
distance
scene point coords.
festures
keys
(values)
act
prod.
hand point coords.
interp.
attention
queries
oulput features
Figure Network architecture of the transformer decoder.
Network training The transformer decoder network is only trained at the feature pre-training stage with the self-consistency losses, and all network weights are frozen afterward in the end- effector optimization stage. All networks are trained on a single GeForce RTX 3090 with an Adam optimizer for 100 iterations.
B Real-Robot Setup Details
100cm
Fig. 8 shows real-robot setup. Four Femto Bolt sensors are mounted at the four corners 100cm above the table, with each edge 100cm. The objects are placed near the center of the table. The dexterous hand is mounted on a UR10e arm, reaching the ob- jects from the left.
Femto Bolt
UR10e
Shadow
Shadow hand parameterization As mentioned in the paper, we conduct all our experiments in the real world with a Shadow Dexterous Hand of DoF. We restrict the DoF at the wrist, focusing the opti- mization on the remaining 22 DoF. The UR10e arm also introduces 6 additional DoFs. Each hand joint parameterized
Figure 8: Real-robot setup.
scaler for its rotation angle. The shadow hand has 4 under-actuated joints. But in our method, we just regard it as a normal actuated joint, which is common practice and the default API from the shadow hand takes these four joints as normal joints. We use "rotate6D" to represent the 3-DoF hand base rotation, which is the first two columns of the rotation matrix.
Scene setup and randomization In the real-robot experiments, for single-object scenes with toy animals, we randomize its pose with arbitrary 7-axis rotations so that the dexterous hand can grasp it from the top without hitting the table. For functional grasping with 3D-printed tools, we put them on boxes so that the robot can reach its handle without hitting the table. For multi-object scenes, we randomize the poses of all objects with full SO(3) rotations.
C Ablation Study Details
Evaluation metrics (Fig. The ablation studies done virtually on pre-captured 3D scene pointclouds to eliminate the variances caused by scene randomization, and thus the 10 evaluation
13

---

scenes in all the ablation settings are identical. In Fig. 6, we evaluate the average distance between the hand-surface query points and the target region (monkey arm) in the scene, as a rough metric indicating how close the hand reaches the target post-optimization. The target region is manually annotated with MeshLab.
Quantitative results Fig. 9 shows the qualitative results on grasping the monkey arm with the same demonstration but different pre-training setups for the transformer decoder. The hand opti- mization trajectories are shown in green.
# pretrain scenes
scenes
scenes
scenes
8 scenes
10 scenes
demo scene
included
demo scene not included
Figure Qualitative results for the ablation study on different pre-training setups.
14


Reviews:
[{"review_id": "spw3KLb6fS", "reviewer": "Reviewer_gKTk", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work introduces a neural attention field using a transformer decoder to model inter-point relevance, enhancing feature representation and focusing on task-relevant regions. They uses a self-supervised framework to train the model on a few 3D scenes and applied to new scenes for grasping purposes from one-shot demonstrations.", "summary_of_recommendation": "Recommend a rejection unless followup draft demonstrates a significant improvement in experimental quality and contributions", "technical_quality": 2, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 2, "quality_of_limitations": 1, "questions_for_rebuttal": "From an applied robotics perspective, I am curious how the authors envision this method affecting downstream applications. Specifically, in a problem setting where the model is trained on grasping one or a few objects and then applied to grasp the same or similar objects in more complex environments. Although fast adaptation is beneficial, the applicability seems limited because new objects with new semantics would still require new demonstrations for re-training.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fcxBl6SOas", "reviewer": "Reviewer_36Qt", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a neural attention field method for one-shot transfer of dexterous grasps. The work builds on previous work [6] by shifting the focus to task-relevant scene points instead of spatial proximity. The work is evaluated in real-world scenarios using a shadow hand.", "summary_of_recommendation": "Dexterous grasping is challenging, but I have some concerns about the experiment setup, as objects are placed much further apart, questioning the claim that this method is more robust to distractors. I am looking forward to the rebuttal.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "1.) Why not simply add a segmentation step to [6]? This would filter out the distractors.\n\n2.) The clutter in the real-world experiment seems to be biased towards this method. It seems like objects are placed more apart in the “Monkey to horse context.” E.g., the power drill in the video at 2:45 is. Can you comment on that?\n\n3.) Actually, can you redo the “Monkey to horse context” experiment and place the objects close together as you did for SparseDFF?\n\n4.) Similarly, the rubber mallet in 2:45 seems to be rotated less than for the baselines. Why does SparseDFF only perform 50% if there are few distractors?\n\n5.) Can you elaborate more on the task-relevance of this feature?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DYitBigPGa", "reviewer": "Reviewer_vfCH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces a neural attention field based on an attention decoder to create semantic-aware features for one-shot dexterous grasping transfer. This approach models inter-point relevance instead of individual point features, enhancing the ability to generalize grasps across different 3D scenes. The authors propose a self-supervised framework for training the attention decoder, eliminating the need for extensive hand-labeled datasets. Experimental results demonstrate that this method significantly improves performance over baseline feature-field-based methods, showing better optimization landscapes and higher success rates in grasping tasks with real robots.", "summary_of_recommendation": "The paper introduces a neural attention field for one-shot dexterous grasping, using a transformer decoder to enhance semantic understanding. The self-supervised learning approach enables efficient training from few samples and shows significant performance improvements over baselines. However, the paper lacks detail on key design choices and does discuss potential costly runtime and data collection process. Despite these issues, the contributions and results justify acceptance.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Questions for Rebuttal\n- How does this method transfer to scenes with unseen objects as distractors?\n- How is the pairing of key points across scenes achieved (lines 151-152)? Does this method require a curated dataset with cross-scene point pairs?\n- What is $\\bar{\\mathbb{Q}}$ in equation 8 $E_{pen}$?\n- Does the initialization of query points (Q) affect the quality of the final grasp pose?\n- How are the scene features calculated, as shown in Appendix Figure A?\n- How does the method perform in scenes with occlusions?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a new method for one-shot grasp transfer for dexterous grasping. A key component is the introduced neural attention field to capture dependencies between 3D points by using an attention mechanism. Grasps are optimized using features from the  neural attention field.

Strengths

- The reviewers agree on that the neural attention field for one-shot grasping is new.
- The paper is well written and easy to read.

Weaknesses

The reviewers have raised several concerns about the paper.

- Limited novelty of the paper regarding the difference between the proposed method and [6].
- Limited experimental evaluations. No comparison with other state-of-the-art baselines. Can object segmentation be used with SparseDFF?
- Clarification of certain experimental setup.

Post-Rebuttal
- The authors provided their answers to address the concerns from the reviewers. However, the reviewers did not participant in the discussion with several messages from the AC.
- The AC checked the rebuttal and agrees with the authors. The proposed method has enough novelty to address the one-shot grasping problem for dexterous hands. It shows improvement over SparseDFF [6].
Decision: Accept