SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulation
Authors: Qianxu Wang
Haotong Zhang
Congyue Deng
Yang You
Hao Dong
Yixin Zhu
Leonidas Guibas
Venue: ICLR 2024 poster (2024)
Keywords: dexterous manipulaiton
one-shot manipulation transfer
distilled feature field
implicit field
Primary area: applications to robotics, autonomy, planning

Abstract: Humans demonstrate remarkable skill in transferring manipulation abilities across objects of varying shapes, poses, and appearances, a capability rooted in their understanding of semantic correspondences between different instances. To equip robots with a similar high-level comprehension, we present SparseDFF, a novel DFF for 3D scenes utilizing large 2D vision models to extract semantic features from sparse RGBD images, a domain where research is limited despite its relevance to many tasks with fixed-camera setups. SparseDFF generates view-consistent 3D DFFs, enabling efficient one-shot learning of dexterous manipulations by mapping image features to a 3D point cloud. Central to SparseDFF is a feature refinement network, optimized with a contrastive loss between views and a point-pruning mechanism for feature continuity. This facilitates the minimization of feature discrepancies w.r.t. end-effector parameters, bridging demonstrations and target manipulations. Validated in real-world scenarios with a dexterous hand, SparseDFF proves effective in manipulating both rigid and deformable objects, demonstrating significant generalization capabilities across object and scene variations.

Published as a conference paper at ICLR 2024
SPARSEDFF: SPARSE-VIEW FEATURE DISTILLATION FOR ONE-SHOT DEXTEROUS MANIPULATION
Qianxu Wang HHatong Zhang', Congyue Deng
2.0 Yang You?,
Hao Dong', Yixin 3hu Leonidas Guibas
1 CFCS, School of Computer Science, Peking University, China
Department of Computer Science, Stanford University, USA
3 Institute for AI, Peking University, China
PKU-WUHAN Institute for Artificial Intelligence, China
congyue stanfordttttaadddddeeoeo edu, yixin, zhu@@ @wu.upppuuucce edu.edu.eeo edu en, guibas @ stanford.edu htttps://helloxww/ee g situub...... Lo//parreDDF
Feature Distillation
Generalized Manipulation
2D features
Pose deformation
Novel category
One-shot demonstration
Scene context
SparseDFF
Figure 1: Overview of SparseDFF. We introduce novel method, SparseDFF, for distilling view-consistent 3D Distilled Feature Field (DFF) from sparse RGBD images, readily generalizable to novel scenes without any modifications or fine-tuning. The DFFs create dense correspondences across scenes, enabling one-shot learning of dexterous manipulations. This approach facilitates seamless manipulation transfer to new scenes, effectively handling variations in object poses, deformations, scene contexts, and categories.
ABSTRACT
Humans demonstrate remarkable skill transferring manipulation abilities across objects of varying shapes, poses, and appearances, a capability rooted in their understanding semantic correspondences hetween different instances. To equip robots with similar high-level comprehension, we present SparseDFFF, a novel DFF for 3D scenes utilizing large 2D vision models to extract semantic features from sparse RGBD images, a domain where research is limited despite its relevance to many tasks with fixed-camera setups. SparseDFF generates view-consistent 3D DFFs, enabling efficient one-shot learning of dexterous manipulations by mapping image features to a 3D point cloud. Central to SFarseDFF is feature refinement network, optimized with contrastive loss between views and a point-pruning mechanism for feature continuity. This facilitates the minimization of feature discrepancies w.r.t. end-effector parameters, bridging demonstrations and target manipulations. Validated in real-world scenarios with a dexterous hand, SparseDFF proves effective in manipulating both rigid and deformable objects, demonstrating significant generalization capabilities across object and scene variations.
INTRODUCTION
Learning from demonstration is a powerful approach for quickly imparting complex skills to robots. Although recent advancements have shown promising results in applying reinforcement learning to dexterous manipulation tasks (Xu et al., 2023; Wan et al., 2023; Li et al., 2023a), the effectiveness of these methods is often contingent and struggles to accommodate the varied demands different tasks. Moreover, these techniques primarily target the manipulation of rigid objects and encounter significant obstacles in real-worr world applications and in scaling to datasets beyond those they were trained on. In stark contrast, humans demonstrate remarkable abilities extrapolate and generalize from observed demonstrations (Lake
the availability of carefully curated demonstration dataset

---

Published as a conference paper at ICLR 2024
& Baroni, 2023; Jiang et al., 2023; Li ct 2024b;a; 2023b; Xie et al., 2021; Lake et al., 2015). For instance, learning to hold a cat by watching someone do so can effortlessly extend to holding various other cats of different breeds, sizes, and appearances even to entirely different animals, such as dogs, otters, or baby tigers, provided they are accessible and amenable. This exceptional capacity for generalization stems from the ability to discern underlying similarities across different instances, despite variations in appearance, pose, or species (Zhu et al., 2020; Fan et al., 2022).
To empower autonomous agents with human-like comprehension and generalization from demonstra- tions. leveraging object and scene representations from large vision models proves to be effective. Despite the prevalence of these models being trained on 2D acquiring and annotating 3D data applying them directly to complex manipulation tasks, such as those requiring dexterous manipulation, remains significant challenge. Recent endeavors have introduced DFFs (Kobayashi et al., 2022), which transform dense 3D feature fields from 2D) image features, thus enhancing understanding of 3D scenes (Kerr et al., 2023) and facilitating interactions (Shen et al., 2023; Lin et al., 2023; Rashid et al., 2023; Ze et al., 2023).
attributed to the hurdles in
Nevertheless, the predominant methodology for constructing these feature fields in 3D vision- often by leveraging techniques akin to NeRF-results in heavy dependence dense camera views for 2D-3D distillation (Shen et al., 2023; Rashid et al., 2023). This dependency constrains interaction scenarios to those with only sparse views available and impedes rapid training and inference, significantly narrowing the model's 's applicability. Additionally, the operations in existing works are relatively rudimentary, typically involving gripper handling rigid objects, which points to the feature field's limited effectiveness (Simeonov et al., 2022; Shen et al., 2023; Lin et al., 2023; Rashid et al., 2023; Ze et al., 2023).
In this work, we present SparseDFF, a novel approach to generating view-consistent 3D DFFs from sparse RGBD observations, facilitating one-shot learning of dexterous manipulations transferable to new scenes. Our principal insight is that the main limitation for feature fields in manipulation is not a lack of visual information or the expressive capacity of the field model, but rather the consistency of local features. We show that a point cloud-based feature field, with enhancements in feature consistency, can provide precise optimization for a 24 DoFs dexterous hand.
More specifically, we project image features onto 3D point cloud, facilitating their propagation across 3D space to form dense feature field. At the heart of SparseDFF lies a lightweight feature refinement network trained only based on a single demonstration, optimized using contrastive loss between pairwise views. Furthermore, we introduce a continuity within each local area. The resultant feature fields create dense correspondences across varying scenes, enabling the establishment of an energy function on the end-effector pose from the original demonstration the target manipulation. This approach allows for the one-shot learning of dexterous manipulations, adaptable to new scenes with variations in object poses, deformations, scene settings, or even differing object categories. Our methodology is validated through real-world experiments with a dexterous hand interacting with both rigid and deformable objects, exhibiting strong generalizations across various objects and scene settings.
strategy to improve fcature
To summarize, our contributions are threefold:
We introduce novel framework for one-shot learning of dexterous manipulations, leveraging semantic scene understanding distilled into 3D feature fields.
We devise an efficient method for deriving view-consistent 3D features from 2D image models, incorporating a lightweight feature refinement network and point pruning mechanism. This facilitates the application of our network to new scenes, predicting consistent features without requiring any adjustments or fine-tuning,
Our real-world experiments with a dexterous hand affirm our method's effectiveness, demonstrating its robustness and superior generalization ability in varied scenarios.
2 RELATED WORK
Implicit Fields for Manipulation The identification of point-wise correspondences facilitates the transfer of manipulation policies across diverse objects. In contrast to methods based on key points (Manuelli et al., 2019; Xue et al., 2023; Florence et al., 2018), recent efforts focus developing dense feature fields through implicit representations (Simeonov et al., 2022; Wu et al., 2023a; Ryu al., 2022; Dai et al., 2023; Simeonov et al., 2023; Weng et al., 2023; Urain et al., 2023; Zhao et al.,
2

---

Published as a conference paper at ICLR 2024
2022; Wu & Zhao, 2022). The integration of large vision models has propelled research towards leveraging DFFs for enabling few-shot or one-shot policy learning, 6-DOF grasps (Shen et al., 2023), sequential actions (Lin et al., 2023), and language-guided manipulations (Rashid et al., 2023). Yet, these approaches either depend on dense view inputs (Shen et al., 2023; Rashid et al., 2023), requiring extensive camera movement around the scene, or utilize single-view features (Lin et al., 2023), which may suffice for parallel grippers but fall short for dexterous manipulations due to spatial complexities.
Of particular note Ze al. (2023), which, despite using multiple camera views to synthesize unseen views through neural rendering and extracting features from pre-trained models like StableDiffusion, shows advancement in connecting sparse with dense observations. Nonetheless, the processes of synthesizing and propagating unseen views demand significant effort. Moreover, these efforts are predominantly focused on simple manipulations using parallel grippers, with intricate dexterous manipulations largely remaining unaddressed.
Another pertinent work is by Karunratanakul et al. (2020), which employs a signed distance field to illustrate interactions between human hands and objects. Our work parallels Karunratanakul et al. (2020) in optimizing hand parameters from a 3D field. However, unlike their direct depiction of hand-object distances, we construct an implicit feature field to define energy functions for end-effector parameters, offering a novel approach to complex dexterous manipulations.
Distilling 2D Features into 3D The works of Zhi et al. (2021), Siddiqui et al. (2023), and Ren et al. (2022) demonstrate the lifting of semantic information from 2D segmentation networks to 3D, showing that averaging language embeddings over views can produce distinct 3D segmentations. Kobayashi et al. (2022) and Tschernezki et al. (2022) delve into integrating pixel-aligned image features from models like LSeg or DINO (Caron et al., 2021) into 3D Neural Radiance Fields (NeRF), highlighting their impact on manipulating 3D geometry. Further, Peng et al. (2023) and Kerr et al. (2023) explore distilling
wiip---iiiieneeeee image features, such as those from CLIP (Radford et al., 2021). into 3D scenes without the need for fine-tuning, yet their reliance on dense view acquisition for 3D feature extraction poses challenges for scenarios limited to sparse camera setups.
Dexterous Grasping Dexterous manipulation, central to advanced robotic applications, necessitates nuanced understanding and control, akin human-like grasping capabilities (Salisbury Craig, 1982; Dogar & Srinivasa, 2010; Andrews & Kry, 2013; Dafle et al., 2014; Kumar et al., 2016; Qi et al., 2023; Liu et al., 2022). The field prioritizes dexterous grasping due to its foundational role in hand-object interactions. Analytical approaches (Bai & Liu, 2014; Dogar & Srinivasa, 2010; Andrews& & Kry, 2013) focus on direct modeling of hand and object dynamics, offering varying simplification levels, while recent strides in learning- based methods have introduced state (Chen et al., 2022; Christen et al., 2022; Andrychowicz et al., 2020; She et al., 2022; Wu et al., 2023b) and vision-based strategies (Mandikal & Grauman, 2021; 2022; Li et al., 2023a; Qin et al., 2023; Xu et al., 2023; Wan et al., 2023), targeting realistic scene comprehensions.
Despite their advances, these methods depend on large demonstration datasets for training, showing limited generalization beyond trained scenarios. Notably, Wei et al. (2023) proposes a grasp synthesis algorithm that generalizes within similar object shapes using minimal demonstrations, focusing on geometric and physical constraints for functional grasping. Our approach distinguishes itself by em- ploying semantic visual features for dexterous manipulation, facilitating cross-category generalization from a singular demonstration, thus broadening the scope of generalization in dexterous grasping.
3 METHOD
Given a 3D point cloud X, we aim to first construct a continuous feature field f(.X)): R3 - RC surrounding the scene, as illustrated in Fig. 2. This field provides semantic understandings for inter- scene correspondences, extending beyond geometric descriptors (Secs. 3.1 and 3.2). Utilizing a lightweight feature network, trained once on the source scene, enables direct adaptation to target scenes without additional fine-tuning. We then introduce pruning method to enhance feature continuity, akin to the classical Hough voting (Qi et al., 2019). Following this, we employ the demonstrated end-effector pose to formulate an energy function between source and target scenes via the feature fields. This function facilitates the optimization of the end-effector pose in the target scene while conforming physical constraints (Sec. 3.3), depicted in Fig. 3.
3

---

Published as a conference paper at ICLR 2024
Feature fusion
gpun Reflned feat.
Dine
Refine. layers
proj. head O
Contrastive loss
Dimo
O 0+000 Reffine. layers
Feature refinement
proj. bead
Point pruning
Distilled Feature Field
Figure Constructing sparse-view DFFs. (a) Starting with the aggregation of DINO features, we form an initial 3D DFF. (b) Next, lightweight network then refines these features, trained solely on single demonstration and employing contrastive loss to improve field consistency. (c) Finally, a pruning algorithm assesses points through feature similarity in their vicinity. Points with minimal votes eliminated.
3.1 3D FEATURE DISTILLATION
In contrast to previous works (Kerr et al., 2023; Rashid et al., 2023; Ze et al., 2023) that directly reconstruct continuous implicit feature fields alongside NeRF representations, our method first distills features onto discrete 3D points before propagating them into the surrounding space (Fig. 2a), akin to the approach of Kerbl et (2023). Formally, for any given point cloud feature set F [f.). the feature f at query point E IR3 is determined by
1/1g x
f
w.f, where W. Σ 1/11q x,|
(1)
Consider 3D scene observed by several cameras positioned around it, yielding set of K sparsely sampled RGBD scans. Each sean comprises 2D image and 2.5D point cloud X E RRNx33 establishing 1 to correspondences between image pixels and 3D points. Combining the point clouds of K views produces a comprehensive 3D point cloud of the scene X = Us X E RN3
As large vision models such DINO (Caron et al., 2021; Oquab al., 2023) have demonstrated emergent object correspondence properties even when trained without supervision, an intuitive approach is directly apply DINO (Oquab al., 2023) to RGB images and back pack-project each point cloud X using the pixel-point correspondences, yielding per-point features F; RRNxXC integrating the features of all views results in the point features of the entire scene F Uv F& E RNC However, DINO lacks strict multiview invariance, causing local feature inconsistencies.
Addressing the issue of local feature discrepancies, we devise a lightweight feature refineeen network s, consisting shallow per-point MLP, as depicted in Fig. 2b. This network can (i) be efficiently self- self-supervisedly trained on single source scene, (ii) obtain high-qualing consistent feature, and (iii) directly apply in new scenes without any modification. The efficacy of this feature refinement is demonstrated and discussed in Sec. 4.3. Formally, given any pair of 2.5D point clouds X₁ = ) = {x/m) from different views with per-point DINO features fex- fim. applying &p yields the refined features f (ffrn) fy (ffm..) The intuition is ensure that neighboring features are similar and distant ones are distinct, following Xie et al. (2020).
contrastive learning (Chen et al., 2020) to optimize the weights in so. The refined features are passed through a projection head to obtain projected features gen gim- The contrastive learning objective is defined by finding the overlapping region of the views with pairs of points (distance |lmm). In each training iteration, a minibatch of N pairs is randomly sampled as positive examples, with the other 2(N - 1) possible pairs of unmatched points serving negative examples. The contrastive loss for a positive pair of examples (Xte Xlm) is formulated as
Computationally,
exp exp(sim((e) (gen 8/m)
bm = log
(2)
[ exp(sim (gkn: (aim(g)g:::)/7) g()/7)*
with sim(u, u, v) as cosine similarity. 1 [t indicating * n, and 7 as a temperature parameter. After training the source scene, g removed, retaining only şo for novel scenes.
4

---

Published as a conference paper at ICLR 2024
(a) Optimization objective
(b) Optimization process
Test-time Demonstration
Pointeloud
Feature difference
Hand pose 8
Query points QI
((OL8 X
F8g
K(91
Pointeloud
((QI8,X)
End-effector optimization
End-effector optimization
E ()
Figure 3: End-effector optimization. (a) We sample query points on the end-effector and compute their features using the learned 3D feature field. Minimizing the feature differences energy function facilitates the transfer of the end-effector pose from the source demonstration the target manipulation. (b) The color gradient the hand indicates the optimization steps from start end.
3.2 POINT PRUNING
To improve feature consistency within the merged point cloud = {xx), we introduce pruning mechanism akin to Hough voting (Xie et al., 2020), depicted in Fig. 2c. mechanism operates based on refined point features f', focusing on the similarity features among neighboring points.
Specifically, each point is evaluated within radius For every neighboring point x, B(x,, we assess the feature difference between f and f.. If the discrepancy If," (5-f) -fyl falls below predefined threshold 6, then x; secures vote from Xj. The voting is formalized as:
(xx:) = # (x, B(x,r)) [F fi]l < 8).
leading to the exclusion of the bottom 20% of points with the least votes.
This pruning mechanism addresses potential discontinuities between adjacent points from different views, a common issue when integrating DINO features, which are generally continuous within the same image but may diverge across views. By pruning points that lack consensus across views, enhance feature consistency and reliability across the point cloud, mitigating the impact of point cloud noise and DINO feature imperfections.
3.3 END-EFFECTOR OPTIMIZATION
For our dexterous hand end-effector, represented by the joint pose parameters 3, we optimize these parameters in the target scene X using the feature fields derived from both the source demonstration scene X with hand parameters 3, and the target scene; please refer to Fig. 3 for visualization.
Starting with randomly sampling Q points on the hand surfaces in both the source and target manipulation, we generate query point sets Q Q] E RQx3 conditioned on the hand parameters. Prioritizing the fingers for their crucial role in dexterous manipulation, density on them than on the palm. These query points are then processed through our learned 3D feature fields to extract feature sets (. X). f(Q): 3, X) RQxC The objective is to minimize the feature disparity between the demonstration and target hand poses via an loss, formulated as an energy function E(3 X, X, 3) w.r.t. B:
ensure a higher sampling
E(3 1 X) f(Q)3, X)].
We integrate repulsion energy functions to prevent hand-object and self peff---eeetraioss drawing from Wang et al. (2023) to ensure the action's physical viability. The inter-penetration and self-penetration energy functions are as follows:
Eepen (x
Σ
IXE d( x, 2Q), Expen (1)
Σ p.qeQ
lp/q max(6 d)p,
(5)
Moreover, to avoid potential physical damage from extreme hand poses, pose constraint Epose penalizes out-of-limit hand pose. The overall optimization combines these terms:
B X, () + Epen X) + Aspen EspenE) Apose Epose (3).
(6)
In implementation, Apen 10-1, Aspen 10-2 Apase
10-2

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS
We evaluate our model through real-world experiments with a robot hand, opting for direct assessment in real-world settings to leverage the superior stability of large vision models like DINO (Caron et al., 2021; Oquab et al., 2023) on real images over synthetic ones.
Environment Our experimental setup features a Shadow Dexterous Hand, which has 24 Degrees of Freedoms (DoFs). We restrict the 2 DoFs at the wrist, focusing the optimization on the remaining 22 DoF's This hand is mounted on a URI 10e arm, adding 6 Dol's enabling it to reach any point on tabletops sized either Imx 1.2m or 1mx x 1m. Precautions are taken to prevent the dexterous hand from contacting the table surface directly, and the range of motion is limited by the hand's elbow.
For RGBD scans, we use four Azure Kinect DK sensors, positioned at each corner of the table and aimed towards its center, ensuring comprehensive capture of the scene. These sensors are pre- calibrated and fixed at specific heights. Post-processing is applied to the captured scans to remove background elements.
In single-object experiments, the object's 's point cloud is segmented using SAM (Kirillov et al., 2023). For multi-object scenarios, we apply physical constraints to limit the experimental area to the tabletop and employ RANSAC (Fischler & Bolles, 1981) to exclude the table surface from the analysis.
Tasks and evaluation Our method undergoes quantitative evaluation centered on the grasping of both rigid and deformable objects, with grasping providing a straightforward measure of success or failure. Initially, for each trial, demonstration is set up within a virtual environment on an object scan by manually positioning a dexterous hand model on the object's point cloud using MeshLab. After this setup. our feature network takes 20000 iterations for adaptation, roughly 300 seconds using a single NVIDIA GeForce RTX 3090. Once trained, the network is applied unchanged to different real-world scenes to optimize the hand pose for 300 iterations, roughly 20 seconds using a single NVIDIA GeForce RTX 3090.
During testing, object placements are varied within the hand's reach, and our approach tested 10 times to determine its success rate. Each trial begins with deriving an initial grasping pose by our end-effector optimization process, succeeded by a predetermined lifting motion-either directly upwards or a mix of upward and backward movements, depending on the object's physical properties. A trial is deemed successful if the object is securely lifted from the table without being dropped.
Baselines We compare our approach against naive DFF baseline, where DINO image features directly sack-projected onto the point cloud following the method described Peng et al. (2023), with interpolation then applied to populate the 3D space. This comparison leverages the nascent field of one-shot learning for dexterous manipulation, using identical end-effector optimization processes between our method and the baseline for fair comparison. In scenarios with rigid objects, also benchmark against UniDexGraspt* (Wan et al., 2023). Due to UniDexGrasp*+** vision model instability with our real-world, noisy point clouds, we evaluate its state-based model in simulations, conducting 100 trials per setup to gauge success rates.
4.1 RIGID OBJECTS
Our assessment of rigid object grasping is to validate the versatility of our method across different poses, shapes, and object categories. The evaluation focuses on specific setups:
Box: A demonstration with the Cheez-It box from the YCB dataset (Calli et al., 2015b; 2017; 2015a) (ID=3) is utilized for initial evaluation. The testing extends to this same box and another
cracker box, highlighting variations in geometry and appearance.
Drill: The demonstration involves a functional grasp of the Drill from the YCB dataset (ID=35) by its handle. The testing includes the same drill in various poses.
Bowl: For this setup, three 3D-printed bowls are used. The demonstration shows grasp by the rim of Bowll, with subsequent tests on all three bowls, each presented in different poses, including a cat-shaped bowl with unique shape and complex geometry.
Bowl Mug: Demonstrating the method's 's generalization ability, the grasp learned on Bowll is transferred to three distinct 3D- printed Mugs. This setup tests the method's categorical generaliza- tion from Bowls to Mugs, which have related functionalities but distinct geometries.

---

Published as a conference paper at ICLR 2024
Demoustration
Novel
Novel shape
Demonstration
Novel shape
(a)
Novel object category
Demonstration
Novel pose
(b)
Figure 4: Qualitative results on rigid objects grasping. Each panel illustrates the initial grasping pose, determined via end-effector optimization, followed by frame capturing the successful lift-off of the target object. Grasping Box and transferring the skill to Boxes in new poses, including distinct box Box2. (h) functional grasp of drill by its handle. (c) Transferring learned grasp on Bowll to bowls with varied shapes (top row) and cross-category generalization to Mugs (hottom row).
Tab. outlines the suc- cess rates achieved by our method versus the baselines in rigid object grasping tasks. Our ap- proach consistently passes the baseline perfor- mances across all config- urations. While the base-
Table I: Success rates on rigid object grasping. Given the instability of the vision-basse model of UniDexGrasp** (Wan al., 2023) with noisy point clouds, evaluate its state-based model within simulation environments virtually replicated objects. denotes experiments conducted simulation.
Demo.
Boxl
Drill Drill
Bowll Mug.
Target
Boxt
Box2
Bowll
Bowl2
CatBowl
Fleating BeerBarel
UniDexGrappr*
7.7%
66.9%
37.7%
31.9%
26.3%
24.7%
25.5%
6.2%
90%
0%
100%
90%
Ours
100%
100%
100%
100%
80%
60%
80%
40%
90%
lines show promise in simpler scenarios especially when the manipulation target closely matches the demonstrated object-their performance significantly drops the source and target objects become more disparate, notably in transitions like from Bowll to CatBowl Bowll to Mugs. Conversely, method exhibits strong success rates even in these complex transitions.
Fig. showcases our qualitative results. Specifically, the Boxes and Drill from the YCB dataset are depicted in Figs. 4a and 4b, demonstrating our method's ability to generalize across rigid transformations of object poses. The transfer from Box1 to Box2 also illustrated in Fig. 4a, showcasing method's 's udaptability to variations in geometry and appearance. In Fig. 4c, we show the learned grasping from a demonstration on Bowll, extending its applicability to diverse bowls, including the geometrically complex CatBowl (top row), well the cross-category generalization from Bowls to Mugs (bottom row). Refer to Supplementary Videofor additional qualitative results.
4.2 DEFORMABLE OBJECTS
Our evaluation extends to deformable objects to showcase the method's adaptability across various deformations, object types, and scene contexts. The setups include:
This category involves three plush toys-two Bears of different sizes, and a Monkey characterized by its flexibility and elongated limbs. Each toy is demonstrated in specific pose, with subsequent evaluations exploring diverse poses and levels of deformation. Monkey - SmallBear: Additionally, we explore the method's capability to transfer grasping knowledge between the Monkey and the SmallBear. This task underscores the challenges of adapting between different objects and their respective deformations.
SmallBear, BigBear, Monkey:
Monkey in Context: A complex scene setup features the Monkey amidst various background items, with each trial randomizing object placements. This environment tests the method's performance in scenarios where the Monkey may be partially obscured by surrounding objects. In comparison, the grasp demonstration is performed with the Monkey isolated from these complicating factors.
The success rates for the de- formable objects are detailed in Tab. 2. Our method markedly out- performs the baseline, demon- strating exceptional success rates, especially in scenarios that
Table 2: Success rates on deformable object grasping.
Demo.
Monkey MonkeyScene
BigBcar BigBear
SmallBcar SmallBear Monkey
Target
Monkey
SmallBear
DFF
90% 100 %
40% 100%
0% 60 %
20% 80 %
90% %
0%
Ours
90 %
50%

---

Published as a conference paper at ICLR 2024
Demonstration
Novel pose
Novel object
Demonstration
Novel pose
Novel object
(a)
Novel context
Demonstration
Novel pose
(b)
Figure 5: Qualitative results on deformable objects grasping. For each successful grasp, we show the initial grasping pose and frame demonstrating the successful lift of the object off table. (a) Learning to grasp SmallBear and transferring this skill to various poses and to the Monkey. (b) Learning to grasp BigBear by the nose is challenging due to its small nose. (c) Learning to grasp the Monkey, showcasing adaptability to significant deformations and transfers SmallBear. Additionally, challenging scenario is presented where the Monkey surrounded by multiple objects, showing the capability to handle interactions and occlusions.
Demonstration
Novel pose
Demonstration
Novel object
(a) head caressing
(b) butt patting
Figure 6: Pet toy animals. (a) Head caressing transferred from single, lying Monkey scene with the Monkey hugging BigBear, exemplifying the method's adaptability to varying scene compositions and interactions. (b) Butt patting is transferred from the Monkey to the SmallBear, whether the SmallBear alone or in different scene contexts, underlining the method's versatility across various scenarios and object interactions.
quire generalization. These find-
ings are consistent with the outcomes observed in the evaluations involving rigid objects.
Fig. 5 showcases qualitative results in dealing with deformable objects. In Fig. 5a, demonstrate the grasp of a SmallBear toy by its body, illustrating our method's ' capacity for generalization across different poses and Monkey toy, thereby underscoring the versatility of approach. Fig. 5b shows a unique instance of grasping BigBear by its nose, reflecting our method's adaptability to the physical characteristics of diverse objects. Fig. 5c focuses grasping the Monkey toy, emphasizing the method's flexibility with extensive deformations and its ability to generalize from the Monkey to the SmallBear, across varying object forms and structures. Additionally, Fig. 5c presents complex test setup with the Monkey amidst dynamic scene, requiring not just target object identification but also an intricate understanding of its pose, deformation, and relationships with surrounding objects, even in cases of occlusion. These instances highlight our method's 's robustness and flexibility in handling a range of challenging scenarios, showcasing its potential for real-world applications. Further qualitative results and discussion can be found in the Supplementary Video.
Beyond grasping Our framework's capabilities extend beyond simple grasping to encompass a variety of hand-object interactions. Fig. 6 depicts two distinct scenarios involving interactions with toy animals, demonstrating our method's adaptability. In Fig. 6a, the interaction entails caressing the head of Monkey toy in various poses. The initial demonstration features the Monkey lying flat with its arms outstretched, while subsequent tests, adapts to hugging a BigBear, seamlessly adjusting to this new context. Fig. 6b focuses patting the butts of the toys. This action, demonstrated on the Monkey, successfully generalized to the SmallBear in various settings, highlighting our method's ability adapt to different scenarios and object interactions.
4.3 ABLATION STUDIES
Ablations feature refinement Our ablation the feature refinement network, discussed in Sec. 3.1, is visualized in Fig. 7. This network plays crucial role in enhancing the feature field's consistency. The comparison highlights the energy fields used for end-effector pose optimization, showcasing the advantages of incorporating the refinement network. With refinement, we observe a focused distribution of low-energy values at positions matching the hand demonstration, indicating
8

---

Published as a conference paper at ICLR 2024
w/o Teat. refinement
Full model
Source
Target
Source fent.
Target feat.
Energy field
Source feat.
Target feat.
Energy field
Figure 7: Ablation study on feature refinement. Given the source and target objects (left), compare the and target features, the energy fields between model without the feature refinement network (middle), the full model incorporating it (right), Point cloud features are visualized using RGB coloring, derived by applying PCA to reduce the features to three components. The energy fields, calculated based on the feature differences between the source and target scenes, are illustrated to reflect differences: yellow areas signify smaller differences and thus higher similarity, whereas purple areas indicate larger differences, emphasizing feature dissimilarity.
Test input
w/o pruning
Full model
Test input
w/o pruning
Full model
Figure 8: Ablations on point pruning. We show the end-effector grasping poses in the test scene (left), comparing the results with (middle) and without (right) point pruning: the demonstration is omitted for simplicity. Features within the point clouds are visualized in RGB, achieved by reducing feature dimensions to three components using PCA. The optimized hand positions highlighted in blue.
better feature consistency. Conversely, without refinement, the energy distribution appears more scattered, lacking precise low-energy zones.
Ablations on point pruning Fig. 8 qualitatively demonstrates the impact of point pruning (Sec. 3.2) using two examples. This procedure not only eliminates outlier points but also boosts the local feature consistency, significantly enhancing the optimization stability and the accuracy of end-effector poses.
CONCLUSIONS
In this study, tackled the challenge of one-shot learning for dexterous manipulations, harnessing semantic correspondences from pre-trained vision models to enhance robotic grasping capabilities. By distilling sparse-view RGBD observations into a consistent 3D feature field, we developed method that significantly advances the adaptability and generalization of robotic manipulators across various objects and scenes. Our approach not only demonstrated robust generalization new object poses and categories in real-world settings but also showcased the potential for practical applications in dynamic environments. Future work will explore the integration of additional sensory inputs, such tactile feedback, for further improving the precision and reliability of robotic grasping.
Acknowledgment The authors thank NVIDIA for their support of GPUs and hardware. C. Deng, You, and L. Guibas supported in part by the Toyota Research Institute University 2.0 Program and a Vannevar Bush Faculty Fellowship. Q. Wang, H. Zhang, and Y. Zhu are supported in part by the National Science and Technology Major Project (20227D01 14900) and the Beijing Nova Program.

---

Published as a conference paper at ICLR 2024
REFERENCES
Sheldon Andrews and Paul G Kry. Goal directed multi-finger manipulation: Control policies and analysis. Computers & Graphics, 37(7):830-88)) -300, 2013.
OpenAl: Marcin Andrychowicz. Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. International Journal of Robotics Research (IJRR), 39(1):3-00, 2020. 3
Yunfei Bai and C Karen Liu. Dexterous manipulation using both palm and fingers. In International Conference on Robotics and Automation (ICRA), 2014.
Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. The yeb object and model set: Towards common benchmarks for manipulation research. In International Conference Robotics and Automation (ICRA), 2015a. 6
Berk Calli, Aaron Walsman, Arjun Singh, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. Benchmarking in manipulation research: The ycb object and model set and benchmarking protocols. arXiv preprint arXiv: 2015b. 6
Berk Calli, Arjun Singh, James Bruce, Aaron Walsman, Kurt Konolige, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. International Journal of Robotics Research (LJRR), 36(3):261-268. 2017.
dataset for robotic manipulation research.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. International Conference on Computer Vision (ICCV), 2021. 3, 4,
Tao Chen, Jie Xu, and Pulkit Agrawal. system for general in-hand object re-orientation. In Conference Robot Learning (CoRL), 2022. 3
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), 2020. 4
Sammy Christen, Muhammed Kocabas, Emre Aksan, Jemin Hwangbo, Jie Song, and Otmar Hilliges.
D-grasp: Physically plausible dynamic grasp synthesis for hand-object interactions. In Conference Computer Vision and Pattern Recognition (CVPR), 2022. 3
Nikhil Chavan Dafle, Alberto Rodriguez, Robert Paolini, Bowei Tang, Siddhartha Srinivasa, Michael Erdmann, Matthew T Mason, Ivan Lundberg, Harald Staab, and Thomas Fuhlbrigge. Extrinsic dexterity: In-hand manipulation with external forces, In International Conference on Robotics and Automation (ICRA). 2014.3
Qiyu Dai, Yan Zhu, Yiran Geng, Ciyu Ruan, Jiazhao Zhang, and He Wang. Graspnerf: multiview- based 6-dof grasp detection for transparent and specular objects using generalizable nerf. In International Conference on Robotics and Automation (ICRA), 2023. 2
Mehmet R Dogar and Siddhartha S Srinivasa. Push- grab-grasping with dexterous hands: Mechanics and method. In International Conference on Intelligent Robots and Systems (IROS), 2010. 3
Lifeng Fan, Manjie Xu, Zhihao Cao, Yixin Zhu, and Song-Chun Zhu. Artificial social intelligence:
comparative and holistic view. CAAI Artificial Intelligence Research, 1(2):144-160, 2022. 2
Martin Fischler and Robert Bolles. Random sample consensus: a paradigm for model fitting with applications image analysis and automated cartography. Communications of the ACM, 24 (6):381-395, 1981.
Peter R Florence, Lucas Manuelli, and Russ Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. In Conference on Robot Learning, pp. 373-385.
PMLR, 2018. 2
10

---

Published as a conference paper at ICLR 2024
Guangyuan Jiang, Manjie Xu, Shiji Xin, Wei Liang, Yujia Peng, Chi Zhang, and Yixin Zhu. Mewl: Few-shot multimodal word learning with referential uncertainty. International Conference on Machine Learning (ICML), 2023. 2
Korrawe Karunratanakul, Jinlong Yang, Yan Zhang, Michael Black, Krikamol Muandet, and Siyu Tang. Grasping field: Learning implicit representations for human grasps. In International Conference on 3D Vision (3DV), 2020.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (TOG), 42(4):1-14, 2023.
Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. arXiv preprint arXiv:2303.09553 2023. 2, 3,4
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.006443 2023. 6
Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 2,
Vikash Kumar, Emanuel Todorov, and Sergey Levine. Optimal control with learned local models: Application to dexterous manipulation. International Conference on Robotics and Automation (ICRA), 2016.
Brenden M Lake and Marco Baroni. Human-like systematic generalization through a meta-learning neural network. Nature, 15--71, 2023.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science. (136626)::332-13333 2015. 2
Puhao Li, Tengyu Liu, Yuyang Li, Yixin Zhu, Yaodong Yang, and Siyuan Huang. Gendexgrasp: Generalizable dexterous grasping. In International Conference Robotics and Automation (ICRA), 2023a. 1,3
Qing Li, Siyuan Huang, Yining Hong. Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu. A minimalist dataset for systematic generalization of perception, syntax, and semanties. In International Conference on Learning Representations (ICLR), 2023b.
Qing Li, Yixin Zhu, Yitao Liang, Ying Nian Wu, Song-Chun Zhu, and Siyuan Huang. Neural- symbolic recursive machine for systematic generalization. International Conference Learning Representations (ICLR), 2024a. 2
Shiqian Li, Kewen Wu, Chi Zhang, and Yixin Zhu. I-phyre: Interactive physical reasoning. In International Conference on Learning Representations (ICLR), 2024b.
Xingyu Lin, John So, Sashwat Mahalingam, Fangchen Liu, and Pieter Abbeel. Spawnnet: Learning generalizable visuomotor skills from pre-trained networks, arXiv preprint arXiv:2307.03566) 2023. 2,
Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 3
Priyanka Mandikal and Kristen Grauman. Learning dexterous grasping with object-centric visual affordances. In International Conference on Robotics and Automation (ICRA), 2021. 3
Priyanka Mandikal and Kristen Grauman. Dexvip: Learning dexterous grasping with human hand pose priors from video. In Conference Robot Learning (CoRL), 2022.
Lucas Manuelli, Wei Gao, Peter Florence, and Russ Tedrake. kpam: Keypoint affordances for category-level robotic manipulation. The International Symposium of Robotics Research, 2019.
11

---

Published as a conference paper at ICLR 2024
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXhv:230.007193, 200 4, 6
Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In Conference on Computer Vision and Putlern Recognition (CVPR), 2023. 3,
Charles Qi, Or Litany, Kaiming He, and Leonidas Guibas. Deep hough voting for 3d object detection in point clouds. In International Conference on Computer Vision (ICCV), 2019. 3
Haozhi Qi, Ashish Kumar, Roberto Calandra, Yi Ma, and Jitendra Malik. In-hand object rotation via rapid motor adaptation. In Conference on Robot Learning (CoRL), 2023.
Yuzhe Qin, Binghao Huang, Zhao-Heng Yin, Hao Su, and Xiaolong Wang. Dexpoint: Generalizable point cloud reinforcement learning for sim-to-real dexterous manipulation. In Conference Robot Learning (CoRI.), 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021. 3
Adam Rashid, Satvik Sharma, Chung Min Kim, Justin Kerr, Lawrence Yunliang Chen, Angjoo Kanazawa, and Ken Goldberg. Language embedded radiance fields for zero-shot task-oriented grasping. In 7th Annual Conference on Robot Learning, 2023. 2, 3,
Zhongzheng Ren, Aseem Agarwala, Bryan Russell, Alexander G Schwing, and Oliver Wang. Neural volumetric object selection. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 3
Hyunwoo Ryu, Hong-in Lee, Jeong- Hoon Lee, and Jongeun Choi. Equivariant descriptor fields: Se (3) equ-vvaivr maaegy-based models for end-to-end visual robotic manipulation learning. In The Eleventh International Conference on Learning Representations, 2022. 2
J Kenneth Salisbury and John J Craig. Articulated hands: Force control and kinematic issues. International Journal of Robotics Research (IJRR), 1(1):4-17, 1982. 3
Qijin She, Ruizhen Hu, Juzhan Xu, Min Liu, Kai Xu, and Hui Huang. Learning high-dof reaching- and-grasping via dynamic representation of gripper-object interaction. ACM Transactions on Graphics (TOG), 41(4):1-14, 2022.
William Shen, Ge Yang, Alan Yu, Jensen Wong, Leslie Pack Kaelbling, and Phillip Isola. Distilled feature fields enable few-shot manipulation. In Conference on Robot Learning (CoRL), 2023. 2, 3
Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Norman Müller, Matthias Nießner, Angela Dai, and Peter Kontschieder. Panoptic lifting for 3d scene understanding with neural fields. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 3
Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B Tenenbaum, Alberto Rodriguez, Pulkit
Agrawal, and Vincent Sitzmann. Neural descriptor fields: Se (3)-eqvivr object representations for manipulation. In International Conference on Robotics and Automation (ICRA), 2022. 2
Anthony Simeonov, Yilun Du, Yen-Chen Lin, Alberto Rodriguez Garcia, Leslie Pack Kaelbling. Tomás LLozano-Péres and Pulkit Agrawal. Se (3)-equivariant relational rearrangement with neural descriptor fields. In Conference on Robot Learning (CoRL), 2023. 2
Vadim Tschernezki, Iro Laina, Diane Larlus, and Andrea Vedaldi. Neural feature fusion fields: 3d distillation of self-supervised 2d image representations. In International Conference on 3D Vision (3DV), 2022. 3
Julen Urain, Niklas Funk, Jan Peters, and Georgia Chalvatzaki. Se diffiusssioss Learning smooth cost functions for joint grasp and motion optimization through diffusion. In International Conference on Roboties and Automation (ICRA), 2023. 2
12

---

Published as a conference paper at ICLR 2024
Weikang Wan, Haoran Geng, Yun Liu, Zikang Shan, Yaodong Yang. Li Yi, and He Wang. Unidex- grasp++: Improving dexterous grasping policy learning via georetry-aware curriculum and itera- tive generalist-speciall learning. arXiv preprint arXiv:2304.0000 2023. 1,3,6,
Ruicheng Wang, Jialiang Zhang, Jiayi Chen, Yinzhen Xu, Puhao Li, Tengyu Liu, and He Wang. Dexgraspnet: A large-scale robotic dexterous grasp dataset for general objects based on simulation. In International Conference on Robotics and Automation (ICRA), 2023. 5
Wei Wei, Peng Wang. and Sizhe Wang- Generalized anthropomorphic functional grasping with minimal demonstrations. arXiv preprint arXiv:2303. 77808, 2023. 3
Thomas Weng, David Held, Franziska Meier, and Mustafa Mukadam. Neural grasp distance fields for robot manipulation. In International Conference on Robotics and Automation (ICRA), 2023.
Ruihai Wu and Yan Zhao. Vat-mart: Learning visual action trajectory proposals for manipulating 3d
articulated objects. In International Conference on Learning Representations (ICLR), 2022, 2022.
Ruihai Wu, Chuanruo Ning, and Hao Dong. Learning foresightful dense visual affordance for deformable object manipulation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 10947-10956, 2023a. 2
Yueh-Hua Wu, Jiashun Wang, and Xiaolong Wang. Learning generalizable dexterous manipulation from human grasp affordance. In Conference on Robot Learning (CoRL., 2023b. 3
Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-trainnn pre-training for 3d point cloud understanding. European Conference on Computer Vision (ECCV), 2020. 4, 5
Sirui Xie, Xiaojian Ma. Peiyu Yu, Yixin Zhu, Ying Nian Wu, and Cong-Chun Zhu. Halma: Hu- manlike abstraction learning meets affordance rapid problem solving. In ICLR Workshop on Generalization beyond the training distribution in brains and machines, 2021. 2
Yinzhen Xu, Weikang Wan, Jialiang Zhang, Haoran Liu, Zikang Shan, Hao Shen, Ruicheng Wang, Haoran Geng, Yijia Weng, Jiayi Chen, et al. Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 3,3
Zhengrong Xue, Zhecheng Yuan, Jiashun Wang, Xueqian Wang, Yang Gao, and Huazhe Xu. Useek: Unsupervised se (3)-equivariant 3d keypoints for generalizable manipulation. In International Conference on Robotics and Automation (ICRA), 2023. 2
Yanjie Ze, Ge Yan, Yueh-Hua Wu, Annabella Macaluso, Yuying Ge, Jianglong Ye, Nicklas Hansen, Li Erran Li, and Xiaolong Wang. Gnfactor: Multi-task real robot learning with generalizable neural feature fields. In 7th Annual Conference on Robot Learning, 2023. 2, 3,
Yan Zhao, Ruihai Wu, Zhehuan Chen, Yourong Zhang, Qingnan Fan, Kaichun Mo, and Hao Dong. Dualafford: Learning collaborative visual affordance for dual-grrpper manipulation. arXiv e-prints, pp. arXiv-200 2207, 2022.
Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew Davison. In-place scene labelling and understanding with implicit scene representation. In Conference Computer Vision and Pattern Recognition (CVPR), 2021. 3
Yixin Zhu, Tao Gao, Lifeng Fan, Siyuan Huang, Mark Edmonds, Hangxin Liu, Feng Gao, Chi Zhang, Siyuan Qi, Ying Nian Wu, Josh Tenenbaum, and Song-Chun Zhu. Dark, beyond deep: paradigm shift to cognitive ai with humanlike common sense. Engineering, 6(3):310-345, 2020. 2
13


Reviews:
[{"review_id": "q0OTpATc9T", "reviewer": "Reviewer_Txdk", "summary": "The broader goal of the paper is to enable the transfer of robotic dexterous grasps from one object to a similar object (which can happen with understanding the inherent similarities of the 3D shape instances despite the variations in appearances, poses, or categories). More precisely, given a source scene, source hand-object grasp and a target scene, what could be the target grasps.\nTo do so, they leverage 2D vision models (namely DINO) to learn features in 2D space and then distill or back-project those features in 3D. The similarly of features in 3D space allows transfer of grasps from one object to another.\nCompared to prior works, which leverage dense multi-view images to infer dense 2D features for each three 3D point and simply average the multi-view features, the paper operates in sparse view setting. Rather than simply out the point features from multiple view, the paper paper learns a feature refinement network on each point features and defines a contrastive learning approach to bring points closer to each in 3D other more closer in feature space and points farther from each other in 3D space more farther in feature space.\nFinally, leveraging the projected and refined 3D features from sparse views, the paper performs the task mentioned in first bullet, mapping source grasp (on sourc e scene) to target scene.", "questions": "I would like authors to address the points raised in weaknesses section. \n1. What happens when there is noise in pose and/or depth? How will that impact refinement module?\n2. How does the method compare with baselines like neural descriptor fields and follow up works?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Writing is good, and the paper is easy to follow.\nThe motivation of creating a generalized robotic manipulator and leveraging 3D priors for that, seems exciting and promising.", "weaknesses": "[Novelty Issue] Lack of any exciting factor or in some sense novelty: The difference from prior work in DFF (distill feature field) is that the paper operates in sparse view setting, where simple fusion of features from multiple views doesn’t perform best. To overcome this loss of views from prior work, they refine the point features baed on the insight that points close in 3D, should have similar features. This seems a very natural and obvious technique of pruning or refining points which have incorrect feature consistency w.r.t 3D consistency.\n\n[Assumptions on GT depth and camera pose] Secondly, following up on the novelty part, the paper makes the assumption of having GT depth maps and also GT camera poses. Errors in either of them will make the above refinement step tricky.\n\n[Additional experimental comparison] Experimental Comparison against baselines like (Neural descriptor fields and follow-up) where the goal is similar but rather than using the large vision model, an object category specific feature descriptor is learned.\n\n[Less Relevant] Newer papers like Lseg, Conceptfusion have projected LLM/ vision-LLM features on the 3D scenes, comments on using them as feature backbones will be appreciated.\n\n[Less Relevant] Papers learning joint hand object poses, leaning visual affordances from images, also seem to be relevant related works, comparison against them in related work would be appreciated (Affordance Diffusion: Synthesizing Hand-Object Interactions, papers on hand-object interaction: HOI etc).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1Il4S22OY5", "reviewer": "Reviewer_dYVL", "summary": "To mitigate the dense view requirement in the distilled feature field (DFF) for the application of one-shot dexterous manipulation, the authors introduce sparseDFF, which utilizes a sparse collection of RGB-D scans of a scene. Feature points are reprojected using depth, followed by a feature refinement process on these reprojections. Contrastive loss and point pruning are employed to enhance feature consistency within each local neighborhood, and an energy function is formulated to aid in reducing feature discrepancies. Performance on grasping benchmarks demonstrate the proposed methods surpasses DFF while significantly outperforms UniDexGrasp++.", "questions": "See the raised questions in Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The necessity of as few as 4 views for transferring manipulation skills is noteworthy, as this method can be readily generalized to novel scenes.\n- Point feature refinement, the minimization of feature discrepancies using an energy function, and point pruning are specifically designed for applications with RGB-D scans as input.\n- Utilizing DINO feature distillation for diverse downstream grasping tasks markedly outperforms the previous baseline (DFF) and UniDexGrasp++.", "weaknesses": "**[Clearance]** \n- The authors are encouraged to establish connections regarding why distilling DINO features is advantageous and elucidate its applications in downstream tasks.\n- The definition of **one-shot** should be explained in the manuscript (abstract or introduction).\n- The input should accurately be described as \"multi-view RGB-D scans\" rather than \"Given a 3D point cloud X\", in the method section. And the dimension of the variable should be added.\n- Regarding the motivation of using DINO, while the authors have highlighted, \"This field offers semantic understandings for inter-scene correspondences that transcend geometric descriptors\", how does it contribute specifically to image matching deep models like LOFTR?\n\n**[Method]** \nThe authors propose \"discard the 20% of points that accumulate the fewest votes.\".How was this hyper-parameter for the pruning ratio determined? Were multiple-stage or iterative pruning strategies considered?\n\n**[Experiments]** \n- How did the inference performance compare with baseline methods?\n- Was depth information also utilized by DFF?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IqHMrHnH7g", "reviewer": "Reviewer_sPkV", "summary": "This work applies 3D feature fields for manipulation. The key contribution lies in introducing a sparse view setting, where unlike previous works that use dense RGB views, this work uses sparse RGB-D views. A point-based sparse 3D feature field construction method is introduced to improve the 3D information aggregation quality and the grasping task performance.", "questions": "N.A.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The introduction of sparse RGBD camera setting.\nThe method of sparse DFF to reconstruct 3D feature fields from sparse RGBD inputs.\nReasonable experiment design and analysis.", "weaknesses": "There are existing sparse-view NeRF methods (e.g., [1,2]) that applies similar ideas as the sparse DFF, some of which are not extremely hard to apply to the normal DFF (e.g,, [1]). It is fairer to allow baselines to also utilize the depth information introduced in this work (e.g., introducing depth supervision similar as [1] in DFF). \n\n[1] Depth-supervised NeRF: Fewer Views and Faster Training for Free\n[2] MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zJEZ4hapaN", "reviewer": "Reviewer_hwb9", "summary": "This paper presents an innovative method to obtain view-consistent 3D DFFs from sparse RGBD data, enabling one-shot learning of complex manipulations that can be adapted to unfamiliar settings. The key contribution of SparseDFF comprises a lightweight feature refinement network, optimized using a contrastive loss applied to pairs of views after projecting image features onto the 3D point cloud. Furthermore, by establishing consistent feature fields in both the source and target scenes, they design an energy function that simplifies the process of minimizing feature differences with respect to the end-effector parameters between the demonstration and the target manipulation.", "questions": "### Question 1:\nIn the contrast learning process, a distance of 1cm is set as the threshold to distinguish between similar and dissimilar parts. Can the authors provide clarification on how they precisely define this distance?\n### Question 2:\nIs distance truly an effective criterion for distinguishing between similarity and dissimilarity?\n### Question 3:\nIs there a typographical error in Equation (3)? Why does it contain both an equation symbol and an inequality symbol?\n### Question 4:\nAdditionally, is the pruning process defined in Equation (3) considered reasonable?\n### Question 5:\nCould this pruning process potentially lead to the removal of critical edge information?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper presents a captivating approach to 3D feature learning, involving the creation of a point-cloud-based 3D representation and the utilization of the DINO feature extractor. This 3D representation based on point clouds can enable one-shot dexterous manipulation. As demonstrated by the experimental results, the proposed method exhibits robust performance across various settings.", "weaknesses": "In light of the experimental findings presented in this paper, it is respectfully suggested that the method described may not be particularly captivating. For a more comprehensive critique, kindly refer to the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)