Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation
Authors: William Shen
Ge Yang
Alan Yu
Jansen Wong
Leslie Pack Kaelbling
Phillip Isola
Venue: CoRL 2023 Oral (2023)
Keywords: Neural Fields
Foundation Models
Scene Understanding
Robot Manipulation

Abstract: Self-supervised and language-supervised image models contain rich knowledge of the world that is important for generalization. Many robotic tasks, however, require a detailed understanding of 3D geometry, which is often lacking in 2D image features. This work bridges this 2D-to-3D gap for robotic manipulation by leveraging distilled feature fields to combine accurate 3D geometry with rich semantics from 2D foundation models. We present a few-shot learning method for 6-DOF grasping and placing that harnesses these strong spatial and semantic priors to achieve in-the-wild generalization to unseen objects. Using features distilled from a vision-language model, CLIP, we present a way to designate novel objects for manipulation via free-text natural language, and demonstrate its ability to generalize to unseen expressions and novel categories of objects. Project website: https://f3rm.csail.mit.edu

Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation
William Shen"l, Ge Yang* Alan Yu', Jansen Wong',
Leslie Pack Kaelbling', Phillip Isola'
MIT CSAIL, Institute for Artificial Intelligence and Fundamental Interactions
Abstract: Self-supervised and language-supervised image models contain rich knowledge the world that is important for generalization. Many robotic tasks, however, require a detailed understanding of 3D geometry, which is often lacking in 2D image features. This work bridges this 2D-to-3D gap for robotic manip- ulation by leveraging distilled feature fields to combine accurate 3D) geometry with rich semantics from 2D foundation models. We present few-shot learning method for 6-DOF grasping and placing that harnesses these strong spatial and semantic priors to achieve in-the-wild generalization to unseen objects. Using fea- tures distilled from a vision- -aiiunla model, CLIP, we present a way to designate novel objects for manipulation via free-text natural language, and demonstrate its ability to generalize unseen expressions and novel categories of objects. Project website: https: //f3rm. csail. mit.
Introduction
What form of scene representation would facilitate open-set generalization for robotic manipulation systems? Consider warehouse robot trying to fulfill an order by picking up an item from cluttered storage bins filled with other objects. The robot is given product manifest, which contains the text description it needs to identify the correct item. In scenarios like this, geometry plays an equally important role as semantics, as the robot needs to comprehend which parts of the object geometry afford a stable grasp. Undertaking such tasks in unpredictable environments - where items from diverse set can deviate markedly from the training data, and can be hidden or jumbled amidst clutter underscores the critical need for robust priors in both spatial and semantic understanding.
In this paper, we study few-shot and language-guidde manipulation, where a robot is expected to pick up novel objects given few grasping demonstrations or text descriptions without having previously seen a similar item. Toward this goal, we build our system around pre-trained image embeddings, which have emerged as reliable way to learn commonsense priors from internet-scale data [1, 2, 3].
Figure illustrates how our system works. The robot first scans a tabletop scene by taking sequence of photos using an RGB camera mounted on a selfie stick (Figure 1, left). These photos are used to construct a neural radiance field (NeRF) of the tabletop, which, crucially, is trained to render not just RGB colors but also image features from a pre-trained vision foundation model [4, 1]. This produces scene representation, called Distilled Feature Field (DFF), that embeds knowledge from 2D feature maps into a 3D volume (Figure 1, middle). The robot then references demonstrations and language instructions grasp objects specified by user (Figure right).
Distilled Feature Fields (DFFs) were introduced in computer graphics for tasks such as decomposing and editing images [5, 6]. The main contribution of this work is to study the use of DFFs instead for robotic manipulation. We evaluate the robot's s ability to generalize using features sourced from self-super vision transformers (DINO ViT, see [4]). These features have been shown to be effective out-of-the-box visual descriptors for dense correspondence [7]. We also source features
Equal contribution. Correspondence to [willshen, gaaaeeenngggngggeeee dcwail. mit. edu
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Baymax
3D l'eature Field
Extract Dense Features
Coffee Mug
S.Cnan Scene
Distill Features
3. Language Guided Manipulation
Figure 1: Distilled Feature Fields Enable Open-Ended Manipulation. (1) Robot uses selfie stick to sean RGB images of the scene (camera frustums shown).
(2) Extract patch-level dense features for the images from 2D foundation model, and distill them into a feature field (PCA shown) along with modeling a NeRF. (3) We can query CLIP feature fields with language to generate heatmaps and infer 6-DOF grasps on novel objects given only ten demonstrations.
from vision- laiguuge model, CLIP [1], which is strong zero-shot learner on various vision and visual avrestion-answere tasks.
One challenge that makes distilled feature fields unwieldy for robotics is the long time takes to model each scene. To address this, we build upon the latest NeRF techniques, and employ hierarchi- cal hashgrids significantly reduce the modeling time [8, 9, 10]. When it comes to vision-language features, CLIP is trained to produce image-level features, whereas 3D feature distillation requires dense 2D descriptors. Our solution is to use the MaskCLIP [11] reparameterization trick, which extracts dense patch-level features from CLIP while preserving alignment with the language stream.
We demonstrate that Distilled Feature Fields enable open-ended scene understanding and can be leveraged by robots for 6-DOF object object manipulation. We call this approach Feature Fields for Robotic Manipulation (F3RM). We present few-shot learning experiments where robot is able to handle open-set generalization to objects that differ significantly in shape, appearance, materials, and poses. We also present language-guided manipulation experiments where our robot grasps or places objects in response to free-text natural language commands. By taking advantage the rich visual and language priors within 2D foundation models, our robot generalizes to new categories of objects that were not seen among the four categories used the demonstrations.
grasping and placing tasks,
2 Problem Formulation
We consider the class of manipulation problems that can be parameterized via single rigid-body transformation T E SE(3), and focus on grasping and placing tasks. We parameterize 6-DOF grasp or place pose as = (R, t) in the world frame (see Figure 2), where R is the rotation matrix, and is the translation vector. each scene, the robot is given set of RGB images (I} with their corresponding camera poses.
Few-Shot Manipulation. We aim to build robots that can manipulate objects given only a few demonstrations of a task, such as grasping mug by its handle. During learning, each demonstration D consists of the tuple ([I), T where [I) are RGB camera views of the scene and T is a pose that accomplishes the desired task. During testing, the robot is given multiple images (I') of new scene which may contain distractor objects and clutter. The robot's goal to predict pose T that achieves the task. We want test for open-ended generalization: the new scene contains related but previously unseen objects that differ from the demo objects in shape, size, pose, and material.

---

Open- Language-Guided Manipulation. We extend few-shot manipulation to include open- text conditioning via natural language. Given a text description of an object, the robot's objective to grasp the objects that match this description. The robot has access to a few demonstrations for multiple object categories. During lesting, the user provides the robot with a text query L* to specify which object manipulate and negative texts 4 reject distractors. In practice, L can be sampled automatically. The object category care about at test time does not necessarily appear in the demonstrations. We explicitly seek open-ended generalization to new object categories, given just few demonstrations limited to small class of object categories.
3 Feature Fields for Robotic Manipulation (F3RM)
We present Feature Fields for Robotic Manipulation (F3RM), our approach for distilling pre- pre-trained representations from vision and vision-language models into 3D feature fields for open-ended robotic manipulation. Doing so involves solving three separate problems. First, how to produce the feature field of a scene automatically at reasonable speed; second, how to represent and infer 6-DOF grasping and placing poses; and finally, how to incorporate language guidance to enable open-text commands. We include a formal summary of NeRFs [12] in Appendix A.I.
3.1 Feature Field Distillation
Distilled Feature Fields (DFFs) [5, 6] extend NeRFs by including an additional output to reconstruct dense 2D features from vision model fiis- The feature field f is a function that maps 3D position x to a feature vector (x). We assume that f does not depend on the viewing direction d.' Supervision of this feature field f is provided through the rendered 2D feature maps, where each feature vector given by the feature rendering integral between the near and far plane (t,, and Ey):
F(r) - f.' (()))()))(()))) di with T()) exx (-f(o(t)))b)))
where is the camera ray corresponding particular pixel, is the distance along the ray, is the density field from NeRF, and [t)) represents the accumulated transmission from the near plane tn to t. Observe this is similar to the volume rendering integral (Eq. 5).
Feature Distillation. We begin with set of 2D feature maps {1 Jim where I/ = fris (I) for each RGB image I. We optimize by minimizing the quadratic loss Licat = ErER [F (r) [| where I/ (r) is the target feature vector from I', and (rr) is estimated by a discrete approximation of the feature rendering integral in Eq. I.
Extracting Dense Visual Features from CLIP. tures from ViTs is to use the key/value embeddings from the last layer before pooling [4]. While this approach has yielded great results for vision-only models on image segmentation and image- to-image correspondence tasks [7, 15], it removes the transformations needed to project the visual features into the shared feature space with the language stream in lisoon-language models such as CLIP [1, 16). Re-aligning the dense features typically requires additional training, which negatively affects the model's open-text generalization.
The common practice for extracting dense fea-
Rather than using CLIP as image-level feature (Alg. 1, App. A.2), we extract dense features from CLIP using the MaskCLIP reparameterization trick (Alg. 2) [11]. These features retain a sufficient alignment with the language embedding to support zero-shot language guidance our experiments (see Fig.7). We present pseudocode for this technique in Appendix A.2. A second modification we apply is to interpolate the position encoding (see 4) to accommodate larger images with arbitrary aspect ratios. This is needed because CLIP uses small, fixed number of input patches from square crop. These two techniques combined enable us extract dense, high-resolution patch-level 2D features from RGB images at about 25 frames per second and does not require fine-tuning CLIP.
Prior work that uses the association of image pixels in 3D to supervise learning 2D image features makes the same assumption [13, 14].
3

---

Festuros Query Points
Task Emmhediing
Demo Emboediigs
ZM
1
Query Points
TL
Esample Object
Feature Field
DOF Gripper Pose
(a) Collect Demonstrations in VR
(b) Sample Feature Vectors (c) Average Over n Demos
Figure 2: Representing 6-DOF Poses. (a) Recording the gripper pose T in virtual reality (VR) on an example mug. (b) We approximate the continuous local field via a fixed set of query points in the gripper's s canonical frame. (c) We concatenate feature vectors these query points, then average over n (we use n = 2) demonstrations. This gives a task embedding ZM for the task M.
3.2 Representing 6-DOF Poses with Feature Fields
We wish represent the pose of the gripper in demonstration by the local 3D feature field in the gripper's coordinate frame. We approximate this local context via discrete set of query points and the feature vectors measured at each point. We sample a fixed set of Ng query points X' = {x E R*)N, in the canonical gripper frame for each task M from a 3D Gaussian. We adjust its mean and variance manually to cover parts the object we intend to target, as well as important context cues (e.g.. body of the mug when grasping the handle) and free-space (Fig.2b). For a 6-DOF gripper pose T. we sample the feature field f each point in the query point cloud, transformed by T (Fig.2b).
To account for the occupancy given by the local geometry, we weigh the features by their corre- sponding alpha values from the density field a of the NeRF model, integrated over the voxel. At point x in the world frame, this produces the a-weighted features
fa (x) axx) f(x), where (xx)
exp(-o(x) E (0, 1),
and 8 is the distance between adjacent samples. We sample of features (fa(x) x E TX} using the transformed query points TX, and concatenate along the feature-dimension into vector, ZT E If), The query points X' and demo embedding ZT thus jointly encode the demo pose T.
We specify each manipulation task M by a set of demonstrations (D}. We average zT over demos for the same task obtain task embedding ZM N,,1 (Fig. 2c). This allows us to reject spurious features and focus relevant parts of the feature space. This representation scheme is similar to the one used in Neural Descriptor Fields [17]. The main distinction is that NDF is trained from scratch on object point clouds, whereas our feature field is sourced from 2D foundation models that are trained over internet-scale datasets. The capabilities that emerge at this scale hold the potential for open-ended generalization beyond the few examples that appear in the demonstrations.
Inferring 6-DOF Poses. Our inference procedure involves coarse pre-filtering step for the trans- lational DOFs, and an optimization- a dense voxel grid over the workspace, where each voxel v has a grid-size 8. We remove free space by rejecting voxels with alphas a(v) < éfree We then remove voxels that are irrelevant to the task, using the cosine similarity between the voxel feature (v) and the task embedding ZM. To get the complete 6-DOF poses T = (T), we uniformly sample N, rotations for each remaining voxel v.
fine-tuning step for the rotational DOFs. First, sample
Pose Optimization.
We optimize the initial poses with the following cost function
Jpose (T) cos 2T. Zar)
(3)
using the Adam optimizer [18] to search for poses that have the highest similarity to the task embed- ding Zar. After each optimization step, we prune poses that have the highest costs. We also reject poses that in collision by thresholding the number of overlapping voxels between voxelized gripper model and the scene geometry. This leaves us with ranked list of poses that we feed into motion planner in PyBullet [19, 20). We execute the highest-ranked grasp or place pose that has

---

T
be the text embeddings of the negative queries, We compute the softmax over the pair-wise cosine Pick up the Bowl
the
User
Text Festures
User
Teat Features
arg mix
X
fron CLIP
Fealuncs st Query Points
d
Selected
T Lang!
Jooce
minimine
Avergge over XcTX
Demo
(a) Retrieving Demonstrations
(b) Language-Guided Pose Optimization
Figure 3: Pipeline for Language-Guided Manipulation. (a) Encode the language query with CLIP, and compare its similarity to the average query point features over set of demos. The mug lip demos have the highest similarity to "Pick up the Bowl". (b) Generate and optimize grasp proposals using the CLIP feature field by minimizing Fung- We use the selected demo from (a) in Tpese. and compute the language-guidance weight with the text features and average query point features.
valid motion plan. Observe that our scheme operates over the entire feature field, and does not rely on assumptions about objectness such segmentation masks object poses.
3.3 Open-Text Languagee-Guiee Manipulation
Natural language offers a way to extend robotic manipulation to an open-set of objects, serving as an attractive alternative when photos of the target object are inaccurate or unavailable. our language-guided few-shot manipulation pipeline, the learning procedure and the representation for the demonstrations remain consistent with Section 3.2. At test time, the robot receives open-text language queries from the user that specify the object of interest manipulate. Our language- guided pose inference procedure comprises three steps (see Fig.3): (i) retrieving relevant demonstrations, (ii) initializing coarse grasps, and (iii) language-guided grasp pose optimization.
Retrieving Relevant Demonstrations. We select the two demonstrations whose average feature F2 (averaged among the query points of each demo pose T*) is closest to the text embedding = embcLıP(Lt) (Fig.3u). We found that using the positive query text (L+) alone is sufficient. This means finding the demonstration that maximizes the cosine similarity cos(q. Fa). Note that the objects used in the demonstrations do not have to come from the same category as the target object. For instance, asking the robot to pick up the "measuring beaker" or "bowl" leads to the robot choosing the demonstration of picking up mug by its lip (Fig.4).
Initializing Grasp Proposals. We speed up grasp pose inference by first running a coarse proposal step where we filter out regions the feature field that are irrelevant to the text query. We start by sampling a dense voxel grid among the occupied regions by masking out free space (see Sec.3.2). Afterward, we prune down the number of voxels by keeping those more similar to the positive query E (1.....n)
1 than any one of the negative queries I._ Formally, let q, = embuup [,,]
similarity between the voxel's feature f, (v) and the ensemble (q. qí 9 ... , and identify the closest negative query q We remove voxels that are closer to q than the positive query q. The cosine similarity between the voxel embedding and [q. pair forms a binomial distribution that allows us reject voxels that have < 50% probability of being associated with the negative query. Finally, to get the set of initial poses T (T), we sample N, rotations for each remaining voxel.
Landuage-Guided Grasp Pose Optimization. To incorporate language guidance, we first com- pute Jpose from Eq.3 using the two demonstrations retrieved in the first step. We then assign lower cost to regions that are more similar to the language query q by computing language-guidang weight C₂ mean (q @ (x)). and multiply it with Jpone (Fig.3b) x
Jang T) mean q @ [x(x)
Jpone( (T).
(4)
TE
5

---

(a) Top Grasps
(b) Robot Execution
(a) Grasp mug (lip)
(b) Grasp screwdriver
(c) Grasp caterpillar
(d) Place cup on rack
(c) Top 10 Grasps
(d) Robot Exccution.
Figure 4: Five Grasping and Place Tasks. (a) grasping a mug by its lip or handle (Fig.2); (b) screwdriver by the handle; (c) the caterpillar by its ears; and (d) placing a cup onto a drying rack. Gripper poses indicate one of two demonstrations.
Figure 5: Generalizing to Novel Objects. (Top Row) Mug is much bigger than the ones used for demonstration. (Bottom Row) This rack has shorter pegs with a square cross- section. Demo rack cylindrical (cf. Fig.4d).
The first term, C.. is the normalized cosine similarity between the text embedding q and the average a-weighted query point feature for pose T. We iteratively update the pose T via gradient descent while pruning using the procedure from Section 3.2 till convergence.
4 Results
4.1 Learning to Grasp from Demonstrations
We consider five 6-DOF grasping and placing tasks and provide two demonstrations per task (Fig.4). To label the demonstrations, we load NeRF-reconstruuted point cloud into virtual reality, and use hand controller move the gripper to the desired pose (Fig.2a). We compare the performance of three types of distilled features: (1) DINO VIT, (2) CLIP ViT, and (3) CLIP ResNet. We consider three baselines, including (1) using density from the NeRF, (2) the intermediate NeRF features, and (3) the RGB color value as features, and compare against MIRA [21], recent work which uses NeRFs to render orthographic viewpoints for pixel-wise affordance predictions. For each task, we evaluate in ten scenes that contain novel objects in arbitrary poses and distractor objects. The novel objects belong to the same or related object category as the demo objects, but differ in shape, size, material and appearance. We reset the scenes about the same configuration for each compared method. We include the full details on the experimental setup in Appendix A.4.
We present the success rates in Table and examples of robot executions in Figure 5. While the baselines using density, RGB color values, or intermediate features from NeRF achieve respectable performance, they struggle to identify the semantic category of the objects we care about, especially in complex scenes with more distractors. We find that DINO and CLIP feature fields exhibit im- pressive generalization capabilities and have complementary advantages. The DINO ViT has good part-level understanding of object geometry with 7/19 failure cases caused by inaccuracies in the grasp rotations and occasionally, the translations. In comparison, 21 /2 failures for CLIP ViT and ResNet combined may be attributed to this issue. We find that CLIP favors semantic and categorical information over geometric features, which are essential for grasping and placing objects. DINO, on the other hand, struggles with distinguishing target objects from distractor objects that contain sim- ilar visual appearance to the objects used in the demonstrations. CLIP struggles less in this regard. The fusion between semantic features and detailed 3D geometry offers way to model multiple objects piled tightly together: in Figure 6b, for instance, caterpillar toy is buried under other toys. Figure 6c shows robot grasping the caterpillar, and dragging it from the bottom of the pile.

---

(a) Demonstration (1 of
(b) Feature Field of Cluttered Scene
(c) Robot Execution
Figure 6: Grasping a Cluttered Scene. (a) Demonstration for grasping the caterpillar in its
DINO feature field (color is PCA, red dots show query points). (b) A cluttered scene with several toys on top of each other. Inset shows the top 10 inferred grasps. Observe the caterpillar's ears share the same features with the demo. (c) Robot successfully grasps the caterpillar.
Mug lip
Mug handle
Caterpillar car
Screwdriver
Cup on rack
Total
handle
Color 7/10
Material
7/10
MIRA [21]
1/10
2/10
6/10
3/10
3/10
15/50
Relational
4/10
Density
5/10
5/10
10/10
2/10
5/10
27/50
Intermedlate
2/10
2/10
1/10
3/10
1/10
9/50
General
4/10
RGB
4/10
3/10
9/10
1/10
4/10
21/50
OOD
9/10
DINO VIT
5/10
4/10
8/10
6/10
8/10
31/50
Total 3//50
CLIP VIT
7/10
7/10
8/10
6/10
6/10
34/50
CLIP ResNet
9/10
6/10
9/10
8/10
7/10
39/50
Table 2: Success rates of Language-Guided Ma- nipulation. Language query success rates across semantic categories.
Table 1: Success rates on grasping and placing tasks. We compare the success rates over ten evaluation scenes given two demonstrations for each task. We consider run successful if the robot grasps or places the correct corresponding object part for the task.
4.2 Language-Guided Object Manipulation
We set up 13 table-top scenes to study the feasibility of using open-text language and CLIP feature fields for designating objects to manipulate. We reuse the ten demonstrations from the previous section (Sec. 4.1), which span four object categories (see Fig.4). We include three types of objects in our test scenes: (1) novel objects from the same categories as the demonstrations, (2) out-of- distribution (OOD) objects from new categories that share similar geometry as the demonstrated items (e.g.. bowls, measuring beakers, utensils), and (3) distractor items that we desire the system ignore. Success metric: we consider a language query successful if the robot stably grasps the target object and places it in plastic bin at known location. We include more details in Appendix A.7.
We break down the success rates by category in Table 2, and show the robot's s execution sequence for an example scene in Figure (video). This scene contained eleven objects, four were sourced from the YCB object dataset (the apple, black marker, mango, and a can of SPAM), the rest collected from the lab and bought online. We present five successful grasps (Figure 7). The robot failed to grasp the stainless steel jug by its handle due a small error in the grasp rotation. This is a typical failure case six out of 19 failures stem from these poor grasp predictions with rotational or translational errors. The remaining 13/19 failed grasps are due to CLIP features behaving like bag-of-words and struggling to capture relationships, attributes, and ordinal information within sentences [22]. For instance, in queries such as "black screwdriver" and "mug on a can of spam," " CLIP paid more attention to other black objects and the can of spam, respectively. We found that retrieving demos via text occasionally (particularly in the rack scene) benefits from prompt engineering.
In total, our robot succeeds in 31 out of 50 language queries, including both fairly general queries (e.g.. mug, drying rack) and ones that specify properties, such as color, material, and spatial rela- tions (c.g., screwdriver on the block). Notably, our robot generalizes to out-of-distribuiia object categories including bowls, rolls of tape, whiteboard markers and utensils using demonstrations only on mugs and screwdrivers. Although this success rate far from practical for industrial use, overall strategy of using 2D visual priors for 3D scene understanding leverage the rapid advancements in VLMs, which hold significant potential for improving performance.
7

---

"Baymax"
"olue screwdriver"
"bowl"
"red screwdriver" "transpurent jug"
Figure 7: Language-Guider Guided Manipulation Execution. (Top Row) Heatmaps given the language queries. (Bottom Row) Robot executing grasps sequentially without rescanning. CLIP can behave like bag-offwwod words, shown by the bleed to the blue bowl for "blue screwdriver."
Related Work
Open-Ended Generalization via Language. number of prior work use natural language for fask-seecification and long-horizon planning in applications such as tabletop and mobile manipula- tion [23, 24, 25, 26] and navigation [27, 28). A recent line of work seeks to replicate the success of vision and language models in robotics, by jointly pre-training large foundation models on behavior data [29, 30). The goal of our work is different. F3RM seeks to incorporate geometric information with any pre-trained features by lifting imagery and language prior knowledge into 3D.
Dense 2D Visual Descriptors. [31, 32] use dynamic 3D reconstruction from RGB-D videos to provide association labels between pixels from different video frames. Dense Object Nets eliminate the need for dynamic reconstruction, using multi-view RGB-D images of static scenes in the context of robotics [13, 33J. NeRF-Superrision extends descriptor learning to thin structures and reflective materials via NeRFs for 3D reconstruction [14]. In contrast, recent work indicates that excellent dense correspondence can emerge at a larger scale without the need for explicit supervision [4, 34].
Geometric Aware Representations for Robotics. Geometric understanding is essential part of mapping [35, 36, 37], grasping [38, 39, 17, 40]. and legged locomotion [41]. These work either require direct supervision from 3D data such as point clouds, or try to learn representations from posed 2D images or videos [13, 14]. [21, 42, 43] leverage neural scene representations to take advantage of their ability to handle reflective or transparent objects and fine geometry. Our work incorporates pre-trained vision foundation models to augment geometry with semantics.
3D Feature Fields. number of recent work integrate 2D foundation models with 3D neural fields
contexts other than robotic manipulation [44, 45, 46, 47, 48]. See Appendix A.8 for comprehen- sive overview. Our work shares many similarities with LERF [49]. However, unlike the multi-scale image-level features used in LERF, we extract dense patch-level features from CLIP. Additionally, we take step further by exploring the utilization of feature fields for robotic manipulation.
6 Conclusion
We have illustrated way to combine 2D visual priors with 3D) geometry to achieve open-ended scene understanding for few-shot and language-guided robot manipulation. Without fine-tuning, Distilled Feature Fields enable out-of-the box generalization over variations in object categories, material, and poses. When the features are sourced from vision-language models, distilled feature fields offer
various levels of semantic granularity.
Limitations. Our system takes Im 40s to collect 50 images of the scene, and 90s to model the NeRF and feature field. This highlights the need to develop generalizable NeRFs that can recover geometry quickly with just few views [9, 43), opening the possibility for closed-loop dynamic ma- nipulation. More generally, novel view synthesis generative process too different from image generation with GANs [50] and diffusion models [51]. These alternatives, to which our philosophy equally applies, hold promise for solving general-purpose visual and geometric understanding.

---

Acknowledgement
We gratefully acknowledge support from Amazon.com Service LLC, Award #2D-06310236, from National Science Foundation under Cooperative Agreement PHY-2019786 (The NSF Al stitute for Artificial Intelligence and Fundamental Interactions, http://aiff.org/ from NSF grant 2214177; from AFOSR grant FA9550-22-1 ----0-29----, from ONR MURI grant N00014-22-1-27400 from ARO grant W91 INF-NN-F---- 22- 1-0034; from the MIT-IBM Watson Lab; and from the MIT Quest for In- telligence. The authors also thank Tomás Lozano-Pérez, Lin Yen-Chen and Anthony Simeonov for their advice; Boyuan Chen for initial discussions; Rachel Holladay for her extensive help with setting the robot; and Tom Silver for providing feedback on an earlier draft.
References
[1] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International Conference on Machine Learning. PMLR, 2021.
[2] G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. OpenCLIP, July 2021. URL. poctps://dol org/10. 50/1/521//ee/ /ee//// 5143773.
[3] C. Jia, Yang, Y. Xia, Y.-T. Chen, Parekh, H. Pham, Le, Y.-H. Sung, 7. Li, and Duerig- Scaling Up Visual and Vision-Language Representation Learning with Noisy Text Supervision. In International Conference Machine Learning. PMLR, 2021.
[4] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, 2021.
[5] V. Tschernekki, I. Laina, D. Larlus, and A. Vedaldi. Neural Feature Fusion Fields: 3D Dis- tillation of Self-Supervised 2D Image Representations. In Proceedings of the International Conference on 3D Vision (3DV), 2022.
[6] S. Kobayashi, E. Matsumoto, and V. Sitzmann. Decomposing NeRF for Editing via Feature Field Distillation. Advances in Neural Information Processing Systems, volume 35, 2022.
[7] S. Amir, Y. Gandelsman, S. Bagon, and T. Dekel. Deep ViT Features Dense Visual Descrip- tors. ECCVW What is Motion For?, 2022.
[8] T. Müller, A. Evans, C. Schied, and A. Keller. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Transactions on Graphics (ToG), 2022.
[9] A. Yu, V. Ye, M. Tancik, and A. Kanazawa. pixelNeRF: Neural radiance fields from or few images. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, 2021.
[10] M. Tancik, E. Weber, E. Ng. R. Li, B. Yi, J. Kerr, T. Wang, A. Kristoffersen, Austin, K. Salahi, A. Ahuja, D. McAllister, and A. Kanazawa. Nerfstudio: A Modular Framework for Neural Radiance Field Development. In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH '23, 2023.
[11] C. Zhou, C. C. Loy, and B. Dai. Extract free dense labels from clip. In European Conference on Computer Vision (ECCV), 2022.
[12] B. Mildenhall, P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing Scenes Neural Radiance Fields for View Synthesis. In European Conference on Computer Vision (ECCV), 2020.

---

[13] P. R. Florence, L. Manuelli, and R. Tedrake. Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation. In Proceedings of the 2nd Conference on Robot Learning (CoRL), June 2018.
[14] L. Yen-Chen, P. Florence, Supervision: Learning dense object descriptors from neural radiance fields. In IEEE Con- ference on Robotics and Automation (ICRA), 2022.
Barron, T.-Y. Lin, A. Rodriguez, and P. Isola. NeRF-
[15] M. Hamilton, Z. Zhang. B. Hariharan, N. Snavely, and W. T. Freeman. Unsupervised Semantic Segmentation by Distilling Feature Correspondences. In International Conference on Learning Representations, 2022. URL htw.s: //tp:////////eeieeerree
[16] Y. Rao, W. Zhao, G. Chen, Y. Tang, Z. Zhu, G. Huang, J. Zhou, and J. Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, 2022.
[17] A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitz- mann. Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation. In IEEE Conference on Robotics and Automation (ICRA), 2022.
[18) D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
[19] E. Coumans and Y. Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://pybullet.com org, 2016.
[20] C. Garrett. PyBullet Planning. https: com/ hwtp:////a///pppplttteelaaaaaaia
2018.
[21] L. Yen-Chen, Florence, A. Zeng. J. T. Barron, Y. Du, W.-C. Ma, A. Simeonov, A. R. Garcia, and P. Isola. MIRA: Mental imagery for robotic affordances. In Conference on Robot Learning (CoRL), 2022.
[22] M. Yuksekgonul, Bianchi, P. Kalluri, D. Jurafsky, and J. Zou. When and Why Vision- Language Models Behave like Bags-Or-Words, and What Do About It? In International Conference Learning Representations, 2023. URL https://openeee id=KRLUvxh8uaX.
net/forum?
[23] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021.
[24] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Proceedings of the 6th Conference on Robot Learning (CoRL), 2022.
[25] S. Karamcheti, S. Nair, A. S. Chen, T. Kollar, C. Finn, D. Sadigh, and P. Liang. Language- driven representation learning for robotics. In Robotics: Science and Systems (RSS), 2023.
[26] S. Stepputtis, J. Campbell, M. Phielipp, S. Lee, C. Baral, and H. Ben Amor Language- Conditioned Imitation Learning for Robot Manipulation Tasks. Advances in Neural Infor- mation Processing Systems, 33, 2020.
[27] A. Stone, Xiao, Y. Lu, K. Gopalakrishnan, K.-H. Lee, Q. Vuong, P. Wohlhart, B. Zitkovich, Xia, C. Finn, et Open-world object manipulation using pre- pre-trained languaeee mod- els. arXiv preprint arXiv:2303.00900 2023.
[28] D. Shah, Osinski, B. Ichter, and S. Levine. LM-nav: Robotic navigation with large pre- trained models of language, vision, and action. In 6th Annual Conference Robot Learning, 2022. URL https: http///wpp//eeeee///////ee wttp:i/wwwweeefooneneeeeee http:/orrumtt---/AAAsseeeee
10

---

[29] D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng. I. Mordatch, and P. Florence. PaLM-E: An Embodied Multimodal Language Model. arXiv preprini arXiv:2303.0337. Mar. 2023.
[30] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang. I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manju- nath, Mordatch, Nachum, C. Parada, J. Peralta, Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. RT-1: Robotics Transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, Dec. 2022.
[31] C. B. Choy, J. Gwak, S. Savarese, and M. Chandraker. Universal Correspondence Network. In Advances in Neural Information Processing Systems, June 2016.
[32] Schmidt, R. Newcombe, and D. Fox. Self-supervised visual descriptor learning for dense correspondence. IEEE Robotics and Automation Letlers (RA-L., Apr. 2017.
[33] Florence, L. Manuelli, and R. Tedrake. Self-Supervised Correspondence in Visuomotor Policy Learning. IEEE Robotics and Automation Letters (RA-I.), 5(2):492-499, Apr. 2020.
[34] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec. V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, R. Howes, P.-Y. Huang, H. Xu, V. Sharma, S.-W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, Mairal, Labatut, A. Joulin, and P. Bojanowski. DINOv2: Learning robust visual features without supervision. (rXiv:2304.0719: 2023.
[35] J. Leonard and H. F. Durrant-Whit Whyte. Simultancous map building and localization for an autonomous mobile robot. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), volume 3, pages 1442-1447, 1991.
[36] H. Whyte,, D. Rye, and E. Nebot. Localization of Autonomous Guided Vehicles. In Robotics Research, pages 613-625. Springer London, 1996.
[37] H. Durrant-Whae Whyte and T. Bailey. Simultaneous localization and mapping: part i. IEEE Robotics and Automation Magazine, 13(2):99-100, 2006. doi: 10. 109.0........
[38] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg. Dex- net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. In Robotics:
Science and Systems (RSS), 2017.
[39] X. Yan, J. Hsu, M. Khansari, Y. Bai, A. Pathak, A. Gupta, J. Davidson, and H. Lee. Learning 6-dof grasping interaction via deep geometry-awand 3d representations. In IEEE International Conference on Robotics and Automation (ICRA), pages 37733 IEEE, 2018.
[40] Urain, N. Funk, J. Peters, and G. Chalvatzaki. SE(3)-DiftssionFieldd: Learning smooth cost functions for joint grasp and motion optimization through diffusion. IEEE International Conference Robotics and Automation (ICRA), 2023.
[41] R. Yang, G. Yang, and X. Wang. Neural volumetric memory for visual locomotion control. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[42] Kerr, L. Fu, H. Huang, Y. Avigal, M. Tancik, J. Ichnowski, Kanazawa, and K. Goldberg. Evo-NeRF: Evolving NeRF for Sequential Robot Grasping of Transparent Objects. In 6th Annual Conference Robot Learning, 2022. URL https:///penreeier net/forum?id Yrr.

---

[43] Q. Dai, Y. Zhu, Y. Geng, C. Ruan, J. Zhang, and H. Wang. GraspNeRF: Multiview-basee 6-DoF Grasp Detection for Transparent and Specular Objects Using Generalizable NeRF. In IEEE International Conference on Robotics and Automation (ICRA), 2023.
[44] S. Peng, K. Genova, C. M. Jiang, A. Tagliasacchi, M. Pollefeys, and T. Funkhouser. Open- Scene: 3D Scene Understanding with Open Vocabularies. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[45] N. M. M. Shafiullah, C. Paxton, L. Pinto, S. Chintala, and Szlam, Clip-fields: Weakly supervised semantic fields for robotic memory. In Robotics: Science and Systems (RSS), 2023.
[46] B. Bolte, A. S. Wang, J. Yang, M. Mukadam, M. Kalakrishnan, and C. Paxton. USA-Net: Unified Semantic and Affordance Representations for Robot Memory. ArXiv, 0abos//33.121.4.. 2023.
[47] C. Huang, O. Mees, A. Zeng, and W. Burgard. Visual Language Maps for Robot Navigation. In IEEE International Conference on Robotics and Automation (ICRA), London, UK, 2023.
[48] K. Jatavallabhula, A. Kuwajerwala, Q Gu, M. Omama, T. Chen, S. Li, G. Iyer, S. Saryazdi, N. Keetha, A. Tewari, J. Tenenbaum, C. de Melo, M. Krishna, L. Paull, F. Shkurti, and A. Tor- ralba. Conceptfusion: Open-set multimodal 3d mapping. In Robotics: Science and Systems (RSS), 2023.
[49] J. Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik. Lerf: Language embedded radiance fields. International Conference on Computer Vision (ICCV), 2023.
[50] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Bengio. Generative adversarial nets. In Advances Neural Information Processing Systems, volume 27, 2014.
[51] Y. Song. Sohl-Dickstein, Kingma, A. Kumar, S. Ermon, and B. Poole. Score- Based Generative Modeling through Stochastic Differential Equations. In International Con- ference Learning Representations, 2021. URL https: httL:httt////epeee http:////ppp/eeeeee//eeeeeeeeeee net wwwefoorrmme oorm? idd= PXTIG12RRHS.
[52] J. T. Kajiya and B. P. Von Herzen. Ray tracing volume densities. ACM SIGGRAPH computer graphics, 1863),165-174, 1984.
[53] N. Max. Optical models for direct volume rendering. IEEE Trans. Vis. Comput. Graph., 1(2): 99-108, June 1995.
[54] J. L. Schönberger and J.-M. Frahm. Structure-from-Mttioe Revisited. In Conference on Com- puter Vision and Pattern Recognition (CVPR), 2016.
[55] J. L. Schönberger, E. Zheng, M. Pollefeys, and J.-M. Frahm. Pixelwise View Selection for Unstructured Multi-Viee Stereo. In European Conference on Computer Vision (ECCV), 2016.
[56] C.-H. Lin, W.-C Ma, A. Torralba, and S. Lucey. Barf: Bundle-adjusting neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5741 5751, 2021.
[57] Z. Wang, Wu, W. Xie, M. Chen, and V. A. Prisacariu. NeRF--; Neural Radiance Fields Without Known Camera Parameters. arXiv preprint arXiv:2102.07066 2021.
[58] A. Simeonov, Y. Du, L. Yen-Chen, A. Rodriguez, L. P. Kaelbling, T. L. Perez, and P. Agrawal. SE(3) Eqqiivaraat Relational Rearrangement with Neural Descriptor Fields. In Proceedings of the 6th Conference Robot Learning (CoRL). PMLR, 2022.
12

---

[59] V. Zhong, T. Rocktäschel, and E. Grefenstette. RTFM: Generalising to New Environment Dynamics via Reading. In International Conference on Learning Representations, 2020. URL.
https://arxiv. www.oooggga............ /a//ab//1999 /91s//911... 08210.
[60] D. Bahdanau, F. Hill, J. Leike, E. Hughes, A. Hosscini, P. Kohli, and E. Grefenstette. Learning to understand goal specifications by modelling reward. In International Conference on Learn- ing Representations, 2018. http:t /ptps////opeeeeeeeeeeeee net/ f orum ? CmC99.
[61] R. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cambridge Uni- versity Press, Mar. 2004.
[62] Li, Z. Shen, Wang, L.. Shen, and L. Bo. Compressing volumetric radiance fields to 1 mb. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4222-4231, 2023.
13

---

Appendix
A.1 Neural Radiance Fields (NeRFs)
6D, vector-valued continuous function that maps from position = (x, y/ z) and normalized viewing direction d = (dg dy, dg), to the differential density a and emitted color (r.g. g. b). In practice, this is achieved via two neural networks which partially share parameters: 1) the density network (x) which depends only on the position x; and 2) the color network c(x, d) which depends on both the position x and viewing direction d.
Neural radiance fields [12] model
Novel-View Synthesis. NeRF synthesizes image by casting ray from the camera origin through the center of each pixel. Points along the ray are parameterized r /d, where i is the distance of the point to the camera origin o. The color C(r) of the ray between the near and far scene bounds tn and tt is given by the volume rendering integral [52]
4 (())o( dt, ["' 1.
dt, T(t) exp (-f(1 ((r))do)
C(r) - -
(5)
where Ttt) is the accumulated transmittance along the ray from Ft.. to
Modeling a Scene with NeRFs. For a scene, we are given a dataset of N RGB images (1)
with camera poses {T) Y=1* each iteration, we sample a batch of rays R (T) and optimize
and by minimizing the photometric loss Ligh ErER || C(r) I(r) where I(r) is the RGB
value the pixel corresponding to ray E R, and C(r) is the color estimated by the model using discrete approximation of Equation [12, 53].
A.2 Dense 2D Feature Extraction via MaskCLIP
We provide pseudo code for the MaskCLIP method [11] for extracting dense, patch-level features from the CLIP model [1] below. Algorithm is the computation graph of the last layer of vanilla CLIP. Algorithm 2 MaskCL.IP's modified graph. Note that the two linear transformations via W, and Wan can be fused into a single convolution operation. We provide our feature extraction code in our GitHub repository (https: /ttp://ggitttt f
Algorithm Image Feature (Original)
Algorithm Dense Features (MaskCLIP 11)
def forwardx))
q, k, W_qkv self (x)
def forward(x):
3 v (q[:1] * tooffmma (dim==1) *
v W_v vel).)n_((x) z W_out
x = + W_out @
self mlp(pelf.....mmmmmmmmmmmmmmmmllm 1n_2(x))
return 2[1:] # all but the CLS token
x[:1]
the CLS token
A.3 Feature Fields
Implementation Details. Memory for caching the 2D feature map a significant system bottle- neck that does appear with RGB reconstruction because high-dimensional features, up-scaled to the RGB image resolution, can grow to than 40 GB for standard NeRF dataset. We solve this issue by reconstructing patch-level feature maps without up-scaling them to pixel resolution. We speed up our feature distillation by building off' newer NeRF implementations using hierarchical hash grids [8] based on Nerfacto [10].
Feature Field Quality. F3RM benefits from neural feature fields' ability reconstruct detailed 3D geometry. We offer such example in Figure A8. Notice the difference in resolution, between the source 2D feature map (middle), and the final feature field.
14

---

(a) RGB Image
(b) Raw DINO VIT Feat.
(c) Distilled Feat.
Figure A8: Level of Detail. (a) Mesh strainer and whisk. (b) Raw feature map from DINO VIT, very low in resolution. Colors correspond to PCA of the features. (c) 3D feature fields recover higher level of detail than the source 2D feature maps. Inset corresponds to (b) in its original size for comparison.
(a) CLIP Feature Mean Squared Error
(b) DINO Feature Mean Squared Error
0.27
MLP Head
MLP Head
Hash Grid
0.18
Hash Grid
MSE
0.26
www.mm
MSE
0.17
0.25
0.16
0.15
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
Step
Step
Figure A9: Feature Error During Feature Distillation. The mean squared error on held-out set of feature maps for (a) CLIP and (b) DINO using the MLP head and hash grid architectures described in Section A.3.1. The hash grid architecture consistently achieves lower error.
A.3.1 Ablation on Feature Field Architecture
We implement our feature field as a hierarchical hash grid [8] that takes 3D position x as input, and outputs the feature vector. We compare this against a MLP head that takes the intermediate features output by NeRF as input, which is similar to the architectures in [5, 6]. We first train a NeRF on images collected by the robot of tabletop scene, then distill features for 10000 steps over 3 seeds.
Figure A9 shows that the hash grid architecture achieves lower mean squared error (MSE), because signals and finer details. While the difference in the MSE seems
is able to capture marginal between these two architectures, the hash grid-based architecture qualitatively results in significantly more well-defined semantic boundaries between objects as shown in Figure A10.
A.4 Experimental Setup
We provide details about our experimental setup used across our experiments for learning grasp from demonstrations and language- innuuge--guded object manipulation.
Physical Setup. We collect RGB images with a RealSense D415 camera (the depth sensor is not used) mounted on a selfie stick. The selfie stick used to increase the coverage of the workspace, as a wrist-mounted camera can only capture small area of the workspace due kinematic limitations. We program Franka Panda arm to pick up the selfie stick from magnetic mount, scan 50 1280 x 720 RGB images of the scene following fixed trajectory of three helical passes at different heights, and place the selfie stick back on the mount.
15

---

(a) MLP Head
(b) Hash Grid
Figure A10: Comparing Feature Field Architectures. We show the similarity heatmaps for the language query "metal mug" on the scene shown Fig.7. (a) When using the MLP head, regions unrelated the metal mug exhibit high similarity, as shown by the red bleed onto objects including the red screwdriver and tip of the whiteboard marker. (b) In contrast, the hash grid architecture results in significantly less bleed and more well-defined semantic boundaries between objects.
To calibrate the camera poses, we run COLMAP [54, 55] on scan of a dedicated calibration scene with calibration markers placed by the robot at known poses. We use these objects to solve for the transformation from the COLMAP coordinate system to the world coordinate system. These camera poses are reused subsequent scans. Given that the true camera poses vary due small differences in how the robot grasps the selfie-stick, we optimize them as part of NeRF modeling to minimize any errors and improve reconstruction quality [10, 56, 57].
Labeling Demonstrations. We label demonstrations in virtual reality (VR) using web-based 3D viewer based Three.js that we developed which supports the real-time rendering of NeRFs, point clouds, and meshes. Given NeRF of the demonstration scene, sample point cloud and export it into the viewer. We open the viewer in Meta Quest headset to visualize the scene, and move gripper to the desired 6-DOF pose using the hand controllers (see Fig.2a).
NeRF and Feature Field Modeling. We downscale the im- ages to 640 480 to speed up modeling of the RGB NeRF, and use the original 1280 720 images input the vision model for dense feature extraction. We optimize the NeRF and feature field sequentially for 2000 steps each, which takes at most 90s (average is 80s) on a NVIDIA RTX 3090, including the time to load the vision model into memory and extract features.
Feature Type
Resolution
DINO ViT
98 x 55
CLIP VIT
42 x 24
CLIP ResNet
24 X 14
Table 3: Feature Map Resolu- tions. Resolutions of the fea- tures output by the vision models given 1280 720 RGB image.
In experiments, we distill the features their original fea- ture map resolution which is significantly smaller than the RGB images (see Table 3). We achieve this by transforming the cam- era intrinsics to match the feature map resolutions, and sampling rays based on this updated camera model. The specific models we used were dino viv.iii for DINO VIT, VHT-L/140336px for CLIP VIT, and RN50x64 for CLIP ResNet.
A.5 Ablation on Number of Training Views
Although robot scans 50 images per scene in experiments, demonstrate that it is possi- ble to use a significantly smaller number of views for NeRF and feature field modeling without significant loss in quality. To investigate this, we ablate the number of training images by evenly subsampling from the 50 scanned images and modeling a NeRF and feature field.
Figure All qualitatively compares the RGB, depth, and segmentation heatmaps. We observe an increase in floaters as reduce the number of training images, with approximately 20 images being the lower bound before drastic decline in quality.
16

---

49 Training Images
30 Training Images
20 Training Images
18 Training Images
Figure A11: Ablating Number of Training Views. We qualitatively compare feature fields trained on different numbers of views. (Top Row) Segmentation heatmap for "Baymax" from CLIP feature field overlaid on the RGB image from NeRF. (Bottom Row) Depth map rendered from NeRF.
A.6 Learning to Grasp from Demonstrations
Sampling Query Points. We use N₂ = 100 query points across all the tasks shown in Fig.4 As other works have observed [17, 58], the downstream performance can vary significantly across different samples of the query points. To address this issue, we sample five sets of query points over different seeds for each task, and run the grasp optimization procedure across set of test scenes used for method development. We select the query points that achieved the highest success rate the test scenes. The covariance of the Gaussian is manually tuned to fit the task.
Grasp Pose Optimization. We first discuss how we initialize the grasp poses. We consider a tabletop workspace of size 0.7 0.8 x 0.35 meters, and sample a dense voxel grid over the workspace with voxels of size 0. 0.0075m (we use 0.005m for the cup on racks experiment), where each voxel (1c, y/ 2) represents the translation for grasp pose.
Next, we compute the alpha value a(v) for each voxel using the NeRF density network and filter out voxels with a<v) < éfree 0.1. This removes approximately 98% of voxels by ignoring free space. The cosine similarity of the voxel features (v) is thresholded with the task embedding ZM to further filter out voxels. This threshold adjusted depending on the task and type of feature distilled, and typically cuts down 80% of the remaining voxels. Finally, we uniformly sample N,- 8 rotations for each voxel get the initial grasp proposals T.
We minimize Equation 3 to find the grasp pose that best matches the demonstrations using the Adam optimizer [18] for 50 steps with a learning rate of 5e-3. This entire procedure takes 15s on average, but could easily be sped up.
Grasp Execution. We reject grasp poses which cause collisions by checking the overlap between voxelized mesh of the Panda gripper and NeRF geometry by querying the density field a. We input the ranked list of grasp poses into an inverse kinematics solver and BiRRT motion planner in PyBullet [19, 20), and execute the highest-ranked grasp with a feasible motion plan.
A.6.1 Baselines
We provide implementation details of the four baselines used in our few-shot imitation learning experiments. The first three baselines use NeRF-based outputs as features for the query point-based pose optimization:
Density: we use the alpha a (0, 1) values for NeRF density ensure the values are scaled consistently through different scenes, the density values output by the density field unbounded.
17

---

(a) Test Scene with Screwdriver
(b) Affordance Prediction
(c) Predicted Grasp
Figure A12: MIRA Failure Case. (a) Test scene with screwdriver and other distractors for the screwdriver task (Fig.4b). (b) The orthographic render of the view selected by MIRA, we show the RGB (top) and depth (bottom) renders. The pixel circled in cyan indicates the action with the highest pixel-wise affordance across views. (c) The predicted 6-DOF grasp incorrectly targets the silicone brush, as it shares resemblance to screwdriver from top-down perspective.
Intermediate Features: we use the features output by the intermediate density embedding MLP in Nerfacto [10], which have dimensionality of 15.
RGB: we use [r, g, b, ai as the feature for this baseline. a is used to ensure that this baseline pays attention both the color and geometry, as we found that using RGB only with the alpha- weighted feature field f, (Eq.2) collapsed RGB values 0, 0) for free space, which corre- sponds to the color black.
MIRA Baseline. The fourth baseline we consider Mental Imagery for Robotic Affordances (MIRA) [21], a NeRF-based framework for 6-DOF pick-and-place from demonstrations that renders orthographic views for pixel-wise affordance prediction. Orthographic rendering ensures that an object has the same size regardless of its distance to the camera, and used to complement the translation equivariance of the Fully Convolutional Network (FCN) for predicting affordances.
MIRA formulates each pixel in a rendered orthographic view as a 6-DOF action T = (R, t)., with the orientation of the view defining the rotation and the estimated depth from NeRF defining translation t. The FCN is trained to predict the pixels the rendered views corresponding to the demonstrated 6-DOF actions, and reject pixels sampled from set of negative views. During inference, MIRA renders several orthographic views of the scene and selects the pixel that has the maximum affordance across all views.
In our experiments, we train separate FCN for 20000 steps for each task in Fig.4 specified by two demonstrations, and sample negative pixels from datasets consisting solely of distractor objects. We use data augmentation following Yen-Cher et al. [21]'s provided implementation and apply random SE(2) transforms to the training views. Given test scene, we 50 RGB images described in Appendix A.4, render 360 orthographic viewpoints randomly sampled over an upper hemisphere looking towards the center of the workspace, and infer a 6-DOF action.
MIRA was designed for suction cup grippers and does not predict end-effector rotations. We at- tempted to learn this rotation, but found that the policy failed to generalize. To address this issue and give MIRA the best chance of success, we manually select the best end-effector rotation to achieve the task. We additionally find that MIRA often selects floater artifacts from NeRF, and manually filter these predictions out along with other unreasonable grasps (e.g., grasping the table itself).
Given that MIRA is trained from scratch given just two demonstrations, we find that it struggles to generalize and is easily confused by floaters and distractors despite data augmentations and negative samples. MIRA additionally reasons over 2.5D by using the rendered RGB and depth from NeRF inputs to the FCN, while our query point-based formulation reasons explicitly over 3D. Because of this, we observe that MIRA can fail when there occlusions distractor objects that look
18

---

(a) Novel Scene
(b) DINO VIT Heatmap
(c) CLIP ViT Heatmap
Figure A13: Comparing DINO and CLIP feature fields. We depict the cosine similarity for the task of grasping a mug by the handle. Two demos are provided on a red and a white mug (cf. Fig.3b). (b) DINO overfits the red color of the apple, while (c) CLIP captures higher-level semantics, and identifies the metal mug.
like the demonstration objects from certain viewpoints. For example, one of the demonstrations for the screwdriver task was a top-down grasp on a screwdriver standing vertically in a rack (Fig.4b). Figure A12 depicts a test scene for the screwdriver grasping task where MIRA incorrectly selects the silicone brush looks similar to a screwdriver from a top-down 2.5D view. DINO and CLIP ResNet feature fields successfully grasp the screwdriver in this scene, highlighting the benefits of using pretrained features and reasoning explicitly over 3D geometry.
A.6.2 DINO Failure Cases
Our experiments show that DINO struggles with distractor objects which have high feature similarity to the demonstrations, despite not representing the objects and their parts we care about (Fig.A A13)). We observe that DINO has the tendency to overfit to color. On the other hand, CLIP struggles far less with distractors due to its stronger semantic understanding (Fig.A13c).
A.7 Gunded Manipulation
For the language -uniged-gguedddddde experiments, we distilled CLIP VIT features from ViT-L/140336px We reuse the 10 demonstrations from the learning to grasp from demonstrations section and their asso- ciated Ng = 100 query points, and sample N, rotations for each voxel when initializing the grasp proposals. We minimize the language-guided cost function Equation 4 for 200 steps with Adam using learning rate of x 10
Retrieving Demonstrations via Text In practice, we compare the user's embedded text query with the task embedding ZM for each task M which is specified by two demonstrations. We randomly sample object names as our negatives (L-)
Inference Time. The inference time optimize for a grasp pose given language query is 6.9 seconds average. We did not make any substantial attempts to speed this up, but note that reducing the number of optimization steps (we use 200 steps but observe convergence usually within 50- 100 steps), pruning more aggressively, and improving the implementation will significantly reduce inference time
A.8 Additional Related Work
We provide a more comprehensive overview of the related work discussed in Section 5.
Open-Ended Generalization via Language. number of prior work use natural language for task-specification and long-horizon planning in applications such as tabletop and mobile manipula- tion [23, 26], navigation [27, 28], and more generally, sequential decision-making in games [59, 60]. A recent line of work seeks to replicate the success of vision and language models in robotics, by jointly pre-training large foundation models on behavior data [29, 30]. One can refer to [25] for
19

---

more comprehensive coverage. The goal of our work is different. We seek to find way to incorpo- rate geometric information with any pre-trained features. F3RM a model-agnostic approach for lifting imagery and language prior knowledge into 3D. In this regard, we are more closely connected to CLIPort [23], which focuses on top-down grasping, and PerAct [24], which trains voxel grid conditioned behavior transformer from scratch given waypoints in motion trajectories.
Semantic Understanding and Dense 2D Visual Descriptors. Learning visual descriptors for dense correspondence estimation is fundamental problem in computer vision. Among the earliest works, Choy et al. [31] and Schmidt et al. [32] used dynamic 3D reconstruction from RGB-D videos to provide labels of association between pixels from different video frames. Dense Object Nets tackle this problem in the context of robotics [13], and eliminate the need for dynamic reconstruction using multi-view RGB-D images of static scenes [33]. NuRr-Supervision [14] leverages NeRF' for 3D reconstruction, extending descriptor learning to thin structures and reflective materials that pose challenges for depth sensors. Unlike these prior works, recent work in vision foundation models shows that self-supervised vision transformers offer excellent dense correspondence [4]. When scaled to large, curated dataset, such models offer phenomenal few-shot performance on various dense prediction tasks [34].
Geometric Aware Representations for Robotics. Geometric understanding has been long- standing problem in computer vision [61] and is an essential and mission-critical part of mapping and navigation [35, 36, 37], grasping [38, 39, 17, 40], and legged locomotion [41]. These work either require direct supervision from 3D data such as Lidar point clouds, or try to learn represen- tations from posed 2D images or vidcos [14]. More recently, the robot grasping community has experimented with neural scene representations to take advantage their ability to handle reflective or transparent objects and fine geometry [21. 42, 43]. Our work incorporates pre-trained vision and vision-language foundation models to augment geometry with semantics.
3D Feature Fields in Vision and Robotics A number of recent work integrate 2D foundation models with 3D neural fields in contexts other than robotic manipulation. For example, Open- Scene [44] and CLIP-Fields [45] distill 2D features into neural field by sampling points from existing 3D point clouds or meshes. USA-Net [46] and VLMap [47] build 3D feature maps from RGB-D images and VLMs for navigation. Our approach uses RGB images and integrates geomet- ric modeling via NeRF with feature fusion in single pipeline. ConceptFusion [48] uses surfels to represent the features, which requires 50- 100s GB per scene. Distilled feature fields offer significant reduction in space. Each of our scenes requires between 60 - 120 MBs and could be further compressed using lower-rank approximations [62]. Our work shares many similarities with LERF [49]. However, unlike the image-level features used in LERF, which are derived from a hier- archy of image crops, we extract dense, patch-level features from CLIP. Additionally, we take a step further by exploring the utilization of these feature fields for robotic manipulation.
20


Reviews:
[{"review_id": "rUuRU-alRAN", "reviewer": "Reviewer_JsoY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper employs Distilled Feature Fields (DFF) for robotic manipulation. It utilizes DFF as a robust, generalizable, and semantic-aware scene representation. The authors demonstrate this representation's efficacy in tasks such as few-shot manipulation and language-conditioned manipulation.", "summary_of_recommendation": "Although the idea is interesting and the experiments are solid, this paper is more about applying CV techniques to robotic tasks with some modification (to make it faster and more available etc.). So I vote for weak accept.\n\nAfter rebuttal:\nI still think the main method is still built on top of *Neural Feature Fusion Fields*, showing interesting applications to robotics. So I keep my vote to weak accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please address the queries highlighted in the weaknesses section.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n1. The concept of utilizing DFF as a visual representation within robotics appears logical. It's both 3D and semantic-aware, a feat current representations can achieve separately, but not in conjunction.\n2. The task design is intelligent, maximizing the benefits of DFFs. The action space is a 6D transformation; with the proposed action reparameterization, the action is also mapped into the 3D feature space, creating homogeneity between the representations of both observation and action.\n3. The experimental results successfully demonstrate the proposed method's efficacy.\n4. The exposition is exceptionally clear and easy to follow.\n\nWeaknesses:\n1. The task design, while inventive, is also somewhat limited. How could this representation be generalized to accommodate closed-loop manipulations?\n2. Have alternatives to action reparameterization been considered? For instance, could the query points be set as the midpoints between the two grippers post-transformation?\n3. On line 186, could you provide examples of objects with high feature similarities to the demonstrations? Under what circumstances would CLIP outperform DINO?", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "jSZzrJ9BkQ", "reviewer": "Reviewer_7BHn", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper focuses on enhancing robotic manipulation tasks by bridging the gap between 2D image features and the detailed understanding of 3D geometry required for such tasks. The authors leverage NeRF representations to get a dense 3D scene representation. CLIP features are distilled into the NeRF model, allowing it to render features in 3D. The grasping and manipulation demonstrations are connected to 3D feature distributions by observing the pose of the demonstrated grasp and storing the nearby CLIP features and their relative 3D locations. With features from the vision-language model CLIP, the authors demonstrate the capability to designate novel objects for manipulation via free-text natural language. The results show that the model can generalize to novel object instances effectively.", "summary_of_recommendation": "I believe the work would be valuable to the community. \n\n**Post-rebuttal update:** The authors have answered my questions. I appreciate the improvement in the quality of the manuscript. My recommendation to accept this paper remains.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Refer to the questions in the \"Weakness\" section.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Strengths:**\n- The paper introduces a novel approach to derive dense 3D features using a pretrained vision-language model CLIP with NeRF-like representation.\n- The paper presents a novel algorithm for integrating both language features as well as stored visual features toward grasping and placing novel objects.\n- The paper is well-written and includes supplementary material with details on the distillation procedure and extraction of dense features from CLIP along with the pseudo-code. \n- Real-world experiments and results are shown to substantiate the effectiveness of the proposed method.\n\n**Weakness:**\n- It is unclear what the paper defines as novel objects. Are these the objects from the same category shown in the demonstration? Or is the work generalizable to a new category of objects? To what degree can the language labels for the objects vary? \n- The paper clearly states that the computation is a limitation. It will be beneficial to see the actual time it takes during the inference time upon the query.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "dIh2OnqVJu", "reviewer": "Reviewer_azcv", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper studies the topic of VLM feature distillation for robotics grasping. Specifically, it proposed a method to distill high-resolution dense 2D features to 3D grids using NeRF. The distilled 3D feature grids is further used in grasping pose matching, with an emphasis on  generalization to open-world objects. Experiments have been conducted on grasping and language-guided object manipulation in the real-world setting. Results demonstrate the effectiveness of the proposed distillation method and strong generalization of the features in grasp pose matching.", "summary_of_recommendation": "The reviewer does not find obvious caveats of the paper, besides the limitations (dynamic scenes and the latency) discussed. It is a great work that involves VLM feature, NeRF, and robotic learning all together. I highly recommend accepting the paper at the conference.\n", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Besides Figure 8 in the paper, it would be good to provide additional quantitative results and ablation studies (e.g., interpolation method, architecture changes). The reviewer also recommends adding more qualitative results to the final version. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is well-structured with clear descriptions of the method, experimental results, and limitations. The reviewer finds the supplementary document and the video helpful.\n\nFor example, the findings summarized in Table 1 are very reasonable, as CLIP feature contains robust semantic information and DINO feature contains more geometric information. The differences are well supported by the experiment results on the cluttered scene.\n\nIn section 3.2, the local geometry transformation is a very effective design for tasks including robotic grasping and manipulation. It might be good to mention the prior work that shares a similar design [NewRef1-2].\n\nReferences\n- Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics, RSS 2017.\n- Learning 6-DOF Grasping Interaction via Deep Geometry-aware 3D Representations, ICRA 2018.\n\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "IVPkpr__9YG", "reviewer": "Reviewer_j1dX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes to obtain high-resolution feature fields via InstantNGP and other engineering tricks to improve training time, and then use such feature field to construct 6-DOF poses with iterative optimisation by retrieving the closest poses presented in the demonstrations. The proposed framework is evaluated through real-world grasping tasks, showing good generalisation in novel objects. ", "summary_of_recommendation": "Good method, but needs some extra clarification. Good results on robotics and video.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. L149 maximising the cosine similarity?\n2. Are these 10 demonstrations carefully chosen throughout the entire experiment? How robust it is for a different set of demonstrations?\n3. Better explanations for the method section.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is clearly in a good quality, with well-motivated and well-designed framework. However, I found some explanations in the method section a bit confusing, and some other improvements could be done for the experiments.\n\n1.\tI found no problem understanding the framework design up till L112, then it starts to get a bit confusing. I understand the framework starts with constructing a feature neural field with CLIP features, and adding some engineering tricks such as removing the global token and using InstantNGP to improve the training speed and construct high-resolution local information. But later in Section 3.2, I failed to understand i) why and what are the query points; ii) are we learning per-feature 6DOF pose via a learnable map? iii) what and why we need the encoding of a task, and why average the features over the demos represents the encoding for a task? I thought the features should represent task-agnostic semantic information, so we may do object-level grasping?\n2.\tAs using a similar strategy in Neural Descriptor Fields, do we have a prior knowledge knowing that the object we want to grasp has a similar object in the query demo? If so, I believe we need do a proper ablation on how the performance would be changed based on different number/variations of the query demos.\n3.\tGeneralisation over text prompts. Though the paper presents a good generalisation on object categories. I am wondering how the generalisation is on language prompts? Or simply the language prompts her are simply object category names? \n4.\tOther design choices. As since the proposed network is mainly based on the pre-trained vision-language features. A much simpler and cleaner framework would be just to learn how the transformation from the learned feature map to grasping point based on a text prompt, without using demonstrations as queries, which I believe is a strong baseline. \n ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel method for robot manipulation by distilling image features from a vision-language model to 3D scenes. The method enables 6D grasping and language-guided grasping of unseen objects using few-shot demonstrations.

The reviewers achieve a consensus on accepting the paper with high scores. Therefore, I recommend to accept the paper as an oral presentation.
Decision: Accept (Oral)