Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping
Authors: Adam Rashid
Satvik Sharma
Chung Min Kim
Justin Kerr
Lawrence Yunliang Chen
Angjoo Kanazawa
Ken Goldberg
Venue: CoRL 2023 Oral (2023)
Keywords: NeRF
Natural Language
Grasping
Semantics

Abstract: Grasping objects by a specific subpart is often crucial for safety and for executing downstream tasks. We propose LERF-TOGO, Language Embedded Radiance Fields for Task-Oriented Grasping of Objects, which uses vision-language models zero-shot to output a grasp distribution over an object given a natural language query. To accomplish this, we first construct a LERF of the scene, which distills CLIP embeddings into a multi-scale 3D language field queryable with text. However, LERF has no sense of object boundaries, so its relevancy outputs often return incomplete activations over an object which are insufficient for grasping. LERF-TOGO mitigates this lack of spatial grouping by extracting a 3D object mask via DINO features and then conditionally querying LERF on this mask to obtain a semantic distribution over the object to rank grasps from an off-the-shelf grasp planner. We evaluate LERF-TOGO’s ability to grasp task-oriented object parts on 31 physical objects, and find it selects grasps on the correct part in 81% of trials and grasps successfully in 69%. Code, data, appendix, and details are available at: lerftogo.github.io

Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping
Adam Rashid Satvik Sharma*, Chung Min Kim, Justin Kerr, Lawrence Yunliang Chen, Angjoo Kanazawa, Ken Goldberg
Object Query
Top Feasible Grasps
LERF for Task-Oriented Grasping
"rose"
"plant stem
Crush
"bread knife"
"handle"
Cut
blue sunglosses"
"eur hook
Shatter
Re-rank Grasps
Figure 1: Learning-bait based grasp planners primarily consider object geometry, potentially yielding suboptimal grasps. LERF-TOGO uses natural language to select the target object with LERF [1] (in orange), and resamples grasps towards on object subparts using conditional LERF queries (in blue) for safe, task-oriented grasps.
Grasping objects by a specific subpart is often crucial for safety and for executing downstream tasks. We propose LERF-TOGO, Language Embedded Ra- diance Fields for Task Miis-Orieeted Grasping of Objects, which uses vision- languaee models zero-shot to output a grasp distribution over an object given natural lan- guage query. To accomplish this, first construct LERF of the scene, which distills CLIP embeddings into multi-scale 3D language field queryable with text. However. LERF has no sense of object boundaries, so its relevancy outputs often return incomplete activations over an object which are insufficient for grasping. LERF-TOGO mitigates this lack of spatial grouping by extracting a 3D object mask via DINO features and then conditionally querying LERF on this mask obtain semantic distribution over the object to rank grasps from an off-the-shelf grasp planner. We evaluate LERF-TOGO ability to grasp task-oriented object parts on 31 physical objects, and find it selects grasps the correct part in 81 % of trials and grasps successfully in 69%. Code, data, appendix, and details are available at: derftogo.githubiir
Abstract:
Keywords: NeRF, Grasping, Semantics, Natural Language
Denotes equal contribution, alphabetically ordered
UC Berkeley
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Introduction
Many common objects must be grasped appropriately to avoid damage or facilitate performing task: a knife by its handle, a flower by its stem, or sunglasses by their frame. Learning-based grasp- ing systems exhibit impressive robustness on grasping arbitrary objects [2. 3, 4, 5, 6, 7, 8, 9, 10]. but these systems typically measure grasp success based on whether the object was lifted [11, 12, 13, 14, 15]. Critically, these methods ignore an object's semantic properties: even if a robot could locate your favorite sunglasses, rather than safely grasp at the frame it may shatter the lenses. This abil- ity to grasp object part based on a desired task and constraints is called task-oriented grasping, and while well-studied [16, 17, 18, 19, 20, 21], previous methods collect specific object affordance datasets and struggle to scale to diverse set of objects. Instead, the flexibility of natural lan- guage has the potential for specifying what and where to grasp. In this work, we propose LERF for Task-Oriented Oriented Grasping on Objects (LERF-TOGO), a method which enables task-oriented grasping through natural language by using large llngua-la models in a zero-shot manner.
LERF-TOGO takes input object and task-orienteed object part name in natural language (i.e. "flower; stem "), and outputs ranking over viable grasps on this object from which the robot should grasp. We build on recent work Language Embedded Radiance Fields (LERF) [1], which takes in calibrated RGB images and trains a standard NeRF in tandem with a scale-conditioned CLIP [22] feature field. Given a sentence prompt query, it outputs a 3D relevancy heatmap representing simi- larity to the query. However, these heatmaps may fail to highlight the full object (e.g., highlight only the bristles of brush), which may cause issues when directly deployed a task-oriented grasping task (grasp the "handle" of brush). LERF-TOGO improves upon LERF's capabilities by predicting a 3D object mask using 3D DINO [23] features explicitly during inference. We propose method of conditional LERF querying which restricts an object sub-part query to the object mask, leveraging the multi-scale nature of LERF to isolate specific regions within an object. LERF-TOGO then uses GraspNet [15] to generate grasps, re-ranking them based on the geometric and semantic distribu- tions. We implemented system with appropriate regularizations which allows LERF-TOGO to operate on physical robot and evaluate on 39 common household objects. experiments, 96% of grasps the correct object, 82% on the correct object part, and 69% result in successful grasp.
This work contributes LERF-TOGO, an algorithm for producing task-based semantic grasp distri- butions over an object by first extracting a coarse 3D object mask with DINO [23] features to locally grow a relevant region, then conditioning a LERF query on this mask to isolate sub-parts of an ob- ject. We design a robotic system which integrates LERF-TOGO on a physical robot to reconstruct a LERF of scene, then execute task-oriented grasps through natural language to grasp semantically meaningful object parts. See the project website at: cerftogo.ithubb..
2 Related Work
2.1
Task-Oriented Grasping
Task oriented grasping studies how to grasp objects by specific parts based on a use case. It has been studied by probabilistically modeling human grasps [19], extracting geometric features from labeled object parts [21], training on part-affordance datasets simulation [24], or transferring category-speciffe part grasps to new instances
[25]. Recent works [26, 17] train object-part grasp networks by leveraging object part and manipulation affordance datasets for range of household objects. Kokic et al. [18] use videos of humans interacting with objects guide grasps towards the same part. Decomposing objects into parts has also long been studied as a co-segmentation task in vision [27, 28]. Recent approaches use pretrained vision features to discover common parts within sets of objects [29]. This technique has been applied at scale to segment parts of objects based on a canonical object [30] or detect object affordances from example images of human usage [16]. Though effective, it assumes access to a canonical image of each object and pre-existing part labels or demonstrations, which are restrictive in real-world applications. In contrast, LERF-TOGO uses off the-heh-hhlf vision- vision-language models trained at scale, so it captures long-tail object and parts more casily without the use of affordance datasets.
2

---

SSuccpaan
Extensson Cord
CCleaning Spray
Too 5% Graios LERF TOGO
'Plug"
Bouy
"Body
"Screwrriver'
"Mug"
"Clorax Wipes"
Too 5% Grasos LERP TOGO
"Meial Rosi"
"Houdie"
"Hondle"
Rimm or Lip"
"Body"
Body
"Protrading Towel
"Lid"
Figure 2: Task-Oriented Grasps: LERF-TOGO grasps target objects by different parts with differ- ent natural language queries (in quotes), Left: Crops of target objects. Right: Top row visualizes the relevancy distribution across the object mask, and bottom row shows top 5% of resampled grasps.
2.2 Neural Radiance Fields (NeRF)
Neural Radiance Fields (NeRF) [31] are an attractive representation for high quality scene recon- struction from pose RGB images, with an explosion of recent work visual quality [32, 33, 34, 35, 36, 37, 38), large-scale scenes [39, 40, 41], optimization speed [42, 43, 44, 45], dynamic scenes [46, 47, 48], and more. Because of its high-quality reconstruction and differentiable prop- erties, NeRF has been widely explored in robotics for navigation and mapping [32, 49, 50, 51], manipulation [52, 53, 54, 55], and for synthetic data generation [56]. This work is most similar to works such as Evo-NeRF [54] which use NeRF as real-time scene reconstruction to grasp objects. However, in contrast to previous works which only use RGB information, in this work we must include additional semantic information in 3D to select grasps falling on relevant target objects.
Several prior works explore using semantic outputs inside NeRF. Semantic-NeRF [57], Panoptic Lifting [58], and Panoptic Neural Fields [59] distill semantic categories from semantic segmentation networks into 3D to improve the 3D consistency of labels, particularly noting the denoising effect of averaging multiple views. Other works such as Distilled Feature Fields [60] or Neural Feature Fusion Fields [61] distill feature vectors from DINO and LSeg [62], and show they can used for editing and scene segmentation. We build off of LERF [1]. which described in the next section.
2.3 LERF Preliminaries
Language Embedded Radiance Fields (LERF) [1] is recent representation that distills CLIP fea- tures into NeRF, LERF imputs RGB images with camera poses and outputs a 3D field of DINO embeddings as well as a scale-conditioned CLIP field. This supports querying points in 3D for CLIP embeddings different physical scales, capturing different semantics given different amounts of context. Given text query. a relevancy value (from 1) can be generated at any 3D point by calculating the cosine similarity between LERF-queried embeddings and the CLIP embedding of query text. During this query process, grid search the scale parameter retrieves the scale with the highest activation. LERF is particularly attractive for task-oriented grasping because 1) its multi-scale parameterization allows queries at both object-level and part-level scales 2) LERF uses outputs from a pre-trained CLIP model without fine-tuning, which supports a variety of long-tail object queries not included in object or part segmentation datasets. LERF, however, tends to pro- duce nonuniform activations on object queries because it lacks spatial grouping as shown in Fig. 5. In this work we show how explicitly use the DINO feature field to obtain object masks to enable down-stream object part queries related to task- trientree grasping.

---

2.4 Natural Language in Robotics
With the advent of large pretrained language and vision-language models, several works have explored building 3D map representations to guide robot navigation. VL.-Maps [63] and Open- Scene [64] build a 3D language embedding from pretrained open-vocab detectors [65, 62], which can be used to navigate to target queries. CLIP-Fields [66]. ConceptFusion [67], and NLMaps- SayCan [68] take more of a region-proposal hased approach, querying CLIP on the outputs of some region proposal methods and fusing them into 3D point clouds for downstream navigation tasks. Region-based zero-shot methods retain more language understanding than fine-tuned fea- tures but run the risk of missing objects by insufficiently masking input images. Semantic Ab- straction [69] avoids this by extracting relevancy from Ission-anguuae models using Chefer et al. [70] and uses these for composing multiple language queries with spatial relationships. Language has also been studied in the context of robot manipulation. Mees and Burgard [71] use the MAt- (Net [72] vision-language model for object rearrangement, and CLIPort [73] uses language under- standing from CLIP to train a Linguage-conditione pick and place module from demonstrations. PerAct [74] uses language-conditiones demonstrations with 3D scene transformer learn di- verse tasks, MOO [75] uses the outputs from OWL-ViT condition manipulation policy for grasping objects, and large-scale demonstration datasets like RT-1 [76] train on massive language- conditioned demonstration trajectories. In contrast to many other language-con -enguugg-coddiioonedd approaches, demonstrations or robot exploration.
LERF-TOGO uses internet-scale vision models purely zero-shot and does not require fine-tuning on
3 Problem and Assumptions
Given a planar surface (table or workbench) containing a set of objects, the objective is for a robot to grasp and lift target object specified using natural language. This query (e.g.. "sunglasses; ear hooks.") includes both the object query ( ("sunglasses") and the object part query, which specifies the part to grasp the object by ("ear hooks"). We experiment with lifting this assumption in Sec. by leveraging LLM for providing part queries. We assume access robot manipulator with a parallel jaw gripper and calibrated wrist-mounted RGB camera, and the objects in the scene are graspable by the robot. We also assume the object query specifies present single object.
Method
Given object and object part query, LERF-TOGO outputs a ranking of viable grasps the object part. To accomplish this, it first captures the scene and reconstructs a LERF (Details in Section B). Given a text query, LERF can generate 3D relevancy map that highlights the relevant parts of the scene (Sec. 2.3). Second, a 3D object mask is generated using the LERF relevancy for the object query and DINO-based semantic grouping (Sec. 4.1). Third, a 3D part relevancy map is generated with a conditional LERF query over the object part query and the 3D object mask (Sec. 4.2). The part relevancy map is used to produce semantic grasp distribution.
4.1 3D Object Extraction
An important limitation of LERF is its lack of spatial grouping within objects: for example given "can opener". LERF tends to highlight regions of the object that most obviously identify that object (e.g. the metal cogs on the can opener as shown by the orange star in Fig. 3). However, since the region that visually identifies the object and the desired grasp location (e.g. handle) can differ significantly, this problematic. LERF inherently exhibits such local behavior because trains on local crops of input images, causing CLIP embeddings surrounding the handle be unaware if it belongs a can opener. LERF-TOGO overcomes this by finding 3D object mask given language query, which groups the object part together with the LERF activation. To create the object mask, we leverage the 3D DINO embeddings [23] (self-Distillation with NO labels) present within LERF during inference, because DINO embeddings have been shown to exhibit strong object awareness and foreground- backgruud distinction [29, 23, 77].
First, we obtain coarse object localization from LERF by rendering top-down view of the scene and querying the object. We produce a foreground mask by thresholding the first principal compo-

---

1. Scene Reconstruction + LERF
2. Object + Part Selection
3. Grasp Selection
3D Object Mask
Conditional LERF Query
6DDoF Grasp Generation
Mask: "can opener" "
Re Re-rank with Semantic Relevancy
Capture Scene Train LERF S LERF
Point Cloud Extraction
"handle"
Relevancy Query on 3D Mask
Semantic Grasp Distribution
Grouping w/ 3D DINO
"coR opener
Language Queries
Mask Result
3D Relevancy: "hundle"
Grasp Execution
Figure 3: LERF-TOGO Pipeline: After reconstructing the scene with wrist-mounted camera, we render an object-centric point cloud around the highest LERF activation. We next extract 3D object mask by flood-filling the DINO features this point cloud, condition an object part query on this object mask. Finally, we sample grasps and re-rank them according to 3D object part relevancy.
nent of the top-down rendered DINO embeddings, and constrain the relevancy query to this mask to find the most relevant 3D point. We then refine this single-point localization into a complete object mask. We render an object-centric point cloud around this 3D point by deprojecting NeRF depth from multiple views, and then iteratively grow the object mask by including neighboring points to the frontier which lie within threshold DINO similarity (similar to floodfill). The output of this process is set of 3D points lying the target object. See the Appendix for more details.
4.2 Conditional LERF Queries
Another important challenge of using CLIP is its tendency to behave bag-of-word. [22]: the activation for "mug" behaves very similarly to "mug handle" because CLIP latches onto individual words, not the grammatical structure of sentences. To mitigate this phenomenon, LERF-TOGO introduces a conditional method of querying LERF relevancy by composing two related queries, similarly to how composing prompts has shown promise generative modeling for guiding specific properties [78]. Because LERF is scale-conditioned, during inference it searches over scales for given query and returns the relevancy at the scale with the highest activation. To condition LERF query, LERF-TOGO searches only on the points within the 3D object mask. Intuitively, this results in distribution over the object's 3D geometry representing the likelihood that given point is the desired object part, which can be used for biasing grasps towards this region.
4.3 Grasping
Grasp Sampling Ensuring complete coverage of grasps on objects is critical to avoid missing spe- cific object parts. We use GraspNet [15], which can generate 6-DOF parallel jaw grasps from a monocular RGBD point cloud, but from a single view it often misses key grasps on target object parts. To mitigate this, and to leverage the full 3D geometry available within NeRF, we create hemisphere of virtual cameras oriented towards the scene's center. For every virtual camera, convert the scene's 's point cloud to the camera coordinate frames before providing it as input to the pretrained GraspNet model. To obtain the final set of grasps for the scene, we combine the generated grasps from the virtual cameras using non-maximum suppression to remove duplicates.
Grasp Ranking Given the grasps in the previous step (the geometric distribution), we now combine it with the semantic distribution across an object obtained from LERF-TOGO. The semantic score Ssem for given grasp is computed as the median LERF relevancy of points within the grasp volume. The geometric score 8eom is the confidence output from GraspNet, indicating grasp quality based
5

---

Mugs
Fragile le
Tools
F Flowers
Knives
Cords
Popsicles
Martinis
Kitchen
Figure 4: Left: Select scenes used during experiments, Right: example grasps using LERF-TOGO.
Object
Object
LERF-TOGO
"Sunsspan"
'Hande
Query Object
"Disk brash"
"Haandi""
O 'Serah braué"
3D Objject Mask
X
Part: "Hanalle
Figure 5: Ablation and Limitations: Left: Without 3D object masking and conditional querying, LERF cannot capture oblong object shapes. Right: CLIP can sometimes fail on generic prompt queries, like the poor activation the box. Additionally, LERF-TOGO struggles with groups of connected objects the object mask groups them all together.
on geometric cues. To balance relevance and success likelihood, we combine the grasp score s 0.958 0.05sgeom. ensure that consider the most relevant grasps while slightly biasing towards confident grasps.
Experiments
Part-Oriented Grasping We evaluate LERF-TOGO on a wide variety of 31 different objects and 49 total object parts to grasp (Fig. 4). For each object, we select an object query by describing sufficiently to unambiguously differentiate between other objects the scene. We use semantic descriptions when possible, and add visual descriptions only when such descriptions are ambiguous (i.e using color differentiate multiple mugs in scene). We provide part query for each object by describing natural place for robot to grasp and lift (i.e. "handle", "plant stem", "ear hook", "frame"). In addition, several objects include different grasp locations. A grasp is successful if it lifts the correct object using the appropriate subpart at least 10cm vertically, and the object remains securely within the gripper jaws throughout. For each query, we measure 1) whether the selected grasp was on the correct object, 2) whether the selected grasp was on the correct object part, and 3) whether the grasp successfuly lifted the object from the table. Every scene is reconstructed once in the beginning, after which the objects are removed sequentially (i.e., objects are removed one-by- one) with no updates in the scene representation. For full list of object queries see the Appendix.
ConceptFusion [79]
LEER-TOGO
Correct Object Correct Part
77%
96% %
Successfuly Lifted 39%
82 %
69
Table 1: Part-Ban Part-Based Grasping Results: Results are reported across 49 different prompts and 12 scenes. See the Appendix for complete list of scenes and queries.

---

Task-Oriented Grasping LERF-TOGO accepts a natural language part query as input, allowing to be used alongside large language models (LLMs) to generate parts hased on the task. To investigate if the LLM can also generate the object part, we use an LLM (ChatGPT) to generate the object and part query automatically via few-shot prompting. Results are shown in Table 3. The prompt and all tasks are included in the Appendix. Given the task and the list of objects in the scene, the LLM is tasked with generating the correct object and object part pair (object, part). We used majority voting scheme to query the LLM. Given the task, the LLM provides seven candidates that we use to select the pair (object, part) that appears in majority of the responses. We also mention details to integrate with an LLM planner in Section D.3 in the Appendix.
5.1 Comparisons
SemAbs [77]
OWL-ViT [69]
Human- -umaan
LLM-Query
Correct Object
80%
85
Correct Object
96 %
96%
Correct Part
35%
50 %
Correct Part
82 %
71%
Table 2: Single View Comparisons: Results are reported across 20 different prompts and 5 scenes. See the Appendix for a complete list of scenes and queries.
Table 3: Task-Oriented Success: Results ing an L.LM to choose the object part given a task specification. Results are reported across 49 different prompts and 12 scenes.
ConceptFusion [79] generates multimodal point cloud of scene by fusing RGBD images and their extracted features together. To query ConceptFusion, object and part prompt (i.e. "mug handle") and rank grasps via the highest similarity. We report the object and part success without physical evaluation. More details are in Appendix Section D.2.
provide it with the concatenated
Semantic Abstraction [69] takes a single RGBD frame as input and text query and outputs relevancy heat map over the image. A query success if the majority of the heatmap overlaps with the object part. In instances where activations are detected on other objects, it is considered successful if the highest activation is on the desired object part. Detailed results can be found in Table 2 and the Appendix.
OWL-VIT [80] is an ooen- detector which takes in an RGB image and text prompts and outputs segmentation maps. We provide OWL-ViT single input image that encompasses all object parts for fair comparison. To obtain an object mask, we use the object prompt establish initial bounding box. This box serves order to deem the part box as successful, we visually confirm that it aligns with the object part. Results can be found in Table 2 and the Appendix.
region identify the highest-scoring part within the region. In
Results
Part-Oriented LERF-TOGO overall achieves a 69% success rate for physically grasping and lifting objects by the correct part. The selected grasp was located around the correct object part 82% of the time, with the remaining failures being grasp execution failures. For context, the highest confidence geometric grasp on an object mask only lies on the correct part 18% of the time, suggesting LERF- TOGO meaningfully biases the grasp distribution to the object part. Selected task-oriented queries are visualized in Fig. 2: the distribution of grasps drastically shifts based on the given part query, and can focus task-orienter grasps on multiple different regions per object based on the language prompt. LERF-TOGO shows strong language understanding performance for object selection (96%), able differentiate between very fine-grae sine-graieed language queries like color, appearance ("matte" vs "shiny"), or semantically similar categories ("popsicle" vs "lollipop") "). It also can recognize long-tail tail object queries like "ethernei dongle", " "cork" or "martini glass". owing to its usage of CLIP zero-shot.
Orientree Combining few-shot LLM prompting with TOGO identifies the correct prim- itive with 92% success and produces grasps on the correct object with 71% success across 49 tasks on 39 different objects. The LLM could identify the object in the scene with the same success rate as the human, giving the correct object and part pairs for tasks like "scrub the dishes" and "cut the
7

---

steak". However, the LLM had a lower success rate (71%) compared to the human (82%) for ob- ject part selection. This is because CLIP, and by extension LERF-TOGO, can be sensitive to subtle variations in wording like "body" vs. "base" resulting in different LERF activations and thus grasps.
ConceptFusion LERF-TOGO out-performs ConceptFusion by 43% at task-oriented grasping be- cause it can capture multi-scale semanties, while ConceptFusion is limited to one CLIP embedding per point. This makes hierarchical querying difficult, and is reflected by the fact that ConceptFusion performs similarly to LERF-TOGO at selecting the correct object, but suffers at selecting the right object part. Due to its lack of cale-conditioning. ConceptFusion frequently emphasizes sections of the table due to the inclusion of both the objects and the table itself in the mask proposals (Fig. 9).
Semantic Abstraction achieves overall object detection rate of 80% and part detection of 35%. The method tends to produce empty relevancy responses when queried for specific object parts, potentially owing to its averaging across multiple scales which drowns out smaller part features. When presented with the object and part, the method highlights all of the object, owing to CLIP's bag of words behavior, a characteristic addressed by LERF-TOGO'S compositional queries. (Fig. D.4).
OWL-ViT achieves 85% accuracy for object localization, struggling on very long-tail objects that not encountered within the detection datasets. This behavior is amplified for object part queries, where queries tend to be long-tail such as "measuring tape" and "ethernel dongle" (Fig. 12).
Failures The primary failure modes of LERF-TOGO are mistaking visually similar object parts for one another (eg teapot spout for a handle), missing subtle geometries like the small teacup handle or spray trigger, or confusing very close categories like steak and bread knives. We also observe prompt sensitivity with part queries: for example "bottle neck" more strongly localizes grasps than "neck", and without more prompt tuning "body" sometimes fails to highlight the bases of bottles.
Object Extraction Ablation: Without 3D object masking and conditional querying. LERF-TOGO suffers with oblong objects, as shown in Fig. 5. We compare against querying LERF individually for the object and part, and multiplying their results together. This produces fragmented results which can ignore relevant parts of the object for part queries.
7 Limitations and Future Work
One limitation of LERF-TOGO is speed: the entire end-to-end the process takes a few minutes which can be impractical for time-sensitive applications. Future work on additional regularizations and optimizations LERF training may reduce computation time. Another key limitation of LERF- TOGO is with groups of connected foreground objects, as the DINO flood-fill mask will include the whole foreground group instead of isolating the individual components. Supporting hierarchy within foreground groups is critical to enable such cases. If the object query matches multiple objects in the scene (identical objects, or generic query like "mug" in the mugs scene), the system will arbitrarily choose only one of them. LERF-TOGO does not consider referrin//comparative expressions (e.g., "mug next to the plate", "biggest mug"). In addition, though we present a method for obtaining object part queries from input task descriptions via LLMs, future work we will evaluate its performance diverse set of tasks.
8 Conclusion
This paper presents LERF-TOGO, a method for using lasiun-language models zero-shot with Lan- guage Embedded Radiance Fields to grasp objects and their parts via language. By improving the spatial grouping of LERF relevancy outputs, LERF-TOGO can support hierarchical part queries conditioned on the full object. Results indicate performs strongly at language-guided grasping, with grasps landing the correct object 96% of the time, and furthermore can direct grasps to the correct object parts 81% of the time.
8

---

Acknowledgements
This research was performed at the AUTOLAB at UC Berkeley affiliation with the Berkeley Al Research (BAIR) Lab. The authors were supported in part by donations from Toyota Research Institute, Bosch, Google, Siemens, and Autodesk. Chung Min, Justin, and Lawrence are supported in part by the National Science Foundation Graduate Research Fellowship Program. Any opinions, findings, and conclusions, recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the Sponsors. We would like to thank our colleague Brent Yi for his work on Viser, the visualization tool of our experimental setup. We thank our colleagues who provided helpful feedback and suggestions, in particular Simeon Adebola and Julia Isaac.

---

References
[1] J. Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik. Lerf: Language embedded radiance fields. ICCV, 2023.
[2] K. Kleeberger and M. Huber. Single shot 6d object pose estimation, 2020.
[3] K. Bousmalis, A. Irpan, P. Wohlhart, Y. Bai, M. Kelcey, M. Kalakrishnan, L. Downs, J. Ibarz, Pastor, K. Konolige, S. Levine, and V. Vanhoucke. Using simulation and domain adaptation to improve efficiency of deep robotic grasping, 2017.
[4] S. Song, A. Zeng, J. Lee, and T. Funkhouser. Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations. Robotics and Automation Letters, 2020.
[5] Tobin, R. Fong, A. Ray, Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation the real world, 2017.
[6] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine. Qt-opt: Scalable deep reinforcement learning for usion-basedd robotic manipulation, 2018.
[7] S. Levine, Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal Robotics Research, 37(4-5):411-336 2018. do::10.117700777344491777
[8] A. Mousavian, C. Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for object manipulation, 2019.
[9] Mahler, M. Matl, V. Satish, M. Danielczuk, B. DeRose, S. McKinley, and K. Goldberg. Learning ambidextrous robot grasping policies. Science Robotics, 4(26):camu4984, 2019.
[10] L. Pinto and A. Gupta. Supersizing self supervipevi:::on Learning to grasp from 50k tries and 700 robot hours. In 20/6 IEEE international conference on robotics and automation (ICRA), pages 3406-36 3413. IEEE, 2016.
[11] T.-Y. Lin, M. Maire, Belongie, L. Bourdev, R. Girshick, Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollár. Microsoft coco: Common objects in context, 2015.
[12] A. Depierre, E. Dellandréa, and L. Chen. Jacquard: A large scale dataset for robotic grasp detection. In 2018 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 351 151166. IEEE, 2018.
[13] I. Lenz, H. Lee, and A. Saxena. Deep learning for detecting robotic grasps. The International Journal Robotics Research, 34(4-5):705-72) 2015.
[14] B. Calli, A. Singh. A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar. The yeb object and model set: Towards common benchmarks for manipulation research. 2015 international conference on advanced robotics (ICAR), pages 510-517. IEEE, 2015.
[15] H.-S. Fang, C. Wang, M. Gou, and Lu. Irallont- A large-scale benchmark for general object grasping. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pages 11444-11453 2020.
[16] G. Li, V. Jampani,
D. Sun, and I.. Sevilla-Lara. Locate: Localize and transfer object parts for weakly supervised affordance grounding. In Proceedings of the IEEE International Conference on Computer Vision, 2023.
[17] A. Murali, W. Liu, K. Marino, S. Chemova, and A. Gupta. Same object, different grasps: Data and semantic knowledge for task-orienter grasping. In Conference on Robot Learning, 2020.
10

---

[18] M. Kokic, D. Kragic, and J. Bohg. Learning task-oriented grasping from human activity datasets. IEEE Robotics and Automation Letters, 5(2):3352-3359, 2020.
[19] D. Song, C. H. Ek, K. Huebner, and D. Kragic. Task-based robot grasp planning using proba- bilistic inference. IEEE transactions on robotics, 31(3):546 -50:: 2015.
[20] R. Detry, J. Papon, and L.. Matthies. Task-oriented grasping with semantic and geometric scene understanding. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3266-32662 3273. IEEE, 2017.
[21] A. Myers, L. Teo, Fermüller, and Y. Aloimonos. Affordance detection tool parts from geometric features. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 1374-1381. IEEE, 2015.
[22] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021.
[23] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pages 9650-9660, 2021.
[24] F-J. Chu, R. Xu, and P. A. Vela. Learning affordance segmentation for real-world robotic manipulation via synthetic images. IEEE Robotics and Automation Letters, 4(2): 140-1147, 2019. doi:10.1 109/ 289449....
[25] N. Vahrenkamp, L. Westkamp. N. Yamanobe, E. E. Aksoy, and T. Asfour. Part-based grasp planning for familiar objects. In 2016 IEEE-RAS 16th International Conference on Humanoid Robots (Hummnoidss, pages 919-925, 2016. 0000101100HUUAANNNS.200.........
[26] Y. Song, P. Sun, Ren, Y. Zheng, and Y. Zhang. Learning 6-dof tne-grained grasp detection based on part affordance grounding. arXiv preprint arXiv:2301. //564, 2023.
[27] D. Hochbaum and V. Singh. An efficient algorithm for co-segmentation. In 2009 IEEE 12th International Conference Computer Vision, pages 269 276. IEEE, 2009.
[28] Y. Chai, V. Lempitskky and A. Zisserman. Bicos: A bi-level co-segmentation method for image classification. In 2011 International conference computer vision, pages 2579-2586. IEEE, 2011.
[29] S. Amir, Y. Gandelsman, S. Bagon, and T. Dekel. Deep vit features as dense visual descriptors. ECCVW What is Motion For?, 2022.
[30] P. Sun, S. Chen, C. Zhu, F. Xiao, P. Luo, S. Xie, and Z. Yan. Going denser with open- vocabulary part segmentation. arXiv preprint 2rXiv:2305.11172 2023.
[31] B. Mildenhall, P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, pages 405-42 Springer, 2020.
[32] M. Adamkiewicz, T. Chen, A. Caccavale, R. Gardner, P. Culbertson, J. Bohg, and M. Schwa- ger. Vision-only robot navigation in neural radiance world. IEEE Robotics and Automation Letters, 7(2):4606-4613, 2022.
[33] T. Barron, Mildenhall, M. Tancik, Hedman, R. Martin-Brualla, and P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855-5864, 2021.
11

---

[34] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Mip-nerf 360: Un- bounded anti-aliased neural radiance fields. In Proceedings of the IEEFJCVE Conference on Computer Vision and Pattemn Recognition, pages 5470-5479, 2022.
[35] L. Ma, X. Li, J. Liao, Q. Zhang, X. Wang. J. Wang, and P. V. Sander. Deblur-nerf: Neural radiance fields from blurry images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12861-12870, 2022.
[36] X.
Huang. Q. Zhang, Y. Feng, H. Li, X. Wang, and Q. Wang. Hdr-nerf: High dynamic range neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18398 18998-1840 18408, 2022.
[37] S. Sabour, S. Vora, D. Duckworth, Krasin, D. J. Fleet, and A. Tagliasacchi. Robustnerf: Ignoring distractors with robust losses. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 20626-20636, 2023.
[38] J. Philip and V. Deschaintre. Radiance field gradient scaling for unbiased near-camera training. arXiv preprint arXiv:2305.02756, 2023.
[39] M. Tancik, E. Weber, E. Ng. Li, B. Yi, J. Kerr, T. Wang, A. Kristoffersen, Austin, K. Salahi, et al. Nerfstudio: A modular framework for neural radiance field development. arXiv preprini arXiv:2302.04264 2023.
[40] Wang, Y. Liu, Z. Chen, L. Liu, Z. Liu, Komura, C. Theobalt, and W. Wang. F2-nerf: Fast neural radiance field training with free camera trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4150-4159, 2023.
[41] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. arXiv preprint arXiv:2304.06706 2023.
[42] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with mul- tiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4):1-15, 2022.
[43] A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su. Tensorf: Tensorial radiance fields. In Com- puter Vision-ECCC 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII, pages 333-350. Springer, 2022.
[44] S. Fridovich-Keil, G. Mcanti, F. R. Warburg, B. Recht, and A. Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12479-12488, 2023.
[45] A. Yu, S. Fridovich-Kan Keil, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa. Plenoxels: Radi- ance fields without neural networks. arXiv preprint
2021.
[46] K. Park, U. Sinha, Hedman, Barron, S. Bouaziz, D. B. Goldman, R. Martin Martin-Brualla, and M. Seitz. Hypernerf: higher-dimensional representation for topologically varying neural radiance fields. arXiv preprint urXiv:2106.13222 2021.
[47] Z. Li, Q. Wang, F. Cole, R. Tucker, and N. Snavely. Dynibar: Neural dynamic image- ren- dering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 4284, 2023.
[48] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Nogus D-NeRF: Neural Radiance Fields for Dynamic Scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[49] Z. Zhu, S. Peng, V. Larsson, W. Xu, H. Bao, Z. Cui, M. R. Oswald, and M. Pollefeys. Nice- slam: Neural implicit scalable encoding for slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12786-12777 12796, June 2022.
12

---

[50] E. Sucar, S. Liu, J. Ortiz, and A. J. Davison. imap: Implicit mapping and positioning in real- time. Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 6229-62223 6238, October 2021.
[51] A. Rosinol, J. J. Leonard, and L. Carlone. Nerf-slam: Real-time dense monocular slam with neural radiance fields, 2022.
[52] Y. Li, S. Li, V. Sitzmann, Agrawal, and A. Torralba. 3d neural representations for visuomotor control. In Conference Robot Learning, pages 112-123, PMLR, 2022.
[53] D. Driess, I. Schubert, P. Florence, Li, and M. Toussaint. Reinforcement learning with neural radiance fields. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[54] J. Kerr, L. Fu, H. Huang, Y. Avigal, M. Tancik, Ichnowski, A. Kanazawa, and K. Gold- berg. Evo-neRF: Evolving neRF for sequential robot grasping transparent objects. In 6th Annual Conference on Robot Learning, 2022. URL Bxr45keYrf.
[55] J. Ichnowski*, Y. Avigal* * J. Kerr, and K. Goldberg. Dex-NeRF: Using neural radiance field to grasp transparent objects. In Conference on Robot Learning (CoRL), 2020.
[56] A. Byravan, J. Humplik, I.. Hasenclever, A. Brussee, Nori, T. Haarnoja, B. Moran, S. Bohez, Sadeghi, B. Vujatovic, and N. Heess. Nerf2real: Sim2real transfer of vision-guided bipedal motion skills using neural radiance fields, 2022.
[57] S. Zhi, T. Laidlow, S. Leutenegger, and J. Davison. In-place scene labelling and under- standing with implicit scene representation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 15838-15847, 2021.
[58] Y. Siddiqui, L. Porzi, S. R. Buló, N. Müller, M. Nießner, A. Dai, and P. Kontschieder. Panoptic lifting for 3d scene understanding with neural fields, 2022.
[59] A. Kundu, K. Genova, X. Yin, A. Fathi, C. Pantofaru, L. Guibas, A. Tagliasacchi, F. Del- laert, and T. Funkhouser. Panoptic Neural Fields: Semantic Object-Aware Neural Scene Representation. In CVPR, 2022.
[60] S. Kobayashi, E. Matsumoto, and V. Sitzmann. Decomposing nerf for editing via feature field distillation. In NeurIPS, volume 35, 2022. URL botps://arxtts org/pdf/ pptpppdff.02..2///2 15585. pdf.
[61] V. Tschernezki, I. Laina, D. Larlus, and A. Vedaldi. Neural feature fusion fields: 3d distillation of supervepee 2d image representations. In Proceedings of the International Conference on 3D Vision (3DV), 2022.
[62] B. Li, K. Q. Weinberger, Belongie, V. Koltun, and Ranftl, Language-driven semantic segmentation. arXiv preprint trxiv:2201.03566, 2022.
[63] C. Huang, O. Mees, A. Zeng, and W. Burgard. Visual language maps for robot navigation. arXiv preprint arXiv:2210.05714, 2022.
[64] S. Peng, K. Genova, C. Jiang, A. Tagliasacchi, M. Pollefeys, T. Funkhouser, et al. Openscene:
3d scene understanding with open vocabularies. arXiv preprint arXiv:2211.15654. 2022.
[65] G. Ghiasi, X. Gu, Y. Cui, and T.-Y. Lin. Open-vocabulary image segmentation. arXiv preprint 1rX1v:2112.12144, 2021.
[66] N. M. M. Shafiullah, C. Paxton, L. Pinto, S. Chintala, and A. Szlam. Clip-fields: Weakly supervised semantic fields for robotic memory. arXiv preprint arXiv:2210.05660 2022.
13

---

[67] K. M. Jatavallabhula, A. Kuwajerwala, Q. Gu, M. Omama, T. Chen, S. Li, G. Iyer, S. Saryazdi, N. Keetha, A. Tewari, et al. Conceptfusion: Open-set multimodal 3d mapping. arXiv preprint arXiv:2302.07221,, 2023.
[68] B. Chen, F. Xia, B. Ichter, K. Rao, K. Copalakrishnan, M. S. Ryoo, A. Stone, and D. Kap- pler. Open-vocabulary queryable scene representations for real world planning. arXiv preprint 2XXiv2209.098744 2022.
[69] H. Ha and S. Song. Semantic abstraction: Open-world 3D scene understanding from 2D vision-language models. In Proceedings of the 2022 Conference Robot Learning, 2022.
[70] H. Chefer, Gur, and L. Wolf. Generic attention explainability for interpreting bi- modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 397-406, 2021.
[71] O. Mees and W. Burgard. Composing pick-and-place tasks by grounding language, 2021.
[72] L. Yu, Z. Lin, X. Shen, J. Yang, X. Lu, M. Bansal, and T. L. Berg. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1307-1315, 2018.
[73] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021.
[74] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Proceedings of the 6th Conference on Robot Learning (CoRI.), 2022.
[75] A. Stone, T. Xiao, Y. Lu, K. Gopalakrishnan, K.-H. Lee, Q. Vuong, P. Wohlhart, B. Zitkovich, Xia, C. Finn, et al. Open-world object manipulation using pre-trained vasion-language mod- els. arXiv preprint arXiv:2303.00000 00905, 2023.
[76] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:22/2.068)
2022.
[77] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniee, V. Khalidov, P. Fernandez, D. Haz- iza, F. Massa, A. El-Nouby, al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
[78] N. Liu, S. Li, Y. Du, A. Torralba, and B. Tenenbaum. Compositional visual generation with composable diffusion models. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII, pages 423 439. Springer, 2022.
[79] K. Jatavallabhula, A. Kuwajerwala, Q. Gu, M. Omama, Chen, Li, G. Iyer, S. Saryazdi, N. Keetha, Tewari, J. Tenenbaum, C. de Melo, M. Krishna, L. Paull, Shkurti, and Tor- ralba. Conceptfusion: Open-set multimodal 3d mapping. arXiv, 2023.
[80] M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy, A. Ma- hendran, A. Arnab, M. Dehghani, Shen, X. Wang, X. Zhai, T. Kipf, and Houlsby, Simple open-vocabulary object detection with vision transformers, 2022.
[81] S. F. Bhat, R. Birkl, D. Wofk, P. Wonka, and M. Müller, Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302. 12288, 2023.
[82] Guangcong, Z. Chen, C. C. Loy, and Z. Liu. Sparsenerf: Distilling depth ranking for few-shot novel view synthesis. Technical Report, 2023.
[83] M. Niemeyer, J. T. Barron, Mildenhall, M. M. Sajjadi, A. Geiger, and N. Radwan. Reg- nerf: Regularizing neural radiance fields for view synthesis from sparse inputs, 2021.
14

---

A Appendix
A.1 Implementation Details
We implement LERF-TOGO on top of the Nerfacto method from Nerfstudio [39]. For faster con- vergence and smoother optimization, we modify several parameters from the original LERF paper. We use a smaller hashgrid with 16 levels and a maximum resolution of 256, and find that using larger MLPs for the density, color, and transient output heads of NeRF results in faster convergence and better ability handle specularities and robot shadows. We introduce weight decay of le-7 to the LERF network which smooths training. In addition, we compress the DINO embeddings into 128 dimensions and supervise these vectors rather than the original DINO outputs, but we do not normalize the resulting vectors like Tschernezki et al. [61]. While constructing the CLIP em- bedding pyramid for LERF, we use crops ranging from 5% to 35% of image height with 6 pyramid levels, biasing the pyramid to smaller crops as LERF-TOGO is primarily interested object and part queries.
B Robot Capture
Scene Reconstruction The robot uses a wrist-mounted camera to capture the scene with a hemi- spherical trajectory centered at the workspace, similar Evo-NeRF [54]. The capture has a radius of 45 cm and ares from +100 around the workspace horizontally and an inclination range of 30 to 75%. We capture images while the arm moves at 15 cm/sec at rate of 3 hz, resulting in around 60 images per capture. We discard blurry images by analyzing the variance of the image Laplacian, ensuring the images are high quality. While the robot moves, we pre-process each image to extract DINO features, multi-scale CLIP, and ZoeDepth [81], which are used during LERF training. See Appendix B for details.
Robot Capture Region Size The robot captures the scene along hemispherical trajectory arcing 1000 around the workspace horizontally ("1/2 hemisphere" in Fig. B.) When this horizontal sweep angle is reduced to fraction of the range, the quality of the 3D object mask degrades, sometimes selecting the incorrect object altogether. LERF's semantic field is supervised on features of the scene images, thus the quality is heavily correlated with the distribution of images viewing the object. This lowers the quality of the 3D DINO embeddings used for the mask generation.
LERF Training Steps In experiments the LERF scene representation is trained to 2k steps. As shown in Fig. B, objects/object parts (e.g.. "spray nozzle", "bottle") can be detected in steps as low as 1k, but more fine- tme-grained or smaller parts (e.g., "handle") may take longer (2-3k steps). This is consistent with what LERF reports: "fine-grained features take more steps [to emerge]""
3D Object Extraction Given the initial 3D point with the highest LERF activation, we create an object-centric point cloud by rendering six different views looking at the 3D coordinate. The views are - around the upwards vector through the 3D point. For DINO floodfill, the threshold DINO similarity is defined as first projecting the current DINO embedding onto the first PCA component of the top-down image, then taking the L2 norm of the difference between the current embedding and the DINO embedding at the initial 3D point.
NeRF Regularization NeRF encounters difficulties in reconstructing texture-less planar surfaces, especially in the presence of specularities. This limitation is prominent in our table-top scenes, where the glossy surface and metallic objects can result in depth renderings with jagged missing regions. These missing regions can cause LERF renderings to spuriously activate and degrade the performance of grasp networks, so we apply depth regularization to mitigate this issue. We adopt the local depth ranking loss proposed in SparseNeRF [82] and use ZoeDepth [81] as the underlying depth model. We found this performs better than smoothness priors [83, 45] because it retains more fine-grained geometry. Additionally, we use the gradient scaling approach from Philip and Deschaintre [38], which significantly reduces the number of near camera floaters and enables more robust grasping directly from point clouds rendered from the NeRF.
15

---

Robot Capture Hemisphere Size
LERI Training Steps
Robot hemisphore
hemisphere
1/2 hemisphere
1k
2k
"Pyple spray bodbey body
Blue mags Nemale"
Figure 6: Left: Decreasing the robot scan region degrades the quality of 3D relevancy map generated by LERF-TOGO, Right: LERF-TOGO relevancy map converges earlier for larger objects and parts.
Robot Scene
LERF relevancy
LERF-TOGO relevancy
Robot Soene
LERF relevancy
LERF-TOGO relevancy
"Small cupy
"mug:
Figure 7: the case of identical objects or ambiguous object queries, LERE-TOGO picks a single object in the scene but does not propose grasps for all the objects in the category.
Poses obtained from cameras in motion are slightly inaccurate, which we found could result in oversmoothed geometry with depth regularization. To overcome this, we optimize the NeRF for the first 500 steps without any regularization allow the camera poses to settle, then anneal the depth regularization loss term from 0 to 100% over the next 1500 steps. Interestingly, we find staged training not only preserves thin features better but also speeds LERF optimization. We hypothesize this is because supervising the language field on un-converged density in free space results in a poor network initialization, while beginming LERF optimization after geometry has been largely removed from free space allows a smoother learning signal.
C Grasping
Point Cloud Extraction To extract scene-wide point cloud for grasping, Nerfstudio [39]. which deprojects randomly sampled rays' depth from the train camera poses, then filters with outlier rejection. We then crop the point cloud to the workspace of the robot. For object centric point clouds, we deproject depth from views radially surrounding the object of interest.
the method in
Motion Planning grasp considered feasible if the robot can perform collision-fres trajectory with the following poses: the pre-grasp, grasp, and post-grasp configurations. The pre-grasp pose is positioned 5cm along the z-axis of the robot end effector, which allows the gripper to approach the target grasp pose with minimal additional motion. The post-grasp pose is located 10cm above the grasp pose, along the z-axis of the world frame. The UR5's IK solver calculates the set of viable joint configurations at these poses, and we calculate the trajectory as linear interpolation between them. We additionally allow for a 180 degree rotation at the last wrist joint, as parallel-jaw grasps are rotationally symmetric. This facilitates the motion planning process, the robot's camera mount prone to colliding with the robot arm.
D Experiments
D.I Setup Details
We UR5 arm with Logitech BRIO webcam at 1600x896 resolution, with all camera settings frozen before each capture prevent color discrepancies among images. The camera mount points
16

---

Robot scene
Grasp Distribution, S=W'Sam Smm (1-W) Squar
"power strip; plug"
w=0.0
w=0.2
w=0.4
w-0.6
w=0.8
w=0.95
Robot
"Dalsy; plant stem"
w=0.0
W=0.2
w=0.4
w=0.6
W=0.8
w=0.95
Figure 8: Grasp score weighting: Varying weight between geometric grasp score and semantic grasp score shifts the grasp distribution. A high semantic grasp weight (w = 0.95) is required, since geometric grasps may be biased away from small and grainnd object parts of interest. Both geometric and semantic scores are in the range [O, 1].
LERF
ConceptFusion
"Black Mug"
"Handle"
"Black Mug"
"Handle"
"Screwdriver"
"Handle"
"Screwdriver"
"Handle" "
ConceptFusion performs well on object- level queries, but struggles with sub-object part queries because of its lack of multi-scale semantics.
Figure 9: Comparison with LERF and ConceptFusion:
orthogonally to the gripper axis, maximize the reachable workspace of the camera while pointing towards the workspace center. During robot capture, pre-computation of DINO, CLIP, and ZoeDepth parallelized across 3 NVIDIA 4090 GPUs to achieve real-time performance, and all subsequent operations are carried out a single 4090. Capturing a scene takes 30 seconds, training the LERF to 2k iterations takes 78 seconds, and finally querying LERF-TOGO takes 10 seconds.
D.2 Comparison to Baselines
ConceptFusion We provide ConceptFusion with depth generated from the NeRF, which results in high-quality point clouds for grasping. To represent the paper, we use the OpenCLIP ViT-H/14 model, which is larger than the ViT-B/16 model for LERF-TOGO.
Semantic Abstraction Since the method takes single image, provide the method with input image observing all object parts for fair comparison. We provide it with part queries ways and take the best performance: I) the concatenated object and part prompt (i.e. "mug handle") "), and 2) the object and part as separate queries.
D.3 Integration with an LLM Planner
LERF-TOGO can integrate as a module with an LLM planner to combine task-oriented grasps for robotic manipulation tasks. We define set of robotic manipulation primitives (grasp, press, twist, pick&place, pour) and prompt the LLM to output the correct primitive for given task. We the same majority voting scheme in the previous section to select both the correct robotic primitive and the pair (object, part). Now, given a task (c.g. 'uncork the wine'), LLM can specify the action
17

---

Scene
Object Query Part Query
Kitchen
(black matte spoon, handle), (shiny black spoon, handle), (teapot, handle), (dish scrub brush, handle), (dust brush, handle)
Flowers
(daisy, plant stem), (rose, plant stem)
Mugs
(blue mug, handle), (pink teacup, handle), (turqouise mug, handle), (white mug, handle), (black mug, handle)
Tools
(Measuring tape, base), (screwdriver, handle), (wire cutters, handle) (soldering iron, handle), (hammer, handle)
Knives
(bread knife, handle), (steak knife, handle), (box cutter, handle)
Martinis
(red martini glass, stem), (grey martini glass, stem)
Fragile
(camera, strap), (pink sunglasses. earhooks)
(blue sunglasses, carhooks), (lightbulb, screw)
Cords
(power strip, plug), (power strip, base), (ethernet dongle, usb) (ethernet dongle, ethernet)
Messy
(ice cream, cone), (green lollipop, stick), (blue lollipop, stick)
Pasta
(wine, cork), (wine, bottle neck), (saucepan, lid knob)
(saucepan, handle), (corkscrew, handle)
Cleaning
(clorox, wet towel), (clorox, lid), (clorox, body)
(tissue box, box), (tissue box, tissue)
Bottles
(meyer's 's cleaning spray, spray trigger), (meyer's ' cleaning spray, bottle neck)
(meyer's 's cleaning spray, body) (purple cleaning spray, body) (purple cleaning spray, spray trigger). (purple cleaning spray, bottle neck)
Electronics
(e-stop. red button),, (computer mouse, scroll wheel), (controller, buttons), (spacemouse, button). (white controller; button) (controller, joystick), (e-stop, red button) Table Complete list of object and part queries
accomplish the task ('grasp') and the pair of object and object part (e.g. 'wine' and 'cork'). The prompt and all tasks are included in the Appendix.
D.4 LLM Interface
We provide the full prompt to the LLM below. For any given task and scene the OBJECT.LIST is replaced with list of objects within the scene and TASK is replaced with the desired task:
Answer the question if you are robot with parallel jaw gripper that has access to only the objects in the object list. Follow the exact format.
First line should describe what basic action is needed to do the task from the following set of actions: press, grasp, twist, pick place.
second line should only be an object from the object list followed by object part that the robot would touch to do this task. VERY IMPORTANT: the basic action is pick & place, only then have third 1ine with 'Place: to specify the object to place on. \
Object list: ['pot', 'knife', 'spoon', 'black pan'] \ Q: How can safely pick up a pan? \
Basic Action: grasp \
Sequence: black pan handle
Object list: ['mechanical keyboard', 'knife', 'TV', > camera'] \ Q: How I safely hit the spacebar keyboard? \
18

---

Basic action: press
Sequence: 1. mechanical keyboard 2. spacebar)
Object list: ['green mug', 'blue spoon', 'ook'' 'knife'] \ Q: How can cut block of cheese?
Basic action: grasp \
Sequence: 1. knife 2. handle \
Object list: ['salt shaker', 'knife', 'fork', 'white pan']
Q: How can safely lift a salt shaker?
Basic Action: grasp \
Sequence: salt shaker 2. base
Object list: ['red cup', bluu cup', 'mug', 'bowl '] \ Q: How do I stack the red cup the blue cup?
Basic action: pick & place
Sequence: 1. red cup 2. rim
Place: blue cup
Object list: ['door knob', 'black mug', 'green dish brush', 'shiny knife'] \ Q: How do I open door knob? \
Basic action: twist \
Sequence: 1. door knob 2. rim
Object list: ['dryer', 'washing machine', 'sunglasses']
Q: How I turn on washing machine?
Basic action: twist
Sequence: 1. washing machine 2. dial
Object list: ['paper towel roll', 'mug', 'teacup', 'headphones', 'pen') ']
Q: How do I grab a paper towel?
Basic action: grasp \
Sequence: paper towel roll 2. paper towel \
Object list: ['magnifying glass', 'blue spoon', 'fork', 'knife'] \
Q: How do I pick up a magnifying glass?
Basic action: grasp \
Sequence: 1. magnifying glass 2. handle
Object list: ['teddy bear', 'toy block', 'mouse', 'saucepan', 'hammer"
Q: How grab a teddy bear?
Basic action: grasp \
Sequence: 1. teddy bear 2. head
Object list: ['green mug', 'blue spoon', 'fork', 'knife'] \ Q: How do I put the mug in the cabinet?
Basic action: pick & place
Sequence: green mug handle Place: cabinet
Object list: {OBJECT_LIST)
Q: How can I safely (TASK)? Basic action:
19

---

Power Strip
Power Strip Plug
Power Strip Base
Plug
USB
Ethernet Port
Base
Ethernet Dongle
Ethernet Port USB
Camera
Camera Strap
Lightbulb
Lightbulb Screw
Screw
Blue Sunglasses
Pink Sunglasses
Earhooks
Blue Sunglasses Earhooks
Figure 10: Semantic Abstraction results for object and object part localization (cont.)
20

---

White Mug
Pink Teacup
Pink Teacup Handle
Turquoise Mug
Turquoise Mug Handle
Blue Mug Handle
Handle
Black Mug
Black Mug Handle
Wire Cutters
Wire Cutters Handle
Soldering Iron
Soldering Iron Handle
Screwdriver
Screwdriver Handle
Measuring Tape
Measuring Tape Base
Handle
Hammer
Hammer Handle
Base
Figure 11: Semantic Abstraction results for object and object part localization
21

---

Scene
Tasks
Kitchen
"grab the matte spoon', passs the shiny spoon', 'grah the teapot', 'scrub the dishes', 'dust the book'
Flowers
give the daisy', "give the rose'
Mugs
ggabb the blue mug', grab the pink teacup', 'grab the turquoise mug'
'grab the white mug'. "grab the black mug'
Tools
'grab the measuring tape'. "give the screwdriver', 'cut the wire' 'hold the soldering iron', 'swing the hammer'
Knives
cut the bread', 'cut the steak', 'cut the box'
Martinis
'deliver the grey martini glass', 'deliver the red martini glass'
Fragile
'take a picture', "wear the pink sunglasses' 'wear the blue sunglasses', 'move the lightbulb"
Cords
"grab the power strip', 'plug in the power strip' 'plug the ethernet into the ethernet dongle',
plug in the usb into the ethernet port'
Messy
eaa the ice cream', 'eat the green lollipop', 'eat the blue lollipop'
Pasta
'uncork the wine', 'grab the saucepan', 'open the saucepan', 'grab the corkscrew's 'lift the wine'
Cleaning
'grab a wet towel from the clorox', 'close the box of clorox'
"hold the clorox', 'get tissue', 'hold the tissue box"
Bottles
ggab the meyers cleaning spray', "open the meyers cleaning spray', sppra the meyers cleaning spray', 'grab the purple cleaning spray". "open the purple cleaning spray', 'spray the purple cleaning spray'
Electronics
'hold down the e-stop', 'hit the scroll wheel', "click the controller', 'click on that website with the spacemouse", 'click the white controller', 'turn video game character', 'release the e-stop" Table 5: Complete list of tasks for each scene
22

---

Power Strip Base
Ethernet Dongle Ethernet Port
Power Strip Plug
Ethernet Port Usb
Lightbulb Screw
Blue Sunglasses Earhooks
Pink Sunglasses Earhooks
Camera Strap
White Mug Handle
Blue Mug Handle
Black Mug Handle
Pink Teacup Handle
Turquaise Mug Handle
Hammer Handle
Soldering Handle
Wire Cuttare Handle
Screwdriver Handle
Measuring Tape Base
Daisy Plant Stem
Rose Plant Stem
Figure 12: OWL-ViT results for object and object part localization
23


Reviews:
[{"review_id": "vU0pzQoJZm", "reviewer": "Reviewer_5Sor", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work presents a pipeline LERF-TOGO that leverages LERF (Language Embedded Radiance Fields) alongside DINO-informed segmentation to perform semantically aware grasping with the help of language instructions. Given the description of the object of interest, LERF is used to identify where that object lies in the scene. A 3D object mask is segmented using flood fill starting from the point of interest identified by LERF. The flood fill is thresholded based on DINO features. Given the segmented object of interest, a language description of the part of interest is provided and used to further identify the portion of the segmented object that is of interest. GraspNet is used to derive grasps given multiple view angles, and proposed grasps are then weighted based on their geometric validity and semantic relevance. The results show that the proposed approach selects the grasps on the correct parts of the object of interest.", "summary_of_recommendation": "I believe the work would be valuable to the community. I would like the issues raised to be addressed in the rebuttal phase.\n\n**Post-rebuttal update:** The authors have answered my questions. I appreciate the improvement in the quality of the manuscript. My recommendation to accept this paper remains.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please refer to the comments in the \"Weakness\" section.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Strengths:**\n- The experiments clearly demonstrate successful execution on the real robot with significantly high rates.\n- The paper presents a way to perform task-oriented grasping while utilizing knowledge of general (foundational) models trained for grasping, vision-language understanding, and segmentation.\n- The proposed approach does not use fine-tuning or demonstrations to learn grasps or affordances.\n- The paper is well written, covering all the details necessary to understand the proposed pipeline. \n\n**Weakness or areas of improvement:**\n- It is unclear whether the authors employ the Evo-NeRF’s early stop training method. \n- The authors show that they can deal with a scenario where the scene has multiple mugs if the description specifies the unique properties of the mug of interest. It is unclear if the method can handle cases where the description still fails to disambiguate a few instances of the object in the scene.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "josi-zVEQ2k", "reviewer": "Reviewer_aACe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper aims to enable a robot to grasp a specific object at a specific part, where the “object + part” combination is specified in natural language. The main idea is to use a neural radiance field, which has been trained to render CLIP features (the recently proposed LERF), to represent the 3D scene, and use LERF to enable part-specific grasping. The main contributions are an implementation of the LERF-based pipeline for part-specific grasping, showing how to link LERF relevancy maps with generated 6-DoF grasps, and proposing a two-stage pipeline for using LERF to output relevancy maps that are faithful to the specified object parts. The method is evaluated in the real world in terms of the ability to identify the correct object, the correct part, and to execute a successful grasp at this part. The results show the proposed method outperforms one using a similar recently proposed scene representation and enables physically successful grasping 69% of the time. ", "summary_of_recommendation": "The core capability that's developed and demonstrated has definite utility, the pipeline shows a robotics use case for an interesting and powerful new tool (LERF), and the real-world demonstrations are convincing. However, the paper is both framed too broadly (\"task-oriented\" rather than \"part-oriented\" grasping) and the key technical components are not communicated clearly. \n\n---\n\nUpdating my score to weak accept after the rebuttal period", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Add LERF preliminaries, and introduce + explain the method more clearly\n- The term \"object part query\" is never properly defined or introduced. I found this term vague and hard to understand (i.e., “*conditions the object part query over this mask”*)\n- Clarifying details on the method:\n    - What does the “flood-fill” do?  \n    - How are you able to render an “object-centric point cloud” without the mask already available? Is this a crop of the scene near the 3D coordinate with the highest LERF activation?\n    - You’re deciding which points belong to the same object by thresholding how similar the DINO features are for the other nearby points in this crop?\n- Details on \"robot capture\", \"scene reconstruction\", and \"NeRF regularization\" can be reduced significantly, with the rest moving to the appendix. \n- Another relevant baseline is \"Semantic Abstraction\" (mentioned above). They also compute multi-scale relevancy maps using CLIP, and they have the added advantage of using a scene encoder so they can operate with new scenes. It's not necessarily critical to compare to this, although that would be great, but the paper should least comment on the similarities and how the two works relate. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary of strengths and weaknesses**\n\nStrengths:\n\n- The demonstrated ability to perform \"part-specific\" grasping is useful\n- Additionally, showcasing one way LERF can be used for representing + interacting with scenes in a semantic fashion for a robotics application is also a valuable contribution\n- The general approach is intuitive and easy to digest, most likely making this a component that can be built upon and/or integrated into more full-stack systems. \n- The paper includes good demonstrations of real-world capability\n\nWeaknesses:\n\n- The main weakness is the presentation of the method. While the high-level intuition can be understood pretty easily, I greatly struggled to understand many core details of how the pipeline actually works. It’s a major problem that there are absolutely no preliminaries on LERF included in the paper. The explanation of using the 3D-distilled DINO features for obtaining the object mask was very unclear.\n- A second weakness is the mismatch between the framing of the paper, as *task*-oriented grasping, whereas in my view, the results shown only go so far as *part*-oriented grasping. It might sound pedantic, but these are not the same. Open-set part segmentation could have been a baseline you used. It’s both more non-trivial, and also potentially more valuable/interesting to provide actual “functional/long-term/downstream” task specifications and see what this could have done.\n- Ablation evaluation - besides the full pipeline and integrating with grasping, the main innovation/contribution is the modifications to allow the part specification to work. But there are probably lots of other ways this could have been done, and showing at least some more exhaustive comparison (ideally quantitative) would be very beneficial\n    - For example, the section *Object Extraction Ablation* says: *Without 3D object masking and conditional querying, LERF-TOGO suffers with oblong objects, as shown in Fig. 5. We compare against querying LERF individually for the object and part, and multiplying their results together. This produces fragmented results which can ignore relevant parts of the object for part queries —*. Ideally, there would be more of this kind of evaluation and analysis, since the specific design choices that allow part-specific queries are the main technical innovation (besides the general integration of LERF with a grasp predictor). \n\n**Details**\n\n*Technical Quality -*: Good. Strengths: The paper uses both new and (now-)standard tools in a simple but effective combination. The authors investigated and reported on key technical issues that arise when attempting this pipeline in a naive fashion, and include other details for running the whole system (compute, pre-processing, reconstruction, etc.). Weaknesses: The evaluations and ablations could have been stronger for supporting the claim that this is a good way to enable (i) LERF to perform well with part-specific queries and (ii) LERF relevancy maps to be combined with a grasp prediction model. \n\n*Clarity -*: Fair. Strengths: The general problem setup and purpose of the paper are clear. Weaknesses: The clarity of the whole method was quite low. It was very hard for me to understand a significant portion of how the pipeline actually works and the core changes that were needed to allow LERF relevancy maps to be used properly. \n\n*Originality -*: Fair. Strengths: The changes needed to make LERF work with part queries appeared to be important for the pipeline to operate. Weaknesses: The components in the pipeline are not new (LERF with 3D-distilled CLIP features and DINO features, 6-DoF graspnet). The idea of computing the relevancy map “hierarchically” (using only the coarsely segmented object to find the required object part), while being important to enable the robotics application, is also not a very significant innovation. \n\n*Impact/Significance -*: Fair. Strengths: Part-specific grasping from language is likely to be useful in full-stack manipulation pipelines. Weaknesses: There are lots of 3D-distilled feature field/semantic point cloud pipelines being proposed, and there are lots of ways this overall pipeline could have been designed. From the evaluations and demonstrations, it's hard to walk away from this paper knowing \"this is the right way to do it\". Furthermore, the paper claims to be solving “task-oriented grasping” but in my view, it’s really addressing “part-oriented grasping”, which (while valuable) is a less significant contribution.\n\n**Other comments**\n- The paper is missing a citation for a very similar related work: \"Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models\", H. Ha, et al., CoRL 2022. Further comments in \"questions for rebuttal\" below. ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "DB6QZb1Onts", "reviewer": "Reviewer_B1ts", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper focuses on task specific grasp generation by leveraging the prior work of Language Embedded Radiance Fields (LERF) which proposed grounding language embeddings into a NeRF scene for open ended language queries onto it. The zero shot performance of LERF is used to conditionally query for a relevancy mask on an extracted object's point cloud from the overall scene. This mask then guides the task based grasping by focusing grasp generation on an object's region of interest. The usage of LERF also allows for text based queries to be sent as a high level command to the robot giving an accesible natural language interface.  ", "summary_of_recommendation": "This is nicely presented paper with clear details and motivation. However I feel it to be a minor extension to the key ideas presented in LERF as the major change seems to be in the spatial grouping within the scene. The idea is definitely worth exploring and the results seem to be encouraging so its not bad in that aspect but perhaps some additional contributions, perhaps along real world robotics aspects could help readers appreciate it even more.  ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- What would happen if the object scene changes? Can we adapt the model to deal with a simple task, for example clearing out a set of objects on table? Basically how can the model deal with such fixed (and perhaps known) changes from the recorded scene\n\n- An ablation / possible observations on (1) effect of LeRF training steps could be shown to tell its effects ; (2) on the proportion of two scores used in calculating the s score for grasp sampling.\n\n- Here a hemisheprical region is used to get scene images. What would be the impact of reducing the area to half of what is used? This would be interesting to see on a robots where the image capture area might be limited (e.g. Fetch robot where the camera is attached to the robot)", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper extends the ideas presented in LERF and tackles some of issues in it, especially with respect to part based grasping. The paper is of good technical quality with clear motivation and rationale for certain design decisions. I also found the presentation of the paper quite convincing with sufficient clairty in figures and writing. My only concern is the authors could have added further studies on real world robotics like static scene assumption, restrictions of camera capture etc. which can help in a compelling contribution.\n\n**Strengths:**\n\n- Utilizing LeRF for grasp selection is a good decision due to their strong zero shot performance and usefulness for open ended queris via pretraind model embeddings. \n\n- Using DINO features for constructing Object masks overcomes the LeRF limitation of missing objectness\n\n- Using 3D object masks for grasp selection (ranking) helps in grasping appropriate object part given in the query\n\n\n**Weakness:**\n- To construct a LeRF, multiview images of the object scene are needed. The authors have shown experiments on an arm robot with camera mounted on the on top of the gripper. Dealing with robots such as Fetch whose camera is fixed to the torso will be challenging task.\n\n- Train LeRF needs some time. Here it is stated that it takes around 30 and 70 seconds for image capturing and training respectively. A faster way to train LeRF can to be checked for (near) real-time sensitive applications (Note: I acknowledge that its mentioned in the limitations but some experiment on this can help the paper?)\n\n- It is assumed that the object query part will be given in the query. Giving the appropriate object part to grasp every time needs to be addressed.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "ChIE6EQfgP", "reviewer": "Reviewer_bAJo", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper studies task-oriented grasping, in which objects must be grasped by a specific part (to avoid, e.g., damaging the object). The approach used builds on top of Language Embedded Radiance Fields (LERFs). For a given scene, the approach first constructs a LERF of the scene. Then, DINO features are used to generate a 3D mask for the object. Grasps are then sampled by quering LERF within this mask. The 3D mask helps constrain the LERF outputs as it has no concept of objectness. The authors test their approach on a set of 31 physical objects, and find that 81% of generated grasps are on the correct object part.", "summary_of_recommendation": "This paper proposes a simple and elegant approach for task-oriented grasping. The main contribution is on the systems level, combining various techniques (such as LERF, DINO, GraspNet, etc.) to build a robotic system and show its effectiveness. While there are limitations of the approach, I find the results quite interesting and would be of value to the community. Overall I am enthusiastic about this paper.\n", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "While the evaluation of the system is thorough, I am wondering whether it would be possible to have more baseline comparisons with existing methods. As of now, the main baseline comparison to a different method is with ConceptFusion, which does not have physical evaluation.\n\nMinor comments:\n- Figure 2: Swap pictures of screwdriver and cleaning spray on left side to match the right side\n- Table 1: Swap rows for part and object to match ordering in the text\n- Table 1 says 82%, should it be 81%?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The importance of task-oriented grasping is well-motivated and the authors propose a simple and elegant approach\n- The approach requires no dataset collection, instead leveraging the knowledge contained in large pretrained visual language models\n- The authors run extensive experiments on a real-robot with 31 real objects and show convincing qualitative results\n- In addition to quantitative results, the authors also run ablations of the system and present analysis of the failure modes which I find interesting and informative\n\nMost of the main weaknesses I see have all been acknowledged by the authors in the limitations section, namely:\n- Training a LERF for each new scene takes on the order of minutes per grasp, which limits the practicality of the approach until the training time can be cut down significantly\n- Even if LERF training is sped up significantly, having to take 60 new images with the robot arm moving to various angles for each new grasp also takes a long time, and is difficult to speed up unless additional cameras are placed in the scene, which would also limit the practicality of the approach\n- The authors argue that previous methods which collect datasets for specific object don't scale to novel objects. However, the proposed approach still requires a human to annotate the correct part to grasp for each novel object. Perhaps an LLM might be able to propose parts but it would need to be shown that LLMs have sufficient understanding of this type of object manipulation.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel method for task-oriented grasping by reconstructing a Language Embedded Radiance Field.

After the rebuttal, the reviewers achieve consensus on accepting the paper.
Majority concerns of the paper have been addressed.

Based on the scores of the reviewers and the novelty of the paper, I recommend to accept the paper as an oral presentation.
Decision: Accept (Oral)