GraspSplats: Efficient Manipulation with 3D Feature Splatting
Authors: Mazeyu Ji
Ri-Zhao Qiu
Xueyan Zou
Xiaolong Wang
Venue: CoRL 2024 (2024)
Keywords: Zero-shot manipulation
Gaussian Splatting
Keypoint Tracking

Abstract: The ability for robots to perform efficient and zero-shot grasping of object parts is crucial for practical applications and is becoming prevalent with recent advances in Vision-Language Models (VLMs). To bridge the 2D-to-3D gap for representations to support such a capability, existing methods rely on neural fields (NeRFs) via differentiable rendering or point-based projection methods. However, we demonstrate that NeRFs are inappropriate for scene changes due to its implicitness and point-based methods are inaccurate for part localization without rendering-based optimization. To amend these issues, we propose GraspSplats. Using depth supervision and a novel reference feature computation method, GraspSplats can generate high-quality scene representations under 60 seconds. We further validate the advantages of Gaussian-based representation by showing that the explicit and optimized geometry in GraspSplats is sufficient to natively support (1) real-time grasp sampling and (2) dynamic and articulated object manipulation with point trackers.
With extensive experiments on a Franka robot, we demonstrate that GraspSplats significantly outperforms existing methods under diverse task settings. In particular, GraspSplats outperforms NeRF-based methods like F3RM and LERF-TOGO, and 2D detection methods. The code will be released.

GraspSplats: Efficient Manipulation with 3D Feature Splatting
Mazeyu Ji", Ri-Zhao Qiu", Xueyan Zou, Xiaolong Wang
UC San Diego
"Equal contribution
h github.io
The ability for robots to perform efficient and zero-shot grasping of object parts crucial for practical applications and is becoming prevalent with recent advances in Vision-Language Models (VLMs). To bridge the 2D-to-3D gap for representations support such capability, existing methods rely neu- ral fields (NeRFs) via differentiable rendering point-based projection methods. However, we demonstrate that NeRFs are imappropriate for scene changes due to their implicitness and point-based methods are inaccurate for part localiza- tion without rendering-basee optimization. To amend these issues, we propose GraspSplats. Using depth supervision and a novel reference feature computa- tion method, GraspSplats generates high-quality scene representations in under 60 seconds. We further validate the advantages of Gaussian-based representation by showing that the explicit and optimized geometry in GraspSplats is sufficient to natively support (1) real-time grasp sampling and (2) dynamic and articu- lated object manipulation with point trackers. With extensive experiments on a Franka robot, we demonstrate that GraspSplats significantly outperforms exist- ing methods under diverse task settings. In particular, GraspSplats outperforms NeRF-based methods like F3RM and LERF-TOGO, and 2D detection methods.
Abstract:
Keywords: Zero-shot manipulation, Gaussian Splatting, Keypoint Tracking
Introduction
Efficient zero-shot manipulation with part-level understanding is crucial for downstream robotics ap- plications. Consider a kitchen robot deployed to new home: given recipe with language instruc- tions, the robot pulls a drawer by its handle, grasps a tool by its grips, and then pushes the drawer back. To perform these tasks, the robot must dynamically understand part-level grasp affordances to interact effectively with objects. Recent work, such as [1, 2. 31, explores this understanding by embedding reference features from large-scale pre-trained vision models (e-g.. CLIP [4]) into Neu- ral Radiance Fields (NeRFs). However, those methods [2. 1] offer only a static understanding of the scene at the object level and require minutes to train the scene, necessitating costly retraining after any scene changes. This limitation significantly hinders practical applications involving ob- ject displacements, requiring part-level understanding. On the other hand, point-based methods such as [5], which perform back-projection of 2D features, are efficient in feature construction but struggle with visual occlusion and often fail to infer ane-grained spatial relationships without further optimization.
In addition dynamic and part-level scene understanding, achieving fine manipulation requires the robot to have a strong understanding of both the geometry and semantics of the scene. For this capability to emerge from coarse 2D visual features, further optimization is necessary to bridge the 2D-to-3D gap. NeRF-based methods [6, 2, 1] facilitate this understanding through differentiable rendering. However, NeRFs [7, 6, 2, 1] are fundamentally implicit representations, making them difficult to edit to accommodate scene changes, thus leading to static assumption. To address
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Pick up tools by their grips
Track truck and unload
Grusp mugs by handles
Lift teapot by the handle
Grasp fruit from turntable
Clean cluttered tools by grips
Serve struck ball
Open cabinet by handle
Figure I: GraspSplats supports diverse robotics tasks using feature-enhanced 3D Gaussians. Com-
pared to existing NeRF-based methods [1, 2]. GraspSplats transforms the feature representation to reflect object motions in real-time with point tracking from one or more cameras, which makes possible to perform zero-shot dynamic and articulated object manipulation by parts.
dynamic problems, some works [8, 9, 10, 11] commonly predict grasp poses using 3D dense cor- respondences, where reliable grasps are identified based on keypoints in reference state and then applied to various viewpoints or object placements. However, these methods face challenges in tracking object states over time and handling identical objects.
To this end, we propose GraspSplats. Given posed RGBD frames from calibrated camera, Grasp- Splats constructs high-fidelity representation as collection of explicit Gaussian ellipsoids via 3D Gaussian Splatting (3DGS) [12. 13]. GraspSplats reconstructs scene in under 30 seconds and supports efficient part-level grasping for static and rigid transformation, which enables manipula- tion such tracking part objects that is not possible with existing methods. GraspSplats initializes Gaussians from the coarse geometry of depth frames; while computing reference features for each input view real time using MobileSAM [14] and MaskCLIP [15]. These Gaussians are further optimized for geometry, texture, and semantics via differentiable rasterization. The user can sup- ply an object name query (e.g., 'mug') and part query (e.g., 'handle") ') for GraspSplats to efficiently predict part-level affordance and generate grasp proposals. GraspSplats directly generates grasp- ing proposals using explicit Gaussian primitives in milliseconds, for which extend existing antipodal grasp generator [16, 17J. In addition, we further exploit the explicit representation to main- tain high-qualin representations under object displacement. Using point tracker [18], GraspSplats coarsely edits the scene to capture rigid transformations and further optimizes it with partial scene reconstruction.
We implemented GraspSplats desktop-grade computer with real Franka Research (FR3) robot to evaluate its efficacy in tabletop manipulation. Every component in GraspSplats is efficient and empirically runs magnitude (10x) faster than existing work [2, I] - computing 2D reference features, optimizing the 3D representation, and generating 2-finger grasp proposals. This makes it possible to simultaneously generate GraspSplats representation in parallel to arm scans. In ex- periments, GraspSplats outperforms NeRF-based methods like F3RM and LERF-TOGO, and other basnt-based methods.
Our contribution is threefold:
framework that advocates 3DGS for grasping representation. GraspSplats elliciently constructs scenes with geometry, texture, and semanties supervision, which outperforms baselines zero-shot part-based grasping in terms of both accuracy and efficiency.
Techniques towards editable high-fidelity representation, which goes beyond zero-shot ma- nipulation in static scenes into dynamic and articulated object manipulation.
Extensive real-robot experiments that validate GraspSplats as an effective tool for zero-shot grasping in both static and dynamic scenes, which demonstrates the superiority of method over NeRF-based or posnt-based methods.

---

2 Related Work
Language-guided Manipulation. To support zero-shot manipulation, robots must leverage priors learned from internet-scale data. There have been some recent works [19, 20, 21, 22, 23, 24, 5J that use 2D foundation vision models (CLIP [4], SAM [25]. or GroundingDINO [26]) to build open- vocabulary 3D representations. However, these methods mostly rely on simple 2D pack-projection Without further rendering- based optimization, they generally fail to provide precise part-level in- formation. Recently, building on LERF [6], researchers [1, 2, 27, 28] have found that combining feature distillation with neural rendering yields promising representations for robotics manipulation, as it offers both high- Sigh-quality semantics and geometry. Notably, LERF-TOGO [2] proposed condi- tional CLIP queries and DINO regularization for zero-shot manipulation by parts. F3RM [1] learned grasping from few-shot demonstrations. Evo-NeRF [29] focuses on NeRF specialized for stacked transparent objects. However, these methods are based on NeRFs [7], which is fundamentally im- plicit. Though certain NeRF representations can be adapted model dynamic movement, such as grid-based methods [1], dynamic scenes are more natural to be modeled with explicit methods.
Grasp Pose Detection, Grasp pose detection has been a long-standing topic [30, 31, 32, 33, 16, 17, 34, 35, 36] robotics manipulation. Existing methods can be roughly divided into two cat- egories: end-to-end and sampling- streamlined pipelines for grasp poses and incorporating learned semantic priors (e.g., mugs grasped by the handle). However, these methods often require the testing data modalities (e.g.. viewpoint, object category, and transformations) to match training distribution exactly. For instance, LERF- TOGO [2] resolves viewpoint variation of GraspNet [32] by generating hundreds of point clouds for input using different transformations, requiring significant computational time. Sampling-basee methods [16, 17]. on the other hand, do not learn semantic priors but offer reliable and rapid re- sults when explicit representations are available. In this work, we find that the explicit Gaussian primitives are natural to be connected sampling-based methods [16, 17], and features embedded in GraspSplats complements the semantic priors via language guidance. This intuitive combination allows efficient and accurate sampling of grasping poses in dynamic and cluttered environments.
approaches. End-to-end methods [32, 33, 35, 36] offer
Concurrent Work. Some concurrent methods [37, 38, 39, 40, 41, 42, 43] interface Gaussian Splat- ting [12] with 2D features [37, 38, 39] and robotics applications [40, 41, 42, 43]. GraspSplats based on Feature Splatting [38] for its enginceringly optimized implementation. Among methods for robotics applications, GaussianGrasper [40] mostly supports object-level queries for static scenes and displaces objects only when the robot moves them. On the other hand, SplatSim [43] and ManiGaussian [42] require expert demonstrations for policy learning based on GS. where we inves- tigate zero-shot grasping. Most related to our work, SplatMover [41] also studies zero-shot dynamic grasping. Compared to SplatMover, GraspSplats (1) efficiently constructs Ceature-enhancee GS under minute; (2) shows the efficacy of grasp planning directly on explicit Gaussians than Grasp- Net [32]; and (3) adapts point-tracking method to displaces object representations in real-time.
3 Efficient Manipulation with 3D Feature Splatting
Problem Formulation. We assume a robot with parallel gripper. a calibrated in-wrist RGBD camera, and a calibrated third-person view camera. Given scene containing set of objects, the objective is for the robot to grasp and lift an object via language queries (e.g., 'kitchen knife ) Optionally, a part query may be further supplied to specify the part to grasp (e.g., 'handle") for task- oriented manipulation [2]. It is worth noting that, unlike previous works [1, 2], we do not assume that the scene is static. Instead, we aim to design a more generalized algorithm where part-level grasping affordance and sampling can be done continuously even with object movement.
Background.
We render depth D, color è, and features F following existing work [38, 33, 39,37];
i-1
D.F.CC = Edd fa c.) az II - a1)))
iEN
1=1
3

---

MobileSAM
Initial Scene Scan
Hierarchical Features
Feature Splatting
Cropped Object Putches
CLIP
Part Feat. F Fart
Obj-level Masks
Masked Average Pooling
Feat. ubb
Image-level CLIP
Figure 2: GraspSplats employs two techniques to efficiently construct feature-enhanced 3D Gaus- sians: hierarchical feature extraction and dense initialization from geometry regularization, which reduces the overall runtime to 1/10 of existing GS methods [38]. (High-dimenniooaal features are visualized using PCA and the visualized Gaussian ellipsoids are trained without densification).
where d,, fúr and e is the distance to camera, latent feature, and color. a; is per-gaussian opacity, and the indices N ordered by d, We provide complete details in appendix Sec. A.
Overview. To support open-ended grasping, GraspSplats proposes three key components. The overviews given in Fig. 2 and Fig. First, way construct scene representation efficiently with novel reference features and geometric regularization. Second, way to generate grasp pro- posals directly on 3D Gaussians, using 3D conditional language queries and extended antipodal grasp proposer [16, 17]. Finally, a way to edit Gaussians under object displacement which enables dynamic and articulated object manipulation.
3.1 Constructing Feature-enhanced 3D Gaussians
We use differentiable rasterization [12, 38] lift 2D features 3D representation. Though existing works in Teature-enhanced GS offers lert- understanding [38, 391, one commonly overlooked weakness is the expensive overhead before the scene optimization begins. This overhead can be further dissected to (I) costly reference feature computation [2] or (2) densification of sparse Gaus- sians [38] originated from SIM preprocessing [12, 44], which we address in this work.
Efficient Hierarchical Reference Feature Computation. Existing methods [2. 39, 40] spend most compute efforts to regularize coarse CLIP features - either through thousands of multi-scale queries [2] or mask-based regularization [38, 39, 40] through costly grid sampling [25].
We propose way efficiently regularize CLIP using MobileSAMV2 [14]. We generate hierar- chical features, object-level and part-level, specialized for grasping. Given input image, Mo- bileSAMV2 [14] predicts class-agnostic bounding boxes obj := {( Mi, wi, hi) and set of object masks [M}. For object-level feature, we first use MaskCLIP [15] to compute coarse CLIP features of the entire image Fc E HR' xC We then follow Qiu et al. [38] and use Masked Average Pooling to regularize object-level CLIP features with (M), which we detail Sec. F.
For part-level features, MaskCLIP [15]. Since Doty incorporates object priors learned from the SA-IB dataset [25], N is significantly smaller than the number of patches needed from uniform queries [6] for efficient inference. We then interpolate the features to remap them into the original image shape and average over multiple instances to form F part for part-level supervision.
extract image patches from Dobj for batched inference
During differentiable rasterization, introduce shallow MLP with two output branches that takes in the rendered features F from Eq. intermediate features. The first branch renders the object- level feature FD and the second branch renders the part-level feature F Fotg. part MLP(F),

---

Initial Gaussians
Deformed Gaussians
Semantic-aware Sampling
Planning & Execution
Real-time Object Tracking
Grusp "yellow mug" by "handle".
Object-lovel Query
V
Part-level
-0
Query
Figure 3: Given an initial state of Gaussians and RGB-D observations from one or more cameras, GraspSplats tracks the 3D motion of objects specified via language, which used to deform the Gaussian representations in real-time. Given object-part text pairs, GraspSplats proposes grasping poses using both semantics and geometry of Gaussian primitives milliseconds.
where F obj and F part are supervised using F obj and F part with cosine loss. We scale the part-level term in the joint loss Loby + X Lprrt with À = 2.0 to emphasize part-level segmentation.
Geometry Regularization via Depth, Existing feature-enhanced GS methods [38, 39, 37] have no supervision for geometry. In GraspSplats, we project points from depth images centers of the initial Gaussians. In addition, we use depth supervision during training. Empirically, this addi- tional geometric regularization significantly reduces the training time and better surface geometry.
3.2 Static Scene: Part-level Object Localization and Grasp Sampling
To support efficient zero-shot part-level grasping, GraspSplats performs object-level query, con- ditional part-level query, and grasp sampling. Unlike NeRF-based approaches [2], which requires costly rendering to extract sungugee-alignet features and geometry from implicit MLPs, GraspSplats operates directly Gaussian primitives for efficient localization and grasping queries.
Open-vocabulary Object Querying. We first perform object-level open-vocabulary query (e.g.. mug), where we take language queries to select objects for grasping, with optional negative queries to filter out other objects. We do so by directly identifying 3D Gaussians whose isotropic CLIP features more closely align with positive queries over negative queries. The feature-text comparison process follows standard CLIP practices [4, I] and is detailed in Sec. E.
Open-vocabulary Conditional Part-level Querying. As discussed by Rashid et al. [2], CLIP ex- hibits bag-of-words-like behavior (e.g., the activation of 'mug handle' tends to contain both mugs and handles). Thus, is necessary to perform conditional queries. While LERF-TOGO [2] requires a two-step frender-voxelization process; GraspSplats natively supports CLIP queries conditioned on Gaussian primitives. In particular, given an object segmented from the previous operation, we simply repeat the procedure with the new part-level query and limit the of Gaussians to the segmented object. A qualitative example of this part-level conditioning is given in Fig. 3.
Grasp Sampling using Gaussian Primitives. We perform grasp sampling directly on the Gaussian primitives for streamlined grasping. To do so, we combine GraspSplats with GPG, a sampling- based grasp proposer 16, 17]. first define workspace Robj as the 3D space expanded from the segmented object part. The expansion radius is the sum of both the longest axes of the scales of the Gaussian primitives and the gripper's collision radius. Then, we sample N points from Royj- Within the neighborhood Rp of these sampled points where Rp refers to the area within specified distance from the selected point, we aggregate Gaussian primitives with rendered normals and compute the reference coordinate system for grasp sampling with the average normal direction
M(p) = Σ a(g)n(g)T
(2)
n
5

---

Latency4
Grasping Successt
Method
Training
Grasping
Static
Dynamic
Tracking Anything [47]
3.1s
41.9%
45%
ConceptGraphs" [24]
~30s
0.7s
51.1%
LERF-TOGO [2]
10min
9.9s
65.1%
F3RM [1]
~3min
1.6s
72.1%
GraspSplats (Ours)
60s
1.3s
81.4%
74.2%
Table I: Comparison NeRF and 2D-based methods on latency, static/dynamic successful rate averaged across variations in motions and objects/parts. Latencies reported for (1) the time for (re)building the representation; and (2) grasp latency given task texts. ": reproduced variants that use GraspNet [32] objects segmented. i: methods require offline batch processing that does not cope with dynamic scenes. We report detailed object-/part-level successes in appendix Sec. C.
where f()) denotes the unit surface normal of the gaussian primitive g. In the reference frame of each sampled point p. we perform local grid search to find candidate grasps, where the finger of the gripper at terminal candidate grasps contacts the geometry of the segmented part. The details are given in Sec. G.
3.3 Dynamic Scene: Real-time Tracking and Optimization
Using representations optimized for semantics and geometry, natural to extend GraspSplats to track object displacements and edit the Gaussian primitives real time. It is worth noting that such operation is challenging for existing NeRe-based methods [2, 1].
Multi-view Object Tracking with keypoints. Assume one or more calibrated cameras without egocentric motions. Given an object language query, we segment its 3D Gaussian primitives and rasterize a 2D mask to the camera. We then discretize the rendered mask into a set of points as input point tracker [18], which continuously tracks the 2D coordinates of given points. We translate these 2D correspondences into 3D using depth. To filter out noisy correspondences, simple DBSCAN [45] clustering algorithm filter out 3D outliers. Finally, for the remaining correspondence points, use the Kabsch [46] algorithm to solve for the SE(3) transformation. which we apply to the segmented 3D Gaussians primitives. For multiple cameras, we append the estimated 3D correspondences from all cameras to the system of equations for the Kabsch algorithm. Note that the displacement can exerted either by the arm or other external forces.
Partial Fine- Fine-Tuning. Edited scenes may have undesirable artifacts for regions unobserved during the initial reconstruction (e.g., surface underneath displaced objects). Optionally, GraspSplats sup- ports partial scene re-training using object masks rendered before and after the displacement, which is much more efficient than a complete reconstruction.
Experiments
In this section, we conduct experiments to validate the efficacy of GraspSplats. Specifically, our experiments aim to address the following research questions:
Main Results Why is GraspSplats preferable than existing NeRF- and point-based methods?
Ablation Sudy- What were the design choices?
Experiment Protocol. We evaluate GraspSplats several static and dynamic task settings. The full implementation details are provided in Appendix Sec. B.
Static zero-shot part-level manipulation. We experiment with 8 different re-arrangements of objects (4 are cluttered, shown in Fig. 1) with 24 objects and 43 trials. The common objects with part queries including kitchenware, tableware, toys, and tools.

---

Method
Segment
Grasp Sampling
Methad
Query Time!
Succ.
Tracking Anything"
2.5 + 0.1s
0.6 0.05s
GraspNet-100 [2]
10.3s
76.7%
ConceptGraph"
0.1 + 0.05s
0.6 + 0.05s
GraspNet-1 [32]
0.6s
65. %
LERE-TOGO
5.1 + 0..3s
4.8 + 0.7s
F3RM [1]
6.9s
F3RM
1.0 0.1s
6.9 - 0.45s
GraspSplats
0.5s
81.4 %
GraspSplats
0.8 0.1s
0.5 + 0.06s
Table 3: Query time and success rate
Table 2: Grasping lateney breakdown. standard
of different grasp sampling methods mea- sured in the static scene. LERF-TOGO [2] uses multi-view inferences that require 100 GraspNet passes.
deviations are reported over 10 runs. ": our own reproductions that use GPG [16] for grasping.
Dynamic zero-shot part-level manipulation. Objects in this setting are similar to static grasp- ing (40 trials on 24 objects). After the initial scan, human operator rearranges the scene. To specifically evaluate tracking performance, we experiment with three types of dynamic motion:
Easy. Objects translated without rotation.
Medium. Objects are rotated 180%, with some occlusions from hand during rotation.
Hard. Objects are translated and rotated simultaneously, with certain occlusions.
4.1 Main Results
We demonstrate the efficacy of GraspSplats over NeRF-based methods [2, 1J. 2D+Depth point-cloud based methods [47, 25] methods, and scene-gras greeggra based method [24]. We compare with two recent NeRF-based methods, LERF-TOGO [2] and F3RM [1]. Since F3RM requires human demonstra- tions, we implement zero-shot variant, F3RM", which uses GraspNet [32] generate grasps based on rendered depth and features, For the projection-based baselines, we adapt TrackAnything [47], which combines SAM [25] and GroundingDINO [26] for segmentation. Per-frame segmented depth is used to build point clouds for imput to GraspNet [32] to generate grasps. For the scene-graphs based method, we use ConceptGraphs [24] for segmentation and GraspNet for grasp generation. The results for latency, and success rate for static and dynamic scenes are shown in Table. We
claim the following advantages for GraspSplats:
GraspSplats builds part-level representations more efficiently than NeRFs. As shown in Ta- ble. 1, GraspSplats is more efficient in both representation building and grasp sampling than LERF- TOGO [2] and F3RM [1]. Besides up to 10x training speed, GraspSplats achieves better grasping success rate with 16.3 points better than LERF-TOGO, and 9.3 points better than F3RM. addi- tion, both baselines fall short under cluttered scenes with object overlapping without regularizing the object boundary in the reference feature.
GraspSplats outperforms 2D foundation models. To demonstrate how optimized representations improve grasping success, we compare GraspSplats against baselines [47, 24] that use off-the-shelf 2D vision foundation models (SAM [25] and GroundingDINO [26]). Due to existing methods' reliance on GroOndingDINO [26] that is not trained on part-level data, they yield reasonable perfor- mance on object-level grasping but fail almost every part-level query.
Our method has supreme capabilities for accurate dynamic scene modeling. GraspSplats ex- ploits the benefit of explicit representations, which allows editing reconstructed representations without compromising the rich geometric and semantic scene features. We demonstrate that our real-time tracking algorithm allows displacing reconstructed objects effectively, which works in dy- namic scenes with a high success rate. This would otherwise be challenging scenario for implicit methods. Additionally, our approach reconstructs scenes with greater detail than 2D-based methods, resulting in higher success rate in dynamic scenarios.
4.2 Ablation Study
Reference Feature. We show advantages of our hierarchical features in Tab. 4. Specifically, since there are no existing scene-scale part-level segmentation datasets, we manually annotate object-
7

---

Track yellow ball and reset it to the serve point.
Reset yellow truck if it stops.
Open the cabinet door by the gray handle, and grasp the pineapple.
Figure 4: Qualtative examples of GraspSplats performing zero-shot task execution in weal-world en- vironments. Given object-part text queries (italicized in descriptions), GraspSplats executes grasp- ing followed by heuristic trajectories. From left to right: illustration of scene change; grasp poses sampled by GraspSplats; execution of grasping. Videos can be found on the website.
level and part-level masks for 4 scenes for evaluation. GraspSplats outperforms LERF [6] on open- vocabulary object-/part-level segmentation. Qualitative results are provided in Fig. 8.
Grasp Sampling. We ablate different grasping methods Table.
GraspNet-n represents the grasps accumulated after running Grasp- Net [32] n times from different viewpoints, test-time augmentation technique used in [2]. GraspSplats demonstrates superior stability and speed in all cases. In other words, when the semantic affordance is satis- factorily provided, then geometrically informed sampling based on explicit Gaussian representation yields good results. We provide further breakdown sampling efficiency Table. 2 and qualitative illustrations in Fig. 9.
Method
loUt
LERF [6]
39.0
GraspSplats
50.7
Table 4: object/part IoU
Qualitative Results. Fig. 4 shows GraspSplats executing various zero-shot tasks. The heuristic policies can be easily composed to involve various object-part text queries and trajectories.
More ablation studies, such as initialization from depth, can be found in appendix Sec. C. We also present failure analysis and qualitative examples of 2D rendered heatmaps in the appendix.
Conclusion and Limitations
In this work, we present GraspSplats, a novel representation for zero-shot task-oriented manipula- tion. GraspSplats efficient building feature-enhanced 3DGS and grasp sampling. We further enhance GraspSplats with point trackers directly edit optimized representation to capture the dy- namies of objects, which would be challenging for implicit NeRF-hased methods.
Limitations. We highlight the most pronounced limitations to facilitate future research and provide detailed failure analysis in the appendix Sec. H. The current object corresponding algorithm based on the Kabsch [46] algorithm assumes that objects undergo rigid transformation. While the Gaussian representation conceptually applicable to more general deformable objects (e.g., dough and clay), this is investigated in the current work. In addition, the tracking sensitive to fast rotation and the resulting visual occlusions and motion blurs, which could be potentially addressed rendcring based optimization problem.

---

Acknowledgement
This work was supported, in part, by NSF CAREER Award IIS-2240014.
References
[1] W. Shen, G. Yang, A. Yu, J. Wong, L. P. Kaelbling, and P. Isola. Distilled feature fields enable few-shot language-guidd guided manipulation. In Conference on Robot Learning. PMLR, 2023.
[2] A. Rashid, Sharma, M. Kim, J. Kerr, L. Chen, A. Kanazawa, and Goldberg. Lan- guage embedded radiance fields for zero-shot task-oriented grasping. In Conference on Robot Learning. PMLR, 2023.
[3] S. Kobayashi, E. Matsumoto, and V. Sitzmann. Decomposing nerf for editing via feature field distillation. NeurIPS, 2022.
[4] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In ICML. PMLR, 2021.
[5] W. Huang, C. Wang, R. Zhang, Y. Li, Wu, and Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. In Conference on Robot Learning- PMLR, 2023.
[6] Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik. Lerf: Language embedded radiance fields. ICCV, 2023.
[7] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[8] R. Florence, L. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. arXiv preprint arXiv:1806.08756 2018.
[9] L. Manuelli, W. Gao, P. Florence, and R. Tedrake, kpam: Keypoint affordances for category- level robotic manipulation. In The International Symposium of Robotics Research, pages 132- 157. Springer, 2019.
[10] A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitz- mann. Neural descriptor fields: Se (3)-equivariant object representations for manipulation. In 2022 International Conference on Robotics and Automation (ICRA), pages 6394-6400. IEEE, 2022.
[11] L. Yen-Chen P. Florence, J. T. Barron, T.-Y. Lin, A. Rodriguez, and P. Isola. Nerf-supervision Learning dense object descriptors from neural radiance fields. In 2022 international conference on robotics and automation (ICRA), pages 6496-6503. IEEE, 2022.
[12] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (ToG), 2023.
[13] G. Chen and W. Wang. A survey on 3d gaussian splatting. arXiv preprint arXiv:2401.03890, 2024.
[14] C. Zhang, D. Han, S. Zheng, J. Choi, T.-H. Kim, and C. S. Hong. Mobilesamy2: Faster segment anything to everything. arXiv preprint 2rXiv:2312.09579, 2023.
[15] C. Zhou, C. C. Loy, and Dai. Extract free dense labels from clip. In ECCV, 2022.
[16] A. Ten Pas, M. Gualtieri, K. Saenko, and R. Platt. Grasp pose detection in point clouds. The International Journal of Robotics Research, 2017.

---

[17] M. Gualtieri, A. Ten Pas, K. Saenko, and R. Platt. High precision grasp pose detection in dense clutter. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2016.
[18] N. Karaev, I. Rocco, B. Graham, N. Neverova, A. Vedaldi, and C. Rupprecht. Cotracker: It is better to track together. arXiv preprint arXin:2307.07635, 2023.
[19] Liu, Y. Orru, C. Paxton, N. M. M. Shafiullah, and L. Pinto. Ok-robot: What really matters in integrating open-knowledge models for robotics. arXiv preprint arXiv:2401.12202 2024.
[20] B. Chen, F. Xia, B. Ichter, K. Rao, K. Gopalakrishnan, M. S. Ryoo, A. Stone, and D. Kappler. Open-vocabulary queryable scene representations for real world planning. In ICRA, 2023.
[21] C. Huang, O. Mees, A. Zeng, and W. Burgard. Visual language maps for robot navigation. In ICRA, 2023.
[22] K. M. Jatavallabhula, A. Kuwajerwala, Q Gu, M. Omama, T. Chen, A. Maalouf, S. Li, G. Iyer, S. Saryazdi, N. Keetha, et al. Conceptfusion: Open-set multimodal 3d mapping. arXiv preprint arXiv:2302.07241, 2023.
[23] N. H. Yokoyama, Ha, D. Batra, Wang, and B. Bucher. Vifm: Vision-language frontier maps for zero-shot semantic navigation. In ICRA, 2024.
[24] Q Gu, A. Kuwajerwala, S. Morin, K. M. Jatavallabhula, B. Sen, A. Agarwal, C. Rivera, W. Paul, K. Ellis, R. Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. arXiv preprint arXiv:2309.16650 2023.
[25] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, al. Segment anything. In ICCV, 2023.
[26] S. Liu, 7. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li Yang, H. Su, J. Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 2023.
[27] R.-Z. Qiu, Y. Hu, G. Yang, Y. Song, Y. Fu, J. Ye, J. Mu, R. Yang, N. Atanasov, S. Scherer, et al. Learning generalizable feature fields for mobile manipulation. arXiv preprint arXiv:2403.07563 2024.
[28] Y. Wang, Z. Li, M. Zhang, K. Driggs-Campbell, J. Wu, L. Fei-Fei, and Y. Li. D fields: Dynamic 3d descriptor fields for zero-shot generalizable robotic manipulation. arXiv preprint arXiv:2309.166 16118, 2023.
[29] J. Kerr, L. Fu, H. Huang, Y. Avigal, M. Tancik, J. Ichnowski, A. Kanazawa, and K. Goldberg. Evo-nerf: Evolving nerf for sequential robot grasping of transparent objects. In Proceedings of The 6th Conference on Robot Learning, 2023.
[30] J. Redmon and A. Angelova. Real-time grasp detection using convolutional neural networks. In ICRA, 2015.
[31] Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. Aparicio, and K. Goldberg. Dex- 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. Robotics:
Science and Systems, 2017.
[32] H.-S. Fang, C. Wang, M. Gou, and C. Lu. Graspnet- Ibillio:: A large-scale benchmark for general object grasping. In CVPR, 2020.
[33] H.-S. Fang, C. Wang, H. Fang. M. Gou, J. Liu, H. Yan. W. Liu, Xie, and C. Lu. Anygrasp: Robust and efficient grasp perception in spatial and temporal domains. IEEE Transactions Robotics, 2023.
10

---

[34] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact-graspnec Efficient 6-dof grasp generation in cluttered scenes. In ICRA, 2021.
[35] A. Mousavian, C. Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In ICCV, 2019.
[36] M. Liu, Z. Chen, X. Cheng, Y.Ji, R.-Z Qiu, R. Yang, and X. Wang. Visual whole-body control for legged loco-manipulation. arXiv preprint arXiv:2003.16667 2024.
[37] S. Zhou, H. Chang, S. Jiang, Z. Fan, Z. Zhu, D. Xu, P. Chari, S. You, Z. Wang, and Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting enable distilled feature fields. In CVPR, 2024.
[38] R. Qiu, G. Yang, W. Zeng. and X. Wang- Feature splatting: Language-driven physics-based scene synthesis and editing. arXiv preprini arXiv:2404.01 0122444..12.,.... 2024.
[39] M. Qin, W. Li, J. Zhou, H. Wang, and H. Pfister. Langsplat: 3d language gaussian splatting. In CVPR, 2024.
[40] Y. Zheng, X. Chen, Y. Zheng, S. Gu, R. Yang, B. Jin, P. Li, C. Zhong, Z. Wang. L. Liu, et al. Gaussiangrasper: 3d language gaussian splatting for upen-vocabulary robotic grasping. arXiv preprint arXiv:2403.094: 09637, 2024.
[41] O. Shorinwa, J. Tucker, A. Smith, A. Swann, T. Chen, R. Firoozi, M. D. Kennedy, and M. Schwager. Splat-mover: multi-stage, open-vocabulary robotic manipulation via editable gaussian splatting. CoRL, 2024.
[42] G. Lu, S. Zhang, Z. Wang, C. Liu, J. Lu, and Y. Tang. Manigaussian: Dynamic gaussian splatting for multi-task robotic manipulation. In ECCV, 2024.
[43] M. N. Qureshi, S. Garg, F. Yandun, D. Held, G. Kantor, and A. Silwal. Splatsim: Zero- shot sim2real transfer of rgb manipulation policies using gaussian splatting, arXiv preprint a409 10161, 2024.
[44] Y. Fu, S. Liu, A. Kulkamni, J. Kautz, A. A. Efros, and X. Wang. Colmap-free 3d gaussian splatting. arXiv preprint arXiv:2312.07504, 2023.
[45] M. Ester, H.PP Kriegel, J. Sander, X. Xu, et al. A density- based algorithm for discovering clusters in large spatial databases with noise. In kdd, 1996.
[46] W. Kabsch. A solution for the best rotation relate two sets of vectors. Acta Crystallographica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography, 1976.
[47] H. K. Cheng, S. W. Oh, B. Price, A. Schwing, and J.-Y. Lee. Tracking anything with decoupled video segmentation. In ICCV, 2023.
[48] G. Wu, T. Yi, Fang, L. Xie, X. Zhang, W. Wei, W. Liu, Q. Tian, and X. Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528, 2023.
[49] J. L. Schonberger and J.-M. Frahm. cinetture-foo-mmotie revisited. In CVPR, 2016.
[50] A. Singh, J. Sha, K. S. Narayan, T. Achim, and P. Abbecl. Bigbird: A large-scale 3d database of object instances. In ICRA, 2014.
[51] C. Chi, 7. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. arXiv preprint arXiv:2402. 10329, 2024.
[52] T. Ren, Liu, A. Zeng, Lin, K. Li, Н. Cao, Chen, X. Huang, Y. Chen, F. Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint axXiv:2401.14159, 2024.
11

---

Method
Process Timel
Train Iteration + 10,000
Method
Object-level
Part-level
Colmap-S [12]
11.6
LERF-TOGO [2]
81.5%
63.0%
Colmap-D [48]
GraspSplats 623.0
3,000
F3RM [1]
85.2%
77.8%
0.7
3,000
GraspSplats
96.3%
85.2%
Table 5: Comparison of different initialization
Table Analysis of object-level and part- level grasping success rate under static scenes. Note that this table is computed us- ing 27 object-part pairs different from the runs presented in Table. 1.
schemes. Processing time is averaged across scenes. Train iteration reports the updates needed for convergence in increments 1,000.
A Neural Rendering Background
Point and surface splatting methods represent scene explicitly via a mixture of 2D or 3D Gaussian ellipsoid. In the case of Gaussian Splatting, the geometry is represented as collection of 3D Gaussian, each being the tuple {X, }} where E R is the centroid of the Gaussian and Σ is its covariance matrix in the world frame. This gives the probability density function
G(X, Σ) exp x --
(3)
Gaussian splatting decomposes into scaling matrix S and rotation matrix R via RSS R The color information in the texture is encoded with spherical harmonics map (H.) which is conditioned on the viewing direction Φ
To optimize for features, existing methods tend to append an additional vector fo E IRd to each Gaussian, which rendered in view-independent manner because the semantics of an object shall remain the same regardless view directions. The rasterization procedure starts with culling the mixture by removing points that lay outside the camera frustum. The remaining Gaussians projected to the image plane according to the projection matrix W of the camera, which is then sorted from low to high using the distance from the virtual camera origin. This projection also induces the following transformation on the covariance matrix E:
Σ J W WEWWJ
where the Jacobian the projection matrix W. We can then render both the color and the visual features with the splatting algorithm:
{F.
C = E(.
[I (1-a)).
(5)
VEN
f=1
where a; is the opacity of the Gaussian conditioned E' and the indices i N are in the ascending order determined by their distance to the camera origin.
Following the convention [38], GraspSplats assumes that per-gaussian feature vector f, is isotropic. The rendered depth, images, and features are then supervised using L2 loss. Note that all recent works [38, 37, 39] follow a similar paradigm as Eq. 1.
B Implementation Details
We calibrate cameras using Colmap [49] with Aruco visual markers. For all settings, the in-wrist camera obtains an initial scan of the scene by going in a tabletop Bezier curve. After which object- part text queries are provided. Though GraspSplats conceptually supports multi multi-camera tracking, we only use single RGB-D camera in experiments. Manipulation success is defined as lifting the object (optionally, by the parts specified) for least 3 seconds without re-attempting.
C More Ablation Studies
Initialization from Depth. In Table. 5, we ablate the effeet of geometric regularization by compar- ing several variants to initialize geometry for Gaussian. In particular, while the original Gaussian
12

---

Splatting [12] paper adopts sparse Colmap [49] initialization from RGB images (Colmap-S) (0), some recent works have found that dense Colmap reconstruction (Colmap-D) serves as better initializa- tion that results faster convergence [48]. GraspSplats adopt geometric regularization and directly initializes centers of Gaussians using depth inputs, which leads to much more efficient training.
Success Rate Breakdown. As shown in Table. 6, more clearly demonstrate the comparison of part-level segmentation capabilities, we tested the success rates of 27 object-part pairs in unclustered scenes, which differs from Table. 1. It clearly shows that our approach has nearly a 100% success rate on object-level grasping while maintaining very high success rate on part-level grasping. This a benefit from both the feature-enhanced 3D Gaussians and part-level supervision from SAM.
D Part-level Features Determination
Given object-level bounding boxes, crop and wrap images square patches obtain CLIP features. After per-patch features computed, create a part-level feature map by aggregating per-patch features using the inverse function of the cropping and wrapping process. For pixels that are assigned with more than two feature vectors, the features are averaged. Pixels with no assigned features are ignored during rendering baase feature distillation.
E Object-level CLIP Queries
Specifically, GraspSplats follows standard CLIP querying practices [1, 6, 2, 38] and takes a posi- tive vocabulary with negative vocabularies. By default, the negative vocabularies include canonical words (i.e., 'objects' and 'things') "), but can be optionally extended with user queries. To be specific, given a language set (Lg .i...., Ln I+} containing + 1 words, where L+ is the positive query and the remaining L- are negative queries. The CLIP model is used to encode each word L3 into a 768-dimensional feature vector F Fiext, E 767,
F F CLIICeeeeooee encode i= 0.
For each Gaussian primitive j. there a 16-dimensional view-independent feature vector F Fimeett IR16 This is decoded into 768-dimensional CLIP feature representation FCLP. R676,
FCLIP,5 Decoder(F Fatent..)
Then, we calculate the cosine similarity between its CLIP feature representation FELIP,j and each query L; in the L:
F FeLI,, FExt. [F Fissstt
sim FrexT,) =
Then, we apply softmax function to the similarities between it and all queries Lę to enhance the similarity for the positive query:
S, softmax((sim(FFunn) Fres.)
Fiex,,))
Here, S, the similarity vector for Gaussian primitive j. After applying a temperature softmax, the similarity for the positive query L is selected:
simpositie.j Sy
The selecting Gaussians whose similarity L passes certain threshold 0.6 will be regarded the grasping object. After Gaussians are selected, we apply the DBSCAN [45] clustering algo- rithm to filter out outliers.
13

---

F Reference Feature Computation
Following [14], Dobj achieves high recall by intentionally including more false positives, which ensures recall of objects. MobileSAMV2 [14] then uses Dayj as priors to generate object-level segmentation masks {M}.
To generate object-level features, For a given object mask M, we use Masked Average Pooling (MAP) to aggregate an object-level feature vector
W1 = Fc)(M. ΣiEre M(i)
Fc(0)
[F
(6)
Diere M(i)
where is a pixel coordinate. We then construct Foby by assigning w.
To generate part-level features, for given bounding box, crop and wrap the patches (224, 224). The image patch is then processed by MaskCLIP generate feature map of shape (28, 28, 768). We then interpolate the generated feature map to match the size of the original bounding box, and paste it onto the part-level feature map. If a pixel assigned more than one feature, we average all assigned features.
G Grasp Sampling Algorithm
We define F(p) [us (p)) (p)] as the orthogonal reference frame at point p where vi u2)), (p)),. correspond to the normal direction, the secondary direction, and the minimum direction of (p). We search 2D grid G Y x Φ. For each (y, Φ) E G, we apply translations and rotations relative to F(p), then push the gripper along the negative x-axis until finger the base of the gripper contacts the point cloud. Let Tzy.o be the homogeneous transformation describing translations in the x, y plane and rotation about the z-axis. The gripper hy.o at grid cell á contacts the point cloud at: ) = F(p)Tz", .v.0. where x² is the minimum distance along the negative x-axis at which the gripper contacts the point cloud, the number of segmented objects Now, within the gripper's 's closed region exceeds set threshold NiA, i... Norj > Ni,, the gripper hy.o added to the candidate grasp set H. Finally, we geometry-awee scoring model [17,50 50] to rank the grasps and select the grasp pose with the highest score.
H Limitations and Failure Cases
We present comprehensive analysis of limitation, or failure modes, of the current implementation of GraspSplats. We hope they will help inspire future research.
General Deformation. The current scene deformation algorithm based on the Kabsch algorithm assumes that objects undergo rigid transformation. While the Gaussian repre- sentation is conceptually applicable to more general deformable objects (e.g., dough and clay), this investigated in the current work.
Tracking Failure. In addition, the tracking is sensitive fast rotation and the resulting visual occlusions and motion blurs, which could be potentially addressed optimiza- tion problem using semantic and geometric priors fused in GraspSplats without assuming consistent object views. We illustrate failure example in Fig. 6, where GraspSplats is tasked to track yellow toy duck with texture-less body.
Manipulation Policy. GraspSplats focuses on the grasping of the object parts via language guidance. However, in reality, it can be interesting to incorporate a more complex manip- ulation policy. For example, though GraspSplats supports the manipulation of articulated object, such as cabinets, by heuristic policy to pull with respect the normal of the sur- face, it currently has the quasi-static assumption. Thus, it does not support more complex manipulation. Future work may explore if it is possible learn more complex manipula- tion policies on top of the powerful representations of GraspSplats. In addition, though the
14

---

Figure 5: Grasp execution enror: The leftmost image is the captured image, the middle shows the segmented grip the pliers, and the rightmost image displays the generated grasp. The red grasp has the highest score.
Figure 6: Tracking failure: The leftmost image shows the initially sampled feature points, with blue indicating valid feature points and green indicating lost tracking points. The two images the right display the feature points' status as the object moves, where valid feature points are continuously lost.
semantic design provides good boundary between foreground and background, it does not differentiate between objects of the same types (illustrated in Fig. 5).
Requirement for Scene Scanning. GraspSplats requires scanning of initial objects on the tabletop, where the scanning poses are generated from preprogrammed poses. Future work may include active reconstruction policy to automate such process.
I Hardware Configuration
We use the Franka Research robot (FR3) as the main experiment platform. In addition to one Intel Realsense D435 cameras mounted on the end effector, we also use inputs from two third-person view Intel Realsense D435 cameras facilitate scene reconstruction. We use the UMI gripper [51] as the end effector. The arm and cameras are connected to desktop computer with single RTX 4090 and an Intel 13-00kk CPU, on which GraspSplats is deployed.
J Multi-state Task Policy Rollout
The examples of complex tasks are designed illustrate potential use cases rather than showcase highly intricate operations. The tasks primarily focus on object manipulation, such updating an object's position, grasping, resetting, and pick-and-place actions. For instance, in the toy experiment in the supplementary video, after grasping the car, the robot resets the car by placing it back in its original position with the same orientation. During pick-and-place tasks, the placement orientation aligned with the grasping orientation, and the placement location is determined by querying the center another object.
These operations highlight the system's capabilities in basic object manipulation. However, chal- lenges arise in tasks that involve precise placement, especially when the object's orientation shifts
15

---

H
(b) Regularized Depth
(c) Regularized Normal
a) RGB Image
(d) Unregularized Depth (e) Unregularized Normal
Figure 7: Qualitative results of depth supervision. Rendered RGB image, which is not noticeably different with or without depth supervision. (b) Rendered depth image with depth supervision. Notice the red circle surface is accurately rendered as flat. (c) Rendered normal image with depth supervision. Notice that the object edges are smoother. (d) Unregularized depth, which contains artifacts on flat patterns. (e) Unregularized normal, which noisy near the object edge.
during grasping due to the inherent instability or rotation caused by the gripper's contact. This issue is particularly evident when the gripper obstructs the view of the object, complicating the perception of the object's orientation during placement. These challenges and potential failure cases are further discussed in the supplementary material.
K Effect of Depth Supervision
We have provided quantitative evidence in the paper to validate the effectiveness of the depth ini- tialization in training. For the other aspect of geometric regularization, depth supervision during the training, we provide qualitative samples in Fig. 7.
L Text Query Comparison
The 2D response maps for the same text query generated by different algorithms are displayed in Fig. 8. Grounded SAM[52] in TrackAnything(47] demonstrates effective segmentation at the object level; however, it fails to distinguish parts of objects and sometimes does not respond at all. Additionally, it tends to merge multiple closely positioned objects into one. LERF[6] similarly exhibits weak responses at the part level, with very unclear segmentation between objects, making it nearly impossible to separate similar objects that are close together. F3RM[L] provides higher- quality features compared to LERF and can respond some part-level queries, but it also struggles with accurately distinguishing between similar, closely positioned objects. In contrast, our algorithm exhibits near 2D segmentation capabilities and highly sensitive part-level queries, allowing for precise differentiation of different parts within distinct objects.
M GraspNet vs Grasp Sampling
In our comparative analysis grasp sampling methods, we employed GraspNet[32) with collision detection LERFTOGO[2], sampling viewpoints on a hemisphere defined by theta and phi pa- rameters. Specifically, we tested configurations with 3x3, 5x5, and 10x10 viewpoints, effectively running GraspNet 9, 25, and 100 times respectively. The resulting grasps for these configurations are illustrated in the first three images is shown in Fig. 9, Notably, the red circles highlight regions
16

---

(a) Cup
(b) Cup (Rim)
(c) Headphone
(d) Headphone
(band)
(e) Plier
(f) Plier
(Handle)
TrackAnything LERF
F3RM
GraspSplats
Figure 8: From left to right are the responses of TrackAnything (Grounded SAM), LERF, F3RM, and our algorithm to the same text query.
Method
Query Time!
GraspNet-100 100
10.3
GraspNet-25
3.2
GraspNet-9
1.2
Ours
0.5
Table 7: Time efficiency comparison of different grasp sampling methods including ours and GraspNet with different number of executions.
where valid grasps were not generated. In contrast, for our sampling-hase based method, we configured the system to sample 3000 points within the workspace, utilizing 16 threads of 12900K for parallel processing. The outcome of this approach is depicted in the last image. We also compare the time efficiency of the algorithms as is shown in Tab.
It is worth noting that while the GS grasps directly generated with Gaussians, the point cloud data (ped) is used solely for visualization purposes. This comparison underscores the efficiency and potential limitations of each method in terms of grasp generation and computational demands.
17

---

(a) GraspNet- 9
(a) GraspNet 25
(a) GraspNet 100
(d) Ours
Figure 9: The first three images from left right show the grasps obtained by running GraspNet with 9, 25, and 100 different viewpoints, respectively. The last image shows the results obtained by our method, which sampled 3000 points in the workspace.
18


Reviews:
[{"review_id": "CZ4d4wdNls", "reviewer": "Reviewer_s84c", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes GraspSplats, a system for utilizing Gaussian Splatting (GS) for grasping. They modify GS for grasping, adding joint object-level/part-level reasoning, integration with an external tracker to handle object motions, and add a sampling-based grasp method using the resulting representation. They compare to existing language-guided zero-shot grasping methods on a real system and show favorable results.", "summary_of_recommendation": "The proposed work seems well motivated and shows strong performance, but is missing some important details in the experiments and should compare to existing grasping methods with GS. Additionally, their tracking relies on language guidance before hand, which seems to severely limit the applicability of the proposed tracking method. More baselines and clarity in experimentations and addressing this limitation would strengthen the work.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "How many grasps are attempted for the results shown in Tables 1 and 3? How many comparisons are made in Table 2? What is the breakdown of performance of each method between object-level and part-level grasping?\n\nWhat are the variance in grasp timing for Table 1? The timing numbers seem like estimates, but if inference time is an important part of the value of the proposed method, there should be more clarity. What is the breakdown of timing w.r.t. subcomponents like finetuning the representation or evaluating grasp proposal?\n\nIs the partial fine-tuning presented in Sec. 3.3 used in experiments?\n\nThe explanation of how part-level features are determined is unclear - how do you perform interpolation to go from labels at parts to labels across the scene?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NpfjHncQ0t", "reviewer": "Reviewer_jdjs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose GraspSplats, a zero-shot object part manipulation algorithm based on 3D Gaussian Splatting. The proposed algorithm takes RGB-D frames from a calibrated camera and reconstructs the scene using 3D Gaussian ellipsoids. The 3D Gaussians are initialized from the depth frames, and GraspSplats uses MobileSAMv2 and MaskCLIP to compute object and part-level features. To handle changes in the scenes, GraspSplat uses multi-view object tracking. Then, zero-shot language-guided manipulation is performed by supplying the object and part name that needs to be manipulated to generate CLIP queries to identify the 3D Gaussians that better aligns. Then, a sampling-based grasp proposer is leveraged to compute candidate grasps around the . GraspSplats demonstrated to be successful in real robot experiments, and the paper presents a comparison of its performance against Tracking Anything, LERF-TOGO, and a zero-shot variant of F3RM in both static and dynamic scenes.", "summary_of_recommendation": "I am in between weak accept and weak reject, but I am currently leaning towards weak reject. I believe that the robotics community could benefit from this paper's findings, especially for object-part manipulation. However, I am concerned that the experimental part is not strong enough as detailed in my review. |||| Update after rebuttal: After reading other reviewers' comments and the rebuttal, I will raise my rate to Weak Accept. Authors have clarify and addressed most of my concerns and I am convinced by the answer given to the other reviewers.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- Improve the clarity of Figures 2 and 4.\n- Improve the experimental part including explicit (voxel and/or point-cloud) representations.\n- Add the missing information in the experimental section and improve the Table 2 results by adding a more significant experiment/dataset.\n- Add grasping qualitative comparison to baselines.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kldgSYvH9s", "reviewer": "Reviewer_vXka", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces GraspSplats, an approach for object and part-level language-guided grasping that leverages 3D Gaussian Feature Splatting as the underlying scene representation. By using 2D point trackers, the Gaussians can be dynamically updated as objects are moved around in the scene. This is a significant advantage compared to implicit representations including NeRFs. The authors additionally show that the normals of the Gaussians can be used for real-time antipodal grasping. Extensive results in the real world demonstrate the improved speed of GraspSplats and its ability to handle dynamic scenes of varying levels of difficulty.", "summary_of_recommendation": "While the paper presents several incremental improvements to Feature 3DGS, these improvements are critical for moving towards real-world deployability. The ability to enable dynamic scene updates and real-time antipodal grasping is a significant improvement over NeRF-based representations", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "See issues raised in the weaknesses above.\n\n**Other questions**\n1. One advantage of neural fields is being able to use RGB images and reconstruct transparent and specular objects. Did you experiment with such objects? Does using RGB-D and depth supervision degrade these properties?\n2. For moving the Gaussians as objects move dynamically, could you further optimize them based on the incoming image observations? How do you think that would compare to using the 2D tracker?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tnJ9YKxSsf", "reviewer": "Reviewer_WC1J", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This method proposes Gaussian splatting-based language-embedded feature fields for task-oriented grasping in dynamic scenes. Given the scene, the method first constructs a GaussianSplat that models the scene in terms of color, depth, and dense visual features. The features are then passed to shallow MLPs to predict object and part features. These features are used together with language-based queries to find the corresponding grasp region on the scene. From this region, grasp samples are generated through an antipodal grasp sampler (GPG). These grasp samples are ranked using geometry-aware scoring methods, and the best grasp is selected. Additionally, the paper uses a tracker to update the underlying feature field by rigidly transforming the Gaussians of the GaussianSplat. This makes the model robust to the changes in the scene.", "summary_of_recommendation": "I think paper is well written and demonstrated. However, core idea is using gaussian splatting instead of Nerfs for learning feature fields. It is not a significant novelty.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "My main concerns are:\n\n* As I see there is no quantitative analysis part-level grasping. \n* The paper requires some clarity improvements.\n\nOther than that, the authors can also address other comments on my review.\n\nUpdate: My concerns were addressed.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers are in agreement that this paper addresses an important topic, is relevant to the conference, and provides some novelty in the form of extracting hierarchical features in Gaussian splat representations. The reviewers also generally agree in the strength of the contribution and it providing interesting results to the robot learning community.

However, reviewers also have a variety of concerns which the authors should address in the rebuttal / discussion phase.
Those that jumped out at me as most important are:
- The lack of evidence to support the claim that the approach outperforms purely point-based or voxel-baesd approaches
- The potential fragility of the tracker as an important limitation and discussion of how this could be overcome
- Issues with clarity in the presentation

Following the rebuttal I think the improved experiments to compare to a point-based representation and the additional tracking results address the majority of the reviewers concerns. While some might argue about the level of novelty of the paper, I think it provides an interesting investigation that would be of utility to the robot learning community.
Decision: Accept