GraspSplats: Efficient Manipulation with 3D Feature Splatting

Authors: mazeyu  ji
ri-zhao  qiu
xueyan  zou
xiaolong  wang
u c san
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-05-16 21:58:17

Abstract: The ability for robots to perform efficient and zero-shot grasping of object parts is crucial for practical applications and is becoming prevalent with recent advances in Vision-Language Models (VLMs). To bridge the 2D-to-3D gap for representations to support such a capability, existing methods rely on neural fields (NeRFs) via differentiable rendering or point-based projection methods. However, we demonstrate that NeRFs are inappropriate for scene changes due to its implicitness and point-based methods are inaccurate for part localization without rendering-based optimization. To amend these issues, we propose GraspSplats. Using depth supervision and a novel reference feature computation method, GraspSplats can generate high-quality scene representations under 60 seconds. We further validate the advantages of Gaussian-based representation by showing that the explicit and optimized geometry in GraspSplats is sufficient to natively support (1) real-time grasp sampling and (2) dynamic and articulated object manipulation with point trackers.
With extensive experiments on a Franka robot, we demonstrate that GraspSplats significantly outperforms existing methods under diverse task settings. In particular, GraspSplats outperforms NeRF-based methods like F3RM and LERF-TOGO, and 2D detection methods. The code will be released.

Main paper: 
 introduction: 
 Efficient zero-shot manipulation with part-level understanding is crucial for downstream robotics applications. Consider a kitchen robot deployed to a new home: given a recipe with language instructions, the robot pulls a drawer by its handle, grasps a tool by its grips, and then pushes the drawer back. To perform these tasks, the robot must dynamically understand part-level grasp affordances to interact effectively with objects. Recent work, such as [1, 2, 3] , explores this understanding by embedding reference features from large-scale pre-trained vision models (e.g., CLIP [4] ) into Neural Radiance Fields (NeRFs). However, those methods [2, 1] offer only a static understanding of the scene at the object level and require minutes to train the scene, necessitating costly retraining after any scene changes. This limitation significantly hinders practical applications involving object displacements, or requiring part-level understanding. On the other hand, point-based methods such as [5] , which perform back-projection of 2D features, are efficient in feature construction but struggle with visual occlusion and often fail to infer fine-grained spatial relationships without further optimization.In addition to dynamic and part-level scene understanding, achieving fine manipulation requires the robot to have a strong understanding of both the geometry and semantics of the scene. For this capability to emerge from coarse 2D visual features, further optimization is necessary to bridge the 2D-to-3D gap. NeRF-based methods [6, 2, 1] facilitate this understanding through differentiable rendering. However, NeRFs [7, 6, 2, 1] are fundamentally implicit representations, making them difficult to edit to accommodate scene changes, thus leading to a static assumption. To address dynamic problems, some works [8, 9, 10, 11] commonly predict grasp poses using 3D dense correspondences, where reliable grasps are identified based on keypoints in a reference state and then applied to various viewpoints or object placements. However, these methods face challenges in tracking object states over time and handling identical objects.To this end, we propose GraspSplats. Given posed RGBD frames from a calibrated camera, Grasp-Splats constructs a high-fidelity representation as a collection of explicit Gaussian ellipsoids via 3D Gaussian Splatting (3DGS) [12, 13] . GraspSplats reconstructs a scene in under 30 seconds and supports efficient part-level grasping for static and rigid transformation, which enables manipulation such as tracking part objects that is not possible with existing methods. GraspSplats initializes Gaussians from the coarse geometry of depth frames; while computing reference features for each input view in real time using MobileSAM [14] and MaskCLIP [15] . These Gaussians are further optimized for geometry, texture, and semantics via differentiable rasterization. The user can supply an object name query (e.g., 'mug') and part query (e.g., 'handle') for GraspSplats to efficiently predict part-level affordance and generate grasp proposals. GraspSplats directly generates grasping proposals using explicit Gaussian primitives in milliseconds, for which we extend an existing antipodal grasp generator [16, 17] . In addition, we further exploit the explicit representation to maintain high-quality representations under object displacement. Using a point tracker [18] , GraspSplats coarsely edits the scene to capture rigid transformations and further optimizes it with partial scene reconstruction.We implemented GraspSplats on a desktop-grade computer with a real Franka Research (FR3) robot to evaluate its efficacy in tabletop manipulation. Every component in GraspSplats is efficient and empirically runs a magnitude (10×) faster than existing work [2, 1] -computing 2D reference features, optimizing the 3D representation, and generating 2-finger grasp proposals. This makes it possible to simultaneously generate GraspSplats representation in parallel to arm scans. In experiments, GraspSplats outperforms NeRF-based methods like F3RM and LERF-TOGO, and other point-based methods.Our contribution is threefold:• A framework that advocates 3DGS for grasping representation. GraspSplats efficiently reconstructs scenes with geometry, texture, and semantics supervision, which outperforms baselines on zero-shot part-based grasping in terms of both accuracy and efficiency.• Techniques towards an editable high-fidelity representation, which goes beyond zero-shot manipulation in static scenes into dynamic and articulated object manipulation.• Extensive real-robot experiments that validate GraspSplats as an effective tool for zero-shot grasping in both static and dynamic scenes, which demonstrates the superiority of our method over NeRF-based or point-based methods. 
related work: 
 Language-guided Manipulation. To support zero-shot manipulation, robots must leverage priors learned from internet-scale data. There have been some recent works [19, 20, 21, 22, 23, 24, 5] that use 2D foundation vision models (CLIP [4] , SAM [25] , or GroundingDINO [26] ) to build openvocabulary 3D representations. However, these methods mostly rely on simple 2D back-projection. Without further rendering-based optimization, they generally fail to provide precise part-level information. Recently, building on LERF [6] , researchers [1, 2, 27, 28] have found that combining feature distillation with neural rendering yields promising representations for robotics manipulation, as it offers both high-quality semantics and geometry. Notably, LERF-TOGO [2] proposed conditional CLIP queries and DINO regularization for zero-shot manipulation by parts. F3RM [1] learned grasping from few-shot demonstrations. Evo-NeRF [29] focuses on NeRF specialized for stacked transparent objects. However, these methods are based on NeRFs [7] , which is fundamentally implicit. Though certain NeRF representations can be adapted to model dynamic movement, such as grid-based methods [1] , dynamic scenes are more natural to be modeled with explicit methods.Grasp Pose Detection. Grasp pose detection has been a long-standing topic [30, 31, 32, 33, 16, 17, 34, 35, 36] in robotics manipulation. Existing methods can be roughly divided into two categories: end-to-end and sampling-based approaches. End-to-end methods [32, 33, 35, 36] offer streamlined pipelines for grasp poses and incorporating learned semantic priors (e.g., mugs grasped by the handle). However, these methods often require the testing data modalities (e.g., viewpoint, object category, and transformations) to match training distribution exactly. For instance, LERF-TOGO [2] resolves viewpoint variation of GraspNet [32] by generating hundreds of point clouds for input using different transformations, requiring significant computational time. Sampling-based methods [16, 17] , on the other hand, do not learn semantic priors but offer reliable and rapid results when explicit representations are available. In this work, we find that the explicit Gaussian primitives are natural to be connected to sampling-based methods [16, 17] , and features embedded in GraspSplats complements the semantic priors via language guidance. This intuitive combination allows efficient and accurate sampling of grasping poses in dynamic and cluttered environments.Concurrent Work. Some concurrent methods [37, 38, 39, 40, 41, 42, 43] interface Gaussian Splatting [12] with 2D features [37, 38, 39] and robotics applications [40, 41, 42, 43] . GraspSplats is based on Feature Splatting [38] for its engineeringly optimized implementation. Among methods for robotics applications, GaussianGrasper [40] mostly supports object-level queries for static scenes and displaces objects only when the robot arm moves them. On the other hand, SplatSim [43] and ManiGaussian [42] require expert demonstrations for policy learning based on GS, where we investigate zero-shot grasping. Most related to our work, SplatMover [41] also studies zero-shot dynamic grasping. Compared to SplatMover, GraspSplats (1) efficiently constructs feature-enhanced GS under 1 minute; (2) shows the efficacy of grasp planning directly on explicit Gaussians than Grasp-Net [32] ; and (3) adapts point-tracking method to displaces object representations in real-time.3 Efficient Manipulation with 3D Feature Splatting Problem Formulation. We assume a robot with a parallel gripper, a calibrated in-wrist RGBD camera, and a calibrated third-person view camera. Given a scene containing a set of objects, the objective is for the robot to grasp and lift an object via language queries (e.g., 'kitchen knife').Optionally, a part query may be further supplied to specify the part to grasp (e.g., 'handle') for taskoriented manipulation [2] . It is worth noting that, unlike previous works [1, 2] , we do not assume that the scene is static. Instead, we aim to design a more generalized algorithm where part-level grasping affordance and sampling can be done continuously even with object movement.Background. We render depth D, color Ĉ, and features F following existing work [38, 39, 37] : where d i , f i , and c i is the distance to camera, latent feature, and color. α i is per-gaussian opacity, and the indices i ∈ N are ordered by d i . We provide complete details in appendix Sec. A.EQUATIONTo support open-ended grasping, GraspSplats proposes three key components. The overviews are given in Fig. 2 and Fig. 3 . First, a way to construct a scene representation efficiently with novel reference features and geometric regularization. Second, a way to generate grasp proposals directly on 3D Gaussians, using 3D conditional language queries and an extended antipodal grasp proposer [16, 17] . Finally, a way to edit Gaussians under object displacement which enables dynamic and articulated object manipulation. 
constructing feature-enhanced 3d gaussians: 
 We use differentiable rasterization [12, 38] to lift 2D features to 3D representation. Though existing works in feature-enhanced GS offers part-level understanding [38, 39] , one commonly overlooked weakness is the expensive overhead before the scene optimization begins. This overhead can be further dissected to (1) costly reference feature computation [2] or (2) densification of sparse Gaussians [38] originated from SfM preprocessing [12, 44] , which we address in this work.Efficient Hierarchical Reference Feature Computation. Existing methods [2, 39, 40] spend most compute efforts to regularize coarse CLIP features -either through thousands of multi-scale queries [2] or mask-based regularization [38, 39, 40] through costly grid sampling [25] .We propose a way to efficiently regularize CLIP using MobileSAMV2 [14] . We generate hierarchical features, object-level and part-level, specialized for grasping. Given an input image, Mo-bileSAMV2 [14] predicts class-agnostic bounding boxesD obj := {(x i , y i , w i , h i )} Ni=1 and a set of object masks {M}. For object-level feature, we first use MaskCLIP [15] to compute coarse CLIP features of the entire image F C ∈ R H ′ ×W ′ ×C . We then follow Qiu et al. [38] and use Masked Average Pooling to regularize object-level CLIP features with {M}, which we detail in Sec. F.For part-level features, we extract image patches from D obj for batched inference on MaskCLIP [15] . Since D obj incorporates object priors learned from the SA-1B dataset [25] , N is significantly smaller than the number of patches needed from uniform queries [6] for efficient inference. We then interpolate the features to remap them into the original image shape and average over multiple instances to form F part for part-level supervision.During differentiable rasterization, we introduce a shallow MLP with two output branches that takes in the rendered features F from Eq. 1 as intermediate features. The first branch renders the objectlevel feature FD and the second branch renders the part-level feature Fobj , Fpart = MLP( F),Real-time Object TrackingGrasp "yellow mug" by "handle".Part-level Query where Fobj and Fpart are supervised using F obj and F part with cosine loss. We scale the part-level term in the joint loss L obj + λ • L part with λ = 2.0 to emphasize part-level segmentation.Geometry Regularization via Depth. Existing feature-enhanced GS methods [38, 39, 37] have no supervision for geometry. In GraspSplats, we project points from depth images as centers of the initial Gaussians. In addition, we use depth as supervision during training. Empirically, this additional geometric regularization significantly reduces the training time and better surface geometry.To support efficient zero-shot part-level grasping, GraspSplats performs object-level query, conditional part-level query, and grasp sampling. Unlike NeRF-based approaches [2] , which requires costly rendering to extract language-aligned features and geometry from implicit MLPs, GraspSplats operates directly on Gaussian primitives for efficient localization and grasping queries.Open-vocabulary Object Querying. We first perform object-level open-vocabulary query (e.g., mug), where we take language queries to select objects for grasping, with optional negative queries to filter out other objects. We do so by directly identifying 3D Gaussians whose isotropic CLIP features more closely align with positive queries over negative queries. The feature-text comparison process follows standard CLIP practices [4, 1] and is detailed in Sec. E.Open-vocabulary Conditional Part-level Querying. As discussed by Rashid et al. [2] , CLIP exhibits bag-of-words-like behavior (e.g., the activation of 'mug handle' tends to contain both mugs and handles). Thus, it is necessary to perform conditional queries. While LERF-TOGO [2] requires a two-step (render-voxelization) process; GraspSplats natively supports CLIP queries conditioned on Gaussian primitives. In particular, given an object segmented from the previous operation, we simply repeat the procedure with the new part-level query and limit the set of Gaussians to the segmented object. A qualitative example of this part-level conditioning is given in Fig. 3 .Grasp Sampling using Gaussian Primitives. We perform grasp sampling directly on the Gaussian primitives for streamlined grasping. To do so, we combine GraspSplats with GPG, a samplingbased grasp proposer [16, 17] . We first define a workspace R obj as the 3D space expanded from the segmented object part. The expansion radius is the sum of both the longest axes of the scales of the Gaussian primitives and the gripper's collision radius. Then, we sample N points from R obj . Within the neighborhood R p of these sampled points where R p refers to the area within a specified distance from the selected point, we aggregate Gaussian primitives with rendered normals and compute the reference coordinate system for grasp sampling with the average normal direction where n(g) denotes the unit surface normal of the gaussian primitive g. In the reference frame of each sampled point p, we perform a local grid search to find candidate grasps, where the finger of the gripper at terminal candidate grasps contacts the geometry of the segmented part. The details are given in Sec. G.EQUATIONUsing representations optimized for semantics and geometry, it is natural to extend GraspSplats to track object displacements and edit the Gaussian primitives in real time. It is worth noting that such operation is challenging for existing NeRF-based methods [2, 1] .Multi-view Object Tracking with keypoints. Assume one or more calibrated cameras without egocentric motions. Given an object language query, we segment its 3D Gaussian primitives and rasterize a 2D mask to the camera. We then discretize the rendered mask into a set of points as input to a point tracker [18] , which continuously tracks the 2D coordinates of given points. We translate these 2D correspondences into 3D using depth. To filter out noisy correspondences, we use a simple DBSCAN [45] clustering algorithm to filter out 3D outliers. Finally, for the remaining correspondence points, we use the Kabsch [46] algorithm to solve for the SE(3) transformation, which we apply to the segmented 3D Gaussians primitives. For multiple cameras, we append the estimated 3D correspondences from all cameras to the system of equations for the Kabsch algorithm. Note that the displacement can be exerted either by the arm or other external forces.Partial Fine-Tuning. Edited scenes may have undesirable artifacts for regions unobserved during the initial reconstruction (e.g., surface underneath displaced objects). Optionally, GraspSplats supports partial scene re-training using object masks rendered before and after the displacement, which is much more efficient than a complete reconstruction. 
experiments: 
 In this section, we conduct experiments to validate the efficacy of GraspSplats. Specifically, our experiments aim to address the following research questions:• Main Results -Why is GraspSplats preferable than existing NeRF-and point-based methods?• Ablation Study -What were the design choices?Experiment Protocol. We evaluate GraspSplats on several static and dynamic task settings. The full implementation details are provided in Appendix Sec. B.• Static zero-shot part-level manipulation. We experiment with 8 different re-arrangements of objects (4 are cluttered, shown in Fig. 1 ) with 24 objects and 43 trials. The common objects with part queries including kitchenware, tableware, toys, and tools. 10.3s 76.7% GraspNet-1 [32] 0.6s 65.1% F3RM [1] 6.9s -GraspSplats 0.5s 81.4% Table 3 : Query time and success rate of different grasp sampling methods measured in the static scene. LERF-TOGO [2] uses multi-view inferences that require 100 GraspNet passes.• Dynamic zero-shot part-level manipulation. Objects in this setting are similar to static grasping (40 trials on 24 objects). After the initial scan, a human operator rearranges the scene. To specifically evaluate tracking performance, we experiment with three types of dynamic motion:-Easy. Objects are translated without rotation.-Medium. Objects are rotated 180 • , with some occlusions from hand during rotation.-Hard. Objects are translated and rotated simultaneously, with certain occlusions.We demonstrate the efficacy of GraspSplats over NeRF-based methods [2, 1], 2D+Depth point-cloud based methods [47, 25] methods, and scene-graph based method [24] . We compare with two recent NeRF-based methods, LERF-TOGO [2] and F3RM [1] . Since F3RM requires human demonstrations, we implement a zero-shot variant, F3RM * , which uses GraspNet [32] to generate grasps based on rendered depth and features. For the projection-based baselines, we adapt TrackAnything [47] , which combines SAM [25] and GroundingDINO [26] for segmentation. Per-frame segmented depth is used to build point clouds for input to GraspNet [32] to generate grasps. For the scene-graphbased method, we use ConceptGraphs [24] for segmentation and GraspNet for grasp generation.The results for latency, and success rate for static and dynamic scenes are shown in Table . 1. We claim the following advantages for GraspSplats:GraspSplats builds part-level representations more efficiently than NeRFs. As shown in Table. 1, GraspSplats is more efficient in both representation building and grasp sampling than LERF-TOGO [2] and F3RM [1] . Besides up to 10x training speed, GraspSplats achieves better grasping success rate with 16.3 points better than LERF-TOGO, and 9.3 points better than F3RM. In addition, both baselines fall short under cluttered scenes with object overlapping without regularizing the object boundary in the reference feature.GraspSplats outperforms 2D foundation models. To demonstrate how optimized representations improve grasping success, we compare GraspSplats against baselines [47, 24] that use off-the-shelf 2D vision foundation models (SAM [25] and GroundingDINO [26] ). Due to existing methods' reliance on GroundingDINO [26] that is not trained on part-level data, they yield reasonable performance on object-level grasping but fail almost every part-level query.Our method has supreme capabilities for accurate dynamic scene modeling. GraspSplats exploits the benefit of explicit representations, which allows editing reconstructed representations without compromising the rich geometric and semantic scene features. We demonstrate that our real-time tracking algorithm allows displacing reconstructed objects effectively, which works in dynamic scenes with a high success rate. This would otherwise be a challenging scenario for implicit methods. Additionally, our approach reconstructs scenes with greater detail than 2D-based methods, resulting in a higher success rate in dynamic scenarios.Reference Feature. We show advantages of our hierarchical features in Tab. 4. Specifically, since there are no existing scene-scale part-level segmentation datasets, we manually annotate object-Track yellow ball and reset it to the serve point.Reset yellow truck if it stops.Open the cabinet door by the gray handle, and grasp the pineapple. level and part-level masks for 4 scenes for evaluation. GraspSplats outperforms LERF [6] on openvocabulary object-/part-level segmentation. Qualitative results are provided in Fig. 8 .Grasp Sampling. We ablate different grasping methods in Table . 3.IoU↑ LERF [6] 39.0 GraspSplats 50.7Table 4 : object/part IoU GraspNet-n represents the grasps accumulated after running Grasp-Net [32] n times from different viewpoints, a test-time augmentation technique used in [2] . GraspSplats demonstrates superior stability and speed in all cases. In other words, when the semantic affordance is satisfactorily provided, then geometrically informed sampling based on explicit Gaussian representation yields good results. We provide further breakdown of sampling efficiency in Table . 2 and qualitative illustrations in Fig. 9 .Qualitative Results. Fig. 4 shows GraspSplats executing various zero-shot tasks. The heuristic policies can be easily composed to involve various object-part text queries and trajectories.More ablation studies, such as initialization from depth, can be found in appendix Sec. C. We also present failure analysis and qualitative examples of 2D rendered heatmaps in the appendix. 
conclusion and limitations: 
 In this work, we present GraspSplats, a novel representation for zero-shot task-oriented manipulation. GraspSplats is efficient in building feature-enhanced 3DGS and grasp sampling. We further enhance GraspSplats with point trackers to directly edit optimized representation to capture the dynamics of objects, which would be challenging for implicit NeRF-based methods.Limitations. We highlight the most pronounced limitations to facilitate future research and provide detailed failure analysis in the appendix Sec. H. The current object corresponding algorithm based on the Kabsch [46] algorithm assumes that objects undergo rigid transformation. While the Gaussian representation is conceptually applicable to more general deformable objects (e.g., dough and clay), this is not investigated in the current work. In addition, the tracking is sensitive to fast rotation and the resulting visual occlusions and motion blurs, which could be potentially addressed as a rendering-based optimization problem.  
 
 Appendix: Point and surface splatting methods represent a scene explicitly via a mixture of 2D or 3D Gaussian ellipsoid. In the case of Gaussian Splatting, the geometry is represented as a collection of 3D Gaussian, each being the tuple {X , Σ} where X ∈ R 3 is the centroid of the Gaussian and Σ is its covariance matrix in the world frame. This gives the probability density functionEQUATIONGaussian splatting decomposes it into a scaling matrix S and a rotation matrix R via Σ = RSS ⊤ R ⊤ . The color information in the texture is encoded with a spherical harmonics mapc i = SH ϕ (d i ), which is conditioned on the viewing direction ϕ.To optimize for features, existing methods tend to append an additional vector f i ∈ R d to each Gaussian, which is rendered in a view-independent manner because the semantics of an object shall remain the same regardless of view directions. The rasterization procedure starts with culling the mixture by removing points that lay outside the camera frustum. The remaining Gaussians are projected to the image plane according to the projection matrix W of the camera, which is then sorted from low to high using the distance from the virtual camera origin. This projection also induces the following transformation on the covariance matrix Σ:EQUATIONwhere J is the Jacobian of the projection matrix W. We can then render both the color and the visual features with the splatting algorithm:EQUATIONwhere α i is the opacity of the Gaussian conditioned on Σ ′ and the indices i ∈ N are in the ascending order determined by their distance to the camera origin.Following the convention [38] , GraspSplats assumes that per-gaussian feature vector f i is isotropic. The rendered depth, images, and features are then supervised using L2 loss. Note that all recent works [38, 37, 39] follow a similar paradigm as Eq. 1.We calibrate cameras using Colmap [49] with Aruco visual markers. For all settings, the in-wrist camera obtains an initial scan of the scene by going in a tabletop Bezier curve. After which objectpart text queries are provided. Though GraspSplats conceptually supports multi-camera tracking, we only use a single RGB-D camera in experiments. Manipulation success is defined as lifting the object (optionally, by the parts specified) for at least 3 seconds without re-attempting.Initialization from Depth. In Table. 5, we ablate the effect of geometric regularization by comparing several variants to initialize geometry for Gaussian. In particular, while the original Gaussian Splatting [12] paper adopts sparse Colmap [49] initialization from RGB images (Colmap-S); some recent works have found that dense Colmap reconstruction (Colmap-D) serves as a better initialization that results in faster convergence [48] . GraspSplats adopt geometric regularization and directly initializes centers of Gaussians using depth inputs, which leads to much more efficient training. Success Rate Breakdown. As shown in Table . 6, to more clearly demonstrate the comparison of part-level segmentation capabilities, we tested the success rates of 27 object-part pairs in unclustered scenes, which differs from Table . 1. It clearly shows that our approach has nearly a 100% success rate on object-level grasping while maintaining a very high success rate on part-level grasping. This is a benefit from both the feature-enhanced 3D Gaussians and part-level supervision from SAM.Given object-level bounding boxes, we crop and wrap images to square patches to obtain CLIP features. After per-patch features are computed, we create a part-level feature map by aggregating per-patch features using the inverse function of the cropping and wrapping process. For pixels that are assigned with more than two feature vectors, the features are averaged. Pixels with no assigned features are ignored during rendering-based feature distillation.Specifically, GraspSplats follows standard CLIP querying practices [1, 6, 2, 38] and takes a positive vocabulary with negative vocabularies. By default, the negative vocabularies include canonical words (i.e., 'objects' and 'things'), but can be optionally extended with user queries. To be specific, given a language set L = {L - 0 , L - 1 , . . . , L - n , L + } containing n + 1 words, where L + is the positive query and the remaining L -are negative queries. The CLIP model is used to encode each word L i into a 768-dimensional feature vector F text,i ∈ R 768 :F text,i = CLIP encode(L i ), i = 0, 1, . . . , nFor each Gaussian primitive j, there is a 16-dimensional view-independent feature vector F latent,j ∈ R 16 This is decoded into a 768-dimensional CLIP feature representation F CLIP,j ∈ R 768 :F CLIP,j = Decoder(F latent,j )Then, we calculate the cosine similarity between its CLIP feature representation F CLIP,j and each query L i in the set L :sim(F CLIP,j , F text,i ) = F CLIP,j • F text,i ∥F CLIP,j ∥∥F text,i ∥Then, we apply a softmax function to the similarities between it and all queries L i to enhance the similarity for the positive query:S j = softmax({sim(F CLIP,j , F text,i )} n i=0 )Here, S j is the similarity vector for Gaussian primitive j. After applying a temperature softmax, the similarity for the positive query L + is selected:sim positive,j = S j [n]The selecting Gaussians whose similarity to L + passes a certain threshold τ = 0.6 will be regarded as the grasping object. After Gaussians are selected, we apply the DBSCAN [45] clustering algorithm to filter out outliers.Following [14] , D obj achieves high recall by intentionally including more false positives, which ensures recall of objects. MobileSAMV2 [14] then uses D obj as priors to generate object-level segmentation masks {M}.To generate object-level features, For a given object mask M, we use Masked Average Pooling (MAP) to aggregate an object-level feature vectorEQUATION)where i is a pixel coordinate. We then construct F obj by assigning w.To generate part-level features, for a given bounding box, we crop and wrap the patches to (224, 224) . The image patch is then processed by MaskCLIP to generate feature map of shape (28, 28, 768) . We then interpolate the generated feature map to match the size of the original bounding box, and paste it onto the part-level feature map. If a pixel is assigned more than one feature, we average all assigned features.We defineF (p) = [v 3 (p)v 2 (p)v 1 (p)]as the orthogonal reference frame at point p where v 1 (p), v 2 (p), v 3 (p), correspond to the normal direction, the secondary direction, and the minimum direction of M (p). We search a 2D grid G = Y × Φ. For each (y, ϕ) ∈ G, we apply translations and rotations relative to F (p), then push the gripper along the negative x-axis until a finger or the base of the gripper contacts the point cloud. Let T x,y,ϕ be the homogeneous transformation describing translations in the x, y plane and rotation about the z-axis. The gripper h y,ϕ at grid cell y, ϕ contacts the point cloud at: F (h y,ϕ ) = F (p)T x * ,y,ϕ , where x * is the minimum distance along the negative x-axis at which the gripper contacts the point cloud. If the number of segmented objects N obj within the gripper's closed region exceeds a set threshold N th , i.e., N obj > N th , the gripper h y,ϕ is added to the candidate grasp set H. Finally, we use a geometry-aware scoring model [17, 50] to rank the grasps and select the grasp pose with the highest score.We present a comprehensive analysis of limitation, or failure modes, of the current implementation of GraspSplats. We hope they will help inspire future research.• General Deformation. The current scene deformation algorithm based on the Kabsch algorithm assumes that objects undergo rigid transformation. While the Gaussian representation is conceptually applicable to more general deformable objects (e.g., dough and clay), this is not investigated in the current work.• Tracking Failure. In addition, the tracking is sensitive to fast rotation and the resulting visual occlusions and motion blurs, which could be potentially addressed as an optimization problem using semantic and geometric priors fused in GraspSplats without assuming consistent object views. We illustrate a failure example in Fig. 6 , where GraspSplats is tasked to track a yellow toy duck with texture-less body.• Manipulation Policy. GraspSplats focuses on the grasping of the object parts via language guidance. However, in reality, it can be interesting to incorporate a more complex manipulation policy. For example, though GraspSplats supports the manipulation of articulated object, such as cabinets, by heuristic policy to pull with respect to the normal of the surface, it currently has the quasi-static assumption. Thus, it does not support more complex manipulation. Future work may explore if it is possible to learn more complex manipulation policies on top of the powerful representations of GraspSplats. In addition, though the semantic design provides a good boundary between foreground and background, it does not differentiate between objects of the same types (illustrated in Fig. 5 ).• Requirement for Scene Scanning. GraspSplats requires scanning of initial objects on the tabletop, where the scanning poses are generated from pre-programmed poses. Future work may include an active reconstruction policy to automate such a process.We use the Franka Research robot (FR3) as the main experiment platform. In addition to one Intel Realsense D435 cameras mounted on the end effector, we also use inputs from two third-person view Intel Realsense D435 cameras to facilitate scene reconstruction. We use the UMI gripper [51] as the end effector. The arm and cameras are connected to a desktop computer with a single RTX 4090 and an Intel i9-13900k CPU, on which GraspSplats is deployed.The examples of complex tasks are designed to illustrate potential use cases rather than showcase highly intricate operations. The tasks primarily focus on object manipulation, such as updating an object's position, grasping, resetting, and pick-and-place actions. For instance, in the toy car experiment in the supplementary video, after grasping the car, the robot resets the car by placing it back in its original position with the same orientation. During pick-and-place tasks, the placement orientation is aligned with the grasping orientation, and the placement location is determined by querying the center of another object.These operations highlight the system's capabilities in basic object manipulation. However, challenges arise in tasks that involve precise placement, especially when the object's orientation shifts during grasping due to the inherent instability or rotation caused by the gripper's contact. This issue is particularly evident when the gripper obstructs the view of the object, complicating the perception of the object's orientation during placement. These challenges and potential failure cases are further discussed in the supplementary material.We have provided quantitative evidence in the paper to validate the effectiveness of the depth initialization in training. For the other aspect of geometric regularization, depth supervision during the training, we provide qualitative samples in Fig. 7 .The 2D response maps for the same text query generated by different algorithms are displayed in Fig. 8 . Grounded SAM [52] in TrackAnything [47] demonstrates effective segmentation at the object level; however, it fails to distinguish parts of objects and sometimes does not respond at all. Additionally, it tends to merge multiple closely positioned objects into one. LERF [6] similarly exhibits weak responses at the part level, with very unclear segmentation between objects, making it nearly impossible to separate similar objects that are close together. F3RM [1] provides higherquality features compared to LERF and can respond to some part-level queries, but it also struggles with accurately distinguishing between similar, closely positioned objects. In contrast, our algorithm exhibits near 2D segmentation capabilities and is highly sensitive to part-level queries, allowing for precise differentiation of different parts within distinct objects.In our comparative analysis of grasp sampling methods, we employed GraspNet [32] with collision detection as LERFTOGO [2] , sampling viewpoints on a hemisphere defined by theta and phi parameters. Specifically, we tested configurations with 3x3, 5x5, and 10x10 viewpoints, effectively running GraspNet 9, 25, and 100 times respectively. The resulting grasps for these configurations are illustrated in the first three images as is shown in Fig. 9 . Notably, the red circles highlight regions where valid grasps were not generated. In contrast, for our sampling-based method, we configured the system to sample 3000 points within the workspace, utilizing 16 threads of 12900K for parallel processing. The outcome of this approach is depicted in the last image. We also compare the time efficiency of the algorithms as is shown in Tab. 7It is worth noting that while the GS grasps are directly generated with Gaussians, the point cloud data (pcd) is used solely for visualization purposes. This comparison underscores the efficiency and potential limitations of each method in terms of grasp generation and computational demands.  


Decision: The reviewers are in agreement that this paper addresses an important topic, is relevant to the conference, and provides some novelty in the form of extracting hierarchical features in Gaussian splat representations. The reviewers also generally agree in the strength of the contribution and it providing interesting results to the robot learning community.

However, reviewers also have a variety of concerns which the authors should address in the rebuttal / discussion phase.
Those that jumped out at me as most important are:
- The lack of evidence to support the claim that the approach outperforms purely point-based or voxel-baesd approaches
- The potential fragility of the tracker as an important limitation and discussion of how this could be overcome
- Issues with clarity in the presentation

Following the rebuttal I think the improved experiments to compare to a point-based representation and the additional tracking results address the majority of the reviewers concerns. While some might argue about the level of novelty of the paper, I think it provides an interesting investigation that would be of utility to the robot learning community.

Reviews: review: title: title: GraspSplats Review
paper_summary: paper_summary: This paper proposes GraspSplats, a system for utilizing Gaussian Splatting (GS) for grasping. They modify GS for grasping, adding joint object-level/part-level reasoning, integration with an external tracker to handle object motions, and add a sampling-based grasp method using the resulting representation. They compare to existing language-guided zero-shot grasping methods on a real system and show favorable results.
main_review: main_review: To the reviewer's understanding, the novelty is in a) the inclusion of both object-level and part-level features, b) the use of an external tracker to enable language-guided tracking, and c) integration with a grasping proposal method. To the reviewer's understanding, the usage of depth data to seed the GS is relatively common and not a significant modification.

The evaluation should compare to the cited paper [33], as a baseline that also utilizes a) utilizes GS for grasping, and b) can incorporate changes quickly (relatively speaking). In fact, an obvious baseline of re-seeding the GS from the current representation upon a change is missing. Even if not real-time, this is an important comparison to make.

A strength of this work is their proposed separation of object-level vs. part-level reasoning. More detailed results on how this distinction results in improved part-level tracking would strengthen the paper.

An important limitation of the proposed tracking method is the need for a language query ahead of time to know which object will move in order to identify the keypoints to track. In the real world, multiple objects may be moving or the object that is moving may not be known a priori. Perhaps a method to automatically identify and track motion would help avoid this issue.

In the experiments, it would be good to separate out results in Tables 1 and 2 into object-level and part-level results. This helps identify in which scenarios the proposed adjustments really help. Additionally, since efficiency is a major selling point, the timing results throughout this paper should be much more precise - they currently read more like rough estimates.

The use of the word "deformation" here seems misplaced - this rather evokes deformable objects (i.e. non-rigid). Considering the tracking explicitly assumes rigidity, this is especially misleading.
questions: questions: How many grasps are attempted for the results shown in Tables 1 and 3? How many comparisons are made in Table 2? What is the breakdown of performance of each method between object-level and part-level grasping?

What are the variance in grasp timing for Table 1? The timing numbers seem like estimates, but if inference time is an important part of the value of the proposed method, there should be more clarity. What is the breakdown of timing w.r.t. subcomponents like finetuning the representation or evaluating grasp proposal?

Is the partial fine-tuning presented in Sec. 3.3 used in experiments?

The explanation of how part-level features are determined is unclear - how do you perform interpolation to go from labels at parts to labels across the scene?
limitations: limitations: 2
review_summary: review_summary: The proposed work seems well motivated and shows strong performance, but is missing some important details in the experiments and should compare to existing grasping methods with GS. Additionally, their tracking relies on language guidance before hand, which seems to severely limit the applicability of the proposed tracking method. More baselines and clarity in experimentations and addressing this limitation would strengthen the work.
title: title: GraspSplats Paper - Official Review
paper_summary: paper_summary: The authors propose GraspSplats, a zero-shot object part manipulation algorithm based on 3D Gaussian Splatting. The proposed algorithm takes RGB-D frames from a calibrated camera and reconstructs the scene using 3D Gaussian ellipsoids. The 3D Gaussians are initialized from the depth frames, and GraspSplats uses MobileSAMv2 and MaskCLIP to compute object and part-level features. To handle changes in the scenes, GraspSplat uses multi-view object tracking. Then, zero-shot language-guided manipulation is performed by supplying the object and part name that needs to be manipulated to generate CLIP queries to identify the 3D Gaussians that better aligns. Then, a sampling-based grasp proposer is leveraged to compute candidate grasps around the . GraspSplats demonstrated to be successful in real robot experiments, and the paper presents a comparison of its performance against Tracking Anything, LERF-TOGO, and a zero-shot variant of F3RM in both static and dynamic scenes.
main_review: main_review: ### Strengths

- Using 3D Gaussian splatting for dynamic scene representation is not completely novel, see [1]. The way this work approaches it, the fact that they handle object parts, and the nice experimental results make this work novel enough and significant for the community.
- The proposed algorithm seems to work well in dynamic environments which are typically hard and a limitation of previous approaches.
- The authors provide a qualitative video with real robot demonstrations for an extensive and varied set of tasks that seem complex enough.
- The authors provide further qualitative study in the supplementary material. Including depth supervision effects, qualitative comparison of text queries for different approaches, GraspNet vs Grasp Sampling, and an extended study of the real robot failure cases.

### Weaknesses

- Figure 2 and Figure 4 are not very clear and hard to understand. I believe that these figures might be more useful and help to understand the method better if they would show where/how MobileSAMV2 and MaskCLIP come into play, and details are missing.
- The paper claims that the proposed representation should be better than explicit (voxel or point) representation but they haven’t provided any comparison to any baselines using point clouds or voxel. I believe that this comparison is needed to motivate the use of GraspSplats.
- A comparison to [1] or at least a citation to this work is missing.
- In Section 4.2, the author presents a comparison of part-level segmentation against LERF. However, I don’t think this comparison is very strong since they evaluate only 4 scenes. A comparison on another dataset (e.g., [1]) or a bigger number of annotated scenes is necessary.
- Details for the experimental setup are missing: How many trails do the authors carry out per type of dynamic motion (easy, medium, hard)?
- Real robot video comparisons of the other baselines are missing (Tracking Anything, LERF).

[1] Object-Aware Gaussian Splatting for Robotic Manipulation, ICRA24 Workshop 3D Manipulation, Li et al.
[2] PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding, CVPR19. Mo, et al.
questions: questions: - Improve the clarity of Figures 2 and 4.
- Improve the experimental part including explicit (voxel and/or point-cloud) representations.
- Add the missing information in the experimental section and improve the Table 2 results by adding a more significant experiment/dataset.
- Add grasping qualitative comparison to baselines.
limitations: limitations: 3
review_summary: review_summary: I am in between weak accept and weak reject, but I am currently leaning towards weak reject. I believe that the robotics community could benefit from this paper's findings, especially for object-part manipulation. However, I am concerned that the experimental part is not strong enough as detailed in my review. |||| Update after rebuttal: After reading other reviewers' comments and the rebuttal, I will raise my rate to Weak Accept. Authors have clarify and addressed most of my concerns and I am convinced by the answer given to the other reviewers.
title: title: Review for GraspSplats: Efficient Manipulation with 3D Feature Splatting
paper_summary: paper_summary: The paper introduces GraspSplats, an approach for object and part-level language-guided grasping that leverages 3D Gaussian Feature Splatting as the underlying scene representation. By using 2D point trackers, the Gaussians can be dynamically updated as objects are moved around in the scene. This is a significant advantage compared to implicit representations including NeRFs. The authors additionally show that the normals of the Gaussians can be used for real-time antipodal grasping. Extensive results in the real world demonstrate the improved speed of GraspSplats and its ability to handle dynamic scenes of varying levels of difficulty.
main_review: main_review: **Strengths**
- The paper is well-written and easy to follow. The related work is comprehensive and the problem formulation is clear.
- The proposed hierarchical feature extraction leverages object masks to get part-level features. This is an improvement over hierarchies of multi-scale crops which is both expensive to extract and distill. Both object and part-level features are distilled into the Gaussian primitives.
- Extensive real-world experiments and ablation studies to demonstrate the robustness of the proposed approach.
- The appendix and video is useful for showcasing the results. The demos with the object being moved around are impressive.

**Weaknesses**
- The explanation for the grasp sampling in Section 3.2 was not clear to me. How do you get $R_p$ from the sampled points and what is $M$? It seems like this formulation is based on GPG but it would be good to keep this self-contained within this paper or explain it through a figure.
- Limited information on the Partial Fine-Tuning in Section 3.3 and how it works. Are additional Gaussians potentially added here?
- The object tracking is inherently based on the results of a 2D point tracker on a single view. This means that tracking will fail if objects are rotated in such a way that the points become out of view or the object becomes occluded.
questions: questions: See issues raised in the weaknesses above.

**Other questions**
1. One advantage of neural fields is being able to use RGB images and reconstruct transparent and specular objects. Did you experiment with such objects? Does using RGB-D and depth supervision degrade these properties?
2. For moving the Gaussians as objects move dynamically, could you further optimize them based on the incoming image observations? How do you think that would compare to using the 2D tracker?
limitations: limitations: 3
review_summary: review_summary: While the paper presents several incremental improvements to Feature 3DGS, these improvements are critical for moving towards real-world deployability. The ability to enable dynamic scene updates and real-time antipodal grasping is a significant improvement over NeRF-based representations
title: title: Sound main idea, but needs some improvements.
paper_summary: paper_summary: This method proposes Gaussian splatting-based language-embedded feature fields for task-oriented grasping in dynamic scenes. Given the scene, the method first constructs a GaussianSplat that models the scene in terms of color, depth, and dense visual features. The features are then passed to shallow MLPs to predict object and part features. These features are used together with language-based queries to find the corresponding grasp region on the scene. From this region, grasp samples are generated through an antipodal grasp sampler (GPG). These grasp samples are ranked using geometry-aware scoring methods, and the best grasp is selected. Additionally, the paper uses a tracker to update the underlying feature field by rigidly transforming the Gaussians of the GaussianSplat. This makes the model robust to the changes in the scene.
main_review: main_review: Overall, the idea looks sound and straightforward. The main novelty is using Gaussian splats for modeling language-embedded feature fields instead of neural radiance fields. The paper's main claim is that Gaussian splats are superior for distilling feature fields, and the experiments demonstrate this claim both in speed and performance. However, I have the following questions and concerns:

* I understand that Gaussian splats are faster than Nerfs, but I don’t understand why they perform better regarding learned features, as shown in Table 2. This table shows that somehow, gaussian splats learn better dense visual features than Nerf (As I assume, prediction of F_part and F_obj should be similar). Moreover, in lines 241-242, the paper mentions that Nerf-based methods perform better on static scenes.
* Grasp success is only estimated using whether the robot can lift the object for 3 seconds. This means the grasp success results do not consider the grasped part. However, one of the main contributions of the work was task-oriented grasping.
* In the supplementary video, complex language queries correspond to multistep plans. How did you process language queries such that these plans are generated? For example, in one of the tasks (toy car experiment in 1.58), the robot picks up the car when it stops and puts it in its initial position. However, I expect this method only to handle the car grasping part. How is the rest accomplished? The paper does not discuss these.
* The results in Table 2 are only from 4 scenes, which is relatively low. 
* The experiment sections contain a very minimal qualitative analysis of the method. I noticed that most of these analyses (also limitations) are included in the supplementary video and material. Some of these should be moved to the main paper.

Clarity:

There are some clarity issues on paper:
* In equation 1, N, as I understand it, is the number of Gaussians in the Gaussian Splat, but this is not discussed anywhere. 
Lines 148 to 152 explain how F_part is estimated, but it is quite unclear. what does uniform query mean? Why is N significantly smaller than the number of patches needed from uniform queries? In line 151, how F_part is estimated is very ambiguous (which features are you talking about in this line?). It would be better to write this part more mathematically for easy understanding.
* Why do you use cosine loss to train F_part and F_object MLPs?
* Between lines 169-170, how queries are selected is unclear. Maybe you can write this part more mathematically as well.
* The Open-vocabulary Conditional Part-level Querying section (Lines 172-178) discusses how part queries lead to activation of both the object and the part (e.g. object and part). To prevent this, you segment the object; however, why does this prevent activation of the object as it still appears in the image?

Other Comments:
* The paper discusses that it is not possible to model dynamic scenes with implicit representations. While I agree it is better to model dynamic scenes with explicit methods, this is not exactly true. For example, it would be quite easy to apply rigid transformations to the SDF-based methods. Furthermore, many works model dynamic scenes with Nerfs using deformation fields (Some examples are [1], [2]).
* There is a line of work in robotics that predicts grasp poses based on descriptors or dense correspondences [3-7]. This line of work is very relevant to this paper, as they also use such representations for grasp prediction.

References:

[1] Park, Keunhong, et al. "Nerfies: Deformable neural radiance fields." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[2] Lin, Chen-Hsuan, et al. "Barf: Bundle-adjusting neural radiance fields." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

[3] Florence, Peter R., Lucas Manuelli, and Russ Tedrake. "Dense object nets: Learning dense visual object descriptors by and for robotic manipulation." arXiv preprint arXiv:1806.08756 (2018).

[4] Manuelli, Lucas, et al. "kpam: Keypoint affordances for category-level robotic manipulation." The International Symposium of Robotics Research. Cham: Springer International Publishing, 2019.

[5] Simeonov, Anthony, et al. "Neural descriptor fields: Se (3)-equivariant object representations for manipulation." 2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022.

[6] Yen-Chen, Lin, et al. "Nerf-supervision: Learning dense object descriptors from neural radiance fields." 2022 international conference on robotics and automation (ICRA). IEEE, 2022.

[7] Sucar, Edgar, Kentaro Wada, and Andrew Davison. "NodeSLAM: Neural object descriptors for multi-view shape reconstruction." 2020 International Conference on 3D Vision (3DV). IEEE, 2020.
questions: questions: My main concerns are:

* As I see there is no quantitative analysis part-level grasping. 
* The paper requires some clarity improvements.

Other than that, the authors can also address other comments on my review.

Update: My concerns were addressed.
limitations: limitations: 3
review_summary: review_summary: I think paper is well written and demonstrated. However, core idea is using gaussian splatting instead of Nerfs for learning feature fields. It is not a significant novelty.