GaussianCut: Interactive segmentation via graph cut for 3D Gaussian Splatting
Authors: Umangi Jain
Ashkan Mirzaei
Igor Gilitschenski
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Vision
Segmentation
Graph cut
Gaussian Splatting
Primary area: machine_vision
TL;DR: Object selection and segmentation using graph cut for scene represented using 3D gaussian splatting

Abstract: We introduce GaussianCut, a new method for interactive multiview segmentation of scenes represented as 3D Gaussians. Our approach allows for selecting the objects to be segmented by interacting with a single view. It accepts intuitive user input, such as point clicks, coarse scribbles, or text. Using 3D Gaussian Splatting (3DGS) as the underlying scene representation simplifies the extraction of objects of interest which are considered to be a subset of the scene's Gaussians.  Our key idea is to represent the scene as a graph and use the graph-cut algorithm to minimize an energy function to effectively partition the Gaussians into foreground and background. To achieve this, we construct a graph based on scene Gaussians and devise a segmentation-aligned energy function on the graph to combine user inputs with scene properties. To obtain an initial coarse segmentation, we leverage 2D image/video segmentation models and further refine these coarse estimates using our graph construction. Our empirical evaluations show the adaptability of GaussianCut across a diverse set of scenes. GaussianCut achieves competitive performance with state-of-the-art approaches for 3D segmentation without requiring any additional segmentation-aware training

GaussianCut: Interactive Segmentation via Graph Cut for 3D Gaussian Splatting
Umangi Jain, Ashkan Mirzaei, and Igor Gilitschenski
University of Toronto
{umangi, ashkan, gilitschenski toronto. edu
Foreground Gaussians
Multiview Images
Pretrained 3DGS
Caussianul
"A Truck"
Sparse Point Clicks
Scribbles
Text Prompt
Background Gaussians
Figure I: Our method, GaussianCut, enables interactive object(s) selection. Given an optimized Gaussian Splatting model for scene with user inputs (clicks, scribbles, text) on any viewpoint, GaussianCut partitions the set of Gaussians as foreground and background.
Abstract
We introduce GaussianCut, new method for interactive multiview segmentation of scenes represented as 3D Gaussians. Our approach allows for selecting the objects to be segmented by interacting with single view. accepts high-level user input, such as point clicks, coarse scribbles, or text. Using 3D Gaussian Splatting (3DGS) as the underlying scene representation simplifies the extraction of objects of interest which are considered to be a subset of the scene's Gaussians. Our key idea is represent the scene as a graph and use the graph-cut algorithm minimize energy function to effectively partition the Gaussians into foreground and background. To achieve this, we construct graph based on scene Gaussians and devise segmentation-alignen energy function on the graph to combine user inputs with scene properties. To obtain an initial coarse segmentation, we leverage 2D image/video segmentation models and then use graph cut to further refine these coarse estimates. Our empirical evaluations show the adaptability of GaussianCut across diverse of scenes. GaussianCut achieves competitive performance with srate-of-the-as approaches for 3D segmentation without requiring any changes the underlying Gaussian- based scene representation. Project page: https: / //umangi- jain. jain.github..... io/ gaassiiannnii
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Recent advances in 3D scene representation have enabled unprecedented quality in 3D view synthesis without requiring specialized equipment or an excessively high computational budget. Fully leverag- ing these advances requires tools for scene understanding and manipulation specifically designed to operate on such representations. Object selection and segmentation often serve as a crucial first step in both scene understanding and editing tasks. While 2D image segmentation has been widely studied, developing analogous techniques for 3D remains challenging. One key challenge is accounting for the choice of underlying 3D scene representation in the segmentation method.
3D Gaussian Splatting (3DGS) [23] offers an explicit representation of a scene using a set of Gaussians. The nature of this representation motivates the idea that each Gaussian corresponds to either the segmented object or the background. Prior works in 3DGS segmentation involve augmenting each Gaussian with low-dimensional feature, that is jointly optimized with the parameters of the Gaussians [6, 44, 55]. This supervised by 2D features, which provide semantic information that can be used for segmentation. While this enables a 3D consistent segmentation, it significantly increases the fitting time and the already high memory footprint of the method. Thus, enabling 3DGS segmentation without modifying the optimization process is an important research challenge.
We address this challenge by proposing GaussianCut, a novel method for selecting and segmenting objects of interest in 3D Gaussian scenes. Our work taps directly into the representation created by 3DGS and maps each Gaussian to either the foreground or background. The proposed process mirrors the interactive nature of 2D segmentation tools, where users can engage through clicks, prompts, or scribbles. We require such user input on a single image and perform the object selection process in two steps. First, we obtain dense multiview segmentation masks from the user inputs using a video segmentation model. Suhsequently, we construct weighted graph, where each node represents Gaussian. Graph cut then partitions the graph into two disjoint subsets by minimizing an energy function, which quantifies the cost of cutting the edges connecting the subsets. This approach effectively segments the selected foreground object from the background by using the energy function as a measure of dissimilarity between the nodes. An overview of the process is provided in Figure 1.
Our main contribution is novel approach for segmentation in scenes obtained from 3DGS. Its main technical novelties are twofold: 1) we propose a method for graph construction from a 3DGS model that utilizes the properties of the corresponding Gaussians to obtain edge weights, and 2) based on this graph, we propose and minimize an energy function (Equation 3) that combines the user inputs with the inherent representation of the scene. Our experimental evaluations show that GaussianCut obtains high- fideliyy segmentation outperforming previous segmentation baselincs.
2 Related Work
2D image segmentation is a long studied problem in computer vision [17, 41, 57]. Recently, models like Segment Anything [25] and SEEM [64], have revolutionized 2D segmentation by employing interactive segmentation. A range of methods have also been developed for 3D segmentation, each tailored different forms of representation, including voxels [10, 36), point clouds [42, 43), meshes [49, 62], and neural representations [7, 32, 48, 52]. The impressive capabilities of Neural Radiance Fields (NeRFs) [35] in capturing scene information implicitly have inspired numerous studies to explore 3D segmentation for NeRFs. Recent works have also explored segmentation with Gaussians as the choice for scene representation [6, 22, 44, 55, 63].
Training 3D segmentation with 2D masks/features: In addition to the wide adaptation of founda- tional models for 2D images [58], they are also used extensively by 3D editing and segmentation models. SAM has been used as an initial mask to facilitate 3D segmentation [6. 7, 33] and also for distillation into NeRF [59] and 3DGS models [63]. Semantic-NeRF [61] proposed 2D label propa- gation to incorporate semantics within NeRF so it can produce 3D consistent masks. MVSeg [37] propagates 2D mask to different views using video segmentation techniques. ISRF [18] distills semantic features into the 3D scene of voxelized radiance fields. Nearest neighbor feature matching then identifies high-confidence seed regions. 2D features have also been used for facilitating the grouping of Gaussians [55] and for hierarchical semantics using language in 3DGS [44]. Distilled
Feature Fields (DFF) [27] and Neural Feature Fusion Fields [50] distill 2D image embeddings from
LSeg [30] and DINO [5] to enable segmentation and editing. SA3D [7] uses SAM iteratively to

---

get 2D segments and then uses depth information to project these segments into 3D mesh grids. SANcRF-HQ [33] aggregates 2D masks in 3D space to enable segmentation with NeRFs.
Segmentation in 3D Gaussian Splatting: Gaussian Grouping [55], SAGA [6], LangSplat [44], CoSSegGaussians [11] and Feature 3DGS [63] require optimizing 3DGS model with additional identity or feature per Gaussian, which is usually supervised by 2D image features. These semantic features allow segmentation through user interaction. Gaussian Grouping and LangSplat also allow for textual prompts to segment objects supported through multimodal models like CLIP or grounding- DINO [31]. Feature-bass based methods alter the fitting process of 3DGS by adding additional attributes for each Gaussian and it facilitates learning features for everything in the scene. While useful, this limits the flexibility of interactivity with a single object. Our method is more flexible in choosing specific object(s) as we generate the 2D masks after the user interaction. Adding additional parameters also increases the fitting time for 3DGS. Moreover, such methods often rely on video segmentation models as they require 2D features from all training viewpoints. In contrast, we can operate on an arbitrary number of 21) masks, including just a single mask.
Graph cut for 3D segmentation: Boykov and Jolly [4] introduced a novel global energy function for interactive image segmentation using graph cut [15, 19]. Several follow-up works improved image segmentation using graph cut by designing a better energy function [13], improving optimization [3, 20], and reducing user input requirements [51]. Adapting energy minimization methods for 3D volumes has been difficult, requiring several modifications [20] to manage the higher memory demands. NVOS [45] trains a special multi-layer perceptron (MLP) to predict voxels in the foreground and background and applies graph cut voxels as post-processing. However, this requires additional training and increases memory consumption. Guo er al. [21] propose 3D instance segmentation of 3D point clouds using graph cut. It involves constructing superpoint graph and training separate graph neural network for predicting the edge weights. Unlike their work, our method is a post hoc application and does not require any additional training, Our graph construction and edge weights have also been tailored specifically for 3D Gaussian Splatting.
Concurrent with our work, Segment Anything in 3D Gaussians (SAGD) [22], also performs interac- tive segmentation using 3D Gaussian Splatting without requiring any segmentation aware training. However, their primary focus is on refining object boundaries by decomposing boundary Gaussians.
3 Method
3.1 Preliminaries
3D Gaussian Splatting (3DGS) [23] is technique for creating 3D representation of scenes based on a set of Gaussian ellipsoids G. 3DGS facilitates real-time rendering and provides high-quality reconstruction. In this representation, each 3D Gaussian is characterized by a set of optimizable parameters. These include 3D position μ E IR3, spherical harmonics (SH) coefficients (for color) a E 3 +1) (where d is the degree of spherical harmonics), scale E R3, rotation E R.A, and opacity a E RR. The optimization process involves iteratively rendering scenes and comparing the rendered images with the training views, interleaved with adaptive density control that handles the creation and deletion of the number of Gaussians. The differentiable rendering pipeline in 3DGS uses tile-based rasterization following [28] to ensure real-time rendering. 3DGS performs splatting by depth sorting the Gaussians and a-blending [14] them to project in 2D. The set of differentiable parameters for g. D = (1) Br, 8;, Ii ao (1) are optimized from set of posed images.
Graph cut refers to the problem of partitioning the vertices V of a graph G with edges E weighted by hees into two disjoint, non-empty sets such that the sum of the weights of the edges between the two sets is minimized. This minimum-cost partitioning is known as the minimum cut. Segmentation can be reformulated as graph cut. This is achieved by forming a graph that includes the segmented entities (e.g. pixels, voxels, Gaussians) and introducing weighted edges. This allows defining an energy function, which includes unary terms representing the cost of assigning a node to set based on individual properties, and pairwise terms that incorporate the cost of assigning neighboring nodes to different sets. The objective of the minimization is to find a cut that optimizes the overall energy, balancing individual preferences and neighborhood interactions. An efficient way for computing this algorithm [2].
minimum cut in graph is the
3

---

Seqmentation images
Optibbed Splatting model
Separated foreground background objects
Video Sepmentation
Rasterization with masks
Rander each componant seperately
Graph construction
input for regrnentation
Track low.grountbankkgguued Ikeinood Gaumsier
Gaansia graph cut
Figure 2: Overall pipeline of GaussianCut. User input from any viewpoint is passed video segmentation model to produce multi-view masks. We rasterize every view and track the contribution of each Gaussian to masked and unmasked pixels. Then, Gaussians formulated nodes in undirected graph and we adapt graph cut to partition the graph. The red edges in the graph highlight the set of edges graph cut removes for partitioning the graph.
3.2 Overview
Given set of posed RGB images I {L.) f=1 and an optimized reconstruction of the scene using set of Gaussians G, we define the task of interactive 3D segmentation as follows: given a user input (point clicks, scribbles, text) on any image E Z, the objective in two non-empty disjoint sets S and T such that S represents the set of Gaussians representing the object(s) of interest and T represents the given 3D scene without these object(s). The extracted subset of Gaussians can be rendered from any viewpoint to effectively cutout the 3D representation of the foreground without retraining. Innately, the other of Gaussians can rendered to remove foreground object(s). Figure shows an overview of our pipeline.
partition the of Gaussians
In order densify the object selection information provided by the user, we off-the-shelf video segmentation model to obtain dense segmentation masks from multiple views (discussed in section 4). For transferring the segmentation masks the 3DGS representation, Gaussians that map to the selected region in the masks (discussed in section 3.3). However, the masks used for propagation are not 3D consistent (as the underlying image segmentation model is 3D-unware). Moreover, the eπrоrs from 2D masks can propagate in the traced Gaussians and thereby provide noisy 3D segmentation. To obtain a precise set of foreground Gaussians, formulate the set of Gaussians G as nodes in graph (discussed in section 3.4) and leverage graph cut to split the nodes into two sets: the foreground S and the background
trace the 3D
3.3 Mapping user inputs to Gaussians
We first feed the sparse single-view annotations by the user (e.g., point clicks) to multiview/video segmentation model to obtain coarse segmentation masks across multiple training views. We then propagate the information from the 2D masks onto the set of Gaussians. For already optimized 3DGS model of a scene, G, we obtain n masks M == {M'} from video segmentation model
corresponding any n viewpoints I = {I {I - Here, MD indicates the set of foreground pixels in the viewpoint I', For each Gaussian g E G, maintain weight, Wg, that indicates the likelihood of the Gaussian belonging to the foreground. To obtain the likelihood term wi pertaining to mask for Gaussian g. we unproject the posed image I back to the Gaussians using inverse rendering and utilizing the mask information,
EpEmS (p(P)T/ w/
(I)
EpED (p)TT (p)

---

where (p) and T. (p) denote the opacity and transmittance from pixel for Gaussian g. Combining over all the masks,
Σ, EpErs dg(P)Tz(p) "9 EpErE og (p) (p)
(2)
This likelihood, Wg. captures the weighted ratio of the contribution of Gaussian g to the masked pixels relative the total number of pixels influenced by it. The complementary value of ug. 1- provides the likelihood of Gaussian g contributing to the background. The value of wa is updated using 2D segmentation masks during rasterization. Since the rasterization of 3DGS is remarkably fast, each pass typically takes less than a second to update. GaussianEditor [8] also learns an additional tracing parameter but unlike our approach, it maps Gaussians to different semantic classes.
Having the likelihoods Wgg, naive approach to extract the 3D representation of the foreground is to threshold the Gaussians and prune those with values below a certain threshold т. We denote this approach "coarse splatting". Figure 4 demonstrates coarse splatting renders for plant in the 360-garden scene [1]. Note that the renderings produced by coarse splatting are not accurate, particularly around the edges. This is due to two main reasons: 1) the 2D segmentation models are 3D inconsistent and can be imperfect, leading to artifacts in the final Gaussian cutout, and 2) lifting 2D masks to 3D can introduce its own artifacts.
3.4 Gaussian graph construction
After rasterization with the masks, each Gaussian g E G in the 3DGS representation characterized og Wg where w,y captures the user requirement and the other
with parameters D2 := (Mg 3 parameters encapsulate the inherent properties of the scene. To fuse the two sources of information and obtain a precise set of foreground Gaussians, we formulate the optimized 3DGS model as an undirected weighted graph G == (G,E), where each Gaussian in 9 is a node and E represents the set of edges connecting spatially adjacent Gaussians. We define the neighborhood W C G x G of node (Gaussian) as its k-nearest Gaussians in terms of their 3D position. The intuition behind constructing the edges is that Gaussians that map to the same object would be close spatially.
Sy: gg
Gaussian graph cut partitions the Gaussians G into two disjoint and non-empty sets S C and T C G. that represent the foreground and background Gaussians, respectively. Our objective is to infer the foreground/backgroum label V/ E {0, 1] of each Gaussian g. Let the unary term (,(-)) represent the likelihood of node g being part of foreground or background and the pairwise term (..-((..-) reflect the edge connection between node g and g'. To obtain the label for each Gaussian g. graph cut minimizes the aggregate of both unary and pairwise terms given by:
E = Σ
Mo) +
Σ 9.9'EN
(Pg Dgr),
(3)
where À provides a trade-off between the two terms.
Neighboring pairwise weights (n-links): The pairwise term models the correlation between neigh- boring nodes. The neighbors for a node are based on its spatial proximity other nodes. The edge weight between each pair of neighbors is combination of its spatial distance and color similarity. While segments of an object can have multiple colors, and they often do, neighboring nodes with dissimilar colors can still be identified and grouped based on their spatial proximity. This ensures that parts of an object, despite varying in color, can be linked if they are close in space. For color similarity, we only use the zero-degree spherical harmonic to capture the ambient colors without any angular dependence. The correlation between the neighboring nodes formulated as
Va.a (Pg, ) f(ug:: myy' X ( (3))
where A, is hyperparameter balancing the contribution of position and color similarity, and the function f estimates similarity as f(x. y) = exp(->/lx yl13) (y is a positive scalar).
Unary weights (1-links): We designate two terminal nodes for the graph cut algorithm, the source and the sink node. These terminals represent the foreground (source) and the background (sink) in segmentation tasks. t-links connect all the nodes to both the terminal nodes and the edge weight for
5

---

munnkon Relmunneen
Figure 3: Visualization results of different objects in the following scenes: truck from Tanks and Temples [26], kitchen from Mip-NeRF 360 [1], tools from Shiny [54].
these links represents the pull of that node towards each terminal node. We assign the edge weights connecting each non-terminal node to the source node to reflect its likelihood of belonging to the foreground set S (and as belonging to the background set T for edges connecting to the sink node).
Gaussian tracking, from section 3.3, provides the connection of each Gaussian y to the source and sink terminal nodes, using Wy and 1 Wgg respectively. However, these weights can be noisy estimates. Therefore, we introduce an additional ferm to the edge weights that captures the similarity of node y to the other nodes that are well-connected to the terminal nodes. To do so, we identify high-confidence nodes for both the source and the sink terminals. A Gaussian is considered as a high-confidence node for the source terminal if W and for sink terminal if w. ~ O. Since computing the similarity of a node to all the nigh-coonidence nodes is computationally expensive, we cluster all the high- confidence nodes (denoted as F and B for the source and sink, respectively) based on their position. For each node g, we then determine the closest cluster by finding 9 = argmin, of the source, and similarly Iu for the sink. Consequently, the unary term based the user input is,
(1
g. Hgr) for
(Da g1 Ho) = (y du Hug (Dg:Dor)
if y/ 1, y/
11 11- Wy + Xu A.bg..n (Dg. Don)
We minimize the objective E in Equation 3 to partition the set of nodes G as S (foreground Gaussians) and T (background Gaussians). To render the foreground object from any viewpoint, we simply render the Gaussian collected in S with T as background.
4 Experimental Setup
Datasets: For quantitative evaluation, we test the scenes from LLFF [34], Shiny [54], SPIn-NeRF [37], and 3D-OVS [29]. All selected scenes from the LLFF and Shiny datasets real-worle front-facing scenes, with 20-62 images each. SPIn-NeRF provides a collection of scenes from some of the widely-used NeRF datasets [16, 26, 34, 35, 56]. It contains combination of front- front-facing and 360 inward-facing real-world scenes. 3D-OVS contains scenes featuring long-tail objects.
Input types: Our model accepts all input processed by Sra-Track [9]. It uses grounding- DINO [31] to process text inputs. For the LLFF scenes used in NVOS [45], we follow their input scribbles to obtain the initial mask. For SPIn-NeRF and Shiny,
clicks (each scene typically requires 1-4 clicks). For the 3D-OVS dataset evaluation, we use text query as input (results in Table 11).
Evaluation metries: Different Image-Based Based Rendering (IBR) models represent 3D scenes in different ways. Thus, obtaining universal ground-truft 3D masks is difficult. To avoid this challenge, we evaluate the segmentation mask of the projected 2D rendering from the scene. The ground-truth 2D masks are typically obtained from professional image segmentation tools. NVOS provides one ground-truth mask for every scene in LLFF. SPIn-NeRF and 3D-OVS provide masks for multiple images every scene. Shiny dataset does not contain any ground-truth masks so we create our own ground-truth touuh mask. For evaluation, we generate 2D foreground masks by rendering the Gaussians

---

from the desired viewpoint. We use pixel classification accuracy (Acc) and foreground intersection- over-union (IoU) for evaluating the segmentation masks.
Following NVOS, we also assess the photo-realistic appearance of the segmented object by rendering it against black background. We trim both the rendered image and the ground-truth image the foreground object by applying bounding box that fits the ground-truth mask. This prevents the evaluation from being biased by the background, especially when the object of interest relatively small. The metrics we report are PSNR, SSIM [53], LPIPS [60].
Implementation details: To obtain segmentation masks from the user inputs (used in Section 3.3), we leverage the advancements in video segmentation models. The user selects the foreground objects on Io, and we obtain dense masks for multiple views using SAM-Track [9]. Note that the use of the video segmentation model is done to enhance the performance further and our method can also work with single image mask (Table 4). We use KD-Trre for efficiently finding the l nearest neighbors construct the edges between the nodes.
For all the evaluations, resize the longer image size to 1008, as commonly practiced in novel- view synthesis. We optimize 3DGS model for each of the considered scenes, without making any changes to the original 3DGS code. For coarse splatting, we keep the cut-off threshold 0.9 for front-facing views and = 0.3 for the 360 inward facinng scenes. This disparity stems because parts of objects might not be observed from every viewpoint for the latter and also because of the relative ineffectiveness of video tracking for fnwrrd-facing scenes (Figure 8). For graph cut, we keep = 0.1 for all other weights, the number of neighbors for every node as 10, and the number clusters for high-confidence nodes as 4 for sink and 1 for source. X, An, Au can be adjusted depending the scene and the quality of coarse splatting but generally, An = A and 0.5 give decent results.
for neighboring pairwise position weights and
Baselines: Our comparison includes selection of baseline models such as NVOS [45], MVSeg [37], Interactive segmentation of radiance fields (ISRF) [18], Segment Anything 3D with NeRFs (SA3D) [7]. Segment Any 3D Gaussians (SAGA) [6]. Segment Anything in 3D Gaussians
(SAGD) [22], Gaussian Grouping [55], and SpngSplat [44]. Unlike our approach, SAGA, Gaussian Grouping, and LangSplat alter the Gaussian optimization process by learning additional features per Gaussian that increases the optimization time (Table 9). SAGD a concurrent work also de- signed for 3DGS segmentation and has yet been published. Thus, their results may be subject change. SAGD, similar to our approach, does not require any segmentation-awal training and uses a cross-view label voting approach segment selected objects. All the baselines allow for selecting objects using clicks, except LangSplat, for which we use text queries. Further details on baseline implementation provided in Appendix A.I.
5 Results
5.1
Quantitative results
We take the scenes from LLFF dataset used in NVOS. NVOS contains a reference image with input scribbles and target view with an annotated 2D segmentation mask. As shown in Table 1, GaussianCut outperforms other approaches. Unlike ISRF, SAGA, Gaussian Grouping, and LangSplat, GaussianCut works on pretrained representations and does not require any changes to the training process. Owing to the fast rasterization, 3DGS-based approaches can also
Dataset from NVOS:
Table 1: Quantitative results for 2D mask seg-
Table 2: Quantitative results on the SPIn-
mentation on NVOS dataset [45].
NeRF dataset [37].
Method
IoU (%)T
Acc (56)
Method
loU (%)
Acc (%)
graph cut (3D) [45, 46]
39.4
73.6
MVSeg
90.4
98.8
NVOS [45]
70.1
92.0
ISRF [18]
71.5
ISRF [18]
83.8
96.4
SA3D [7]
92.4
98.8
SA3D
90.3
98.2
SAGD [22]
89.7
98.1
SAGD [22]
72.1
91.7
SAGA [6]
88.0
98.5
SAGA [6]
90.9
98.3
Gaussian Grouping [55]
90.6
98.2
Gaussian Grouping [55]
88.4
99.0
LangSplat [44]
74.0
94.0
LangSplat [44]
69.5
94.5
GaussianCut (Ours)
92.5
98.4
GaussianCut (Ours)
92.9
99.2
7

---

Table 3: Object rendering results on NVOS [45].
Table 4: Quantitative results on Shiny [54].
Metrics
SSIMT
PSNR (dB)f
LPIPS]
Scenes
IoU (
Ace (%)
graph cut (3D) [43, 46]
0.600
15.03
0,415
SA3D [7]
93.3
98.5
NVOS [45]
0.767
18.40
0.213
SAGD [22]
83.3
84.7
SA3D [7]
0.794
20.76
0.198
Coarse Splatting
94.3
99,4
GaussianCut (Ours)
0.840
22.45
0.132
GaussianCut (Ours)
95.0
99.5
render foreground Gaussians in real-time. To compare the rendering quality of the segmented objects using 3DGS, we train a NeRF model at the same resolution and segment it using SA3D. Table shows the photo-realistic quality of the foreground image against a black background. Gaussian Splatting provides significant gains over NVOS and SA3D for rendering quality, providing boost of +4.05 dB PSNR and +1.69 dB PSNR, respectively.
Dataset from SPIn-NeRF: We compare our model on all scenes from the SPIn-NeRF dataset, which includes four 360 inward-facing scenes and six front-facing scenes. Our model gives an overall better performance compared to other baselines. Compared to MVSeg, on 360 scenes such as lego and truck, GaussianCut provides an absolute IoU gain 14.3% and 10.5%, respectively. Our model also performs better compared to other 3DGS baselines shown in Table 2. The 360 scenes for ISRF were run at one-fourth resolution due to memory constraints. We show the scene-wise results in Table 10. Feature-based 3DGS segmentation methods, such as Gaussian Grouping and LangSplat, outperform GaussianCut on certain scenes, but their interactivity can be limited if the optimized features do not delineate the object of interest. We show such cases in Figure 5. Moreover, also show in Figure 11 that segmentation masks from GaussianCut contain finer details and a better segmentation quality than the ground-truth masks provided in SPIn-NeRF.
We test the segmentation performance of our model on four scenes from the Shiny dataset: tools, pasta, seasoning, and giants. We create ground-truth masks for 4 test-view images for each scene and compare our model against non-feature learning-baa -based baselines: SA3D and SAGD, We also report the performance of Coarse Splatting (no graph cut) in Table 4. Figure 10
Dataset from Shiny:
also shows the quality of segmented images for all the scenes.
5.2 Qualitative results
Table Ablation of the energy function averaged over the seven scenes from LLFF dataset. NVOS
Similar to SA3D, we perform segmentation in three modes: object segmentation, part segmen- tation, and rext-prompting based segmentation. Figure 3 shows object and part segmentation. GaussianCut can retrieve complex objects (such
Energy
Single
86.6
Coarse
91.2
w/o n-link spatial similarity
92.4
w/o n-link color similarity
92.3
as truck, lego bulldozer, mirror) precisely. It can segment smaller part segmentation (man- hole cover, lego wheels, and socket wrenches). Our method can also retrieve multiple objects together (socket wrenches and metallic bowl extracted together).
w/o t-link cluster similarity
91.5
GaussianCut (Ours)
92.5
Figure 4 demonstrates the performance of objection selection using text input. We do qualitative comparison of GaussianCut with ISRF, SA3D, and SAGD. Feature-based 3DGS methods run into memory issues for this scene. Out of these, SA3D and SAGD also use text-based prompt to segment the plant. ISRF uses stroke for segmentation. GaussianCut retrieves finer details in the plant with a higher perceptual quality. We also show the rendered image from different viewpoints. It can also be seen that coarse splatting (before the Gaussian graph cut) misses finer details, such as the decorations on the plant, which can be retrieved using GaussianCut.
5.3 Ablation and sensitivity study
Number of views: To obtain the 2D segmentation masks, we run the camera on fixed trajectory and get rendering from different viewpoints (spiral trajectory for front-facing and circular for 360 inward scene). We limit the number of frames to 30 for front-facing and 40 for 360" scenes. Using more segmentation masks can boost performance, however it might not be always preferred, especially for scenes with large number of training views. SAM-Track can also handle segmentation for unordered multi-frame images. Table 6 shows the effect of varying the number of masks two scenes from the
8

---

Figure 4:
Qualitative comparison: 3D segmentation results GaussianCut using text on 360-
garden [1] scene. Compared to ISRF [18], SA3D [7], SAGD [22], GaussianCut segment contain finer details. The graph cut component of GaussianCut also retrieves fine details (like decorations on the plant) that missed in coarse splatting.
SPIn-NeRF dataset (images used were unordered). Since 3DGS offers fast rasterization, the overall time cost for segmentation does not grow linearly with the number of masks as the time taken for segmentation dominates. We also show the qualitative performance with single mask (no video segmentation model) and just scribbles (no image segmentation model) in Figure 13.
Sensitivity of each term in the graph cut energy function: In order to understand the contribution of each term in the energy function, Table 5 shows the average IoU the NVOS dataset with each term removed from Equation 4 and 5. Each term contributes to the overall performance and the cluster similarity. in particular, gives a significant boost.
Sensitivity of graph cut algorithm the number of neighbors (number of edges for each node) and the number of high- confidence clusters. As the number of neighbors increases, the number of edges in the graph also increases (so does the time taken for graph cut). As seen Table 7, adding more edges can help in modeling more long-distance correlations. However, after a limit, the effects of adding more edges diminish. Adding large number of clusters for the high-confidence nodes, Table 8, does affect the performance drastically and the optimal number can vary depending on the scene. We show sensitivity other hyper-paramene parameters Appendix
We test the sensitivity of our Gaussian graph cut
Discussion
Our results demonstrate that 3DGS allows for direct segmentation using a pretrained model. Devel- opments in 2D segmentation and tracking have played crucial role in 3D) segmentation. We observe that GaussianCut not only generates 3D consistent masks but also improves the segmentation quality of 2D masks by capturing more details (Figure 12). This is more prominent for 360" scenes, where the tracker can partially or fully miss the object of interest (Figure 8).
Table 6: Performance of GaussianCut with varying the number of views passed to the video segmen- tation models. The number in parenthesis is the percentage of total views for the scene.
Number of views
5 (10%)
9(20%)
21 (50%)
43 (100 %)
Coarse Splatting Fortress
96.1
96.3
96.5
96.8
GaussianCut on Fortress
97.7
97.8
97.8
97.9
Time Cost
51
55
59
71
Number views
11 (10%)
21(20%)
51 (50%)
102 (100 %)
Coarse Splatting on Lego
85.5
88.0
88.4
88.9
GaussianCut Lego
87.3
89.1
89.2
89.2
Time Cost (s)
58
62
72
90

---

Table 7: Ablation on the number of neighbors.
Table 8: Ablation on the number of clusters for high-confidence nodes.
#Neighbors
10
50
100
Horns
91.9
93.6
93.8
94.3
#Clusters
5
20
Time (s)
18
57
209
410
Fortress
97.3
97.8
97.6
97.5
Truck
93.3
95.7
95.3
95.2
Horns 93.8
93.9
94.0
94.0
Time (s)
32
96
393
738
Truck
95.6
95.7
95.6
95.5
Time requirement: Since we use pretrained 3DGS models, the optimization time for the Gaussians remains the same as in [23] (it took under 15 minutes for every scene we used). For inference, masked rasterization of Gaussians fast and the time taken for graph cut grows roughly linearly with the number of Gaussians. Table 9 shows detailed breakdown of time taken in each step: preprocessing (obtaining the features from 2D image/video segmentation models), fitting time (3DGS optimization time), and segmentation time (time taken to obtain the segmented output). Compared to feature-based methods, like Gaussian Grouping, LangSplat, and SAGA, our method does not require any alteration to the fitting process and, therefore, has shorter fitting time. While the segmentation time is higher for GaussianCut, it still has much shorter overall time requirement. All reported times have been obtained using NVIDIA RTX 4090 GPU and an Intel Core i9- 13900FF CPU.
Table 9: Comparison of segmentation time (in seconds) on the NVOS benchmark.
Method
Preprocessing time
Fitting time
Segmentation time
Performance (loU)
SAGA [6]
71.17 +: 22.74
1448.50 i 205.07
0.35 0.05
90.9
Gaussian Grouping [55]
13.72 1: 2000.34 :: 1922.19
2096.07 251.96
0.55 0.09
90.6
LplaSplat [44]
1346.92
247.00
0.82 0.02
74.0
Coarse Splatting (Ours)
6.11 + 0,38
510.97 = 106.42
19.48 4: 4.31
91.2
Gaussian (Ours)
6.11 + 0.38
510.97 106.42
88.77
33.68
92.5
While 3DGS has a higher footprint than NeRF-based models, several recent works reduce the memory footprint with limited loss of quality [12, 38-40J. Our method only stores one additional parameter Wy for every Gaussian and is less memory-intensive than methods requiring learning feature field [6, 13].
Memory requirement:
GaussianCut can address some inaccuracies in 2D video segmentation models, but it may still lead to partial recovery when the initial mask or tracking results significantly off (Figure 7). By segmenting directly at the level Gaussians, the resulting segmentation boundaries may inherit inaccuracies of the underlying representation that are not visible in the unsegmented scene due to bleh--b.endig.. While GaussianCut does not require additional scene optimization, our method can still take up a few minutes for the graph cut component, which makes the segmentation not real-time. The implementation could be improved by applying graph cut on a subset of Gaussians. We leave this to future work. Additionally, extending energy function include feature similarity term (in equation 3) is another potential improvement. We also discuss some failure cases in section B.
Limitations:
7 Conclusion
In this paper, we introduce GaussianCut, a novel approach that taps into the underlying explicit representation of 3D Gaussian Splatting to accurately delineate 3D objects. Our approach takes an optimized 3DGS model along with sparse user inputs on any viewpoint from the scene. We use video segmentation models to propagate the mask along different views and then track the Gaussians that splat to these masks. In order to enhance the precision of partitioning the Gaussians, we model them as nodes in an undirected graph and devise an energy function that can be minimized using graph cut. Our approach shows the utility of explicit representation provided by 3DGS and can also be extended for downstream use cases of 3D editing and scene understanding.
8 Acknowledgment
This work was supported by the Sony Research Award Program and NSERC Alliance.
10

---

References
[1] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.
Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022.
[2] Yuri Boykov and Vladimir Kolmogorov. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. TPAMI, 2004.
[3] Yuri Boykov, Olga Veksler, and Ramin Zabih. Fast approximate energy minimization via graph cuts. TPAMI, 2001.
[4] Yuri Y Boykov and M-P Jolly. Interactive graph cuts for optimal boundary & region segmenta- tion of objects in nd images. In ICCV, 2001.
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[6] Jiazhong Cen, Jiemin Fang, Chen Yang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, and Qi Tian. Segment any gaussians. arXiv, 2023.
[7] Jiazhong Cen, Zanwei Zhou, Jiemin Fang, Wei Shen, Lingxi Xie, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, et al. Segment anything in 3d with nerfs. NeurIPS, 2023.
[8] Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang, Xiaofeng Yang. Yikai Wang. Zhongang Cai, Lei Yang, Huaping Liu, and Guosheng Lin, Gaussianeditor: Swift and controllable 3d editing with gaussian splatting, In CVPR, 2024.
[9] Yangming Cheng, Liulei Li, Yuanyou Xu, Xiaodi Li, Zongxin Yang, Wenguan Wang, and Yi Yang. Segment and track anything. arXiv, 2023.
[10] Özgün Çięek, Ahmed Abdulkadir, Soeren Lienkamp, Thomas Brox, and Olaf Ronneberger. 3d u-net: learning dense volumetric segmentation from sparse annotation. In MICCAI, 2016.
[11] Bin Dou, Tianyu Zhang, Yongjia Ma, Zhaohui Wang, and Zejian Yuan. Cosseggaussians: Compact and swift scene segmenting 3d gaussians. arXiv, 2024.
[12] Zhiwen Fan, Kevin Wang, Kairun Wen, Zehao Zhu, Dejia Xu, and Zhangyang Wang. Light- gaussian: Unbounded 3d gaussian compression with 15x reduction and 200+ fps. arXiv, 2023.
[13] Pedro F Felzenszwalb and Daniel P Huttenlocher. Efficient graph-based image segmentation. IJCV, 2004.
[14] James D Foley. Computer graphics: principles and practice, volume 12110. Addison-Wesley Professional, 1996.
[15] Lester Randolph Ford and Delbert Ray Fulkerson. Flows networks. Princeton university press, 2015.
[16] Sara Kridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa, Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
[17] Alberto Garcia-Garcia, Sergio Orts-Escolano, Sergiu Oprea, Victor Villena-Martinez, and Jose Gaddia-Rodriguez. A review on deep learning techniques applied to semantic segmentation.
arXiv, 2017.
[18] Rahul Goel, Dhawal Sirikonda, Saurabh Saini, and PJ Narayanan. Interactive segmentation of radiance fields. In CVPR, 2023.
[19] Andrew V Goldberg and Robert E Tarjan. A new approach to the maximum-flow problem. JACM, 1988.
[20] Leo Grady. Random walks for image segmentation. TPAMI, 2006.
[21] Haoyu Guo, He Zhu, Sida Peng, Yuang Wang, Yujun Shen, Ruizhen Hu, and Xiaowei Zhou. Sam-guided graph cut for 3d instance segmentation. arXiv, 2023.
11

---

[22] Xu Hu, Yuxi Wang, Lue Fan, Junsong Fan, Junran Peng, Zhen Lei, Qing Li, and Zhaoxiang Zhang. Semantic anything 3d gaussians. arXiv, 2024.
[23] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023.
[24] Justin Kerr, Chung Min Kim, Ken Goldberg. Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In ICCV, 2023.
[25] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Wan-Yen Lo, et al. Segment anything. arXiv, 2023.
[26] Arno Knapitsch, Jaesik Park, Qian- Yi Zhou, and Vladlen Koltun. Tanks and temples: Bench- marking large-scale scene reconstruction. ToG, 2017.
[27] Sosuke Kobayashi, Eicchi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. NeurIPS, 2022.
[28] Christoph Lassner and Michael Zollhofer. Pulsar: Efficient sphere-basee based neural rendering. In CVPR, 2021.
[29] Kunhao Liu, Fangneng Zhan. Jiahui Zhang, Muyu Xu, Yingchen Yu, Abdulmotaleb El Sad- dik, Christian Theobalt, Eric Xing, and Shijian Lu. Weakly supervised 3d open- ppe-boocbbllary segmentation. NeurIPS, 2023.
[30] Quande Liu, Youpeng Wen, Jianhua Han, Chunjing Xu, Hang Xu, and Xiaodan Liang. Open- world semantic segmentation via contrasting and clustering vision-language embedding. In ECCV, 2022.
[31] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv, 2023.
[32] Yichen Liu, Benran Hu, Junkai Huang, Yu- Yu-Wing Tai, and Chi-Keung Tang. Instance neural radiance field. In ICCV, 2023.
[33] Yichen Liu, Benran Hu, Chi-Keung Tang. and Yu- WunWing Tai. Sanerf-hq: Segment anything for nerf in high quality. arXiv, 2023.
[34] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. TOG, 2019.
[35] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106 2021.
[36] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 3DV, 2016.
[37] Ashkan Mirzaei, Tristan Lumentado-Ammstrong Konstantinos Derpanis, Jonathan Kelly, Marcus A Brubaker, Igor Gilitschenski, and Alex Levinshtein. Spin-nerf: Multiview segmen- tation and perceptual inpainting with neural radiance fields. In CVPR, pages 20669-20679, 2023.
[38] KL Navaneet, Kossar Pourahmadi Meibodi, Soroush Abbasi Koohpayegani, and Hamed Pirsi- avash. Compact3d: Compressing gaussian splat radiance field models with vector quantization. arXiv, 2023.
[39] Simon Niedermayr, Josef Stumpfegger, and Rüdiger Westermann. Compressed 3d gaussian splatting for accelerated novel view synthesis. arXiv, 2023.
[40] Panagiotis Papantonakis, Georgios Kopanas, Bernhard Kerbl, Alexandre Lanvin, and George Drettakis. Reducing the memory footprint of 3d gaussian splatting. PACMCGIT, 2024.
12

---

[41] Bo Peng, Lei Zhang, and David Zhang. A survey of graph theoretical approaches to image segmentation. Pattern Recognition, 2013.
[42] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, 2017.
[43] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. NeurIPS, 2017.
[44] Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d language gaussian splatting. arXiv, 2023.
[45] Zhongzheng Ren, Aseem Agarwala, Bryan Russell, Alexander G Schwing, and Oliver Wang. Neural volumetric object selection. In CVPR, 2022.
[46] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. "grabcut" interactive foreground extraction using iterated graph cuts. ACM transactions on graphics (TOG), 23(3):309-314, 2004.
[47] Myrna C Silva, Mahtab Dahaghin, Matteo Toso, and Alessio Del Bue. Contrastive gaussian clustering: Weakly supervised 3d scene segmentation. arXiv, 2024.
[48] Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv, 2021.
[49] Wenming Tang and Guoping Qiu. Dense graph convolutional neural networks on 3d meshes for 3d object segmentation and classification. Image and Vision Computing, 2021.
[50] Vadim Tschernezi,, Iro Laina, Diane Larlus, and Andrea Vedaldi. Neural feature fusion fields:
3d distillation of self-supervised 2d image representations. In 3DV, 2022.
[51] Zhuowen Tu. Auto-context and its application to high-level vision tasks. In CVPR, 2008
[52] Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mehdi SM Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth. Nesf: Neural semantic fields for generalizable semantic segmentation of 3d scenes. arXiv, 2021.
[53] Zhou Wang, Alan C Bovik, Hamid Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. 77P, 2004.
[54] Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwa- janakorn. Nex: Real-time view synthesis with neural basis expansion. In CVPR, 2021.
[55] Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke. Gaussian grouping: Segment and edit anything in 3d scenes. arXiv, 2023.
[56] Lin Yen-Chen, Pete Florence, Jonathan T Barron, Tsung- Yi Lin, Alberto Rodriguez, and Phillip Isola. Nerf-supervisiin: Learning dense object descriptors from neural radiance fields. In ICRA, 2022.
[57] Nida M Zaitoun and Mushah Aqel. Survey on image segmentation techniques. Procedia Computer Science, 65:797-806, 2015.
[58] Chaoning Zhang, Fachrina Dewi Puspitasari, Sheng Zheng, Chenghao Li, Yu Qiao, Taegoo Kang, Xinru Shan, Chenshuang Zhang, Caiyan Qin, Francois Rameau, et al. A survey segment anything model (sam): Vision foundation model meets prompt engineering. arXiv, 2023.
[59] Hao Zhang, Fang Li, and Narendra Ahuja. Open-nerf: Towards open vocabulary nerf decompo- sition. In WACV. 2024.
[60] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features perceptual metric. In CVPR, 2018.
13

---

[61] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison. In-place scene labelling and understanding with implicit scene representation. In ICCV, 2021.
[62] Kaichen Zhou, Lanqing Hong, Enze Xie, Yongxin Yang, Zhenguo Li, and Wei Zhang. Serf: Fine-grained interactive 3d segmentation and editing with radiance fields. arXiv, 2023.
[63] Shijie Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang, and Achuta Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting enable distilled feature fields. arXiv, 2023.
[64] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li. Jianfeng Wang, Lijuan Wang. Jianfeng Gao, and Yong Jae Lec. Segment everything everywhere all at once. NeurIPS, 2024.
14

---

A Implementation details
We run our segmentation algorithm on 3D Gaussian Splatting representations, following the code provided by Kerbl et al. [23]. All scenes optimized for 30,000 steps using the default parameters. We use SAM-Track [9] as the video segmentation model.
For the datatset used in NVOS [45], we use the provided reference image with the user scribbles for a fair comparison. For the SPIn-NeRF dataset [37], we use the first image in the directory as reference image for the user input. The scenes reported throughout the paper are selected from the following datasets:
NVOS (LLFF [34] subset): flower, fortress, fern, horns, orchids, trex, leaves SPIn-NeRF (collection from some widely used datasets [16, 26, 34, 35, 56]): orchids, leaves,
fortress, horns, truck, lego bulldozer
Shiny [54]: giants, tools, seasoning, pasta
Mip-NeRF [1]: cycle, garden, bonsai
LERF [24]: figurines
3D-OVS [29]: lawn, sofa, bed, bench, room
The Gaussians optimized for 3DGS can have ambiguous structures as they not geometrically constrained. When partitioning the Gaussians as foreground or background, the boundary Gaussians can appear as having shard-like artifacts (or "spiky" Gaussians). Since the goal of this work is to effectively characterize a Gaussian as foreground or background, we render the foreground mask by overriding the colors of background Gaussians. To generate object assets, our algorithm can be combined with Gaussian decomposition based approached [22].
Mask evaluation:
A.1
Baseline implementation details
A.1.1 Gaussian Grouping
Gaussian Grouping [55] learns additional feature per Gaussian that can be used group Gaussians belonging to the same object. We use SAM-Track to get all the 2D segmentation masks. While the default implementation of Gaussian Grouping DEVA [9] masks,
chose SAM-Trace for both GaussianCur and Gaussian Grouping to maintain consistency in mask quality across the methods. Similar GaussianCut, However, Gaussian Grouping uses a mask for segmenting everything and can, therefore, sometimes
clicks to segment objects in NVOS and SPin-NeRF benchmarks.
Evaluation image
Ground-truth mask
LangSplat score map
Evaluation image
Ground-truth mask
LangSplat score map
Figure 5: Limitation of Splat on Trex and Leaves scenes from NVOS benchmark. Parts of the trex can not be extracted the top row. the bottom row, background leaves are also selected along with front leaf.
15

---

produce over-segmented object segments. To handle such cases and obtain the final segmentation, we aggregate all the segments that constitute the object. We also use the same number of 2D segmented masks as the number of training views. In contrast, for GaussianCut, we limit the number of masks to for front-facing scenes and 40 for 360 inward-facing scenes. To prevent memory issues in Gaussian Grouping, we restrict the total number of Gaussians across all scenes to 2M.
A.1.2 LangSplat
We obtained all 2D features at feature level "part" for LangSplat [44]. Since could not use clicks and scribbles to obtain the segment, we have used text queries. We tried multiple text queries for each scene and reported the results on the best performing query. For certain scenes, text queries can constrain the selection of an object. For instance, in Figure 5, multiple instances the leaves get a high relevancy score when segmenting the front leaf.
B Limitations
Iepaion
poidin
Figure 6: Trex scene from the LLFF [34] dataset. Left: reference image with scribbles provided by NVOS [45]. Center: segmentation mask provided by SAM [25]. The obtained mask misses finer details and also groups multiple intricate features together. Right: segmented using GaussianCut. While it adds finer details (like near the ribs), the tail still contains some background elements.
The performance of our method depends on the robustness of image and video segmentation models. For all the scenes tested, we do not tune SAM-Track and use the default settings. SAM-Track (built on SAM) can provide coarse segments, even on the reference image, especially for irregular scenes, as shown in Figure GaussianCut improves the segmentation details of SAM but there still remains scope for improving the segmentation performance further for the more intricate patterns.
Similar image-segmentation models, video-segmentttiion models can also have inaccurate segmen- tation masks. This issue is more pronounced in complex 360 scenes, where an object can entirely
Input image
Propagated segmentation mask using video segmentation model www.eeeeennnnnn..
Initial segmentation mask
Segmented object from corresponding viewpoints
Figure 7: SAM-Track fails to capture major sections of the bicycle when its orientation significantly deviates from the initial position. Even in the reference image, the segmentation mask omits finer details such the bicycle wheel rims, pedals, and bottle holder. GaussianCut improves segmentation by eliminating substantial portions of the bench to isolate the bicycle, and it partially restores the visibility of the wheel rims. Despite these improvements, the segmentation remains imprecise.
16

---

Input image
Propegated segmentatin mask using video segmentation model
Initial segmentation mask
Segmented object from corresponding viewpoints
Figure 8: GaussianCut precisely retrieves fine details, such the mirrors the front of the truck, even in instances where video-segmentation model struggles to maintain consistency across different views in the scene.
change its orientation, which can lead the trackers to fail in segmenting all views effectively. We illustrate two instances in Figure 7 and 8, where GaussianCut corrects the inaccuracies SAM-Track with varying levels of effectiveness.
Reference image
Bonsai
Initial mask
Reference image
Garden
Initial mask
Reference image
Figurines
Initial mask
Figure 9: Visualization of selected objects the Mip-NeRF and LERF dataset. Initial object
selection, based on point clicks, and the reference image is shown on the left.
C More visualization results
present additional segmentation visualizations for 360 inward scenes taken from Mip-NeRF [1] and LERF datasets [24] in Figure 9. GaussianCut segments complex and irregular features, including
17

---

the leaves and wire in the bonsai lego, the detailed decorations of the plant and table in the garden scene, well as the cord, viewfinder, and flashbulb of the vintage camera.
D
Additional results
D.1 Shiny dataset
Reference Image
SA3D
Ours
Ensunkong
Figure 10: Qualitative results on the Shiny dataset, compared against SA3D [7]. The points used as user inputs highlighted in the reference image.
Scenes in Shiny [54] dataset have complex shapes of objects (pasta), are placed in more cluttered environments (tools), or possess a subtler distinction between the foreground and background colors (giants). We test four scenes from the Shiny dataset and label four images from each scene as ground-truth. Table 4 shows the improvement of GaussianCut against coarse splitting and SA3D [7] with an overall +0.7% and +1.7% absolute gain in foreground mloU, respectively. Qualitative results from the dataset are shown Figure 10. GaussianCut can retrieve fine details (like the strands of the pasta) more accurately.
18

---

D.2 SPIn-NeRF
Table 10 shows the performance of GaussianCut on each scene of the SPIn-NeRF dataset. Further- more, we show in Figure 11 that the quality of the mask produced by GaussianCut contains finer details than the ground-truth labels from SPIn-NeRF.
(a) Image
(b) SPin-NeRF
(c) Ours
(d) Segment
Truck
Pinecone
Orchids
Lego
Figure Qualitative comparison of segmentation masks obtained from GaussianCur and the ground- truth used in SPIn-NeRF dataset.
19

---

Table 10:
Quantitative results on each scene in the SPIn-NeRF dataset.
Scenes
MVSeg [37]
ISRF [18]
SA3D [6]
SAGD
Gaussian Grouping [55]
LangSplat [+4]
GaussianCu)
Orchids
92.7
84.2
85.4
89.4
88.2
87.8
Leaves
94.9
87.2
97.2
87.9
96.7
46.5
96.3
Room
95.6
67.6
88.2
86.5
96.2
64.5
95.2
Horns
92.8
84.3
92.5
87.1
93.6
Fortress
97.7
92.6
98.3
96.6
97.8
95.8
97.7
Fork
19.9
90.0
70.1
85.4
Pinecane
93.4
60.0
92.9
92.0
92.2
54.4
91.9
Truck
85.2
79.2
90.8
93.0
95.9
53.8
95.7
Lego
89.2
90.9
71.5
92.4
89.7
88.4
69.5
92.9
D.3 3D-OVS
To test the performance our method using text queries, we test on the 3D-OVS [29] dataset and compare it against Gaussian Grouping [55], LangSplat [44], and Contrastive Gaussian Clustering [47] in Table 11. We use the grounding-DING integration in SAM-Track to obtain the initial segments. The baseline numbers reported in Table
different text query for some objects than [47]. This was done to ensure that we have a decent initial mask from SAM-Track the goal of our work is not to improve language understanding 3D models.
taken from [47]. We
D.4 Qualitative comparison with 2D segmentation model
The objects segmented by GaussianCut exhibit fine details, as depicted in the masks presented in Figure 12. Although our method uses SAM predictions initial mask, the segregation of Gaussians provides information with greater precision compared to SAM alone.
E
Additional sensitivity analysis and ablations
E.1
Binary weights for coarse splatting
As mentioned in Section 3.3, the likelihood term Wy for each Gaussian g is obtained by taking weighted ratio of g's contribution on the masked pixels compared to the total number pixels affects. Instead of using weighted assignment, either contributes to a foreground pixel or doesn't 't. For the n viewpoints, I == {P) that have corresponding masks M == {M})_T
also have hard binary assignment where
E, EpEn 1(T 0)
Wlg = Σ3Epe I(T2) > 0)
(6)
which reflects the ratio of the number of pixels that has contributed to MD and P. As shown in Table 12, since soft assignment has marginally better performance, it is our default implementation.
E.2 Sensitivity of hyperparameters
We share a default setting in section which performs reasonably well on all our datasets. The sensitivity of each parameter can be very scene-dependent For instance, in a scene where parts of an object have different colors, a very high weight on the color similarity can affect adversely. We show the effect of À (controls the pull of neighboring terminal edges) and y (decay constant of the similarity function) on two scenes Table 13 and Table 14, respectively. The reported metric is IoU.
E.3 Threshold of coarse splatting
For the four 360-degree inward scenes in the SPIn-NeRF benchmark, show sweep of the threshold (default is 0.3 our implementation) used for Coarse Splatting. GaussianCut outperforms all the thresholds considered for coarse splatting shown in Table 15.
20

---

(a) RGB
(b) SAM
(c) Ours
Garden
Bonsai
Truck
Figure 12: Visualization of segmentation masks from SAM and GaussianCut.
21

---

Table 11: Quantitative evaluation on 3D-OVS [29] dataset.
CGC refers to Contrastive Gaussian
Clustering method.
Item
Gaussian Grouping [55]
LangSplat [44]
CGC [47]
GaussianCut (Ours)
Banana
17.8
Leather Shoe
97.6
71.4
97.9
96.4
Camera
95.3
3.3
85.0
95.3
Bed
Hand
96.8
6.9
95.5
94.0
Red Bag
98.5
81.2
98.1
98.8
White Sheet
99.0
25.5
99.0
98.9
Average
34.3
Wall
98.7
42.6
98.7
97.7
Wood
97.8
88.0
98.0
96.2
Egg Tart
95.4
87.4
97.2
93.2
Bench
Orange Cat
94.5
96.4
97.3
97.6
Green grape
0.0
93.7
95.5
94.6
Offrrod car
93.7
95.2
36.0
92.1
Average
73.7
84.8
96.1
95.4
Wall
99.4
24.3
43.6
99.1
Chicken
0.0
94.7
91.4
91.2
Room
Basket
87.9
98.3
97. .4
96.3
Rabbit
96.3
45.3
97.0
93.8
Dinosaur
92.7
6.8
94.9
93.1
Basehall
97.5
68.4
97.4
95.6
Average
79.0
56.3
86.9
94.9
Pikachu
48.8
89.6
0.0
94.3
UNO cards
95.8
79.6
95.4
95.5
Nintendo switch
90.9
89.8
92.6
93.7
Sofa
Gundam
16.1
79.5
Xbox controller
$9.9
67.8
Sofa
97.4
0.0
44.0
98.0
Average
68.1
67.7
67.5
95.0
Apple
96.3
94.0
93.8
88.8
Cap
98.4
98.4
97.9
92.0
Lawn
Stapler
94.7
96.2
95.6
88.1
Headphone
94.5
91.7
70.2
84.2
Hand soap
96.0
95.2
93.7
91.3
Lawn
99.2
99.5
99.3
94.1
Average
96.5
95.8
91.8
89.8
Table 12: Comparison of soft and hard weight assignment of wg-
Scene
Soft assignment
Hard assignment
Fern (NVOS)
83.06
82.56
Fortress (NVOS)
97.97
98.12
Leaves (N (NVOS)
95.95
95.60
Lego (SPIn-NeRD)
89.18 18.18
88.95
Pinecone (SPIn-NeRF)
91.89
91.99
E.4 How important are the 2D segmentation masks?
In order to understand the extent to which the performance of our model depends on the initial 2D segmentation mask, we do the masked rasterization with just scribbles, a single mask, and multi- view masks. Figure 13 shows the segmentation result of Coarse Splatting and GaussianCut. The effectiveness of GaussianCut is heightened further when the initial segmentation mask is sparse. Table 16 also shows the performance improvement when running GaussianCut directly scribbles.
22

---

Table 13: Performance comparison for differ- ent À values.
Table 14: Performance comparison for differ- ent values.
Scene
0.5
À 4
Scene
7 -= 0.5
7
1
7
4
Fortress
97.67
97.99
97.95
97.80
Fortress
96.12
97.95
97.56
96.04
Lego
89.15
89.18
89.18
88.49
Lego
89.20
89.18
89.18
89.19
Table 15: Coarse splatting bascline with different thresholds.
Threshold
IoU
Acc
Coarse@O..
88.47 + 4.85
98.96 + 0.53
Coarse@O:: 0.3
89.67 = 3.18
98.94 0.72
Coarse 0.5
87.76 + 3.06
98.45
1.50
Coarse @ 0.7
83.30 + 6.04
97.58 2.84
Coarse@ 0..9
72.13 11.26
96.08 4.69
GaussianCut
90.55 = 3.76
99.18 + 0.41
Soriables
Input
Coarse Splatting
Ground Truth
CauRs Cur
Figure 13:
We compare coarse splatting (w/o graph cut) and GaussianCut. Scribbles refer to using direct input, single mask refers to taking the mask from one viewpoint, and multi-view masks refer to using video segmentation. The effectiveness of GaussianCut becomes more prominent when the inputs are sparse.
Table 16: Segmentation performance with just user seribbles for NVOS scenes.
Scene
Scribbles
Scribbles (with graphcut)
GaussianCut
Fern 8.17
47.97
83.06
Flower
7.48
85.30
95.37
Fortress
15.12
95.67
97.95
Trex
6.74
50.44
83.43
Orchids
6.17
85.25
95.80
23

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The introduction clearly lists the contribution of this work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have written about the limitations in the main paper. We have also included few failure cases in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]

---

Justification:
Our method does not have any theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, we mention all the hyperparameters and the off-the-shelf models have used. When building on another codebase, we use their default settings (or mention explicitly if something is changed).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
25

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Code is available at: http:/hhwww.ooommmaaaaaaaaia///a
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips.coo ctsaps.ccppublcc/// http//uublcggguudeeeee guides/ /CodeSubmi ssi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we list it in the implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Factors for stochasticity are less.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
26

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Provided in the discussion section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //wurri//////..eee/eee
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "Lp0Kqz6Ykn", "reviewer": "Reviewer_Qza2", "summary": "The paper presents a method for segmentation in 3D Gaussian Splatting. Given a prompt in 2D, the proposed approach is capable of segmenting objects of interest from the 3D Gaussians. Specifically, the method first performs 2D segmentation in all training views, which are then propagated into 3D using a technique similar to visual culling. This achieves a coarse 3D segmentation in the initial stage. Subsequently, the method employs graph cuts to refine the 3D segmentation. The paper validates the method on multiple datasets using 2D segmentation metrics and demonstrates the potential of the proposed approach.", "questions": "Please address the concerns that I described above. During rebuttal, I would expect author justify better the proposed methods according to my suggestions I made above. Before, I currently vote for reject.", "limitations": "Yes, they did it well.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper proposes refining 3D segmentation via graph cuts using carefully-designed energy functions. This step is practical and effective. \n\nIt’s surprising to see that the method, even without 2D mask supervision, can still outperform baselines like LERF.", "weaknesses": "The paper has several weaknesses as below: \n\nWriting: \n- The method is not well-motivated in the introduction section. Specifically, in lines 24-39, the problem defined in the paper is hard to understand. The authors might want to rewrite this section to make the motivation clearer.\n- Part of the implementation is not clear to me. Specifically, I’m uncertain if the method is training-free for segmentation when provided with a pretrained GS model. If so, it would be beneficial to make this clear, as being training-free is a significant strength compared to other methods.\n\nMethod: \n- I’m not entirely convinced by the training-free approach in the proposed method. In my view, learning a per-Gaussian feature for segmentation offers more flexibility. Therefore, I suggest the authors justify their method by demonstrating what the proposed method can achieve that learnable features cannot.\n\nExperiments: \n- The important comparison between baselines that use 3DGS such as GaussianEditor and LangSplat. The paper only reports the baselines that use NeRF, which is generally worse than 3DGS in the segmentation task. In other words, the baselines are insufficient to justify the proposed method.\n- Both qualitative and quantitative results of 3D segmentation are missing in the result section. From my viewpoint, it’s essential to this paper since the proposed method aims to perform 3D segmentation. While 2D segmentation partially shows the performance of the proposed method, it’s not the correct metric to validate the proposed method. The paper should not avoid this evaluation because 3D GT segmentation is missing in the datasets used in the paper. In this case, I would suggest the author trying some other datasets like ScanNet where 3D GT is available.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for their feedback on our paper. We thank the reviewer’s suggestion for helping to improve the clarity of our work \n\n> Clearer motivation for the introduction \n\nOur work is motivated by leveraging the explicit nature of 3DGS representation. With user inputs and the underlying geometry of the scene captured by the Gaussians, objects can be segmented from a scene, without requiring any additional optimization of the 3DGS model. We provide a clearer motivation below and we will include the changes in the revised version of the paper.\n\n3D Gaussian Splatting represents a scene using a set of Gaussians, thereby offering an explicit representation of the scene. Prior works in 3DGS scene segmentation involve augmenting the set of Gaussian with a per-Gaussian feature, that is optimized during the Gaussian fitting, supervised by 2D features. These features provide semantic information and can be used for segmentation. Since 3DGS stores the parameters of each Gaussian explicitly, the size of the feature embedding exacerbates the already high memory footprint of the method.  Therefore, such methods have relied on learning low-dimensional features per Gaussian. While this enables a 3D consistent segmentation, optimizing a per-Gaussian feature significantly increases the fitting time of the Gaussians to the scene. Our proposed methodology is a post hoc technique that operates directly on an optimized 3DGS field without requiring any segmentation-aware fitting. We directly tap into the representation and map each Gaussian to its corresponding object(s). We do this by formulating the Gaussians in a scene as an undirected graph and partitioning the set of Gaussians allowing for the extraction of subsets that represent specific objects.\n>  Is the method training-free for segmentation when provided with a pretrained GS model?\n\n\nYes, our approach is training-free when provided with a pre-trained GS model. We have highlighted this in the abstract and we kindly refer the reviewer to Figure 1 which shows that GaussianCut operates directly on a pre-trained GS model and user inputs. We will also modify our writing to explain this more clearly. To summarize our implementation, we take an already optimized GS model with user inputs on any one image. The user input is processed into dense masks and the masks are also propagated to multiple images using a video segmentation model. We do a \"coarse splatting\", which assigns each Gaussian a likelihood ratio, which is the fraction of pixels splatted in the foreground (as per the 2D mask) and the total number of pixels splatted by that Gaussian. This step does not require any further GS optimization. It is computed by simply rasterizing each view that has a corresponding 2D segmentation mask. Finally, we formulate a graph from the Gaussians that uses this additional likelihood term and the inherent properties of the Gaussians already learned in the initial fitting. \n\nAs mentioned by the reviewer, this is indeed significant as it provides segmentation without requiring any changes to the fitting process. As noted in Table 2 (global response), it also saves substantial time compared to optimizing features.\n\n> Comparison with other 3DGS and per-Gaussian feature baselines\n\nWe thank the reviewer for suggesting additional baselines. We would like to clarify that we do provide 3DGS-based baselines. We provide SAGA[a] and SAGD[b] results on the LLFF and SPIn-NeRF datasets (Tables 1, 3, 9), both of which are based on 3DGS. We also compare against Gaussian Grouping, LangSplat, and CGC (all of which are based on 3DGS) in Table 10 (appendix). We had not compared against them on all datasets because our method is focused more on interactive segmentation and these methods propose learning a feature field for all the scene elements (a more detailed distinction between these baselines and our method is provided in the global response). \nBased on the feedback, we have reported Gaussian Grouping and LangSplat comparison on the NVOS benchmark in Table 1 (global response).\n> 3D evaluations missing\n\nWe completely agree with the reviewer that 3D GT is indeed a better metric to evaluate 3DGS segmentation. As noted by the reviewer, most datasets and approaches do not provide such ground-truth and benchmarks for this evaluation. SPIn-NeRF, Shiny, and 3D-OVS, while not 3D consistent, provide masks for multiple views to show the efficacy of methods to some extent. \nObtaining the ScanNet dataset requires approval from the authors, which could take up to a week. Since we did not have that much time and the 3DGS baselines (SAGA, LangSplat, Gaussian Grouping) have a larger runtime, especially for scenes with a higher number of images, we could not include the quantitative results for the rebuttal. We will include results from Replica or ScanNet in the final paper. \n\n[a] Segment Any 3D Gaussians. arXiV, 2023 \\\n[b] SAGD: Boundary-Enhanced Segment Anything in 3D Gaussian via Gaussian Decomposition, arXiV, 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0YsnTZYEaY", "reviewer": "Reviewer_BR3k", "summary": "This paper proposes GaussianCut for interactive 3D segmentation. The GaussianCut takes a trained 3DGS model and the user prompt as inputs. The SAM model first transfers the user prompt into an initial mask. Then the 3DGS model renders multiple view images and an existing video-tracking model is used for segmenting 2D masks across multiple views. With the masks on multiple views, the splatted Gaussians are identified with two likelihood parameters. Then the graph-cut method is applied for the Gaussian points where each Gaussian is a node, and the edge models the foreground and background relations. Results on multiple benchmarks show the effectiveness of the proposed method.", "questions": "The novelty of the proposed method should be justified. The proposed method is more like a post-processing. Some important method details and comparisons are missing.", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is well-written and easy to follow.\n- Using Graph Cut to segment the 3D Gaussians makes sense and sounds interesting.\n- The result on multiple benchmarks shows the proposed method achieves new SOTA performance.", "weaknesses": "- The proposed method is kind of straightforward. The framework of GaussianCut is a combination of all existing models. The SAM model is used to obtain masks from user prompts. The SAM-Track model is used to generate Multiview masks. 3DGS model can explicitly model relations between 3D Gaussians and 2D pixels. Graph Cut model is used to separate the Gaussian points. This paper should explain the key contributions of GaussianCut itself.\n\n- Details of associating 3D Gaussians with 2D masks should be given. As alpha blending will assign different weights to different Gaussians for a single pixel, For the Gaussians splatted into one pixel, are they assigned different weights just for this pixel? Or a hard (binary) assignment is used?\n\n- The time cost of the proposed method is shown in Tables 6 and 7. However, the comparisons with other SOTA methods in terms of time cost should be included to illustrate the speed advantage/disadvantage of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive review. We would like to respond to the questions and concerns that they have posed.\n\n> Key contributions clarification\n\nWe kindly refer the reviewer to the global response which provides a detailed explanation of our key contributions.\n\n> Details of associating 3D Gaussians with 2D masks\n\nGaussians are assigned coarse estimates based on their weights during alpha-rendering at rasterization, i.e., for the Gaussians splatted onto one pixel, it is assigned a different weight just for this pixel. This weight is based on its contribution at alpha blending. However, we had experimented with binary weights as well and we did not observe any major performance difference. The binary weights also give a similar performance in our experiments. We will include these details about the mapping of masks to the Gaussians in the revised paper. Thank you for the feedback.\n\n> Time cost analysis\n\nFor a detailed speed analysis, we kindly refer the reviewer to Table 2 (global response).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DQesV73plM", "reviewer": "Reviewer_24kC", "summary": "This paper presents GaussianCut, a new method for foreground/background segmentation of 3D Gaussian scenes. GaussianCut relies on 2D image/video segmentation masks, which are generated for a subset of the training images. GaussianCut propagates these masks to the 3D Gaussians by projecting each Gaussian onto each mask and averaging the corresponding mask values. It then assembles the Gaussians into a graph where each Gaussian is connected to its nearest neighbors, defines several energy terms (for nodes and edges), and uses these energy terms alongside a graph cut to partition the Gaussians. Unlike several baselines, GaussianCut can be applied to pre-trained 3D Gaussian scenes without retraining/fine-tuning.", "questions": "- Line 171-172: The authors state that \"Gaussians that map to the same object would be closer spatially.\" This seems reasonable, but isn't always the case. For example, dull specular highlights are often represented via transparent surfaces and \"clouds\" inside objects. Does the proposed method handle these cases well?\n- The proposed method has a number of hyperparameters. How sensitive is it to these hyperparameters?\n- How good is the coarse splatting baseline with well-chosen hyperparameters? I think visualizing a sweep of the threshold in the appendix would convince the reader that the proposed approach works better no matter what threshold is chosen.\n- How are the frames ordered before they are passed to the video segmentation model? How sensitive is the method to this ordering?\n- To what extent does the proposed method's performance rely on SAM-Track's quality? Do any baselines rely on different video segmentation models, and if so, how do the metrics change when those are updated to use SAM-Track?", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- In Figure 4, the qualitative results of the proposed method look much sharper than those of the baselines. Also, the quality metrics for the proposed method improve over the baselines slightly.\n- Figures 5 and 7 make it clear that the proposed method is fairly robust to bad 2D input segmentations.\n- The paper is clear and easy to follow.\n- The proposed method is simple.", "weaknesses": "- One major weakness is that none of the tables include the runtimes of the proposed method and the baselines. This is important because segmentation is arguably much more useful if it runs at interactive speeds. The paper also does not list the runtime of the simpler coarse splatting variant/ablation. I think it is crucial for the paper to list both the training-time overhead and segmentation runtime of each method.\n- Another major weakness is that the paper includes/omits baselines in the tables very inconsistently. For example, SAGD and ISRF are featured in Figure 4, but not in Table 3. On the other hand, MVSeg and SAGA are featured in the table, but not the figure. I would be much more confident in the results if the baselines were always included (when appropriate).\n\nOverall, I would be happy to raise the paper's score if the above two major weaknesses were addressed. My score for contribution is held back by the fact that some of the baselines (e.g., SAGA) are significantly faster to run despite having similar quality.\n\n- Another smaller weakness is that the paper does not include visualizations of the energy terms, high-confidence clusters, etc. These would be very helpful for building the reader's intuition for the method.\n\nMinor nitpicks:\n- Line 55/56: The following sentence should perhaps include citations: \"Recent works have also explored segmentation with Gaussians as the choice for scene representation.\"\n- Line 101: I was confused about what is meant by \"decomposing the boundary Gaussians.\"\n- Line 144: \"Guassians\"\n- Line 169: The graph is defined as $(|\\mathcal{G}|, \\mathcal{E})$. This should be $(\\mathcal{G}, \\mathcal{E})$ instead.\n- Line 170: The definition of the neighborhood is listed as $\\mathcal{N} \\subseteq |\\mathcal{G}| \\times |\\mathcal{G}|$, but the neighborhood is defined in terms of nodes ($\\mathcal{N} \\subseteq \\mathcal{G}$) and not edges (which would be $\\mathcal{N} \\subseteq \\mathcal{G} \\times \\mathcal{G}$ anyways) in the next sentence.\n- Line 264: Units (dB) should be listed for the PSNR differences.\n- Line 296-297: The time cost for segmentation technically grows linearly, but the constant factor is big enough that this doesn't matter. I would update the sentence here to be more precise.\n- The single mask ablation (line 298) should probably be included in Table 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful and constructive feedback.  We have provided clarifications throughout and added additional experiments to address the concerns they raised. We appreciate the reviewers’ attention to detail and will incorporate all the suggestions made by them.\n\n> Runtimes of proposed method and baselines\n\nWe have added all the runtimes suggested by the reviewer and we kindly refer them to Table 2 (global response).\n\n> Inconsistent baselines\n\nWe ran all the baselines on the SPIn-NeRF dataset (10 scenes). For ISRF, we get OOM issues at 1008 resolution for 360-degree inward scenes so we resize it to 1/4th resolution.\n\n| Method  | IoU  | Acc  |\n|--|-|-|\n| MVSeg  | 90.9 | 98.9 |\n| SA3D| 92.4 | 98.9 |\n| SAGD  | 89.7 | 98.1 |\n| SAGA | 88.0 | 98.5 |\n| ISRF| 71.5 | 95.5 |\n| Coarse Splatting  | 91.9 | 98.9 |\n| GaussianCut | 92.9 | 99.2 |\n\nFor the qualitative results in Figure 4, since MVSeg (part of SPIn-NeRF) is designed for inpainting task, extracting the segmentation module was challenging. For SAGA, we run out of memory for the garden scene in RTX 4090 GPU. It's because the scene has considerably more images (185 images, ~4.4 million Gaussians) and SAGA stores a 32-dimensional feature for each Gaussian. Therefore, we were not able to include SAGA in Figure 4. We will also include all the baselines on all the datasets in the final version.\n\n> Visualizations of the energy terms\n\nWe will be happy to provide visualizations that can improve the clarity of our method. However, it is not clear to us how energy function can be visualized. The weights are assigned on edges, not nodes, which makes visualization tougher. Also, since the energy function is not optimized over time (it is minimized using min-cut algorithms), we can not plot the overall energy either. We would like to request the reviewer to provide suggestions on how meaningful visualizations can be included.\n\n> I was confused about what is meant by \"decomposing the boundary Gaussians.\"\n\nSAGD[a] also does not require any segmentation-aware training given an optimized 3DGS model. They decompose Gaussians (split a single Gaussian into two) that may lie at the boundary of an object\n\n> The single mask ablation (line 298) should probably be included in Table 5.\n\nWe provide the ablation with a single mask for the NVOS benchmark below. We will also include this in Table 5 in the revised paper.\n| Method | IoU  |\n|---|--|\n| Single mask | 86.6 |\n| Coarse | 91.2 |\n| GaussianCut | 92.6 |\n\n> \"Gaussians that map to the same object would be closer spatially.\" assumption justification\n\nOur proposed method is scene agnostic, i.e., it works regardless of the distribution of the underlying Gaussians. With the recent advances in point-based and Gaussian-based techniques, these artifacts might be mitigated and since our model works on a pre-trained 3DGS, it can adapt to such advances. This assumption worked well for the all scenes we tested. \n\n> Sensitivity to hyper-parameters\n\nWe share a default setting in the paper which performs reasonably on all our datasets. The sensitivity of each parameter can be very scene-dependent. For instance, in a scene where parts of an object have different colors, a very high weight on the color similarity can affect adversely.  We show the effect of $\\lambda$ (controls the pull of neighboring vs terminal edges)  and $\\sigma$ (decay constant of the similarity function) on two scenes. The reported metric is IoU.\n| Scene| $\\lambda=0.5$ | $\\lambda=1$ | $\\lambda=2$ | $\\lambda=4$ |\n|--|-|-|--|--|\n| Fortress | 97.67 | 97.99 | 97.95 | 97.80 |\n| Lego | 89.15 | 89.18 | 89.18 | 88.49 |\n\n| Scene  | $\\sigma=0.5$ | $\\sigma=1$ | $\\sigma=2$ | $\\sigma=4$ |\n|--|-|-|-|-|\n| Fortress | 96.12 | 97.95 | 97.56 | 96.04 |\n| Lego | 89.20| 89.18 | 89.18 | 89.19 |\n\n> Coarse splatting baseline with well-chosen hyperparameters\n\nFor the four 360-degree inward scenes in the SPIn-NeRF benchmark, we show a sweep of the threshold (default is 0.3). GaussianCut outperforms all the thresholds considered for coarse splatting. It is worth noting that adjusting the threshold of coarse splatting also improves the quality of graph cut (as we directly use these for terminal links weights).\n\n| Threshold  | IoU  | Acc |\n|-|-|-|\n| Coarse@0.1   | 88.47 $\\pm$ 4.85  | 98.96 $\\pm$ 0.53   |\n| Coarse@0.3  | 89.67 $\\pm$ 3.18  | 98.94 $\\pm$ 0.72   |\n| Coarse@0.5 | 87.76 $\\pm$ 3.06  | 98.45 $\\pm$ 1.50   |\n| Coarse@0.7 | 83.30 $\\pm$ 6.04  | 97.58 $\\pm$ 2.84   |\n| Coarse@0.9      | 72.13 $\\pm$ 11.26 | 96.08 $\\pm$ 4.69   |\n| GaussianCut w/ Coarse@0.3  | 90.55 $\\pm$ 3.76  | 99.18 $\\pm$ 0.41   |\n\n>How are the frames ordered before they are passed to the video segmentation model? How sensitive is the method to this ordering?\n\nWe run the camera on a fixed trajectory to obtain rendering from different viewpoints (spiral trajectory for front-facing and round for 360-degree inward scene). We limit the number of frames to 30 for front-facing and 40 for 360-degree scenes. However, all the training images can also be used for coarse splatting (Table 6). Although it might not be preferred for scenes that have a large number of training images. SAM-Track is quite good for unordered multi-frame images as well. The results in Table 6 were obtained after directly giving all training images to SAM-Track. \n\n> To what extent does the proposed method's performance rely on SAM-Track's quality? Do any baselines rely on different video segmentation models, and if so, how do the metrics change when those are updated to use SAM-Track?\n\nAll our experiments are based on SAM-Track and we did not experiment with another model. Our method can work on any video segmentation model and its quality can affect the final performance. Although we start with SAM-Track, our mask quality is improved significantly (Figure 11).\n\n[a] SAGD: Boundary-Enhanced Segment Anything in 3D Gaussian via Gaussian Decomposition, arXiV, 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MzdUS4KFV3", "reviewer": "Reviewer_DeYw", "summary": "The paper proposes GaussianCut for interactive multi-view scene segmentation using 3D Gaussian Splatting. It first accepts user input (clicks, scribbles or text, similar to SAM) on single images, and them aim to segment the corresponding 3D Gaussians. The method constructs a graph based on scene Gaussian, and then using graph-cut algorithm to minimize the energy function. Segmentation tracking method are employed to provide the initial segmentation masks. The experiments are performed on LLFF, Shiny, SPIn-NeRF and 3D-OVS.", "questions": "How will GaussianCut perform when video segmentation/track fail due to large motion? which may lead the coarse segmentation contain large portions of errors.\n\nHow many Gaussians are considered during the graph construction? The most concerning part for me is the tech novelty of the paper.", "limitations": "The limitation is discussed in the paper.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper has a good presentation and clear writing, where figures are clean for the readers to understand the method pipeline.\n\n2. The paper has a good motivation on extracting objects from explicit 3D Gaussians. Graph cut is well known for image segmentation on pixels, and the extension to 3D Gaussians is interesting.\n\n3. The quantitative results in Table 1, 2, 3 and 4 show the reasonable segmentation performance brought by GaussianCut.", "weaknesses": "1. Although extracting objects from explicit 3D Gaussians has a good motivation, this has been studied in previous works like LangSplat [a] and Gaussian Grouping [b], but these two very relevant methods are missing quantitative comparisons in the main paper. It is not clear what are the advantages of the proposed Gaussian Cut comparing to [a] and [b]. In Table 10 (should move to the main paper), [a][b] are compared but the performance of Gaussian Grouping is still on par or even better than the proposed GaussianCut. To understand the paper novelty, the paper should clarify these main differences / advantages in the main paper, and also include the detailed running speed comparison. \n\n2. Using video tracking masks to obtain coarse segmentation is not new, which has been explored in [b, c], but not compared. Since [a][b] both lift SAM's masks to 3D, these 2 methods can also perform click/scribble based 3D segmentation\n\n3. The limited performance improvement of the introduced spatial, color and cluster similarities in Table 5, which show the improvement of the proposed n-links and t-links are minor. Also, the extension of Graph cut from 2d pixels to 3D Gaussians seems very straight-forward.\n\n4.  More comparisons on benchmarks like Replica (setting proposed by Panoptic Lifting [d]) or Lerf-Mask [b]  are desired.\n\n[a] Langsplat: 3d language gaussian splatting. CVPR, 2024.\n\n[b] Gaussian Grouping: Segment and Edit Anything in 3D Scenes. ECCV, 2024.\n\n[c] CoSSegGaussians: Compact and Swift Scene Segmenting 3D Gaussians with Dual Feature Fusion. arXiv, 2024.\n\n[d] Panoptic Lifting for 3D Scene Understanding with Neural Fields. CVPR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for accessing our work and providing valuable feedback. We provide clarifications to the concerns and questions they raise.\n\n> Justify the tech novelty. Graph cut from 2d pixels to 3D Gaussians seems very straightforward.\n\nExtending graph cut from 2D pixels to 3D Gaussians involves several non-trivial design choices and findings. We kindly refer the reviewer to the global response.\n\n> Comparison with LangSplat and Gaussian Grouping and detailed running speed comparison\n\nWe kindly refer the reviewer to Table 1 (global response) for comparison with LangSplat and Gaussian Grouping, and a detailed speed comparison in Table 2 (global response). For the 3D-OVS dataset, we provided a comparison with these two baselines (Table 10). We would like to clarify that the last line of Table 10 is the average over the Lawn scene (not the overall average). Our overall average is much higher than other baselines as shown below. A reason behind this is that Gaussian Grouping and LangSpalt optimize features for all the Gaussians and it therefore limits interactivity with specific objects. Interactivity here refers to the choices made when selecting the 2D supervision mask. Once a supervision mask is chosen, the underlying feature field is optimized based on that. On the other hand, for GaussianCut, the 2D masks are chosen based on the user input. \n\nOverall average on the 3D-OVS dataset. The reported metric is IoU.\n| 3D-OVS  | Gaussian Grouping | LangSplat | CGC   | Ours  |\n|---|----|--|-------|-------|\n| Average IoU | 82.92    | 67.78     | 87.50 | 94.38 |\n\nSince the code for CoSSegGaussians is not public yet, we can’t provide a comparison against that.\n\n> Dependence on video segmentation model\n\nWe use video segmentation models for better performance, but our method can work with just one mask as well (Table 4 and Figure 1 in the rebuttal pdf). While our model performance does suffer when using a single mask, it can still perform reasonable segmentation. Langsplat, Gaussian Grouping, CoSSegGaussians, however,  require a segmentation mask for all training views. \n\n> Limited improvement from proposed n-links and t-links\n\nResults in Table 5 show the average performance on 7 scenes. While the effect of each individual component might not seem a lot on average, the effect can be significant depending on the scene. We show two scenes from the SPIn-NeRF dataset where removing the n-links can have a significant effect. GaussianCut also retrieves fine details (as shown in Figure 4 and 10) which are otherwise missed when just using the semantic maps. The numbers in the table are IoU / classification accuracy.\n| Scene  | Coarse         | w/o n spatial  | w/o n color  | w/o cluster    | GaussianCut     |\n|--------|----------------|----------------|--------------|----------------|-----------------|\n| Lego   | 88.03/99.63    | 88.43/99.66    | 88.52/99.66  | 89.18/99.69    | 89.18/99.70     |\n| Truck  | 93.32/97.76    | 95.47/98.49    | 95.60/98.54  | 95.67/98.57    | 95.70/98.60     |\n\n\n> Comparison on more datasets like Replica and Lerf-mask\n\nWe thank the reviewer for suggesting additional datasets. We show quantitative results on four datasets: LLFF from NVOS, SPIn-NeRF, 3D-OVS, and Shiny, and have also compared performance against SA3D, ISRF, SAGA, SAGD, LangSplat, Gaussian Grouping. In addition, we take scenes from mip-NeRF and LERF (Figure 8) to show qualitative results. If needed, we can provide quantitative results on more benchmarks for the revised paper. \n\n>How will GaussianCut perform when video segmentation/track fails due to large motion? which may lead the coarse segmentation containing large portions of errors.\n\nWe kindly refer the reviewer to Figure 5-7. We have also added extreme case (no video segmentation model) qualitative result in Figure 1 (rebuttal pdf). The quantitative results (IoU / classification accuracy) are shown below:\n\n\n| Scene  | Single Image (Coarse) | Single Image (graphcut) | GaussianCut  |\n|--------|-----------------------|-------------------------|--------------|\n| Truck  | 55.63 / 83.37          | 71.83 / 90.23            | 95.7 / 98.6   |\n| Lego   | 72.92 / 98.88          | 79.98 /99.26            | 89.2 / 99.7   |\n\n\n> How many Gaussians are considered during the graph construction? \n\n\nWe consider all the Gaussians to construct the graph. The number is same as base 3DGS model, ranging from ~850k to 4M for all of our scenes (time taken between 40 sec to 4 mins).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rqiB27DkLw", "reviewer": "Reviewer_MadC", "summary": "The paper proposes a method for interactive multiview segmentation of scenes represented as a set of 3D gaussians (3D gaussian splatting). The method takes as an input multiple views of a scene, constructs 3D Gaussian splatting representation of the scene, takes interaction from the user (scribbles, points, or text prompts), uses a video segmentation method (prior work) to get object masks from user input in each view, and then constructs a graph with nodes corresponding to the gaussians. A binary energy function on this graph is designed which encourages similar gaussians (both in space and color) to be connected with edges of higher weight. Also for each Gaussian there is a likelihood term representing how likely it is to be foreground/background. Finally this energy function is optimized with a graph cut and gives a separation of the cloud of gaussians into two sets: foreground/background. After separation in 2 sets, foreground/background can be visualized separately using the corresponding set of Gaussians. Experimental results outperform prior work.", "questions": "Without the mask refinement method (just from user scribbles), by how much the performance goes down?", "limitations": "Limitations are discussed.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method is well motivated. The energy function makes sense.", "weaknesses": "Computational cost is not real time.\n\nMapping user input to gaussians could be better described. For someone not familiar with 3D Gaussian splatting prior work, it is hard to understand how it works. An illustration could help. \n\nThe cluster similarity in unary terms (Eq. 3) seems to have a big impart on the performance, but is kind of ad-hock not so well motivated. The problem might be a result of 'shortcutting' bias in the energy formulation (cuts separating a small component are cheaper). What you are doing here is kind of reminiscent of saying that pixels close to the foreground scribbles should be more likely in the foreground (if I think of an analogy to 2D segmentation).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback and kind words about our work. \n\n> Computational cost is not real time.\n\nThe reviewer is right in noting that the graph cut algorithm does not run in real-time. However, once segmented into foreground and background Gausisans, it can be rendered in real-time. We kindly refer the reviewer to Table 2 (global response) to see a detailed analysis of pre-processing, training, and segmentation time for our method. Since our method does not perform changes to the 3DGS optimization, the overall time is much less than the baselines. Exploring real-time segmentation is an exciting future direction for this project. \n\n\n> Mapping user input to gaussians could be better described. \n\nIn 3DGS, we can keep track of the Gaussians splatting to a particular pixel in 2D while rasterizing from that viewpoint. So for a particular Gaussian, we can compute the ratio of pixels splatted in the foreground (as per the 2D mask) and the total number of pixels splatted by that Gaussian. This gives us the coarse estimate. Thank you for highlighting this. We will add an equation and an illustration to make this more clear to the reader.\n\n\n> Intuition for the cluster similarity term\n\nThe intuition behind adding a cluster similarity term is to improve the accuracy of foreground identification. If a Gaussian is in a region with other Gaussians that are likely foreground, it might also be foreground. We start with a coarse estimate of each Gaussian's likelihood of being foreground or background, derived from coarse splatting. However, this initial estimate can have errors due to inaccuracies in the 2D segmentation masks.\nNodes with high confidence (weights $w_g$ close to 1) of belonging in the foreground can serve as prototypes for other similar nodes. Directly finding the closest high-confidence node for each Gaussian is computationally expensive. Therefore, we cluster these high-confidence nodes to reduce the computational load. We also present an ablation study on the number of clusters in Table 8. After experimenting with various heuristics, this clustering approach proved to be quite effective.\n\n\n>Without the mask refinement method (just from user scribbles), by how much the performance go down?\n\n\nWe kindly refer the reviewer to Figure 12 which shows this effect qualitatively. If we do not use any 2D mask (just rely on the user scribbles), we only get the area around the scribble from coarse splatting. Applying graphcut has much more significant benefits in this case. We show the quantitative results on five scenes from the LLFF dataset (scribbles following NVOS bnchmark). The numbers in the table are IoU, classification accuracy.\n\n\n| Scene    | Scribbles       | Scribbles (with graphcut) | GaussianCut       |\n|----------|------------------|--------------------------|--------------------|\n| Fern     | 8.17, 74.50      | 47.97, 84.52             | 83.06, 94.60       |\n| Flower   | 7.48, 79.16      | 85.30, 97.61             | 95.37, 98.91       |\n| Fortress | 15.12, 84.15     | 95.67, 99.56             | 97.95, 99.61       |\n| Trex     | 6.74, 88.53      | 50.44, 91.96             | 83.43, 97.83       |\n| Orchids  | 6.17, 84.84      | 85.25, 97.55             | 95.80, 99.31       |\n\n\nThe performance drop is natural as the default implementation uses masks from multiple views but graph cut on 3D Gaussians can still retrieve major parts of the objects even with just the scribbles. We can include more qualitative results in the appendix to show this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their helpful and valuable comments and appreciate that they found our paper well-written, easy to follow (DeYw, 24kC, BR3k), and well-motivated (MadC, DeYw). We are delighted to see that the reviewers recognize the performance improvement of our training-free approach on 3D segmentation benchmarks (Qza2, BR3k, 24kC, MadC), and found the extension of graph cuts to 3D Gaussian Splatting interesting (BR3k, DeYw).

We want to use this general response to clearly highlight our key contributions following the suggestions of reviewers BR3k and DeYw, and differentiate our work from feature-based learning approaches (Qza2, DeYw). We ran additional experiments to compare against feature-based baselines (LangSplat[a] and Gaussian Grouping[b]) following the suggestion of reviewers Qza2 and DeYw and show that our model outperforms these baselines. Additionally, we also show a detailed time analysis in Table 2 (global response) following suggestions from reviewers BR3k and 24kC.  We found the feedback constructive and will happily incorporate changes suggested by the reviewers. 

>  Key contribution (DeYw, BR3k)

Our key contribution is in proposing a method for object selection that is training-free given a pretrained 3DGS model. 
- **Training-free**: Our method is a post hoc technique and unlike prior work on optimization of per-feature Gaussian [a,b], our technique saves significant optimization time (Table 2) and memory  (as we only store one additional parameter per Gaussian). This is not only novel in the sense of a new capability but it is also a surprising result. Given the larger computational budget of training-based approaches, one would expect them to achieve higher performance.
- **Extension of graph cuts to 3D Gaussians**: Our work also contributes to Graph cut-based segmentation research. Although thoroughly explored for image segmentation, the extension to 3D Gaussians is non-trivial and has not been considered in previous work. The energy function proposed in our work contains several non-trivial design choices, including, modeling distances to neighbors (which is typically not considered in images), and n-links and t-links design. 
- **Leverage underlying geometry information**: 2D semantic maps (like SAM), while very robust, can be inefficient at segmenting finer details (Figure 11). Gaussians optimized for a scene capture the geometry of the scene, and our method utilizes these (using the color and position similarity) to retrieve fine details. Feature field-based methods can also miss finer details (e.g., plant decorations in Figure 4).

> Comparison against feature-based methods (Qza2, DeYw)

Per-Gaussian feature optimization baselines[a-c] alter the fitting process of 3DGS by adding an additional attribute for each Gaussian. Below, we compare our approach with feature-based methods
- **Use case**: Feature-based methods learn features for everything in the scene. While useful, it can limit the flexibility of interactivity with a single object. Our method is more flexible in choosing specific object(s) using positive/negative clicks, or scribbles as we generate the 2D masks after the user interaction.
- **Optimization time**: The fitting time of feature-based methods (Table 2 global response) as well as the memory footprint of storing additional features increases significantly, which might not be desirable in all applications. 
- **Reliance on video segmentation model**: Since we use the 2D masks only while rasterization, we do not require masks from all the viewpoints and can also work with just a single mask (Figure 12, Table 4). However, feature-based models require masks for all training views.
- **Complimentary rather than competing**: Rather than seeing feature-based methods as a replacement for our method, we see them as complementary. Our energy function can be modified to also include a feature similarity term in equation 2. We see this as an interesting extension of our work.

Table 1: NVOS results using Gaussian Grouping (GG) and LangSplat (LS). Our approach gives an overall better performance. LangSplat also fails to give good segmentation results for two scenes (Figure 2 of rebuttal pdf).
| Scene | IoU (GG) | IoU (LS) | IoU (ours) | 
|-|-|-|-|
| Horns   | 93.61 | 95.99 | 97.03 |
| Fern     | 87.06  | 83.97|83.07| 
| Orchids  | 86.80 | 96.25| 95.81 |
| Flower   | 95.27  | 95.24 | 95.37 |
| Leaves   | 93.00| 29.26  | 95.96  |
| Fortress | 97.06  | 97.70  | 97.95  |
| Trex     | 81.66  | 19.46  | 83.44  |
| Average  | 90.64   | 73.98   | 92.66  |

> Run time analysis (BR3k, 24kC, MadC)

We compare the run time of our method with [a-d]. We take an average of the run time over 7 scenes from the NVOS benchmark. We divided the run time analysis into 3 stages: pre-processing (this step involves obtaining features for images), fitting time (which involves optimizing the 3DGS/NeRF model), and segmentation time (time between obtaining user prompts and producing segmentation output). Since we operate directly on the pre-trained 3DGS model, our fitting time is significantly lower than other 3DGS-based approaches.

Table 2: Segmentation time (in seconds) on NVOS benchmark
|Method|Preprocessing time|Fitting time|Segmentation time|Performance (IoU)|
|-|-|-|-|--|
|SA3D (NeRF-based)|0|309.14 $\pm$ 18.85|33.89 $\pm$ 13.04|90.3 |
|Gaussian Grouping|13.72 $\pm$ 4.63|2096.07 $\pm$ 251.96|0.55 $\pm$ 0.09|90.6|
|LangSplat| 2000.34 $\pm$ 1222.19|1346.92 $\pm$247.00 |0.82 $\pm$ 0.02 | 74.0|
| SAGA|71.17 $\pm$ 22.74|1448.50 $\pm$ 205.07|0.35 $\pm$ 0.05 |90.9|
| Coarse Splatting| 6.11 $\pm$ 0.38 | 510.97 $\pm$ 106.42 |19.48 $\pm$ 4.31 | 91.2 |
| GaussianCut | 6.11 $\pm$ 0.38 | 510.97 $\pm$ 106.42| 88.77 $\pm$ 33.68 | 92.5 |

[a]Langsplat: 3d language gaussian splatting. CVPR, 2024. \
[b]Gaussian Grouping: Segment and Edit Anything in 3D Scenes. ECCV, 2024. \
[c]Segment Any 3D Gaussians. arXiV, 2023 \
[d]Segment Anything in 3D with NeRFs, NeurIPS, 2023

Meta-review: This paper proposed a new interactive segmentation method for 3D Gaussian scenes. The reviews of this paper are slightly mixed. Two reviewers gave “Borderline reject” while three reviewers were very positive on the paper (2 “Weak Accept” and 1 “Strong Accept”). After carefully reading the authors’ rebuttal and all reviewers’ comments. The AC agrees with the majority of reviewers that this paper has made valuable contributions on this challenging task and showed satisfactory segmentation results on 3D Gaussian scenes. Furthermore, the idea of introducing Graph Cut to segment 3D Gaussians without retraining/fine-tuning is novel and interesting. To prepare the camera-ready version of this paper, the authors are required to make the necessary revisions suggested by the reviewers. This decision was discussed with and approved by the SAC.
Decision: Accept (poster)