Title: GaussianCut: Interactive segmentation via graph cut for 3D Gaussian Splatting
Venue: NeurIPS 2024
Accepted: yes
Keywords: 3D Vision

Segmentation

Graph cut

Gaussian Splatting
Abstract: We introduce GaussianCut, a new method for interactive multiview segmentation of scenes represented as 3D Gaussians. Our approach allows for selecting the objects to be segmented by interacting with a single view. It accepts intuitive user input, such as point clicks, coarse scribbles, or text. Using 3D Gaussian Splatting (3DGS) as the underlying scene representation simplifies the extraction of objects of interest which are considered to be a subset of the scene's Gaussians.  Our key idea is to represent the scene as a graph and use the graph-cut algorithm to minimize an energy function to effectively partition the Gaussians into foreground and background. To achieve this, we construct a graph based on scene Gaussians and devise a segmentation-aligned energy function on the graph to combine user inputs with scene properties. To obtain an initial coarse segmentation, we leverage 2D image/video segmentation models and further refine these coarse estimates using our graph construction. Our empirical evaluations show the adaptability of GaussianCut across a diverse set of scenes. GaussianCut achieves competitive performance with state-of-the-art approaches for 3D segmentation without requiring any additional segmentation-aware training

GaussianCut: Interactive Segmentation via Graph Cut for 3D Gaussian Splatting

Abstract

We introduce GaussianCut, a new method for interactive multiview segmentation of scenes represented as 3D Gaussians. Our approach allows for selecting the objects to be segmented by interacting with a single view. It accepts high-level user input, such as point clicks, coarse scribbles, or text. Using 3D Gaussian Splatting (3DGS) as the underlying scene representation simplifies the extraction of objects of interest which are considered to be a subset of the scene's Gaussians. Our key idea is to represent the scene as a graph and use the graph-cut algorithm to minimize an energy function to effectively partition the Gaussians into foreground and background. To achieve this, we construct a graph based on scene Gaussians and devise a segmentation-aligned energy function on the graph to combine user inputs with scene properties. To obtain an initial coarse segmentation, we leverage 2D image/video segmentation models and then use graph cut to further refine these coarse estimates. Our empirical evaluations show the adaptability of GaussianCut across a diverse set of scenes. GaussianCut achieves competitive performance with state-of-the-art approaches for 3D segmentation without requiring any changes to the underlying Gaussian-based scene representation. Project page: https: //umangi-jain.github.io/gaussiancut/ .

1 Introduction

Recent advances in 3D scene representation have enabled unprecedented quality in 3D view synthesis without requiring specialized equipment or an excessively high computational budget. Fully leveraging these advances requires tools for scene understanding and manipulation specifically designed to operate on such representations. Object selection and segmentation often serve as a crucial first step in both scene understanding and editing tasks. While 2D image segmentation has been widely studied, developing analogous techniques for 3D remains challenging. One key challenge is accounting for the choice of underlying 3D scene representation in the segmentation method.

3D Gaussian Splatting (3DGS) [23] offers an explicit representation of a scene using a set of Gaussians. The nature of this representation motivates the idea that each Gaussian corresponds to either the segmented object or the background. Prior works in 3DGS segmentation involve augmenting each Gaussian with a low-dimensional feature, that is jointly optimized with the parameters of the Gaussians [6, 44, 55]. This is supervised by 2D features, which provide semantic information that can be used for segmentation. While this enables a 3D consistent segmentation, it significantly increases the fitting time and the already high memory footprint of the method. Thus, enabling 3DGS segmentation without modifying the optimization process is an important research challenge.

We address this challenge by proposing GaussianCut, a novel method for selecting and segmenting objects of interest in 3D Gaussian scenes. Our work taps directly into the representation created by 3DGS and maps each Gaussian to either the foreground or background. The proposed process mirrors the interactive nature of 2D segmentation tools, where users can engage through clicks, prompts, or scribbles. We require such user input on a single image and perform the object selection process in two steps. First, we obtain dense multiview segmentation masks from the user inputs using a video segmentation model. Subsequently, we construct a weighted graph, where each node represents a Gaussian. Graph cut then partitions the graph into two disjoint subsets by minimizing an energy function, which quantifies the cost of cutting the edges connecting the subsets. This approach effectively segments the selected foreground object from the background by using the energy function as a measure of dissimilarity between the nodes. An overview of the process is provided in Figure 1.

Our main contribution is a novel approach for segmentation in scenes obtained from 3DGS. Its main technical novelties are twofold: 1) we propose a method for graph construction from a 3DGS model that utilizes the properties of the corresponding Gaussians to obtain edge weights, and 2) based on this graph, we propose and minimize an energy function (Equation 3) that combines the user inputs with the inherent representation of the scene. Our experimental evaluations show that GaussianCut obtains high-fidelity segmentation outperforming previous segmentation baselines.

2 Related Work

2D image segmentation is a long studied problem in computer vision [17, 41, 57]. Recently, models like Segment Anything [25] and SEEM [64], have revolutionized 2D segmentation by employing interactive segmentation. A range of methods have also been developed for 3D segmentation, each tailored to different forms of representation, including voxels [10, 36], point clouds [42, 43], meshes [49, 62], and neural representations [7, 32, 48, 52]. The impressive capabilities of Neural Radiance Fields (NeRFs) [35] in capturing scene information implicitly have inspired numerous studies to explore 3D segmentation for NeRFs. Recent works have also explored segmentation with Gaussians as the choice for scene representation [6, 22, 44, 55, 63].

Training 3D segmentation with 2D masks/features: In addition to the wide adaptation of foundational models for 2D images [58], they are also used extensively by 3D editing and segmentation models. SAM has been used as an initial mask to facilitate 3D segmentation [6, 7, 33] and also for distillation into NeRF [59] and 3DGS models [63]. Semantic-NeRF [61] proposed 2D label propagation to incorporate semantics within NeRF so it can produce 3D consistent masks. MVSeg [37] propagates a 2D mask to different views using video segmentation techniques. ISRF [18] distills semantic features into the 3D scene of voxelized radiance fields. Nearest neighbor feature matching then identifies high-confidence seed regions. 2D features have also been used for facilitating the grouping of Gaussians [55] and for hierarchical semantics using language in 3DGS [44]. Distilled Feature Fields (DFF) [27] and Neural Feature Fusion Fields [50] distill 2D image embeddings from LSeg [30] and DINO [5] to enable segmentation and editing. SA3D [7] uses SAM iteratively to

get 2D segments and then uses depth information to project these segments into 3D mesh grids. SANeRF-HQ [33] aggregates 2D masks in 3D space to enable segmentation with NeRFs.

Segmentation in 3D Gaussian Splatting: Gaussian Grouping [55], SAGA [6], LangSplat [44], CoSSegGaussians [11] and Feature 3DGS [63] require optimizing a 3DGS model with an additional identity or feature per Gaussian, which is usually supervised by 2D image features. These semantic features allow segmentation through user interaction. Gaussian Grouping and LangSplat also allow for textual prompts to segment objects supported through multimodal models like CLIP or groundingDINO [31]. Feature-based methods alter the fitting process of 3DGS by adding additional attributes for each Gaussian and it facilitates learning features for everything in the scene. While useful, this limits the flexibility of interactivity with a single object. Our method is more flexible in choosing specific object(s) as we generate the 2D masks after the user interaction. Adding additional parameters also increases the fitting time for 3DGS. Moreover, such methods often rely on video segmentation models as they require 2D features from all training viewpoints. In contrast, we can operate on an arbitrary number of 2D masks, including just a single mask.

Graph cut for 3D segmentation: Boykov and Jolly [4] introduced a novel global energy function for interactive image segmentation using graph cut [15, 19]. Several follow-up works improved image segmentation using graph cut by designing a better energy function [13], improving optimization [3, 20], and reducing user input requirements [51]. Adapting energy minimization methods for 3D volumes has been difficult, requiring several modifications [20] to manage the higher memory demands. NVOS [45] trains a special multi-layer perceptron (MLP) to predict voxels in the foreground and background and applies graph cut on voxels as post-processing. However, this requires additional training and increases memory consumption. Guo et al. [21] propose 3D instance segmentation of 3D point clouds using graph cut. It involves constructing a superpoint graph and training a separate graph neural network for predicting the edge weights. Unlike their work, our method is a post hoc application and does not require any additional training. Our graph construction and edge weights have also been tailored specifically for 3D Gaussian Splatting.

Concurrent with our work, Segment Anything in 3D Gaussians (SAGD) [22], also performs interactive segmentation using 3D Gaussian Splatting without requiring any segmentation-aware training. However, their primary focus is on refining object boundaries by decomposing boundary Gaussians.

3 Method

3.1 Preliminaries

3D Gaussian Splatting (3DGS) [23] is a technique for creating a 3D representation of scenes based on a set of Gaussian ellipsoids G . 3DGS facilitates real-time rendering and provides high-quality reconstruction. In this representation, each 3D Gaussian is characterized by a set of optimizable parameters. These include 3D position µ ∈ R 3 , spherical harmonics (SH) coefficients (for color) β ∈ R 3( d 2 +1) (where d is the degree of spherical harmonics), scale s ∈ R 3 , rotation r ∈ R 4 , and opacity σ ∈ R . The optimization process involves iteratively rendering scenes and comparing the rendered images with the training views, interleaved with adaptive density control that handles the creation and deletion of the number of Gaussians. The differentiable rendering pipeline in 3DGS uses tile-based rasterization following [28] to ensure real-time rendering. 3DGS performs splatting by depth sorting the Gaussians and α -blending [14] them to project in 2D. The set of differentiable parameters for G , D := { µ i , β i , s i , r i σ i } |G| i =1 , are optimized from a set of posed images.

Graph cut refers to the problem of partitioning the vertices V of a graph G with edges E weighted by { w e } e ∈E into two disjoint, non-empty sets such that the sum of the weights of the edges between the two sets is minimized. This minimum-cost partitioning is known as the minimum cut . Segmentation can be reformulated as graph cut. This is achieved by forming a graph that includes the segmented entities (e.g. pixels, voxels, Gaussians) and introducing weighted edges. This allows defining an energy function, which includes unary terms representing the cost of assigning a node to a set based on individual properties, and pairwise terms that incorporate the cost of assigning neighboring nodes to different sets. The objective of the minimization is to find a cut that optimizes the overall energy, balancing individual preferences and neighborhood interactions. An efficient way for computing this minimum cut in a graph is the Boykov-Kolmogorov algorithm [2].

User input for segmentation

Gaussian graph cut

Video Segmentation

Segmentation mask on multi-view images

Rasterization with masks

Track foreground/background likelihood of Gaussians

Separated foreground and background objects

Render each component separately

Graph construction

3.2 Overview

Given a set of posed RGB images I = { I i } k i =1 and an optimized reconstruction of the scene using a set of Gaussians G , we define the task of interactive 3D segmentation as follows: given a user input (point clicks, scribbles, or text) on any image I 0 ∈ I , the objective is to partition the set of Gaussians in two non-empty disjoint sets S and T such that S represents the set of Gaussians representing the object(s) of interest and T represents the given 3D scene without these object(s). The extracted subset of Gaussians S can be rendered from any viewpoint to effectively cutout the 3D representation of the foreground without retraining. Innately, the other set of Gaussians T can be rendered to remove the foreground object(s). Figure 2 shows an overview of our pipeline.

In order to densify the object selection information provided by the user, we use an off-the-shelf video segmentation model to obtain dense segmentation masks from multiple views (discussed in section 4). For transferring the segmentation masks to the 3DGS representation, we trace the 3D Gaussians that map to the selected region in the masks (discussed in section 3.3). However, the masks used for propagation are not 3D consistent (as the underlying image segmentation model is 3D-unware). Moreover, the errors from 2D masks can propagate in the traced Gaussians and thereby provide a noisy 3D segmentation. To obtain a precise set of foreground Gaussians, we formulate the set of Gaussians G as nodes in a graph (discussed in section 3.4) and leverage graph cut to split the nodes into two sets: the foreground S and the background T .

3.3 Mapping user inputs to Gaussians

We first feed the sparse single-view annotations by the user (e.g., point clicks) to a multiview/video segmentation model to obtain coarse segmentation masks across multiple training views. We then propagate the information from the 2D masks onto the set of Gaussians. For an already optimized 3DGS model of a scene, G , we obtain n masks M := { M j } n j =1 from a video segmentation model corresponding to any n viewpoints I := { I j } n j =1 . Here, M j indicates the set of foreground pixels in the viewpoint I j . For each Gaussian g ∈ G , we maintain a weight, w g , that indicates the likelihood of the Gaussian belonging to the foreground. To obtain the likelihood term w j g pertaining to mask j for Gaussian g , we unproject the posed image I j back to the Gaussians using inverse rendering and utilizing the mask information,

where σ g ( p ) and T j g ( p ) denote the opacity and transmittance from pixel p for Gaussian g . Combining over all the n masks,

This likelihood, w g , captures the weighted ratio of the contribution of Gaussian g to the masked pixels relative to the total number of pixels influenced by it. The complementary value of w g , 1 -w g provides the likelihood of Gaussian g contributing to the background. The value of w g is updated using n 2D segmentation masks during rasterization. Since the rasterization of 3DGS is remarkably fast, each pass typically takes less than a second to update. GaussianEditor [8] also learns an additional tracing parameter but unlike our approach, it maps Gaussians to different semantic classes.

Having the likelihoods w g , a naive approach to extract the 3D representation of the foreground is to threshold the Gaussians and prune those with values below a certain threshold τ . We denote this approach as 'coarse splatting'. Figure 4 demonstrates coarse splatting renders for a plant in the 360-garden scene [1]. Note that the renderings produced by coarse splatting are not accurate, particularly around the edges. This is due to two main reasons: 1) the 2D segmentation models are 3D inconsistent and can be imperfect, leading to artifacts in the final Gaussian cutout, and 2) lifting 2D masks to 3D can introduce its own artifacts.

3.4 Gaussian graph construction

After rasterization with the masks, each Gaussian g ∈ G in the 3DGS representation is characterized with parameters D g := { µ g , β g , s g , r g σ g , w g } , where w g captures the user requirement and the other parameters encapsulate the inherent properties of the scene. To fuse the two sources of information and obtain a precise set of foreground Gaussians, we formulate the optimized 3DGS model as an undirected weighted graph G := ( G , E ) , where each Gaussian in G is a node and E represents the set of edges connecting spatially adjacent Gaussians. We define the neighborhood N ⊆ G × G of a node (Gaussian) as its k -nearest Gaussians in terms of their 3D position. The intuition behind constructing the edges is that Gaussians that map to the same object would be close spatially.

Gaussian graph cut partitions the Gaussians G into two disjoint and non-empty sets S ⊂ G and T ⊂ G , that represent the foreground and background Gaussians, respectively. Our objective is to infer the foreground/background label y g ∈ { 0 , 1 } of each Gaussian g . Let the unary term ϕ g ( · , · ) represent the likelihood of node g being part of foreground or background and the pairwise term ψ g,g ′ ( · , · ) reflect the edge connection between node g and g ′ . To obtain the label for each Gaussian g , graph cut minimizes the aggregate of both unary and pairwise terms given by:

where λ provides a trade-off between the two terms.

Neighboring pairwise weights (n-links) : The pairwise term models the correlation between neighboring nodes. The neighbors for a node are based on its spatial proximity to other nodes. The edge weight between each pair of neighbors is a combination of its spatial distance and color similarity. While segments of an object can have multiple colors, and they often do, neighboring nodes with dissimilar colors can still be identified and grouped based on their spatial proximity. This ensures that parts of an object, despite varying in color, can be linked if they are close in space. For color similarity, we only use the zero-degree spherical harmonic to capture the ambient colors without any angular dependence. The correlation between the neighboring nodes is formulated as

where λ n is a hyperparameter balancing the contribution of position and color similarity, and the function f estimates similarity as f ( x , y ) = exp( -γ ∥ x -y ∥ 2 2 ) ( γ is a positive scalar).

Unary weights (t-links) : We designate two terminal nodes for the graph cut algorithm, the source and the sink node. These terminals represent the foreground (source) and the background (sink) in segmentation tasks. t -links connect all the nodes to both the terminal nodes and the edge weight for

munkong

?

these links represents the pull of that node towards each terminal node. We assign the edge weights connecting each non-terminal node to the source node to reflect its likelihood of belonging to the foreground set S (and as belonging to the background set T for edges connecting to the sink node).

Gaussian tracking, from section 3.3, provides the connection of each Gaussian g to the source and sink terminal nodes, using w g and 1 -w g , respectively. However, these weights can be noisy estimates. Therefore, we introduce an additional term to the edge weights that captures the similarity of node g to the other nodes that are well-connected to the terminal nodes. To do so, we identify high-confidence nodes for both the source and the sink terminals. A Gaussian g is considered as a high-confidence node for the source terminal if w g ≈ 1 and for a sink terminal if w g ≈ 0 . Since computing the similarity of a node to all the high-confidence nodes is computationally expensive, we cluster all the highconfidence nodes (denoted as F and B for the source and sink, respectively) based on their position. For each node g , we then determine the closest cluster by finding g f = argmin g ′ ∈F f ( µ g , µ g ′ ) for the source, and similarly g b for the sink. Consequently, the unary term based on the user input is,

We minimize the objective E in Equation 3 to partition the set of nodes G as S (foreground Gaussians) and T (background Gaussians). To render the foreground object from any viewpoint, we simply render the Gaussian collected in S with T as background.

4 Experimental Setup

Datasets: For quantitative evaluation, we test the scenes from LLFF [34], Shiny [54], SPIn-NeRF [37], and 3D-OVS [29]. All selected scenes from the LLFF and Shiny datasets are real-world front-facing scenes, with 20-62 images each. SPIn-NeRF provides a collection of scenes from some of the widely-used NeRF datasets [16, 26, 34, 35, 56]. It contains a combination of front-facing and 360 ◦ inward-facing real-world scenes. 3D-OVS contains scenes featuring long-tail objects.

Input types: Our model accepts all input processed by SAM-Track [9]. It uses grounding-DINO [31] to process text inputs. For the LLFF scenes used in NVOS [45], we follow their input scribbles to obtain the initial mask. For SPIn-NeRF and Shiny, we use clicks (each scene typically requires 1-4 clicks). For the 3D-OVS dataset evaluation, we use text query as input (results in Table 11).

Evaluation metrics: Different Image-Based Rendering (IBR) models represent 3D scenes in different ways. Thus, obtaining universal ground-truth 3D masks is difficult. To avoid this challenge, we evaluate the segmentation mask of the projected 2D rendering from the scene. The ground-truth 2D masks are typically obtained from professional image segmentation tools. NVOS provides one ground-truth mask for every scene in LLFF. SPIn-NeRF and 3D-OVS provide masks for multiple images in every scene. Shiny dataset does not contain any ground-truth masks so we create our own ground-truth mask. For evaluation, we generate 2D foreground masks by rendering the Gaussians

from the desired viewpoint. We use pixel classification accuracy (Acc) and foreground intersectionover-union (IoU) for evaluating the segmentation masks.

Following NVOS, we also assess the photo-realistic appearance of the segmented object by rendering it against a black background. We trim both the rendered image and the ground-truth image to the foreground object by applying a bounding box that fits the ground-truth mask. This prevents the evaluation from being biased by the background, especially when the object of interest is relatively small. The metrics we report are PSNR, SSIM [53], LPIPS [60].

Implementation details: To obtain segmentation masks from the user inputs (used in Section 3.3), we leverage the advancements in video segmentation models. The user selects the foreground objects on I 0 , and we obtain dense masks for multiple views using SAM-Track [9]. Note that the use of the video segmentation model is done to enhance the performance further and our method can also work with a single image mask (Table 4). We use KD-Tree for efficiently finding the k nearest neighbors to construct the edges between the nodes.

For all the evaluations, we resize the longer image size to 1008, as commonly practiced in novelview synthesis. We optimize 3DGS model for each of the considered scenes, without making any changes to the original 3DGS code. For coarse splatting, we keep the cut-off threshold τ = 0 . 9 for front-facing views and τ = 0 . 3 for the 360 ◦ inward-facing scenes. This disparity stems because parts of objects might not be observed from every viewpoint for the latter and also because of the relative ineffectiveness of video tracking for inward-facing scenes (Figure 8). For graph cut, we keep γ = 0 . 1 for neighboring pairwise position weights and γ = 1 for all other weights, the number of neighbors for every node as 10, and the number of clusters for high-confidence nodes as 4 for sink and 1 for source. λ , λ n , λ u can be adjusted depending on the scene and the quality of coarse splatting but generally, λ n = λ u = 1 and λ = 0 . 5 give decent results.

Baselines: Our comparison includes a selection of baseline models such as NVOS [45], MVSeg [37], Interactive segmentation of radiance fields (ISRF) [18], Segment Anything in 3D with NeRFs (SA3D) [7], Segment Any 3D Gaussians (SAGA) [6], Segment Anything in 3D Gaussians (SAGD) [22], Gaussian Grouping [55], and LangSplat [44]. Unlike our approach, SAGA, Gaussian Grouping, and LangSplat alter the Gaussian optimization process by learning additional features per Gaussian that increases the optimization time (Table 9). SAGD is a concurrent work also designed for 3DGS segmentation and has not yet been published. Thus, their results may be subject to change. SAGD, similar to our approach, does not require any segmentation-aware training and uses a cross-view label voting approach to segment selected objects. All the baselines allow for selecting objects using clicks, except LangSplat, for which we use text queries. Further details on baseline implementation are provided in Appendix A.1.

5 Results

5.1 Quantitative results

Dataset from NVOS: We take the seven scenes from LLFF dataset used in NVOS. NVOS contains a reference image with input scribbles and a target view with an annotated 2D segmentation mask. As shown in Table 1, GaussianCut outperforms other approaches. Unlike ISRF, SAGA, Gaussian Grouping, and LangSplat, GaussianCut works on pretrained representations and does not require any changes to the training process. Owing to the fast rasterization, 3DGS-based approaches can also

render foreground Gaussians in real-time. To compare the rendering quality of the segmented objects using 3DGS, we train a NeRF model at the same resolution and segment it using SA3D. Table 3 shows the photo-realistic quality of the foreground image against a black background. Gaussian Splatting provides significant gains over NVOS and SA3D for rendering quality, providing a boost of +4 . 05 dB PSNR and +1 . 69 dB PSNR, respectively.

Dataset from SPIn-NeRF: We compare our model on all scenes from the SPIn-NeRF dataset, which includes four 360 ◦ inward-facing scenes and six front-facing scenes. Our model gives an overall better performance compared to other baselines. Compared to MVSeg, on 360 ◦ scenes such as lego and truck, GaussianCut provides an absolute IoU gain of 14.3% and 10.5%, respectively. Our model also performs better compared to other 3DGS baselines as shown in Table 2. The 360 ◦ scenes for ISRF were run at one-fourth resolution due to memory constraints. We show the scene-wise results in Table 10. Feature-based 3DGS segmentation methods, such as Gaussian Grouping and LangSplat, outperform GaussianCut on certain scenes, but their interactivity can be limited if the optimized features do not delineate the object of interest. We show such cases in Figure 5. Moreover, we also show in Figure 11 that segmentation masks from GaussianCut contain finer details and a better segmentation quality than the ground-truth masks provided in SPIn-NeRF.

Dataset from Shiny: We test the segmentation performance of our model on four scenes from the Shiny dataset: tools, pasta, seasoning, and giants. We create ground-truth masks for 4 test-view images for each scene and compare our model against non-feature learning-based baselines: SA3D and SAGD. We also report the performance of Coarse Splatting (no graph cut) in Table 4. Figure 10 also shows the quality of segmented images for all the scenes.

5.2 Qualitative results

Similar to SA3D, we perform segmentation in three modes: object segmentation, part segmentation, and text-prompting based segmentation. Figure 3 shows object and part segmentation. GaussianCut can retrieve complex objects (such as truck, lego bulldozer, mirror) precisely. It can segment smaller part segmentation (manhole cover, lego wheels, and socket wrenches).

Our method can also retrieve multiple objects together (socket wrenches and metallic bowl are extracted together).

Figure 4 demonstrates the performance of objection selection using text input. We do a qualitative comparison of GaussianCut with ISRF, SA3D, and SAGD. Feature-based 3DGS methods run into memory issues for this scene. Out of these, SA3D and SAGD also use a text-based prompt to segment the plant. ISRF uses stroke for segmentation. GaussianCut retrieves finer details in the plant with a higher perceptual quality. We also show the rendered image from different viewpoints. It can also be seen that coarse splatting (before the Gaussian graph cut) misses finer details, such as the decorations on the plant, which can be retrieved using GaussianCut.

5.3 Ablation and sensitivity study

Number of views: To obtain the 2D segmentation masks, we run the camera on a fixed trajectory and get rendering from different viewpoints (spiral trajectory for front-facing and circular for 360 ◦ inward scene). We limit the number of frames to 30 for front-facing and 40 for 360 ◦ scenes. Using more segmentation masks can boost performance, however it might not be always preferred, especially for scenes with a large number of training views. SAM-Track can also handle segmentation for unordered multi-frame images. Table 6 shows the effect of varying the number of masks on two scenes from the

munkong

0

SPIn-NeRF dataset (images used were unordered). Since 3DGS offers fast rasterization, the overall time cost for segmentation does not grow linearly with the number of masks as the time taken for segmentation dominates. We also show the qualitative performance with a single mask (no video segmentation model) and just scribbles (no image segmentation model) in Figure 13.

Sensitivity of each term in the graph cut energy function: In order to understand the contribution of each term in the energy function, Table 5 shows the average IoU on the NVOS dataset with each term removed from Equation 4 and 5. Each term contributes to the overall performance and the cluster similarity, in particular, gives a significant boost.

Sensitivity of graph cut hyper-parameters: We test the sensitivity of our Gaussian graph cut algorithm on the number of neighbors (number of edges for each node) and the number of highconfidence clusters. As the number of neighbors increases, the number of edges in the graph also increases (so does the time taken for graph cut). As seen in Table 7, adding more edges can help in modeling more long-distance correlations. However, after a limit, the effects of adding more edges diminish. Adding a large number of clusters for the high-confidence nodes, in Table 8, does not affect the performance drastically and the optimal number can vary depending on the scene. We show sensitivity to other hyper-parameters in Appendix E.

6 Discussion

Our results demonstrate that 3DGS allows for direct segmentation using a pretrained model. Developments in 2D segmentation and tracking have played a crucial role in 3D segmentation. We observe that GaussianCut not only generates 3D consistent masks but also improves the segmentation quality of 2D masks by capturing more details (Figure 12). This is more prominent for 360 ◦ scenes, where the tracker can partially or fully miss the object of interest (Figure 8).

Time requirement: Since we use pretrained 3DGS models, the optimization time for the Gaussians remains the same as in [23] (it took under 15 minutes for every scene we used). For inference, masked rasterization of Gaussians is fast and the time taken for graph cut grows roughly linearly with the number of Gaussians. Table 9 shows a detailed breakdown of time taken in each step: preprocessing (obtaining the features from 2D image/video segmentation models), fitting time (3DGS optimization time), and segmentation time (time taken to obtain the segmented output). Compared to feature-based methods, like Gaussian Grouping, LangSplat, and SAGA, our method does not require any alteration to the fitting process and, therefore, has a shorter fitting time. While the segmentation time is higher for GaussianCut, it still has a much shorter overall time requirement. All reported times have been obtained using an NVIDIA RTX 4090 GPU and an Intel Core i9-13900KF CPU.

Memory requirement: While 3DGS has a higher footprint than NeRF-based models, several recent works reduce the memory footprint with limited loss of quality [12, 38-40]. Our method only stores one additional parameter w g for every Gaussian and is less memory-intensive than methods requiring learning a feature field [6, 13].

Limitations: GaussianCut can address some inaccuracies in 2D video segmentation models, but it may still lead to partial recovery when the initial mask or tracking results are significantly off (Figure 7). By segmenting directly at the level of Gaussians, the resulting segmentation boundaries may inherit inaccuracies of the underlying representation that are not visible in the unsegmented scene due to alpha-blending. While GaussianCut does not require additional scene optimization, our method can still take up to a few minutes for the graph cut component, which makes the segmentation not real-time. The implementation could be improved by applying graph cut on a subset of Gaussians. We leave this to future work. Additionally, extending our energy function to include a feature similarity term (in equation 3) is another potential improvement. We also discuss some failure cases in section B.

7 Conclusion

In this paper, we introduce GaussianCut, a novel approach that taps into the underlying explicit representation of 3D Gaussian Splatting to accurately delineate 3D objects. Our approach takes in an optimized 3DGS model along with sparse user inputs on any viewpoint from the scene. We use video segmentation models to propagate the mask along different views and then track the Gaussians that splat to these masks. In order to enhance the precision of partitioning the Gaussians, we model them as nodes in an undirected graph and devise an energy function that can be minimized using graph cut. Our approach shows the utility of explicit representation provided by 3DGS and can also be extended for downstream use cases of 3D editing and scene understanding.

8 Acknowledgment

This work was supported by the Sony Research Award Program and NSERC Alliance.

References

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR , 2022.

Yuri Boykov and Vladimir Kolmogorov. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. TPAMI , 2004.

Yuri Boykov, Olga Veksler, and Ramin Zabih. Fast approximate energy minimization via graph cuts. TPAMI , 2001.

Yuri Y Boykov and M-P Jolly. Interactive graph cuts for optimal boundary & region segmentation of objects in nd images. In ICCV , 2001.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV , 2021.

Jiazhong Cen, Jiemin Fang, Chen Yang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, and Qi Tian. Segment any 3d gaussians. arXiv , 2023.

Jiazhong Cen, Zanwei Zhou, Jiemin Fang, Wei Shen, Lingxi Xie, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, et al. Segment anything in 3d with nerfs. NeurIPS , 2023.

Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang, Xiaofeng Yang, Yikai Wang, Zhongang Cai, Lei Yang, Huaping Liu, and Guosheng Lin. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting. In CVPR , 2024.

Yangming Cheng, Liulei Li, Yuanyou Xu, Xiaodi Li, Zongxin Yang, Wenguan Wang, and Yi Yang. Segment and track anything. arXiv , 2023.

Özgün Çiçek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger. 3d u-net: learning dense volumetric segmentation from sparse annotation. In MICCAI , 2016.

Bin Dou, Tianyu Zhang, Yongjia Ma, Zhaohui Wang, and Zejian Yuan. Cosseggaussians: Compact and swift scene segmenting 3d gaussians. arXiv , 2024.

Zhiwen Fan, Kevin Wang, Kairun Wen, Zehao Zhu, Dejia Xu, and Zhangyang Wang. Lightgaussian: Unbounded 3d gaussian compression with 15x reduction and 200+ fps. arXiv , 2023.

Pedro F Felzenszwalb and Daniel P Huttenlocher. Efficient graph-based image segmentation. IJCV , 2004.

James D Foley. Computer graphics: principles and practice , volume 12110. Addison-Wesley Professional, 1996.

Lester Randolph Ford and Delbert Ray Fulkerson. Flows in networks . Princeton university press, 2015.

Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR , 2022.

Alberto Garcia-Garcia, Sergio Orts-Escolano, Sergiu Oprea, Victor Villena-Martinez, and Jose Garcia-Rodriguez. A review on deep learning techniques applied to semantic segmentation. arXiv , 2017.

Rahul Goel, Dhawal Sirikonda, Saurabh Saini, and PJ Narayanan. Interactive segmentation of radiance fields. In CVPR , 2023.

Andrew V Goldberg and Robert E Tarjan. A new approach to the maximum-flow problem. JACM , 1988.

Leo Grady. Random walks for image segmentation. TPAMI , 2006.

Haoyu Guo, He Zhu, Sida Peng, Yuang Wang, Yujun Shen, Ruizhen Hu, and Xiaowei Zhou. Sam-guided graph cut for 3d instance segmentation. arXiv , 2023.

Xu Hu, Yuxi Wang, Lue Fan, Junsong Fan, Junran Peng, Zhen Lei, Qing Li, and Zhaoxiang Zhang. Semantic anything in 3d gaussians. arXiv , 2024.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics , 42(4), 2023.

Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In ICCV , 2023.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv , 2023.

Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ToG , 2017.

Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. NeurIPS , 2022.

Christoph Lassner and Michael Zollhofer. Pulsar: Efficient sphere-based neural rendering. In CVPR , 2021.

Kunhao Liu, Fangneng Zhan, Jiahui Zhang, Muyu Xu, Yingchen Yu, Abdulmotaleb El Saddik, Christian Theobalt, Eric Xing, and Shijian Lu. Weakly supervised 3d open-vocabulary segmentation. NeurIPS , 2023.

Quande Liu, Youpeng Wen, Jianhua Han, Chunjing Xu, Hang Xu, and Xiaodan Liang. Openworld semantic segmentation via contrasting and clustering vision-language embedding. In ECCV , 2022.

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv , 2023.

Yichen Liu, Benran Hu, Junkai Huang, Yu-Wing Tai, and Chi-Keung Tang. Instance neural radiance field. In ICCV , 2023.

Yichen Liu, Benran Hu, Chi-Keung Tang, and Yu-Wing Tai. Sanerf-hq: Segment anything for nerf in high quality. arXiv , 2023.

Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. TOG , 2019.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021.

Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 3DV , 2016.

Ashkan Mirzaei, Tristan Aumentado-Armstrong, Konstantinos G Derpanis, Jonathan Kelly, Marcus A Brubaker, Igor Gilitschenski, and Alex Levinshtein. Spin-nerf: Multiview segmentation and perceptual inpainting with neural radiance fields. In CVPR , pages 20669-20679, 2023.

KL Navaneet, Kossar Pourahmadi Meibodi, Soroush Abbasi Koohpayegani, and Hamed Pirsiavash. Compact3d: Compressing gaussian splat radiance field models with vector quantization. arXiv , 2023.

Simon Niedermayr, Josef Stumpfegger, and Rüdiger Westermann. Compressed 3d gaussian splatting for accelerated novel view synthesis. arXiv , 2023.

Panagiotis Papantonakis, Georgios Kopanas, Bernhard Kerbl, Alexandre Lanvin, and George Drettakis. Reducing the memory footprint of 3d gaussian splatting. PACMCGIT , 2024.

Bo Peng, Lei Zhang, and David Zhang. A survey of graph theoretical approaches to image segmentation. Pattern Recognition , 2013.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR , 2017.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. NeurIPS , 2017.

Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d language gaussian splatting. arXiv , 2023.

Zhongzheng Ren, Aseem Agarwala, Bryan Russell, Alexander G Schwing, and Oliver Wang. Neural volumetric object selection. In CVPR , 2022.

Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. " grabcut" interactive foreground extraction using iterated graph cuts. ACM transactions on graphics (TOG) , 23(3):309-314, 2004.

Myrna C Silva, Mahtab Dahaghin, Matteo Toso, and Alessio Del Bue. Contrastive gaussian clustering: Weakly supervised 3d scene segmentation. arXiv , 2024.

Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv , 2021.

Wenming Tang and Guoping Qiu. Dense graph convolutional neural networks on 3d meshes for 3d object segmentation and classification. Image and Vision Computing , 2021.

Vadim Tschernezki, Iro Laina, Diane Larlus, and Andrea Vedaldi. Neural feature fusion fields: 3d distillation of self-supervised 2d image representations. In 3DV , 2022.

Zhuowen Tu. Auto-context and its application to high-level vision tasks. In CVPR , 2008.

Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mehdi SM Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth. Nesf: Neural semantic fields for generalizable semantic segmentation of 3d scenes. arXiv , 2021.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. TIP , 2004.

Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn. Nex: Real-time view synthesis with neural basis expansion. In CVPR , 2021.

Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke. Gaussian grouping: Segment and edit anything in 3d scenes. arXiv , 2023.

Lin Yen-Chen, Pete Florence, Jonathan T Barron, Tsung-Yi Lin, Alberto Rodriguez, and Phillip Isola. Nerf-supervision: Learning dense object descriptors from neural radiance fields. In ICRA , 2022.

Nida M Zaitoun and Musbah J Aqel. Survey on image segmentation techniques. Procedia Computer Science , 65:797-806, 2015.

Chaoning Zhang, Fachrina Dewi Puspitasari, Sheng Zheng, Chenghao Li, Yu Qiao, Taegoo Kang, Xinru Shan, Chenshuang Zhang, Caiyan Qin, Francois Rameau, et al. A survey on segment anything model (sam): Vision foundation model meets prompt engineering. arXiv , 2023.

Hao Zhang, Fang Li, and Narendra Ahuja. Open-nerf: Towards open vocabulary nerf decomposition. In WACV , 2024.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR , 2018.

Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison. In-place scene labelling and understanding with implicit scene representation. In ICCV , 2021.

Kaichen Zhou, Lanqing Hong, Enze Xie, Yongxin Yang, Zhenguo Li, and Wei Zhang. Serf: Fine-grained interactive 3d segmentation and editing with radiance fields. arXiv , 2023.

Shijie Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang, and Achuta Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. arXiv , 2023.

Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. NeurIPS , 2024.

A Implementation details

We run our segmentation algorithm on 3D Gaussian Splatting representations, following the code provided by Kerbl et al. [23]. All scenes are optimized for 30,000 steps using the default parameters. We use SAM-Track [9] as the video segmentation model.

For the datatset used in NVOS [45], we use the provided reference image with the user scribbles for a fair comparison. For the SPIn-NeRF dataset [37], we use the first image in the directory as the reference image for the user input. The scenes reported throughout the paper are selected from the following datasets:

NVOS (LLFF [34] subset): flower, fortress, fern, horns, orchids, trex, leaves

SPIn-NeRF (collection from some widely used datasets [16, 26, 34, 35, 56]): orchids, leaves, fortress, horns, truck, lego bulldozer

Shiny [54]: giants, tools, seasoning, pasta

Mip-NeRF [1]: cycle, garden, bonsai

LERF [24]: figurines

3D-OVS [29]: lawn, sofa, bed, bench, room

Mask evaluation: The Gaussians optimized for 3DGS can have ambiguous structures as they are not geometrically constrained. When partitioning the Gaussians as foreground or background, the boundary Gaussians can appear as having shard-like artifacts (or 'spiky' Gaussians). Since the goal of this work is to effectively characterize a Gaussian as foreground or background, we render the foreground mask by overriding the colors of background Gaussians. To generate object assets, our algorithm can be combined with Gaussian decomposition based approached [22].

A.1 Baseline implementation details

A.1.1 Gaussian Grouping

Gaussian Grouping [55] learns an additional feature per Gaussian that can be used to group Gaussians belonging to the same object. We use SAM-Track to get all the 2D segmentation masks. While the default implementation of Gaussian Grouping uses DEVA [9] masks, we chose SAM-Track for both GaussianCut and Gaussian Grouping to maintain consistency in mask quality across the methods. Similar to GaussianCut, we use clicks to segment objects in NVOS and SPIn-NeRF benchmarks. However, Gaussian Grouping uses a mask for segmenting everything and can, therefore, sometimes

Evaluationimage

Ground-truthmask

LangSplat score map

Evaluationimage

Ground-truthmask

LangSplatscoremap

produce over-segmented object segments. To handle such cases and obtain the final segmentation, we aggregate all the segments that constitute the object. We also use the same number of 2D segmented masks as the number of training views. In contrast, for GaussianCut, we limit the number of masks to 30 for front-facing scenes and 40 for 360 ◦ inward-facing scenes. To prevent memory issues in Gaussian Grouping, we restrict the total number of Gaussians across all scenes to 2 M.

A.1.2 LangSplat

We obtained all 2D features at feature level 'part' for LangSplat [44]. Since we could not use clicks and scribbles to obtain the segment, we have used text queries. We tried multiple text queries for each scene and reported the results on the best performing query. For certain scenes, text queries can constrain the selection of an object. For instance, in Figure 5, multiple instances of the leaves get a high relevancy score when segmenting the front leaf.

B Limitations

False negative

False positive

The performance of our method depends on the robustness of image and video segmentation models. For all the scenes tested, we do not tune SAM-Track and use the default settings. SAM-Track (built on SAM) can provide coarse segments, even on the reference image, especially for irregular scenes, as shown in Figure 6. GaussianCut improves the segmentation details of SAM but there still remains scope for improving the segmentation performance further for the more intricate patterns.

Similar to image-segmentation models, video-segmentation models can also have inaccurate segmentation masks. This issue is more pronounced in complex 360 ◦ scenes, where an object can entirely

Inputimage

Initialsegmentationmask

Segmentedobjectfromcorrespondingviewpoints

Inputimage

Propagatedsegmentationmaskusingvideosegmentationmodel

Initialsegmentationmask

Segmentedobjectfromcorrespondingviewpoints

change its orientation, which can lead the trackers to fail in segmenting all views effectively. We illustrate two instances in Figure 7 and 8, where GaussianCut corrects the inaccuracies of SAM-Track with varying levels of effectiveness.

Reference image

Initial mask

Reference image

Initial mask

Garden

Reference image

Initial mask

Figurines

C More visualization results

We present additional segmentation visualizations for 360 ◦ inward scenes taken from Mip-NeRF [1] and LERF datasets [24] in Figure 9. GaussianCut segments complex and irregular features, including

the leaves and wire in the bonsai lego, the detailed decorations of the plant and table in the garden scene, as well as the cord, viewfinder, and flashbulb of the vintage camera.

D Additional results

D.1 Shiny dataset

SA3D

Ours

Amunkong

Scenes in Shiny [54] dataset have complex shapes of objects (pasta), are placed in more cluttered environments (tools), or possess a subtler distinction between the foreground and background colors (giants). We test four scenes from the Shiny dataset and label four images from each scene as ground-truth. Table 4 shows the improvement of GaussianCut against coarse splitting and SA3D [7] with an overall +0.7% and +1.7% absolute gain in foreground mIoU, respectively. Qualitative results from the dataset are shown in Figure 10. GaussianCut can retrieve fine details (like the strands of the pasta) more accurately.

D.2 SPIn-NeRF

Table 10 shows the performance of GaussianCut on each scene of the SPIn-NeRF dataset. Furthermore, we show in Figure 11 that the quality of the mask produced by GaussianCut contains finer details than the ground-truth labels from SPIn-NeRF.

Lego

Orchids

Pinecone

Truck

(a) Image

(b) SPin-NeRF

(c) Ours

(d) Segment

D.3 3D-OVS

To test the performance of our method using text queries, we test on the 3D-OVS [29] dataset and compare it against Gaussian Grouping [55], LangSplat [44], and Contrastive Gaussian Clustering [47] in Table 11. We use the grounding-DINO integration in SAM-Track to obtain the initial segments. The baseline numbers reported in Table 11 are taken from [47]. We use a different text query for some objects than [47]. This was done to ensure that we have a decent initial mask from SAM-Track as the goal of our work is not to improve language understanding in 3D models.

D.4 Qualitative comparison with 2D segmentation model

The objects segmented by GaussianCut exhibit fine details, as depicted in the masks presented in Figure 12. Although our method uses SAM predictions as an initial mask, the segregation of Gaussians provides information with greater precision compared to SAM alone.

E Additional sensitivity analysis and ablations

E.1 Binary weights for coarse splatting

As mentioned in Section 3.3, the likelihood term w g for each Gaussian g is obtained by taking a weighted ratio of g 's contribution on the masked pixels compared to the total number of pixels it affects. Instead of using weighted assignment, we can also have a hard binary assignment where a g either contributes to a foreground pixel or it doesn't. For the n viewpoints, I := { I j } n j =1 that have corresponding masks M := { M j } n j =1 ,

which reflects the ratio of the number of pixels that g has contributed to in M j and I j . As shown in Table 12, since soft assignment has marginally better performance, it is our default implementation.

E.2 Sensitivity of hyperparameters

We share a default setting in section 4 which performs reasonably well on all our datasets. The sensitivity of each parameter can be very scene-dependent. For instance, in a scene where parts of an object have different colors, a very high weight on the color similarity can affect adversely. We show the effect of λ (controls the pull of neighboring vs terminal edges) and γ (decay constant of the similarity function) on two scenes in Table 13 and Table 14, respectively. The reported metric is IoU.

E.3 Threshold of coarse splatting

For the four 360-degree inward scenes in the SPIn-NeRF benchmark, we show a sweep of the threshold τ (default is 0.3 in our implementation) used for Coarse Splatting. GaussianCut outperforms all the thresholds considered for coarse splatting as shown in Table 15.

(a) RGB

(b) SAM

(c) Ours

Garden

Truck

E.4 How important are the 2D segmentation masks?

In order to understand the extent to which the performance of our model depends on the initial 2D segmentation mask, we do the masked rasterization with just scribbles, a single mask, and multiview masks. Figure 13 shows the segmentation result of Coarse Splatting and GaussianCut. The effectiveness of GaussianCut is heightened further when the initial segmentation mask is sparse. Table 16 also shows the performance improvement when running GaussianCut directly on user scribbles.

Scribbles

Singlemask

Multi-viewmasks

Input

CoarseSplatting

Ground Truth

GaussianCut

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The introduction clearly lists the contribution of this work.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have written about the limitations in the main paper. We have also included a few failure cases in the appendix.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our method does not have any theoretical result.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes, we mention all the hyperparameters and the off-the-shelf models we have used. When building on another codebase, we use their default settings (or mention explicitly if something is changed).

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code is available at: https://github.com/umangi-jain/gaussiancut Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Yes, we list it in the implementation details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Factors for stochasticity are less.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Provided in the discussion section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for interactive multiview segmentation of scenes represented as a set of 3D gaussians (3D gaussian splatting). The method takes as an input multiple views of a scene, constructs 3D Gaussian splatting representation of the scene, takes interaction from the user (scribbles, points, or text prompts), uses a video segmentation method (prior work) to get object masks from user input in each view, and then constructs a graph with nodes corresponding to the gaussians. A binary energy function on this graph is designed which encourages similar gaussians (both in space and color) to be connected with edges of higher weight. Also for each Gaussian there is a likelihood term representing how likely it is to be foreground/background. Finally this energy function is optimized with a graph cut and gives a separation of the cloud of gaussians into two sets: foreground/background. After separation in 2 sets, foreground/background can be visualized separately using the corresponding set of Gaussians. Experimental results outperform prior work.\n\nSTRENGTHS:\nThe proposed method is well motivated. The energy function makes sense.\n\nWEAKNESSES:\nComputational cost is not real time.\n\nMapping user input to gaussians could be better described. For someone not familiar with 3D Gaussian splatting prior work, it is hard to understand how it works. An illustration could help. \n\nThe cluster similarity in unary terms (Eq. 3) seems to have a big impart on the performance, but is kind of ad-hock not so well motivated. The problem might be a result of 'shortcutting' bias in the energy formulation (cuts separating a small component are cheaper). What you are doing here is kind of reminiscent of saying that pixels close to the foreground scribbles should be more likely in the foreground (if I think of an analogy to 2D segmentation).\n\nQUESTIONS:\nWithout the mask refinement method (just from user scribbles), by how much the performance goes down?\n\nLIMITATIONS:\nLimitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes GaussianCut for interactive multi-view scene segmentation using 3D Gaussian Splatting. It first accepts user input (clicks, scribbles or text, similar to SAM) on single images, and them aim to segment the corresponding 3D Gaussians. The method constructs a graph based on scene Gaussian, and then using graph-cut algorithm to minimize the energy function. Segmentation tracking method are employed to provide the initial segmentation masks. The experiments are performed on LLFF, Shiny, SPIn-NeRF and 3D-OVS.\n\nSTRENGTHS:\n1. The paper has a good presentation and clear writing, where figures are clean for the readers to understand the method pipeline.\n\n2. The paper has a good motivation on extracting objects from explicit 3D Gaussians. Graph cut is well known for image segmentation on pixels, and the extension to 3D Gaussians is interesting.\n\n3. The quantitative results in Table 1, 2, 3 and 4 show the reasonable segmentation performance brought by GaussianCut.\n\nWEAKNESSES:\n1. Although extracting objects from explicit 3D Gaussians has a good motivation, this has been studied in previous works like LangSplat [a] and Gaussian Grouping [b], but these two very relevant methods are missing quantitative comparisons in the main paper. It is not clear what are the advantages of the proposed Gaussian Cut comparing to [a] and [b]. In Table 10 (should move to the main paper), [a][b] are compared but the performance of Gaussian Grouping is still on par or even better than the proposed GaussianCut. To understand the paper novelty, the paper should clarify these main differences / advantages in the main paper, and also include the detailed running speed comparison. \n\n2. Using video tracking masks to obtain coarse segmentation is not new, which has been explored in [b, c], but not compared. Since [a][b] both lift SAM's masks to 3D, these 2 methods can also perform click/scribble based 3D segmentation\n\n3. The limited performance improvement of the introduced spatial, color and cluster similarities in Table 5, which show the improvement of the proposed n-links and t-links are minor. Also, the extension of Graph cut from 2d pixels to 3D Gaussians seems very straight-forward.\n\n4.  More comparisons on benchmarks like Replica (setting proposed by Panoptic Lifting [d]) or Lerf-Mask [b]  are desired.\n\n[a] Langsplat: 3d language gaussian splatting. CVPR, 2024.\n\n[b] Gaussian Grouping: Segment and Edit Anything in 3D Scenes. ECCV, 2024.\n\n[c] CoSSegGaussians: Compact and Swift Scene Segmenting 3D Gaussians with Dual Feature Fusion. arXiv, 2024.\n\n[d] Panoptic Lifting for 3D Scene Understanding with Neural Fields. CVPR, 2023.\n\nQUESTIONS:\nHow will GaussianCut perform when video segmentation/track fail due to large motion? which may lead the coarse segmentation contain large portions of errors.\n\nHow many Gaussians are considered during the graph construction? The most concerning part for me is the tech novelty of the paper.\n\nLIMITATIONS:\nThe limitation is discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents GaussianCut, a new method for foreground/background segmentation of 3D Gaussian scenes. GaussianCut relies on 2D image/video segmentation masks, which are generated for a subset of the training images. GaussianCut propagates these masks to the 3D Gaussians by projecting each Gaussian onto each mask and averaging the corresponding mask values. It then assembles the Gaussians into a graph where each Gaussian is connected to its nearest neighbors, defines several energy terms (for nodes and edges), and uses these energy terms alongside a graph cut to partition the Gaussians. Unlike several baselines, GaussianCut can be applied to pre-trained 3D Gaussian scenes without retraining/fine-tuning.\n\nSTRENGTHS:\n- In Figure 4, the qualitative results of the proposed method look much sharper than those of the baselines. Also, the quality metrics for the proposed method improve over the baselines slightly.\n- Figures 5 and 7 make it clear that the proposed method is fairly robust to bad 2D input segmentations.\n- The paper is clear and easy to follow.\n- The proposed method is simple.\n\nWEAKNESSES:\n- One major weakness is that none of the tables include the runtimes of the proposed method and the baselines. This is important because segmentation is arguably much more useful if it runs at interactive speeds. The paper also does not list the runtime of the simpler coarse splatting variant/ablation. I think it is crucial for the paper to list both the training-time overhead and segmentation runtime of each method.\n- Another major weakness is that the paper includes/omits baselines in the tables very inconsistently. For example, SAGD and ISRF are featured in Figure 4, but not in Table 3. On the other hand, MVSeg and SAGA are featured in the table, but not the figure. I would be much more confident in the results if the baselines were always included (when appropriate).\n\nOverall, I would be happy to raise the paper's score if the above two major weaknesses were addressed. My score for contribution is held back by the fact that some of the baselines (e.g., SAGA) are significantly faster to run despite having similar quality.\n\n- Another smaller weakness is that the paper does not include visualizations of the energy terms, high-confidence clusters, etc. These would be very helpful for building the reader's intuition for the method.\n\nMinor nitpicks:\n- Line 55/56: The following sentence should perhaps include citations: \"Recent works have also explored segmentation with Gaussians as the choice for scene representation.\"\n- Line 101: I was confused about what is meant by \"decomposing the boundary Gaussians.\"\n- Line 144: \"Guassians\"\n- Line 169: The graph is defined as $(|\\mathcal{G}|, \\mathcal{E})$. This should be $(\\mathcal{G}, \\mathcal{E})$ instead.\n- Line 170: The definition of the neighborhood is listed as $\\mathcal{N} \\subseteq |\\mathcal{G}| \\times |\\mathcal{G}|$, but the neighborhood is defined in terms of nodes ($\\mathcal{N} \\subseteq \\mathcal{G}$) and not edges (which would be $\\mathcal{N} \\subseteq \\mathcal{G} \\times \\mathcal{G}$ anyways) in the next sentence.\n- Line 264: Units (dB) should be listed for the PSNR differences.\n- Line 296-297: The time cost for segmentation technically grows linearly, but the constant factor is big enough that this doesn't matter. I would update the sentence here to be more precise.\n- The single mask ablation (line 298) should probably be included in Table 5.\n\nQUESTIONS:\n- Line 171-172: The authors state that \"Gaussians that map to the same object would be closer spatially.\" This seems reasonable, but isn't always the case. For example, dull specular highlights are often represented via transparent surfaces and \"clouds\" inside objects. Does the proposed method handle these cases well?\n- The proposed method has a number of hyperparameters. How sensitive is it to these hyperparameters?\n- How good is the coarse splatting baseline with well-chosen hyperparameters? I think visualizing a sweep of the threshold in the appendix would convince the reader that the proposed approach works better no matter what threshold is chosen.\n- How are the frames ordered before they are passed to the video segmentation model? How sensitive is the method to this ordering?\n- To what extent does the proposed method's performance rely on SAM-Track's quality? Do any baselines rely on different video segmentation models, and if so, how do the metrics change when those are updated to use SAM-Track?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes GaussianCut for interactive 3D segmentation. The GaussianCut takes a trained 3DGS model and the user prompt as inputs. The SAM model first transfers the user prompt into an initial mask. Then the 3DGS model renders multiple view images and an existing video-tracking model is used for segmenting 2D masks across multiple views. With the masks on multiple views, the splatted Gaussians are identified with two likelihood parameters. Then the graph-cut method is applied for the Gaussian points where each Gaussian is a node, and the edge models the foreground and background relations. Results on multiple benchmarks show the effectiveness of the proposed method.\n\nSTRENGTHS:\n- This paper is well-written and easy to follow.\n- Using Graph Cut to segment the 3D Gaussians makes sense and sounds interesting.\n- The result on multiple benchmarks shows the proposed method achieves new SOTA performance.\n\nWEAKNESSES:\n- The proposed method is kind of straightforward. The framework of GaussianCut is a combination of all existing models. The SAM model is used to obtain masks from user prompts. The SAM-Track model is used to generate Multiview masks. 3DGS model can explicitly model relations between 3D Gaussians and 2D pixels. Graph Cut model is used to separate the Gaussian points. This paper should explain the key contributions of GaussianCut itself.\n\n- Details of associating 3D Gaussians with 2D masks should be given. As alpha blending will assign different weights to different Gaussians for a single pixel, For the Gaussians splatted into one pixel, are they assigned different weights just for this pixel? Or a hard (binary) assignment is used?\n\n- The time cost of the proposed method is shown in Tables 6 and 7. However, the comparisons with other SOTA methods in terms of time cost should be included to illustrate the speed advantage/disadvantage of the proposed method.\n\nQUESTIONS:\nThe novelty of the proposed method should be justified. The proposed method is more like a post-processing. Some important method details and comparisons are missing.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method for segmentation in 3D Gaussian Splatting. Given a prompt in 2D, the proposed approach is capable of segmenting objects of interest from the 3D Gaussians. Specifically, the method first performs 2D segmentation in all training views, which are then propagated into 3D using a technique similar to visual culling. This achieves a coarse 3D segmentation in the initial stage. Subsequently, the method employs graph cuts to refine the 3D segmentation. The paper validates the method on multiple datasets using 2D segmentation metrics and demonstrates the potential of the proposed approach.\n\nSTRENGTHS:\nThe paper proposes refining 3D segmentation via graph cuts using carefully-designed energy functions. This step is practical and effective. \n\nIt’s surprising to see that the method, even without 2D mask supervision, can still outperform baselines like LERF.\n\nWEAKNESSES:\nThe paper has several weaknesses as below: \n\nWriting: \n- The method is not well-motivated in the introduction section. Specifically, in lines 24-39, the problem defined in the paper is hard to understand. The authors might want to rewrite this section to make the motivation clearer.\n- Part of the implementation is not clear to me. Specifically, I’m uncertain if the method is training-free for segmentation when provided with a pretrained GS model. If so, it would be beneficial to make this clear, as being training-free is a significant strength compared to other methods.\n\nMethod: \n- I’m not entirely convinced by the training-free approach in the proposed method. In my view, learning a per-Gaussian feature for segmentation offers more flexibility. Therefore, I suggest the authors justify their method by demonstrating what the proposed method can achieve that learnable features cannot.\n\nExperiments: \n- The important comparison between baselines that use 3DGS such as GaussianEditor and LangSplat. The paper only reports the baselines that use NeRF, which is generally worse than 3DGS in the segmentation task. In other words, the baselines are insufficient to justify the proposed method.\n- Both qualitative and quantitative results of 3D segmentation are missing in the result section. From my viewpoint, it’s essential to this paper since the proposed method aims to perform 3D segmentation. While 2D segmentation partially shows the performance of the proposed method, it’s not the correct metric to validate the proposed method. The paper should not avoid this evaluation because 3D GT segmentation is missing in the datasets used in the paper. In this case, I would suggest the author trying some other datasets like ScanNet where 3D GT is available.\n\nQUESTIONS:\nPlease address the concerns that I described above. During rebuttal, I would expect author justify better the proposed methods according to my suggestions I made above. Before, I currently vote for reject.\n\nLIMITATIONS:\nYes, they did it well.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}