Segment then Splat: Unified 3D Open-Vocabulary Segmentation via Gaussian Splatting
Authors: Yiren Lu
Yunlai Zhou
Yiran Qiao
Chaoda Song
Tuo Liang
Jing Ma
Huan Wang
Yu Yin
Venue: NeurIPS 2025 poster ()
Keywords: 3D Reconstruction
3D Scene Understanding
Open Vocabulary Segmentation
Differentiable Rendering
Primary area: applications

Abstract: Open-vocabulary querying in 3D space is crucial for enabling more intelligent perception in applications such as robotics, autonomous systems, and augmented reality. However, most existing methods rely on 2D pixel-level parsing, leading to multi-view inconsistencies and poor 3D object retrieval. Moreover, they are limited to static scenes and struggle with dynamic scenes due to the complexities of motion modeling.
In this paper, we propose Segment then Splat, a 3D-aware open vocabulary segmentation approach for both static and dynamic scenes based on Gaussian Splatting. 
Segment then Splat reverses the long established approach of "segmentation after reconstruction'' by dividing Gaussians into distinct object sets before reconstruction. Once reconstruction is complete, the scene is naturally segmented into individual objects, achieving true 3D segmentation. This design eliminates both geometric and semantic ambiguities, as well as Gaussian–object misalignment issues in dynamic scenes. It also accelerates the optimization process, as it eliminates the need for learning a separate language field.
After optimization, a CLIP embedding is assigned to each object to enable open-vocabulary querying. Extensive experiments on various datasets demonstrate the effectiveness of our proposed method in both static and dynamic scenarios.

Segment then Splat: Unified 3D Segmentation via Gaussian Splatting
Yiren Lu', Yunlai Zhou', Yiran Qiao', Chaoda Song',
Tuo Liang',
Jing Ma', Huan Wang?. Yu Yin 1
Case Western Reserve University
Westlake University
[yiren. lu, yunlai zhou, yiran.qiao, chaoda.
tuo. liang, jing ma5, yu. qu.yin)@case.cm edu
2 "wanghuan@westlale edu.. cn
https://vulab-at github. io/ Sheneenttthen-Sppe
Abstract
Open-vocabulary querying 3D space crucial enabling more intelligent perception in applications such robotics, autonomous systems, and augmented reality. However, most existing methods rely on 2D pixel-level parsing, leading multi-view inconsistencies and poor 3D object retrieval. Moreover, they limited to static scenes and struggle with dynamic scenes due to the complexities of motion modeling. In this paper, propose Segment then Splat, 3D-au 3D-aware open vocabulary segmentation approach for both static and dynamic scenes based Gaussian Splatting. Segment then Splat reverses the long-established approach "segmentation after reconstruction" " by dividing Gaussians into distinct object before reconstruction. Once reconstruction complete, the scene naturally segmented into individual objects, achieving true 3D segmentation. This design eliminates both geometric and semantic ambiguities, as well as Gaussian-object misalignment issues in dynamic scenes. It also accelerates the optimization process, eliminates the need for learning a separate language field. After optimization, a CLIP embedding is assigned to each object to enable open-vocabulary querying. Extensive experiments on various datasets demonstrate the effectiveness of our proposed method in both static and dynamic scenarios.
Introduction
3D open-vocabulary querying marks pivotal step in language-driven interaction with 3D envi- ronments, removing the need for predefined labels. This capability is vital for large-scale scene exploration, scene understanding, robotic navigation [1.3] and manipulation 7 where free-form text bridges human language and machine perception.
3D Gaussian Splatting (3DGS) 8 has been widely adopted 3D representation due to its efficient training and real-time rendering capabilities. While 3DGS has demonstrated remarkable performance in scene reconstruction and novel view synthesis, it lacks inherent semantic understanding. limiting its applicability tasks that require natural language-driven retrieval and reasoning.
To solve this issue, most existing works [9 incorporate a separate language field alongside Gaussian Splatting reconstruction. By rendering the language field into 2D feature maps, they enable pixel-based querying by retrieving relevant pixels based on the input text embedding. However, this approach essentially performs segmentation in space rather than partitioning Gaussians in 3D> space, leading to several drawbacks: 1) Inconsistent 2D segmentation across different views, leading
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Input
Reconstruct Splat
Pixel-level Segment
"Chopsticks" CLIP
Rosket
Similarity
Training Images
Reconstruction
Image
Map
Queried Mask
Object Query Result
(8) Truditional Splat then Segment Pipeline
SIM
Segment
Reconstruct Splat
"Chopstioks"
CLIP
Similarity
Point Clouds
Objecct-
Trained Object-
"Chopstickr"
spceilic Gaussians
Reconstruction
spocific Gaussians
Gnussians
Object Query Resull
(b) The Proposed Segment then Splat Pipeline
Figure I: Traditional 3D Open- Vocabulary Segmentation vs. our Segment-then-Sppe Pipeline. (a) The traditional Seat-thhn-Segmenn pipeline learns language field alongside the reconstruction of the entire scene. During object queries, it renders Gaussian language embeddings into 2D feature map to identify relevant pixels based on the input text embedding. (b) In contrast, our Segment-then- Splat pipeline first initializes Gaussians into object-specific sets before reconstruction, ensuring more precise object-Gaussian Gaussian correspondence and improving segmentation accuracy.
to inaccurate object boundaries. 2) Failure to capture true 3D object information, complicating 3D object extraction and limiting downstream tasks like robot navigation and 3D manipulation. 3) Inapplicability to dynamic scenes, since Gaussians may have varying semantic meanings at different time steps, preventing straightforward extensions to time-var ainevvryyigg moving objects.
Recently, few works 13 have explored direct segmentation in 3D space. However, these approaches require predefined number of objects for clustering [13] or are limited to foreground segmentation [14] and also cannot be directly applied to dynamic scenes.
Despite the differences in segmentation strategies, whether pixel-based or 3D-based, all existing approaches follow a "reconstruction then segmentation" (i.e., splat then segment) paradigm. This approach inherently results in imprecise object boundaries, as each Gaussian may encode geometric and semantic information from multiple objects, leading to geometric and semantic ambiguity.
In this paper, we propose Segment then Splat, a unified framework for 3D-wware open- pocabbocyyylary segmentation that can be applied to both static and dynamic scenes. Unlike existing methods that adopt a "splat then segment" approach, our method reverses the process by first initializing each object with specific set of Gaussians, shown in Fig. During training, each set of Gaussians is assigned a unique object ID) and contributes only to its corresponding object, guided by 2D multi-view mask supervision. By doing so, each Gaussian is dedicated to single object and thus learns more accurate object geometry. Moreover, since the Gaussian-object correspondence is strictly maintained, our method can be directly applied to dynamic scenes without the risk of Gaussian-object misalignment (i.e., one Gaussian may represent different objects at different time steps). Finally, Segment then Splat requires only one pass of reconstruction and does not depend on learning an additional feature field, significantly improving efficiency. In summary, our key contributions include:
We propose Segment then Splat, novel paradigm that segments Gaussians into object sets before reconstruction. This enables unified static/dynamic open- vocc-uabbyl segmentation, eliminates auxiliary language fields, and significantly reduces training complexity.
Our framework features robust object tracking module that maintains spatial -entrall consistency of object-specific Gaussians in the scene, ensuring accurate segmentation and motion modeling while preventing misalignment.
Learning object-specific Gaussians from the outset preserves explicit object- corre- spondence, eliminating geometric and semantic ambiguity, yielding superior 3D geometries and multi-level segmentation granularity.

---

Extensive experiments demonstrate state-of-the-art performance across diverse static and dynamic datasets in 3D open-vocabulary segmentation, object geometry accuracy, and computational efficiency.
2 Related Work
2.1 3D & 4D Gaussian Splatting
3D Gaussian Splatting (3DGS) [8] is a widely recognized 3D scene representation that introduces anisotropic 3D Gaussians and an efficient differentiable splatting scheme. This enables high-quality explicit scene representation with efficient training and real-time rendering. However, since it was originally designed for static scenes, 3DGS lacks the capability to model dynamic environments.
To address this limitation, Dynamic 3D Gaussians [15] employs a table-based strategy, storing each Gaussian's 's mean and variance at every timestamp. 4D Gaussian Splatting [16] extends 3DGS into four dimensions, adding a temporal component to facilitate dynamic scene reconstruction. Meanwhile, Deformable 3D Gaussians [17 leverages multi-layer perceptron (MLP) to learn per-timestamp positions, rotations, and scales for each Gaussian, effectively capturing object motion and deformation over time. 4DGaussians
[18] utilizes multi-resolution HexPlanes [9] to decode features for temporal deformation of 3D Gaussians, while STG [20] uses a temporal opacity term and polynomial function for each Gaussian, yielding more detailed representation of dynamic scenes.
Despite these advancements, the above methods act solely as scene representations. After reconstruc- tion, they do not support additional interaction provide information beyond geometry and texture. In contrast, our approach enhances interaction by integrating CLIP [21] embeddings into Gaussian Splatting. This assigns semantic meaning each Gaussian, enabling open- open-vocabulary segmentation where users can retrieve, organize, and query objects using natural language prompts.
2.2 Language Embedded Scene Representation
Prior to the emergence of radiance field representations, many studies explored the integration of feature embeddings into point cloud representations enhance scene understanding (2 266 More recently, a growing body of work has focused on incorporating language features [9] 27 28 29] into radiance fields (e.g., NeRF and 3DGS). LERF 9 features in NeRF. Specifically, it extracts pixel-level CLIP embeddings from multi-scale image crops and trains them alongside NeRF to enable ugen-vocabulary 3D queries. Building on this idea, LEGaussians [ incorporates uncertainty and semantic feature attributes into each Gaussian, while introducing quantization strategy to compress high-level language and semantic features. Spnat 10 employs a scene-wise language autoencoder to learn language features within a scene-specific latent space, and incorporate SAM mask to enable clear object boundaries in rendered feature images. Despite these advancements, all the above methods essentially perform 2D segmentation when conducting vyen-vocabulary segmentation, as they compare rendered 2D language features with the input text query embedding. OpenGaussian [13] leverages contrastive learning to assign feature embedding to each Gaussian, then applies K-means clustering to group Gaussians into multiple object clusters, thereby realizing 3D segmentation. Similarly, GaussianCut [14] utilizes graphcut 30 32
pioneered this approach by embedding CLIP
to segment Gaussians into foreground and background based on user input.
However, all of the above-mentioned methods adhere a "splat then segment" (i.e., reconstruction then segmentation) pipeline, which inherently results in imprecise object boundaries, since each Gaussian may contain geometry and semantic information from different objects. Furthermore, these methods struggle with dynamic scenes due to Gaussian-object misalignment, preventing their direct application to non-static environments. Dynamic 3D Gaussian Distillation (DGD) [33 distills the feature from LSeg 3 into a feature field to achieve open- vocabulary segmentation. 4D LanaSplat (35) further leverages Multimodal Large Language Models (MLLMs) through object-wise video prompting to enhance both menessenitivee and time-agnostic open- apen-vocabulary understanding. Similarly, 4-LEGS (36] distills temporal language features into 4DGS for event localization from text prompts. Nevertheless, these methods require specific modifications for dynamic scenarios and still suffer from abject-Gaussia misalignment.
In contrast, we introduce Segment then Splat, which overturns the long-established "splat then segment" paradigm. Instead, approach first initializes object-specific Gaussians for each object
3

---

Robust Tracking
Gaussian Intialization
Specif-ppeiffe
Optimization Reconstraction
Recomstrucced
Tracking
CLIP Embedding Association
Tracking
Object
Mendie Aino O'ject
Resster Depiliatte Oijives
Recoverz frowe Trasding
Mi
Clouds
Point
Randden Objicrr Sampling hesalie))
"Pig"
CLIP
CLIP
1-
oby
Rendered Imgs (') of *8 M,")
"perfume"
Render:
Dynomic
CLIP
obg
Rasterioct
Trained Object-
"sofa"
Cropped
Objoct Poul
Statie
specific Gaussians
Object Masks
Figure Demonstration of Segment then Splat pipeline. We first extract multi- masks for each object through robust tracking module, then object IDs are assigned to each initial Gaussian based on these masks, forming distinet object-specific sets. During optimization, object specific loss Lubj is used to enforce Gaussian-object correspondence and thus resulting in more accurate object geometries. Finally, a CLIP embedding is assigned to each Gaussian group for open-vocabulary queries.
before performing the reconstruction process. By enforcing object-Gaussing correspondence, our method achieves more accurate object geometries and compatibility with dynamic scenes.
3 Method
We introduce Segment then Splat, a unified approach for 3D open- vocabulayy segmentation based Gaussian Splatting, illustrated in Fig. The process begins by extracting multi-view masks for each object through robust object tracking module (Sec. 3.2) ensuring reliable detection and is assigned object ID according to these masks, partitioning the entire scene into multiple object-specifec Gaussian sets (See. 3.3 During optimization & reconstruction (See. B.4). each Gaussian contributes exclusively to its assigned object, preserving Gaussian-obiee correspondence and resulting in more accurate object geometries. After reconstruction, CLIP embeddings are associated with each group of Gaussians (Sec. 3.5). enabling open open-vocabulary queries. Besides, three levels of granularity (large, middle, and small) are introduced to facilitate object retrieval different scales.
mask generation. Next, each Gaussian initialized by COLMAP 37 38
3.1 Preliminary: 3D and 4D Gaussian Splatting
3DGS. 3D Gaussian Splatting represents scene using a collection of 3D ellipsoids, each modeled as an anisotropic 31D Gaussian. Each Gaussian is parameterized by mean æ, which defines the center of the ellipsoid, a covariance matrix Σ, which determines its shape, as shown in Eq. (T. The color of the Gaussian is defined using spherical harmonics.
G(x) (()))>
During rendering, the 3D Gaussians are first transformed into camera coordinates and projected onto the image plane as 2D Gaussians. The final pixel color is then computed through alpha blending, which integrates the weighted Gaussian colors from front to back:
Ō=
Σ C:O: (1- TEN
j=1
where G is the color of each Gaussian, and o; is the alpha value of the ith Gaussian.
4DGS. 4D Gaussian Splatting is an extension 3D Gaussian Splatting, capable of modeling dynamic scenes. Following Deformable 3D Gaussian Splatting [17], we incorporate deformation field to capture scene dynamics:
(ōx, år, ds) = (())
where Fo represents deformation field, which takes Gaussian mean a and time as input and outputs the deformation (ōx, ör, òs) time t. 7() denotes positional encoding.

---

3.2 Robust Object Tracking
Given set of input images {4i our goal is to extract multi-view masks for all objects at different granularity levels (i.e., large, middle and small) in the scene. We begin by leveraging Segment Anything (SAM) [9] with grid-based point prompting to obtain initial static object masks of different granularities in the first input frame Tg. Subsequently, SAM2 [40] is employed to track objects throughout the sequence based on the extracted mask from To. However, this process presents several challenges: 1) Some objects may not appear in the first frame, leading to their exclusion from tracking. 2) Due to grid-based prompting, one pixel may be tracked multiple times into different masks, either belonging to a single object or a part of an object. 3) If an object temporarily disappears and reappears later in the scene, tracking may be lost. To overcome these issues and improve the robustness of object tracking, we propose three targeted post pesscesccsssngg strategies that dynamically detect new objects, resolve mask conflicts, and recover from tracking failures. These strategies transform SAM- -bsdddded segmentation into a flexible, scene-adaptive tracking pipeline, which serves as the critical foundation for subsequent steps in our methodology-
Detect Any New Objects. To capture newly appearing objects, we introduce a detection mechanism at fixed intervals of At. Specifically, we compare the segmented region ratio between frames 4+ and 4. A significant decline in this ratio indicates the potential presence of new objects. At this point, we re-segment the scene and analyze the intersection between the new and previous segmentation results. Objects with minimal overlap with prior masks are identified as new objects and added to the static segmentation results. SAM2 then continues tracking based on this updated segmentation.
Resolving Multiple Trackings of Pixel. To ensure that each pixel is assigned to only one object within a given granularity level, we employ an Intersection over Union (loU)-based filtering approach. For each pair of masks in Ji if their loU exceeds a predefined threshold, the smaller object is discarded in favor of the larger one, as it can be segmented separately at finer granularity level.
Handling Lost Tracking. When an object's tracking is lost, it may be incorrectly treated as new object upon its reappearance in subsequent frames, leading to multiple instances representing the same object. In these scenarios, we resolve this issue using the approach described in Sec. 3.3
3.3 Object-Specific Gaussian Initialization
As our method follows a "segmentation then reconstruction" strategy, we first segment the Gaussians initialized by COLMAP into distinct sets, each representing different object. Each Gaussian is assigned three object IDs, corresponding to three granularity levels. To determine these IDs, we analyze the visibility of each Gaussian center across all views and identify the corresponding object mask region which it resides. After object IDs are determined, we handle the lost tracking issue stated in Sec. 3.2 When an object is tracked multiple times different instances, the corresponding Gaussians should share a similar geometric center and appearance (c.g. color). Therefore, we refine the segmentation by merging Gaussians and its corresponding object mask, if they exhibit closely aligned "gcometric-appeaaance distance", defined as follows:
d( G Gý) - Xa M, M, 2 + (1 A.) -
(4)
where G; and Gj are two sets of Gaussians representing different objects, M; and My denote the mean Gaussian centers, representing object geometric centers, and C; and C; are the average colors of the respective Gaussian sets. Aa is the weight to balance geometric distance and appearance distance. Additionally, since COLMAP provides only sparse reconstruction of the scene, some objects may not be covered and thus lack corresponding Gaussians. We compensate these missing objects by randomly initializing Gaussians. Furthermore, we generate set of background Gaussians to fill small unsegmented regions.
3.4 Optimization & Reconstruction
Optimization Goal. After initializing Gaussians as distinct object sets, they are used to reconstruct the scene. During this process, we enforce constraints to ensure that each set of Gaussians contributes only to its corresponding object. Specifically,, we introduce additional object-level loss term,
5

---

denoted as Lobj, alongside the standard rendering loss Lrender:
Genderr (1 X,) (oCc((4 Cds Lai = (M @ 1, 12).
(5)
(6)
where 1, is the rendered 2th image, and 1 and M. represent the rendered yd object in the 7th image and its corresponding mask, respectively, as described in Sec. S.c.3.2 @ denotes element-wise multiplication. The overall loss function is follows:
L Lrender + Lubj-
Throughout the entire reconstruction process, all densification and cloning operations are performed strictly within each object-specific Gaussian set, preserving the Gaussian-object correspondence. Besides, Gaussian persistence mechanism is designed ensure that each set of Gaussians is not com- pletely pruned, thereby preventing scenarios where an object could lose all its associated Gaussians.
Optimization Efficiency. Since the number of objects in scene can range from few to hundred, computing Lorj for every object becomes computationally infeasible, as it would require K times more rendering operations per iteration. To address this, randomly sample m objects per iteration to apply Latg. balancing efficiency and optimization effectiveness.
Optimization on Multiple Granularities. Noticed that we have three level granularities: large, middle and small. To ensure effective optimization, we must account for these varying scales when determining the optimization order. Since smaller objects are always part of larger ones, they should be optimized first. the order is reversed, i.e., optimizing smaller objects after larger ones, the internal structure of the larger objects may become disorganized again, shown in Fig. Thus the Cory will finally be formulated as:
Lobjs: Libj + Lobi_M,
stagel
Lobj
stage2
Lubj_s + Lubj + Labj L.
stage3
where Lobis, Lubj M. Lubj 1. represent per-object loss for small, middle, large level respectively.
Partial Mask Filtering. In 3D- aware segmentation, objects can be reconstructed even from views where they would typically appear occluded. Consequently, multi-view 2D masks provided supervision can introduce discrepancies, as they do not account for occluded regions revealed during rendering, as shown lead
Optimize small-level first
Optimize mid-level first
X
Small-Leve ID
102
102
Initallzzdd Sparss
101
103
Gaussians 103s
Initialioal Spance
101
162
Gaussiars with IDs
Superviso Small Objects
Supervise Mid-level Objects
in Fig. [5] This discrepancy to incorrect constraints, ultimately distorting the geometric structure of the 3D objects. The original 3DGS framework can partially resolve this issue by leveraging multi-view con- sistency, but errors from unreliable masks persist. To robustly address this, we propose partial mask fil- tering strategy applied at the end of training. Specifically, we render each reconstructed object into 2D images, compute their Intersection- Over-Union (IoU) against the provided masks, and discard masks exhibiting low IoU scores. This ensures that only consistent, accurate masks inform the final optimization, significantly enhancing the geometric fidelity of the reconstructed 3D objects.
107
102:
102
for
102
Iny
10t
112
101
lor
y03
181
101
183
102
102
Obfict
Chastic Small-leve une
Onject
Otject
Otjest
Otjest
Figure 3: demonstration how the optimization order af- fects reconstruction. Optimizing small-level objects first pre- serves both small- and middle-level structures, while starting with middle-level ones leads to well-maintained middle-level but chaotic small-level regions due to lack supervision.

---

SD OVS
3D OVB
Ourz
OpenGauastar
Gat
Dell
Torget Object Querles
Cad
Targes Object Queries
LEBF OVS
Rublec
Speen
Spelide
Tanget Object Quariees
Object Queries
Figure 4: Qualitative comparison on static scenes. Compared to baseline methods, our approach accurately retrieves the correct object and produces sharper segmentation boundaries. In contrast, 2D pixel-based methods exhibit ambiguous boundaries, while OpenGaussian either misses parts of the object incorrectly groups irrelevant objects together.
3.5 CLIP Embedding Association
Unlike previous methods [2 that supervise Gaussian language embeddings indirectly via feature maps, leading to inconsistent embeddings for Gaussians belonging to the same object, approach directly assigns a unified language embedding to each object-specific Gaussian set. This ensures consistent semantic embeddings within each object, significantly improving segmentation accuracy and boundary precision. The CLIP embedding of each object is calculated as follows:
fp
1
Σ M.
(Crop M. & I;));
T2 Mî
where fp is the language embedding of the pth object, Mfrr denotes the partial masks that are excluded in the previous section. CL)P; represents the CLIP image encoder and crop(.) denotes the cropping function extract the mask region.
Open-vocabulary segmentation. Given an input text prompt, we perform open vocabulary query following the below strategy:
Jn CLIP:
(10)
Greturn arg max cos(f fp),
(11)
where fy is the CLIP embedding of the input text prompt given by CLIP text encoder CLIP, (-). and gretm the object that best matches the query, determined by maximizing the cosine similarity cos (-) between the query embedding and the object embeddings.
4 Experiments
4.1
Setups
Baselines. We categorize the baselines into two groups based on their querying strategies:
pixel-bused segmentation and 3D-hased segmentation. For static scenes, we use LangSplat (10). LEGaussians [1 and Gaussian Grouping [41] as the 2D pixel-based basclines. For the 3D basclines, choose OpenGaussian 13]. and we also adapt LangSplat and LEGaussians for 3D segmentation by selecting Gaussians instead of pixels to evaluate their performance. In dynamic scenes, we adopt a zero-shot image segmentation model CLIP-LSeg 34 the 2D pixel-based baseline and DGD 33
as the 3D-based approach.
Datasets & Evaluation Metrics. To assess the segmentation performance of our proposed method, we conduct experiments on two static scene datasets (i.e., 3DOVS dataset 442 and LERF LERF_OVS dataset [9) and two dynamic scene datasets (i.e., HyperNeRF dataset 43 and Neu3D dataset
We use mean intersection over union (mloU) for open-vocabulary segmentation and report optimization time (in minutes) for training efffciincy.
44

---

Implementation Details. The new object detection stride At in the robust object tracking is set to 10. Following the original 3D Gaussian Splatting, we set A, in Cender to 0.2. For distance, we set X to 0.5. In each iteration, we sample object per granularity for 3DOVS to compute Lat and 3 objects per granularity for the remaining datasets. The mloU threshold for partial mask filtering is set to 30%. We train the smaller-scale 3DOVS data for 20K iterations, and larger-scale datasets (i.e.. LERF_ OVS, HyperNeRR, and Neu3D) for 40K iterations. All experiments conducted using RTX A6000 GPU.
LERF_OVS,
4.2 Veen--occbullry Query
2D V.S. 3D segmentation. One thing to be mentioned is that, due to the nature of 3D segmentation and the evaluation used for open- open- vocabulary segmentation, there is slight reduction our mloU score. This is because 3D segmentation directly retrieves the Gaussians associated with object, revealing occluded parts that are not visible some ground truth masks, as shown in Fig. 5
apen-vocabulary
Results on 3DOVS dataset. The quantitative results on the 3DOVS are presented in Tab. [ (a). Our method outperforms all baseline ap-
Table 1:
Quantitative segmentation results for static (a) and dynamic (b) scenes.
proaches. The qualitative comparison is shown
(a) Static scenes
in Fig. Notably, 2D pixel-based methods tend to produce relatively ambiguous bound- aries, whereas approach, leveraging the Seg- ment then Splat strategy, achieves significantly clearer object boundaries. For 3D segmenta- tion methods, OpenGaussian requires a prede- fined number of objects K for clustering. Since 3DOVS contains fewer objects compared to LERF OVS, manually reduced its preset K. However, even after this adjustment, Open- Gaussian still only retrieves parts of certain ob- jects. Although LangSplat and LEGaussians are modified segment Gaussians instead of pixels, their performance remains suboptimal. This is because each Gaussian's 's language embedding lacks direct supervision and may encode multi- ple object semantics, leading to inaccurate seg- mentation. Additionally, since our method follows single- pass reconstruction process and the scene scale is relatively small, our training time significantly shorter compared to the baselines.
LERE OVS
3DOVS
Mothod
mloU1
Timel
mloU1
Time
2D
LangSplai [0]
LEGaussians
46.37
18.79
72.00
62.00
52.12
82,49
68.90
55,90
G-Grouping
77.00
76.24
LangSplat [0
16.76
62.00
47,31
68.90
3D
Opentinussin
LEGaussian
42.43
69.75
72.00
51.00
59.40
Ours
52.10
50.75
88,53
9.40
Dynamic
HyperNeRF
Neu3D
Method
mloUf
Time.
mloUt
Time.
2D
LSeg
34
15.71
1.49
DGD Ours
331
7.83
1564.5
1.65
1733
69.48
218
44.00
161.3
Results on LERF _OVS dataset. The quan- titative results on the LERF OVS dataset are also presented in Tab. [ tive results are shown in Fig. Similar to the 3DOVS dataset, 2D pixel-based methods pro- duce less precise object boundaries, while our method demonstrates significantly improved re- sults. For 3D-based methods, OaenGaussian performs much better on LERF_OV ROO_ compared to 3DOVS. However, since OpenGaussian re- quires predefined number of clusters, some ob- jects may be incorrectly grouped together (c.g., the rubber duck and the Rubik's 's cube). More- over, due to its "splat then segmentation" strat- egy, its object boundaries remain less accurate than those produced by our method.
(a) and the qualita-
"Controller"
Original Image
Mask
2D Seg
Ours
Figure 5: Comparison between 2D pixel-based segmentation and our 3D segmentation. Unlike 2D pixel-based methods, which are limited by oc- clusions, our approach can retrieve the complete object even from occluded view.
Results on HyperNeRF dataset. The quantitative results and qualitative results on HyperNeRF dataset are presented Tab. (b) and Fig. respectively. Since our method explicitly enforces Gaussian-object correspondence, it can be directly applied to dynamic scenes, achieving good segmentation performance without the Gaussian-object misalignment issue encountered by previous approaches. In contrast, methods such DGD, which relies on learning language field supervised

---

HyperNoRF
r-t
odio
T=2
Hand
Object Queries
Chickon
Object
Chicken
Object Querles
Chiclien
DGO
Ours
T
Ours
Man
Man
Banny
Bunny
Object Queries
Object Querles
Figure 6: Qualitative comparison dynamic scenes. As method enforce object-Gaussial correspondence, applies directly to dynamic scenes and performs well, whereas DGD and LSeg tend to include irrelevant content and are not able to retrieve small objects (e.g., bunny painting).
via 2D feature maps, suffer from misalignment, as a single Gaussian might represent multiple distinct objects across different time steps. As illustrated in Fig. 6 this misalignment results in retrieving irrelevant Gaussians. Moreover, because DGD does directly supervise the language embeddings of each Gaussian, Gaussians located far apart may share similar embeddings, further deteriorating segmentation quality. addition, our method achieves nearly a ten-fold improvement optimization speed compared to DGD, as learning dynamic language field is computationally intensive. We omit training time results for LSeg, as is zero-shot method requiring no additional optimization.
Results on Neu3D dataset. Quantitative results and qualita- tive results are shown in Tab. I and Fig. 6 Similar to the observa- tions on the HyperNeRR dataset, many irrelevant Gaussians are re- trieved due to object-Gaussiian misalignment issue and the "splat then segment" strategy. Besides, both LSeg and DGD fail when re- trieving relatively small objects (e.g., bunny painting).
Table 2: Ablation studies: (Top) Number of supervised objects per iteration. (Middle) Partial mask filtering. (Bottom) Tracking module ablation.
(a) Number of supervised objects per iteration
Static Scene
Dynamic Scene
kitehen
chickchicken
split-cookic
oob
Time!
mloU?
Time
mloU?
Time.
mioU?
Time]
mloU?
26
$1.09
44
33.97
146
73.11
180
77.76
202
42
55.61
10.83
172
75.29
253
80.17
53
56.03
63
40.77
185
75.21
258
81.52
62
81.74
Partial mask filtering strategy
Static Semne
Dynamic Scene
4.3 Ablation Study
ramen
Jitchen
chickobicken
split cookie
Method
mloUt
mintit
mioUt
mlloff
PSNRT
Number of Supervised Objects. In this ablation study, we ex- amine the relationship between the number of supervised ob- jects in each granularity per iter- ation and the segmentation per- formance. We pick two static scenes from LERF_ OVS and two dynamic scenes from HyperNeRR dataset. The results are presented in Tab. (a). As expected, increasing the number of supervised objects per iteration generally enhances segmentation performance. However, beyond certain threshold, the performance gain becomes marginal. scenes with fewer objects (e.g., chickchicken and split-cookie), performance quickly converges after certain number of supervised objects. In contrast, more complex scenes containing many objects (e.g., ramen and waldo kitchen) continue to show performance improvements, though at diminished rate. Additionally, training time increases
who MP
42.19
23.11
31.94
30.13
72.65
29.32
80.23
32.76
Ours
54.38
24.54
40.71
31,33
74,89
29.02
80,30
33,04
(c) Tracking module ablation
Methed
teatime
figurines ORR1
split-cookle
Dup
Dup
Dup
ORR?
Dupl.
SAM2
0.889
01858
0.683
4
0.970
0
0,934
0.96.
18
1,000
0.961
0.948
1.000
+lost track
0.934
0.961
0
0.948
0
1,000

---

proportionally with the number of supervised objects. To balance training time and performance, we choose three objects for LERF OVS, Neu3D and HyperNeRF in our experiments.
Partial Mask Filtering, In this ablation study, we investigate the impact of the partial mask filtering strategy segmentation performance well reconstruction quality. As shown in Tab. 2 (b), scenes with larger number of objects (e.g., ramen, kitchen tend to have more occluded views, leading more performance gain when applying partial mask filtering. contrast, scenes with fewer objects (e.g., chickchicken, split-cookie) exhibit smaller performance gain. Overall, the results validate the effectiveness of the partial mask filtering strategy, particularly in complex scenes with high object density.
Robust Object Tracking. We conduct ablation study each component of our robust object tracking module, as shown in Tab. (c). The evaluation performed on three static scenes from LERF_ OVS and dynamic scene from HyperNeRF. Leveraging ground-truth labels, adopt two
metrics: Object Recall Rate (ORR), defined as
number of tracked objects
ORR
(12)
number of GT objects
where k is the number of ground-truth frames, measures the percentage of objects successfully tracked throughout the sequence; and the number of duplicate trackings (Dup). which quantifies how often the same object is redundantly tracked as multiple instances. Results show that the new object detection module increases ORR by capturing newly appeared objects, albeit the cost of more duplicate instances. In contrast, the multi-track and lost-track handling modules effectively reduce Dup, as described in Sec. 3.2 The slight drop in ORR observed in the Figurines scene when adding the multi track module is caused by an isolated tracking failure at a fine- iiee-graned level in a single frame. However, this minor failure does not affect the final reconstruction, as sufficient information is retained from other views.
5 Conclusion
We propose Segment then Splat, a unified framework for 3D neee-vocabulary segmentation based on Gaussian Splatting. We reverse the long-established "segment after reconstruct" approach to form a "segment then reconstruct" pipeline. By maintaining consistent object- Sajject-Gaussian correspondence throughout the process, Segment then Splat effectively eliminates both geometric and semantic ambiguities, resulting in more precise object boundaries and improved segmentation performance. Furthermore, because this correspondence is explicitly established, our method naturally extends to dynamic scenes without concerns of misalignment between objects and Gaussians during dynamic modeling. Extensive experiments across diverse static and dynamic datasets demonstrate the superior performance and robustness of our framework.
6 Limitation and Future Direction
Our method relies on SAM2 for initializing object tracking. In extremely complex scenarios with high-density and visually similar objects. tracking may fail, resulting in suboptimal open-vocabulary segmentation, which is an issue commonly shared by tracking-based approaches (e.g., Gaussian Grouping). This limitation could potentially be addressed by incorporating techniques such as Kalman filtering (45) to improve the temporal stability of SAM2. Another limitation is that our method cannot effectively handle text queries involving relational descriptions across multiple objects, such as "a sheep sitting on the chair in front the table." Similar to existing baselines, method encodes textual embeddings from masked regions local patches corresponding to individual objects, and therefore lacks explicit modeling of inter-object relationships or contextual understanding. Developing this capability is crucial for advancing open- vecabuacay understanding, and we plan to explore it in future work.
10

---

References
[1] Dexter Ong, Yuezhan Tao, Varun Murali, Igor Spasojevic, Vijay Kumar, and Pratik Chaudhari. Atlas navigator: Active task-driven language-embedded gaussian splatting. arXiv preprint arXiv:2502.20386, 2025.
[2] Guangzhao Dai, Jian Zhao, Yuantao Chen, Yusen Qin, Hao Zhao, Guosen Xie, Yazhou Yao, Xiangbo Shu, and Xuelong Li. Unitedvin: Generalizable gaussian splatting for continuous vision-language navigation. arXiv preprint arXiv:2411.1605. 2024.
[3] Keiko Nagami, Timothy Chen, Javier Yu, Ola Shorinwa, Maximilian Adang, Carlyn Dougherty, Eric Cristofalo, and Mac Schwager. Vista: Open-vocabulle task-relevant robot exploration with online semantic gaussian splatting. arXiv preprint (iXi::2557.0122555 2025.
[4] Adam Rashid, Satvik Sharma, Chung Min Kim, Justin Kerr, Lawrence Yunliang Chen, Angjoo Kanazawa, and Ken Goldberg. Language embedded radiance fields for shot task-oriented grasping. In 7th Annual Conference on Robot Learning. 2023. /tttts://opprreviii net/ Paaerrummid----P8I000mm
[5] William Shen, Ge Yang, Alan Yu, Jansen Wong, Leslie Pack Kaelbling, and Phillip Isola. Distilled feature fields enable few-shot language- -uiddeeggdddd manipulation. In 7th Annual Conference on Robot Learning, 2023.
[6] Yuhang Zheng, Xiangyu Chen, Yupeng Zheng, Songen Gu, Runyi Yang. Bu Jin, Pengfei Li, Chengliang Zhong, Zengmao Wang. Lina Liu, et al. Gaussiangrasper: 3d language gaussian splatting for open-vocabulary robotic grasping. IEEE Robotics and Automation Letters, 2024.
[7] Mazeyu Ji, Ri-Zhao Qiu, Xueyan Zou, and Xiaolong Wang. Graspsplats: Efficient manipulation with 3d feature splatting. In Conference on Robot Learning, pages 1443-1460, PMLR, 2025.
[8] Bernhard Kerbl, Georgios Kopanas, Thomas Leimküühlrr, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions Graphics, 42(4), July 2023. URL htttlhttts////tttt /tttpp::/rreo--eee sam. inria. fr fr/fungraph//3d 3d
[9] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In International Conference on Computer Vision (ICCV), 2023.
[10] Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang. and Hanspeter Pfister. Langsplat: 3d language gaussian splatting. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 20051-20060, 2024.
[11] Jin-Chuan Shi, Miao Wang, Hao-Bin Duan, and Shao-Hua Guan. Language embedded 3d gaus- sians for open-vocabulary scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5333-5343, 2024.
[12] Shijie Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang, and Achuta Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21676 11776-2166 21685, 2024.
[13] Yanmin Wu, Jiarui Meng, Haijie Li, Chenming Wu, Yahao Shi, Xinhua Cheng, Chen Zhao, Haocheng Feng, Errui Ding, Jingdong Wang. et al. Opengaussian: Towards point-level 3d gaussian- based open vocabulary understanding. arXiv preprint crXiv:2406.02058, 2024.
[14] Umangi Jain, Ashkan Mirzaei, and Igor Gilitschenski. Gaussiancut: Interactive segmentation via graph cut for 3d gaussian splatting. Advances in Neural Information Processing Systems, 2025.
[15] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 2024 International Conference on 3D Vision (3DV), pages 800-809. IEEE, 2024.

---

[16] Zeyu Yang. Hongye Yang, Zijie Pan. and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. In International Conference on Learning Representations (ICLR), 2024.
[17] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In Proceedings of the IEEE/CVF conference computer vision and paltern recognition, pages 20331-20341, 2024.
[18] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 203 10310-203220 20320, 2024.
[19] Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 130-141, 2023.
[20] Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8508-8520, 2024.
[21] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference machine learning. pages 8748-8763. PmLR, 2021.
[22] Songyou Peng, Kyle Genova, Chiyu "Max" Jiang, Andrea Tagliasacchi, Mare Pollefeys, and Thomas Funkhouser. Openscene: 3d scene understanding with open vocabularies. In CVPR, 2023.
[23] Ayça Takmaz, Elisabetta Fedele, Robert W. Sumner, Mare Pollefeys, Federico Tombari, and Francis Engelmann. OpenMask3DD: Open-Vocabulary 3D Instance Segmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[24] Rui Huang, Songyou Peng, Ayca Takmaz, Federico Tombari, Marc Pollefeys, Shiji Song, Gao Huang, and Francis Engelmann. Segment3d: Learning fine-grained class-agnostic 3d segmentation without manual labels. European Conference on Computer Vision (ECCV), 2024.
[25] Yuliang Sun, Xudong Zhang, and Yongwei Miao. A review of point cloud segmentation for understanding 3d indoor scenes. Visual Intelligence, 2(1): 14, 2024.
[26] Yizhou Wang, Longguang Wang, Qingyong Hu, Yan Liu, Ye Zhang, and Yulan Guo. Panoptic segmentation of 3d point clouds with gaussian mixture model in outdoor scenes. Visual Intelligence, 201):10, 2024.
[27] Wenbo Zhang, Lu Zhang, Ping Hu, Liqian Ma, Yunzhi Zhuge, and Huchuan Lu. Boot- straping clustering of gaussians for view-consistent 3d scene understanding. arXiv preprint arXiv:2411.19551, 2024.
[28] Xingxing Zuo, Pouya Samangouei, Yunwen Zhou, Yan Di, and Mingyang Li. Fmgs: Foundation model embedded 3d gaussian splatting for holistic 3d scene understanding. International Journal of Computer Vision, pages 1-17, 2024.
(29] Ri-Zhao Qiu, Ge Yang, Weijia Zeng, and Xiaolong Wang. Feature splatting: Language-driven physics-based scene synthesis and editing. arXiv preprint urXiv:2404.01223. 2024.
[30] Yuri Y Boykov and M-P Jolly. Interactive graph cuts for optimal boundary & region segmenta- tion of objects in nd images. In Proceedings eighth IEEE international conference on computer vision. ICCV 2001, volume pages 105-112 IEEE, 2001.
[31] Lester Randolph Ford and Delbert Ray Fulkerson. Flows in networks. 2015.
[32] Andrew V Goldberg and Robert E Tarjan. A new approach to the maximum-fiee problem. Journal of the ACM (JACM), 35(4):921-940, 1988.
12

---

[33] Isaac Labe, Noam Issachar, Itai Lang, and Sagie Benaim. Dgd: Dynamic 3d gaussians distillation. European Conference on Computer Vision, pages 361-378. Springer, 2024.
[34] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language- driven semantic segmentation. arXiv preprini arXiv:2201.03544 2022.
[35] Wanhua Li, Renping Zhou, Jiawei Zhou, Yingwei Song, Johannes Herter, Minghan Qin, Gao Huang, and Hanspeter Pfister. 4d langsplat: 4d language gaussian splatting via multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 22001-22011, 2025.
[36] Gal Fiebelman, Tamir Cohen, Ayellet Morgenstern, Peter Hedman, and Hadar Averbuch-Elow 4-legs: 4d language embedded gaussian splatting. In Computer Graphics Forum, page e70085. Wiley Online Library.
[37] Johannes Lutz Schönberger, Enliang Zheng, Mare Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. European Conference on Computer Vision (ECCV), 2016.
[38] Johannes Lutz Schönberger and Jan-Michacl Frahm. cenettre-ffrommmene revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[39] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yer Lo, al. Segment anything. In Proceedings of the IEEF/CVF international conference on computer vision, pages 4015-4026, 2023.
[40] Nikhila Ravi, Valentin Gabeur, Yuan- Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma. Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.0m 00784 14, 2024. httptttss///are wwwpiii.vor//bbaae org/abs/ wwwp///...0.... 00714
[41] Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke. Gaussian grouping: Segment and edit anything in 3d scenes. In ECCV, 2024.
[42] Kunhao Liu, Fangneng Zhan, Jiahui Zhang, Muyu Xu, Yingchen Yu, Abdulmotaleb El Sad- dik, Christian Theobalt, Eric Xing, and Shijian Lu. Weakly supervised 3d open-vocabulary segmentation. Advances in Neural Information Processing Systems, 36:53433-53456, 2023.
[43] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan Goldman, Ricardo Martin-Brualle and Steven M. Seitz. Hypemner: A representation for topologically varying neural radiance fields. ACM Trans. Graph. 40(6), dec 2021.
[44] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5521-5531 2022.
[45] Cheng- Yen Yang, Hsiang- Wei Huang, Wenhao Chai, Zhongyu Jiang, and Jenq-Neng Hwang. Samurai: Adapting segment anything model for zero-shot visual tracking with motion-aware memory, 2024. URL http::////aaat / org/ argg/bbs/21111 11922
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims made in the abstract and introduction accurately reflect paper's contribution and scope.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See the limitations in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Implementation detail included in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
The code is released at http:://gttubbbbommmuyyy///eeeet
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cpippi.ccppulli//uuuii wwwpc//ggue////eee CoodeSubeiiieeee ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training and test details are provided in Sec. 4.1 Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We provide numerical mean for all the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources are stated in Sec. 4.1 implementation details and time of execution is included in Sec. 4.2 experiment results
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: As discussed in Sec. [ 3D open-vocabulary querying enables richer scene un- derstanding which can benefit applications like autonomous driving and robot manipulation.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not release data or model that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the datasets used in the paper cited.
Guidelines:
existing assets. The authors should cite the original paper that produced the code package or dataset.
The answer NA means that the paper does
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The paper does not involve LLMs any important components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neuriie 220p:////c...eceeeeeeee//22////0 for what should or should not be described.
20


Reviews:
[{"review_id": "wP4pU16QRM", "reviewer": "Reviewer_5uyD", "summary": "The paper introduces \"Segment then Splat,\" a novel 3D open-vocabulary segmentation method using Gaussian Splatting that segments objects before reconstruction. This reverses the traditional \"splat then segment\" approach, maintaining precise object-Gaussian correspondence for improved 3D geometry and segmentation accuracy. It supports both static and dynamic scenes by preventing Gaussian-object misalignment and includes a robust object tracking module for spatial-temporal consistency. By associating CLIP embeddings with object-specific Gaussians, it enables efficient open-vocabulary querying without extra language fields. Experiments show state-of-the-art performance in accuracy and efficiency across diverse datasets", "questions": "1. I noticed that the training process adopts a three-stage optimization strategy, progressively encouraging the model to focus on objects of increasing scale, from small parts to larger wholes. However, the paper seems to lack a detailed experimental analysis that specifically evaluates the model’s performance on different object hierarchies or granularities, particularly for relatively smaller subparts. I am especially interested in understanding whether the model can accurately identify and segment these fine-grained, smaller objects (subparts) with precision comparable to larger objects. Such an analysis would provide valuable insights into the robustness and granularity of the proposed segmentation approach.\n\n2. If possible, I would like to hear an explanation about Weaknesses 3 above. Of course, if my understanding is incorrect, please feel free to point it out.\n\n3. During training, the number of supervised objects per iteration is identified as a crucial hyperparameter influencing segmentation performance, as shown in the ablation study in paper. It appears that in none of the static scenes does performance fully saturate with increasing numbers of supervised objects. I wonder if this observation might be due to insufficient supervision over certain part and subpart-level objects, which may dilute the learning signal when only random sampling is employed. If possible, I kindly request the authors to provide some analysis or insight into this aspect.\n\n4. It would be helpful if the authors could include some failure cases or typical scenarios where the proposed method does not perform satisfactorily, in order to better understand its limitations and potential areas for improvement.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "#### Strengths:\n\n1. The paper is clearly written and well-structured, making it easy to follow.\n\n2. The concept of “Segment then Splat” represents a highly innovative and bold departure from the conventional strategy of explicitly modeling semantic fields in open-vocabulary Gaussian approaches. By reversing the traditional “splat then segment” paradigm, this method fundamentally eliminates the reliance on separate language fields for semantic understanding. Moreover, it effectively addresses challenges such as the misalignment between Gaussians and objects, particularly prominent in dynamic scenes. The subsequent experimental results robustly validate the effectiveness of this approach, demonstrating its practical viability and advantages over prior methods.\n\n3. The experimental performance is impressive: across a variety of both static and dynamic datasets, Segment then Splat achieves state-of-the-art results in 3D open-vocabulary segmentation as well as computational efficiency. Extensive qualitative visualizations further illustrate the model’s ability to preserve the integrity of 3D structures while delivering sharp, well-defined object boundaries. \n\n4. Another strength of the proposed framework is its natural compatibility with dynamic scenes. By maintaining a strict one-to-one correspondence between Gaussians and objects, the method effectively prevents Gaussian-object misalignment caused by object motion. This capability allows Segment then Splat to seamlessly unify the treatment of both static and dynamic environments, overcoming a major limitation of existing methods that struggle with time-varying or moving objects.\n\n#### Weaknesses:\n\n1. Although the paper adopts the Segment then Splat approach to avoid explicitly incorporating semantic parameters into Gaussians, this increases complexity during the preprocessing stage. Additionally, at inference time, the model still requires some matching method to assign semantic information to Gaussians. This leads to a more fragmented model pipeline rather than an end-to-end framework.\n\n2. For more complex scenes, compared to explicitly embedding semantic information into Gaussians, this method may be more difficult to handle. This is because the grouping prior relies heavily on high-precision grouping initialization and complex tracking logic, which may be fragile or error-prone in challenging scenarios.\n\n3. If I understand correctly, during experiments comparing with baselines, the ground truth used is at the object-level rather than pixel-level. This might introduce some unfairness to methods optimized at pixel-level such as LangSplat. If my understanding is incorrect, please kindly correct me. Otherwise, providing additional quantitative comparison with pixel-level ground truth would be beneficial. Of course, due to techniques like Partial Mask Filtering used during training, some performance gap is expected and a slight disadvantage could be acceptable.\n\n4. Although the paper includes a section discussing limitations, it does not comprehensively illustrate failure cases or conditions under which the method’s performance degrades. Presenting such results would help readers gain a more complete understanding of the model’s robustness and boundaries.", "was_revised": true, "final_justification": "Overall, I find the method proposed in this paper to be novel. Although it inevitably involves a lot of engineering and somewhat complicated design, I lean towards a weak accept.", "rebuttal": "Thank you for your thoughtful and encouraging review. We sincerely appreciate your valuable feedback and your recognition of our method's novelty and strong performance. We are delighted that you highlighted the innovation of the \"Segment then Splat\" concept and acknowledged its advantages for dynamic scenes. Your support is very motivating.\n\n# \\[W1\\] More fragmented model pipeline.\n\nAlthough our method introduces a more complex preprocessing step, it eliminates the need to train an additional feature field. Moreover, the CLIP embedding association is performed during training, not at inference time. At inference, we simply compare the input text embedding with the precomputed embeddings of each Gaussian group to retrieve the most relevant object, ensuring an efficient and lightweight retrieval process.\n\n# \\[W2\\] Complex scenes may be challenging.\n\nIn complex scenes where semantic information is dense and noisy, language field-based methods tend to struggle with entangled representations. In contrast, our approach associates language embedding with object-specific Gaussian groups, leading to more robust and disentangled representation.\n\nThe tracking logic has shown reliable performance on LERF-OVS, 3DOVS, Neu3D and HyperNeRF dataset. We conduct additional experiment on MipNeRF360 dataset, which contains scenes with hundreds of cluttered objects (i.e. Counter and Bonsai), to demonstrate the effectiveness and reliability of our approach:\n\n| Method        | Counter mIoU | Time | Bonsai mIoU | Time | Kitchen mIoU | Time | Garden mIoU | Time | Room mIoU | Time | Avg mIoU | Avg Time |\n|---------------|--------------|------|-------------|------|---------------|------|--------------|------|------------|------|----------|-----------|\n| Ours          | **55.32**    |**36.0**|**63.80**    |**45.0**|**68.59**      |**48.0**|**44.09**      |**86.0**| 42.90      | 44.0 |**54.94** |**51.80**   |\n| OpenGaussian  | 49.57        | 48.0 | 39.08       | 58.0 | 31.22         | 60.0 | 15.10        | 130.0| 41.88      |**42.0**| 35.37    | 67.60     |\n| G-Grouping    | 41.76        | 50.0 | 56.36       | 62.0 | 58.15         | 71.0 | 36.25        | 128.0| 30.84      | 63.0 | 44.67    | 74.80     |\n| LEGaussian    | 54.98        | 50.0 | 62.46       | 52.0 | 66.77         | 60.0 | 40.12        | 103.0|**45.58**    | 61.0 | 53.98    | 65.20     |\n\n# \\[W3, Q2\\] Object-level ground truth is not fair for those pixel-level methods. \n\nSorry for the confusion. During the experiments the ground truth used is actually **pixel-level**, not object-level. This aligns with the original setting of all the baselines. For 3D comparison with LangSplat and LEGaussian, we modify their inference code to extract text-related Gaussians instead of pixels and render the extracted Gaussians to image plan to evaluate on pixel-level ground truth. Based on this evaluation strategy, the performance of the original baselines will not degrade, thus the comparison is fair for baseline methods. While our approach may suffer from small performance degradation due to the pixel-level ground truth.\n\n# \\[Q1\\] Performance analysis on different object granularities.\n\nThank you for pointing this out. We separate the ground truth objects in LERF\\_OVS dataset into two categories: 1\\) large/whole objects (e.g. ramen, bear, sheep), represented by (L), 2\\) small/sub-parts (e.g. onion segment (in the ramen), bear nose, hooves (of the sheep)), represented by (S). And provide the quantitative results in the following table: \n\n| Method         | Ramen (L) | Ramen (S) | Kitchen (L) | Kitchen (S) | Teatime (L) | Teatime (S) | Figurines (L) | Figurines (S) | Avg (L) | Avg (S) |\n|----------------|-----------|-----------|--------------------|--------------------|-------------|-------------|----------------|----------------|---------|---------|\n| Ours           | **47.24** | **59.22** | **51.60**          | **3.68**           | **69.42**   | **53.58**   | 54.61          | 33.11          | **55.72**|**41.06**|\n| OpenGaussian   | 27.85     | 0.16      | 41.89              | 0.00               | 64.67       | 52.82       | **59.96**      | **33.57**      | 48.59   | 27.03   |\n| G-Grouping     | 47.17     | 2.26      | 15.01              | 0.00               | 60.37       | 37.90       | 59.02          | 1.68           | 45.39   | 17.45   |\n| LEGaussian     | 18.69     | 4.69      | 23.00              | 0.00               | 42.38       | 48.73       | 0.90           | 1.13           | 21.24   | 15.16   |\n\n\n# \\[Q3\\] Supervision may be insufficient due to object random sampling.\n\nThank you for the excellent point. You are correct that random sampling can dilute the supervision signal. In our settings, we chose to supervise 3 objects to balance the performance and optimization efficiency. As shown in Tab.2(a), while performance does not fully saturate, we observe a clear convergence trend: the improvement from supervising 1 to 3 objects is substantial, while gains from 3 to 5 (and beyond) become marginal. In scenes with a large number of objects (e.g., *Ramen*), achieving full saturation may require supervising more objects or running additional optimization iterations.\n\n# \\[W4, Q4\\] Limitations and potential improvements.\n\nOne limitation is that sometimes the associated embeddings with each Gaussian set cannot be successfully queried by text-input. This is likely due to our current use of a simple averaging strategy when assigning CLIP embeddings. In cases where an object is heavily occluded in certain viewpoints, the embedding from that view may ruin the entire object embedding and result in query failure. This is also a future direction, developing a more robust and adaptive embedding assignment strategy could further enhance retrieval accuracy and overall performance."}, {"review_id": "2e02TFxRX6", "reviewer": "Reviewer_TP1s", "summary": "The paper proposes a 3D open-vocabulary segmentation method called \"Segment then Splat\", which reverses the traditional \"Splat-then-Segment\" method. Instead of conducting segmentation after scene reconstruction, the method segments objects at the 3D Gaussian initialization stage, ensuring object-specific Gaussian sets before reconstruction. After reconstruction, semantic embeddings from CLIP are assigned to each object to enable open-vocabulary querying.", "questions": "1. Scalability: \n\nHow does the method perform on large-scale scenes with many objects? Does the computational cost scale linearly or non-linearly with the number of objects?\n\n2. Robustness of Object Tracking: \n\nIn cases where SAM2 fails to track objects accurately (e.g., cluttered scenes or objects with similar appearances), how does this affect the segmentation and reconstruction performances?\n\n3. Dynamic Scene Complexity: \n\nHow does the method handle highly dynamic scenes with significant object deformation or occlusion? Are there some failure cases?\n\n4. Post-Processing Dependence: \n\nThe partial mask filtering strategy appears to be a post-hoc correction for mask inaccuracies. How significant is its impact on the final results, and does this suggest limitations in the present pipeline?", "limitations": "Yes, the authors have discussed the limitations on page 9.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. Reversal of the Traditional Pipeline: \n\nThe proposed \"Segment then Splat\" pipeline introduces a straightforward modification to the traditional \"Splat-then-Segment\" approach, addressing issues like Gaussian-object misalignment in dynamic scenes.\n\n2. Dynamic Scene Support: \n\nBy enforcing object-Gaussian correspondence and using a strong object tracking module, the method gains the ability to handle dynamic scenes better than other methods.\n\n3. Comprehensive Evaluation: \n\nExperiments on both static and dynamic datasets provide sufficient evidence of the method’s effectiveness compared to other methods.\n\n4. Qualitative Results: \n\nThe paper demonstrates sharper segmentation boundaries and better object retrieval performance, particularly in cases where previous methods struggled with multi-object scenes or dynamic scenes.\n\n**Weaknesses**\n\n1. Limited Novelty: \n\nThe \"Segment then Splat\" pipeline is conceptually simple, consisting of two straightforward steps:\n- Perform 3D segmentation on point clouds before reconstruction.\n- Use CLIP embeddings for semantic matching.\nThis method lacks a unique technique design, as it essentially combines existing techniques (SfM-based segmentation + CLIP-based matching) in a straightforward manner.\n\n2. Pipeline Complexity: \n\nThe method requires per-object 3DGS reconstruction, which makes the pipeline computationally expensive and potentially impractical for large-scale or complex scenes with many objects.\n\n3. Scalability Concerns: \n\nThe computational efficiency of the method for scenes with hundreds or thousands of objects is unclear, as the need for object-specific Gaussian reconstruction could introduce lots of overhead.\n\n4. Lacking Discussions about the OV 3D Detection settings\n\nConsidering the research contexts are very related, the submission should also discuss the relation with methods of OV-3DDet settings [1, 2, 3]. I'm curious about their differences and whether this method has the potential for OV-3DDet applications\n\n\n[1] Pengkun Jiao, Na Zhao, Jingjing Chen, and Yu-Gang Jiang. Unlocking Textual and Visual Wisdom: Open-Vocabulary 3D Object Detection Enhanced by Comprehensive Guidance from Text and Image. In ECCV, 2025.\n\n[2] Yang Cao, Yihan Zeng, Hang Xu, and Dan Xu. CoDA: Collaborative Novel Box Discovery and Cross-Modal Alignment for Open-Vocabulary 3D Object Detection. In NeurIPS 2023.\n\n[3] Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. Open-Vocabulary Point-Cloud Object Detection without 3D Annotation. In CVPR 2023", "was_revised": true, "final_justification": "I have carefully read the rebuttal. Thank the authors for their efforts and thorough analysis. Most of my concerns have been addressed. However, I still think the novelty of this work is somewhat limited. As they pointed out, the key idea of this work is to \"reverse the long-established 'Splat then Segment' approach.\" From my perspective, this represents only a minor contribution to novelty, as it appears to be a relatively straightforward modification. Nonetheless, I appreciate the experiments they conducted to address the concerns regarding efficiency.\n\nIn addition, the responses to Q2, Q3, and Q4 have effectively resolved my concerns. Finally, I would suggest including more detailed discussions on Open-Vocabulary 3D detection in your revision. Doing so will not diminish the contribution but will instead benefit the community in terms of both OV-3DSeg and OV-3DDet.\n\nOverall, I will raise my score to 4.", "rebuttal": "We thank the reviewer for the thoughtful feedback and for recognizing our proposed “Segment-then-Splat” approach as a novel reversal of the traditional pipeline, the comprehensiveness of our evaluation, and our method’s ability to support dynamic scenes. We would like to take this opportunity to clarify the following points.\n\n# \\[W1\\] Novelty\n\nWe appreciate the reviewer’s perspective. While our “Segment then Splat” pipeline may appear conceptually simple, we would like to argue in the following aspects:\n\n1) We reverse the long established “Splat then Segment” approach in Gaussian Splatting based open-vocabulary segmentation tasks.   \n2) Based on (1) we establish an object-centric reconstruction approach, achieving a better object boundary localization as shown in Fig.4 compared to baseline “Splat then Segment” methods. Unlike prior methods, our approach does not require training an additional feature field, making it more efficient and easier to optimize.  \n3) Based on (1), we eliminate the Gaussian-object misalignment issue in dynamic modeling, thus can be directly applied to dynamic scenes without additional modification to form a unified 3D open-vocabulary segmentation approach.\n\n# \\[W2, W3, Q1\\] Efficiency, Scalability and Performance on Larger Scenes\n\nThank you for pointing this out. In the current implementation we proposed a random sampling strategy to compensate for the overhead introduced by object-specific supervision during training. To evaluate the scalability of our method, we conducted additional experiments on the MipNeRF360 \\[2\\] dataset, which includes highly cluttered scenes with hundreds of objects (e.g., Bonsai and Counter). The performance and efficiency metrics are summarized in the following table:\n\n| Method        | Counter mIoU | Time | Bonsai mIoU | Time | Kitchen mIoU | Time | Garden mIoU | Time | Room mIoU | Time | Avg mIoU | Avg Time |\n|---------------|--------------|------|-------------|------|---------------|------|--------------|------|------------|------|----------|-----------|\n| Ours          | **55.32**    |**36.0**|**63.80**    |**45.0**|**68.59**      |**48.0**|**44.09**      |**86.0**| 42.90      | 44.0 |**54.94** |**51.80**   |\n| OpenGaussian  | 49.57        | 48.0 | 39.08       | 58.0 | 31.22         | 60.0 | 15.10        | 130.0| 41.88      |**42.0**| 35.37    | 67.60     |\n| G-Grouping    | 41.76        | 50.0 | 56.36       | 62.0 | 58.15         | 71.0 | 36.25        | 128.0| 30.84      | 63.0 | 44.67    | 74.80     |\n| LEGaussian    | 54.98        | 50.0 | 62.46       | 52.0 | 66.77         | 60.0 | 40.12        | 103.0|**45.58**    | 61.0 | 53.98    | 65.20     |\n\nAs shown, our method outperforms baseline approaches without any modification to the training pipeline, while maintaining superior efficiency. This suggests that our random sampling strategy effectively balances supervision and training cost, even in more complex scenes.\n\nHowever, such overhead on larger scenes can still increase linearly if we need either more objects to be sampled per iteration or extend the training iteration when facing extreme cases.   \nTo further improve the training efficiency, the rasterization backend can be modified to support batch rendering on the objects (e.g. one pass rasterization for all the objects). Notably, the GSplat \\[1\\] backend already supports batch rasterization for a single set of Gaussians. By modifying the rasterizer to add Gaussian masks during rasterization, we can render different objects in batch, thus enabling a more efficient and scalable training process.  Although we have not implemented this yet, we consider it a promising future refinement, and we believe the efficiency issue can be fully resolved through such engineering strategies.\n\n\\[1\\] Ye V, Li R, Kerr J, et al. gsplat: An open-source library for Gaussian splatting\\[J\\]. Journal of Machine Learning Research, 2025, 26(34): 1-17.\n\n# \\[Q2\\] Robustness of Object Tracking\n\nSAM2 itself may easily fail to track objects accurately in complex scenarios (e.g., *Figurines* in Tab.2(c)), due to issues like lost tracking, duplicate tracking etc.. And that’s the reason why we designed a robust tracking module to handle these issues, ensuring reliable supervision for the following object-specific reconstruction. And we also conduct further experiment on MipNeRF360 dataset to demonstrate the reliability of our tracking module on complex scenes with cluttered objects, please refer to **the table in the above section**.\n\n# \\[Q3\\] Highly Dynamic and Occluded Scenes\n\nThe ability to handle significant object deformation primarily depends on the underlying 4D Gaussian Splatting (4DGS) method used. Our approach is compatible with any 4DGS pipeline, thus, the capacity to model complex dynamics is inherently limited by the dynamic modeling capabilities of the chosen 4DGS method.   \nFor highly occluded scenarios, the *Figurines* in the LERF-OVS dataset and *Kitchen* in MipNeRF360 dataset can serve as representative examples. In these challenging settings, our proposed pipeline and tracking method demonstrate better performance over baseline methods.\n\n# \\[Q4\\] Partial Mask Filtering\n\nSorry for the confusion. Partial mask filtering is not a correction for mask inaccuracies. The mask provided by our tracking module is accurate enough. The motivation for incorporating this strategy is that the rendered object Gaussians may reveal the full object structure, even in views where the object is heavily occluded, as it can learn from other views. A demonstration is shown in Fig. 5 of the main paper. In such cases, direct supervision using the SAM mask from the occluded view would impose an incorrect constraint, potentially distorting the learned object structure. Partial mask filtering helps avoid this by preventing such misleading supervision signals. The ablation study for partial mask filtering **has already been done** in main paper Tab. 2(b). In highly occluded scenes such as *Ramen* and Waldo-Kitchen, partial mask filtering significantly improves both segmentation and reconstruction performance.\n\n# \\[W4\\] Discussion on Open-Vocabulary 3D Detection\n\nThank you for pointing this out. Open-vocabulary 3D detection and open-vocabulary 3D segmentation are indeed closely related tasks. The key distinction lies in the level of spatial precision required: 3D segmentation aims to recover accurate **object boundaries**, whereas 3D detection only requires **coarse localization via bounding boxes**. Our approach can naturally be extended to open-vocabulary 3D detection by simply computing a 3D bounding box that encloses the extracted object-specific Gaussians."}, {"review_id": "O083CF4Nh7", "reviewer": "Reviewer_Yvnk", "summary": "The paper presents Segment then Splat, a 3D-aware open-vocabulary segmentation approach for both static and dynamic scenes based on Gaussian Splatting. Different with traditional splat then segment pipelines used in this task, the proposed method reverses the process by first initializing each object with a specific set of Gaussians. Extensive experiments show state-of-the-art performance across diverse static and dynamic scenes in 3D open-vocabulary segmentation.", "questions": "Please refer to the weaknesses.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n+ The writing is easy to follow.\n+ The proposed method is reasonable and practical.\n+ The proposed method outperforms other baselines in 3D-based segmentation\n\nWeaknesses:\n- Since the proposed method is based on 3D segmentation, it is also capable of being evaluated based on the rendered 2D images and then evaluated on 2D segmentation masks. However, Table 1 only includes the 3D-based segmentation results for the proposed method. Including these aspects of experiments can be better for the reader to evaluate the proposed method when compared to other baseline methods, as a common setting in this research area.\n- Experiments on Mip-NeRF360. In addition, OpenGaussian offers a new annotated Mip-NeRF360 for 3D open-vocabulary segmentation. The authors should conduct their experiments on this dataset since it contains more views per scene, so that other 2D methods can be better evaluated regardless of the reconstruction tools they used.\n- Design. Since the proposed methods are based on per-object segmentation and then conduct reconstruction on it, the inherent limitation of the proposed method is to deal with the text query referring to multiple objects. However, I do not see any discussion or experimental results on this. Offering this kind of information can provide a more comprehensive understanding of these categories of “segment-then-splat” approaches.", "was_revised": true, "final_justification": "Thanks for the clarification. The rebuttal addresses most of my concerns. I would suggest the authors review their submission carefully to avoid any ambiguity. Despite this, I appreciate the effort the authors made. Based on these aspects, I will maintain my initial score.", "rebuttal": "We thank the reviewer for recognizing the reasonableness, practicality and performance of our proposed method. Your support is greatly appreciated and we would like to take this opportunity to clarify several points.\n\n# \\[W1\\] Evaluation on 2D Segmentation Mask\n\nThank you for pointing this out, and we apologize for the confusion. The evaluation results presented in Tab.1 are computed based on **2D segmentation masks**, in order to align with the evaluation protocols used by all baseline methods.  \nFor the 3D part in Tab.1, we modify 2D-based baselines (e.g., LEGaussian and LangSplat) to perform segmentation directly on **3D Gaussians** instead of pixels. However, to maintain a fair and consistent comparison, **all methods are ultimately evaluated on their rendered 2D segmentation maps**. This ensures comparability across different approaches while preserving the integrity of 3D-aware segmentation modeling.\n\n# \\[W2\\] Experiments on Mipnerf360\n\nThank you for your suggestion. We have conducted additional experiments on the Mipnerf360 dataset. And the results are shown in the table below, where our method performs well on this more challenging dataset and surpasses all baseline methods: \n\n| Method        | Counter mIoU | Time | Bonsai mIoU | Time | Kitchen mIoU | Time | Garden mIoU | Time | Room mIoU | Time | Avg mIoU | Avg Time |\n|---------------|--------------|------|-------------|------|---------------|------|--------------|------|------------|------|----------|-----------|\n| Ours          | **55.32**    |**36.0**|**63.80**    |**45.0**|**68.59**      |**48.0**|**44.09**      |**86.0**| 42.90      | 44.0 |**54.94** |**51.80**   |\n| OpenGaussian  | 49.57        | 48.0 | 39.08       | 58.0 | 31.22         | 60.0 | 15.10        | 130.0| 41.88      |**42.0**| 35.37    | 67.60     |\n| G-Grouping    | 41.76        | 50.0 | 56.36       | 62.0 | 58.15         | 71.0 | 36.25        | 128.0| 30.84      | 63.0 | 44.67    | 74.80     |\n| LEGaussian    | 54.98        | 50.0 | 62.46       | 52.0 | 66.77         | 60.0 | 40.12        | 103.0|**45.58**    | 61.0 | 53.98    | 65.20     |\n\n\n# \\[W3\\] Text query referring to multiple objects\n\nYes, this is indeed a limitation of our current method. Since object embeddings are derived solely from objects themselves, our model does not explicitly capture spatial or relational context between multiple objects. However, this limitation is shared by all baseline methods \\[1\\]\\[2\\]\\[3\\]\\[4\\]\\[5\\], as their embeddings are constructed in a similar object-isolated manner.  \nIncorporating a global scene-level representation, such as a scene graph that models relationships between objects (e.g., proximity, co-occurrence, or functional interaction), could potentially address this issue. We consider this a promising direction for future work and appreciate the reviewer for highlighting this important point.\n\n\\[1\\] Shi J C, Wang M, Duan H B, et al. Language embedded 3d gaussians for open-vocabulary scene understanding\\[C\\]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 5333-5343.  \n\\[2\\] Qin M, Li W, Zhou J, et al. Langsplat: 3d language gaussian splatting\\[C\\]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 20051-20060.  \n\\[3\\] Ye M, Danelljan M, Yu F, et al. Gaussian grouping: Segment and edit anything in 3d scenes\\[C\\]//European conference on computer vision. Cham: Springer Nature Switzerland, 2024: 162-179.  \n\\[4\\] Wu Y, Meng J, Li H, et al. Opengaussian: Towards point-level 3d gaussian-based open vocabulary understanding\\[J\\]. Advances in Neural Information Processing Systems, 2024, 37: 19114-19138.  \n\\[5\\] Labe I, Issachar N, Lang I, et al. Dgd: Dynamic 3d gaussians distillation\\[C\\]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 361-378."}, {"review_id": "R4T1YNyviv", "reviewer": "Reviewer_NzNg", "summary": "This submission proposes a heuristic approach to segment (potentially dynamic) scenes observed in multiple images and assign CLIP descriptors to them.", "questions": "How are the 3 scales determined?  Does the approach assume a know metric scale?  How does the approach work for a camera moving through a scene which can observe objects both from close-by and far away?", "limitations": "Compared to fully open vocabulary approaches to query scenes, here the method is limited to only return one of the previously reconstructed objects at one of the 3 scales.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "3D scene understanding is still a challenging and relevant problem, \n\nbut highly related prior work is missing and the heuristics are somewhat limiting (3 scales, how are those determined?), strongly relies on SAM2, how multi-view consistency is achieved is not well explained, seems could easily fail.  \n\nThe method limits its prior work mostly to Gaussian Splatting-based works, while there is a lot of highly related work with alternative representations that is ignored, OpenScene, OpenMask3D, Segment3D.  In particular the two latter ones are addressing the same problem, but use a pointcloud/mesh representations.", "was_revised": true, "final_justification": "I hope the authors will be able to include a discussion of the OpenMask3D/Segment3D related work.  For the rest, the issue I had were mostly addressed.  Maybe one small remaining issue, the authors (and many others) talk about training a Gaussian Splat representation, which is nonsense, they are fitting a Gaussian Splat representation.", "rebuttal": "Thank you for your review, we sincerely appreciate your comments and would like to clarify the following points.\n# Robustness and Strong Reliance on SAM2\n\nWhile we use SAM2 to obtain the initial tracking results, our proposed robust tracking module is designed to handle potential failure cases of SAM2. Our pipeline is evaluated across four datasets (e.g., LERF-OVS, 3DOVS, Neu3D, and HyperNeRF), ensuring its robustness. Additionally, we conduct further experiments on the more challenging MipNeRF360 dataset, including scene with hundreds of cluttered objects (e.g., *Counter* and *Bonsai*) to showcase the reliability and effectiveness of our method, the results are provided below:\n\n| Method        | Counter mIoU | Time | Bonsai mIoU | Time | Kitchen mIoU | Time | Garden mIoU | Time | Room mIoU | Time | Avg mIoU | Avg Time |\n|---------------|--------------|------|-------------|------|---------------|------|--------------|------|------------|------|----------|-----------|\n| Ours          | **55.32**    |**36.0**|**63.80**    |**45.0**|**68.59**      |**48.0**|**44.09**      |**86.0**| 42.90      | 44.0 |**54.94** |**51.80**   |\n| OpenGaussian  | 49.57        | 48.0 | 39.08       | 58.0 | 31.22         | 60.0 | 15.10        | 130.0| 41.88      |**42.0**| 35.37    | 67.60     |\n| G-Grouping    | 41.76        | 50.0 | 56.36       | 62.0 | 58.15         | 71.0 | 36.25        | 128.0| 30.84      | 63.0 | 44.67    | 74.80     |\n| LEGaussian    | 54.98        | 50.0 | 62.46       | 52.0 | 66.77         | 60.0 | 40.12        | 103.0|**45.58**    | 61.0 | 53.98    | 65.20     |\n\n\n# Multiview Consistency\n\nWe are not entirely sure what the reviewer means by “multiview consistency.” In our pipeline, mask consistency is maintained by SAM2 along with our robust tracking module. Additionally, our proposed object-centric reconstruction ensures that the 3D object geometry is consistent across multiple views, and the CLIP embedding association enforces embedding consistency for the same object across different viewpoints.\n\n# Three scales of SAM\n\nSorry for the confusion. The 3 scales are semantic levels pre-defined in the Segment Anything (SAM) \\[1\\] as “whole”, “part” and “subpart”. These levels are used to capture varying degrees of semantic granularity within an object. Please kindly refer to the SAM paper for more details. As these scales are semantic-wise, they are not related to metric (geometric) scales.\n\n\\[1\\] Kirillov, Alexander, et al. \"Segment anything.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023\\.\n\n# Scenes with camera moving close-by and far-away\n\nThank you for the question. All scenes in the LERF-OVS dataset, on which we conducted our experiments, feature camera trajectories that observe objects both from close-up and from a distance, and our approach performs robustly under these conditions. Please check the LERF \\[2\\] paper and website for the visualization of their dataset.\n\n\\[2\\] Kerr J, Kim C M, Goldberg K, et al. Lerf: Language embedded radiance fields\\[C\\]//Proceedings of the IEEE/CVF international conference on computer vision. 2023: 19729-19739.\n\n# Missing comparison to point cloud-based related works\n\nThank you for mentioning these related works. However, OpenScene, OpenMask3D, and Segment3D operate under a different problem setting compared to ours. All of these methods assume access to existing 3D geometry (e.g., point clouds) as input, whereas our method takes only multi-view 2D images as input and performs open-vocabulary segmentation after reconstructing the scene geometry.  \nAs a result, the segmentation performance in our method and the baseline ones inherently influenced by the quality of the reconstruction, introducing a distinct set of challenges not present in the above-mentioned works. Therefore, a direct comparison would not be fair or meaningful, as the input assumptions and task formulations differ significantly. We will go through these works and consider discussing them in the related work section.\n\n# The method is not fully open-vocabulary\n\nThough our method can only return the reconstructed objects at 3 different semantic levels, namely, “whole objects”, “parts”, and “sub-parts”, these levels are designed to **comprehensively cover the semantic granularity** of most objects and their components in the scene.  \nAs a result, arbitrary text queries can still be matched to the most relevant reconstructed object or part through CLIP-based embedding comparison. While the system does not generate new segmentations on-the-fly, it remains **practically open-vocabulary** in that it supports **flexible and diverse text queries** without being constrained to a fixed category set."}, {"review_id": "iD34DUDaGY", "reviewer": "Reviewer_cG1h", "summary": "This paper proposes ‘segment then splat’, a 3D open-vocabulary scene understanding method under the framework of 3D-GS. Compared with previous semantics reconstruction then understanding methods like LangSplat and LEGaussians, the proposed method first segments the whole scene into multiple objects and attach semantics information to each object. Experiments results demonstrate the effectiveness of the proposed method.", "questions": "1. I think it is better to focus on the segmentation task like GaussianGrouping. Though the paper claims it focuses on open-vocabulary segmentation, the main method focuses on improving the class-agnostic segmentation.\n2. My main concern is about the novelty, which I think can not be solved by minimal modification to the current paper. I encourage the author to totally revise the paper and method and resubmit.", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The motivation and the proposed method are easy to understand.\n2. Writing is clear.\n3. Experimental results are good, showing the effectiveness of the proposed method.\n\nWeaknesses:\n1. There is a misalignment between the theme of the paper and the proposed method. Though the paper claims the proposed method focuses on open-vocabulary 3D segmentation, the main part of the proposed method focuses on segment the 3D scene in a class-agnostic paradigm (similar to Gaussian Grouping). The semantic understanding is achieved by simple multi-view semantics feature aggregation.\n2. The whole pipeline is not new. The paradigm of Segment then Splat’ has been proposed by OpenGaussian. Discussion is insufficient. The only new thing is the support for dynamic scene, which is not a core challenge of open-vocabulary 3D understanding.\n3. How to obtain the 42.43 mIoU of OpenGaussian in Table 1? The mIoU reported in their paper is 38.36.\n4. Many important citations are missing, including but not limited to N2F2 [1], SAGA [2], GARField [3], Feature3DGS [4], GOI [5], and others. I strongly recommend that the authors conduct a more thorough review of related work in this field.\n5. The main contribution of the proposed method is unclear. The 3D segmentation method looks like a collection of tricks (Line 148-160) added on GaussianGrouping. However, the reliability of such hand-crafted calibration is in doubt. As shown in Table 2, these tricks do not provide large performance gain.\n\n[1] Bhalgat, Y., Laina, I., Henriques, J. F., Zisserman, A., & Vedaldi, A. (2024, September). N2f2: Hierarchical scene understanding with nested neural feature fields. In European Conference on Computer Vision (pp. 197-214). Cham: Springer Nature Switzerland.\n[2] Cen, J., Fang, J., Yang, C., Xie, L., Zhang, X., Shen, W., & Tian, Q. (2025, April). Segment any 3d gaussians. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 39, No. 2, pp. 1971-1979).\n[3] Kim, C. M., Wu, M., Kerr, J., Goldberg, K., Tancik, M., & Kanazawa, A. (2024). Garfield: Group anything with radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 21530-21539).\n[4] Zhou, S., Chang, H., Jiang, S., Fan, Z., Zhu, Z., Xu, D., ... & Kadambi, A. (2024). Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 21676-21685).\n[5] Qu, Y., Dai, S., Li, X., Lin, J., Cao, L., Zhang, S., & Ji, R. (2024, October). Goi: Find 3d gaussians of interest with an optimizable open-vocabulary semantic-space hyperplane. In Proceedings of the 32nd ACM International Conference on Multimedia (pp. 5328-5337).", "was_revised": true, "final_justification": "I thank the authors for their response. After reading the response I still believe the method focuses on class-agnostic 3D segmentation. I think it requires a more holistic way to integrate the semantic understanding part with the segmentation part. Considering the technical contribution of the paper, I raise my score from 2 to 3.", "rebuttal": "We thank the reviewer for recognizing the effectiveness of our method and providing feedbacks. We would like to take this opportunity to clarify the following points.\n\n# \\[W1, Q1\\] Clarification of Motivation and Contribution\n\nThank you for your feedback. While we appreciate the reviewer’s perspective, we would like to respectfully clarify our motivation and contributions. Our primary focus is on open-vocabulary segmentation, and our core motivation is to assign each Gaussian an accurate and discriminative language embedding to enable precise text-driven segmentation. We achieve this by eliminating semantic ambiguities and establishing correspondence between Gaussian groups and objects (e.g., Gaussians belonging to the same object have identical embeddings and each Gaussian only represents one object). \n\nSpecifically, we design a robust tracking module and develop Gaussian-specific optimization strategy, all these are served for establishing object-Gaussian correspondence and eliminating ambiguity. These components are central to our method and serve the overall goal of improving open-vocabulary segmentation performance.\n\nTherefore, we respectfully disagree with the assessment that our method primarily focuses on class-agnostic segmentation. While class-agnostic grouping is used as an intermediate step, our full pipeline is thematically and technically aligned with the goal of open-vocabulary 3D segmentation.\n\n# \\[W2, W5, Q2\\] Novelty \n\nWe would like to emphasize the novelty and contributions of our proposed pipeline, and clarify how it fundamentally differs from previous works.\n\nAll prior methods (e.g., OpenGaussian, LangSplat, Gaussian Grouping, etc.) follow a “Splat-then-Segment” paradigm. Taking OpenGaussian (mentioned by the reviewer) as an example, it first **reconstructs** the scene using 3D Gaussian Splatting while simultaneously learning a feature field via contrastive loss. It then clusters the Gaussians based on the learned features, and finally assigns CLIP embeddings to each cluster to enable open-vocabulary **segmentation**.\n\n“Splat-then-Segment” will cause several issues.\n\n1. **Geometric and Semantic Ambiguities:** Without object-level constraints during reconstruction, Gaussians near object boundaries may represent multiple objects and carry mixed semantic information, leading to ambiguity.  \n2. **Object-Gaussian Misalignment in Dynamic Scenes**: In dynamic scenes, a single Gaussian may represent different objects across time, requiring impractical time-varying embeddings for open-vocabulary segmentation. Thus, direct adaptation of “Splat-then-Segment” methods to dynamic scenes is fundamentally limited.\n\nWe propose Segment-then-Splat, we first “segment” Gaussians into object-specific groups then during reconstruction, each group of Gaussians only contributes to its own object, thus eliminating the ambiguity and misalignment issue inherent in “Splat-then-Segment” approaches.  \n Our method achieves superior segmentation performance compared to baselines, particularly around object boundaries (see Fig. 4 in the main paper), and can be directly applied to dynamic scenes without additional modification.\n\n# \\[W3\\] Higher mIoU of OpenGaussian compared to the original paper\n\nThank you for your observation. The original OpenGaussian implementation holds out every eighth image (i.e., llffhold=8) during training to follow the standard evaluation protocol for reconstruction. However, this results in some segmentation-labeled images being inadvertently included in the training set, which is inconsistent with the evaluation protocols followed by other baseline methods.  \nTo ensure a fairer evaluation and to align with the settings used by other baseline methods, in segmentation experiments, we exclude all labeled images from the training set and use only the unlabeled images for training. This adjustment allows more images to be used for reconstruction compared to the llffhold=8 setting,  thus leading to higher reconstruction quality and, consequently, improved segmentation performance.\n\n# \\[W4\\] Missing Citations\n\nThank you for pointing this out. As some of the referenced works are not directly focused on open-vocabulary segmentation, we did not include them primarily in the current version. However, we acknowledge their relevance and will carefully review these works and consider including appropriate citations and discussions in the final version of the paper.\n\n# \\[W5\\] Reliability of the Robust Tracking Module\n\nWe have conducted comprehensive experiments on 4 datasets including LERF-OVS, 3DOVS, Neu3D, and HyperNeRF ensuring its robustness. Additionally, we conduct new experiment on the more challenging MipNeRF360 dataset, including scene with hundreds of cluttered objects (e.g., *Counter* and *Bonsai*) to showcase the reliability and effectiveness of our method, the results are provided below:\n\n| Method        | Counter mIoU | Time | Bonsai mIoU | Time | Kitchen mIoU | Time | Garden mIoU | Time | Room mIoU | Time | Avg mIoU | Avg Time |\n|---------------|--------------|------|-------------|------|---------------|------|--------------|------|------------|------|----------|-----------|\n| Ours          | **55.32**    |**36.0**|**63.80**    |**45.0**|**68.59**      |**48.0**|**44.09**      |**86.0**| 42.90      | 44.0 |**54.94** |**51.80**   |\n| OpenGaussian  | 49.57        | 48.0 | 39.08       | 58.0 | 31.22         | 60.0 | 15.10        | 130.0| 41.88      |**42.0**| 35.37    | 67.60     |\n| G-Grouping    | 41.76        | 50.0 | 56.36       | 62.0 | 58.15         | 71.0 | 36.25        | 128.0| 30.84      | 63.0 | 44.67    | 74.80     |\n| LEGaussian    | 54.98        | 50.0 | 62.46       | 52.0 | 66.77         | 60.0 | 40.12        | 103.0|**45.58**    | 61.0 | 53.98    | 65.20     |"}]

Author rebuttal:

Meta-review: The paper introduces a 3D segmentation method that splits Gaussians used in 3D Gaussian splat scene representations into distinct object sets before reconstruction. Once the reconstruction is complete, the scene is naturally segmented into individual objects, achieving 3D segmentation. As a result, the method maintains precise object-Gaussian correspondence for improved 3D geometry and segmentation accuracy. 

The paper received five reviews: four borderline accepts and one borderline reject. Positive reviewers highlighted several strengths: (a) the intellectual novelty of the “Segment-then-Splat” idea, (b) strong experimental results, and (c) natural compatibility with dynamic scenes. Weaknesses raised during the reviews and author--reviewer discussion included: (a) the technical contribution is somewhat limited and primarily related to class-agnostic segmentation, (b) increased preprocessing complexity, and (c) issues with robustness to large occlusions. The authors acknowledged these limitations.

Overall, four out of five reviewers felt that the significance of the “Segment-then-Splat” idea and the strong results outweighed the weaknesses. The AC concurs with this assessment and recommends acceptance. The authors are strongly encouraged to incorporate the additional experiments, results, and clarifications from the rebuttal and discussion into the final version. They should also carefully address cG1h’s remarks and the other weaknesses raised by the reviewers, particularly in the limitations section.
Decision: Accept (poster)