Title: Segment then Splat: Unified 3D Open-Vocabulary Segmentation via Gaussian Splatting
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D Reconstruction

3D Scene Understanding

Open Vocabulary Segmentation

Differentiable Rendering
Abstract: Open-vocabulary querying in 3D space is crucial for enabling more intelligent perception in applications such as robotics, autonomous systems, and augmented reality. However, most existing methods rely on 2D pixel-level parsing, leading to multi-view inconsistencies and poor 3D object retrieval. Moreover, they are limited to static scenes and struggle with dynamic scenes due to the complexities of motion modeling.
In this paper, we propose Segment then Splat, a 3D-aware open vocabulary segmentation approach for both static and dynamic scenes based on Gaussian Splatting. 
Segment then Splat reverses the long established approach of "segmentation after reconstruction'' by dividing Gaussians into distinct object sets before reconstruction. Once reconstruction is complete, the scene is naturally segmented into individual objects, achieving true 3D segmentation. This design eliminates both geometric and semantic ambiguities, as well as Gaussian–object misalignment issues in dynamic scenes. It also accelerates the optimization process, as it eliminates the need for learning a separate language field.
After optimization, a CLIP embedding is assigned to each object to enable open-vocabulary querying. Extensive experiments on various datasets demonstrate the effectiveness of our proposed method in both static and dynamic scenarios.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes ‘segment then splat’, a 3D open-vocabulary scene understanding method under the framework of 3D-GS. Compared with previous semantics reconstruction then understanding methods like LangSplat and LEGaussians, the proposed method first segments the whole scene into multiple objects and attach semantics information to each object. Experiments results demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The motivation and the proposed method are easy to understand.\n2. Writing is clear.\n3. Experimental results are good, showing the effectiveness of the proposed method.\n\nWeaknesses:\n1. There is a misalignment between the theme of the paper and the proposed method. Though the paper claims the proposed method focuses on open-vocabulary 3D segmentation, the main part of the proposed method focuses on segment the 3D scene in a class-agnostic paradigm (similar to Gaussian Grouping). The semantic understanding is achieved by simple multi-view semantics feature aggregation.\n2. The whole pipeline is not new. The paradigm of Segment then Splat’ has been proposed by OpenGaussian. Discussion is insufficient. The only new thing is the support for dynamic scene, which is not a core challenge of open-vocabulary 3D understanding.\n3. How to obtain the 42.43 mIoU of OpenGaussian in Table 1? The mIoU reported in their paper is 38.36.\n4. Many important citations are missing, including but not limited to N2F2 [1], SAGA [2], GARField [3], Feature3DGS [4], GOI [5], and others. I strongly recommend that the authors conduct a more thorough review of related work in this field.\n5. The main contribution of the proposed method is unclear. The 3D segmentation method looks like a collection of tricks (Line 148-160) added on GaussianGrouping. However, the reliability of such hand-crafted calibration is in doubt. As shown in Table 2, these tricks do not provide large performance gain.\n\n[1] Bhalgat, Y., Laina, I., Henriques, J. F., Zisserman, A., & Vedaldi, A. (2024, September). N2f2: Hierarchical scene understanding with nested neural feature fields. In European Conference on Computer Vision (pp. 197-214). Cham: Springer Nature Switzerland.\n[2] Cen, J., Fang, J., Yang, C., Xie, L., Zhang, X., Shen, W., & Tian, Q. (2025, April). Segment any 3d gaussians. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 39, No. 2, pp. 1971-1979).\n[3] Kim, C. M., Wu, M., Kerr, J., Goldberg, K., Tancik, M., & Kanazawa, A. (2024). Garfield: Group anything with radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 21530-21539).\n[4] Zhou, S., Chang, H., Jiang, S., Fan, Z., Zhu, Z., Xu, D., ... & Kadambi, A. (2024). Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 21676-21685).\n[5] Qu, Y., Dai, S., Li, X., Lin, J., Cao, L., Zhang, S., & Ji, R. (2024, October). Goi: Find 3d gaussians of interest with an optimizable open-vocabulary semantic-space hyperplane. In Proceedings of the 32nd ACM International Conference on Multimedia (pp. 5328-5337).\n\nQUESTIONS:\n1. I think it is better to focus on the segmentation task like GaussianGrouping. Though the paper claims it focuses on open-vocabulary segmentation, the main method focuses on improving the class-agnostic segmentation.\n2. My main concern is about the novelty, which I think can not be solved by minimal modification to the current paper. I encourage the author to totally revise the paper and method and resubmit.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

comments: SUMMARY:
This submission proposes a heuristic approach to segment (potentially dynamic) scenes observed in multiple images and assign CLIP descriptors to them.

STRENGTHS AND WEAKNESSES:
3D scene understanding is still a challenging and relevant problem, 

but highly related prior work is missing and the heuristics are somewhat limiting (3 scales, how are those determined?), strongly relies on SAM2, how multi-view consistency is achieved is not well explained, seems could easily fail.  

The method limits its prior work mostly to Gaussian Splatting-based works, while there is a lot of highly related work with alternative representations that is ignored, OpenScene, OpenMask3D, Segment3D.  In particular the two latter ones are addressing the same problem, but use a pointcloud/mesh representations.

QUESTIONS:
How are the 3 scales determined?  Does the approach assume a know metric scale?  How does the approach work for a camera moving through a scene which can observe objects both from close-by and far away?

LIMITATIONS:
Compared to fully open vocabulary approaches to query scenes, here the method is limited to only return one of the previously reconstructed objects at one of the 3 scales.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Segment then Splat, a 3D-aware open-vocabulary segmentation approach for both static and dynamic scenes based on Gaussian Splatting. Different with traditional splat then segment pipelines used in this task, the proposed method reverses the process by first initializing each object with a specific set of Gaussians. Extensive experiments show state-of-the-art performance across diverse static and dynamic scenes in 3D open-vocabulary segmentation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ The writing is easy to follow.\n+ The proposed method is reasonable and practical.\n+ The proposed method outperforms other baselines in 3D-based segmentation\n\nWeaknesses:\n- Since the proposed method is based on 3D segmentation, it is also capable of being evaluated based on the rendered 2D images and then evaluated on 2D segmentation masks. However, Table 1 only includes the 3D-based segmentation results for the proposed method. Including these aspects of experiments can be better for the reader to evaluate the proposed method when compared to other baseline methods, as a common setting in this research area.\n- Experiments on Mip-NeRF360. In addition, OpenGaussian offers a new annotated Mip-NeRF360 for 3D open-vocabulary segmentation. The authors should conduct their experiments on this dataset since it contains more views per scene, so that other 2D methods can be better evaluated regardless of the reconstruction tools they used.\n- Design. Since the proposed methods are based on per-object segmentation and then conduct reconstruction on it, the inherent limitation of the proposed method is to deal with the text query referring to multiple objects. However, I do not see any discussion or experimental results on this. Offering this kind of information can provide a more comprehensive understanding of these categories of “segment-then-splat” approaches.\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a 3D open-vocabulary segmentation method called \"Segment then Splat\", which reverses the traditional \"Splat-then-Segment\" method. Instead of conducting segmentation after scene reconstruction, the method segments objects at the 3D Gaussian initialization stage, ensuring object-specific Gaussian sets before reconstruction. After reconstruction, semantic embeddings from CLIP are assigned to each object to enable open-vocabulary querying.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. Reversal of the Traditional Pipeline: \n\nThe proposed \"Segment then Splat\" pipeline introduces a straightforward modification to the traditional \"Splat-then-Segment\" approach, addressing issues like Gaussian-object misalignment in dynamic scenes.\n\n2. Dynamic Scene Support: \n\nBy enforcing object-Gaussian correspondence and using a strong object tracking module, the method gains the ability to handle dynamic scenes better than other methods.\n\n3. Comprehensive Evaluation: \n\nExperiments on both static and dynamic datasets provide sufficient evidence of the method’s effectiveness compared to other methods.\n\n4. Qualitative Results: \n\nThe paper demonstrates sharper segmentation boundaries and better object retrieval performance, particularly in cases where previous methods struggled with multi-object scenes or dynamic scenes.\n\n**Weaknesses**\n\n1. Limited Novelty: \n\nThe \"Segment then Splat\" pipeline is conceptually simple, consisting of two straightforward steps:\n- Perform 3D segmentation on point clouds before reconstruction.\n- Use CLIP embeddings for semantic matching.\nThis method lacks a unique technique design, as it essentially combines existing techniques (SfM-based segmentation + CLIP-based matching) in a straightforward manner.\n\n2. Pipeline Complexity: \n\nThe method requires per-object 3DGS reconstruction, which makes the pipeline computationally expensive and potentially impractical for large-scale or complex scenes with many objects.\n\n3. Scalability Concerns: \n\nThe computational efficiency of the method for scenes with hundreds or thousands of objects is unclear, as the need for object-specific Gaussian reconstruction could introduce lots of overhead.\n\n4. Lacking Discussions about the OV 3D Detection settings\n\nConsidering the research contexts are very related, the submission should also discuss the relation with methods of OV-3DDet settings [1, 2, 3]. I'm curious about their differences and whether this method has the potential for OV-3DDet applications\n\n\n[1] Pengkun Jiao, Na Zhao, Jingjing Chen, and Yu-Gang Jiang. Unlocking Textual and Visual Wisdom: Open-Vocabulary 3D Object Detection Enhanced by Comprehensive Guidance from Text and Image. In ECCV, 2025.\n\n[2] Yang Cao, Yihan Zeng, Hang Xu, and Dan Xu. CoDA: Collaborative Novel Box Discovery and Cross-Modal Alignment for Open-Vocabulary 3D Object Detection. In NeurIPS 2023.\n\n[3] Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. Open-Vocabulary Point-Cloud Object Detection without 3D Annotation. In CVPR 2023\n\nQUESTIONS:\n1. Scalability: \n\nHow does the method perform on large-scale scenes with many objects? Does the computational cost scale linearly or non-linearly with the number of objects?\n\n2. Robustness of Object Tracking: \n\nIn cases where SAM2 fails to track objects accurately (e.g., cluttered scenes or objects with similar appearances), how does this affect the segmentation and reconstruction performances?\n\n3. Dynamic Scene Complexity: \n\nHow does the method handle highly dynamic scenes with significant object deformation or occlusion? Are there some failure cases?\n\n4. Post-Processing Dependence: \n\nThe partial mask filtering strategy appears to be a post-hoc correction for mask inaccuracies. How significant is its impact on the final results, and does this suggest limitations in the present pipeline?\n\nLIMITATIONS:\nYes, the authors have discussed the limitations on page 9.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces \"Segment then Splat,\" a novel 3D open-vocabulary segmentation method using Gaussian Splatting that segments objects before reconstruction. This reverses the traditional \"splat then segment\" approach, maintaining precise object-Gaussian correspondence for improved 3D geometry and segmentation accuracy. It supports both static and dynamic scenes by preventing Gaussian-object misalignment and includes a robust object tracking module for spatial-temporal consistency. By associating CLIP embeddings with object-specific Gaussians, it enables efficient open-vocabulary querying without extra language fields. Experiments show state-of-the-art performance in accuracy and efficiency across diverse datasets\n\nSTRENGTHS AND WEAKNESSES:\n#### Strengths:\n\n1. The paper is clearly written and well-structured, making it easy to follow.\n\n2. The concept of “Segment then Splat” represents a highly innovative and bold departure from the conventional strategy of explicitly modeling semantic fields in open-vocabulary Gaussian approaches. By reversing the traditional “splat then segment” paradigm, this method fundamentally eliminates the reliance on separate language fields for semantic understanding. Moreover, it effectively addresses challenges such as the misalignment between Gaussians and objects, particularly prominent in dynamic scenes. The subsequent experimental results robustly validate the effectiveness of this approach, demonstrating its practical viability and advantages over prior methods.\n\n3. The experimental performance is impressive: across a variety of both static and dynamic datasets, Segment then Splat achieves state-of-the-art results in 3D open-vocabulary segmentation as well as computational efficiency. Extensive qualitative visualizations further illustrate the model’s ability to preserve the integrity of 3D structures while delivering sharp, well-defined object boundaries. \n\n4. Another strength of the proposed framework is its natural compatibility with dynamic scenes. By maintaining a strict one-to-one correspondence between Gaussians and objects, the method effectively prevents Gaussian-object misalignment caused by object motion. This capability allows Segment then Splat to seamlessly unify the treatment of both static and dynamic environments, overcoming a major limitation of existing methods that struggle with time-varying or moving objects.\n\n#### Weaknesses:\n\n1. Although the paper adopts the Segment then Splat approach to avoid explicitly incorporating semantic parameters into Gaussians, this increases complexity during the preprocessing stage. Additionally, at inference time, the model still requires some matching method to assign semantic information to Gaussians. This leads to a more fragmented model pipeline rather than an end-to-end framework.\n\n2. For more complex scenes, compared to explicitly embedding semantic information into Gaussians, this method may be more difficult to handle. This is because the grouping prior relies heavily on high-precision grouping initialization and complex tracking logic, which may be fragile or error-prone in challenging scenarios.\n\n3. If I understand correctly, during experiments comparing with baselines, the ground truth used is at the object-level rather than pixel-level. This might introduce some unfairness to methods optimized at pixel-level such as LangSplat. If my understanding is incorrect, please kindly correct me. Otherwise, providing additional quantitative comparison with pixel-level ground truth would be beneficial. Of course, due to techniques like Partial Mask Filtering used during training, some performance gap is expected and a slight disadvantage could be acceptable.\n\n4. Although the paper includes a section discussing limitations, it does not comprehensively illustrate failure cases or conditions under which the method’s performance degrades. Presenting such results would help readers gain a more complete understanding of the model’s robustness and boundaries.\n\nQUESTIONS:\n1. I noticed that the training process adopts a three-stage optimization strategy, progressively encouraging the model to focus on objects of increasing scale, from small parts to larger wholes. However, the paper seems to lack a detailed experimental analysis that specifically evaluates the model’s performance on different object hierarchies or granularities, particularly for relatively smaller subparts. I am especially interested in understanding whether the model can accurately identify and segment these fine-grained, smaller objects (subparts) with precision comparable to larger objects. Such an analysis would provide valuable insights into the robustness and granularity of the proposed segmentation approach.\n\n2. If possible, I would like to hear an explanation about Weaknesses 3 above. Of course, if my understanding is incorrect, please feel free to point it out.\n\n3. During training, the number of supervised objects per iteration is identified as a crucial hyperparameter influencing segmentation performance, as shown in the ablation study in paper. It appears that in none of the static scenes does performance fully saturate with increasing numbers of supervised objects. I wonder if this observation might be due to insufficient supervision over certain part and subpart-level objects, which may dilute the learning signal when only random sampling is employed. If possible, I kindly request the authors to provide some analysis or insight into this aspect.\n\n4. It would be helpful if the authors could include some failure cases or typical scenarios where the proposed method does not perform satisfactorily, in order to better understand its limitations and potential areas for improvement.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}