econSG: Efficient and Multi-view Consistent Open-Vocabulary 3D Semantic Gaussians
Authors: Can Zhang
Gim Hee Lee
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Scene Understanding
Gaussian Splatting
Open-Vocabulary 3D Semantic
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: The primary focus of most recent works on open-vocabulary neural fields is extracting precise semantic features
from the VLMs and then consolidating them efficiently into a multi-view consistent 3D neural fields
representation. However, most existing works over-trusted SAM to regularize image-level CLIP without any further refinement. Moreover, several existing works improved efficiency by dimensionality reduction of semantic features from 2D VLMs before fusing with 3DGS semantic fields, which inevitably leads to multi-view inconsistency. In this work, we propose econSG for open-vocabulary semantic segmentation with 3DGS. Our econSG consists of: 1) A Confidence-region Guided Regularization (CRR) that mutually refines SAM and CLIP to get the best of both worlds for precise semantic features with complete and precise boundaries. 2) A low dimensional contextual space to enforce 3D multi-view consistency while improving computational efficiency by fusing backprojected multi-view 2D features and follow by dimensional reduction directly on the fused 3D features instead of operating on each 2D view separately. Our econSG show state-of-the-art performance on four benchmark datasets compared to the existing methods. Furthermore, we are also the most efficient training among all the methods.

Published as a conference paper at ICLR 2025
ECONSG: EFFICIENT AND MULTI-VIEW CONSISTENT OPEN- VEENULOCBBULAY 3D SEMANTIC GAUSSIANS
Can Zhang & Gim Hee Lee Department of Computer Science National University of Singapore can. zhang@u.. nus. edu, gimhee. Teraee.leeenne lee@nus.edu
ABSTRACT
The primary focus of most recent works open-vocabulary neural fields is ex- tracting precise semantic features from the VLMs and then consolidating them efficiently into multi-view consistent 3D neural fields representation. However, most existing works over-trusted SAM to regularize image-level CLIP without any further refinement. Moreover, several existing works improved efficiency by di- mensionality reduction of semantic features from 2D VLMs before fusing with 3DGS semantic fields, which inevitably leads to multi-view inconsistency. this work, we propose econSG for open-vocabulary semantic segmentation with 3DGS. Our econSG consists of: 1) A Confidence-region Guided Regularization (CRR) that mutually refines SAM and CLIP to get the best of both worlds for precise semantic features with complete and precise boundaries. 2) A low dimen- sional contextual space to enforce 3D multi-view consistency while improving computational efficiency by fusing backprojected multi-view 2D features and fol- low by dimensional reduction directly the fused 3I) features instead of operat- ing on each 2D view separately. Our econSG shows state-of-the-art performance four benchmark datasets compared to the existing methods. Furthermore, we are also the most efficient training among all the methods. Our source code is available at: https: httsss//llussindaae. github 1o/ So/econSSprooectt.
INTRODUCTION
The advances in neural 3D scene representation techniques have revolutionized many research and applications in computer vision and graphics. Among these neural 3D scene representation tech- niques, Neural Radiance Field (NeRF) (Mildenhall et al. 202T) stands out for its ability to learn 3D neural fields directly from 2D images with excellent performance in important real-world applica- tions such as novel view synthesis. Recently, the explicit 3D Gaussian Splatting (3DGS) (Kerbl et al.) 2023) has been proposed remarkable reconstruction quality while maintaining high training and rendering efficiency. Con- current neural 3D scene representation techniques, large risual-language models (VLMs) such as the CLIP model (Radford et al., 2021) have shown extremely strong capability in zero-shot transfer the open-world setting for various downstream tasks such as image semantic segmentation, etc.
alternative to the implicit NeRF. This technique has demonstrated
Neural 3D scene representation and multi-modality foundation models are rapidly advancing in par- allel. This progress has led to research on open-vocabulary 3D scene understanding, where the neural rendering capabilities of neural fields are leveraged to align VLMs with 3D scenes. To this end, almost all existing works (Kerr al. [2023] Liu et al., 2024 Qin al. 20033] Liao et al. 2024] Shi et al.) 2023 Zhou et al. 2024,] Ye et al., 2023) Guo et al.) 2024) unanimously adhered to the fundamental pipeline of first extracting semantic features from the given multi-view images using open-world 2D visual-language models (VLMs), followed by using these semantic features to train semantic fields appended to NeRF or 3DGS. However, since 2D semantic features extracted inde- pendently from multi-view images can be incomplete and inconsistent, the primary focus of most existing works is on extracting precise semantic features from VLMs and then efficiently consoli- dating them into multi-view consistent 3D neural field representation.

---

Published as a conference paper at ICLR 2025
Early approach LeRF (Kerr al. 2023) leverages CLIP to get semantic features from each of the input multi-view images. However, this often results in semantic features with ambiguous bound- aries since CLIP is trained image-level captions despite the attempt in LeRF to improve granu- larity with multi-scale CLIP features. Several subsequent works utilize Segment Anything Model
(SAM) (Kirillov al.] 2023) DINOv2 (Oquab et al. (0033 to improve the precision of the seman-
tic features from CLIP. 3DOVS (Liu et al.) (024) and LEGaussians (Shi et al. 2023) use semantic
features with better boundaries from DINO to complement CLIP. Feature-3DGS (Zhou et al. 2024) extracts semantic features from either SAM or LSeg (Li et al. 2022). Gaussian Grouping (Ye et al., 2023) leverages only SAM masks, leading to class-agnostic segmentation. Semantic Gaussian (Guo) et al. 2024) and OV-NeRF (Liao et al., 2024) unify 2D CLIP features with class-agnostic instance masks generated from SAM. All the above-mentioned works over-trusted DINOv2 or SAM without making any refinement, which we show empirically (ef. Fig. 3 Column (b) shows missing regions in the mask proposals from SAM) to be imperfect.
Several approaches such as OV-NeRF (Liao et al. 2024 LeRF (Kerr et al. 2023). 3DVVS (Lu
ct al. 2024) and Feature-3DGS (Zhou et al., 2024) naively adopt the same dimension for the 3D neural semantic fields as the high- digeenimnnii semantic features from 2D VLMs, which inevitably incurs high computational complexity for training and querying. Methods such as LangSplat (Qin) et al. 2023) and LeGaussians (Shi et al., 2023) propose the use of autoencoder or quantization to reduce the dimension of the multi-view 2D semantic features, and therefore result in similar reduction of dimension in the 3D neural semantic fields for efficient computation. However, the reduction of feature dimension is carried out in the 2D space before lifting into the 3D space, and this can lead to multi-view inconsistency that hurts performance. Although Gaussian Grouping (Ye et al. 2023) is efficient by learning 3DGS only for lannosti mask rendering, it consequently lacks semantic language information for each Gaussian.
In this paper. we propose Efficient and Multi-view Consistent 3D Semantic Gaussians (econSG), a simple yet effective zero-shot model for 3D semantic understanding. Our proposed econSG con- sists of: 1) Confidence-region Guided Regularization (CRR) to alleviate the incompleteness and ambiguous boundaries of the semantic features obtained from VLMs. In contrast to other ap- proaches which over-trusted SAM or DINO, our CRR is designed to get the best from both worlds of OpenSeg (Ghiasi et al. 2022) and SAM with strong 3D multi-view consistency. Specifically, our CRR first backprojects high confidence OpenSeg semantic features from multiple views using the depth maps obtained from Colmap (Schönberger et al.] 2016).. We then fit a bounding box on the fused features reprojected onto each view to prompt SAM for better region masks. These masks help refine the OpenSeg semantic features towards well-defined boundaries. 2) Low- -ow-Dinesiional 3D Contextual Space to enforce 3D multi-view consistency and enhance computational efficiency. We build 3D contextual space from 3D features obtained by fusing the backprojected multi-view 2D features instead of operating on each 2D view separately. We then pre-train an autoencoder to get the low-dimensional latent semantic space for initializing the 3DGS semantic fields. The encoder of the pre-trained autoencoder is also used to project CRR-refined semantic features into the same latent space to supervise the 3DGS semantic fields. Our model improves efficiency by enabling strong initialization for 3DGS semantic fields while performing optimization and rendering entirely within the low-dimensional latent space.
We summarize our main contributions as follows: Regularization (CRR) to get 2D semantic features with complete and precise boundaries by mutual guidance from OpenSeg and SAM with strong awareness of multi-view consistency. 2) We design an autoencoder with one-time pre-training get the oow-dimesiional 3D contextual space for initial- ization of the 3D neural semantic fields, and enforce multi-view consistency by backprojecting 2D features from CRR into the same dimension as the low-dimensional 3D contextual space for efficient training. 3) Our econSG show state-of-the th-the-r performance on four benchmark datasets compared to existing methods. Furthermore, we are also the most efficient training among all methods.
1) We propose a Confidence-region Guided
2 RELATED WORK
2D Open-vocabulary Segmentation. 2D open-vocabulary segmentation has seen considerable growth due to the availability of vast text-image datasets and computational resources. Advance-
ments in large VLMs (Alayrac et al. 2022 Jia et al. 2021 Radford et al., 2021) have significantly
2

---

Published as a conference paper at ICLR 2025
enhanced zero-shot 2D scene understanding, even for long-tail objects in images. A common ap- proach for zero-shot predictions is to use trained to map images and text labels into unified semantic space. However, these models often produce embeddings at the image level, which are not suitable for tasks requiring pixel-level in- formation. Recent efforts (Ghiasi al. 2022 Kuo al. 2022] Li et al. 2022. (hoou et al.) 2022] Rao et al., (022) aim to bridge this gap by correlating dense image features with language model embeddings, enabling users to detect, classify or segment objects in images with arbitrary text la- bels. Predominant open- vocabulay segmentation methods (e.g. LSeg (Li et al.) (022)) often rely on distilling knowledge from large-scale pre-trained models such as image-text contrastive learn- ing models (e.g. CLIP (Radford et al.) (02T)) and diffusion models(Rombach et al., 2022). These approaches leverage the rich semantic information captured during pre-training to perform segmen- tation tasks. However, the distillation process necessitates fine-tuning on specific datasets with a limited vocabulary which undermines the open-vocabulary capability and results in reduced perfor- mance in recognizing rare classes. OpenSeg (Ghiasi et al. (2022) utilizes weak supervision through image captions without fine-tuning on a specific class set, but its vocabulary is limited compared to CLIP due to a smaller training dataset. In contrast, method bypasses fine-tuning CLIP and effectively handles open world classes.
cross-modal encoders, which are
3D Open-vocabulary Segmentation. The success of 2D open-vocabulary segmentation has in- spired many recent works (Peng et al. [023] Ding et al.] [2023] Nguyen et al.) [2024] Takmaz et al.) |023)oo on 3D apen-vocabylary segmentation for point clouds. Many of these methods follow com- mon design principle: aligning pre-trained 2D open-vocabulary segmentation frameworks such as
LSeg (Li al. 2022) with point cloud feature embeddings. These works primarily rely on point clouds that are relatively more difficult to obtain than multi-view images. To enable multi-view images as input, there has been a significant increase in NeRF-based (Mildenhall et al.] [02T] 3D segmentation. Given that the 2D semantic features derived independently from multi-view images are prone to inconsistency, the primary objective is to learn shared 3D neural representation that enforces consistency by fitting multi-view data into unified representation with a loss function that penalizes inconsistencies among views. Semantic-Neee field which enables the synthesis of segmentation masks from novel views. However, this method requires a large number of annotated labels, which is non-trivial and costed. Some methods (Tsch-
(Zhi et al.] (021) constructs a semantic
emnezk et al. 2022] Fan et al. 2022) utilize the self-supervised feature extractor (e.g. DINO (Caron
et al.] (021)) ) to extract 2D features and distill features into the semantic field. More recently, several NeRI- works (Kerr et al.
2023 Liu et al. 2024: Kobayashi et al. 2022) have explored textual
descriptions combined with CLIP models to achieve open- vocabblcyy 3D semantic understanding.
LERF (Kerr et al.] 2023) grounds the language field within NeRF by optimizing multi-scale em- beddings from CLIP into 3D scenes. 3DOVS (Liu et al. (024) distills voca-vocay multimodal knowledge from CLIP and object boundary information from DINO into the NeRE, [Wang et al.) (2024) focuses on proposing a 3D open-vocabulary segmentation framework that can generalize to unseen scenes. Despite promising results, NeRF-based approaches suffer from slow training and rending. We circumvent these issues by using the more efficient 3D Gaussian Splatting.
3D Gaussian Splatting. 3D Gaussian Splatting (3DGS) (Kerbl et al. 2023 has recently gained pop-
ularity as technique for real-time radiance field rendering and 3D scene reconstruction. Inspired by the success of 3DGS in novel view synthesis, various works (Luiten et al. 2023] Yi et al. [023] Ye et al. 2023) have adapted it for various tasks to leverage its efficient rendering process. How- ever, Gaussian Splatting methods enabling object-level or semantic understanding of the 3D scene are still under-explored yet meaningful. Gaussian Grouping (Ye al. 2023) extends 3DGS beyond mere scene appearance and geometry modeling with instance level modeling based on class-agnostic SAM masks. DGS (Zhou et al. 2024) learns nigh-dimensional semantic fields in 3D Gaus- sians using multi-view CLIP features, leading to high computational cost. LangSplat (Qin et al., 2023) and LEGaussians (Shi et al.] 2023) encode multi-view features from 2D pre-trained VLMs into 3DGS via different feature dimension reduction techniques. However, these approaches suffer from rendering inefficiencies and 3D semantie inconsistencies across training views. FastL.GS (Ji) et al.) 2025) mitigates multiview inconsistencies by using appearance-base mask similarities across views, where they force matched masks across the views to take the same semantic field. In contrast, we do not directly apply inconsistent and imprecise semantics from 2D VLMs across views to optimize 3DGS. Instead, we construct a multi-view consistent 3D embedding space based on geometry for modeling the 3DGS semantic fields.
correspondences and
3

---

Published as a conference paper at ICLR 2025
Multi-view Feature Fusion
Input Images
Image Encoder
SAM
Back-Projections BD
Fusion
Low Dimersional Latant
Semantic Space
2DD
F=2b
g-30
CRR
Average Pooling
9
h
Confidence- guided Region Regulbrization
M
M
Per Pleel Latent Embeddings
Semantic Loss
Iniiallatiinn
Encoder
F1
Object Queries
L semontic
Rendering
Query Embeddiing
CRR
(p ttery
Similarity
Regulartzed
S Segmentation Logit
Learnalee 3DGS
Latent Embeddings
Cross Loss
Semantic Fields
Figure 1: Our econSG framework. Top: Building 3D contextual latent space. We use the image encode from a VLM and our CRR to get 2D features F2D,, which are then back-projecter and fused in 3D to get the high dimensional 3D contextual code M.. An autoencoder [g, h] is learned to map M into the low dimensional space M.. 2) Bottom: 3DGS for semantic fields. We optimize for the 3DGS semantic fields f with Esemantic and Loe supervised by the image fF2D and query T- latent embeddings obtained by the encoder g. respectively. MM is used to initialize f.
PRELIMINARIES: 3D GAUSSIAN SPLATTING
3DGS (Kerbl et al. 2023) explicitly represents the 3D scene set of anisotropic 3D Gaussians, which share similarity with point clouds. Each Gaussian is characterized by center point vector E IR* and a covariance matrix Σ I 3x3, which influences a 3D point æ in the scene following
3D Gaussian distribution: G(x) (2
((=-w)" (x-x), To ensure positive semi-definite
and differential optimization, Σ = RSS R. decomposed into two learnable components: a scaling matrix S R and rotation quaternion matrix R R Additionally, each Gaussian is parameterized by an opacity value o E IR and an appearance feature vector defined by n spherical harmonic (SH) coefficients of SH with degree d. For rendering, 3D Gaussians are projected onto the image plane of the given view by the a- blending function as follows:
{c; IR3 - 1, ....... de), where de the number of coefficients
Σ G,a; [T= (1 az). c is the final color in the rendered image computed by blending ordered Gaussians that overlap onto the pixel. C; R represent color computed from SH coefficients in the jth Gaussian. a is obtained by multiplying the projected 2D covariance matrix >' I2222 with the learned opacity. >' JWZW T J in the camera coordinates is computed using view transform matrix W and the Jacobian matrix J of the affine approximation the projective transformation.
OUR METHOD
Objective, Given the posed images I and the corresponding open-vocabulary queries T from the frozen text encoder of VLM, the goal is to synthesize semantic masks from novel views rendered by 3DGS parameterized by {x, n, R, S, e, f), where f is additional optimizable semantic field add to 3DGS.
Overview. Fig shows an overview of our econSG which consists of: 1) A pre-training stage Region Regularization (CRR) that mutually refines OpenSeg and SAM to get the 2D semantic features. In contrast to (Ye et al. 2023 Liao et al. 2024 CRR avoids over-trusting DINO or SAM
where we first design the Confidence-guided
Guo et al. 2024 Liu al. 2024 Shi al. 2023), which we empinically show to be imperfect (cf. Fig. Column (b)). We then train an autoencoder
for the low-dimensional 3D contextual space to improve the training and query efficiency of the

---

Published as a conference paper at ICLR 2025
3DGS semantic fields in the next stage. Unlike (Qin et al. 2023 Shi et al. 2023 which compress semantic features in the 2D space before 3D fusion, we enhance 3D consistency by first fusing the backprojected 2D semantic features to get the 3D contextual space followed by training autoencoder to get the low-dimensional 3D contextual space. 2) A training stage where we initialize the 3DGS semantic fields with the low-dimensional 3D contextual space, and supervise the training of the rendered low-dimensional 3DGS semantic fields efficiently with the CRR semantic features mapped into the same dimension by the frozen encoder in the pre-trained autoencoder. We also utilize the frozen encoder and CRR to align class semantics with the 3DGS semantic fields.
4.1 IMAGE AND TEXT EMBEDDINGS
We obtain per- pixel semantic feature F from the RGB images I with the image encoder of 2D VLM w2n I F2D Similarly, we use the 2D VLM encoder to get the representative open- vocabulary embeddings T from multi-view object queries T via youery T + T. During training, queries T represent object proposals derived from multi-view inputs, whereas during inference, they correspond to text prompts either provided inference specified by the user.
4.2 CONFIDENCE-GUIDES REGION REGULARIIZTTOO (CRR)
As shown in Fig. 3 the semantic feature map from OpenSeg (Column (a)) and the regional mask proposals from SAM (Column (b)) can be imperfect due to complex background and occlusion, and thus leading to inconsistent and inaccurate semantics across multiple views. We design our CRR for mutual refinement of the per-pixel semantic feature from the 2D VLM and regional mask proposals from SAM as follows:
a: Select pixel embeddings F2D across all views with confidence higher than threshold T1: R SelectConfident (F +i);
b: Back-project semantic features of each pixel in R. into 3D using depthmaps D from Colmap. Average-por parrage-pooloao projeceee R to get multi-view consistent semantic features:
F3D AvgPool (rgPPoo(BackPrroect(RR.. D));
Obtain semantic label for each 3D point according to its similarity with the query embeddings. On the reprojected points, do majority voting on the semantic labels and everage-pooling on the semantic features to get set of 2D semantic masks and their corresponding features and labels: [2, y +
FFD
b
Fit bounding boxes to the re-projected P, and use as input prompts to SAM to get better regional mask proposals:
b S 5- PrroppSsMMEEBBBex(Prooott////
e: Retain P with confidence higher than threshold 72. Assign the semantic label and feature of the high confidence P to the improved SAM regional mask proposal S with the highest IoU score:
D {S, Y} + MaxloUScore (SeirrreeeSeeeCnniddeeeii
T2),
Note that the semantic features F22 __2 and FeD share the same dimension since F3D obtained from average pooling of F22 from multi-view back-projections, and FF2D from the average- pooling of the reprojected FED in each mask P. Steps (a)-(c) enforces multi-view consistency in the semantic features from OpenSeg. Step (d) uses the multi-view consistent semantic mask to improve regional mask proposals from SAM. Finally, Step (c) uses the improved regional mask proposals from SAM further refine the multi-view consistent semantic mask.
4.3 LOW-DIMENSIONAL 3D CONTEXTUAL SPACE
Multi-view Feature Fusion. For each 3D point obtained from Structure-from-Motiin (SfM) using Colmap for the initialization of 3DGS, compute per 3D point feature f3D "f ..... . F2P) from average pooling v the multi-view features of the N, visible corresponding pixels. We build an
5

---

Published as a conference paper at ICLR 2025
initial 3D contextual space by consolidating all point features corresponding to the point cloud from SfM: M .p.....pppp)
Autoencoder. A naive direct rendering of the feature fields is very time-consuming due to the high dimensionaliity of the semantic features F=D since the latent dimensions in 2D foundation models tend to be very large. This problem is further aggravated in complex 3D scenes with lot of dense points. We thus pre-train an autoencoder to map the high dimensional 3D contextual space M into low-dimensional latent space space M, {=i ..... = to improve efficiency. Specifically, the encoder 23D = s(f2D) maps feature 13 with high dimension to a lower dimension latent vector =gD The reconstruction is given by of )(g({2)), where Λ(-) is the decoder and og. is the reconstructed 3D semantic feature. The training objective of the autoencoder on the 3D point features M is as follows:
Lue L12lSp .of) (u) cos < <o T>) + Lee (1) cos < 2/. ( >);
(I)
where L12 and Le.. denote the L2 loss and cross entropy loss, respectively. Using cosine similarity, cos < 0%2 T outputs the semantic label of the reconstructed semantic feature based on query em- beddings and cos < : (7)) > outputs the semantic label of the encoded low dimension semantic feature. ÿ is the pseudo semantic mask generated from 2D segmentation model.
4.4 3DGS SEMANTIC FIELDS
After obtaining the pre-trained autoencoder hg()), we use the encoder g(-) to map: 1) The
initial 3D contextual features the low-dimensional 3D contextual space g : M M,; 2) Per- pixel semantic features to per-pixe low-dimensional semantic features g : F2D + F2D :3) Query semantic features to low-dimensional query semantic features g T=.
We use the low-dimension 3D contextual space M2 to initialize the semantic field f in each 3D Gaussians, and render the 3DGS semantic fields into each view via alpha-blending:
F
Σ St (1-00)) IE
We supervise the rendered semantic fields by their semantic logit Sn, with the semantic mask label Y from CRR using cross-entropy loss: Lee CF(Sm, Y), where the semantic logit is obtained from the cosine similarity between the low-dimensional semantic and query features: S,, cos < F, To >, Furthermore, optionally regularize F to improve feature smoothness with the low- dimensional semantic features F&D using a L2 semantic loss: Lsemantic = L2(F, FTD).
The final supervision loss for optimizing the given scene is formulated as follows:
Lcolor + AzaLce + deem Lsemantic:
(3)
where Lestor is the 3D Gaussian image rendering loss, and A2d and Asem denote hyperparameters to balance the loss terms. In inference, we use Eq. 2 to render the learned 3DGS semantic fields from 3D 2D. We deploy the encoder g(-) from the pre-trained autoencoder get query features T, of open-world queries. By computing activation scores between the rendered 3DGS semantic fields F and query features, we can obtain open-world segmentation predictions.
EXPERIMENTS
We perform series of experiments to demonstrate the effectiveness of our proposed method across various 3D scene understanding tasks. We evaluate our method on the 2D semantic segmentation benchmarks: ScanNet (Dai al. 2017) and Replica (Straub al. 2019). and 3D open-vocabulary segmentation benchmarks: LERF (Kerr et al. 2023) and 3DOVS (Luu et al. 2024) to compare with previous work, and provide results from ablation studies. We further showcase qualitative results the Mip-Nerf366 (Barron et al.) 2022) for exciting open-vocabulary applications such as 3D object localization, 3D object removal, 3D object inpainting, and language-guided editing.

---

Published as a conference paper at ICLR 2025
Table 1: Comparisons of open-vocabulary segmentation 3DOVS dataset. Best results in bold.
Dataset
3DVVS
hed
sofa
lawn
bench
overall
Method
mloU
mAcc
mloU
mAcc
mloU
mAce
mloU
mAce mloU
mAce mlaU
mAce
2D
LSeg
56.0
87.6
4.5
16.5
17.5
77.5
19.2
46.1
6.0
42.7
20.6
54.1
LERF
73.5
86.9
27.0
43.8
73.7
93.5
46.6
79.8
53.2
79.7
54.8
76.7
3DOVS
89.5
96.7
74.0
91.6
88.2
97.3
92.8
98.9
89.3
96.3
86.8
96.2
Feature3DGS
56.6
87.5
12.4
37.3
82.6
20.5
36.7
43.0
25.5
52.4
3D
LEGaussians
45.7
48.2
49.7
44.7
47.4
47.1
LangSplat
73.5
89.7
82.3
98.7
89.9
95.6
95.0
99,4
70.6
92.6
82.3
95.2
econSG (Ours)
94.9
97.4
91.6
98.7
96.3
98.5
95.8
99.4
93.0
97.6
94.3
98.3
Table 2: Comparisons of localization accuracy LERF dataset. Best results in bold.
Dataset
LERF
Method
ramen
figurines
teatime
waldo kitchen
overall
2D
LSeg
14.1
8.9
33.9
27.3
21.1
LERF
62.0
75.0
84.8
72.7
73.6
LangSplat
73.2
80.4
88.1
95.5
84.3
SemanticGaussian
76.8
83.1
89.8
90.9
85.2
3D
LEGaussians
78.6
73.7
85.6
90.1
82.0
econSG (Ours)
83.2
89.3
93.4
90.5
5.1 DATASETS AND EXPERIMENTAL SETTING
Datasets. To measure the semantic segmentation performance open-world scenes, we eval- uate the effectiveness of approach using two established multi-view indoor scene datasets: Replica (Straub et al.] 2019) and Scannet (Dai et al. (017)), and two 3D open-vocabulary segmenta- tion datasets: LERF Kerr al. 2023) and 3DOVS (Liu al., 2024). For both ScanNet and Replica, we construct training and test sets by evenly sampling sequences in each scene. Images are rendered the resolution of 640 x 480. We adopt 20 different semantic class categories for Scannet by following Openscene (Peng et al. 2023), while Replica is annotated with 51 classes for evaluation in Engelmann al. 2024). For LERF and 3DOVS, we follow the settings in LangSplat (Qin et al. 2023) where LERF is extended with ground truth masks annotated for language queries and 3DOVS consists of 20 30 images for each scene with the resolution of 4032 3024. To as-
sess 3D reconstruction quality, we applied our method to Mip-Nerf360 (Barron et al. 2022) and
LERR-Localiiztion (Kerr et al. 2023) by following Gaussian Grouping YYe al. 2023).
Implementation details. For 2D VLMSs, we utilize pixel-level encoders, OpenSeg (Ghiasi et al., 2022) and LSeg(Li et al. 2202)) to extract the per-pixel semantic features each image in indoor scene datasets, and adopt Openclip (Ilharco al. 202l> to extract image-level features for language- guided editing on Mip-Ner1360, LERF and 3DOVS datasets. We then use SAM for mutual refine- ment with the 2D VLMs in CRR to get the semantic features where
0.45, 72 0.6. We use the Adam optimizer with the learning rate 0.0025 for latent semantic fields. For parameters train the image scene, follow the default setting in the original 3DGS (Kerbl et al. 2023). For additional parameters introduced to train the semantic scene, we set Asem 1, = 1. For all
datasets, we train each scene for 30K iterations on one NVIDIA RTX-4090 GPU.
5.2 MULTIVIEW RECONSTRUCTION AND SEGMENTATION
ppen-vocabulary Segmentation Comparison. Tab. and Tab. show quantitative results of open- vocabulary segmentation on 3DOVS dataset and localization accuracy on LERF dataset. Tab. shows segmentation comparison across various scenes on both Scannet and Replica. We compare with 2D-based apen-vocabulary segmentation model LSeg (Li et al., 2022) along with 3D NeRF and 3DGS-based methods. LERF (Kerr et al., 2023) and 3DOVS (Liu et al. 2024) leverage the multi-scale CLIP features from the image patches as supervisions for learning NeRF-based seman- tie field, and thus struggle with both object boundary ambiguities for segmentation and rendering efficiency. Their performance inconsistent and imprecise ground truth semantics from multi-scale features across multiple views. Feature3DGS (Zhou al. 2024) directly applies inconsistent and noisy semantics from training
novel views be greatly degraded because of they generate

---

Published as a conference paper at ICLR 2025
Table Comparison with other methods on segmentation of novel views from Scannet and Replica. Best results highlighted in bold.
Replica
Scannet
Dataset
FPS
sparse-vice view
multi-view
sparse- -ieew
multi-view
mloU
mAcc
mloU
mAcc
mloU
mAcc
mloU
mAcc
LERF
0.2
4.312
17.080
8.285
22.125
14.059
38.734
15.349
40.294
3DOVS
0.3
4.553
19.356
9.081
23.938
14.227
40.584
17.802
42.532
FeS
2.5
9.584
38.245
10.634
36.520
17.552
48.686
18.069
54.101
econSG (Ours)
156
25.513
70.716
33.869
78.564
39.018
74.805
48.205
86.178
Rendered Image
OpenSeg Masks Prediction
Grouping Feature Visualizzton
Masks Prediction (class agnostis)
Masks Prediction (dess agnostic)
Masks Prediction
Feeture Visualizatin
chair
aet
table
Figure 2: Qualitative comparison of our econSG with Gaussian Grouping Replica.
Ye al.
2023
views to optimize high-dimensional semantic field in 3D Gaussians, resulting in high computation costs and inferior segmentation results. LangSplat (Qin al. 2023) and LEGaussians (Shi al., 2023) compress 2D features across all views to improve rendering efficiency on semantic fields, their performance are still hindered by the inherent 3D semantic noises and inconsistency. Se- manticGaussian distills noisy 2D features into additional 3D model for learning 3D semantics while ignoring semantic consistency from the multi-view 2D images. Our model consistently shows the best performance since we introduce the 3D contextual latent space to provide sufficient 3D semantic consistency into the ground truths and design a CRR step to generate clean and complete semantic masks. These components help ensure optimization efficiency and robustness few input images.
with
In Tab. B] we also present the inference speed under the multi-view setting in terms of the frames per second (FPS) metric. NeRF--based methods are generally constrained on rendering efficiency and slow. 3DGS-based models are inefficient from high-dimensional language features in 3D Gaussians. We also perform robustness comparison by evenly sampling sparse training views for optimiza- tion(30 images per-scene our experiments). It shows our model consistently outperforms other methods, proving the proposed components help ensure optimization efficiency and robustness even with few input images. Fig. 2 we visualize the learned semantic fields by showing the rendered latent embeddings in the testing views. We observe that our predictions are of better consistency across views with more complete and well-defined boundaries semantics masks.
Ablation on CRR. We compare our CRR with OpengSeg and SAM, and conduct ablation studies on CRR. OpenSeg in Fig. Faa) shows issues such as ambiguous boundaries and inaccurate dense predictions. This is due to the use of noisy segmentation maps from pre-trained visual encoders for supervision. Fig. b) shows that naive over-trusting of SAM masks to refine boundaries does not work well in complex scenes. Fig. (()) vs. (d) and Fig. e (f) show the without and with
CRR on the training and testing sets, respectively. We can see that our CRR effectively produces semantic fields with clear boundaries.
Analysis on 3D Contextual Latent Space. We show qualitative results of 3D segmentation pre- dictions and contextual feature space Fig. 4 The 3D segmentations derived from the original
8

---

Published as conference paper at ICLR 2025
training views vision language foundation resulta training views
testing
View
View
Tra dag perfictoon
Tood sew pasiicuioe suranicc
Tneting potd plose suraniic
prodited Spendag
prodited
PAE CRR
Figure 3: Ablation confidence-guile guidedd region regularization (CRR) with qualitative results of our econSG on Replica. Panels (a)-(e) are from training views, and panels (f)-(g are from testing views.
Input
OpenSeg
Gaussian Grouping
SAMPro3D
Ours
Masks Prediation
Feature
No Features
Masks
Feature
No Features
Figure 4: Qualitative 3D Segmentation results and comparison of our method. The second and fourth rows illustrate the feature visualization in 3D space.
OpenSeg exhibit significant coarseness and errors due to multi-view inconsistency among the pre- dicted 2D semantic features. Gaussian Grouping shows better object-level boundaries by leveraging SAM object mask IDs as direct supervision. However, SAM can fail in complex scenes leading to incorrect masks in some views. Morever, since SAM segmentations are class-agnostic, the learned semantic embeddings from Gaussian Grouping are only instance-level and cannot queried by text embeddings. SAMPro3D (Xu et al.] (023) proposes to filter low-quality prompts and consoli- date prompts inside the object. However, SAMPro3D not applicable open-vocabulary understanding tasks without feature embeddings. In contrast, model significantly improves the quality 3D contextual space and segmentation predictions as illustrated the last column.
scene
Training Efficiency Analysis. In Tab. 4 we show training and inference time on the "sofa scene" of 3DOVS dataset at different feature dimensions. Compared with LangSplat, our model achieves a Ours:4.9s). This is because LangSplat performs evaluation on the original high-dimensional space while our model directly makes predic- tions in the low-dimensional latent contextual space. Our model can achieve promising efficiency and accuracy due to the low-dimensional 3D latent contextual space that avoids the need for training nigh-dimensional 3DGS semantic fields. The last column of Tab 4 shows that the -ige-dimensional features (e.g. 512 for CLIP features) pose huge memory and computation demands especially on training when the autoencoder removed.
significant speed increase in inference (LangSplat:401.9g
5.3 APPLICATIONS
3D Scene Editing, Fig. 5 Right) shows examples of geguage--gidded editing on object scene from the Bear data (Ye et al. 2023) and scene from Mip-NeRF360 (Barron et al. (022).. We utilize the text encoder to embed the object category names to identify the corresponding 3DGS points and adjust their attributes such coordinates and colors. We first detect regions that invisible in all

---

Published as a conference paper at ICLR 2025
Table 4:
Training efficiency analysis the sofa scene of the 3DOVS dataset.
Ours
Methods
LERF
3DOVS
Langsplat
Feature3DGS
Ours
(remove autocncoder)
Feature dimension
512
128
6
16
32
512
mloU (%)
27.0
74.0
82.3
6.7
91.6
91.8
91.8
OOM
Training time (min)
19.4
66
87
29
32
43
OOM
Inference (s)
121.4
401.9
6.0
4.9
5.2
OOM
Object localization and segmentation
Editing
Prompt
egg
Prompt: green apple
Prompt: stuffed bear
Rendered Images
Gausslan Grouping
Relevancy Map (Ours)
Mask prediction (Ours)
(Ours)
Rendered View
temove [object]'
Inpaint black blurry regions'
Figure 5: Qualitative examples of language- guided segmentation and editing. Segmentation results of the rendering views are compared with Gaussian Grouping on LERF-localization dataset.
Text query: 'ceiling
Retevancy Segmeertttto
Door Image query
office0
office0
Figure 6: A 3D scene can queried using text prompt embedding or images to locate matching 3D points. Colors of image query outlines indicate corresponding matches in the 3D point cloud.
views after deletion and then inpaint these specific areas instead of the entire 2D object regions. We then use the 2D inpainted image in each rendering view to guide the learning of new 3D Gaussians.
Open-Vone Vocabulary 3D Object detection. Fig. 5 (Left) shows examples of object localization and segmentation with text queries. In Fig. 6 we query a 3D scene database to retrieve examples based on their similarity to a given input image. We first encode the query text image using CLIP image encoder and then threshold the cosine similarities between the CLIP features and the 3DGS semantic fields produce 3D object detection and mask.
CONCLUSION
In this paper, we propose econSG for open- open-vocabulary semantic segmentation of 3D scenes. Specif- ically, we propose CRR to get 2D semantic features with complete and precise boundaries by mutual guidance from OpenSeg and SAM with strong awareness of multi-view consistency. We design an autoencoder with one-time pretraining to get the low-dimensional 3D contextual space for initial- ization the 3D neural semantic fields, and enforce multi-view consistency by backprojecting 2D features from CRR into the same dimension the oow-dimensional 3D contextual space for efficient training. Our econSG shows state-of-the ofth--of-the-at performance on four benchmark datasets compared to the existing methods. Furthermone, are also the most efficient training among all the methods.
10

---

Published as a conference paper at ICLR 2025
Acknowledgments. This research work is supported by the Agency for Science, Technology and Research (A*STAR) under its MTC Programmatic Funds (Grant No. M23L760021), and the Tier grant MOE-T2EP20124-0115 from the Singapore Ministry of Education.
REFERENCES
Jean-Baptiste Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances Neural Information Processing Systems, 35:23716- 23736, 2022.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5470-5479, 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828-5839, 2017.
Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Pla: Language- driven open-vocabulary 3d scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Francis Engelmann, Fabian Manhardt, Michael Niemeyer, Keisuke Tateno, Marc Pollefeys, and Federico Tombari. Opennerf: Open set 3d neural scene segmentation with pixel-wise features and rendered novel views. arXiv preprint arXiv:2404.03650, 2024.
Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Nerf- sos: Any-view self-supervised object segmentation on complex scenes. arXiv preprint arXiv:2209.08776, 2022.
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In European Conference on Computer Vision, pp. 540-557. Springer, 2022.
Jun Guo, Xiaojian Ma, Yue Fan, Huaping Liu, and Qing Li. Semantic gaussians: Open-vocabulary scene understanding with 3d gaussian splatting, 2024.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi. Ali Farhadi, and Ludwig Schmidt. Openclip. July 2021. URL hatpps:/ /ww.ooi.o.oo. woo.orrg/1... 5281/ zenodo. 51 1137733 If you use this software, please cite it as below.
Yuzhou Ji, He Zhu, Junshu Tang, Wuyi Liu, Zhizhong Zhang. Xin Tan, and Yuan Xie. Fastlgs: Speeding up language embedded gaussians with feature grid mapping. In Proceedings of the AAAI Conference Artificial Intelligence, 2025.
Chao Jia, Yinfei Yang, Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vasion-language representation learning with noisy text supervision. International conference on machine learning, pp. 4904-4916. PMLfR, 2021.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023.
Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Lan- guage embedded radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 19729 19729-19739, 2023.
11

---

Published as a conference paper at ICLR 2025
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXi::2304.026433 2023.
Sosuke Kobayashi, Eicchi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. Advances Neural Information Processing Systems, 35:23311-23330, 2022.
Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. Open-vocabulary object detection upon frozen vision and language models. In The Eleventh International Conference on Learning Representations, 2022.
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. In International Conference on Learning Representations, 2022. URL.
https:
Guibiao Liao, Kaichen Zhou, Zhenyu Bao, Kanglin Liu, and Qing Li. Ov-nerf: Open-vocabulary neural radiance fields with vision and language foundation models for 3d semantic understanding. arXiv preprinl 24x.v:2402.046644 2024.
Kunhao Liu, Fangneng Zhan, Jiahui Zhang, Muyu Xu, Yingchen Yu, Abdulmotaleb El Saddik, Christian Theobalt, Eric Xing, and Shijian Lu. Weakly supervised 3d open-vocabulary segmen- tation. Advances in Neural Information Processing Systems, 36, 2024.
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308.099 09713, 2023.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65()):99-106, 2021.
Phuc D. A. Nguyen, Tuan Due Ngo, Evangelos Kalogerakis, Chuang Gan, Anh Tran, Cuong Pham, and Khoi Nguyen. Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guid- ance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), 2024.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fermandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
Songyou Peng. Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 815-824, 2023.
Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d lan- guage gaussian splatting. arXiv preprint arXiv:2312.16084, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763, PMLR, 2021.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
18082-18091, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
12

---

Published as a conference paper at ICLR 2025
Johannes Lutz Schönberger, Enliang Zheng, Mare Pollefeys, and Jan-Michate Frahm. Pixelwise view selection for unstructured multi-vien view stereo. In European Conference on Computer Vision (ECCV), 2016.
Jin-Chuan Shi, Miao Wang, Hao-Bin Duan, and Shao-Hua Guan. Language embedded 3d gaussians for open-vocabulary scene understanding. arXiv preprint arXiv:2311.1844 2023.
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. The Replica dataset: digital replica indoor spaces. arXiv preprint arXiv: 2019.
Ayça Takmaz, Elisabetta Fedele, Mare Pollefeys, Federico Tombari, and Francis Engelmann. Open- mask3d: Open-vocabulary 3d instance segmentation. 2023. URL https://arxiv..
abs/ fbs/2336..o 13631
Vadim Tschemezki, Iro Laina, Diane Larlus, and Andrea Vedaldi. Neural feature fusion fields: 3d distillation of self-supervised 2d image representations. 2022 International Conference on 3D Vision (3DV), pp. 443-453. IEEE, 2022.
Yunsong Wang, Hanlin Chen, and Gim Hee Lee. Gov-nesf: Generalizable open-vocabulary neural semantic fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Mutian Xu, Xingyilang Yin, Lingteng Qiu, Yang Liu, Xin Tong, and Xiaoguang Han. Sampro3d: Locating sam prompts in 3d for zero-shot scene segmentation. arXiv preprint arXiv:2311.17707, 2023.
Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke. Gaussian grouping: Segment and edit anything in 3d scenes. arXiv preprint arXhv:2312.00732 2023.
Taoran Yi, Jiemin Fang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussian splatting with point cloud priors. arXiv preprint arXiv:2310.08529, 2023.
Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu. Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. Mvimgnet: A large-scale dataset of multi-view images. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 9150-9161 2023.
Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison. In-place scene la- belling and understanding with implicit scene representation. In Proceedingy of the IEFF/CVF International Conference Computer Vision, pp. 15838-15847 2021.
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision, pp. 696-712. Springer, 2022.
Shijie Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Xu, pradyumna Chari, Suya You, Zhangyang Wang, and Achuta Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 11676-21685, 2024.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 IMPLEMENTATION
Our auto-encoder is implemented by MLPs, which compresses the high-dimensional CLIP features into low-dimensional latent features, i.e. 738 for OpenSeg and 512 for CLIP features. The encoder consist layers with size follows: [256, 128, 64, 32. 6]. The decoder composed with layers by the following dimensions: [16, 32, 64, 128, 256, 256, 768).
A.2 ABLATION STUDY ON CRR.
We perform the ablation study of CRR on Scannet and Replica in Tab. In row "w.CRR(w.o Step a.)', evaluate Step of CRR which performs high-confidence region selection for computing 3D semantic features. In row 'w.CRR (w.o Step d.)", we analyze Step d of CRR which further use SAM refine the projected 2D segmentation maps. For example, in Roomo and scene0494 scenes, our CRR improves performance greatly in terms of both mloU, demonstrating that CRR helps refine segmentation boundaries.
Table Ablation study for CRR on Scannet and Replica.
Setting
Room0
Replica
scene0494
Scannet
mloU
mACC
mloU
mACC
mloU
mACC
mloU
mACC
W.O CRR
13.245
41.307
10.604
29.816
18.674
36.869
17.933
42.133
W. CRR (w.o step a.)
22.336
61.563
23.276
69.364
52.861
84.450
36.549
75.235
w. CRR (w.o step d.)
27.091
68.647
27.146
72.843
57.924
86.583
42.098
83.476
CRR
31.715
72.492
33.869
78.564
63.043
90.574
48,205
86.178
A.3 MORE VISUALIZATION RESULTS
We demonstrate more examples on Scannet dataset for open-vocabulary 3D semantic segmentation in Fig. 7 In Fig. Fig. and Fig. [0, present additional examples of retrieved objects from the 3DOVS and LERF datasets. Our model consistently provides clearer semantic patterns and exhibits reduced noise compared to LEEaussians and LpngSplat. These results demonstrate the effectiveness of our model.
A.4 3D MULTIVIEW CONSISTENCY OF OUR CRR
Fig. shows an illustration of 3D point with semantic label that is consistent (Top) and in- consistent (Bottom) across all multiple views. The steps in our CRR modify the results from OpenSeg/LSeg and SAM to ensure that the 2D features are 3D consistent across all multiple views (Top case). In contrast, other baseline methods overtrusted SAM and/or OpenSeg/LSeg often result in 3D multiview inconsistencies (Bottom case). Consequently, the supervision of 3DGS semantic fields from the features of our CRR tend to lead to good performances.
A.5 VISUALIZATION RESULTS
In Fig 12 we present the vocabuacyyy segmentation results the 3DOVS et al. 2024)
and LERF (Kerr al., 2023) datasets using object attributes such color, texture, and function as queries. For example, when queried with the attribute "plush" in the LERF-teatime scene, our model successfully localizes both the plush sheep and plush bear with accurate segmentation boundaries. When queried with "white" the same scene, model correctly localizes the white plush sheep. This demonstrates the robustness of our model handling diverse open-vocabulary text queries.
we show the visualization results MVImgNet (Yu et al. 2023) datasets for open- vocabulary segmentation using part-level text queries. We chose MVImg Net to show this task be- cause it contains single object scenes. vocabbocay segmentation using part-level text queries too challenging on the LeRF and 3DOVS datasets which contain multiple objects in the scenes. In our experiments and shown in the figures, we object part names text queries segment
In Fig. 13
14

---

Published as a conference paper at ICLR 2025
Scene0494
Scene0693
Scene0389
Scene0004
Figure 7: Qualitative 3D segmentation results of our econSG on the Scannet dataset.
red bag
leather shoe
LEGoussians
LangSplaat
Qura
Figure 8: Qualitative comparison of our econSG with baselines on the 3DOVS dataset. We show the visualization of the retrieved objects in the scene. The quantitative results are Table in the main paper.
the object parts for evaluation. LangSplat generates three scales with SAM and performs evalua- tions using 3DGS models at small, medium, and large scales during inference to select the optimal scale for query. The results show that LangSplat struggles with open-vocabulary part segmenta- tion. For example, when queried with "guitar headstock" (third row), LangSplat identifies the entire guitar instead the headstock. each part-level query. Although our econSG is capable of showing precise part segmentation for this example, we do not claim the full capability of part segmentation. This challenging task out-of-scope and we will leave it for future work.
contrast, our econSG generates precise segmentation maps for
15

---

Published as a conference paper at ICLR 2025
ared Nintendo Switch Joy. con controller
Gundam
Xbox wireless
controller
Pikachu
LEGauselana
LangSplat
Oura
Figure 9: Qualitative comparison of our econSG with baselines on the 3DOVS dataset. We show
the visualization of the retrieved objects in the scene. The quantitative results are in Table I in the main paper.
yellow
glass
chopsticks
wavy noodles
LongSplat
LERF-ramon
Figure 10: Qualitative comparison of our econSG with baselines the LERF dataset. We show the visualization of the retrieved objects in the scene. The quantitative results are in Table 2 in the main paper.
16

---

Published as a conference paper at ICLR 2025
3D Multiview Consistent
3D Point
Red Apple
View
View
3D Multiview Inconsistent
3D Point
Red Apple
Orange
Watermelon
View 3
View
View
Figure 11: Multiview consistency illustration. Top: The semantic label 3D point matches the semantic labels in the 2D images across multiple views. Bottom: The semantic label of the 3D point do not match all the 2D semantic labels in the 2D images across multiple views.
17

---

Published as a conference paper at ICLR 2025
Query:
"digital"
"yellow"
"smooth"
"soft"
3DOVS-bed
Query:
"plush"
"wood"
"glass"
Figure 12: Qualitative results (relevancy maps and segmentations) of our econSG for evaluating the open-vocabulary attribute query.
Query:
Small bpprr
LangSiplet
Lor LongB Midille scale (part)
La ge seale pct)
Langfilat
Dumppnt
gunte "pttariggetong
Maimagater-guuter
Figure 13: Qualitative comparisons between our econSG and LangSplat on MVIngNet dataset for evaluating the open-vocabulary 3D part segmentation.
18


Reviews:
[{"review_id": "wdE7XVfdkJ", "reviewer": "Reviewer_9XDx", "summary": "This paper presents a novel approach for open-vocabulary neural field segmentation. Current methods face two key challenges: 1) they rely too heavily on the inconsistent 2D features from 2D VLMs, and 2) they are computationally expensive due to the need for high-dimensional feature rendering. To address these, the authors propose a Confidence-region Guided Regularization technique to achieve precise boundaries and multi-view consistency, as well as a 3D autoencoder to project high-dimensional features into a lower-dimensional space. The proposed method is evaluated against the current state-of-the-art across multiple benchmarks.", "questions": "Please see weaknesses.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-motivated, addressing the issues of 2D feature inconsistency and the high computational cost associated with high-dimensional features.\n2. The paper provides extensive benchmarking against current state-of-the-art methods.", "weaknesses": "1. The implementation details of the proposed 3D autoencoder are unclear. Could you clarify the architecture of the 3D autoencoder (e.g., is it an MLP or another type of model), and specify its number of parameters? Additionally, how is the autoencoder trained—is it trained per scene? What is the duration of the training process?\n2. During inference, text features are also projected into a low-dimensional space for querying. However, since the autoencoder is not trained on a rich set of diverse text features, information loss is likely during encoding, which may impair the model's ability to handle open-world queries. The authors should conduct experiments to assess the autoencoder's impact on text embeddings by randomly selecting open-world terms and evaluating the feature correlation before and after encoding.\n3. The proposed method requires a set of text queries as input. How are these text queries obtained? Do they need to comprehensively include all objects present in the scene? In the experiments, are the open-vocabulary test queries the same as the input text queries? If so, this could bias comparisons with LangSplat, which does not see the text queries prior to training.\n4. Does the proposed method support multi-scale segmentation similar to LangSplat?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9FqY6MC293", "reviewer": "Reviewer_Mv8f", "summary": "This paper introduces econSG, a new approach for open-vocabulary semantic segmentation within 3D Gaussian Splatting (3DGS), addressing challenges with consistency and efficiency in recent methods. Most current open-vocabulary neural fields rely heavily on Segment Anything Model (SAM) to regularize image-level features extracted from CLIP, often without additional refinement, which can lead to inconsistencies across multiple views. Additionally, some methods use dimensionality reduction on 2D semantic features before fusing them with 3D semantic fields, which compromises multi-view consistency.\n\nEconSG overcomes these issues through two main innovations:\n\n1. Confidence-region Guided Regularization (CRR): This approach mutually refines SAM and CLIP features to achieve complete and precise semantic boundaries with enhanced accuracy.\n2. Low-dimensional Contextual Space: By fusing multi-view 2D features into a single 3D representation before dimensionality reduction, econSG enhances multi-view consistency and computational efficiency.", "questions": "I'm confused with the 3D consistency brought by the CRR module.\n- For step b. From my understanding, the consistency here highly depends on the accuracy of the COLMAP depth maps. I understand that average pooling can partially mitigate issues like complex backgrounds or occlusion in individual views, resulting in more stable, 3D-consistent semantic features. But if some view features are affected by occlusion, noise, or low confidence, simple averaging may dilute the features, potentially affecting consistency. Moreover, if the features from different views have large discrepancies, average pooling alone may not fully eliminate these differences. \n- For step c. If a reprojected point is occluded in most of the views, I believe your majority voting strategy may assign a label of the occlusion  to that point instead of the correct label of that point.\nCan you resolve my confusion about the two points mentioned above? Please elaborate on how you ensure your 3D consistency and how your major voting strategy is implemented. Thanks.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper clearly explains its motivation and methodology, with both the algorithmic process and pipeline illustrations presented in a way that is very clear to the reviewers.", "weaknesses": "1. I believe the second contribution Low-dimensional Contextual Space lacks novelty. Similar idea of mapping high-dimensional features into a low-dimensional space in this task can be seen in previous work like one of the baselines, LangSplat[1]. Even if the authors claim that the contribution is to increase the inference speed by changing the space of rendering, recent work like FastLGS: Speeding up Language Embedded Gaussians with Feature Grid Mapping[2] also proposes the same idea of directly rendering low dimensional semantic features.\n2. The qualitative and quantitative experiments are not well-aligned. The baselines that appeared in the quantitative tables are not mentioned in the qualitative comparisons, which reduces the reliability of data in the tables. If the author can provide qualitative results that can align with the quantitative results and more qualitative results that display your 3D consistency, this work can be more convincing.\n\n[1] Qin, M., Li, W., Zhou, J., Wang, H., & Pfister, H. (2024). Langsplat: 3d language gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 20051-20060).\n[2] Ji, Yuzhou, et al. \"FastLGS: Speeding up Language Embedded Gaussians with Feature Grid Mapping.\" arXiv preprint arXiv:2406.01916 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vJNBbOXbEO", "reviewer": "Reviewer_C9FH", "summary": "This paper aims to refine visual foundation models' output and solve the multi-view inconsistency problem in 3D rendering. Specifically, they proposed a confidence-region guided regularization and back projected 2D features into a latent space shared with 3D features. Rendering is performed via alpha blending, supervised by semantic information, 2D features, and color information.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Experiment results seem to be good.\n2. Incorporating visual foundation models into this field for multi-modal learning is interesting.", "weaknesses": "Currently, I vote for 5, marginally below the acceptance threshold, for the reasons below:\n\n1. My primary concern is the lack of explanation on the open-vocabulary experiment settings. In section 5.1, this manuscript does not clarify how their sampling method ensures an open-vocabulary setting or specify which classes are used in training. It will be helpful to provide more explanation on experiments. Additionally, in Table 1, only a few classes are listed for the class-level comparisons. From my current knowledge, they adopted ‘20 different semantic class categories’ in Scannet. It will be beneficial to include the open-vocabulary experiment results in more classes.\n\n2. Writing. Some parts of the manuscript will benefit from fine-tuning for brevity, i.e., ‘The parallel rapid developments of neural 3D scene representation and large multi-modality foundation models naturally lead to research on open-vocabulary 3D scene understanding by leveraging the neural rendering capability of neural fields to align the visual-language models to 3D scenes’. Condensing long sentences can better retain reader’s attention on the main point. Another small note: it seems that there is a typo in section 4.1 in the text feature’s notation T. \n\n3. Back projecting 2D information into a shared latent space with 3D is not entirely novel. It would be helpful to provide more explanation on their unique contribution for low-dimensional 3D contextual space.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t5oiwWON78", "reviewer": "Reviewer_ZWwm", "summary": "This paper introduces econSG, a novel 3D semantic Gaussian model that tackles the challenges of extracting precise semantic features and maintaining multi-view consistency in open-vocabulary neural fields. The proposed method incorporates Confidence-region Guided Regularization (CRR) and a low-dimensional contextual space, resulting in state-of-the-art performance on four benchmark datasets. Notably, econSG also exhibits superior efficiency in training and Inference compared to existing methods.", "questions": "Since the article repeatedly emphasizes the multi-view consistency, I suggest adding some introductions to related research in the related work or introduction.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The paper introduces CRR as a novel approach to refine semantic features from VLMs, specifically focusing on achieving precise boundaries and completeness. The mutual refinement strategy between OpenSeg and SAM is a significant innovation that addresses the limitations of existing methods in capturing accurate and complete semantic information.\n+ By constructing a low-dimensional 3D contextual space, econSG effectively integrates 2D features from different views, improving the consistency and accuracy of 3D scene representation.\n+ EconSG significantly reduces computational complexity and improves training and querying efficiency by mapping high-dimensional features to a low-dimensional space using a pre-trained autoencoder.\n+ The paper demonstrates a notable  improvement in performance, achieving state-of-the-art results on four benchmark datasets.", "weaknesses": "- Despite mentioning an ablation study on CRR, the paper does not include the corresponding experimental outcomes. This gap is detrimental to evaluating the specific impact of each CRR component on the overall effectiveness of the method.\n- Figure 2 shows that the method proposed in this article tends to segment large objects and directly ignores small objects?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)