3D Vision-Language Gaussian Splatting
Authors: Qucheng Peng
Benjamin Planche
Zhongpai Gao
Meng Zheng
Anwesa Choudhuri
Terrence Chen
Chen Chen
Ziyan Wu
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-modal 3D Scene Understanding
Gaussian Splatting
Open-vocabulary Semantic Segmentation
Open-vocabulary Object Localization
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Recent advancements in 3D reconstruction methods and vision-language models have propelled the development of multi-modal 3D scene understanding, which has vital applications in robotics, autonomous driving, and virtual/augmented reality. However, current multi-modal scene understanding approaches have naively embedded semantic representations into 3D reconstruction methods without striking a balance between visual and language modalities, which leads to unsatisfying semantic rasterization of translucent or reflective objects, as well as over-fitting on color modality. To alleviate these limitations, we propose a solution that adequately handles the distinct visual and semantic modalities, i.e., a 3D vision-language Gaussian splatting model for scene understanding, to put emphasis on the representation learning of language modality. We propose a novel cross-modal rasterizer, using modality fusion along with a smoothed semantic indicator for enhancing semantic rasterization. We also employ a camera-view blending technique to improve semantic consistency between existing and synthesized views, thereby effectively mitigating over-fitting. Extensive experiments demonstrate that our method achieves state-of-the-art performance in open-vocabulary semantic segmentation, surpassing existing methods by a significant margin.

Published as a conference paper at ICLR 2025
3D VISION-LANGUAGE GAUSSIAN SPLATTING
Qucheng Pengl.2 Benjamin Planche" Zhongpai Gao², Meng Zheng", Anwesa Choudhuri' Terrence Chen², Chen Chen', Ziyan Wu²
'Center for Research in Computer Vision, University of Central Florida, Orlando, FL, USA United Imaging Intelligence, Boston, MA, USA
1eirst..ast/euiii eeeaatteiiiieeeeeeeeeeeeeeeeeeee -aauii-aiiaa .cmm chen, chennerrov www. qucheng, pengtucf. {first. last
ABSTRACT
Recent advancements in 3D reconstruction methods and vision-language models have propelled the development of multi-modal 3D scene understanding, which has vital upplications in robotics, autonomous driving, and virtual/augmented re- ality. However, current multi-modal scene understanding approaches have naively embedded semantic representations into 3D reconstruction methods without strik- ing balance between visual and language modalities, which leads to unsatisfying semantic rasterization of translucent or reflective objects, well as over-fitting on color modality. To alleviate these limitations, we propose a solution that ad- equately handles the distinct visual and semantic modalities, i.e., 3D vision- language Gaussian splatting model for scene understanding. to put emphasis on the representation learning of language modality. We propose novel tross-modal rasterizer, using modality fusion along with a smoothed semantic indicator for enhancing semantic rasterization. We also employ camera-view blending tech- nique to improve semantic consistency between existing and synthesized views, thereby effectively mitigating over-fitting. Extensive experiments demonstrate that our method achieves state-of-the of-the-art performance in open-vocaben voccb-ooyyl seman- tic segmentation, surpassing existing methods by significant margin.
INTRODUCTION
zahre
The advancement of 3D reconstruction methods, such as neural radiance fields (NeRF) (Mildenhall et al.) 2020) and 3D Gaussian splatting (3DGS) (Kerbl al.) 2023). has enabled the effec- live acquisition of 3D color representa- tions, facilitating high-fidelity and real- time rendering from novel viewpoints. Moreover, vision- languuge models like CLIP (Radford al., (02T) and LSeg (Li et al.] 2022) have been bridging the gap between color images and seman- tic features in 2D space. Given an in- put image, these models can generate dense 2D language map, i.e., assigning semantically rich language embeddings to each pixel (e.g., pixel depicting person's face can be assigned a language embedding de- scribing said face). Building on these developments, multi-modal 3D scene understanding- -which aims learn effective 3D semantic representations from multi-view images and their corresponding camera poses-hall has made significant progress in recent years. This area research has a wide range of applications across various practical domains, including robotic manipulation (Shorinwa al. 2024.. autonomous driving (Gu et al. 2024), and VR/AR (Guerroudji et al.) 2024)
color opacity
12 Lggo
color features
rasterizer
SOTA
Takrin
semantic feats.
poses
rasterizer
Agem
X semantic representation subordinate to the richer color modality.
3D Gaussians
jint ring
color opacity
OURS
color features
rgb
W W0
semantic feats
sem. Indicator
semantic information emphasized still benefits from color guidance.
Figure 1: Comparison of prior semantic 3DGS work and our novel method. We apply cross-modal rasterization and camera-view-basee regularization for better explo- ration of semantic features.
"This work was done during Qucheng Peng's 's internship at United Imaging Intelligence, Boston, MA, USA. Corresponding author.

---

Published as a conference paper at ICLR 2025
Recent methods (Engelmann et al. 2024 Qin et al. 2024 Shi et al. 2024 Zhou et al. 2024b Yu
et al. 2024a) in muulti-modal 3D scene understanding have adopted the paradigm of embedding se- mantic representations into 3D representation for joint reconstruction training, and utilizing semantic knowledge distilled from off-the-shelf vision-language models to guide the training process. For ex-
ample, OpenNeRF (Engelmann et al. 2024) integrates MipNeRF (Barron et al.] 2021) method with OpenSeg model (Ghiasi et al. 20222, and LangSplat (Qin et al.) 2024) employs both 3DGS (Kerbl al. 2023) method and CLIP [Radford al., 2021) model. These solutions rely on 2D supervision to learn a representation back 2D views for comparison with the input observations (input color images and corresponding 2D language maps inferred by aforementioned vision-language models).
(color and semantic) 3D scene representation, i.e., projecting the learned 3D
However, we argue that these methods have naively embedded semantic representations into 3D reconstruction methods like 3DGS ooriinalyy designed for color information-failln to strike balance between visual and language modalities. For example, they are directly applying the color rasterization function- ignoring that this function relies on a color opacity attribute that does not translate to semantic information. Prior art also tends to ignore the unequal complexity and distribution of color and semantic modalities, and the risk of over- fitting the color information to the detriment of the 3D semantic representations. While the same objects may exhibit different colors from various views, their semantic information remains constant. Conversely, different objects can share similar colors, but less desirable for their semantic representations appear identical. Thus, training color representations may negatively impact the training of 3D semantic representations.
to project 3D RGB information to 2D-to the new language modality,
Given these limitations, our intuition is to strike balance between visual and language modalities, rather than simply embedding language features into RGB-based 3D reconstruction. Therefore, we propose novel framework named 3D languael Gaussian splatting, as shown in Fig. | On one hand, we propose novel modsl rasterizer that prioritizes the rendering of language fea- tures. We integrate semantic features with meaningful information from the color domain through modality fusion, prior to rasterization, to facilitate the robust learning of semantic information. Be- sides, we introduce a language-speciff parameter that enables the meaningful blending of language features from different Gaussians. This methodology yields more accurate representation of se- mantic information, especially for translucent or reflective objects, such as glass and stainless steel, where the usage of color opacity may lead to misinterpretation. On the other hand, also propose a novel camera-viers blending augmentation scheme specific to the semantic modality, Le., blending information across views to synthesize new training samples. This process regularizes the language modality through enhancing semantic consistency between the existing and novel views, leading to more robust 3D semantic representations.
All in all, our 3D vision-language Gaussian splatting can be summarized into the following contri- butions:
We propose a cross-modal rasterizer that places greater emphasis on language features. Modality fusion occurs prior to rasterization, accompanied by a learnable and independent semantic indica- tor parameter for the a-blending of language features, enabling more accurate representation of translucent or reflective objects.
We define a camera-view blending technique for the regularization of semantic representations during training, augmenting the input 2D language maps through cross-modal view, i.e., lever- aging the semantic-consistency prior to alleviate over-fitting on the color modality.
Extensive experiments on benchmark datasets demonstrate that our approach achieves state-of- the-art performance in opea-locybulary semantic segmentation tasks, outperforming existing methods by significant margin.
2 RELATED WORK
3D Neural Representations. Recent advancements in 3D scene representation have made signifi- cant progress, particularly with neural radiance fields (NeRF) (Mildenhall et al.) 2020), which excel in novel view synthesis. However, NeRF's reliance on a neural network for implicit scene repre- sentation can result in prolonged training and rendering times. Methods like Instant-NGP (Müller al. 2022) speed up these processes through hash encoding, while approaches such as 3D Gaussian splatting (3DGS) (Kerbl al. 2023) use explicit neural representations for better alignment with
2

---

Published as a conference paper at ICLR 2025
GPU computations via differential tile rasterization.
Additionally, MipNeRR (Barron et al. 2021 2022)
and Mip-Splatting (Yu et al., 2024b) address aliasing issues, whereas DS-NeRF (Deng et al. and DG-Splattnng (Chung al. (2024) focus sparse view reconstructions, In this paper, we utilizze 3D Gaussian Splatting for 3D neural representations.
Visual Foundation Models. Visual foundation models include both pure vision models, crucial for semantic segmentation and feature extraction, and vision-language models that connect images with natural language. Among pure vision models, the Segment Anything Model (SAM) (Kirillov 2023) excels in zero-shot transfer, generating multi-scale segmentation masks. DINO (Caron et al.
al. 2021) and DINOv2 (Oquab al., 2024), trained in self-supervised manner, deliver fine- grained features for various downstream tasks. In vision-language models, CLIP (Radford et al. 2021 utilizes contrastive learning to align visual and textual features, while I. LSeg (Li et 2022) enhances this by incorporating spatial regularization for refined predictions. APE (Shen et al. functions as a universal perception model for diverse multimodal tasks. this paper, we employ SAM and CLIP to extract ground-truth features for baseline comparisons, ensuring fair evaluation.
2024
Multi-modal 3D Scene Understanding. Existing methods for scene understanding learn multi- modal 3D representations from posed images, enabling rendering from novel viewpoints for tasks like open-vocabulary semantic segmentation. For NeRF-based approaches, LERF (Kerr et al. )023) optimizes language field alongside NeRF, using positions and physical scales to generate CLIP vectors. OpenNeRF (Engelmann et al. 2024) introduces a mechanism for obtaining novel camera poses, enhancing feature extraction. Among 3DGS-based methods, LangSplat (Qin al. adopts two-stage strategy for semantic feature acquisition, while GS-Grouping (Ye et al. extends Gaussian splatting for joint reconstruction and segmentation. Additionally, GOI (Qu al. 2024) employs hyperplane division to select features for improved alignment. Moreover, HUGS Zhou al. (2024a) enables holistic 3D scene understanding by integrating 2D semantics and flow with 3D tracking, effectively lifting them into the 3D space.
2024)
2024
METHODOLOGY
3.1 PROBLEM STATEMENT
According to the vanilla Gaussian splatting (3DGS) paradigm applied to RGB image rendering (erbll et al. )023), scene represented by a set of 3D Gaussians G = [gg) J where A denotes their number. Each Gaussian is defined as g" = {u', > o', J'}, i.e., by its mean position μ² E RR covariance matrix E' E R3x3 opacity Of E R. and color properties cf E Rd (e.g., with de = 3 for RGB parameterization). Images are rasterized by splatting Gaussians through each pixel v into the scene and a-blending the Gaussian contributions to the color C(v), as:
N Σ
((v)
a p²) with p = - 1=1
(I)
where 7/' and E. the 3D-to-2D 2D projections of jL and Σ This scene representation is learned from a training set 7 {(FF, WT), (15, W consisting of several pairs of RGB images I of the target scene and the corresponding camera poses W.
Gaussian splatting models have been expanded to incorporate language-embedding information densely describing the scene (Qin et al. 2024; Zhou et al. 2024b Qu al.) (2024] by adding language feature vector f E RR (with d, feature size) to each 3D Gaussian and similarly rasteriz- ing this modality. Le.. 2D semantic embedding F' pixel can be expressed as:
N F(v) = Σ f II (1-00 (1P))))
1-1
(2)
Even though the two modalities have widely different properties (e.g., translucence only applies to the visual modality and not the semantic one), previous methods have been directly applying the color rasterization process to the language features without any adaption, i.e., only replacing c' (Eq. I} by f (Eq. 2). In this paper, we propose to adapt the usual rasterization scheme to better fit the Tanguage-feature modality.
3

---

Published as a conference paper at ICLR 2025
A] Multimodal Gaussian Representation
B] Data Enrichment
incut poses
position
Jf
CLIP
SAM
covariance
21
color
cross-modal
opacity
rasterization
images
Ennglaee maps
nput mages
language fest.
D) Blending:
Jang. indicator
blending
4a
CJ Cross Modal Rasterization:
- LERPNG
self- aite ntio
ff (o' et!)
rsterrer
lanquege maps
SLIRFIO Ao-
Figure 2: Overview of our proposed framework. A) We propose a novel multi-modal Gaussian splatting model; B) enrich the input images and poses for the model better fit the semantic information. Besides our introduction of a novel semantic indicator parameter I, our additional contributions are: C) a semantic-aware cross-modal rasterization module; and D) a camera view blending augmentation scheme for training regularization.
To train these semantically-enriee enriched 3DGS models, the standard procedure consists of first gener- ating the set of language-feature maps H corresponding to the input images Off-the-shelff pure vision models like SAM (Kirillov et al. 2023) and vision-language models like CLIP (Rad- ford et al., 2021) are typically used, along with an auto-encoder-basee Zhai et al. (2018) dimension reduction as Qin et al. (2024). Once the data is prepared, the 3D Gaussians can be iteratively optimized by minimizing the distance between its rasterized 2D semantic embeddings (c.f. Eq. 2 and the ground-truth 2D semantic embeddings:
L = (I,))T Ewer frem FFW (v), H (u)).
where L is the overall optimization objective. Besides, FW and HW the predicted and ground- truth 2D language maps rendered using camera pose W corresponding to image I (for ease of readability, drop the superseript W in subsequent equations), and Cyem is distance function for 2D semantic maps (e.g., Ll distance).
The resulting 3D Gaussian language representation be leveraged for open-vocabulary tie queries. For example, given a query language vector т, pixel-wise 2D relevancy score map corresponding to view W can be computed as (Radford et al. 2021)):
F(U) ((T) [P)()
F(u) sps) |[())]]|<<>)]]
p(T = exp(
Σ exp(
(4)
SET
where so is the text encoder from the lisione model. Such a relevancy map can be leveraged, egg., for open-vocabulary 2D localization (i.e., by measuring the argmax response the query) or semantic segmentation (i.e., by thresholding the resulting relevancy map).
3.2 SEMANTIC-AWARE RASTERIZATION
Existing multi-modal rasterization methods Qin al. 2024 Zhou 2024b Zuo et 2024 have largely drawn from color-based 3DGS (Kerbl et al. 2023). These rasterizers approach the rendering of 3D semantic information in manner akin RGB rendering (see Sec. 3.1), resulting in an insufficient focus semantic- -spectic- design. To address this gap, we propose novel cross- modal rasterizer that emphasizes semantic-specific design, as illustrated in Fig. A and Fig.
A first noticeable shortcoming is the insufficient integration and exchange between 3D semantic and color features. While these modalities have distinet properties, they offer correlated and complemen- tary information about the scene, with knowledge from one informing the other. Current models fail to leverage this synergy, resulting in inadequate guidance for learning semantic representations from the richer color modality.
To address this shortcoming, we propose integrating 3D semantic and color features prior to ras- terization, rather than treating them independently (Qin al. 2024 Qu et al. 2024), which facili- tates effective knowledge fusion and exchange between modalities. This intuitive improvement to multi-modal 3DGS has surprisingly been ignored in prior work. We decide to correct this oversight
4

---

Published as a conference paper at ICLR 2025
RGB Image
F o
Density Distribution of r'
Figure 3: Empirical differences between color opacity and proposed smoothed semantic indicator. On the left, visualize the difference ( -f in Gaussians modeling the ramen scene. While their color opacity may vary significantly, most Gaussians need their semantic features to be rasterized with minimal blending (c.f. red regions in the difference maps, Le., where 1' >> o'), except for Gaussians representing intangible lighting effects (cff. glares on the bottle, bowl, table, etc.). On the right, we further plot the density distribution of I o' Gaussians for the ramen scene, indicating the different distributions of these two control parameters.
by applying the well-established self-attention mechanism for modality fusion before rasterization, thereby enhancing its effectiveness. We can derive the 3D modality- features u² E RRd++dy for our rasterizer as:
u' = + f Voout (softmax volc' @ f') + f*)
wivldss f*)),
(5)
Vdn
where dn represents the number of heads. Moreover, th, d/ vv Idd + Rdd x Rd and Pout : Rds+td, x dd + Rd.-d, are single-layer linear networks.
Another salient shortcoming the simplistic adoption of e-blenndigg-bas based RGB splatting (Eq. D to rasterize language embeddings, only substituting the 3D color features c with 3D semantic repre- sentations f (Eq. 2). In other words, previous works use the opacity attribute of Gaussians to render not only RGB images but also language maps, c.f. Eq. (either optimizing the opacities values over the two modalities (Shi al.] 2024; Zhou al.) 2024b). or applying frozen opacity values from the RGB-only pre-training during semantic rasterization (Qin et al. (024)). We argue that this ap- proach fails to recognize that in phenomena involving semi-opaque media (e.g., glass, water) and complex light transport effects (e.g., direction-dependent scattering and glare), color opacity cannot be effectively translated to the semantic modality. In most cases, when Gaussians represent tangi- ble elements of the scene (e.g., the surface volume of object), their semantie features should be splatted to 2D without any reduction in opacity. Conversely, intangible Gaussians (e.g., those simulating lighting effects like optical glares) should be excluded from semantic rasterization by enforcing their a-blending weight close to zero.
To address this problem, propose novel a-blending strategy specifically designed for explor- ing semantic information. We introduce a new learnable attribute for our multi-modal Gaussians: a smoothed semantic indicator I [0, 1]. which is applied to the rasterization of language embed- dings, effectively replacing the color opacity parameter for this modality:
U(v) Σ Lid pd (1-OPP))
Σ Wd P (P-PP))
(6)
f=1
j=1
color modality (channels d.)
language modality (channels d.+1 de+dd)
Note that these two processes conducted simultaneously in the rasterizer: By disentangling the semantic rasterization from the opacity-based control and letting the overall model learss an inde- pendent per- Gaussian semantic indicator, allow the model to better fit the semantic information of the scene. This is highlighted in Fig. B which shows how much the distribution semantic indicator values differs from the color opacity after optimization. This is especially notable for
5

---

Published as a conference paper at ICLR 2025
RGB
L.Gaussians
LangSplat
GOI
Ours
Ground Truth
'glass'
'kamaboko'
'cup'
'onion' f
'chopstick'
Figure 4: Qualitative semantic segmentation comparisons on the amen scene (LERF dataset).
RGB
L.Gaussians
LangSplat
GOI
Ours
Ground Truth
seiuri
deint
'niffe'
'ponts
'wevvee
'saaseer
'ottolenghi'
Weekk'
Figure 5: Qualitative semantic segmentation comparisons on the tchen scene (LERF dataset).
semi-transparent or reflective objects (e.g., glass and pot in depicted scene), exhibiting low opacity but high semantic-indicator values (c.f. P o' close to 1).
After rasterization, the resulting pixel information U (v) can be further decomposed as:
U(v) C(v) e F(v),
where C(v) and F(v) are the RGB color and 2D language embeddings at pixel e respectively.
Based on the above, formulate the loss function optimize the overall scene representation as follows, V(I, W) T
Craster (W) Ever [MsE (C (v), + LASE (F" (v), (v)),,
where LMSE represents the mean squared error (MSE) criterion.
3.3 SEMANTIC-AWARE CAMERA VIEW BLENDING
While the novel cross-modal rasterization process proposed Sec. 3.2 facilitates the efficient learn- ing of joint color and semantic representations, this multi-modal approach may be suboptimal for applications focused exclusively on semantic information-suct as open-vocabulary semantic seg- mentation tasks-dus to the adverse effect from richer color modality. On the one hand, though the same object may exhibit different colors from various viewpoints, its semantic information re- mains consistent. On the other hand, different objects might share similar colors, which lead to undesirable identical semantic representations. Therefore, it is crucial to incorporate regularization methods that enhance semantic consistency across diverse views, thereby mitigating the overfitting on semantic representations.
To this end, we propose regularization technique called camera view blending (shown in Fig. 23), designed specifically to augment the semantic modality by leveraging multi-modal information. Inspired by mix-up augmentation Zhang al. (2018), our approach involves blending two ground- truth 2D language maps to create new ones, while providing semi-consistent corresponding camera poses through advanced 6DOF pose interpolation. For two randomly selected samples (I1, Wi) and (T2, W2) from the training set 7", where Wi * W2, we first utilize the camera poses Wi and W2 to synthesize novel camera pose that incorporates both rotation and translation components.
Expressing the input rotations with quaternions 91 and 92, we apply spherical linear interpolation 91-42 (Slerp) (hhoemake 1985). Define arccos [ we and let the interpolation ratio be randomly
sampled from Beta 0.2, 0.2). The quatermion for the novel camera view is then given by:
h: Q1
cos 0
(1 k)
if cos 0 > 0.995
[
cos 0
q6
91
sin
0
kd)
cos 0
22 sin (k0)
if cos 0 S 0.995.
[|
sin 0
cos 0
|/
sin
0

---

Published as a conference paper at ICLR 2025
When the two quaternions are nearly aligned in direction cos 0 > 0.995), we use linear interpola- tion (Lerp) for improved efficiency. If they are not aligned ( cos 8| - 0.995), Slerp is applied.
For the translation component, use linear interpolation synthesize the focal center of the novel view. Given the camera centers & and 22 in the world coordinate system, corresponding to W1 and W2 respectively, the new center is determined with the same interpolation ratio k:
to = kt1 + (1 k))t2
(10)
With the blended quaternion qb and camera center tp. empirically derive the novel camera view Wo, which can be used to render the associated 2D semantic embeddings from the Gaussian model.
Our goal is to enhance the semantic consistency across existing views and their associated synthe- sized views, thereby regularizing the training of 3D semantic representations. Thus, the camera view blending loss L, for semantics with identical interpolation ratio k is:
C. SSIM( 125 CMSE (FW(()) (v), H (w)) with H =HH W: + H (11)
Here FW,(v) (v) denotes the rendered 2D semantic embeddings based on W, using the rasterizer de- scribed in Section 3.2, while 4t"i(o (v) and H (v) represent the ground truth 2D semantic features for W1 and W2 respectively. When the two images differ significantly, regularization may be coun- terproductive. To address this, we utilize the Structural Similarity Index Measure (SSIM) between 1 and 12 as a weighting factor. By enforcing similarity between H Hw)(v) and FWo (w). we enhance the semantic consistency of the 3D representations, thereby mitigating the overfitting issue.
By combining the rasterization loss (Eq. 8 and the camera view blending loss (Eq. 11 according to trade-off hyper- prrree-paraeeter \, the overall objective is represented
Loverall = E, W.).( Craster (W1) + Lraster (W2) + WW,, W2)]
(12)
4 EXPERIMENTS
4.1 SETTINGS
Datasets. We employ 3 datasets for evaluation open-vocabulary semantic tasks. (1) LERF dataset (Kerr et al. 2023). captured using the Polycam application on an iPhone, comprises complex, in-the-wild scenes and specifically tailored for 3D object localization tasks, We report the mean Intersection over Union (mloU) results, alongside localization accuracy results in accordance with Qin al., 2024 (2) 3D-OVS dataset (Liu et 2023) consists of a diverse collection of long-tail objects in varios poses and backgrounds. This dataset is specifically developed for open- voccabuayyy 3D semantic segmentation and includes a complete list of categories. Following (Qin et al., 2024] Zuo et al., 2024), the mioU metric is applied for this dataset. (3) Mip-NeRF 360 dataset (Barron cct al. 2022) contains 9 scenes, each composed of a complex central object or area and detailed background. Following Qu et al. 2024), we provide some evaluations on this dataset in annex.
Implementation. To extract ground-truth semantic features from each image, we utilize the SAM ViT-H model (Kirillov et al.) 2023) alongside the OpenCLIP BTT6 model (Radford al. 2021). The 3D Gaussians for each scene are initialized using sparse point clouds derived from Structure from Motion. For modality fusion, we set de and d to 3, while d, is set to 4. During rasterization, smoothed semantic indicator is initialized in the same manner as color opacity. For each iteration, two camera views and their associated images are selected, and 3D Gaussians iterations. Moreover, the parameter À in Equation [2] is configured to 1.2. The implementation of semantic opacity is done in CUDA and C++, while the other components are in PyTorch. All ex- periments are conducted on Nvidia A100 GPUs. For each scene, our model is trained for 15,000 iterations using an Adam optimizer (Kingmal (014), and the learning rates of different components are shown in the appendix. After convergence, the model is applied to the localization and segmen- tation tasks according to the procedure described Sec.
trained for 15,000
4.2 RESULTS
Baselines. For fair comparison, select the latest works open-vocabulary 3D scene under- standing: Feature-DDGS (Zhou et al. 2024b), LEGaussians (Shi et al. 2024). LangSplat QQin
et al. 2024), GS-Grouping (Ye al. 2024). GOI (Qu et al. 2024), and F FMGS (Zuo al. 2024).
7

---

Published as a conference paper at ICLR 2025
all of which based on 3DGS Kerbl al. 2023), and employ the same segmentation and vision- language models mentioned in Sec. 4.1 to extract ground truth language features. important to note that FMGS (Zuo et al. 2024) does not report mloU results on the LERF dataset and is also not open-sourced, so its mloU results are not listed.
Quantitative Evaluation. Tab. [ and Tab. present the mloU and localization accuracy results on the LERF dataset, respectively. Tab. [ displays the mloU results on the 3D-OVS dataset. Our proposed method achieves state-of-the-ar performance across all scenes, notably outperforming the second best method LangSplat (Qin et al., 2024) by 10.6 in mloU on the LERF dataset. Further experiments on other datasets are listed in the appendix.
Qualitative Evaluation. To facilitate qualitative comparisons, we contrast our method with several baseline approaches, including LEGaussian (Shi al. 2024), LangSplat (Qin al. (024). and GOI (Qu et al. 2024). Fig. and Fig. F illlstraat semantic segmentation results on LERF data, while Fig. ppreeents the localization results on the same dataset. For the localization task, the black bounding boxes with dotted lines represent the ground truth ranges, while the red dots indicate predictions from various methods. Besides, segmentation comparisons on the 3D-OVS dataset are displayed in F and Fig.8 It is clear that our proposed method outperforms the other approaches and is closer to the ground-truth, particularly reflective and translucent areas. Additional results are displayed in the appendix.
4.3 ABLATION STUDIES
Ablation on cross-modal rasterizer. Tab. A presents an ablation study on the data-fusion of the rasterizer. In addition layer MI. and cross-attention for modality Fusion to enhance semantic features for comparison. The results indicate that self-attention modality fusion the most effective method. Besides, single- layer MLP and cross-attention modality fusion do not demonstrate any advantage over the no fusion condition. In Tab. B, we highlight the benefits of our proposed smoothed semantic indicator. We compare it to previous methods for rasterizing semantic embeddings. specifically by applying the color opacity to the language modality (Qin et al., 2024 Zhou et al., 2024b Zuo et al,, 2024 The results indicate that using separate attribute for rasterizing language embeddings is clearly advantageous. However, this new attribute cannot be naively fixed, e.g., to I or 0.5 for all Gaussians. Its values should be learned by the model through 2D supervision, which is why introduce a learnable indicator for the a- a-blending of language modality.
fusion like LEGaussians (Shi et al.) (202)) we also incorporate single-
Ablation on camera-view blending. We also present ablation of different camera-view blending strategies in Tab. [ which includes three variants: Rotation that associates to the operation in Eq 9 Translation that corresponds the operation Eq. TO,, and SSIM that weighs Eq. [T] Our proposed method corresponds to the last row of the table. The results indicate that both Slerp-basee rotation blending and serp-based translation blending contribute to performance improvements. Moreover, the regularization provided by SSIM is essential for controlling the extent of blending.
In Tab. C, we present an ablation study on the interpolation ratio k. In addition fixed values of 0.5, 0.75, and 1.0, we also include experiments based on Even Distribution (U(0, 1)), Gaussian Distribution (N(0, 1)), and our Beta Distribution (Beta(0.2, 0.2)). The results demonstrate that balanced ratio is crucial for acquiring better representations, and that slight disturbances can be beneficial as well.
Table 1: mloU results on the LERF dataset.
Method Feature-3DGS
Venue
ramen
teat ime
figurines
kitchen
avg.
CVPR 24
43.7
58.8
40.5
39.6
45.7
LEGaussians
CVPR'24
46.0
60.3
40.8
39.4
46.9
LangSplat GS uro-Gru GOI Ours
CVPR
51.2
65.1
44.7
44.5
51.4
ECCV'24
45.5
60.9
40.0
38.7
46.3
ACMMM'24
52.6
63.7
44.5
41.4
50.6
61.4
73.5
58.1
54.8
62.0
8

---

Published as a conference paper at ICLR 2025
Table 2: Localization accuracy (%) on the LERF dataset.
Method
Venue
ramen
teatin ime
figurines
kitchen
avg.
reature-3DGi
CVPR 24
69.8
77.2
73.4
87.6
77.0
LEGaussians
CVPR'24 24
67.5
75.6
75.2
90.3
77.2
LangSplat
CVPR
73.2
88.
80.4
95.5
84.3
GS-Grouping
ECCV'24
68.6
75.0
74.3
88.2
76.5
GOI
ACMMM'24
75.5
88.6
82.9
90.4
84.4
FMGS
IJCV'24
90.0
89.7
93.8
92.6
91.5
Ours
92.5
95.8
97.1
98.6
96.0
Table 3: mloU results on the 3D-OVS dataset.
Method
Venue
bed
bench
room
sofa
1awn
avg.
FeaSure-3DGS
CVPR 24
83.5
90.7
84.7
86.9
93.4
87.8
LEGaussians
CVPR 24
84.9
91.1
86.0
87.8
92.5
88.5
LangSplat
CVPR 24
92.5
94.2
94.1
90.0
96.1
93.4
Grourng
2C
83.0
91.5
85.9
87.3
90.6
87.7
GOI
ACMMM 24
89.4
92.8
91.3
85.6
94.1
90.6
FMGS
IJCV'24
80.6
84.5
87.9
90.8
92.6
87.3
Ours
96.8
97.3
97.7
95.5
97.9
97.1
LEGaussians
LangSplat
GO1
Ours
'kamaboko'
'hear
nose
'stainless toaster'
Figure Localization comparisons on LERF Black dotted-line bounding boxes represent the ground-truth targets, and red dots indicate the predictions (considered correct if within GT box).
RGB
LEGaussians
LangSplat
GOI
Ours
Ground Truth
'dull'
'woff'
'cor"
'egg tury'
'mini car'
'enape'
Figure 7: Qualitative semantic segmentation comparisons the bench scene (3D-OVS dataset).
RGB
LEGaussians
LangSplat
GOI
Ours
Ground Truth
Switch controller'
'card'
'Gundamm '
'Pikacku
'vufaa
Nbax controller"
Figure 8: Qualitative semantic segmentation comparisons on the so sofa scene (3D-OVS dataset).

---

Published as a conference paper at ICLR 2025
Table 4: Ablation results of various contributions the LERF dataset, in terms of mloU.
Method
ramen
figurines teatime
kitchen
avg.
No Fusion
57.8
69.6
54.2
51.5
58.3
(A)
Fusion
module
Single-layer MLP Fusion
55.9
68.2
53.0
49.9
56.8
Cross-attention Modality Fusion
57.3
70.7
54.6
52.0
58.7
Sclf-attention Modality Fusion (Ours)
61.4
73.5
58.1
54.8
62.0
B Indicator value
Fixed to 0.5
52.2
70.9
50.6
50.8
56.1
Fixed to 1.0
50.3
67.7
47.1
48.5
53.4
Using Color Opacity
55.9
71.0
54.2
51.3
58.1
Smoothed Semantic Indicator (Ours)
61.4
73.5
58.1
54.8
62.0
(C) Interpolation ratio sampling
Fixed 0.5
58.8
72.6
56,4
52.5
60.1
Fixed to 0.75
56.3
71.0
50.5
51.7
57.4
Fixed to 1.0
55.2
68.3
48.5
51.0
55.8
Even Distribution
58.3
55,4
51.2
59.0
Gaussian Distribution
57.5
71.3
53.9
50.7
58.4
Beta Distribution (Ours)
61.4
73.5
58.1
54.8
62.0
Table Ablation results of Camera View Blending on the LERF dataset, in terms of mloU.
Rotation
Translation
SSIM
ramen
teat i
figurines nes
kitchen avg.
X
55.4
68.2
48.8
50.9
55.8
X
56.6
69.5
49.7
51.5
56.8
V
V
58.3
70.9
51.1
52.4
58.2
55.7
69.3
49.6
51.7
56.6
X
V
57.9
70.2
52.3
52.1
58.1
V
X
59.6
71.8
53.4
53.2
59.5
V
61.4
73.5
58.1
54.8
62.0
4.4 EFFICIENCY ANALYSIS
Finally, demonstrate that, despite the parameter added to the Gaussians, our con- tributions actually benefit the efficiency of the resulting scene representation, i.e., in terms of convergence speed and rendering time. As shown in Tab. 66] our model outperforms the other four baselines efficiency. We attribute this performance to our cooperative training over augmented view/pose batches, and to the increased ability of the Gaussians to accurately fit both target modalities.
Table 6: Efficiency analysis LERF's ramen.
Training
# of
Method
FPS f
Time
Gaussians +
LangSplat GS GS-Grond ES-Grouping
96min
40
86k
130min
76
107k
GOI
73min
42
92k
Ours
65min
80k
CONCLUSION
In this work, we propose 3D visual-language Gaussian splatting for semantic scene understanding, addressing the neglect of language information in current 3DGS approaches. We propose novel cross-modal rasterizer that performs modality fusion followed by language-specific rasterization, utilizing smoothed semantic indicator disable irrelevant Gaussians, e.g., for scenes with com- plex light transport (reflections, translucence, etc.). Moreover, our camera-view blending technique effectively mitigates over-fitting ensuring semantic consistency across both existing and synthesized viewpoints. Comprehensive experiments validate the effectiveness of our framework, demonstrating significant improvements in open-vocabee vocabulayy semantic segmentation compared to prior Future research will explore further modalities that could enrich the scene representation (e.g., semantic features from mixtures of foundation models), as well as extending this work dynamic scenes.
10

---

Published as a conference paper at ICLR 2025
Ethical Statement
We have carefully read the ICLR Code of Ethics. We are sure that all studies and procedures de- scribed in the paper follow the ethical rules in the academia. Our research does not involve human subjects, and also does not contain any potentially harmful insights, methodologies and applications, Furthermore, do not expect any direct disrimiinatonn/ias/ainnss concerns, privacy/security issues relating to this paper.
Reproducibility Statement
All key technical details and quantitative results are provided in the main paper and Appendix, enabling researchers and practitioners in this field to reproduce our work. Additionally. all datasets used in this research are publicly available to the community. We will also release the source code of our method upon acceptance of the paper.
REFERENCES
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin Battin--rualla,, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5855-5864, 2021.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5470-5479, 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Jacyoung Chung, Jeongtack Oh, and Kyoung Mu Lee. Depth-regularized optimization for 3d gaus- sian splatting in few-shot images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 811 111- 2024.
Kangle Deng, Andrew Liu, Jun- 7hu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12891, 2022.
Francis Engelmann, Fabian Manhardt, Michael Niemeyer, Keisuke Tateno, and Federico Tombari. Opennerf: Open set 3d neural scene segmentation with pixel-wise features and rendered novel views. In The Twelfth International Conference on Learning Representations, 2024.
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In European Conference on Computer Vision, pp. 540-557. Springer, 2022.
Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In 2024 IEEE International Con- ference Robotics and Automation (ICRA), pp. 5021-5028. IEEE, 2024.
Mohamed Amine Guerroudji, Kahina Amara, Mohamed Lichouri, Nadia Zenati, and Mostefa Mas- moudi. A 3d disualization-base augmented reality application for brain tumor segmentation. Computer Animation and Virtual Worlds, 35(1):e2223, 2024.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Trans. Graph., 12(4):139-1,,022 2023.
Justin KeTI, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Lan- guage embedded radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19729-19739, 2023.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
11

---

Published as a conference paper at ICLR 2025
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF International Conference Computer Vision, pp. 4015-40266 2023.
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. In International Conference Learning Representations, 2022.
Kunhao Liu, Fangneng Zhan, Jiahui Zhang, Muyu Xu, Yingchen Yu, Abdulmotaleb El Saddik, Christian Theobalt, Eric Xing. and Shijian Lu. Weakly supervised 3d open-vocabulary segmen- tation. Advances Neural Information Processing Systems, 36:53433-5344 2023.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Computer Vision-ECCC 2020, pp. 405-421, 2020.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with a multiresolution hash encoding. ACM transactions on graphics (TOG), 41(4):1-15, 2022.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Journal, pp. 1-31, 2024.
Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d lan- guage gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20051-20066 2024.
Yansong Qu, Shaohui Dai, Xinyang Li. Jianghang Lin, Liujuan Cao. Shengchuan Zhang, and Ron- grong Ji. Goi: Find 3d gaussians of interest with an optimizable open- voccbvocabulary semantic-space hyperplane. In Proceedings of the 32nd ACM international conference on multimedia, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
Yunhang Shen, Chaoyou Fu, Peixian Chen, Mengdan Zhang, Ke Li, Xing Sun, Yunsheng Wu, Shaohui Lin, and Rongrong Ji. Aligning and prompting everything all at once for universal visual perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13193-13203, 2024.
Jin-Chuan Shi, Miao Wang, Hao-Bin Duan, and Shao-Hua Guan. Language embedded 3d gaus- sians for open-vocabulary scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5333-5343, 2024.
Ken Shoemake. Animating rotation with quaternion curves. Proceedings of the 12th annual conference on Computer graphics and interactive techniques, 1985. URL https://api.
Olaolu Shorinwa, Johnathan Tucker, Aliyah Smith, Aiden Swann, Timothy Chen, Roya Firoozi, Monroe David Kennedy, and Mac Schwager. Splat-mover: Multi-stage, open-vocabulary robotic manipulation via editable gaussian splatting. In 8th Annual Conference on Robol Learning, 2024.
Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke. Gaussian grouping: Segment and edit anything in 3d scenes. In European Conference on Computer Vision. Springer, 2024.
Justin Yu, Kush Hari, Kishore Srinivas, Karim El-Refai, Adam Rashid, Chung Min Kim, Justin Kerr, Richard Cheng, Muhammad Zubair Irshad, Ashwin Balakrishna, Thomas Kollar, and Ken Gold- berg. Language-embedded gaussian splats (legs): Incrementally building room-scale representa- tions with mobile robot. In Proceedings of the IEEE International Conference on Intelligent Robots and Systems (IROS), 2024a.
12

---

Published as a conference paper at ICLR 2025
Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias- free 3d gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19447-19456, 2024b.
Junhai Zhai, Sufang Zhang, Junfen Chen, and Qiang He. Autoencoder and its various variants. In 2018 IEEE international conference on systems, man, and cybernetics (SMC), pp. 415-419. IEEE, 2018.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Hongyu Zhou, Jiahao Shao, Xu, Dongfeng Bai, Weichao Qiu, Bingbing Liu, Yue Wang, Andreas Geiger, and Yiyi Liao. Hugs: Holistic urban 3d scene understanding via gaussian splatting. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp- 21336-20 21345, 2024a.
Shijie Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang, and Achuta Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. In Proceedings of the IEEE/CVF Conference on Com- puler Vision and Pattern Recognition, pp. 21676-21685, 2024b.
Xingxing Zuo, Pouya Samangouei, Yunwen Zhou, Yan Di, and Mingyang Li. Fmgs: Foundation model embedded 3d gaussian splatting for holistic scene understanding. International Journal of Computer Vision, pp. 1-17, 2024.
A APPENDIX
A.1 FURTHER IMPLEMENTATION DETAILS
We complement the implementation details shared in Subsection 4. official implementations of 3DGS Kerbl et al. (2023) and LangSplat Qin et al. (2024) made publicly available by their respective authors (c.f. links provided in their papers). Specifically, we edit the LangSplat's 3DGS model to add our novel smoothed indicator, as well as the CUDA rasterization function for joint rendering of color and semantic modalities w.r.t. their respective opacity/indicator control. As explained in the main paper, we also adapt the optimization script for joint learning over both modalities and for integrating the proposed view-blending augmentation, applied to pairs of batched input samples at every iteration. The learning rates applied to the different 3DGS attributes are provided Tab.
solution is based the
Table Learning rates applied the parameters of the Gaussians.
Components
Learning Rates
position
1.6 x 10
scale
5.0 10
rotation
1.0 10
color features
5,0 x 10-3
color opacity
5.0 10-2
semantic features
5.0 10-3
semantic indicator
5.0 x 10-2
For larger version of Sub-figs. C-D representing fusion module and augmentation module respectively, we refer the readers to Fig.
In the ablation cross-modal rasterizer (Sec. 4.3) the cross-attention fusion tested in Tab. 4 is performed by using f as query and c² as both key and value, whereas the adopted self-attention performs cross-modality fusion by using et <b I as query, key, and value.
13

---

Published as a conference paper at ICLR 2025
C) Cress-Modal Rasterization
D) View Blendingg
H LERP(H, ,. M2.
4b
self- attention
rasterierr
o' +
fp
42 KJ
SLERP 42
Figure 9: Larger version of Sub-figs. C-D describing fusion and augmentation modules.
Table mloU results on the Mip-NeRF 360 dataset.
Method
Venue
garden
bonsai
kitchen
room
avg.
Feature-3D
CVPR 24
45.1
46.2
46.8
17.5
38.9
LEGaussians
CVPR 24
48.5
50.4
49.3
55.7
51.0
LangSplat
CVPR '24
50.1
59.1
50.0
62.6
55.5
GS-Grouping
2CCV'24
42.0
43.1
42.2
49.1
44.1
GOI
ACMMM
85.0
91.5
84.3
85.0
86.5
Ours
89.2
93.7
87.6
88.8
89.8
A.2
EVALUATION ON ADDITIONAL MIP-NERF 360 DATASET
A.2.1 QUANTITATIVE RESULTS.
Building on the work of GOI (Qu et al. 2024). we also utilize the Mip-NeRF 360 dataset (Barron al. 2022) for our evaluation. In this context, employ APE (Shen et 2024) as the frozen backbone to extract ground truth 2D features. The results, presented in Tab. 8 clearly demonstrate the effectiveness of our proposed method, highlighting its ability to improve overall performance on this dataset.
A.2.2 QUALITATIVE RESULTS.
In Fig. 10 and Fig. II present qualitative comparisons the Mip-NeRF 360 dataset (Barron
et al. 2022). The results demonstrate that our proposed method significantly outperforms the com- peting approaches.
RGB
LEGaussians
LpngSplat
GOI
Ours
Ground Truth
'icyylee
'able
'rne'
Akybound
Figure 10: Qualitative semantic segmentation comparisons the bonsai scene of Mip-NeRF 360.
RGB
LEGaussians
LangSplat
GOI
Ours
Ground Truth
'glass'
'book*
'ees'
'wouden floor'
'curpee'
Figure 11: Qualitative semantic segmentation comparisons on the room scene of Mip-NeRF 360.
A.3 ADDITIONAL. ABLATION STUDIES
A.3.1 HIGH-LEVEL. ABLATION OF PROPOSED CONTRIBUTIONS.
We first provide some high-level ablation studies Tab. and Tab. 10, highlighting the impact each of our key contributions, iee., proposed modality fusion (" modal. fus."), semantic indicator (*
14

---

Published as a conference paper at ICLR 2025
Table 9: Ablation results of three key contributions the LERF dataset, in terms of mloU.
modal. fus.
sem. indic.
view blend.
ramen
teat ime
figurines
kitchen
avg.
53.3
66.9
46.3
45.3
53.0
54.7
68.5
47.7
45.8
54.2
X
X
55.4
67.2
47.8
48.9
54.8
54.8
67.5
48.2
48.6
54.8
X
V
57.8
69.6
54.2
51.5
58.3
55.9
71.0
54.2
51.3
58.1
V
55.4
68.2
48.8
50.9
55.8
61.4
73.5
58.1
54.8
62.0
Table 10: Ablation results of three key contributions on the 3D-OVS dataset, in terms of mloU.
modal. fus.
sem. indic.
view blend.
bed
bench
room
sofa
lawn
avg.
92.7
94.3
94.1
90.5
96.0
93.5
X
93.0
94.6
94.9
91.6
96.1
94.0
93.6
95.1
95.0
92.2
96.0
94.4
X
93.9
95.2
95.3
91.8
96.2
94.5
X
V
94.5
95.6
95.3
93.0
96.5
95.0
X
94.7
95.2
95.0
92.7
96.4
94.8
V
92.9
94.6
95.2
91.5
96.1
94.1
V
96.8
97.3
97.7
95.5
97.9
97.1
Table 11: Ablation results of modality fusion the 3D-OVS dataset.
Method
bed
bench
room
sofa
lawn
avg.
No Fusion
94.5
95.6
95.3
93.0
96.5
95.0
Single-layer MLP Fusion
93.1
94.7
94.4
91.7
96.0
94.1
Cross-attention Modality Fusion
94.7
95.9
95.6
93.2
96.6
95.2
Self-attention Modality Fusion (Ours)
96.8
97.3
97.7
95.5
97.9
97.1
Table 12: Ablation results of smoothed semantic indicator on the 3D-OVS dataset.
Method
bed
bench
room
sofa
lawn
avg.
Fixed to 0.5
93.4
94.9
95.1
92.0
95.4
94.2
Fixed to 1.0
91.5
92.8
90.9
87.6
92.8
91.1
Using Color Opacity
94.7
95.2
95.0
92.7
96.4
94.8
Smoothed Semantic Indicator (Ours)
96.8
97.3
97.7
95.5
97.9
97.1
sem. indic."), and camera-view blending ("view blend.") components crucial in improving the overall performance.
The results demonstrate that all three
A.3.2 ABLATION OF MODALITY FUSION
Table II
presents ablation study of modality fusion using the OVS-3D dataset. The results indicate that self-attention modality fusion is the most effective method.
A.3.3 ABLATION OF SEMANTIC INDICATOR
Tab. 12 compares our proposed Smoothed Semantic Indicator against three alternatives: Fixed Value at 0.5, Fixed Value at I, and Using Color Opacity on the 3D-OVS dataset. The results demonstrate that fixed values greatly under-perform compared to shared opacity derived from color rendering. Moreover, the learnable Smoothed Semantie Indicator surpasses the of color opacity signifi- cantly. Thus, utilizing a learnable semantic indicator for semantic rasterization proves beneficial for the open-vocabulary segmentation task.
15

---

Published as a conference paper at ICLR 2025
Table 13: Comparison between data-augmentation strategies, downstream open-vocabulary semantic segmentation (mloU) on LERF scenes. Note that the original training sets contain between 124 and 297 images per scene. For the offline augmentation using 3DGS-renderee images and off- the-shelf open-vocabulary models (SAM/CLIP), 120 novel views are added, with their viewpoints sampled via interpolation.
Method
ramen
teat ime
figurines
kitchen
avg.
w/o data augmentation
53.3
66.9
46.3
45.3
53.0
w/ data from 3DGS & SAM/CLIP
55.2
68.1
49.5
48.2
55.3
w/ view blending
61.4
73.5
58.1
54.8
62.0
Table 14: Ablation results of camera view blending the 3D-OVS dataset.
Rotation
Translation
SSIM
bed
bench
room
sofa
lawn
avg.
92.9
94.6
95.2
91.5
96.1
94.1
X
93.5
94.9
95.5
92.2
96.2
94.5
94.8
95.2
96.3
93.5
96.4
95.2
X
94.0
94.8
95.7
92.3
96.3
94.6
X
95.1
95.4
96.2
93.0
96.2
95.2
V
96.2
96.8
96.7
94.6
96.4
96.1
96.8
97.3
97.7
95.5
97.9
97.1
Table 15: Ablation results of interpolation ratio sampling on the 3D-OVS dataset.
Method
bed
bench
room
sofa
lawn
avg.
Fixed to 0.5
95.8
95.9
95.6
93.7
96.5
95.5
Fixed to 0.75
94.1
95.2
95.0
92.5
95.4
94.4
Fixed to 1.0
93.4
94.0
94.4
90.8
95.2
93.6
Even Distribution
94.6
95.1
94.8
93.2
96.1
94.8
Gaussian Distribution
94.0
94.5
94.2
92.9
96.0
94.3
Beta Distribution (Ours)
96.8
97.3
97.7
95.5
97.9
97.1
Table 16: Ablation results of the parameters of Beta distribution on LERF data, in terms of mloU.
Method
ramen
teat ime
figurines kitchen
avg.
Beta(0.1, 0.1)
59.7
72.2
57.3
54.4
60.9
Beta(0.2, 0.2)
61.4
73.5
58.1
54.8
62.0
Beta(0.3, 0.3)
60.7
72.8
57.1
54.2
61.2
Beta(0.4, 0.4)
59.6
72.0
56.8
54.1
60.6
A.3.4 ABLATION OF CAMERA VIEW BLENDING
We first provide further justifications for view-blending augmentation technique. Instead of our mix-up strategy, one could apply the selected off-the-shed language-feature extractor (e.g., CLIP) generate new useudo-ground-trit data. This could be done either:
online, i.e., using the model's predicted color image as input to the language-feature ex- tractor in order generate the target language map evaluate the one rendered by the model.
offline, i.e., using pre-trained color-only 3DGS model of the scene generate novel views, pass these views to the feature extractor, and use all the resulting data to train the language-embedded representation.
However, the computational cost of running the selected Hunguage-feature extractor (CLIP) is too heavy for online usage, and the offline strategy also comes with several downsides.
First, one of our contributions is the joint CUDA-based rasterization function, which enables the fast rendering of both modalities (color and language). While most prior works Qin et al. (204)) train their model in two phases (a first phase to train color-only 3DGS model, then second phase to extend it with language information), we leverage our custom rasterizer to efficiently optimize our vision-language model in single training phase (see Table 8 highlighting training and rendering
16

---

Published as a conference paper at ICLR 2025
Table 17: Ablation results on different levels of disentanglement between the per-modality Gaussian parameters, evaluated on the downstream open-vocabulary semantic-segmenlation task on LERF.
Parameters shared across modalities
Segmentation accuracy (mloU)
position covariance opacity/indic.
features
ramer:
Leat ime
Ligurines
kitchen
avg.
X
55.9
71.0
54.2
51.3
58.1 1
(ours)
X
61,4
73,5
58.1
54,8
62.0
X
47.8
65.6
42.3
40.9
49.2
41.4
60.7
39.2
38.8
45.0
efficiency). The downside of this faster training is that, unlike prior solutions, we do not have access to a pre-trained 3DGS model of the target scene. Adding a pre-training phase for the color-only model would be significant computational overhead. Second, the aforementioned strategy (i.e., rendering additional views using pre-trained 3DGS) only adds fixed/limited number of images to the training set (c.f. offline rendering before the training starts), whereas our augmentation scheme provides new randomized language maps every training iteration (c.f- online augmentation).
We quantitatively compared the suggested offline augmentation strategy to ours in a new experi- ment shared in Tab. [3] which confirms that our augmentation technique does indeed result in more robust representation w.r.t. downstream semantic tasks. Note that future work, would be interesting to investigate how both strategies could be optimally interleaved (e.g.. by pretraining our model with mix-up augmentation; then, after some iterations, augmenting the training data with SNGS/CLIP-rendere pairs).
Additionally, we present an ablation study of various camera view blending strategies in Tab. [4 using the 3D-OVS dataset, which includes three variants: Rotation, Translation, and SSIM. Our proposed method is represented in the last row of the table. The results indicate that both Slerp- based rotation blending and Lerp-based translation blending enhance performance. Furthermore, the regularization provided by SSIM is crucial for managing the extent of blending.
In Tab. 15 further present ablation study on the interpolation ratio used within the camera view blending scheme, performed on 3D-OVS data. Alongside fixed values of 0.5, 0.75, and 1.0, we also include experiments utilizing Even Distribution (U(0, 1)), Gaussian Distribution (N(0, 1)), and our Beta Distribution (Beta(0.2, 0.2)). The results indicate that a balanced ratio is essential for achieving better representations, and that minor disturbances can also yield beneficial effects.
We also change the parameter of the Beta distribution in the blending loss Ls on the LERF dataset, and the results are exhibited in Tab. [6,, showing our selection is reasonable. It important to keep a relatively balanced ratio while introducing some disturbances.
A.3.5 ABLATION ON DISENTANGLEMENT LEVEL BETWEEN THE TWO MODALITIES
The main reason why prior 3DGS- DDGS-based or NeRF- based language-embedded scene representations subordinate the semantic modality to the visual one that the 2D semantic maps are too sparse to regress the underlying 3D scene geometry from them. The visual/shading information provided by the corresponding color images is required for the 3D representation to properly converge. More- over, for some downstream tasks such as 3D editing, it is necessary maintain the same underlying geometry across modalities. For these reasons, we decided to keep all spatial properties of the 3D Gaussians (i.e,, mean position and covariance) common to the two modalities, and disentangle only the modality-specin parameters (color features vs. language features, opacity vs. semantic indicator).
We substantiate the above with an additional experiment, shared in Tab. 17 We evaluate multiple variations of our solution over the open-vocahine vocabulary semantic segmentation task, with different levels of disentanglement between the represented modalities: (1) a model similar to prior work, with only the feature vectors specific to each modality; (2) our proposed solution, introducing per-modality opacity/indicato; (3) a model further introducing per-modality covariance matrices; (4) model with completely separate Gaussians for each modality (similar to training a semantic-only 3DGS model for the segmentation task). As expected, the semantic-only model (4) fails to effectively learn the underlying 3D information and converges poorly, impacting the downstream results on unseen views. Our solution performs the best out of the four.
17

---

Published as conference paper at ICLR 2025
RGB
LEGaussians
LangSplat
GOI
Ours
Ground Truth
'glass'
'naphin'
'heep''
loookie bag'
'platee
Figure 12: Qualitative semantic segmentation comparisons on the t eeat ime scene from LERF.
RGB
..Guussssans
LangSplat
GOI
Ours
Ground Truth
'Pikachu bag'
'patull''
stotue'
Figure 13: Qualitative semantic segmentation comparisons on the f figurines scene from LERF.
RGB
L.Gaaussians
LangSplat
Gor
Ours
Ground Truth
Basebblll'
'baskett
'chicken
'nubbit
'wall'
dinossaur'
Figure 14: Qualitative semantic segmentation comparisons on the room scene (3D-OVS dataset).
RGB
LEGaussians
LangSplat
GOI
Ours
Ground Truth
'headphonee
'apple'
Yhar"
'hand
'lwww'
'stapler'
Figure 15: Qualitative semantic segmentation comparisons on the lawn scene (3D-OVS dataset).
A.4 ADDITIONAL QUALITATIVE RESULTS
A.4.1 RESULTS ON LERF DATA
Fig. 12 and Fig. present additional semantic segmentation results on the LERF dataset, while
Fig. 16 displays further localization results for the same dataset. Our proposed method clearly outperforms other approaches, particularly reflective and translucent areas, where it more closely aligns with the ground truth.
A.4.2 RESULTS ON 3D-OVS DATA
We present additional segmentation comparisons on 3D-OVS dataset in Fig. 14 and Fig. 15 Our proposed method demonstrates superior performance compared to other approaches, showing greater alignment with the ground truth.
A.4.3
IMPACT SEMANTIC-INDICEE
CONTRIBUTION ON RELEVANCY MAPS
In Fig. 17 we share the relevancy maps (represented heatmaps) corresponding various open- vocabulary prompts, comparing the results for color/semantic 3DGS models incorporating our novel smoothed semantic (last row) versus baseline models re-using the color opacity during rasterization of semantic data (mid row). The results clearly demonstrate the effectiveness of the semantic in- dicator in uncovering semantic information, particularly in translucent and reflective objects. This enhanced capability allows for more nuanced understanding the scene, highlighting how the proposed method improves the semantic representations.
18

---

Published as a conference paper at ICLR 2025
LEGaussians
LangSplat
GOI
Ours
'napkin'
'yellow pouf
'door handle'
Figure 16: Additional qualitative examples of object localization in the LERF dataset.
Prompt
"glass" "
"cup"
"knife"
"toaster"
"handle"
Annotation
Shared
Color
Opacity
Smoothed
Eddim
Semantic
Indicator
Figure 17: Qualitative impact of the proposed additional semantic indicator on 2D relevancy maps w.r.t. various open- open-vocayulary prompts.
A.4.4 IMPACT OF CAMERA- VIEW BLENDING ON COLOR/LANGUAGE DISENTANGLEMENT
we demonstrate how camera-view blending helps alleviate over-fitting. Fig. 18 shows examples of multiple objects sharing the same colors. For example, in the first scene, the scissors and spatula share the same color, leading part of the scissors to be misclassified as spatula when camera-view blending is not applied. Similarly, in the second scene, the pot and table share the same wooden texture, causing part of the pot to be misidentified as table by the model without camera-view blending.
In Figs. 18 and 19
Fig. 19 shows examples of objects composed of multiple colors/textrres. In the first example, the camera and the spatula are composed of parts with different colors, and some of these parts end up misclassified by the model without camera-view blending. The same can be observed in the second example: the bike is composed of parts with varied appearances, and the model optimized without augmentation misses many of them; whereas the model optimized with correctly identifies the whole object. The proposed technique enhances semantic consistency across diverse views through regularization.
19

---

Published as a conference paper at ICLR 2025
RGB image
w/o View Blending
w/ View Blending
Ground Truth
spatula
refrigerutor
pot
dry fower
table
fower
Figure 18: Examples of multiple objects sharing similar colors/textures (e.g., scissors and spatula in example #1, pot and table example #2).
RGB image
w/o View Blending
w/ View Blending
Ground Truth
spatula
cut statue
camera
waldo
bike
bonsai tree
table cloth
Examples of objects composed of multiple colors/textures (e.g., multi-part camera and spatula in example #1, bike in example #2).
Figure 19:
A.5 ADDITIONAL QUANTITATIVE RESULTS
A.5.1 EXTENDED RESULTS ON 3D-OVS SCENES
In the main manuscript, we considered the five main scenes of the 3D-OVS dataset, following the protocol adopted in LangSplat Qin al. (2024). To match other experimental protocols, such as FMGS' one which include the additional ta le scene, we extend our evaluation to additional 3D- OVS scenes, for total of seven, ie.,, adding results for table and snacks scenes. The results shared in Tab. 18 confirm the effectiveness of our solution in terms of open-vocabulary semantic segmentation.
Table 18: Evaluation 3D-OVS dataset extended additional scenes.
Method
Venue
bed
bench
zoom
sofa
lawn
table
snacks
avg.
Feature-3DGS
CVPR 24
83.5
90.7
84.7
86.9
93.4
92.4
88.6
88.6
LEGaussians
CVPR'24
84.9
91.1
86.0
87.8
92.5
93.6
87.5
89.1
LangSplat
CVPR 24
92.5
94.2
94.1
90.0
96.1
95.9
92.3
93.6
GS- Grouping
BCCV*24
83.0
85,9
87.3
90.6
88.7
88.8
GOI
ACMMM'24 24
89.4
92.8
91.3
85.6
94.1
95.6
89.3
91.1
FMGS
1JCV'24
80.6
84.5
87.9
90.8
92.6
97.2
83.0
88.2
Ours
96.8
97.3
97.7
95.5
97.9
98.5
94.7
97.0
20

---

Published as a conference paper at ICLR 2025
A.5.2 EXTENDED EFFICIENCY ANALYSIS AND IMAGE QUALITY EVALUATION
We complement the Tab. from the main paper, adding results on other LERF scenes. As shown in Tabs. 22 our proposed method consistently outperforms others in terms of the training time, FPS, number Gaussians, and storage size.
Furthermore, we extend these experiments to measure the quality of the color images synthesized by the trained Gaussian splatting models. The goal is verify that, while this work focuses on semantic accuracy, our solution does not sacrifice visual precision too much. PSNR and SSIM results in Tab, 23 actually show that our model remains competitive on the visual modality, showing the effectiveness of our modality balance. Moreover, comparing to the results from color-only 3DGS (same as SplaSp as this method fixes all 3DGS parameters after its pre- training),, we observe that semantic learning does not have an obvious impact color-modality metrics (i.e., image quality).
Table 19: Efficiency and image-quality analysis LERF's ramen.
Method
Training Time +
FPS
Gaussians # +
Storage Size +
PSNR t
LangSplat
96min
40
86k
180MB
28.5
GS- eS-Grouping
130min
76
107k
230MB
29.2
GOI
73min
42
92k
198MB
29.0
Ours
65min
79
80k
175MB
29.2
Table 20: Efficiency and image-quality analysis on LERF's teatime.
Method
Training Time &
FPS
Gaussians #
Storage Size 4
PSNR t
LangSplat
102min
42
152k
433MB
32.6
GS- GS-Grouping
135min
75
187k
506MB
32.1
GOI
79min
44
170k
467MB
32.3
Ours
67min
78
144k
425MB
32.5
Table 21: Efficiency and image-quality analysis LERF's figur seurinns..
Method
Training Time
FPS
Gaussians #
Storage Size &
PSNR 1
LangSplat
121min
38
94k
190MB
27.9
GS-Grouping
130min
65
116k
245MB
26.5
GOI
104min
39
92k
222MB
27.7
Ours
89min
67
86k
187MB
27.9
A.5.3 PER-CATEGORY EVALUATION OF SEMANTIC INDICATOR CONTRIBUTION
In the following, expand our justification for introducing the semantic indicator provided in Sec. [.2] especially Fig. 3 There, we highlight how re-using the opacity o (optimized w.r.t. the color modality) rasterize language maps-i.e.. entangling visual properties and 2D language projections- harm the semantic accuracy for objects with complex visual properties, such as translucent or highly-reflective objects. Here, we provide further empirical evidence for disentan- gling the rasterization properties of the two modalities, especially for translucent/reflectiee object categories. In Tab, 24 through Tab. 28 we present the mloU results for each ground-trun category, highlighting objects considered translucent or reflective (e.g., glass, metal bottle, etc.). We observe how the introduction of the semantic indicator specially benefits these categories.
In other words: as shown in Fig. translucent and reflective objects are mostly represented by 3D Gaussians with low opacity values (in order to model their complex light transport). Using these low opacity values to rasterize the language features cause unwanted artifacts. This is alleviated by our semantic indicator. Our model correctly learns higher indicator values for Gaussians representing the above-man mentioned object categories (e.g., see large I results for the translucent glass reflective bottle/cup in Fig. B). The improvement on downstream semantic tasks, e.g., open- vocabuloyaylaryy se- mantic segmentation, is thus especially large for these categories. E.g., the segmentation of g lass
21

---

Published as a conference paper at ICLR 2025
Table 22: Efficiency and image-quality analysis LERF's kitchen.
Method
Training Time +
FPS
Gaussians # &
Storage Size +
PSNR t
LangSplat
105min
43
142k
406MB
32.4
GS GS-Grouping
141min
77
177k
455MB
32.3
GOI
86min
45
162k
429MB
31.9
Ours
80min
78
135k
398MB
32.3
Table 23: Evaluation of color-image rendering quality on the LERF dataset.
Method
ramen
teat ime
figurines PSNR
kitchen
avg.
PSNR
SSIM
PSNR
SSIM
SSIM
PSNR
SSIM
PSNR
SSIM
LangSplat (3DGS)
28.5
0.87
32,6
0.83
27.9
0.76
32.4
0.85
30.4
0,83
GS Grouping
29.2
0.85
32.1
0.82
26.5
0.74
32.3
0.84
30.0
0.81
GOI
29.0
0.83
32.3
0.83
27.7
0.77
31.9
0.82
30.2
0.81
Ours
0.86
32.5
27.9
0.76
32.3
0.86
30.5
0.83
is improved by 14.5% and sake cup by 14.3%, whereas the segmentation of non-translucent/oon reflective is improved by 3.6% on average, c.f. Tab. 24
22

---

Published as a conference paper at ICLR 2025
object categories with twanssuueen/reffeecttt parts
translucen/rrelleciiee objects
spoen
30.3
45.4
+15.1
glass
64.7
77.6
+12.9
pot.
68.8
76.5
+7.7
plate
88.6
95.4 9..4
+6.8
object categories with trans/ncent/reffectiiii parts
ladle
17.0
32.4
+15.4
glass
64.1
78.6
+14.5
Table 25: Per-category mloU results for the open- coon-vocabulary semantic segmentation task on LERF's teat ime scene.
coffee .mog
Table 26: Per-category mloU results for the open- open-vocabullee sae--ooobblrrryyy semantic segmentation task on LERF's kit chen scene.
Table 24: Per-calegory mloU results for the open- vocabulary semantic segmentation task on LERF'S ramen scene.
79.1
88.6
+9.5
vessel
28.7
44.2
+15.5
plate
53.6
62.1
+8.5
stainl stainless-poo.
maakemap
36.7
51.0
+14.3
pout
53.2
55.6
+2.4
42.3
51.5
+9.2
brand
52.7
58.1
+5.4
40.3
53.7
+13.4
Loaster
booves
14.6
19.8
+5.2
24.8
45.6
+20.8
coom
10.4
13.9
+3.5
mon-translucent and non- non--eflecctvv object categories
coffee
46.9
51.2
+4.3
knife
46.0
52.5
+6.5
non con--translceett and non non-refiecttl reflective object categories
24.7
29.8
+5.1
cooxies
plate
69.4
78.8
+9.4
76.0
76.5
+0.5
cabinet
non-translucent and non-reflective object categories
29.6
33.7
+4.1
kamaboka
be ar..oee
37.7
44.5
+6.8
24.9
40.3
+15,4
ketchup
20.5
23.2
+2.7
noodles
51.4
56.6
+5.2
napkin
64.3
68.1
+3.8
ottclenghi refzigerat.o
napkin
45.9
51.2
+5.3
sheep apple
77.1
77.6
+0.5
50.7
55.3
+4.6
nori
78.2
80.4
+2.2
85.5 8
85.9
+0.1
60.7
62.1
+1.4
egg
96.4
96.5
+0.1
cookle_bag
71.5
72.2
+0.7
desk
39.3
40.0
+0.7
chopet:
90.0
90.4
+0.4
beat
62.7
66.9
+4.2
Semantic Indicator?
Semantic Indicator?
improvement
Semantic Indicator?
improvement
improvement
23

---

Published as a conference paper at ICLR 2025
punpkin
73.3
75.1
+1.8
Table 27: Per- cor-categoyy mloU results for the open- www.bulccaaaaomeeeeeo semantic segmentation task on LERF's f figurines scene (part 1/2).
Table 28: Per- category mloU results for the open- vocabulary semantic segmentation task on LERF'SS figurines seene (part 2/2).
c.cubbe magicc.ube
74.8
76.2
+1.4
magi
64.8
76.6
+11.8
hat
21.2
24.7
+3.5
Jake
42.8
43.3
+0.5
object categories with transiucent reflective perts
37.5
51.0
+13.5
non-translacent and non-re reflcctivee object categories red_appee
64.5
67.8
+3.3
45.3
60.6
+15.3
pikachu
s.hendle corcelein.hem red.chair
37.6
44.0
+6.4
48.6
+16.8
65.4
green_apple apeneppple
84.2
84.4 4
+0.22
41.9
50.4
+8.5
10e_cream
42.2
45.6
+3,4
28.2
43.7
+15.5
buoy-dock -dock
54.4
58.7
+4.3
49.6
61.3
+11.7
ha tatukcck
56.8
59.2
+2.4
Semantic Indicator?
improvement
walde
78.3
80.5
+2.2
elephant
90.0
90.0
+0.0
Semantic Indicator?
improvement
24


Reviews:
[{"review_id": "bDS9VvVN7s", "reviewer": "Reviewer_Awye", "summary": "This paper focuses on improving the semantic rendering quality within well-reconstructed 3D scenes. With introducing self-attention, semantic indicator and regularization, the  proposed method achieves better semantic segmentation results, especially for \ntransparent objects.", "questions": "(1) What is the different between self-attention and cross-attention mechanism?\n\n(2) Considering the experiment setting is the open-vocabulary, for example, the CLIP feature dimension is 768, how to obtain 768 rendering with d_c+d_f=6? How to perform the dim. Reduction as shown in Figure.2", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n\n2. The designs are well-motivated.\n\n3. The performance shows a significant improvement.", "weaknesses": "(1) Line 215, the u^i is the fused features or position?\n\n(2) In camera interpolation, why use mix-up to generate a feature map instead of using an off-the-shelf OV model to output one? Just like previous methods.\n\n(3) Missing a global ablation study to present the importance of 'self-attention', 'semantic indicator'. 'camera interpolation' over your baseline. Now, it is unclear how the baseline stands in comparison with previous methods and the contributions from the three proposed designs.\n\n(4)  Low efficiency, the Table. 8 shows it takes over one hour to learn the semantics of single 3D scenes, which is significantly longer than typical 3D Gaussian reconstruction (which may take less than 10 mins on A100？), especially considering the iteration is only 15k. What makes the rasterization so slow? In addition, since more attributions are added in Gaussian, it is fair to provide the storage cost of Gaussian models when compared with other works in Table.8 instead of only Gaussian numbers.\n\n(5) The incomplete experiments. The experiments on 3D-OVS are conducted on  5/7 scenes instead of full scenes.\n\n(6) No deep analysis of performance. In my view, the IoU improvements are mainly from the translucent or reflective objects, as shown in Figure.3, no detailed statistics to prove that.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KTPVuAuoj7", "reviewer": "Reviewer_EPr2", "summary": "This paper claims that the performances of current multi-modal scene understanding approaches are limited by the the imbalance between visual and language modalities, which have widely different properties.\nTo alleviate these limitations, this paper proposes several strategies, including a cross-modal rasterizer that places greater emphasis on language features and a camera-view blending technique.\nFinally, the proposed methods achieves state-ofthe-art performance in open-vocabulary semantic segmentation tasks.", "questions": "1. What is the specific relationship between the improvement brought by your method and the details of the scene? In what kind of scene is your improvement more obvious?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of this paper  sounds reasonable. The over-fitting on the color modality may have a negative impact on semantic learning, which is consistent with intuition.\n2. The proposed techniques, including the smoothed semantic indicator  and the mix-up augmentation, are simple but effective.\n3. The performance of the method  outperform existing methods by a significant margin.", "weaknesses": "1. Although the motivation  sounds reasonable and the Figure 3 explain the motivation to some extent, I expect more experiments to further study the differences between color and semantics modalities. Beside showing quantitative, qualitative results and ablation of the proposed strategies, the authors should further discuss the deeper mechanisms. For example, the authors can study the relationship between color over-fitting phenomenon and specific scenes.\n2. The figure 2 is hard to understand. The method is simple but the process in the figure looks complicated. Some fonts are too small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HU5qlUlve6", "reviewer": "Reviewer_Jaap", "summary": "This paper learns language embedded 3dgs field and addresses the often-neglected issue of appearance-semantic misalignment in rasterization. It introduces a camera-view blending technique to enhance semantic consistency between existing and synthesized views. The experimental findings demonstrate that the proposed approach outperforms existing methodologies.", "questions": "I am curious about the impact of semantic learning on rendering quality.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a profound insight into the challenges posed by semi-opaque media and intricate light transport effects, highlighting the limitations in translating color opacity to the semantic domain. This observation is interesting. \n2. The implementation of a single learnable parameter to replace the conventional shared color opacity parameter is both straightforward and efficient.\n3. The paper is commendably structured, presenting a well-motivated narrative and a clear methodology.", "weaknesses": "1. The paper introduces the camera-view blending technique as a method to enhance cross-view semantic consistency. However, it falls short in fully addressing the challenge of different objects sharing similar colors, potentially leading to indistinguishable semantic representations, which the authors claim to have tackled. Further insights on this critical point are needed.\n2. In Section 4.4, the experimental results suggest an increase in rendering speed with the proposed pipeline. However, given the introduction of additional steps and parameters, one would expect a potential decrease in inference speed. The discrepancy in the reported results requires clarification.\n3. The title of the paper, \"3D Vision-Language Gaussian Splatting,\" might be misleading as it predominantly focuses on learning open language semantic fields rather than visual appearances. A revision to better reflect the core contribution of the research is recommended.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "agLJZazLDf", "reviewer": "Reviewer_yCDX", "summary": "This paper proposes an interesting idea on how to fuse multi-modality info, i.e. visual & semantic, for better semantic rasterization in 3DGS. This solution mainly composes of a cross-modal rasterizer, a specially designed semantic indicator and a camera-view blending method. The solution is interesting especially because it offers a better solution for semantic rasterization for reflective or translucent objects, which is a serious problem with the original 3DGS. For me, the designed semantic indicator in the Gaussian representation is novel and useful for the claimed main problem. The main contributions also include the empirical quality result of the semantic indicator on these reflective or translucent objects, and quantity comparison of open-vocabulary semantic segmentation tasks. These results conclude the usefulness of the proposed solution. \n\nThe paper is generally well written and structured, easy to follow and read. I don’t come across many issues for further clarification. But here a few things I may need the authors to respond. Given those issues properly clarified, I would be willing to increase the score.\n\n1、\tEquation (3), any language/semantic should not change due to view angle, so F^W doesn't make sense to me, even the authors mention that superscript W will be omitted for ease of reading. It shouldn’t appear in the original equation at all. Also in the same equation, there lacks a clear definition of what L_sem is, but this should be a minor issue.\n2、\tFigure 3 right, I’m not 100% sure where the symmetry of the density distribution is from, as there lacks sufficient correlation between l and o. \n3、\tTable 8, an average training and inference time should be reported on all scenes of a whole dataset, instead of on just one. \n4、\tThe Related Work session is relatively weak and could be fortified with more related reference included. For instance, “HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting”, which proposes another way of using semantic info in 3DGS (not traditional 2d way mentioned by the authors）and is worthy of being cited. \n5、\tA minor grammar error of Line 356-357.", "questions": "1、\tEquation (3), any language/semantic should not change due to view angle, so F^W doesn't make sense to me, even the authors mention that superscript W will be omitted for ease of reading. It shouldn’t appear in the original equation at all. Also in the same equation, there lacks a clear definition of what L_sem is, but this should be a minor issue.\n2、\tFigure 3 right, I’m not 100% sure where the symmetry of the density distribution is from, as there lacks sufficient correlation between l and o. \n3、\tTable 8, an average training and inference time should be reported on all scenes of a whole dataset, instead of on just one. \n4、\tThe Related Work session is relatively weak and could be fortified with more related reference included. For instance, “HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting”, which proposes another way of using semantic info in 3DGS (not traditional 2d way mentioned by the authors）and is worthy of being cited. \n5、\tA minor grammar error of Line 356-357.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed idea is interesting, especially because it offers a better solution for semantic rasterization for reflective or translucent objects, which is a serious problem with the original 3DGS. For me, the designed semantic indicator in the Gaussian representation is novel and useful for the claimed main problem. The main contributions also include the empirical quality result of the semantic indicator on these reflective or translucent objects, and quantity comparison of open-vocabulary semantic segmentation tasks. These results conclude the usefulness of the proposed solution.", "weaknesses": "A few issues are listed in the next section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qvZnJHURA3", "reviewer": "Reviewer_56RB", "summary": "This paper proposes a novel approach to open-vocabulary 3DGS. It proposes to use another learnable opacity parameter for the blending of semantics, and train a cross-modal fusion module. The authors also propose a semantic-aware camera view blending method, in which they randomly interpolate between training views and supervise the semantic rendering based on the weighted sum of the semantics. The experiments show that the proposed method significantly improves the open-vocabulary semantic segmentation results.", "questions": "1. Is the fusion module per-scene trained? Would it be more effective to train it across different scenes to learn the prior?\n2. According to Table 6 and Table 7, does the view interpolation method alone improve the performance by 6.2 mAP? It's a little counter-intuitive that simply weighted-summing the two views to supervise the interpolated views can lead to such improvements, since the weighted-sum semantics (as shown in Figure 2) do not consider the relative camera poses between the two views. What if you wrap the two views to the interpolated view using the known camera poses and learned depths?\n3. The paper discussed using separate opacity for semantics, then what would the results be if learning separate covariance matrix in each gaussian for color and semantics? Besides, if we simply train two separate sets of Gaussians for color and semantic renderings respectively, would it improve the semantic segmentation results?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is novel and promising. Using another learnable parameter as opacity for semantic blending is an overlooked method that benefits open-vocabulary 3DGS.\n2. The results are promising.", "weaknesses": "Please see the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)