3D Vision-Language Gaussian Splatting

## Abstract

Abstract Recent advancements in 3D reconstruction methods and vision-language models have propelled the development of multi-modal 3D scene understanding, which has vital applications in robotics, autonomous driving, and virtual/augmented reality. However, current multi-modal scene understanding approaches have naively embedded semantic representations into 3D reconstruction methods without striking a balance between visual and language modalities, which leads to unsatisfying semantic rasterization of translucent or reflective objects, as well as over-fitting on color modality. To alleviate these limitations, we propose a solution that adequately handles the distinct visual and semantic modalities, i.e., a 3D vision-language Gaussian splatting model for scene understanding, to put emphasis on the representation learning of language modality. We propose a novel cross-modal rasterizer, using modality fusion along with a smoothed semantic indicator for enhancing semantic rasterization. We also employ a camera-view blending technique to improve semantic consistency between existing and synthesized views, thereby effectively mitigating over-fitting. Extensive experiments demonstrate that our method achieves state-of-the-art performance in open-vocabulary semantic segmentation, surpassing existing methods by a significant margin.

## 1 Introduction

Figure: Figure 1: Comparison of prior semantic 3DGS work and our novel method. We apply cross-modal rasterization and camera-view-based regularization for better exploration of semantic features.
Refer to caption: x1.png

The advancement of 3D reconstruction methods, such as neural radiance fields (NeRF) (Mildenhall et al., ) and 3D Gaussian splatting (3DGS) (Kerbl et al., ), has enabled the effective acquisition of *3D color representations*, facilitating high-fidelity and real-time rendering from novel viewpoints. Moreover, vision-language models like CLIP (Radford et al., ) and LSeg (Li et al., ) have been bridging the gap between color images and semantic features in 2D space.
Given an input image, these models can generate a dense 2D language map, *i.e*., assigning semantically-rich language embeddings to each pixel (*e.g*., a pixel depicting a person’s face can be assigned a language embedding describing said face). Building on these developments, multi-modal 3D scene understanding—which aims to learn effective *3D semantic representations* from multi-view images and their corresponding camera poses—has made significant progress in recent years. This area of research has a wide range of applications across various practical domains, including robotic manipulation (Shorinwa et al., ), autonomous driving (Gu et al., ), and VR/AR (Guerroudji et al., ).

Recent methods (Engelmann et al., ; Qin et al., ; Shi et al., ; Zhou et al., ) in multi-modal 3D scene understanding have adopted the paradigm of *embedding* semantic representations into 3D representation for joint reconstruction training, and utilizing semantic knowledge distilled from off-the-shelf vision-language models to guide the training process. For example, OpenNeRF (Engelmann et al., ) integrates MipNeRF (Barron et al., ) method with OpenSeg model (Ghiasi et al., ), and LangSplat (Qin et al., ) employs both 3DGS (Kerbl et al., ) method and CLIP (Radford et al., ) model.
These solutions rely on 2D supervision to learn a multi-modal (color and semantic) 3D scene representation, *i.e*., projecting the learned 3D representation back to 2D views for comparison with the input observations (input color images and corresponding 2D language maps inferred by aforementioned vision-language models).

However, we argue that these methods have *naively embedded* semantic representations into 3D reconstruction methods like 3DGS—originally designed for color information—failing to strike a balance between visual and language modalities. For example, they are directly applying the color rasterization function—meant to project 3D RGB information to 2D—to the new language modality, ignoring that this function relies on a color opacity attribute that does not translate to semantic information.
Prior art also tends to ignore the unequal complexity and distribution of color and semantic modalities, and the risk of over-fitting the color information to the detriment of the 3D semantic representations. While the same objects may exhibit different colors from various views, their semantic information remains constant. Conversely, different objects can share similar colors, but it is less desirable for their semantic representations to appear identical. Thus, training color representations may negatively impact the training of 3D semantic representations.

Given these limitations, our intuition is to strike a *balance* between visual and language modalities, rather than simply embedding language features into RGB-based 3D reconstruction. Therefore, we propose a novel framework named 3D vision-language Gaussian splatting, as shown in Fig. 1.
On one hand, we propose a novel cross-modal rasterizer that prioritizes the rendering of language features. We integrate semantic features with meaningful information from the color domain through modality fusion, prior to rasterization, to facilitate the robust learning of semantic information. Besides, we introduce a language-specific parameter that enables the meaningful blending of language features from different Gaussians. This methodology yields a more accurate representation of semantic information, especially for translucent or reflective objects, such as glass and stainless steel, where the usage of color opacity may lead to misinterpretation.
On the other hand, we also propose a novel camera-view blending augmentation scheme specific to the semantic modality, *i.e*., blending information across views to synthesize new training samples. This process regularizes the language modality through enhancing semantic consistency between the existing and novel views, leading to more robust 3D semantic representations.

All in all, our 3D vision-language Gaussian splatting can be summarized into the following contributions:

- •
We propose a cross-modal rasterizer that places greater emphasis on language features. Modality fusion occurs prior to rasterization, accompanied by a learnable and independent *semantic indicator* parameter for the $\alpha$-blending of language features, enabling a more accurate representation of translucent or reflective objects.
- •
We define a camera-view blending technique for the regularization of semantic representations during training, augmenting the input 2D language maps through a cross-modal view, *i.e*., leveraging the semantic-consistency prior to alleviate over-fitting on the color modality.
- •
Extensive experiments on benchmark datasets demonstrate that our approach achieves state-of-the-art performance in open-vocabulary semantic segmentation tasks, outperforming existing methods by a significant margin.

## 2 Related Work

3D Neural Representations. Recent advancements in 3D scene representation have made significant progress, particularly with neural radiance fields (NeRF) (Mildenhall et al., ), which excel in novel view synthesis. However, NeRF’s reliance on a neural network for implicit scene representation can result in prolonged training and rendering times. Methods like Instant-NGP (Müller et al., ) speed up these processes through hash encoding, while approaches such as 3D Gaussian splatting (3DGS) (Kerbl et al., ) use explicit neural representations for better alignment with GPU computations via differential tile rasterization. Additionally, MipNeRF (Barron et al., ) and Mip-Splatting (Yu et al., ) address aliasing issues, whereas DS-NeRF (Deng et al., ) and DG-Splatting (Chung et al., ) focus on sparse view reconstructions. *In this paper, we utilize 3D Gaussian Splatting for 3D neural representations.*

Visual Foundation Models. Visual foundation models include both pure vision models, crucial for semantic segmentation and feature extraction, and vision-language models that connect images with natural language. Among pure vision models, the Segment Anything Model (SAM) (Kirillov et al., ) excels in zero-shot transfer, generating multi-scale segmentation masks. DINO (Caron et al., ) and DINOv2 (Oquab et al., ), trained in a self-supervised manner, deliver fine-grained features for various downstream tasks. In vision-language models, CLIP (Radford et al., ) utilizes contrastive learning to align visual and textual features, while LSeg (Li et al., ) enhances this by incorporating spatial regularization for refined predictions. APE (Shen et al., ) functions as a universal perception model for diverse multimodal tasks. *In this paper, we employ SAM and CLIP to extract ground-truth features for baseline comparisons, ensuring a fair evaluation.*

Multi-modal 3D Scene Understanding. Existing methods for scene understanding learn multi-modal 3D representations from posed images, enabling rendering from novel viewpoints for tasks like open-vocabulary semantic segmentation. For NeRF-based approaches, LERF (Kerr et al., ) optimizes a language field alongside NeRF, using positions and physical scales to generate CLIP vectors. OpenNeRF (Engelmann et al., ) introduces a mechanism for obtaining novel camera poses, enhancing feature extraction. Among 3DGS-based methods, LangSplat (Qin et al., ) adopts a two-stage strategy for semantic feature acquisition, while GS-Grouping (Ye et al., ) extends Gaussian splatting for joint reconstruction and segmentation. Additionally, GOI (Qu et al., ) employs hyperplane division to select features for improved alignment.

## 3 Methodology

### 3.1 Problem Statement

According to the vanilla Gaussian splatting (3DGS) paradigm applied to RGB image rendering (Kerbl et al., ), a scene is represented by a set of 3D Gaussians $\mathcal{G}=\{g^{i}\}_{i=1}^{\mathcal{N}}$, where $\mathcal{N}$ denotes their number. Each Gaussian is defined as $g^{i}=\{\mu^{i},\Sigma^{i},o^{i},c^{i}\}$, *i.e*., by its mean position $\mu^{i}\in\mathbb{R}^{3}$, covariance matrix $\Sigma^{i}\in\mathbb{R}^{3\times 3}$, opacity $o^{i}\in\mathbb{R}$, and color properties $c^{i}\in\mathbb{R}^{d_{c}}$ (*e.g*., with $d_{c}=3$ for RGB parameterization).
Images are rasterized by splatting Gaussians through each pixel $v$ into the scene and $\alpha$-blending the Gaussian contributions to the color $C(v)$, as:

$$ $C(v)=\sum_{i=1}^{\mathcal{N}}c^{i}o^{i}P^{i}\displaystyle\prod_{j=1}^{i-1}(1-o ^{j}P^{j})\;\text{ with }\;P^{i}=e^{-\frac{1}{2}(v-\widehat{\mu}^{i})(\widehat {\Sigma}^{i})^{-1}(v-\widehat{\mu}^{i})},$ (1) $$

where $\widehat{\mu}^{i}$ and $\widehat{\Sigma}^{i}$ are the 3D-to-2D projections of $\mu^{i}$ and $\Sigma^{i}$.
This scene representation is learned from a training set $T^{r}=\{(I_{1}^{r},W_{1}^{r}),(I_{2}^{r},W_{2}^{r}),\ldots\}$ consisting of several pairs of RGB images $I$ of the target scene and the corresponding camera poses $W$.

Gaussian splatting models have been expanded to incorporate language-embedding information densily describing the scene (Qin et al., ; Zhou et al., ; Qu et al., ), by adding a language feature vector $f^{i}\in\mathbb{R}^{d_{h}}$ (with $d_{h}$ feature size) to each 3D Gaussian and applying a similar rasterization process to this modality, *i.e*., the 2D semantic embedding $F$ at pixel $v$ can be expressed as:

$$ $F(v)=\sum_{i=1}^{\mathcal{N}}f^{i}o^{i}P^{i}\displaystyle\prod_{j=1}^{i-1}(1-o ^{j}P^{j}).$ (2) $$

Even though the two modalities have widely different properties (*e.g*., translucence only applies to the visual modality and not the semantic one), previous methods have been directly applying the color rasterization process to the language features without any adaption, *i.e*., only replacing $c^{i}$ (Eq. 1) by $f^{i}$ (Eq. 2). In this paper, we propose to adapt the usual rasterization scheme to better fit the language-feature modality.

To train these semantically-enriched 3DGS models, the standard procedure consists of first generating the set of 2D language-feature maps $H$ corresponding to the input images $I$. Off-the-shelf pure vision models like SAM (Kirillov et al., ) and vision-language models like CLIP (Radford et al., ) are typically used, along with an auto-encoder-based Zhai et al. () dimension reduction as in Qin et al. (). Once the data is prepared, the 3D Gaussians can be iteratively optimized by minimizing the distance between its rasterized 2D semantic embeddings (*c.f*. Eq. 2) and the ground-truth 2D semantic embeddings:

$$ $\mathcal{L}=\mathbb{E}_{(I,W)\in T^{r}}\mathbb{E}_{v\in I}\mathcal{L}_{\text{ sem}}(F^{W}(v),H^{W}(v)),$ (3) $$

where $\mathcal{L}$ is the overall optimization objective. Besides, $F^{W}$ and $H^{W}$ are the predicted and ground-truth 2D language maps rendered using camera pose $W$ corresponding to image $I$ (for ease of readability, we drop the superscript $W$ in subsequent equations), and
$\mathcal{L}_{\text{sem}}$
is a distance function for 2D semantic maps.

The resulting 3D Gaussian language representation can be leveraged for open-vocabulary semantic queries. For example, given a query language vector $\tau$, a pixel-wise 2D relevancy score map corresponding to view $W$ can be computed as (Radford et al., ):

$$ $p(\tau|v)=\exp(\frac{F(v)\cdot\varphi(\tau)}{\lVert F(v)\rVert\lVert\varphi( \tau)\rVert})/\sum_{s\in\mathcal{T}}\exp(\frac{F(v)\cdot\varphi(s)}{\lVert F(v )\rVert\lVert\varphi(s)\rVert}),$ (4) $$

where $\varphi$ is the text encoder from the vision-language model.
Such a relevancy map can be leveraged, *e.g*., for open-vocabulary 2D localization (*i.e*., by measuring the argmax response to the query) or semantic segmentation (*i.e*., by thresholding the resulting relevancy map).

### 3.2 Semantic-aware Rasterization

Figure: Figure 2: Overview of our proposed framework. A) We propose a novel multi-modal Gaussian splatting model; B) we enrich the input images and poses for the model to better fit the semantic information. Besides our introduction of a novel semantic indicator parameter $l$, our additional contributions are: C) a semantic-aware cross-modal rasterization module; and D) a camera view blending augmentation scheme for training regularization.
Refer to caption: x2.png

Existing multi-modal rasterization methods (Qin et al., ; Zhou et al., ; Zuo et al., ) have largely drawn from color-based 3DGS (Kerbl et al., ). These rasterizers approach the rendering of 3D semantic information in a manner akin to RGB rendering (see Sec. 3.1), resulting in an insufficient focus on semantic-specific design. To address this gap, we propose a novel cross-modal rasterizer that emphasizes semantic-specific design, as illustrated in Fig. 2A and Fig. 2C.

A first noticeable shortcoming is the insufficient integration and exchange between 3D semantic and color features. While these modalities have distinct properties, they offer correlated and complementary information about the scene, with knowledge from one informing the other. Current models fail to leverage this synergy, resulting in inadequate guidance for learning semantic representations from the richer color modality.

To address this shortcoming, we propose integrating 3D semantic and color features prior to rasterization, rather than treating them independently (Qin et al., ; Qu et al., ), which facilitates effective knowledge fusion and exchange between modalities.
This intuitive improvement to multi-modal 3DGS has surprisingly been ignored in prior work. We decide to correct this oversight by applying the well-established
self-attention mechanism for modality fusion before rasterization, thereby enhancing its effectiveness. We can derive the 3D modality-fused features $u^{i}\in\mathbb{R}^{d_{c}+d_{f}}$ for our multi-modal rasterizer as:

$$ $u^{i}=c^{i}\oplus f^{i}+\psi_{\mathrm{out}}(\text{softmax}(\frac{\psi_{Q}(c^{i }\oplus f^{i})\psi_{K}^{\top}(c^{i}\oplus f^{i})}{\sqrt{d_{h}}})\psi_{V}(c^{i} \oplus f^{i})),$ (5) $$

where $d_{h}$ represents the number of heads. Moreover, $\psi_{Q},\psi_{K},\psi_{V}:\mathbb{R}^{d_{c}+d_{f}}\mapsto\mathbb{R}^{d_{c}+d_
{f}}\times\mathbb{R}^{d_{h}}$ and $\psi_{\mathrm{out}}:\mathbb{R}^{d_{c}+d_{f}}\times\mathbb{R}^{d_{h}}\mapsto
\mathbb{R}^{d_{c}+d_{f}}$ are single-layer linear networks.

Figure: Figure 3: Empirical differences between color opacity and proposed smoothed semantic indicator. On the left, we visualize the difference $l^{i}-o^{i}$ in Gaussians modeling the ramen scene. While their color opacity may vary significantly, most Gaussians need their semantic features to be rasterized with minimal blending (*c.f*. red regions in the difference maps, *i.e*., where $l^{i}\gg o^{i}$), except for Gaussians representing intangible lighting effects (*c.f*. glares on the bottle, bowl, table, *etc*.). On the right, we further plot the density distribution of $l^{i}-o^{i}$ in Gaussians for the ramen scene, indicating the different distributions of these two control parameters.
Refer to caption: x3.png

Another salient shortcoming is the simplistic adoption of $\alpha$-blending-based RGB splatting (Eq. 1) to rasterize language embeddings, only substituting the 3D color features $c$ with 3D semantic representations $f$ (Eq. 2). In other words, previous works use the opacity attribute of Gaussians to render not only RGB images but also language maps, *c.f*. Eq. 2 (either optimizing the opacities values over the two modalities (Shi et al., ; Zhou et al., ), or applying frozen opacity values from the RGB-only pre-training during semantic rasterization (Qin et al., )). We argue that this approach fails to recognize that in phenomena involving semi-opaque media (*e.g*., glass, water) and complex light transport effects (*e.g*., direction-dependent scattering and glare), color opacity cannot be effectively translated to the semantic modality. In most cases, when Gaussians represent *tangible* elements of the scene (*e.g*., the surface or volume of an object), their semantic features should be splatted to 2D without any reduction in opacity. Conversely, *intangible* Gaussians (*e.g*., those simulating lighting effects like optical glares) should be excluded from semantic rasterization by enforcing their $\alpha$-blending weight close to zero.

To address this problem, we propose a novel $\alpha$-blending strategy specifically designed for exploring semantic information. We introduce a new learnable attribute for our multi-modal Gaussians: a smoothed semantic indicator $l^{i}\in[0,1]$, which is applied to the rasterization of language embeddings, effectively replacing the color opacity parameter for this modality:

$$ $U(v)=\underbrace{\sum_{i=1}^{\mathcal{N}}u^{i}_{\bm{1:d_{c}}}o^{i}P^{i} \displaystyle\prod_{j=1}^{i-1}(1-o^{j}P^{j})}_{\text{color modality (channels 1 to }d_{c}\text{)}}\oplus\underbrace{\sum_{i=1}^{\mathcal{N}}u^{i}_{\bm{d_{c} +1:d_{c}+d_{f}}}l^{i}P^{i}\displaystyle\prod_{j=1}^{i-1}(1-l^{j}P^{j})}_{\text {language modality (channels }d_{c}+1\text{ to }d_{c}+d_{f}\text{)}}.$ (6) $$

*Note that these two processes are conducted simultaneously in the rasterizer.*
By disentangling the semantic rasterization from the opacity-based control and letting the overall model learns an independent per-Gaussian semantic indicator, we allow the model to better fit the semantic information of the scene. This is highlighted in Fig. 3, which shows how much the distribution of semantic indicator values differs from the color opacity after optimization. This is especially notable for semi-transparent or reflective objects (*e.g*., glass and pot in depicted scene), exhibiting low opacity but high semantic-indicator values (*c.f*. $l^{i}-o^{i}$ close to 1).

After rasterization, the resulting pixel information $U(v)$ can be further decomposed as:

$$ $U(v)=C(v)\oplus F(v),$ (7) $$

where $C(v)$ and $F(v)$ are the RGB color and 2D language embeddings at pixel $v$ respectively.

Based on the above, we formulate the loss function to optimize the overall scene representation as follows, $\forall(I,W)\in T^{r}$:

$$ $\mathcal{L}_{\mathrm{raster}}(W)=\mathbb{E}_{v\in I}\left[\mathcal{L}_{\mathrm {MSE}}\left(C^{W}(v),I(v)\right)+\mathcal{L}_{\mathrm{MSE}}\left(F^{W}(v),H^{W }(v)\right)\right],$ (8) $$

where $\mathcal{L}_{\mathrm{MSE}}$ represents the mean squared error (MSE) criterion.

### 3.3 Semantic-aware Camera View Blending

Figure: Figure 4: Qualitative semantic segmentation comparisons on the ramen scene (LERF dataset).
Refer to caption: x4.png

Figure: Figure 5: Qualitative semantic segmentation comparisons on the kitchen scene (LERF dataset).
Refer to caption: x5.png

While the novel cross-modal rasterization process proposed in Sec. 3.2 facilitates the efficient learning of joint color and semantic representations, this multi-modal approach may be suboptimal for applications focused exclusively on semantic information—such as open-vocabulary semantic segmentation tasks—due to the adverse effect from richer color modality. On the one hand, though the same object may exhibit different colors from various viewpoints, its semantic information remains consistent. On the other hand, different objects might share similar colors, which can lead to undesirable identical semantic representations. Therefore, it is crucial to incorporate regularization methods that enhance semantic consistency across diverse views, thereby mitigating the overfitting on semantic representations.

To this end, we propose a regularization technique called camera view blending (shown in Fig. 2B), designed specifically to augment the semantic modality by leveraging multi-modal information. Inspired by mix-up augmentation Zhang et al. (), our approach involves blending two ground-truth 2D language maps to create new ones, while providing semi-consistent corresponding camera poses through advanced 6DOF pose interpolation. For two randomly selected samples $(I_{1},W_{1})$ and $(I_{2},W_{2})$ from the training set $T^{r}$, where $W_{1}\neq W_{2}$, we first utilize the camera poses $W_{1}$ and $W_{2}$ to synthesize a novel camera pose that incorporates both rotation and translation components.

Expressing the input rotations with quaternions $q_{1}$ and $q_{2}$,
we apply spherical linear interpolation (Slerp) (Shoemake, ). Define $\theta=\arccos\frac{q_{1}\cdot q_{2}}{\lVert q_{1}\rVert\lVert q_{2}\rVert}$ and let the interpolation ratio $k$ be randomly sampled from $\text{Beta}(0.2,0.2)$. The quaternion for the novel camera view $q_{b}$ is then given by:

$$ $q_{b}=\left\{\begin{aligned} &k\frac{q_{1}}{\lVert q_{1}\rVert}\cdot\frac{\cos \theta}{|\cos\theta|}+(1-k)\frac{q_{2}}{\lVert q_{2}\rVert}~{}~{}~{}~{}~{}~{}~ {}~{}~{}~{}~{}\text{ if }~{}~{}~{}|\cos\theta|>0.995\\ &\frac{q_{1}\sin(\theta-k\theta)}{\lVert q_{1}\rVert\sin\theta}\cdot\frac{\cos \theta}{|\cos\theta|}+\frac{q_{2}\sin(k\theta)}{\lVert q_{2}\rVert\sin\theta}~ {}~{}\text{ if }~{}~{}~{}|\cos\theta|\leq 0.995.\end{aligned}\right.$ (9) $$

When the two quaternions are nearly aligned in direction ($|\cos\theta|>0.995$), we use linear interpolation (Lerp) for improved efficiency. If they are not aligned ($|\cos\theta|\leq 0.995$), Slerp is applied.

For the translation component, we use linear interpolation to synthesize the focal center of the novel view. Given the camera centers $t_{1}$ and $t_{2}$ in the world coordinate system, corresponding to $W_{1}$ and $W_{2}$ respectively, the new center is determined with the same interpolation ratio $k$:

$$ $t_{b}=kt_{1}+(1-k)t_{2}.$ (10) $$

With the blended quaternion $q_{b}$ and camera center $t_{b}$, we empirically derive the novel camera view $W_{b}$, which can be used to render the associated 2D semantic embeddings from the Gaussian model. Our goal is to enhance the semantic consistency across existing views and their associated synthesized views, thereby regularizing the training of 3D semantic representations. Thus, the camera view blending loss $\mathcal{L}_{b}$ for semantics with identical interpolation ratio $k$ is:

$$ $\mathcal{L}_{b}=\mathrm{SSIM}(I_{1},I_{2})\mathbb{E}_{v\in I_{1}}\mathcal{L}_{ \mathrm{MSE}}(F^{W_{b}}(v),H^{W_{b}}(v))\;\text{ with }\;H^{W_{b}}=kH^{W_{1}}+ (1-k)H^{W_{2}}.$ (11) $$

Here $F^{W_{b}}(v)$ denotes the rendered 2D semantic embeddings based on $W_{b}$ using the rasterizer described in Section 3.2, while $H^{W_{1}}(v)$ and $H^{W_{2}}(v)$ represent the ground truth 2D semantic features for $W_{1}$ and $W_{2}$ respectively. When the two images differ significantly, regularization may be counterproductive. To address this, we utilize the Structural Similarity Index Measure (SSIM) between $I_{1}$ and $I_{2}$ as a weighting factor. By enforcing similarity between $H^{W_{b}}(v)$ and $F^{W_{b}}(v)$, we enhance the semantic consistency of the 3D representations, thereby mitigating the overfitting issue.

By combining the rasterization loss (Eq. 8) and the camera view blending loss (Eq. 11), the overall objective is represented as:

$$ $\mathcal{L}_{\mathrm{overall}}=\mathbb{E}_{(I_{1},W_{1}),(I_{2},W_{2})\in T^{r }}[\mathcal{L}_{\mathrm{raster}}(W_{1})+\mathcal{L}_{\mathrm{raster}}(W_{2})+ \lambda\mathcal{L}_{b}(W_{1},W_{2})],$ (12) $$

where $\lambda$ is a trade-off parameter.

## 4 Experiments

### 4.1 Settings

Datasets. We employ 3 datasets for our evaluation on open-vocabulary semantic tasks.
(1) LERF dataset (Kerr et al., ), captured using the Polycam application on an iPhone, comprises complex, in-the-wild scenes and is specifically tailored for 3D object localization tasks. We report the mean Intersection over Union (mIoU) results, alongside localization accuracy results in accordance with (Qin et al., ) (2) 3D-OVS dataset (Liu et al., ) consists of a diverse collection of long-tail objects in various poses and backgrounds. This dataset is specifically developed for open-vocabulary 3D semantic segmentation and includes a complete list of categories. Following (Qin et al., ; Zuo et al., ), the mIoU metric is applied for this dataset.
(3) Mip-NeRF 360 dataset (Barron et al., ) contains 9 scenes, each composed of a complex central object or area and a detailed background. Following Qu et al. (), we provide some evaluations on this dataset in annex.

Implementation. To extract ground-truth semantic features from each image, we utilize the SAM ViT-H model (Kirillov et al., ) alongside the OpenCLIP ViT-B/16 model (Radford et al., ). The 3D Gaussians for each scene are initialized using sparse point clouds derived from Structure from Motion. For modality fusion, we set $d_{c}$ and $d_{f}$ to 3, while $d_{h}$ is set to 4. During rasterization, smoothed semantic indicator is initialized in the same manner as color opacity. For each iteration, two camera views and their associated images are selected, and 3D Gaussians are trained for 15,000 iterations. Moreover, the parameter $\lambda$ in Equation 12 is configured to 1.2. The implementation of semantic opacity is done in CUDA and C++, while the other components are in PyTorch. All experiments are conducted on Nvidia A100 GPUs. For each scene, our model is trained for 15,000 iterations using an Adam optimizer (Kingma, ), and the learning rates of different components are shown in the appendix. After convergence, the model is applied to the localization and segmentation tasks according to the procedure described in Sec. 3.1.

**Table 1: mIoU results on the LERF dataset.**
| Method | Venue | ramen | figurines | teatime | kitchen | avg. |
| --- | --- | --- | --- | --- | --- | --- |
| Feature-3DGS | CVPR’24 | 43.7 | 58.8 | 40.5 | 39.6 | 45.7 |
| LEGaussians | CVPR’24 | 46.0 | 60.3 | 40.8 | 39.4 | 46.9 |
| LangSplat | CVPR’24 | 51.2 | 65.1 | 44.7 | 44.5 | 51.4 |
| GS-Grouping | ECCV’24 | 45.5 | 60.9 | 40.0 | 38.7 | 46.3 |
| GOI | ACMMM’24 | 52.6 | 63.7 | 44.5 | 41.4 | 50.6 |
| Ours |  | 61.4 | 73.5 | 58.1 | 54.8 | 62.0 |

**Table 2: Localization accuracy ($\%$) on the LERF dataset.**
| Method | Venue | ramen | figurines | teatime | kitchen | avg. |
| --- | --- | --- | --- | --- | --- | --- |
| Feature-3DGS | CVPR’24 | 69.8 | 77.2 | 73.4 | 87.6 | 77.0 |
| LEGaussians | CVPR’24 | 67.5 | 75.6 | 75.2 | 90.3 | 77.2 |
| LangSplat | CVPR’24 | 73.2 | 88.1 | 80.4 | 95.5 | 84.3 |
| GS-Grouping | ECCV’24 | 68.6 | 75.0 | 74.3 | 88.2 | 76.5 |
| GOI | ACMMM’24 | 75.5 | 88.6 | 82.9 | 90.4 | 84.4 |
| FMGS | IJCV’24 | 90.0 | 89.7 | 93.8 | 92.6 | 91.5 |
| Ours |  | 92.5 | 95.8 | 97.1 | 98.6 | 96.0 |

**Table 3: mIoU results on the 3D-OVS dataset.**
| Method | Venue | bed | bench | room | sofa | lawn | avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Feature-3DGS | CVPR’24 | 83.5 | 90.7 | 84.7 | 86.9 | 93.4 | 87.8 |
| LEGaussians | CVPR’24 | 84.9 | 91.1 | 86.0 | 87.8 | 92.5 | 88.5 |
| LangSplat | CVPR’24 | 92.5 | 94.2 | 94.1 | 90.0 | 96.1 | 93.4 |
| GS-Grouping | ECCV’24 | 83.0 | 91.5 | 85.9 | 87.3 | 90.6 | 87.7 |
| GOI | ACMMM’24 | 89.4 | 92.8 | 91.3 | 85.6 | 94.1 | 90.6 |
| FMGS | IJCV’24 | 80.6 | 84.5 | 87.9 | 90.8 | 92.6 | 87.3 |
| Ours |  | 96.8 | 97.3 | 97.7 | 95.5 | 97.9 | 97.1 |

Figure: Figure 6: Localization comparisons on LERF. Black dotted-line bounding boxes represent the ground-truth targets, and red dots indicate the predictions (considered correct if within a GT box).
Refer to caption: x6.png

Figure: Figure 7: Qualitative semantic segmentation comparisons on the bench scene (3D-OVS dataset).
Refer to caption: x7.png

Figure: Figure 8: Qualitative semantic segmentation comparisons on the sofa scene (3D-OVS dataset).
Refer to caption: x8.png

**Table 4: Ablation results of modality fusion on the LERF dataset, in terms of mIoU.**
| Method | ramen | figurines | teatime | kitchen | avg. |
| --- | --- | --- | --- | --- | --- |
| No Fusion | 57.8 | 69.6 | 54.2 | 51.5 | 58.3 |
| Single-layer MLP Fusion | 55.9 | 68.2 | 53.0 | 49.9 | 56.8 |
| Cross-attention Modality Fusion | 57.3 | 70.7 | 54.6 | 52.0 | 58.7 |
| Self-attention Modality Fusion (Ours) | 61.4 | 73.5 | 58.1 | 54.8 | 62.0 |

### 4.2 Results

Baselines. For a fair comparison, we select the latest works on open-vocabulary 3D scene understanding: Feature-3DGS (Zhou et al., ), LEGaussians (Shi et al., ), LangSplat (Qin et al., ), GS-Grouping (Ye et al., ), GOI (Qu et al., ), and FMGS (Zuo et al., ), all of which are based on 3DGS (Kerbl et al., ), and employ the same segmentation and vision-language models mentioned in Sec. 4.1 to extract ground truth language features. *It is important to note that FMGS (Zuo et al., ) does not report mIoU results on the LERF dataset and is also not open-sourced, so its mIoU results are not listed.*

Quantitative Evaluation. Tab. 1 and Tab. 2 present the mIoU and localization accuracy results on the LERF dataset, respectively. Tab. 3 displays the mIoU results on the 3D-OVS dataset. Our proposed method achieves state-of-the-art performance across all scenes, notably outperforming the second best method LangSplat (Qin et al., ) by 10.6 in mIoU on the LERF dataset. Further experiments on other datasets are listed in the appendix.

Qualitative Evaluation. To facilitate qualitative comparisons, we contrast our method with several baseline approaches, including LEGaussian (Shi et al., ), LangSplat (Qin et al., ), and GOI (Qu et al., ). Fig. 4 and Fig. 5 illustrate the semantic segmentation results on the LERF dataset, while Figure 6 presents the localization results on the same dataset. For the localization task, the black bounding boxes with dotted lines represent the ground truth ranges, while the red dots indicate predictions from various methods. Besides, segmentation comparisons on the 3D-OVS dataset are displayed in Fig. 7 and Fig. 8. It is clear that our proposed method outperforms the other approaches and is closer to the ground-truth, particularly in reflective and translucent areas. Additional results are displayed in the appendix.

### 4.3 Ablation Studies

Ablation on cross-modal rasterizer. Tab. 4 presents an ablation study on the data-fusion of the rasterizer. In addition to no fusion like LEGaussians (Shi et al., ), we also incorporate single-layer MLP and cross-attention for modality Fusion to enhance semantic features for comparison. The results indicate that self-attention modality fusion is the most effective method. Besides, single-layer MLP and cross-attention modality fusion do not demonstrate any advantage over the no fusion condition. In Tab. 5, we highlight the benefits of our proposed smoothed semantic indicator. We compare it to previous methods for rasterizing semantic embeddings, specifically by applying the color opacity $o^{i}$ to the language modality (Qin et al., ; Zhou et al., ; Zuo et al., ). The results indicate that using a separate attribute for rasterizing language embeddings is clearly advantageous. However, this new attribute cannot be naively fixed, *e.g*., to 1 or 0.5 for all Gaussians. Its values should be learned by the model through 2D supervision, which is why we introduce a learnable indicator for the $\alpha$-blending of language modality.

**Table 5: Ablation results of smoothed semantic indicator on the LERF dataset, in terms of mIoU.**
| Method | ramen | figurines | teatime | kitchen | avg. |
| --- | --- | --- | --- | --- | --- |
| Fixed to 0.5 | 52.2 | 70.9 | 50.6 | 50.8 | 56.1 |
| Fixed to 1.0 | 50.3 | 67.7 | 47.1 | 48.5 | 53.4 |
| Using Color Opacity | 55.9 | 71.0 | 54.2 | 51.3 | 58.1 |
| Smoothed Semantic Indicator (Ours) | 61.4 | 73.5 | 58.1 | 54.8 | 62.0 |

**Table 6: Ablation results of Camera View Blending on the LERF dataset, in terms of mIoU.**
| Rotation | Translation | SSIM | ramen | figurines | teatime | kitchen | avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| $\times$ | $\times$ | $\times$ | 55.4 | 68.2 | 48.8 | 50.9 | 55.8 |
| $\times$ | $\checkmark$ | $\times$ | 56.6 | 69.5 | 49.7 | 51.5 | 56.8 |
| $\times$ | $\checkmark$ | $\checkmark$ | 58.3 | 70.9 | 51.1 | 52.4 | 58.2 |
| $\checkmark$ | $\times$ | $\times$ | 55.7 | 69.3 | 49.6 | 51.7 | 56.6 |
| $\checkmark$ | $\times$ | $\checkmark$ | 57.9 | 70.2 | 52.3 | 52.1 | 58.1 |
| $\checkmark$ | $\checkmark$ | $\times$ | 59.6 | 71.8 | 53.4 | 53.2 | 59.5 |
| $\checkmark$ | $\checkmark$ | $\checkmark$ | 61.4 | 73.5 | 58.1 | 54.8 | 62.0 |

Ablation on camera-view blending. We also present an ablation of different camera-view blending strategies in Tab. 6, which includes three variants: Rotation that associates to the operation in Eq. 9, Translation that corresponds to the operation in Eq. 10 , and SSIM that weighs Eq. 11. Our proposed method corresponds to the last row of the table. The results indicate that both Slerp-based rotation blending and Lerp-based translation blending contribute to performance improvements. Moreover, the regularization provided by SSIM is essential for controlling the extent of blending.

**Table 7: Ablation results of interpolation ratio sampling on the LERF dataset, in terms of mIoU.**
| Method | ramen | figurines | teatime | kitchen | avg. |
| --- | --- | --- | --- | --- | --- |
| Fixed to 0.5 | 58.8 | 72.6 | 56.4 | 52.5 | 60.1 |
| Fixed to 0.75 | 56.3 | 71.0 | 50.5 | 51.7 | 57.4 |
| Fixed to 1.0 | 55.2 | 68.3 | 48.5 | 51.0 | 55.8 |
| Even Distribution | 58.3 | 71.1 | 55.4 | 51.2 | 59.0 |
| Gaussian Distribution | 57.5 | 71.3 | 53.9 | 50.7 | 58.4 |
| Beta Distribution (Ours) | 61.4 | 73.5 | 58.1 | 54.8 | 62.0 |

In Tab. 7, we present an ablation study on the interpolation ratio $k$. In addition to fixed values of $0.5$, $0.75$, and $1.0$, we also include experiments based on Even Distribution (U$(0,1)$), Gaussian Distribution (N$(0,1)$), and our Beta Distribution (Beta$(0.2,0.2)$). The results demonstrate that a balanced ratio is crucial for acquiring better representations, and that slight disturbances can be beneficial as well.

### 4.4 Efficiency Analysis

**Table 8: Efficiency analysis on LERF’s ramen.**
| Method | Training<br>Time $\downarrow$ | FPS $\uparrow$ | $\#$ of<br>Gaussians $\downarrow$ |
| --- | --- | --- | --- |
| LangSplat | 96min | 40 | 86k |
| GS-Grouping | 130min | 76 | 107k |
| GOI | 73min | 42 | 92k |
| Ours | 65min | 79 | 80k |

Finally, we demonstrate that, despite the new parameter $l$ added to the Gaussians, our contributions actually benefit the efficiency of the resulting scene representation, *i.e*., in terms of convergence speed and rendering time. As shown in Tab. 8, our model outperforms the other four baselines in efficiency. We attribute this performance to our cooperative training over augmented view/pose batches, and to the increased ability of the Gaussians to accurately fit both target modalities.

## 5 Conclusion

In this work, we propose 3D visual-language Gaussian splatting for semantic scene understanding, addressing the neglect of language information in current 3DGS approaches. We propose a novel cross-modal rasterizer that performs modality fusion followed by language-specific rasterization, utilizing a smoothed semantic indicator to disable irrelevant Gaussians, *e.g*., for scenes with complex light transport (reflections, translucence, *etc*.). Moreover, our camera-view blending technique effectively mitigates over-fitting, ensuring semantic consistency across both existing and synthesized viewpoints. Comprehensive experiments validate the effectiveness of our framework, demonstrating significant improvements in open-vocabulary semantic segmentation compared to prior art.
Future research will explore further modalities that could enrich the scene representation (*e.g*., semantic features from mixtures of foundation models), as well as extending this work to dynamic scenes.

## Appendix A Appendix

### A.1 Further Implementation Details

We complement the implementation details shared in Subsection 4.1: our solution is based on the official implementations of 3DGS Kerbl et al. () and LangSplat Qin et al. () made publicly available by their respective authors (*c.f*. links provided in their papers). Specifically, we edit the LangSplat’s 3DGS model to add our novel smoothed indicator, as well as the CUDA rasterization function for joint rendering of color and semantic modalities w.r.t. their respective opacity/indicator control.
As explained in the main paper, we also adapt the optimization script for joint learning over both modalities and for integrating the proposed view-blending augmentation, applied to pairs of batched input samples at every iteration. The learning rates applied to the different 3DGS attributes are provided in Tab. 9.

**Table 9: Learning rates applied to the parameters of the Gaussians.**
| Components | Learning Rates |
| --- | --- |
| position | $1.6\times 10^{-4}$ |
| scale | $5.0\times 10^{-3}$ |
| rotation | $1.0\times 10^{-3}$ |
| color features | $5.0\times 10^{-3}$ |
| color opacity | $5.0\times 10^{-2}$ |
| semantic features | $5.0\times 10^{-3}$ |
| semantic indicator | $5.0\times 10^{-2}$ |

### A.2 Evaluation on Additional Mip-NeRF 360 Dataset

#### Quantitative Results.

Building on the work of GOI (Qu et al., ), we also utilize the Mip-NeRF 360 dataset (Barron et al., ) for our evaluation. In this context, we employ APE (Shen et al., ) as the frozen backbone to extract ground truth 2D features. The results, presented in Tab. 10, clearly demonstrate the effectiveness of our proposed method, highlighting its ability to improve overall performance on this dataset.

#### Qualitative Results.

In Fig. 9 and Fig. 10, we present qualitative comparisons on the Mip-NeRF 360 dataset (Barron et al., ). The results demonstrate that our proposed method significantly outperforms the competing approaches.

### A.3 Additional Ablation Studies

**Table 10: mIoU results on the Mip-NeRF 360 dataset.**
| Method | Venue | garden | bonsai | kitchen | room | avg. |
| --- | --- | --- | --- | --- | --- | --- |
| Feature-3DGS | CVPR’24 | 45.1 | 46.2 | 46.8 | 17.5 | 38.9 |
| LEGaussians | CVPR’24 | 48.5 | 50.4 | 49.3 | 55.7 | 51.0 |
| LangSplat | CVPR’24 | 50.1 | 59.1 | 50.0 | 62.6 | 55.5 |
| GS-Grouping | ECCV’24 | 42.0 | 43.1 | 42.2 | 49.1 | 44.1 |
| GOI | ACMMM’24 | 85.0 | 91.5 | 84.3 | 85.0 | 86.5 |
| Ours |  | 89.2 | 93.7 | 87.6 | 88.8 | 89.8 |

Figure: Figure 9: Qualitative semantic segmentation comparisons on the bonsai scene of the Mip-NeRF 360 dataset.
Refer to caption: x9.png

Figure: Figure 10: Qualitative semantic segmentation comparisons on the room scene of the Mip-NeRF 360 dataset.
Refer to caption: x10.png

**Table 11: Ablation results of modality fusion on the 3D-OVS dataset.**
| Method | bed | bench | room | sofa | lawn | avg. |
| --- | --- | --- | --- | --- | --- | --- |
| No Fusion | 94.5 | 95.6 | 95.3 | 93.0 | 96.5 | 95.0 |
| Single-layer MLP Fusion | 93.1 | 94.7 | 94.4 | 91.7 | 96.0 | 94.1 |
| Cross-attention Modality Fusion | 94.7 | 95.9 | 95.6 | 93.2 | 96.6 | 95.2 |
| Self-attention Modality Fusion (Ours) | 96.8 | 97.3 | 97.7 | 95.5 | 97.9 | 97.1 |

#### Ablation of Modality Fusion.

Table 11 presents an ablation study of modality fusionusing the OVS-3D dataset. The results indicate that self-attention modality fusion is the most effective method.

**Table 12: Ablation results of smoothed semantic indicator on the 3D-OVS dataset.**
| Method | bed | bench | room | sofa | lawn | avg. |
| --- | --- | --- | --- | --- | --- | --- |
| Fixed to 0.5 | 93.4 | 94.9 | 95.1 | 92.0 | 95.4 | 94.2 |
| Fixed to 1.0 | 91.5 | 92.8 | 90.9 | 87.6 | 92.8 | 91.1 |
| Using Color Opacity | 94.7 | 95.2 | 95.0 | 92.7 | 96.4 | 94.8 |
| Smoothed Semantic Indicator (Ours) | 96.8 | 97.3 | 97.7 | 95.5 | 97.9 | 97.1 |

#### Ablation of Semantic Indicator.

Tab. 12 compares our proposed Smoothed Semantic Indicator against three alternatives: Fixed Value at 0.5, Fixed Value at 1, and Using Color Opacity on the 3D-OVS dataset. The results demonstrate that fixed values greatly under-perform compared to shared opacity derived from color rendering. Moreover, the learnable Smoothed Semantic Indicator surpasses the use of color opacity significantly. Thus, utilizing a learnable semantic indicator for semantic rasterization proves beneficial for the open-vocabulary segmentation task.

**Table 13: Ablation results of camera view blending on the 3D-OVS dataset.**
| Rotation | Translation | SSIM | bed | bench | room | sofa | lawn | avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\times$ | $\times$ | $\times$ | 92.9 | 94.6 | 95.2 | 91.5 | 96.1 | 94.1 |
| $\times$ | $\checkmark$ | $\times$ | 93.5 | 94.9 | 95.5 | 92.2 | 96.2 | 94.5 |
| $\times$ | $\checkmark$ | $\checkmark$ | 94.8 | 95.2 | 96.3 | 93.5 | 96.4 | 95.2 |
| $\checkmark$ | $\times$ | $\times$ | 94.0 | 94.8 | 95.7 | 92.3 | 96.3 | 94.6 |
| $\checkmark$ | $\times$ | $\checkmark$ | 95.1 | 95.4 | 96.2 | 93.0 | 96.2 | 95.2 |
| $\checkmark$ | $\checkmark$ | $\times$ | 96.2 | 96.8 | 96.7 | 94.6 | 96.4 | 96.1 |
| $\checkmark$ | $\checkmark$ | $\checkmark$ | 96.8 | 97.3 | 97.7 | 95.5 | 97.9 | 97.1 |

#### Ablation of Camera View Blending.

We present an ablation study of various camera view blending strategies in Tab. 13 using the 3D-OVS dataset, which includes three variants: Rotation, Translation, and SSIM. Our proposed method is represented in the last row of the table. The results indicate that both Slerp-based rotation blending and Lerp-based translation blending enhance performance. Furthermore, the regularization provided by SSIM is crucial for managing the extent of blending.

**Table 14: Ablation results of interpolation ratio sampling on the 3D-OVS dataset.**
| Method | bed | bench | room | sofa | lawn | avg. |
| --- | --- | --- | --- | --- | --- | --- |
| Fixed to 0.5 | 95.8 | 95.9 | 95.6 | 93.7 | 96.5 | 95.5 |
| Fixed to 0.75 | 94.1 | 95.2 | 95.0 | 92.5 | 95.4 | 94.4 |
| Fixed to 1.0 | 93.4 | 94.0 | 94.4 | 90.8 | 95.2 | 93.6 |
| Even Distribution | 94.6 | 95.1 | 94.8 | 93.2 | 96.1 | 94.8 |
| Gaussian Distribution | 94.0 | 94.5 | 94.2 | 92.9 | 96.0 | 94.3 |
| Beta Distribution (Ours) | 96.8 | 97.3 | 97.7 | 95.5 | 97.9 | 97.1 |

**Table 15: Ablation results of the parameters of Beta distribution on the LERF dataset, in terms of mIoU.**
| Method | ramen | figurines | teatime | kitchen | avg. |
| --- | --- | --- | --- | --- | --- |
| Beta(0.1, 0.1) | 59.7 | 72.2 | 57.3 | 54.4 | 60.9 |
| Beta(0.2, 0.2) | 61.4 | 73.5 | 58.1 | 54.8 | 62.0 |
| Beta(0.3, 0.3) | 60.7 | 72.8 | 57.1 | 54.2 | 61.2 |
| Beta(0.4, 0.4) | 59.6 | 72.0 | 56.8 | 54.1 | 60.6 |

In Tab. 14, we further present an ablation study on the interpolation ratio used within the camera view blending scheme, performed on the 3D-OVS dataset. Alongside fixed values of $0.5$, $0.75$, and $1.0$, we also include experiments utilizing Even Distribution (U$(0,1)$), Gaussian Distribution (N$(0,1)$), and our Beta Distribution (Beta$(0.2,0.2)$). The results indicate that a balanced ratio is essential for achieving better representations, and that minor disturbances can also yield beneficial effects.

We also change the parameter of the Beta distribution in the blending loss $\mathcal{L}_{b}$ on the LERF dataset, and the results are exhibited in Tab. 15, showing our selection is reasonable. It is important to keep a relatively balanced ratio while introducing some disturbances.

Figure: Figure 11: Qualitative semantic segmentation comparisons on the teatime scene of the LERF dataset.
Refer to caption: x11.png

Figure: Figure 12: Qualitative semantic segmentation comparisons on the figurines scene of the LERF dataset.
Refer to caption: x12.png

Figure: Figure 13: Additional qualitative examples of object localization in the LERF dataset.
Refer to caption: x13.png

### A.4 Additional Qualitative Results

#### Results on LERF Data.

Fig. 11 and Fig. 12 present additional semantic segmentation results on the LERF dataset, while Fig. 13 displays further localization results for the same dataset. Our proposed method clearly outperforms other approaches, particularly in reflective and translucent areas, where it more closely aligns with the ground truth.

Figure: Figure 14: Qualitative semantic segmentation comparisons on the room scene (3D-OVS dataset).
Refer to caption: x14.png

Figure: Figure 15: Qualitative semantic segmentation comparisons on the lawn scene (3D-OVS dataset).
Refer to caption: x15.png

#### Results on 3D-OVS Data.

We present additional segmentation comparisons on the 3D-OVS dataset in Fig. 14 and Fig. 15. Our proposed method demonstrates superior performance compared to other approaches, showing greater alignment with the ground truth.

#### Impact of Semantic-Indicator Contribution on Relevancy Maps.

In Fig. 16, we share the relevancy maps (represented as heatmaps) corresponding to various open-vocabulary prompts, comparing the results for color/semantic 3DGS models incorporating our novel smoothed semantic (last row) versus baseline models re-using the color opacity during rasterization of semantic data (mid row).
The results clearly demonstrate the effectiveness of the semantic indicator in uncovering semantic information, particularly in translucent and reflective objects. This enhanced capability allows for a more nuanced understanding of the scene, highlighting how the proposed method improves the semantic representations.

Figure: Figure 16: Qualitative impact of the proposed additional semantic indicator on 2D relevancy maps w.r.t. various open-vocabulary prompts.
Refer to caption: x16.png

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: (1) What is the different between self-attention and cross-attention mechanism?

(2) Considering the experiment setting is the open-vocabulary, for example, the CLIP feature dimension is 768, how to obtain 768 rendering with d_c+d_f=6? How to perform the dim. Reduction as shown in Figure.2
soundness: 3
strengths: 1. The paper is well-written and easy to follow.

2. The designs are well-motivated.

3. The performance shows a significant improvement.
summary: This paper focuses on improving the semantic rendering quality within well-reconstructed 3D scenes. With introducing self-attention, semantic indicator and regularization, the  proposed method achieves better semantic segmentation results, especially for 
transparent objects.
weaknesses: (1) Line 215, the u^i is the fused features or position?

(2) In camera interpolation, why use mix-up to generate a feature map instead of using an off-the-shelf OV model to output one? Just like previous methods.

(3) Missing a global ablation study to present the importance of 'self-attention', 'semantic indicator'. 'camera interpolation' over your baseline. Now, it is unclear how the baseline stands in comparison with previous methods and the contributions from the three proposed designs.

(4)  Low efficiency, the Table. 8 shows it takes over one hour to learn the semantics of single 3D scenes, which is significantly longer than typical 3D Gaussian reconstruction (which may take less than 10 mins on A100？), especially considering the iteration is only 15k. What makes the rasterization so slow? In addition, since more attributions are added in Gaussian, it is fair to provide the storage cost of Gaussian models when compared with other works in Table.8 instead of only Gaussian numbers.

(5) The incomplete experiments. The experiments on 3D-OVS are conducted on  5/7 scenes instead of full scenes.

(6) No deep analysis of performance. In my view, the IoU improvements are mainly from the translucent or reflective objects, as shown in Figure.3, no detailed statistics to prove that.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. What is the specific relationship between the improvement brought by your method and the details of the scene? In what kind of scene is your improvement more obvious?
soundness: 2
strengths: 1. The motivation of this paper  sounds reasonable. The over-fitting on the color modality may have a negative impact on semantic learning, which is consistent with intuition.
2. The proposed techniques, including the smoothed semantic indicator  and the mix-up augmentation, are simple but effective.
3. The performance of the method  outperform existing methods by a significant margin.
summary: This paper claims that the performances of current multi-modal scene understanding approaches are limited by the the imbalance between visual and language modalities, which have widely different properties.
To alleviate these limitations, this paper proposes several strategies, including a cross-modal rasterizer that places greater emphasis on language features and a camera-view blending technique.
Finally, the proposed methods achieves state-ofthe-art performance in open-vocabulary semantic segmentation tasks.
weaknesses: 1. Although the motivation  sounds reasonable and the Figure 3 explain the motivation to some extent, I expect more experiments to further study the differences between color and semantics modalities. Beside showing quantitative, qualitative results and ablation of the proposed strategies, the authors should further discuss the deeper mechanisms. For example, the authors can study the relationship between color over-fitting phenomenon and specific scenes.
2. The figure 2 is hard to understand. The method is simple but the process in the figure looks complicated. Some fonts are too small.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: I am curious about the impact of semantic learning on rendering quality.
soundness: 3
strengths: 1. The paper provides a profound insight into the challenges posed by semi-opaque media and intricate light transport effects, highlighting the limitations in translating color opacity to the semantic domain. This observation is interesting. 
2. The implementation of a single learnable parameter to replace the conventional shared color opacity parameter is both straightforward and efficient.
3. The paper is commendably structured, presenting a well-motivated narrative and a clear methodology.
summary: This paper learns language embedded 3dgs field and addresses the often-neglected issue of appearance-semantic misalignment in rasterization. It introduces a camera-view blending technique to enhance semantic consistency between existing and synthesized views. The experimental findings demonstrate that the proposed approach outperforms existing methodologies.
weaknesses: 1. The paper introduces the camera-view blending technique as a method to enhance cross-view semantic consistency. However, it falls short in fully addressing the challenge of different objects sharing similar colors, potentially leading to indistinguishable semantic representations, which the authors claim to have tackled. Further insights on this critical point are needed.
2. In Section 4.4, the experimental results suggest an increase in rendering speed with the proposed pipeline. However, given the introduction of additional steps and parameters, one would expect a potential decrease in inference speed. The discrepancy in the reported results requires clarification.
3. The title of the paper, "3D Vision-Language Gaussian Splatting," might be misleading as it predominantly focuses on learning open language semantic fields rather than visual appearances. A revision to better reflect the core contribution of the research is recommended.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1、	Equation (3), any language/semantic should not change due to view angle, so F^W doesn't make sense to me, even the authors mention that superscript W will be omitted for ease of reading. It shouldn’t appear in the original equation at all. Also in the same equation, there lacks a clear definition of what L_sem is, but this should be a minor issue.
2、	Figure 3 right, I’m not 100% sure where the symmetry of the density distribution is from, as there lacks sufficient correlation between l and o. 
3、	Table 8, an average training and inference time should be reported on all scenes of a whole dataset, instead of on just one. 
4、	The Related Work session is relatively weak and could be fortified with more related reference included. For instance, “HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting”, which proposes another way of using semantic info in 3DGS (not traditional 2d way mentioned by the authors）and is worthy of being cited. 
5、	A minor grammar error of Line 356-357.
soundness: 3
strengths: The proposed idea is interesting, especially because it offers a better solution for semantic rasterization for reflective or translucent objects, which is a serious problem with the original 3DGS. For me, the designed semantic indicator in the Gaussian representation is novel and useful for the claimed main problem. The main contributions also include the empirical quality result of the semantic indicator on these reflective or translucent objects, and quantity comparison of open-vocabulary semantic segmentation tasks. These results conclude the usefulness of the proposed solution.
summary: This paper proposes an interesting idea on how to fuse multi-modality info, i.e. visual & semantic, for better semantic rasterization in 3DGS. This solution mainly composes of a cross-modal rasterizer, a specially designed semantic indicator and a camera-view blending method. The solution is interesting especially because it offers a better solution for semantic rasterization for reflective or translucent objects, which is a serious problem with the original 3DGS. For me, the designed semantic indicator in the Gaussian representation is novel and useful for the claimed main problem. The main contributions also include the empirical quality result of the semantic indicator on these reflective or translucent objects, and quantity comparison of open-vocabulary semantic segmentation tasks. These results conclude the usefulness of the proposed solution. 

The paper is generally well written and structured, easy to follow and read. I don’t come across many issues for further clarification. But here a few things I may need the authors to respond. Given those issues properly clarified, I would be willing to increase the score.

1、	Equation (3), any language/semantic should not change due to view angle, so F^W doesn't make sense to me, even the authors mention that superscript W will be omitted for ease of reading. It shouldn’t appear in the original equation at all. Also in the same equation, there lacks a clear definition of what L_sem is, but this should be a minor issue.
2、	Figure 3 right, I’m not 100% sure where the symmetry of the density distribution is from, as there lacks sufficient correlation between l and o. 
3、	Table 8, an average training and inference time should be reported on all scenes of a whole dataset, instead of on just one. 
4、	The Related Work session is relatively weak and could be fortified with more related reference included. For instance, “HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting”, which proposes another way of using semantic info in 3DGS (not traditional 2d way mentioned by the authors）and is worthy of being cited. 
5、	A minor grammar error of Line 356-357.
weaknesses: A few issues are listed in the next section.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Is the fusion module per-scene trained? Would it be more effective to train it across different scenes to learn the prior?
2. According to Table 6 and Table 7, does the view interpolation method alone improve the performance by 6.2 mAP? It's a little counter-intuitive that simply weighted-summing the two views to supervise the interpolated views can lead to such improvements, since the weighted-sum semantics (as shown in Figure 2) do not consider the relative camera poses between the two views. What if you wrap the two views to the interpolated view using the known camera poses and learned depths?
3. The paper discussed using separate opacity for semantics, then what would the results be if learning separate covariance matrix in each gaussian for color and semantics? Besides, if we simply train two separate sets of Gaussians for color and semantic renderings respectively, would it improve the semantic segmentation results?
soundness: 3
strengths: 1. The proposed method is novel and promising. Using another learnable parameter as opacity for semantic blending is an overlooked method that benefits open-vocabulary 3DGS.
2. The results are promising.
summary: This paper proposes a novel approach to open-vocabulary 3DGS. It proposes to use another learnable opacity parameter for the blending of semantics, and train a cross-modal fusion module. The authors also propose a semantic-aware camera view blending method, in which they randomly interpolate between training views and supervise the semantic rendering based on the weighted sum of the semantics. The experiments show that the proposed method significantly improves the open-vocabulary semantic segmentation results.
weaknesses: Please see the questions below.

Meta-review: This paper presents a 3D vision-language Gaussian splatting model for scene understanding. The proposed method includes a semantic indicator, a cross-modal rasterizer, and a camera-view blending augmentation technique. The paper aims to address limitations in existing methods, particularly challenges with translucent and reflective objects, as well as semantic misalignment due to color overfitting.

The reviewers acknowledge the paper's contributions, rigorous experiments, and potential impacts. There are some initial concerns regarding clarify, experiment details, and method novelty, which are all addressed through additional experiments and thorough response. Therefore, AC recommends accepting this paper.

Decision: Accept (Poster)