3D Vision-Language Gaussian Splatting

Authors: Qucheng Peng, Benjamin Planche, Zhongpai Gao, Meng Zheng, Anwesa Choudhuri, Terrence Chen, Chen Chen, Ziyan Wu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Multi-modal 3D Scene Understanding, Gaussian Splatting, Open-vocabulary Semantic Segmentation, Open-vocabulary Object Localization

Abstract:
Recent advancements in 3D reconstruction methods and vision-language models have propelled the development of multi-modal 3D scene understanding, which has vital applications in robotics, autonomous driving, and virtual/augmented reality. However, current multi-modal scene understanding approaches have naively embedded semantic representations into 3D reconstruction methods without striking a balance between visual and language modalities, which leads to unsatisfying semantic rasterization of translucent or reflective objects, as well as over-fitting on color modality. To alleviate these limitations, we propose a solution that adequately handles the distinct visual and semantic modalities, i.e., a 3D vision-language Gaussian splatting model for scene understanding, to put emphasis on the representation learning of language modality. We propose a novel cross-modal rasterizer, using modality fusion along with a smoothed semantic indicator for enhancing semantic rasterization. We also employ a camera-view blending technique to improve semantic consistency between existing and synthesized views, thereby effectively mitigating over-fitting. Extensive experiments demonstrate that our method achieves state-of-the-art performance in open-vocabulary semantic segmentation, surpassing existing methods by a significant margin.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1729341490771

thread_type:
official_review

<thread_object>
creator: reviewer_56rb

<content>
summary:
This paper proposes a novel approach to open-vocabulary 3DGS. It proposes to use another learnable opacity parameter for the blending of semantics, and train a cross-modal fusion module. The authors also propose a semantic-aware camera view blending method, in which they randomly interpolate between training views and supervise the semantic rendering based on the weighted sum of the semantics. The experiments show that the proposed method significantly improves the open-vocabulary semantic segmentation results.

strengths:
1. The proposed method is novel and promising. Using another learnable parameter as opacity for semantic blending is an overlooked method that benefits open-vocabulary 3DGS.
2. The results are promising.

weaknesses:
Please see the questions below.

questions:
1. Is the fusion module per-scene trained? Would it be more effective to train it across different scenes to learn the prior?
2. According to Table 6 and Table 7, does the view interpolation method alone improve the performance by 6.2 mAP? It's a little counter-intuitive that simply weighted-summing the two views to supervise the interpolated views can lead to such improvements, since the weighted-sum semantics (as shown in Figure 2) do not consider the relative camera poses between the two views. What if you wrap the two views to the interpolated view using the known camera poses and learned depths?
3. The paper discussed using separate opacity for semantics, then what would the results be if learning separate covariance matrix in each gaussian for color and semantics? Besides, if we simply train two separate sets of Gaussians for color and semantic renderings respectively, would it improve the semantic segmentation results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730108641714

thread_type:
official_review

<thread_object>
creator: reviewer_ycdx

<content>
summary:
This paper proposes an interesting idea on how to fuse multi-modality info, i.e. visual & semantic, for better semantic rasterization in 3DGS. This solution mainly composes of a cross-modal rasterizer, a specially designed semantic indicator and a camera-view blending method. The solution is interesting especially because it offers a better solution for semantic rasterization for reflective or translucent objects, which is a serious problem with the original 3DGS. For me, the designed semantic indicator in the Gaussian representation is novel and useful for the claimed main problem. The main contributions also include the empirical quality result of the semantic indicator on these reflective or translucent objects, and quantity comparison of open-vocabulary semantic segmentation tasks. These results conclude the usefulness of the proposed solution. 

The paper is generally well written and structured, easy to follow and read. I don’t come across many issues for further clarification. But here a few things I may need the authors to respond. Given those issues properly clarified, I would be willing to increase the score.

1、	Equation (3), any language/semantic should not change due to view angle, so F^W doesn't make sense to me, even the authors mention that superscript W will be omitted for ease of reading. It shouldn’t appear in the original equation at all. Also in the same equation, there lacks a clear definition of what L_sem is, but this should be a minor issue.
2、	Figure 3 right, I’m not 100% sure where the symmetry of the density distribution is from, as there lacks sufficient correlation between l and o. 
3、	Table 8, an average training and inference time should be reported on all scenes of a whole dataset, instead of on just one. 
4、	The Related Work session is relatively weak and could be fortified with more related reference included. For instance, “HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting”, which proposes another way of using semantic info in 3DGS (not traditional 2d way mentioned by the authors）and is worthy of being cited. 
5、	A minor grammar error of Line 356-357.

strengths:
The proposed idea is interesting, especially because it offers a better solution for semantic rasterization for reflective or translucent objects, which is a serious problem with the original 3DGS. For me, the designed semantic indicator in the Gaussian representation is novel and useful for the claimed main problem. The main contributions also include the empirical quality result of the semantic indicator on these reflective or translucent objects, and quantity comparison of open-vocabulary semantic segmentation tasks. These results conclude the usefulness of the proposed solution.

weaknesses:
A few issues are listed in the next section.

questions:
1、	Equation (3), any language/semantic should not change due to view angle, so F^W doesn't make sense to me, even the authors mention that superscript W will be omitted for ease of reading. It shouldn’t appear in the original equation at all. Also in the same equation, there lacks a clear definition of what L_sem is, but this should be a minor issue.
2、	Figure 3 right, I’m not 100% sure where the symmetry of the density distribution is from, as there lacks sufficient correlation between l and o. 
3、	Table 8, an average training and inference time should be reported on all scenes of a whole dataset, instead of on just one. 
4、	The Related Work session is relatively weak and could be fortified with more related reference included. For instance, “HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting”, which proposes another way of using semantic info in 3DGS (not traditional 2d way mentioned by the authors）and is worthy of being cited. 
5、	A minor grammar error of Line 356-357.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730469686751

thread_type:
official_review

<thread_object>
creator: reviewer_jaap

<content>
summary:
This paper learns language embedded 3dgs field and addresses the often-neglected issue of appearance-semantic misalignment in rasterization. It introduces a camera-view blending technique to enhance semantic consistency between existing and synthesized views. The experimental findings demonstrate that the proposed approach outperforms existing methodologies.

strengths:
1. The paper provides a profound insight into the challenges posed by semi-opaque media and intricate light transport effects, highlighting the limitations in translating color opacity to the semantic domain. This observation is interesting. 
2. The implementation of a single learnable parameter to replace the conventional shared color opacity parameter is both straightforward and efficient.
3. The paper is commendably structured, presenting a well-motivated narrative and a clear methodology.

weaknesses:
1. The paper introduces the camera-view blending technique as a method to enhance cross-view semantic consistency. However, it falls short in fully addressing the challenge of different objects sharing similar colors, potentially leading to indistinguishable semantic representations, which the authors claim to have tackled. Further insights on this critical point are needed.
2. In Section 4.4, the experimental results suggest an increase in rendering speed with the proposed pipeline. However, given the introduction of additional steps and parameters, one would expect a potential decrease in inference speed. The discrepancy in the reported results requires clarification.
3. The title of the paper, "3D Vision-Language Gaussian Splatting," might be misleading as it predominantly focuses on learning open language semantic fields rather than visual appearances. A revision to better reflect the core contribution of the research is recommended.

questions:
I am curious about the impact of semantic learning on rendering quality.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705737308

thread_type:
official_review

<thread_object>
creator: reviewer_epr2

<content>
summary:
This paper claims that the performances of current multi-modal scene understanding approaches are limited by the the imbalance between visual and language modalities, which have widely different properties.
To alleviate these limitations, this paper proposes several strategies, including a cross-modal rasterizer that places greater emphasis on language features and a camera-view blending technique.
Finally, the proposed methods achieves state-ofthe-art performance in open-vocabulary semantic segmentation tasks.

strengths:
1. The motivation of this paper  sounds reasonable. The over-fitting on the color modality may have a negative impact on semantic learning, which is consistent with intuition.
2. The proposed techniques, including the smoothed semantic indicator  and the mix-up augmentation, are simple but effective.
3. The performance of the method  outperform existing methods by a significant margin.

weaknesses:
1. Although the motivation  sounds reasonable and the Figure 3 explain the motivation to some extent, I expect more experiments to further study the differences between color and semantics modalities. Beside showing quantitative, qualitative results and ablation of the proposed strategies, the authors should further discuss the deeper mechanisms. For example, the authors can study the relationship between color over-fitting phenomenon and specific scenes.
2. The figure 2 is hard to understand. The method is simple but the process in the figure looks complicated. Some fonts are too small.

questions:
1. What is the specific relationship between the improvement brought by your method and the details of the scene? In what kind of scene is your improvement more obvious?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711661173

thread_type:
official_review

<thread_object>
creator: reviewer_awye

<content>
summary:
This paper focuses on improving the semantic rendering quality within well-reconstructed 3D scenes. With introducing self-attention, semantic indicator and regularization, the  proposed method achieves better semantic segmentation results, especially for 
transparent objects.

strengths:
1. The paper is well-written and easy to follow.

2. The designs are well-motivated.

3. The performance shows a significant improvement.

weaknesses:
(1) Line 215, the u^i is the fused features or position?

(2) In camera interpolation, why use mix-up to generate a feature map instead of using an off-the-shelf OV model to output one? Just like previous methods.

(3) Missing a global ablation study to present the importance of 'self-attention', 'semantic indicator'. 'camera interpolation' over your baseline. Now, it is unclear how the baseline stands in comparison with previous methods and the contributions from the three proposed designs.

(4)  Low efficiency, the Table. 8 shows it takes over one hour to learn the semantics of single 3D scenes, which is significantly longer than typical 3D Gaussian reconstruction (which may take less than 10 mins on A100？), especially considering the iteration is only 15k. What makes the rasterization so slow? In addition, since more attributions are added in Gaussian, it is fair to provide the storage cost of Gaussian models when compared with other works in Table.8 instead of only Gaussian numbers.

(5) The incomplete experiments. The experiments on 3D-OVS are conducted on  5/7 scenes instead of full scenes.

(6) No deep analysis of performance. In my view, the IoU improvements are mainly from the translucent or reflective objects, as shown in Figure.3, no detailed statistics to prove that.

questions:
(1) What is the different between self-attention and cross-attention mechanism?

(2) Considering the experiment setting is the open-vocabulary, for example, the CLIP feature dimension is 768, how to obtain 768 rendering with d_c+d_f=6? How to perform the dim. Reduction as shown in Figure.2
</content>
</thread_object>

</thread>
<thread>
creation date:
1732151888558

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response to Reviewers

comment:
We are very grateful to the reviewers `Awye`, `EPr2`, `Jaap`, `yCDX`, and `56RB` for the constructive feedback, as well as the recognition of our paper's strengths, such as its novelty and effectiveness.

In this global response, we further highlight the strengths of our work shared by reviewers and address the marginal concern and question brought by more than one reviewer. Other individual questions are answered in our per-reviewer responses, and we are looking forward to further discussing with reviewers.

----
## Shared Strengths

### 1. Novel and Effective Method

Reviewers commended the novelty of our method [`yCDX`, `56RB`], highlighting its intuitive yet highly effective contributions [`EPr2`, `Jaap`, `yCDX`]

### 2. Significant Improvements

All reviewers [`Awye`, `EPr2`, `Jaap`, `yCDX`, `56RB`] acknowledge that our method outperforms existing solutions, with many deeming the improvement as "_significant/promising_" [`Awye`, `EPr2`, `yCDX`, `56RB`].


### 3. Well-motivated and Insightful Work

Reviewers overall agree that our work is well-motivated [`Awye`, `EPr2`, `Jaap`] and tackles a "_serious/overlooked problem_" [`yCDX`, `56RB`] impacting prior works. Reviewer `Jaap` adds that our insight about semi-opaque media and intricate light transport effects is "_profound_".

### 4. Well-written and Structured Paper

Several reviewers [`Awye`, `Jaap`, `yCDX`] praised our manuscript as being structured, well-written, and easy to follow. 

----
## Shared Questions

Overall, the questions and concerns brought by the reviewers have little overlap, except for the two listed below. We partly tackle them below and further answer each reviewer in their respective response.

### 1. More Throrough Study of Contributions Impact on Specific Content

Reviewers `EPr2` and `Jaap` fairly remark that we could have more thoroughly studied the impact of our contributions on specific scenes content, such as the positive impact of disentangling color/language rasterization on specific objects with challenging appearances [`EPr2`] or the impact of the cross-view regularization brought by our augmentation technique on objects sharing similar colors [`Jaap`].

We agree that our manuscript could benefit from additional experiments and visualizations highlighting our performance w.r.t. these specific, challenging cases. Therefore, the following experiments and discussions have been added to the Appendix:

- In **new Tables 26-30** and Appendix A.5.3, we analyze the significant impact of disentangling appearance and language rasterization on translucent and reflective objects (w.r.t. downstream semantic tasks). See discussion with Reviewer `Awye` for more details. 
- In **new Figures 18-19** and Appendix A.4.4, we qualitatively highlight and discuss the impact of color over-fitting over specific scenes. We show examples of scenes with multiple objects sharing the same colors, and scenes containing objects composed of multiple colors/textures. Our solution with view-blending augmentation clearly outperforms methods without on these objects, as further discussed with Reviewer `Jaap`.


### 2. Question about Online View-blending Augmentation vs. Offline 3DGS+CLIP-based Augmentation

Both Reviewers `Awye` and `56RB` raised the question of using a pre-trained color-only 3DGS model and or the off-the-shelf feature-extractor model to assist with data augmentation, instead of our proposed view-blending technique. In our responses to both, we highlight how our online augmentation scheme is not only easier to adopt to due its light computational overhead, bu that it also results in better cross-view regularization and more robust language representations. We substantiate this claim with a new experiment, added into Appendix A.3.4 (see **new Table 15** and related discussion there).

----
## New Results and Manuscript Edits

We would also like to highlight that we have submitted an updated version of our manuscript, accounting for the reviewers' suggestions and containing additional results (further discussed in our responses to each reviewer).

To facilitate the reviewers' task, we have highlighted all our changes to the manuscript by using a $\color{blue}{\text{\textbf{blue}}}$ font color.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a 3D vision-language Gaussian splatting model for scene understanding. The proposed method includes a semantic indicator, a cross-modal rasterizer, and a camera-view blending augmentation technique. The paper aims to address limitations in existing methods, particularly challenges with translucent and reflective objects, as well as semantic misalignment due to color overfitting.

The reviewers acknowledge the paper's contributions, rigorous experiments, and potential impacts. There are some initial concerns regarding clarify, experiment details, and method novelty, which are all addressed through additional experiments and thorough response. Therefore, AC recommends accepting this paper.