UniGS: Unified Language-Image-3D Pretraining with Gaussian Splatting

Authors: Haoyuan Li, Zhou Yanpeng, Tao Tang, Jifei Song, Yihan Zeng, Michael Kampffmeyer, Hang Xu, Xiaodan Liang
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: multi-modal learning, 3D gaussian splatting

Abstract:
Recent advancements in multi-modal 3D pre-training methods have shown promising efficacy in learning joint representations of text, images, and point clouds. However, adopting point clouds as 3D representation fails to fully capture the intricacies of the 3D world and exhibits a noticeable gap between the discrete points and the dense 2D pixels of images. To tackle this issue, we propose UniGS, integrating 3D Gaussian Splatting (3DGS) into multi-modal pre-training to enhance the 3D representation. We first rely on the 3DGS representation to model the 3D world as a collection of 3D Gaussians with color and opacity, incorporating all the information of the 3D scene while establishing a strong connection with 2D images. Then, to achieve Language-Image-3D pertaining, UniGS starts with a pretrained vision-language model to establish a shared visual and textual space through extensive real-world image-text pairs. Subsequently, UniGS employs a 3D encoder to align the optimized 3DGS with the Language-Image representations to learn unified multi-modal representations. To facilitate the extraction of global explicit 3D features by the 3D encoder and achieve better cross-modal alignment, we additionally introduce a novel Gaussian-Aware Guidance module that guides the learning of fine-grained representations of the 3D domain. Through extensive experiments across the Objaverse, ABO, MVImgNet and SUN RGBD datasets with zero-shot classification, text-driven retrieval and open-world understanding tasks, we demonstrate the effectiveness of UniGS in learning a more general and stronger aligned multi-modal representation. Specifically, UniGS achieves leading results across different 3D tasks with remarkable improvements over previous SOTA, Uni3D, including on zero-shot classification (+9.36%), text-driven retrieval (+4.3%) and open-world understanding (+7.92%).

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730137075390

thread_type:
official_review

<thread_object>
creator: reviewer_sah2

<content>
summary:
This paper argues that point clouds, as 3D representations, cannot fully capture the intricacies of the 3D world. To address this, it proposes the use of 3D Gaussian primitives instead of point clouds. The authors introduce a pretraining strategy to align the 3D Gaussian features with those of a pretrained vision-language model, establishing a shared visual and textual space through extensive real-world image-text pairs. Additionally, they propose a Gaussian-Aware Guidance module that leverages priors from pretrained point cloud encoders to guide the learning of Gaussian features, enhancing 3D understanding. The proposed approach achieves state-of-the-art performance across various challenging datasets.

strengths:
1. Introduction of 3D Gaussian Primitives: It proposes a novel representation using 3D Gaussian primitives, offering a potential alternative to point clouds for improved 3D modeling.
2. Alignment with Vision-Language Models via Pretraining: The pretraining strategy aligns 3D Gaussian features with a pretrained vision-language model, enabling a shared visual-textual space through large-scale real-world image-text pairs.
3. Gaussian-Aware Guidance Module: The proposed module leverages pretrained point cloud encoders to guide the learning process, helping the 3D Gaussian features improve their understanding and representation capabilities.
4. State-of-the-Art Performance: The method demonstrates superior results, achieving state-of-the-art performance across some of the datasets against point cloud-based counterparts.

weaknesses:
This paper has several weaknesses that should be addressed:

1. Lack of Comparison with Related Works: While the authors contend that 3D Gaussian primitives (3DGS) offer a superior 3D representation, the paper does not provide comparisons with other relevant works, such as [1], [2], [3], and [4]. These works utilize Neural Radiance Fields (NeRF), which offer similar advantages for 3D perception tasks, as mentioned in Lines 63-70. Including such comparisons would strengthen the argument.
2. Insufficient Clarity in the Implementation Details: The paper lacks clear explanations regarding the methodology, leaving certain aspects ambiguous. This raises multiple questions that require further clarification, as outlined in Questions 3 and 4.
3. Unsubstantiated Hypothesis: Without addressing the aforementioned issues, the core hypothesis that 3D Gaussian primitives are a better representation than point clouds remains unconvincing. A thorough comparison and clearer implementation details are necessary to support this claim effectively.
4. Improving these weaknesses could strengthen the paper’s claims, which may lead me to consider raising its rating.

questions:
1. Clarification on NeRF-based Methods: Could you clarify why NeRF-based methods were not discussed in this work? Are there any specific disadvantages that prevented you from considering them as alternatives to 3DGS?

2. Evaluation of Computational Time: Have you evaluated the runtime of your method? While 3D Gaussian primitives may offer several advantages, they require an additional optimization process that point clouds do not, as point clouds represent raw data directly from 3D sensors. This suggests that 3DGS involves additional time for optimization, which is not discussed in the paper. Could you provide more details on this aspect?

3. Initialization with Point Clouds: For clarification, did you use sample raw point clouds from meshes to initialize the Gaussian primitives for the experiments (for ABO and Objaverse) presented in your main results? If so, could you explain why, in some datasets (such as ABO in Table 2), the performance of 3DGS falls short compared to point cloud methods?

4. Clarification on Baseline Performance and Experimental Settings: Could you elaborate on how the performance of the baseline models was obtained? Additionally, how does the experimental setting of the results reported in your paper (Table 2, Lines 328-329, showing 38.17% top-1 accuracy) differ from the Objaverse-LVIS zero-shot classification performance reported in Uni3D[5] (Table 1)? A detailed comparison would clarify any discrepancies.

**References:**

[1] Jeong, et al. (2022). Perfception: Perception using radiance fields. Advances in Neural Information Processing Systems, 35, 26105-26121.

[2] Hu, et al. (2023). Nerf-rpn: A general framework for object detection in nerfs. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,* (pp. 23528-23538)

[3] Li, et al. (2024). GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 21708-21718).

[4] Ballerini, et al. (2024). Connecting NeRFs Images and Text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop (pp. 866-876).

[5] Zhou, et al. (2024). Uni3D: Exploring Unified 3D Representation at Scale. In The Twelfth International Conference on Learning Representations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730638428566

thread_type:
official_review

<thread_object>
creator: reviewer_svju

<content>
summary:
This paper introduces a system for computing a joint embedding of images, text and 3D models (similar to an extension to CLIP). The paper proposes to use gaussian splats as the 3D representation instead of using point clouds to represent 3D shapes.

strengths:
Building joint representations across different modalities is an interesting direction of work. Including 3D shapes can have lots of applications (which would be good to discuss in the paper).

weaknesses:
I miss clear experiments that help understand the benefit of using 3D gaussian splats as the representation of 3D shapes instead of using point clouds. For instance, what numbers do you get in table 3 if you use the architecture used in UniGS but replacing the GS with point clouds?

Can you provide some scenarios in which it is useful to have the representation you propose?

The paper has no visual results. There are only tables with numbers. But it is hard to get an intuition of what the numbers mean and what the results look like just by reading those tables. It would be helpful to show some visual illustrations. You could show examples if 3D retrieval when the query is a 2D image.

questions:
1. Caption in figure 2 could provide some details to help interpret the system sketch. Figure 1, figure 2 and figure 3 seem redundant. It would be clearer if they were integrated into a more comprehensive system description.

2. Section 3.1 is not very helpful, presents standard material and it is not particularly clear. Section 3.2, which introduce the architecture proposed in the paper, is short.

3. Some typos:
Line 32: “to achieve Language-Image-3D pertaining,” -> pretraining?
Line 239: “for understanding the relationships between global position and feature.” What feature?
Line 253: “We donate the process”, -> denote. 

4. Section 3.4 is not very clear.  It will be helpful if figure 3 had the same notation than the one used in section 3.4. For instance, where is f_{fun}, f_{adv}? Then, in line 257 the notation changes again introducing f’_{fun} and f’_{adv}.  Why do you add the single quote ‘?

5. It would be helpful to add figures with some visual examples and comparisons across different methods. How helpful is to use GS versus point clouds?

6. The experimental results section has a lot of tables, but they are not clearly described. For instance, section 4.3 only briefly describes table 5, but it is not clear what the table shows. If this is not central to the paper, this table and section could be moved to the supplementary material section and the extra space can be used to better describe the other experiment and add a figure with visual results.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730666595290

thread_type:
official_review

<thread_object>
creator: reviewer_fg23

<content>
summary:
UniGS introduces 3D Gaussian Splatting (3DGS) into multi-modal pre-training to address the limitations of point clouds in capturing the full complexity of 3D scenes. By representing the 3D world as collections of colored, opaque Gaussians, UniGS bridges the gap between discrete 3D points and dense 2D images. Starting from a pre-trained vision-language model, UniGS aligns 3DGS with text and image representations, creating a unified multi-modal representation. A Gaussian-Aware Guidance module further enhances fine-grained 3D feature learning and cross-modal alignment. Tested on Objaverse, ABO, MVImgNet, and SUN RGBD datasets, UniGS shows substantial improvements over the previous state-of-the-art Uni3D.

strengths:
1. The improvement gain is significant compared with previous methods, which shows the effectiveness of using 3DGS as the unified 3D representation.

2. The paper introduces an innovative Gaussian-Aware Guidance module that utilizes priors from pre-trained point cloud encoders as an initialization to enhance the learning of 3DGS features. This design is effective since it doesn't require training from scratch but can make use of existing models from a different 3D representation.

3. The intuition to use 3DGS as the unified 3D representation is novel and reasonable. To the best of my knowledge, it seems to be the first paper to use 3DGS as the unified 3D representation to bridge multimodal.

weaknesses:
1. Figure 2 could provide an overall description of the information flow (like how this pipeline works in general) in the caption. Also, the figure could be improved by adding some diagrams to represent downstream tasks instead of using text only.

2. I think one significant weakness of using 3DGS as a unified 3D representation is that, usually raw data doesn't use this representation, like point cloud from a Lidar sensor. In this way, this method needs to optimize or process a 3DGS using those raw data (let me know if I understand incorrectly), then leverage this unified 3D representation to conduct downstream tasks. It could be computationally expensive. It seems that this paper does not provide the computation cost for inference, I would appreciate it if authors could include this and discuss this potential weakness.

questions:
1. L56-L60, the paper discusses the weakness of using point clouds as a 3D presentation due to the discrete representation. How about other potential 3D representations? Like depth maps or NeRF, which do not suffer from the discrete representation. I think some discussion among all potential 3D representations will help to justify the choice of using 3DGS as the unified 3D representation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730772480948

thread_type:
official_review

<thread_object>
creator: reviewer_yxjz

<content>
summary:
This paper presents a text-image-3D pre-training framework that leverages 3DGS as the 3D representation for multi-modal representation. Experiments on various datasets are conducted.

strengths:
- The proposed method has good performance on various multi-modal datasets.
- The experimental results are given on two different datasets COCO and VisDrone.

weaknesses:
- Uni3D has used one model to unify the 3D representations from different models, which can be used to align with image and text. What is the main advantage of proposed method using 3DGS.
- The proposed method introduces 3DGS for feature experimentation. Does it increase computational cost.
- It seems that most experiment are not inconsistent with the results in Uni3D. In this paper, the performance are relatively poor. What is the difference.
- I think that it is better to use the similar settings as Uni3D for experiments.

questions:
I suggest that the authors provide more experiments and illustrations for the questions in weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1731087596871

thread_type:
official_review

<thread_object>
creator: reviewer_agij

<content>
summary:
This paper presents a novel multi-modal pre-training method, UniGS, designed to achieve more general and efficient joint representations of text, images, and point clouds. The innovative introduction of 3D Gaussian Splatting (3DGS) and the proposed Gaussian-Aware Guidance module achieve leading results in different 3D tasks. The experiments are solid and can confirm the effectiveness of the proposed method.

strengths:
The proposed approach can achieve state-of-the-art performance on various challenging datasets, which demonstrates the effectiveness in learning strong cross-model representations.

weaknesses:
1）No detailed explanation was given on how negative samples were selected. Do different tasks need to adjust the negative sample strategy?
2）There are significant differences in the structural and spatial characterization of 3DGS and traditional point cloud data. Does it affect the final performance？

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732286911127

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response – Thanks to All Reviewers for Constructive and Insightful Feedback

comment:
We would like to thank all reviewers for their positive affirmations on the novelty and potential impact of this paper, which leverages 3DGS as the 3D representation for learning a more general and stronger multi-modal representation and proposes a novel Gaussian-Aware Guidance module to leverage priors from pre-trained point cloud encoders for better 3D understanding.

Reviewer agiJ found the innovative introduction of 3D Gaussian Splatting and the effectiveness of the proposed Gaussian-Aware Guidance module in different 3D tasks, emphasizing our work is solid and credible. Reviewer yxjz recognized the effectiveness of our proposed framework of text-image-3D pre-training with 3DGS representation. Reviewer fg23 noted that our framework proposes a unique Gaussian-Aware Guidance module for improving 3D comprehension and achieving state-of-the-art performance, emphasizing the intuition and novelty of the article. Reviewer SvjU appreciated the introduction of our approach for aligning 3DGS representation, which is potentially significant for future real-world representation learning. Reviewer SAh2 highlighted that our paper introduces 3D Gaussian Primitives for alignment with Vision-Language Models via Pretraining, achieving state-of-the-art performance with the novel Gaussian-Aware Guidance module.

We believe that we have been able to thoroughly address the Reviewers’ comments by clarifying certain sections of the paper and incorporating additional experiments. Details on these changes can be found in the response to individual comments by the Reviewers. 

Specifically, to address the concerns raised by Reviewer yxJz, we have

1. Included more comparisons of computational cost on FLOPs and runtime.  
2. Included more ablation study results on the computational cost of UniGS.  
3. Included more evaluation results to understand the same setting with Uni3D.  

Additionally, we will open-source all the code and the prepared large-scale datasets to contribute to further research in this area.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The manuscript received positive ratings (6, 6, 6, 6, and 6). Reviewers appreciated the intuition to use 3DGS as the unified 3D representation, the design of the Gaussian-Aware Guidance module, and the performance improvements obtained by the proposed approach on different datasets. Reviewers also raised several concerns in the initial review including, the computational cost, more explanations with respect to results reported in Uni3D, additional ablation regarding using the architecture employed in UniGS but replacing the GS with point clouds, additional comparisons with NeRF-based approaches, and more visual results. Authors provided a rebuttal to address the concerns of the reviewers including, additional details and results regarding negative sample strategy, computational cost comparison, comparisons with NeRF- and Depth-based approaches, additional ablation analysis, and clarification on visual results. Reviewers expressed that most of their concerns are addressed in the rebuttal and remained positive about the manuscript. Given the reviewers comments, rebuttal and discussions, the recommendation is accept. Authors are strongly encouraged to take into consideration reviewers feedback when preparing the revised manuscript.