UniGS: Unified Language-Image-3D Pretraining with Gaussian Splatting
Authors: Haoyuan Li
Zhou Yanpeng
Tao Tang
Jifei Song
Yihan Zeng
Michael Kampffmeyer
Hang Xu
Xiaodan Liang
Venue: ICLR 2025 Poster (2025)
Keywords: multi-modal learning
3D gaussian splatting
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Recent advancements in multi-modal 3D pre-training methods have shown promising efficacy in learning joint representations of text, images, and point clouds. However, adopting point clouds as 3D representation fails to fully capture the intricacies of the 3D world and exhibits a noticeable gap between the discrete points and the dense 2D pixels of images. To tackle this issue, we propose UniGS, integrating 3D Gaussian Splatting (3DGS) into multi-modal pre-training to enhance the 3D representation. We first rely on the 3DGS representation to model the 3D world as a collection of 3D Gaussians with color and opacity, incorporating all the information of the 3D scene while establishing a strong connection with 2D images. Then, to achieve Language-Image-3D pertaining, UniGS starts with a pretrained vision-language model to establish a shared visual and textual space through extensive real-world image-text pairs. Subsequently, UniGS employs a 3D encoder to align the optimized 3DGS with the Language-Image representations to learn unified multi-modal representations. To facilitate the extraction of global explicit 3D features by the 3D encoder and achieve better cross-modal alignment, we additionally introduce a novel Gaussian-Aware Guidance module that guides the learning of fine-grained representations of the 3D domain. Through extensive experiments across the Objaverse, ABO, MVImgNet and SUN RGBD datasets with zero-shot classification, text-driven retrieval and open-world understanding tasks, we demonstrate the effectiveness of UniGS in learning a more general and stronger aligned multi-modal representation. Specifically, UniGS achieves leading results across different 3D tasks with remarkable improvements over previous SOTA, Uni3D, including on zero-shot classification (+9.36%), text-driven retrieval (+4.3%) and open-world understanding (+7.92%).

Published as a conference paper at ICLR 2025
UNIGS: UNIFIED LANGUAGE-IMAGE-D PRETRAIN- ING WITH GAUSSIAN SPLATTING
Haoyuan Li Yanpeng Zhou", Tao Tang', Jifei Song', Yihan Zeng",
Michael Kampffmeyer" Hang Xu², Xiaodan Liang1.4.5
Shenzhen campus of Sun Yat-sen University, Huuawei Noah's Ark Lab, UIT The Arctic University of Norway, *Peng Cheng Laboratory. *Guangdong Key Laboratory of Big Data Analysis and Processing
https://githab.contn com/Li yuan/UniGS
Point Clouds vs 3DGS
UnIGS (Text-Image-3D Pretraining)
Performance
Text
Image
3DGS
Oparmedi
brown toy with yellow cloths
Text encoder
Image encoder
3DGS
bual
Tra-cot
Point Clouds
3DGS
Alignment
LM05
Figure 1: Left: information gap in different 3D representations. Middle: our UniGS, novel unified text-image-33 pre-training framework, leverages 3DGS as the 3D representation. Right: our UniGS learns more general and stronger multi-modal representation.
ABSTRACT
Recent advancements in multi-modal 3D pre- pre-training methods have shown promis- ing efficacy in learning joint representations of text, images, and point clouds. However, adopting point clouds as 3D representation fails to fully capture the intricacies of the 3D world and exhibits noticeable gap between the discrete points and the dense 2D pixels of images. To tackle this issue, we propose UniGS, integrating 3D Gaussian Splatting (3DGS) into multi-modal pre-training enhance the 3D representation. We first rely on the 3DGS representation to model the 3D world as collection of 3D Gaussians with color and opacity, incorporating all the information of the 3D scene while establishing a strong connection with 2D Language-Image-31
images. Then, achieve Language- Image-3D pertaining. UniGS starts with a pre-trained vision-language model to establish shared visual and textual space through extensive real-world image-text pairs. Subsequently, UniGS employs 3D encoder to align the optimized 3DGS with the Language-Ing Image representations learn unified multi-modal representations, To facilitate the extraction of global explicit 3D features by the 3D encoder and achieve better cross-modal alignment, we additionally introduce a novel Gaussian-Aue Aware Guidance module that guides the learning of fine-grained representations of the 3D domain. Through extensive experiments across the Objaverse, ABO, MV MVImgNet and SUN RGBD datasets with zero-shot c classification, text-driven retrieval and open-world world understanding tasks, we demonstrate the effectiveness of UniGS learning a more general and stronger aligned multi-modal representation. Specifically, UniGS achieves leading results across different 3D) tasks with remarkable improvements over previous SOTA, Uni3D, including on zero-shot classification (+9.36%) text-driven retrieval (+4.3%) and open-world understanding (+7.92%).
"Work done as an intern at Huawei Noah's Ark Lab.
Corresponning author.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
The remarkable success of 2D Image-Text pre-training through modality alignment via contrastive
learning (Radford et al. 2021 Sun et al.] 2023: Fang et al. 2023] Schuhmann et al. [022]] [ et al.
2020 Changpinyo al. 2022 Hong al. [021] has recently inspired a line of work pursuing 3D
pre-training (Xue ct al., 2023a
Zeng et al. 2023 Zhou et al. 2024] Liu et al. 2024 Zhang et al.
2022, Huang et al. 2023 Afham et al. 2022). Leveraging diverse large-scal scale 3D datasets, recent
works such as (Liu et al. 2024 Zhou et al. 2024) expand the traditional 2D task to text-image-3D
pertaining by including point clouds as 3D representations, resulting in considerable improvements in 3D zero-shot/open-wor world object detection, classification and retrieval tasks. While point clouds serve as natural step towards 3D representations, there are inherent limitations when using them to represent 3D objects. In particular, as illustrated in Fig, [ point clouds consist of discrete set of points and thus struggle to accurately capture fine- toee-griied geometric details and surface textures of common objects, limiting the performance of 3ID representation learning approaches. Moreover, there exists a noticeable gap between the discrete points and the dense 2D pixels of images, which further hinders the learning of joint multi-modal representations.
On the other hand, 3D Gaussian Splatting (3DGS) (Kerbl et al. 2023) has recently revolutionized 3D scene representations, and offers promising and more efficient alternative facilitate 3D representation learning. Specifically, 3DGS models scenes as a set of 3D Gaussians, which effectively reconstruct the 3D target object as well as provide efficient correspondence between 3D and 2D images through the splatting rendering algorithm. Furthermore, 3DGS offers the advantage of utilizing a more diverse range of data sources as can leverage multi-view images or COLMAP data (Yu et al. 2023) Schonberger & Frahm, (016)) for optimization with minimal overhead and collection time. Additionally, existing point cloud datasets can be used as the initialization for 3D Gaussian locations, further enhancing the capabilities of 3DGS. However, simply retraining existing multi-modal frameworks like CLIP (Zeng et al., 2023) or Uni3D (Zhou et al. 2024) to leverage 3DGS as the 3D representation is not effective. As the spatial connections of 3DGS may be insufficient to capture and express objects due to the fact that 3DGS are not necessarily distributed on the surface of objects.
To this end, we propose UniGS, which leverages 3DGS as the 3D representation for unified language- image-3D pre-training and enhances the performance of 3D understanding. To better model and understand the explicit features of 3DGS, UniGS additionally proposes a novel Gaussian- Guidance module. Specifically, UniGS utilizes a parallel-structure VIT as the 3D encoder con- sisting of a fundamental encoder and advanced encoder, where the fundamental encoder encodes spatial information together with color and the advanced encoder the spatial information with the remaining 3D Gaussian attributes, where pre-trained models can be leveraged for initialization of the fundamental encoder. With priors extracted from the fundamental encoder, the advanced encoder aggregates priors through cross attentiot layers for guiding the 3DGS feature learning, unlocking the superior performance that comes with leveraging 3DGS. Through extensive experiments across the Objaverse (Deitke et al., 2023), ABO (Collins et al. 2022). MVImgNet (Yu et al. 2023) and SUN RGBD (Song et al., 2015) datasets and various tasks, we demonstrate the effectiveness of UniGS in learning more general and stronger multi- molti-modal representation. Specifically, UniGS achieves state-of-the--at results across different 3D tasks with remarkable improvements, including zero-shot classification (+9.36%) text-driven retrieval (+4.3%), and open-world understanding (+7.92%). Our contributions can be summarized as follows:
We propose UniGS, a novel unified text-image-3D pre-training framework, which leverages 3DGS as the 3D representation for learning more general and stronger multi-modal representation.
We propose novel Gaussian-Aware Guidance module to leverage priors from pre-trained point clouds encoders to guide the learning of the Gaussian features for better 3D under- standing.
Our proposed approach achieves state-of-the- performance on various challenging datasets, demonstrating the effectiveness in learning strong cross-model representations.
2

---

Published as a conference paper at ICLR 2025
Multi view Datasets
3DGS envoder
Vanilla optimization
Zero-shot classification
imagee
3DGS
Few-shot classification
Datnects with Point clouds
Alignment
Location- optimization
Image uncoder
Text-driven retrieval
SUN RGBD
Point cloud
image
image-driven retrieval
Images
3D Darasets with mesh
Socne recognition
Locitioniiittaalined optimization
etc.
ABD
encoder
Mesh point
images
Text
Figure 2: The overview of UniGS. UniGS is an innovative, unified, and scalable 3D pretraining framework designed for 3D representation learning. It offers versatile pipelines for various datasets, enabling the efficient 3DGS acquisition enhance 3D representation learning with SoTA CLIP models. UniGS demonstrates exceptional performance across a broad range of benchmarks.
2 RELATED WORK
Multi-modal pretraining via contrastive learning. Leveraging multi-modal data to pre-tras pre-train modality-specific encoders via contrastive alignment has received considerable attention in recent years (Radford et al. 2021; Mu et al., 2022) due to its ability to leverage massive in-the-wild datasets of paired data. While early works have largely been focused on image-text data, promoting research
on text-based image manipulation (Patashnik et al. (021). open vocabulary object detection (Gu
ctt al.
202I Gao et al. 2022), language grounding (Li ct al 2022) and zero-shot segmentation (Xu
et al.
2023 there has been increasing focus on learning 3D representations lately (Xue al.
2023a bj Zeng et al.] 2023 Zhou et al.] 2024). These approaches largely follow the contrastive
learning paradigm by adding a new 3D representation encoder and aligning it with the 2D and/or text modalities. However, due to the difficulty of collecting and constructing 3D representation data, current approaches leverage point clouds. More specially, PointCLIP (Zhang al., 2022) and CLIP2Point (Huang et al.) (023) extract depth maps from point clouds to obtain image- like data that can be leveraged for contrastive pretraining. Since these depth maps lose plenty of spatial information of the original point cloud data structure, CLIP² (Zeng et al. (2023) instead learns 3D encoder directly on point clouds, demonstrating robustness in real-world indoor and outdoor scenarios. However, the flexibility and simplicity of point clouds come at cost as specific shape and texture information of the object surface is lost, leading to ambiguities. In this work, we, therefore, adopt 3DGS (Kerbl et al. 2023) to replace point clouds as the 3D representation alleviate this loss of information.
Zero-shot/Open-wee cloud into 3D voxels (Shi et al.] 2020 Maturana & Scherer) 2015) Qi et al., 2017a) and extract features that can be associated with semantic category information. PointNet++ (Qi et al. 2017b)
Learning in 3D. Considerable progress has been made to project a point
proposes a hierarchical neural network to extract local features with increasing contextual scales
and PointMLP (Ma et al. 2022) proposes a pure residual MLP network while achieving competitive results. More recently, sell selp-super learning YYu et al., 2022] Pang et al.) 2022) and unsupervised
learning (Atham et al. 2022. Liang et al. 2021 et al.) 2022) approaches for 3D understanding
methods have also shown promising performance. However, while the current supervision-bass
are restricted by the annotated training datasets and show poor performance on unseen categories, methods can not directly be transferred to zero-shot tasks
self-supervised and unsupervised-baase with open-world vocabularies due to the limited downstream annotations. Therefore, we construct a language-image-3D dataset for pretraining to learn transferable 3D representations that are aligned to an open-vocabulary language space to facilitate zero-shot transfer.
METHODOLOGY
In this section, we introduce our proposed UniGS in detail. We first review background information of 3DGS and next provide overview of UniGS in Section 3.1 The proposed cross cross-modal contrastive learning framework for multi-modas alignment is then presented in Section berfe we introduce
the details of the Gaussian-Awan Guidance in Section 3.3 We further present details on scaling up
3

---

Published as a conference paper at ICLR 2025
and initializing UniGS in Section 3.4 Finally, we present how ensemble 3DGS datasets from existing datasets in Section 3.5
3.1 PRELIMINARIES AND OVERVIEW
3D Gaussian splatting. 3D Gaussian splatting (Kerbl et al., 2023) proposed an explicit model design and efficient differentiable rendering implementation, which enable faster training and high- quality performance. 3DGS employs a collection anisotropic Gaussian primitives. denoted as
G = {g1. 92. .... 9N J. to represent the scene. Each 3D Gaussian sphere g can be parameterized by the following attributes: (1) located at 31) position μa E 'R (2) with color defined by SH (3+c E R*), (3) opacity a E [0, 1] and 3D shape decomposed into (4) a scaling factor E R and (5) rotation quaternion E R2, where &o denotes the freedom of SH basis. Clearly, Gaussian sphere can be characterized by a high-dimensional feature representation concatenating its position μs, color c, opacity a, scaling factor s, and rotation R. where
g (H,Ga,s
(1)
In the rendering process, splatting pipeline is utilized, projecting 3D Gaussians onto the 2D image plane. This projection splats the 3D Gaussians into 2D counterparts on the image plane, which are then blended using the a- a-blending algorithm to determine the final color composition.
C= T= ca: II a.), RSS R IE
where a denotes the color defined by spherical harmonics (SH) (Kerbl al., 2023) coefficients of each 3D Gaussian, R and S are the matrix representation of It and 8, calculated by the multiplication of 2D Gaussian with covariance Σ and a learned per-point opacity Yu et al. (021).
Overview of UniGS. UniGS facilitates 3D multi-modal representation learning by leveraging the informative and effective 3DGS representation and supports open-world learning. The overview of UniGS is depicted in Fig. [ With Image Encoder and Text Encoder from pre-trained Text-image aligned model (Radford et al. 2021) for a shared latent space, UniGS formulates a parallel-structue dual-branch encoder, which adopts frozen pre-trained point cloud encoder the fundamental encoder for priors and leverages another encoder as the advanced encoder modeling high-level information.
3.2 CROSS- REPRESENTATION ALIGNMENT
To align multi-modal representations of text, image, and 3D domains, UniGS adopts the pre- pre-trained language-iman image model CLIP (Rafford et al. 202T) to provide a common language-iman -imggee--age latent space that serves the target latent space align 3DGS representation (see Fig. 2 To facilitate recognition, the text and image encoders the CLIP model which defines the common latent space are frozen. In particular, we take inspiration from the contrastive loss in (Radford et al. 2021 I and propose Language-3DGS and Image-3DGS Alignment losses to bridge the domain gap among the different modalities.
the transferability of the learned representations and enable zero-shot/open-wor
Zeng et al., (023)
Language-3DGS Alignment. Given a text-image- 3DGS triplet, {Xr, X,, text features, f" E - image features, f' e CRC and 3DGS features, f E C can be obtained through the corresponding modality encoders. The contrastive loss between the text and 3D modality is then utilized to align the text and 3DGS feature representations. Let N denotes the batch size and r the temperature coefficient, the Language-3DGS Alignment training objective L(T, G) can be described as:
(TT = Σ C(i, N SEN
1
1 N
log exp(S! 1 IEN
exp(St (1/TT explS! "Sp(S(-1917)) 1917)
(3)
Σ
4

---

Published as a conference paper at ICLR 2025
3D Encoder
3D feature f
Arontrrsiie
MLP
Text feature f
Image f
frun
fadt
Cross
guidanes
Alln
Uni3a
Encuder
f
CLIP
CrossAm
Image
CNN layers
layers
froe
(u.c)
Funu
(,
Group Divider
Group Divider
Text
Image
Fundamental
Advanced encoder
(1L
(b)
(R.0,..8)
Figure 3: Model overview of UniGS. Let C, a, R denote the location, color, opacity, scale, and rotation attribute of 3DGS. (a) Given a 3DGS input, the pre-trained and frozen branch takes 3DGS locations and color as input while the second branch, which initialized from scratch, focuses on the 3DGS location and the remaining attributes. (b) shows the details of our 3D Encoder and how the prior is leveraged through cross-attention layers.
Image-3DGS Alignment. Similarly, we apply the contrastive loss to align the image and 3DGS features. The Image-3DGS Alignment objective L(I, G) defined as:
1 C(..IG)) G)I,G) N
Σ log iEN
exp(F! 111 exp(f! f//T)+ E exp(f! -f/T)) (4) JENJYi
N
NN
Following (Zeng al., 2023), the final cross- contrastive learning objective LcM (T, I, G) can be obtained by combining the text-3DGS and image-3DGS alignment objective, namely, L(T, G) and G))
LCM LCM = +2L(LT
(5)
where both hyper-parameters Å₁ and d2. are set 0.5.
3.3 GAUSSIAN- GUIDANCE
Admittedly, projecting point clouds into 3D voxels with 3D backbone (Qi et al. 2017a b) can be helpful for understanding the relationships hetween global position and feature derived from non-positional information. However, we observe that the explicit feature of 3DGS will be ignored as the voxelization procedure loses the shape and texture information. To address this issue, Transformer-basee model pattern is adopted for feature-context learning and model scalability.
For better modeling and understanding of the 3D domain represented by 3DGS, we further propose a Gaussian aware Guidance module. Specifically, this module, as highlighted in the dashed box in Fig. 34. formulates Auualbranch 3D encoder, where the fundamental encoder Efun leverages the 3D VTT Encoder pre-trained on 3D point clouds from (Zhou et al.) 2024) to model the low-level features ffun including spatial and color information of 3DGS, while the advanced encoder Fade additionally models the high-level features fade. the relationship between spatial connections and 3DGS feature. As the number of Gaussian spheres allocated to an object or scene represented by 3DGS varies, a group divider is leveraged to process the input 3DGS into fixed number of Gaussian spheres. The CNN layers map low- oww-dimensional raw features consistent with UniDD. Moreover, cross
high-dimensiona. feature space, cross-attentin attention layers are built to extract guidance of embeddings from the fundamental encoder and improve the alignment of the 3DGS features with other domains. We denote the process as CA,, where
Q KT adv V
CA = softmar
dds.
Finally, concatenate the features of ffan and fadto respectively generated by the fundamental encoder and the advanced encoder, and then map it through an MLP to the same dimension as the pre-trained image-text model. Let SA be the acronym of self-attention and fo(-) denote the process
5

---

Published as a conference paper at ICLR 2025
of group divider and CNN layers. The process of encoding 3DGS features based on Gaussian-Aware Guidance can then be represented by:
fuu = (1,, Jado = (11, (,, 8,
(7)
fljm SA(folffan)) Saaoo CA((o(fado)) f = M MM Saau))
(8)
3.4 SCALING UP 3DGS BACKBONES
Scaling Up UniGS. Previous works have achieved good performance by designing specific model architectures and shown case effectiveness in various applications. However, these methods are either limited to certain small-scale dataset data sources like point clouds which are expensive to collect. Instead, with recent advances in large-scare multi-view datasets (Yu et al. (023). our method adopts 3DGS representations and designs ViT-based Encoder to encode the 3D modality. Therefore, our model can naturally solve the difficulties by simply scaling up the data and model size with well-studied unified scaling-up strategies.
Initializing UniGS. Restricted by the scale of the 3D dataset, directly pre-training each 3D backbone for specific 3D tasks leads expensive training costs and may suffer from difficulties in convergence
or overfitting. To overcome this issue, we follow Uni3D (Zhou et al. (2024) and adopt off-the-shelf
pre-trained large models (Sun et al. 2023 Fang et al., 2023 Radford et al. [202]] Caron al. 2021) with ViT-based structure in the other modalities as the initialization of the 3D backbone to transfer pre-training Different from
their rich underlying representational abilities to anguage-image--
Uni3D (Zhou al.] 2024), UniGS further establishes stable cross-modal contrastive learning through the Gaussian Aware Guidance module, which introduces a new perspective of leveraging pre-trained priors for stabilizing the learning of large-scale 3D representations, and provide guidance spatial information for understanding and aligning 3DGS representations with other modalities.
3.5 ENSEMBLING 3DGS DATASETS
To create 3DGS-text tmxteimage triplets for training, we over-sample points from the mesh surface uniformly to capture the details of each object. We random sample N point clouds to initialize the 3D Gaussians, where N is set to 1024 in Objaverse, ABO, and MVImgNet, Next, we use rendered images for 3DGS optimization. More specifically, for ABO (Collins et al. 2022). we uniformly render 72 images covering the whole shape, while obtain the rendered image from (Liu et al. (023) for the Objaverse dataset. Finally, we collect and clean the caption for each dataset. We clean the given caption of ABO and sort ABO objects into 23 classes with 7929 items, while nluan--verfiied and machine- machine-generated generated high-quality captions from (Luo et al., 2024, Dong et al.) (024) are utilized for Objaverse. During the optimization of 3DGS, we control the number of 3D Gaussians by adjusting the 3DGS optimization scheme. In particular, after each 3DGS duplicating and pruning step, we sort the 3D Gaussians by opacity and only keep the top N. To facilitate better optimization results for the SUN RGB-D dataset, we do not restrict N to 1024 as it is under a sparser setting with a single image provided for each scene.
4 EXPERIMENT
In this section, we evaluate the proposed UniGS on a range of 3D understanding tasks, highlighting its ability to learn more expressive and informative 3D representations than prior approaches. In particular, we consider the tasks of object retrieval, zero-shot classification, and scene understanding. We further conduct detailed and comprehensive ablation studies to reveal the impact and power of design for cross modal learning.
4.1 EXPERIMENTAL SETUP
Baseline. We compare UniGS with ZZng et al. 2023 Zhou al.) 2024: Zhang al.) 2024 Qi et al. 2023 2024). To further understand how the 3DGS features, the pre-trained weights, and the parallel structure impact 3D representation learning, we retrain the most relevant model (Uni3D) with different settings for a fair comparison. As for the baseline model, we further report the performance of the original Uni3D model trained with point clouds and the altered version trained with the 3D location attributes (mean value of Gaussians) of 3DGS instead of point clouds for completeness. Note that 3DGS does not necessarily exist on the surface of objects, so there a certain difference between point clouds and the 3D location attributes of 3DGS.

---

Published as a conference paper at ICLR 2025
Avg. Top5
Methods
Representation
Text-image Model
Embedding dim
Top1
Top 10
Objaverse (3D dataset with mesh)
Uni3D
8,400
17.50
22.60
point cloud
EYA02-E-14-plus
1024
CLIP2
7.400
22.20
32.50
point cloud
ViT-B-16
512
CLIP
6.400
20.20
30.90
3DGS
ViT-B-16
512
Uni3D
2.300
8.100
12.00
3DGS location
EVA02-E-14-plus
1024
Uni3D*
16.70
37.10
48.10
point cloud
ViT-B-16
512
Uni3D
10.40
26.20
36.40
3DGS
ViT-B-16
512
Uni3D*
15.80
35.60
47.20
3DGS
ViT-B-16
512
UnirS(Ours)
21.00
39.80
53.50
3DGS
ViT-B-16
512
ABO (3D dataset with mesh)
Uni3D
5.700
19.19
29.49
point cloud
EVA02-E-14-plus
1024
CLIP²
7.090
24.34
38.94
point cloud
ViT-B-16
512
CLIP²
7.650
23.92
37.83
3DGS
ViT-B-16
512
Uni3D
1.670
6,400
11.27
3DGS location
EVA02-E-14-plle
1024
Uni3D*
10.29
29.21
43.67
point cloud
ViT-B-16
512
Uni3D
8.070
25.31
37.41
3DGS
ViT-B-16
512
Uni3D*
10.85
29.76
42.98
3DGS
ViT-B-16
512
UniGS(Ours)
11.27
30.32
43.95
3DGS
ViT-B-16
512
MVImgNet (Multi-view dataset)
CLIP
9.560
31.00
42.19
point cloud
ViT-B-16
512
CLIP
12.12
31.93
45,45
3DGS
ViT-B-16
512
Uni3D
1.400
7.690
11.42
3DGS location
EVA02-E-14-plus
1024
*ni3D*
17.72
48.95
62.24
point cloud
ViT-B-16
512
Uni3D
9.090
29.14
39.63
3DGS
ViT-B-16
512
Uni3D*
17.02
47.09
60.61
3DGS
ViT-B-16
512
UniGS(Ours)
18.65
53.38
66.90
3DGS
ViT-B-16
512
Table 1: Top1, Top5 and Top10 Text-3D retrieval accuracy. Avg.: the mean average retrieval accuracy. denotes training from scratch.
Implementation Details. Following Section 3.5 we collect 146000, 7929, 3483, and 61871 objects, optimized for Objaverse (including Objaverse-LVIS for evaluation only ), ABO, M MVImgNet, and SUN RGBD datasets, respectively. For the retrieval task, we randomly sample 1000 items to form the test set, and use the rest as training set. We train UniGS with learning rate of 1e-4 for 15 epochs for the retrieval task and 50 epochs for the zero-shot classification and scene recognition tasks.
4.2 COMPARISONS TO STATE-OF-THE-ARY
To demonstrate the effectiveness of our proposed method, we evaluate UniGS on the Text-3D retrieval, zero-shot classification, and scene understanding tasks and make comparisons with state-of-the art methods (Zeng et al. 2023 Zhou et al.) 2024). We further report results of Uni3D (Zhou et al.. 2024) trained with 3DGS for completeness. Note that CLIP2 (Zeng et al. 2023) is retrained on our collected dataset for fair comparisons.
Text-3D retrieval. With the learned multi-modal representations of UniGS, we naturally retrieve 3D shapes from the query text or images. Here, we focus on Text-driven retrieval, due to its importance for 3D asset search and downstream applications. Specifically, from the test set by calculating the cosine similarity between the embedding of the query text prompt and 3D shapes in the gallery. We report Top1, Top5, and Top10 accuracy.
retrieve 3D shapes
As shown in Table I UniGS outperforms the current state-of-the-ae approaches across all datasets and improves the Top 1 retrieval accuracy of CLIP2 and Uni3D on the Objaverse dataset by
7

---

Published as a conference paper at ICLR 2025
Avg. Top3
Methods
Representation
Text-image Model
Embedding dim
Top1
Top 5
Objaverse-Lvis (3D dataset with mesh)
Uni3D
38.17
59.92
67.18
point cloud
EVA02-E-14-plu
1024
CLII22
12.35
24.62
32.91
point cloud
ViT-B-16
512
CLIP
10.20
20.47
27.71
3DGS
ViT-B-16
512
Uni3D
5.130
11.20
13.27
3DGS location
EVA02-E-14-plus
1024
Uni3D*
36.72
57.09
65.18
point cloud
ViT- B-16
512
Uni3D
18.48
34.39
43.31
3DGS
ViT-B-16
512
Uni3D*
30.47
48.46
55.87
3DGS
ViT-B-16
512
UniGS(Ours)
38.57
60.54
68.96
3DGS
ViT-B-16
512
ABO (3D dataset with mesh)
Uni3D
68.94
90.49
94.15
point cloud
EVA02-E-14-plus
1024
CLIP²
22.58
43.83
54.56
point cloud
ViT-B-16
512
CLIP
19.06
38.48
48.71
3DGS
ViT-B-16
512
Uni3D
13.34
28.28
42.20
3DGS location
EVA02-E-14-phi
1024
Uni3D*
37.60
59.68
70.22
point cloud
ViT-B-16
512
Uni3D
27.57
50.60
63.69
3DGS
ViT-B-16
512
Uni3D*
37.79
61.08
69.04
3DGS
VIT-B-16
512
UniGS(Ours)
46.97
69.91
79.38
3DGS
ViT-B-16
512
MVImgNet (Multi-view dataset)
CLIP
5,030
12.33
16.96
point cloud
ViT-B-16
512
CLIP
4.310
11.24
14.89
3DGS
ViT-B-16
512
Uni3D
3.680
10.72
14.92
3DGS location
EVA02-E-14-plus
1024
Uni3D*
5.410
12.51
16.68
point cloud
ViT-B-16
512
Uni3D
7,020
15.18
21.05
3DGS
ViT-B-16
512
Uni3D*
4.920
12.42
15.70
3DGS
ViT-B-16
512
UniGS(Ours)
7.650
16.96
22.48
3DGS
VIT-B-16
512
Table Zero-shot classification. Avg.: the mean average classification accuracy. from scratch.
denotes training
Method
Rep.
Avg.
Bed
Bsf.
Chair
Desk
Sofa
Table
Toilet
Btub.
Dresser
NSd.
Uni3D
point clouds
11.04|
56.74
14.33
19.58 25.03
22.93
15.98
24.62
24.00
0.000
1.690
CLIP
point clouds
41.39
1.840
14.00
68.02
30.98
45.44
7.460
13.85
0.000
15.00
3.390
CLIP
3DGS
28.50
1.470
4.000
40.03
1.640
15.20
56.72
4.620
0.000
26.25
30.51
Uni3D*
point clouds
61.72
63.60
59.67
84.33
47.43
79.36
78.97
63.59
74.67
12.92
18.93
Uni3D
3DGS
54.51
58.09
19.00
80.38
17.05
62.40
47.68
56.92
48.00
7,500
11.02
U
3DGS
56.67
74.63
28.00
83.89
28.36
50.88
54.31
7.690
20.00
27.50
19.49
UniGS(Ours)
3DGS
69.64
81.62
32.00 87.46 17.38 79.36 68.74
93.85
96.00
35.00
36.44
Table Recognition on SUN RGBD (dataset with point clouds), Avg.: the mean average Top1 accuracy across all categories. denotes training from scratch.
13.6% and 5.2%, respectively. For reference also include the evaluation results of Uni3D on their larger dataset (in gray).
Zero-shot classification. We evaluate the zero-shot classification performance of UniGS on Objaverse-Lvis, ABO, and MVImgNet without accessing their training sets. We reorganized ABO into 44 major categories, then skipped those with items fewer than 50, and formed 23 categories ultimately for classification evaluation. Similarly, we reorganized the Objaverse and MV ImgNeet datasets into 318 and 95 categories, respectively.
8

---

Published as a conference paper at ICLR 2025
Avg.
Training
Methods
Source
3D points
Backbone
Representation
Topl
Top3
Top 5
Dataset
Additional zero-shot comparisons
Uni3D
36.72
57.09
65.18
point cloud
Uni3D
30.47
48.46
55.87
3DGS
TAMM
no Lvis
1024
EVA02-S
22.70
38.83
47.13
100k
3DGS
ReCon
23.40
41.41
48.95
3DGS
UniGS(Ours)
38.57
60.57
68.96
3DGS
Table 4: Summary of the experimental results on Objaverse-LVIS zero-shot classification. Avg.:
the mean average classification accuracy. All methods are trained from scratch.
Training
Initialization
Avg. Top3
w/ point cloud
opacity
scale & rotation
Top1
Top
46.97
69.91
79.38
0
0
46.29
68.67
76.79
0.4
0
48.49
70.06
77.55
0.4
0.4
49.62
70.61
77.50
Table 5: Zero-shot classification with point clouds on ABO. Avg. denotes mean average classifica- tion accuracy. Results illustrate that properly converting point clouds into 3DGS format can improve performance.
As shown in Table 2, our UniGS significantly outperforms CLIP² improving performance by over 24% for both the Objaverse and ABO datasets. Due to the lack of real-world data the pre-training dataset, the Top1 accuracy of MVImgNet is relatively low. However, UniGS outperforms all the baselines on all datasets, revealing the power of 3DGS representations and the effectiveness UniGS.
Scene recognition. We leverage the SUN RGBD dataset (Song et al. 2015) the scene data and classify objects into 37 categories following the setting of (Song et al. (015). Since SUN RGBD only provides single image for each scene, load the point clouds the initialization for 3I) Gaussians and fix them during the entire 3DGS optimizations. Thus the 3DGS locations of SUN RGBD are equivalent to point clouds.
As shown in Table 3 our UniGS outperforms both CLIP² and Uni3D, improving performance by 28.25% and 7.92% respectively, and achieving increase of 12.97% over directly modeling 3DGS. Moreover, the success of UniGS in SUN RGBD shows the robustness of 3DGS representation to the number multi-view images.
Further comparisons to additional zero-shot 3D object understanding methods. As shown in the Additional zero-shot comparisons part of Table 4 we supplement extra comparisons to TAMM Zhang et al.) (2024) and ReCon Qi et al. (2023) and UniGS significantly outperform SOTA methods, emphasizing effectiveness of 3DGS representation and proposed Gaussian-aware Guidance.
4.3
GENERALIZATION TO POINT CLOUDS
To show the capability UniGS to process vanilla point clouds, we conduct experiments where we convert point clouds into 3DGS to benefit the performance of modeling spatial information.As shown in Table we do this by initializing the scale, opacity, and rotation with default value and show that by including this data in training, UniGS can achieve better performance than trained using 3DGS only. This highlights UniGS's potential to inherit capabilities of the point cloud encoder and that can directly be applied to 3D point clouds.
4.4 ABLATION STUDY.
Ablation study the proposed modules of UniGS. To further study the power of 3DGS representa- tion and the effectiveness of UniGS, we further ablate and evaluate UniGS in zero-shot classification on ABO. We summarize all the experiments and conduct ablation study whether leverage the 3DGS feature, ViT pattern, Pretrained weight (Pre.), Parallel structure (Par.), and Cross attention (Cro.) between Parallel structures. Therefore, shown in Table
analyze and conclude that

---

Published as a conference paper at ICLR 2025
GAG
ExP.
3DGS
VIT
Avg.
Pre.
Par.
Cro.
Data scaling up
X
X
X
X
X
22.62
Data
10k
50K
100k
V
X
X
X
19.35
Avg.
27.67
43.06
46.97
V
V
X
X
38.38
/
V
V
X
27.43
Model scaling up
V
V
V
37.58
Model
UniGS-T UniGS-S UniGS-L
6
40.57
Avg.
42.67
46.97
52.30
46.97
Table 7: UniGS performance with scaling up. Avg.: the mean aver- age Topl zero-shot classification ac- curacy on ABO.
Table 6: Ablation study on the proposed modules. Avg.: the mean average Top1 accuracy across all categories. GAG denotes our Gaussian-Award Guidance.
Comparisons between Expl., Exp2., and Exp3. show that the convolution- -based model, i.e. PointNet++, can not capture the explicit 3DGS feature while the ViT-based model, i.e. Uni3D, can successfully model feature relationships.
Comparisons between Exp3., Exp4., and Exp7. show that loading pretrained weights has certain advantages, but without careful designs, it may not be possible to utilize the explicit features of 3DGS. On the contrary, training directly from scratch can better learn the feature information of 3DGS, revealing that incorrect model design can hinder subsequent learning due to prior knowledge.
Comparisons between Exp3., Exp4., Exp5., and Exp7. show that dividing the model into parallel structures to process color and other 3DGS features separately improve alignment of 3DGS, but it is not enough to achieve superior performance.
Comparisons between Exp5. and Exp7. show the importance of cross attention to utilize the prior from point cloud encoding to guide 3DGS feature encoding.
Comparisons between Exp5., Exp6., and Exp7. show that the pretrained weights are beneficial for the final performance and that the key to UniGS performance lies in the Gaussian-Awes Aware Guidance that reduced the difficulty of overall 3DGS learning and enhances 3DGS understanding through the cross attentions.
Scaling Up. We next explore the effectiveness of scaling up training data and model type in Table The performance under different data and model scales demonstrates that scaling the training data and model size of UniGS can significantly improve the performance 3D representation learning.
CONCLUSION
In this paper, we proposed UniGS, which for the first time includes 3DGS in cross-modal learning as the universal 3D representation supplementing shape and texture information. To this end, parallel structure with cross-attention is proposed to avoid knowledge conflicts and builds knowledge connections via the attention mechanism. We demonstrate that our proposed UniGS achieves superior performance to state-of-the-an approaches and reveals the power and importance of 3DGS for 3D representation learning.
Despite the robust and effective performance of UniGS for 3D representation learning and downstream applications, its current version lacks performance validation out-door scenarios and 3D understanding ability with Large Language Model (LLM), resulting in insufficient perfor- mance improvement and validation for downstream tasks. Moreover, at least one image with a camera pose is required for the optimization of 3DGS, and how further consider a camera-pose free approach (e.g., Emage-to-3DGS3 or a dataset of pure point clouds while maintaining performance, is another exciting direction for future work.
Linitations:
Acknowledgment: This work is supported by National Key Research and Development Program of China(2024y 2OEE000000320333003 100), National Natural Science Foundation of China (NSFC) under Grants No.62476293 and Nansha Key R&D Program under Grant No.2022ZD014 We thank the General Embodied AI Center of Sun Yat-sen University for support of this work.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Rodrigo, Crosspoint: Sel--supervieed cross-modal contrastive learning for 3d point cloud understanding. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 9902-9912, 2022.
Francesco Ballerini, Pierluigi Zama Ramirez, Roberto Mirabella, Samuele Salti, and Luigi Di Stefano. Con- necting nerfs images and text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 866-876, 2024.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski. and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 9650-9660, 2021.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3558 3568, 2021.
Jasmine Collins, Shubham Goel. Kenan Deng, Achleshwar Luthra. Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas Yago Vicente, Thomas Dideriksen, Himanshu Arora, et al. Abo: Datasct and benchmarks for real-world 3d object understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 21126-22 21136, 2022.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsami, Aniruddha Kembhavi, and Ali Farhadi. Ohjaverse: A universe of annotated 3d objects. In Pruceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13142-13153, 2023.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang. Haodong Duan, Maosong Cao, et al. xcooposse::: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401. 16420, 2024.
Zhiwen Fan, Kevin Wang. Kairun Wen, Zehao Zhu, Dejia Xu, and Zhangyang Wang. Lightgaussian: Unbounded 3d gaussian compression with 15x reduction and 200+ fps. arXiv preprint ar 272:4 2023.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9935-19939, 2023.
Mingfei Gao, Chen Xing. Juan Carlos Niebles, Junnan Li. Ran Xu, Wenhao Liu, and Caiming Xiong. Open vocabulary object detection with pseudo bounding-boo box labels. European Conference on Computer Vision. pp. Springer, 2022.
Xiuye Gu, Tsung Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection vision and language knowledge distillation. arXiv preprint arXiv:2104. 13921, 2021.
Weixiang Hong. Kaixiang Ji, Jiajia Liu, Jian Wang. Jingdong Chen, and Wei Chu. Gilbert: Generative vision-language pre-training for image-text retrieval. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1379-1388, 2021.
Tianyu Huang, Bowen Dong. Yunhan Yang, Xiaoshui Huang, Rynson WH Lau, Wanli Ouyang, and Wangmeng Zuo. Clip2point: Transfer clip to point cloud classification with image-depth pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22157-22167, 2023.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühherr and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 10965-10975. 2022.
Hanxue Liang, Chenhan Jiang, Dapeng Feng, Xin Chen, Hang Xu, Xiaodan Liang, Wei Zhang, Zhenguo Li, and Luc Van Gool. Exploring geometry-awe -ware contrast and clustering harmonization for self-supervised 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3293-33933333 3302,, 2021.
Haotian Liu, Mu Cai, and Yong Jac Lee. Masked discrimination for self-supervised learning on point clouds. In European Conference on Computer Vision, pp. 657-675. Springer, 2022.
11

---

Published as a conference paper at ICLR 2025
Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards open-world understanding. Advances Neural Information Processing Systems, 36, 2024.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. 7ero-1-to-3 Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9298-9309, 2023.
Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pretrained models. Advances in Neural Information Processing Systems, 36, 2024.
Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Fu, Rethinking network design and local geometry in point cloud: simple residual mlp framework. arXiv preprint arXiv:2202.07123, 2022.
Daniel Maturana and Sebastian Scherer. Voxnet: convolutional neural network for real-time object recognition. In 2015 JEEE/RSJ international conference on intelligent rohots and systems (IROS), pp- 922-928. IEEE, 2015.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. In European Conference on Computer Vision, pp. 529-544, Springer, 2022.
Yatian Pang. Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. In European conference on computer vision, pp. 604-621. Springer, 2022.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2085-22850 2094,, 2021.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 652-660, 70.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. Advances neural information processing systems, 30, 201 7b.
Di Qi. Lin Su, Jia Song, Edward Cui. Taroon Bharti, and Arun Sacheti. Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data, arXiv preprint arXiv:2001.07966 2020.
Zekun Qi. Runpei Dong. Guofan Fan, Zheng Ge. Xiangyu Zhang, Kaisheng Ma, and Li Yi. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining. In International Conference on Machine Learning, pp. 28223-28243. PMLR, 2023.
Zekun Qi. Runpei Dong, Shaochen Zhang. Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, and Kaisheng Ma. Shapellm: Universal 3d object understanding for embodied interaction. arXiv preprint arXi::240..17766 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Pierluigi Zama Ramirez, Luca Luigi, Daniele Sirocchi, Adriano Cardace, Riccardo Spezialetti, Francesco Ballerini, Samuele Salti, and Luigi Di Stefano. Deep learning on neural fields. arXiv preprint 2rxiv:2312.13277 2023.
Johannes Schonberger and Jan-Michael Frahm. Structure-froo-mmttion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4104-4113 13, 2016.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, el al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25278 25294, 2022.
Shaoshuai Shi. Chaoxu Guo. Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-rcun: Point-voxel feature abstraction for 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 10529 10538, 2020.
Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgh-d scene understanding benchmark suite. In Proceedings of the IEF. conference on computer vision and pattern recognition, pp. 567-576, 2015.
12

---

Published as a conference paper at ICLR 2025
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303. 15389, 2023.
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocaben panoptic segmentation with text-to-image diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2955-2966 2023.
Le Xue, Mingfei Gao, Chen Xing. Roberto Martín Martin-Martin, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning unified representation of language, images, and point clouds for 3d understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1179-1189) 2023a.
Le Xue, Ning Yu, Shu Zhang, Junnan Li, Roberto Martín-Martinn Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Nicbles, and Silvio Savarese. Ulip-2: Towards scalable multimodal pre-training for 3d understanding. arXiv preprint arXiv:2305.00mmmm 08275, 2023b.
Zhiyuan Yang. Yunjiao Zhou, Libua Xie, and Jianfei Yang. T3dnet: Compressing point cloud models for lightweight 3d recognition. arXiv preprint (xXiv:2402.19266 2024.
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng. and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5752-5761, 2021.
Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjic Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. Mvimgnet: A large-scale dataset of multi-view images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9150-9161, 2023.
Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 19313-19322, 2022.
Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit-Ya Yan Yeung, Zhen Yang, Xiandan Liang. and Hang Xu. Clip2: Contrastive language-inage-poin pretraining from real-world point cloud data. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 15244 15253, 2023.
Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointelip: Point cloud understanding by clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 8552 22625556, 2022.
Zhihao Zhang. Shengcao Cao, and Yu-Xiong Wang. Tamm: Triadapter multi-modal learning for shape understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21413-21423, 2024.
Junsheng Zhou, Jinsheng Wang. Baorui Ma. Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation at scale. In ICLR, 2024.
13

---

Published as a conference paper at ICLR 2025
A APPENDIA/SUPPLEMENTI MATERIAL
The outline of the Appendix is as follows:
More Implementation Details;
More Implementation Details of ensemble datasets;
More Implementation Details of training and evaluation;
More ablation study on the quality of 3DGS;
Additional ablation study objaverse zero-shot classification;
Additional ablation study on ABO zero-shot classification;
More comparisons large-scale data;
More comparisons to state-of-the-ar methods;
More experiments with unfrozen fundamental Encoder;
More experiments on generalization 3DGS-driven methods;
More evaluations on zero-shot image-driven retrieval;
Additional comparisons on zero-shot image-driven retrieval;
Future work.
Social impact;
Discussion
B IMPLEMENTATION DETAILS
Here provide more implementation details on the ensemble of the Objavers<Deiike Deitke al. (2023)
ABCCollins et al. (2022), MVImgNe Yu et al. (2023) and SUN RGBLSSong et al. (2015) datasets; as well as the training and evaluation details on the Text-driven retrieval, Zero-shot classification, and scene recognition tasks.
B. DETAILS OF ENSEMBLE DATASETS
Lup al. 2024) and InternLM Intern. M-ooooosem2222 et al. (2024) leveraged for object caption, which will be used for the Text-driven retrieval task. To better utilize the weights pre-trained on colored point clouds, the SH degree is set to 0 to ignore the view-dependent color effects, which also brings the additional advantage of reducing the computation and storage consumption. Moreover, the opacity reset interval is set to 501 to promote the stability of 3DGS. Note that the saving interval should not be multiplier of the opacity reset interval, otherwise the retained results may become unstable. The optimization iteration for Objaverse, ABO, MVImgNet, and SUN RGBD datasets is set 1500, 2000, 3000, and 500, respectively. All datasets can be successfully prepared 6 RTX40000 GPU within days, where 15 scenes can be optimized simultaneously on each GPU.
B.2 DETAILS OF TRAINING AND EVALUATION
We leverage the activation function tanh(-) to convert the features of 3DGS to the range [-1,1] and set the batch size of training and evaluation to 24 and 80, respectively. In terms of time consumption, the whole training process on Objaverse costs 12.5 hours with 6x RTX4090 GPU, where UniGS trained for 15 epochs on the Objaverse training set for the sufficient understanding and alignment of 3DGS representations, which will be used for Zero-shot classification and Text-driven retrieval the inference stage.
Zero-shot classification. After training 15 epochs on Objaverse, UniGS is directly evaluated on the entire Objaverse-Lvis, ABO, and MVimgnet datasets. We reorganize each dataset accelerate the entire evaluation, where Objaverse-Lvis, ABO, and MVImgNet are reorganized into 315, 23, and 95 categories, respectively.
Text-driven retrieval. In Text-driven retrieval, ABO and MVImgNet will be split into training and testing sets, where the testing sets of Objaverse, ABO, and MV MVImgNet contain 1000, 433, and 1450
14

---

Published as a conference paper at ICLR 2025
Tops
83
25
40
20
Accumccy
20
12
Bestist
Dovibeldan
10
20
12
Images
number ofimsges
number imrgges
Figure 4: Additional ablation study of the quality of 3DGS the Text-driven retrieval task. The accuracy of Text-driven retrieval on Objaverse under three optimization pipelines.
Top5
60-
ACCUBDY 50
TesbiesOurs)
RexitleOursi
12
fand
original
35
10
27
36
36
cumber of Imagus
numbar of imaqus
number of in arga
Figure 5: Additional ablation study of the quality of 3DGS the Zero-shot classification task. The accuracy of Zero-shot classification on ABO under three optimization pipelines.
items, respectively. UniGS will be further fine-tuned for 50 epochs on the training set to alleviate impact the text domain across different datasets. Next, 3DGS encoded by UniGS is used to compute similarity and calculate Topk: accuracy across texts of all the items in the testing set.
Scene recognition. As for the scene recognition task on the SUN RGBD dataset, UniGS follows the basic evaluation pattern to directly train 50 epochs on the training set and finally be evaluated on the testing set.
C ADDITIONAL ABLATION STUDY OF THE QUALITY OF 3DGS
As shown in Fig. and Fig. we considered two common optimization settings of 3DGS for the ablation of the number of images: (1) flexible (ours): load surface points as initialization with flexibility of 3DGS location, (2) original: the vanilla optimization from 3DGS.
As shown the results in Fig. with the increasing number images, the overall accuracy generally shows upward trend. Our "fexible" pipeline exhibits stronger robustness and better reconstruction capability to the number of input images. Notably, when the number of images is halved, the overall accuracy can be maintained within 10% of the optimal level. Moreover, as shown in the results in Fig. of the rebuttal PDF, our additional ablation study on the ABO dataset reveals that 3DGS only requirss small number of images to achieve commendable results. More specifically, simpler reconstruction tasks like ABO, our UniGS can maintain its performance even when the image count drastically reduced from 36 to just 4.
D FURTHER COMPARISONS ON LARGE-SCALE DATA
As shown in Table we supplement comparisons between UniGS and Uni3D when scaling up to the dataset used in Uni3D, which is combination of Objaverse and ABO. As shown in the Large-scale training part in Table, UniGS benefits from scaling up of the dataset and still outperforms Uni3D when considering the larger scale setting, clearly demonstrating the benefit of 3DGS over point clouds. Additional comparisons to the official state-of-the of methods with 10000 3D points can be found in Appendix
15

---

Published as a conference paper at ICLR 2025
Table 8: Experimental results of large scale training on Objaverse-LVIS zero-shot classification. Avg.: the mean average classification accuracy. All methods are trained from scratch.
Avg.
Training Dataset
Methods
Source
3D points Backbone
Representation
Top1
Top3
Top
Large-scale training
Uni3D UniGS(Ours)
46.31 72.62 79.78
with Lvis
1024
EVA02-S
800k
3DGS
49.95
75.60 82.38
Table 9: Comparisons to state-of-the-ard methods with the same data Objaverse-I.VIS zero- shot classification. Avg.: the mean average classification accuracy.
Avg. Top3
Dataset
Methods
Source
Backbone
Representation
Top1
Top
train
test
10000 3D points with official model
TAMM
Point-BERT
50.70
73.20
80.60
800k
46k
point clouds
ReCon++-B
ViT-bigG
53.20
75.30
81.50
800k
46k
point clouds
Uni3D-S
with Lvis
EVA02-S
50.34
72.70
79.81
800k
46k
point clouds
Uni3D-S
EVA02-S
49.87
72.39
79.70
800k
6k
point clouds
UniGS-S(Ours)
EVA02-S
51.22
73.64
80.88
46k
6k
3DGS
E FURTHER COMPARISONS TO STATE-OF-THE-ART METHODS
We supplement extra experiments on representations with 10000 3D points and evaluate UniGS on a no- no-bias mini testing following setting of the "Ensembled (with LVIS)" from Uni3D. As shown in T UniGS benefits from the increasing of 3D points, achieving similar Top accuracy compared with Uni3D, TAMM and ReCon++ (regardless of backbones). Note that TAMM and ReCon++ use larger model parameters than UniGS.
Therefore, shown Table and Table with the dataset scaling up and the increasing of 3D poitns, UniGS have great potential to show superior performance over common point clouds, outperforming the same data.
Moreover, UniGS proposed to leverage 3DGS as 3D representation and utilize Gaussian Aware Guidance for better understanding, which can be applied existing 3D understanding work with point clouds for improvement. Since we leverage Uni3D as our fundamental encoder, experiments will focus on the comparisons with Uni3D the original paper to show the effectiveness of 3DGS representation and our Gaussian-Aware Guidance module.
F ADDITIONAL ABLATION STUDY WITH UNFROZEN REPRESENTATION
We freeze the Uni3D encoder UniGS only leverages spatial information of point clouds for explicit feature learning. Therefore, we opt to freeze the Uni3D encoder, utilizing it solely for guidance, which leads relationship modeling and feature understanding of the 3DGS explicit features, instead of directly training our point encoder on 3DGS.
Moreover, we provide additional experiments with unfrozen fundamental encoder in Table 10 Despite the doubled demand for computing resources, UniGS with an unfrozen fundamental encoder still achieves comparable performance.
G GENERALIZATION TO 3DGS-DRIVEN METHODS
We provide more comprehensive results the robustness to 3DGS-derive methods with UniGS trained on 2DGS representation. As shown in the Table above, UniGS pre-trained 2DGS achieve similar performance compared with UniGS pre-trained on 3DGS, illustrating the power our proposed Gaussian-Aware Guidance and the ability migrate other dervvde methods.
16

---

Published as a conference paper at ICLR 2025
Table 10: Additional compartsons state-of-the-art methods on Zero-shot classification. Avg. denotes the mean average classification accuracy. denotes training from scratch and +denotes unfreezing the fundamental encoder.
Ufrozen
Objaverse Avg.
ABO Avg.
Methods
fundamental encoder
Top1
Top3
Top 5
Top1
Top3
Top 5
TAMM*
22.70
38.83
47.13
35.44
54.96
63.40
ReCon *
23.40
41.41
48.95
34.29
55.14
67.69
Uni3D*
30.47
48.46
55.87
37.79
61.08
69.04
UniGS(Ours)* UniGS(Ours)f
x
38.57
60.54
68.96
46.97
69.91
79.38
/
38.74
62.89
71.88
47.53
70.49
78.60
Table 11: Zero-shot classification on Objaverse with other 3DGS-driven methods. Avg. denotes mean average classification accuracy. Results illustrate the ability of UniGS to migrate to other 3DGS-driven methods. 2DGS denotes "2D Gaussian Splatting for Geometrically Accurate Radiance Fields".
Representation
Avg.
Training & Testing
Top1
Top3
Top
3DGS
38.57 38. (-0.46)
60.54 61.28 (+0.74)
68.96
2DGS
70.32 (+1.36)
H ZERO-SHOT IMAGE-DRIVER RETRIEVAL
As shown Table 12 we further evaluate UniGS with Zero-shot Image-driven retrieval in batch, which reveals the alignment between 3DGS and the image domain. Experiment results on ABO and MVImgNet Table 12 demonstrate the power of 3DGS in Image-3D alignment.
I FUTURE WORK
To further validate the scaling ability of UniGS for better performance and 3D understanding. The next step of UniGS is to conduct it on large 3DGS datasets with more 3D points and further explore model architectures for language-image imeee-mmmae-DDD-- alignment. We will establish more pipelines that support the conversion of various datasets into 3DGS and further expand the scale of the 3DGS datasets. Another interesting future direction will be the model architecture exploration, where plan to further explore the connection between fundamental encoder and advanced encoder, and attempt to support training point cloud datasets for larger datasets and benchmarks.
J SOCIAL IMPACT
While there a wide range of application domains where 3D representation learning will be beneficial, such in autonomous driving, augmented/virtual reality, and embodied AI, there also potentially negative application scenarios. For instance, these approaches could be used in malicious contexts obtain private image data through splatting with a specific decoder or for surveillance purposes. Consequently, UniGS is released as a research tool to benefit the academic field only.
K DISCUSSION
K.1
COMPARED TO NERF- APPROACHES
conduct additional comparisons with NeRF-based approaches and UniGS outperforms nerf2elip Ballerini et al., 2024) and nf2vec(Ramirez et al. 2023) with 9.93% and 6.64%, demonstrating significant improvement NeRF-hased approaches cross-modalities learning.
As shown in Table 13
17

---

Published as a conference paper at ICLR 2025
Avg. Top3
Methods
Representation
Text-image Model
Embedding dim
Top1
Top 5
Objaverse-Lvis
CLIP2
28.83
51.43
63.57
point cloud
CLIP2
27.99
50.30
62.76
3DGS
Uni3D*
39.65
60.72
70.51
point cloud
ViT-B-16
512
Uni3D
35.82
58.35
69.63
3DGS
*ni3D*
34.74
56.70
67.12
3DGS
UniGS(Ours)
41.78
62.50
72.24
3DGS
ViT-B-16
512
ABO
CLIP²
15.29
31.74
42.74
point cloud
CLIP
13.80
29.60
40.85
3DGS
Uni3D*
18.25
35.26
45.29
point cloud
ViT-B-16
512
Uni3D
21.14
38.88
49.38
3DGS
Uni3D*
25.30
45.69
57.51
3DGS
UniGS(Ours)
26.69
46.26
56.72
3DGS
ViT-B-16
512
MVImgNet
CLIP
6.900
17.65
26.16
point cloud
CL.IP2
2.160
6.330
10.12
3DGS
Uni3D*
6,380
16.65
25.50
point cloud
ViT-B-16
512
Uni3D
1.410
4.260
6.840
3DGS
Uni3D*
7.940
18.86
27.20
3DGS
UniGS(Ours)
10.55
23.75
33.15
3DGS
ViT-B-16
512
Table 12: Zero-shot Image-driven retrieval. Avg.: the mean average retrieval accuracy. * denotes training from scratch.
Methods
3D Representation
Avg.(%)t
ShapeNetRender
CLIP(I view)
73.60
CLIP(16 view)
82,40
nerf2elip
NeRF
84.00
nf2vec
NeRF
87.30
Uni3D
3DGS location
88.96
UniGS(Ours)
3DGS
93.94
Table 13: Zero-shot classification ShapeNetRender. Avg.:
the mean average Topl classification accuracy. Uni3D and UniGS is trained for 15 epoch on ShapeNetRender.
K.2 UNIGS UNI3D
As shown in Fig. 6 we highlight the difference between UniGS and Uni3D. Our UniGS is also capable of utilizing single model to unify 3D representations from different models, with better performance with 3DGS representation and proposed Gaussian-Aware Guidance. Specifically, when using point clouds unified 3D representation, the main challenge is the divergence between the 3D representation and other modalities. In contrast, UniGS leverages 3DGS as the 3D representation, which effectively reconstructs the 3D target object well provides efficient correspondences between 3D and 2D images.
Moreover, conducted additional experiments fine-tune UniGS using pure 3DGS dataset. As shown in Table [4, UniGS outperforms Uni3D when augmented with point clouds under the same settings Uni3DD. It also achieves higher Top-1 and Top-3 accuracy after fine-tuning on the pure
18

---

Published as a conference paper at ICLR 2025
Uni3D (Text-Image-3D Preiraining)
UniGS (Text-Image-3D Pretraining)
Performance
Text
Image
Points
Text
Image
3DGS
Bohh RL-U Propaiie
brown toy yellow cloths
brown toy yellow cloths
Viliggst
Text encoder
Points encoder
Text encoder
Image encoder
3DGS encoder
encoder
Alignment
Alignment
Figure 6: Left: Uni3D, using 3D point clouds for text-image-3D pre-training. Middle: our UniGS, leveraging 3DGS as the 3D representation with better alignment with image modality. Right: our UniGS learns more general and stronger multi-modal representation.
Table 14:
Comparison results with 10000 points dataset on Objaverse-I.vi zero-shot classifica- tion. denotes fine-tuning on 3DGS datasets.
Avg. Top3
Augment point clouds
Methods
Backbone
Representation
Top1
Top 5
10000 3D points
Uni3D
50.34
72.70
79.81
point clouds
UniGS
EVA02-S-patch14
52.44
75.37
82.71
3DGS
UniGSt
53.16
75.59
82.14
3DGS
x
3DGS dataset. Experimental results Table demonstrate that Uni3D is not fully compatible with both point clouds and 3DGS. However, with proposed Gaussian-Aware Guidance, UniGS exhibits the ability to effectively understand objects in both point clouds and 3DGS, achieving superior results after fine-tuning.
K.3 RUNTIME ANALYSIS
Table 15: Comparisons of forward computational cost Objaverse-L.vi.
Methods
FLOPs(G) Time(ms)
Top Avg.
CLIP²
22.49
232
10.20
TAMM
22.49
233
22.70
Uni3D
47.85
113
30.47
UniGS(Ours)
98.17
233
38.57
As shown in the Table 15 we further evaluate the FLOPs and runtime UniGS and compare them with state-of-the-ee art approaches. With slight increase in runtime, UniGS achieves significant improvement over CLIP?, TAMM, and Uni3D on Objaverse-Lvis zero-shot classification.
Moreover, shown in the Table 16 and Fig. 7 present additional ablation study of UniGS modules on FLOPs in Fig. 7 Specifically, this helps us understand the difference as 76.5% of the total FLOPs (73.38G) due to the CNN layers of the 3D Encoder extract 3D spatial features.
Fortunately, much progress being made in compressing models( Yang et al. 2024) and 3D rep- resentations(Fan al. 2023), and expect these advances to facilitate the development of 3D understanding with 3DGS representation.
K.4 VISUAL COMPARISONS
As shown in Fig. Uni3D may mistakenly retrieve another similar object due to the similarity in point cloud structure. In contrast, UniGS demonstrates superior 3D understanding of object color,
19

---

Published as a conference paper at ICLR 2025
Ablation study on the FLOPs of UniGS
MLP
Advanced Encoder blocks
Cross
Advanced Encoder
Crass-Are & Others
NxVIT Block
Attn
Nx ViT Block
CNN layers
38. 38.3%
0,7%
11.4%
Fundamental Encoder VIT blocks
CNN layers
CNN layers
Group Divider
Group Divider
Fundamental Encoder layers
38.2%
(4, c)
(H4 (,,, R)
Fundamental encoder
Advanced encoder
Figure 7: Additional ablation study on the FLOPs of UniGS. (a) FLOPs of each module, (b) 3D Encoder of UniGS. 76.5% of FLOPs are due to the CNN layers of the fundamental and the advanced encoder extract 3D spatial features, while only 23.5% of the FLOPs is spent for 3D understanding.
Fundamental Encoder
Advanced Encoder
Cross-Attn Others
FLOPs
CNN layers ViT blocks
CNN layers
VIT blocks
36.67
X
X
47,60
X
X
84.31
X
95.24
95.43
95.94
Table 16: Ablation study on the FLOPs of UniGS modules. CNN encoder denotes the CNN
layers extract spatial information from 3D representation into features, and ViT blocks denotes the Transformer blocks understanding object from extracted features. Cross-Attn denotes the Cross- attention layers between Fundamental and Advanced Encoder.
shape, and texture with 3DGS representation and proposed Gaussian- Guidance, resulting in better Image-to-3I retrieval.
20

---

Published as a conference paper at ICLR 2025
Image Query
Uni3D retrieval
Image Query
Uni3D retrieval
wrong
UniGS retrieval
UniGS retrieval
Image Query
Uni3D retrieval
Image Query
Uni3D retrieval
wrong
UniGS retrieval
UnIGS retrieval
(c)
Figure 8: Visual comparisons to Uni3D on Image-to-3D retrieval. With 3D Gaussian Splatting representation capturing image details and a powerful Gaussian-Awaen Guidance module, UniGS outperforms Uni3D in 3D understanding of object color, shape, and texture.
21


Reviews:
[{"review_id": "PIpvXo7rvr", "reviewer": "Reviewer_agiJ", "summary": "This paper presents a novel multi-modal pre-training method, UniGS, designed to achieve more general and efficient joint representations of text, images, and point clouds. The innovative introduction of 3D Gaussian Splatting (3DGS) and the proposed Gaussian-Aware Guidance module achieve leading results in different 3D tasks. The experiments are solid and can confirm the effectiveness of the proposed method.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed approach can achieve state-of-the-art performance on various challenging datasets, which demonstrates the effectiveness in learning strong cross-model representations.", "weaknesses": "1）No detailed explanation was given on how negative samples were selected. Do different tasks need to adjust the negative sample strategy?\n2）There are significant differences in the structural and spatial characterization of 3DGS and traditional point cloud data. Does it affect the final performance？", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J8fJlASgbi", "reviewer": "Reviewer_yxJz", "summary": "This paper presents a text-image-3D pre-training framework that leverages 3DGS as the 3D representation for multi-modal representation. Experiments on various datasets are conducted.", "questions": "I suggest that the authors provide more experiments and illustrations for the questions in weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method has good performance on various multi-modal datasets.\n- The experimental results are given on two different datasets COCO and VisDrone.", "weaknesses": "- Uni3D has used one model to unify the 3D representations from different models, which can be used to align with image and text. What is the main advantage of proposed method using 3DGS.\n- The proposed method introduces 3DGS for feature experimentation. Does it increase computational cost.\n- It seems that most experiment are not inconsistent with the results in Uni3D. In this paper, the performance are relatively poor. What is the difference.\n- I think that it is better to use the similar settings as Uni3D for experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BwhlSg9nzw", "reviewer": "Reviewer_fg23", "summary": "UniGS introduces 3D Gaussian Splatting (3DGS) into multi-modal pre-training to address the limitations of point clouds in capturing the full complexity of 3D scenes. By representing the 3D world as collections of colored, opaque Gaussians, UniGS bridges the gap between discrete 3D points and dense 2D images. Starting from a pre-trained vision-language model, UniGS aligns 3DGS with text and image representations, creating a unified multi-modal representation. A Gaussian-Aware Guidance module further enhances fine-grained 3D feature learning and cross-modal alignment. Tested on Objaverse, ABO, MVImgNet, and SUN RGBD datasets, UniGS shows substantial improvements over the previous state-of-the-art Uni3D.", "questions": "1. L56-L60, the paper discusses the weakness of using point clouds as a 3D presentation due to the discrete representation. How about other potential 3D representations? Like depth maps or NeRF, which do not suffer from the discrete representation. I think some discussion among all potential 3D representations will help to justify the choice of using 3DGS as the unified 3D representation.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The improvement gain is significant compared with previous methods, which shows the effectiveness of using 3DGS as the unified 3D representation.\n\n2. The paper introduces an innovative Gaussian-Aware Guidance module that utilizes priors from pre-trained point cloud encoders as an initialization to enhance the learning of 3DGS features. This design is effective since it doesn't require training from scratch but can make use of existing models from a different 3D representation.\n\n3. The intuition to use 3DGS as the unified 3D representation is novel and reasonable. To the best of my knowledge, it seems to be the first paper to use 3DGS as the unified 3D representation to bridge multimodal.", "weaknesses": "1. Figure 2 could provide an overall description of the information flow (like how this pipeline works in general) in the caption. Also, the figure could be improved by adding some diagrams to represent downstream tasks instead of using text only.\n\n2. I think one significant weakness of using 3DGS as a unified 3D representation is that, usually raw data doesn't use this representation, like point cloud from a Lidar sensor. In this way, this method needs to optimize or process a 3DGS using those raw data (let me know if I understand incorrectly), then leverage this unified 3D representation to conduct downstream tasks. It could be computationally expensive. It seems that this paper does not provide the computation cost for inference, I would appreciate it if authors could include this and discuss this potential weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IB66yKO6ti", "reviewer": "Reviewer_SvjU", "summary": "This paper introduces a system for computing a joint embedding of images, text and 3D models (similar to an extension to CLIP). The paper proposes to use gaussian splats as the 3D representation instead of using point clouds to represent 3D shapes.", "questions": "1. Caption in figure 2 could provide some details to help interpret the system sketch. Figure 1, figure 2 and figure 3 seem redundant. It would be clearer if they were integrated into a more comprehensive system description.\n\n2. Section 3.1 is not very helpful, presents standard material and it is not particularly clear. Section 3.2, which introduce the architecture proposed in the paper, is short.\n\n3. Some typos:\nLine 32: “to achieve Language-Image-3D pertaining,” -> pretraining?\nLine 239: “for understanding the relationships between global position and feature.” What feature?\nLine 253: “We donate the process”, -> denote. \n\n4. Section 3.4 is not very clear.  It will be helpful if figure 3 had the same notation than the one used in section 3.4. For instance, where is f_{fun}, f_{adv}? Then, in line 257 the notation changes again introducing f’_{fun} and f’_{adv}.  Why do you add the single quote ‘?\n\n5. It would be helpful to add figures with some visual examples and comparisons across different methods. How helpful is to use GS versus point clouds?\n\n6. The experimental results section has a lot of tables, but they are not clearly described. For instance, section 4.3 only briefly describes table 5, but it is not clear what the table shows. If this is not central to the paper, this table and section could be moved to the supplementary material section and the extra space can be used to better describe the other experiment and add a figure with visual results.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Building joint representations across different modalities is an interesting direction of work. Including 3D shapes can have lots of applications (which would be good to discuss in the paper).", "weaknesses": "I miss clear experiments that help understand the benefit of using 3D gaussian splats as the representation of 3D shapes instead of using point clouds. For instance, what numbers do you get in table 3 if you use the architecture used in UniGS but replacing the GS with point clouds?\n\nCan you provide some scenarios in which it is useful to have the representation you propose?\n\nThe paper has no visual results. There are only tables with numbers. But it is hard to get an intuition of what the numbers mean and what the results look like just by reading those tables. It would be helpful to show some visual illustrations. You could show examples if 3D retrieval when the query is a 2D image.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aY3vNdxk6q", "reviewer": "Reviewer_SAh2", "summary": "This paper argues that point clouds, as 3D representations, cannot fully capture the intricacies of the 3D world. To address this, it proposes the use of 3D Gaussian primitives instead of point clouds. The authors introduce a pretraining strategy to align the 3D Gaussian features with those of a pretrained vision-language model, establishing a shared visual and textual space through extensive real-world image-text pairs. Additionally, they propose a Gaussian-Aware Guidance module that leverages priors from pretrained point cloud encoders to guide the learning of Gaussian features, enhancing 3D understanding. The proposed approach achieves state-of-the-art performance across various challenging datasets.", "questions": "1. Clarification on NeRF-based Methods: Could you clarify why NeRF-based methods were not discussed in this work? Are there any specific disadvantages that prevented you from considering them as alternatives to 3DGS?\n\n2. Evaluation of Computational Time: Have you evaluated the runtime of your method? While 3D Gaussian primitives may offer several advantages, they require an additional optimization process that point clouds do not, as point clouds represent raw data directly from 3D sensors. This suggests that 3DGS involves additional time for optimization, which is not discussed in the paper. Could you provide more details on this aspect?\n\n3. Initialization with Point Clouds: For clarification, did you use sample raw point clouds from meshes to initialize the Gaussian primitives for the experiments (for ABO and Objaverse) presented in your main results? If so, could you explain why, in some datasets (such as ABO in Table 2), the performance of 3DGS falls short compared to point cloud methods?\n\n4. Clarification on Baseline Performance and Experimental Settings: Could you elaborate on how the performance of the baseline models was obtained? Additionally, how does the experimental setting of the results reported in your paper (Table 2, Lines 328-329, showing 38.17% top-1 accuracy) differ from the Objaverse-LVIS zero-shot classification performance reported in Uni3D[5] (Table 1)? A detailed comparison would clarify any discrepancies.\n\n**References:**\n\n[1] Jeong, et al. (2022). Perfception: Perception using radiance fields. Advances in Neural Information Processing Systems, 35, 26105-26121.\n\n[2] Hu, et al. (2023). Nerf-rpn: A general framework for object detection in nerfs. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,* (pp. 23528-23538)\n\n[3] Li, et al. (2024). GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 21708-21718).\n\n[4] Ballerini, et al. (2024). Connecting NeRFs Images and Text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop (pp. 866-876).\n\n[5] Zhou, et al. (2024). Uni3D: Exploring Unified 3D Representation at Scale. In The Twelfth International Conference on Learning Representations.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Introduction of 3D Gaussian Primitives: It proposes a novel representation using 3D Gaussian primitives, offering a potential alternative to point clouds for improved 3D modeling.\n2. Alignment with Vision-Language Models via Pretraining: The pretraining strategy aligns 3D Gaussian features with a pretrained vision-language model, enabling a shared visual-textual space through large-scale real-world image-text pairs.\n3. Gaussian-Aware Guidance Module: The proposed module leverages pretrained point cloud encoders to guide the learning process, helping the 3D Gaussian features improve their understanding and representation capabilities.\n4. State-of-the-Art Performance: The method demonstrates superior results, achieving state-of-the-art performance across some of the datasets against point cloud-based counterparts.", "weaknesses": "This paper has several weaknesses that should be addressed:\n\n1. Lack of Comparison with Related Works: While the authors contend that 3D Gaussian primitives (3DGS) offer a superior 3D representation, the paper does not provide comparisons with other relevant works, such as [1], [2], [3], and [4]. These works utilize Neural Radiance Fields (NeRF), which offer similar advantages for 3D perception tasks, as mentioned in Lines 63-70. Including such comparisons would strengthen the argument.\n2. Insufficient Clarity in the Implementation Details: The paper lacks clear explanations regarding the methodology, leaving certain aspects ambiguous. This raises multiple questions that require further clarification, as outlined in Questions 3 and 4.\n3. Unsubstantiated Hypothesis: Without addressing the aforementioned issues, the core hypothesis that 3D Gaussian primitives are a better representation than point clouds remains unconvincing. A thorough comparison and clearer implementation details are necessary to support this claim effectively.\n4. Improving these weaknesses could strengthen the paper’s claims, which may lead me to consider raising its rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)