Duoduo CLIP: Efficient 3D Understanding with Multi-View Images
Authors: Han-Hung Lee
Yiming Zhang
Angel X Chang
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Representation
3D Understanding
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: We introduce Duoduo CLIP, a model for 3D representation learning that learns shape encodings from multi-view images instead of point-clouds. The choice of multi-view images allows us to leverage 2D priors from off-the-shelf CLIP models to facilitate fine-tuning with 3D data. Our approach not only shows better generalization compared to existing point cloud methods, but also reduces GPU requirements and training time. In addition, the model is modified with cross-view attention to leverage information across multiple frames of the object which further boosts performance. Notably, our model is permutation invariant to the order of multi-view images while being pose-free. Compared to the current SOTA point cloud method that requires 480 A100 hours to train 1 billion model parameters we only require 57 A5000 hours and 87 million parameters. Multi-view images also provide more flexibility including being able to encode objects with a variable number of images, and performance scales when more views are used. In contrast, point cloud based methods require an entire scan or model of the object. We showcase this flexibility with benchmarks from images of real-world objects. Our model also achieves better performance in more fine-grained text to shape retrieval, demonstrating better text-and-shape alignment than point cloud based models.

Published as a conference paper at ICLR 2025
DUODUO CLIP: EFFICIENT 3D UNDERSTANDING WITH MULTI-VILE IMAGES
Han-Hung Lee', Yiming Zhang" Angel X. Chang 'Simon Fraser University. Canada CIFAR AI Chair, Amii {hla300, yza440, angelx @0sfu. ca http:///hhhhcoomo///ll/ee///oooo
ABSTRACT
We introduce Duoduo CLIP, model for 3D representation learning that learns shape encodings from multi-view images instead of point clouds. The choice of multi-view images allows us leverage 2D priors from off-the-shelf CLIP models to facilitate fine-tuning with 3D data. Our approach not only shows better generalization compared existing point cloud methods, but also reduces GPU requirements and training time. In addition, the model is modified with cross-view attention to leverage information across multiple frames of the object which further boosts performance. Notably, our model is permutation invariant to the order of multi-view images while being pose-free. Compared to the current SOTA point cloud method that requires 480 A100 hours to train 1 billion model parameters we only require 57 A5000 hours and 87 million parameters. Multi-view images also provide more flexibility including being able encode objects with variable number of images, and performance scales when more views are used. In contrast, point cloud based methods require an entire scan or model of the object. We showcase this flexibility with benchmarks from images of weal-world objects. Our model also achieves better performance in more fine-grained text to shape retrieval, demonstrating better text-and-shape alignment than point cloud based models.
INTRODUCTION
Learning representations for 3D shapes is useful for many applications in object generation, shape understanding, grounding and robotics. Recently, there also growing interest in aligning 31> representations to text, to enable text-to-3D shape retrieval (Ruan et al. 2024, (Tang et al., (0233) Wang jet al.) (003) and generation (Chen et al. 22019), or as a basis for 3D feature maps that can be queried with text (Huang et al. 2023a; Jatavallabhula al. 2023
Liu et al. 2023a).
Early work in this domain attempted to train such aligned models on limited data (Chen et al.] 2019: Ruan et al.,
Table 1:
Comparison of training time and GPU used for our method and recent point cloud based methods.
2024. Tang et al. 2023). More recent work (Liu et al.
2023a: Xue et al. 2023) leveraged large pre-trained vision-
language models (e.g. CLIP (Radford et al.
2021 Larger
Method
Time
3D datasets like Objaverse (Deitke al.
2023) enabled
OpenShupel Lip et al,
Uni3D 2100 & di 2014
20230
24x 100 (40GB)
Ax00000 (80GB)
300 hr
20 hr
training of methods that can handle more diverse object
RECON
A800 (80GB)
categories (Liu al.] 2023a Qi et al., 2024 Xue et al.
Ours (Rull)
(48GB)
14.3
2024. Zhou et al.) 2024). However, more and more GPU resources are required for training such methods. These works (Liu al. 2023a Qi al. 2024 Xue
Ours (6 laycrs)
4x (24GB)
14,3 hr
ct al. 2024) predominantly rely on point clouds as their 3D representation. Point clouds are limited in resolution and have a domain gap with real-world images, and thus these point -ase- methods can not effectively leverage 2D priors from CLIP. Uni3D (Zhou et al.] 2024) proposed training regime to incorporate 2D priors with point clouds as input. However, their model requires significant scaling (IB parameters) for the best performance. In this work, we show that our Duoduo CLIP approach which uses multi-vien view images as the shape representation allows more easily adapting fif-the-shelf 2D representation models while enabling more efficient training. As shown in Fig. [b] our model not only has much stronger generalization on shapes that were not seen during training, but also significantly lowers the compute resources required for training (see Tab. 1

---

Published as conference paper at ICLR 2025
A Anentalorh day
55
(14.3 hr w/ 4cA5000)
Duoduo CLIP
Text Encoder
Text Encoder
50
UniaD
Accuracy %
45
Image Encoder
Contraztie Controztie
Image
Contrative
[300 w/ 1xA100) OpenShape
Encoder
Contrastive Loss
35
PCD Encoder
MV
32M
87M
Image Encoder
# Parameters
Doud
(b) Accuracy on Objaverse-LVVS with all models trained on Ensembled (no LVIS). Our Duoduo CLIP is fast train, relatively small while also achieving the best performance.
OpenShape / Uni3D
Duoduo CLIP
(a) We multi-view images, instead of point-clouds, represent the shape.
Figure Duoduo CLIP is model for 3D shape understanding that represents the shape with multi-view images encoder instead of point-cloud (PC) encoder. This makes our model more accurate and efficient than PC methods (OpenShape Liu al. (2023a), Uni3D Zhou al. (2024))
Figure 2: We blend concepts from two distinct objects by identifying the Objaverse shape with the embedding that maximizes similarity to both objects simuitancously. This shows that our model has learned latent space capable of combining geometric features from the mixed objects or capturing their semantic attributes. For more details see App. A.4.2
Multi-view images also offer greater flexibility. For example, in real-world datasets like ScanNet (Dai et al. 2017) or ScanObjectNN (Uy et al. 2019)) 3D mesh is first reconstructed from densely captured RGBD data from which 3D point clouds are sampled. This process introduces errors and artifacts, in contrast to sparse multi-view images which can be directly sub-sampled from the RGB video stream. Our model takes images directly input, making it plug-and-plaa for any method using image features for encoding 3D objects, which is particularly important for real-time robotics applications with sparse views (Gadre et al.] 2023) Garg al. 2024, [Shrddhar al. [0022] Wang al. 2024). Additionally, previous methods (Liu et al. 2023a) that only train synthetic objects have limited performance on real-wer world objects. Our method can be trained on both real photos and reconstructed objects and outperforms prior work real-world objects. Performance further be boosted with additional multi-view images when point clouds are not available, demonstrating our model's potential for scaling more data becomes available.
In summary, our Duoduo CLIP model provides the following: Simpler model. We average em- beddings across views of object obtain 3D representations, while modifying the attention layers to span multiple frames of the shape to enable information flow across frames in the model. Better efficiency. We explore several strategies for efficiently fine-tuning off-the-shelf CLIP models, including training only the attention layers and selectively choosing the number of layers to train without hurting performance. More flexible representation. We show the benefits of multi-view images more flexible 3D representation over point clouds on real-worke objects. Furthermore, we demonstrate better text to shape retrieval with more fine-grained descriptions, and that by maximizing similarity to two different shape embeddings, our learned embedding space can also be used to blend concepts (see Fig. 2)
RELATED WORK
There is much work on 3D representation learning using supervised training QQ et al. 2017 Qian al. 2022) using self-supervised learning (with reconstruction loss (Chen Zhang 2019)
2

---

Published as a conference paper at ICLR 2025
contrastive learning (Xie et al. 2020), masked prediction Yu et al. 2022b), and others). Here focus on work that learns aligned representation spaces between text and either 2D or 3D.
2D-text joint representation. Modeling multiple modalities through shared representation has been explored through carly works by aligning images and text embeddings (Faghri et al.) 2018 Frome Kiros et al., [014] Weston et al., (011). With the introduction of the InfoNCE loss (Oord
et al.
2013
et al. 2018), VIR' ZZang et al. 2022b) proposed using contrastive learning with InfoNCE style
loss to learn joint text and image embeddings. This was popularized by follow-up work such as
CLIP (Radford et al. 202T) and ALIGN (Jia et al. 202T), which demonstrated that by scaling up training on a large-scale dataset, the learned embeddings can be useful for many downstream tasks such as classification, retrieval, grounding and generation. Followup work (Jia et al. [02T] Mu et al.)
2022 Yang et al., 2022 Yu et al., 2022a Zhai et al.) 2022) has since improved upon the formulation or training. There have also been attempts such as MERU (Desai et al., 2023) that use a hyperbolic
embedding space to better model the hierarchical relations between text and images.
3D-text joint representation. Text2Shape (Chen et al. 2019) was one of the first works that introduced captioned text and 3D dataset and used metric learning to learn an aligned space between text and 3D shapes. Follow-up work investigated improvements to the aligned representation for text-to-shape retrieval with triplet loss (Tang et al. 2023) and contrastive losses (Ruan et al., 2024
Wang et al. 2023 Xue et al., (023). Some prior work used of CLIP to align the embeddings (Ruan
et al. 2024, [Toomsonn et al. 2022 Xue et al., 2023). ULIP (Xue et al. 2023) used frozen CLIP text-image encoders, and trained only the point-cloud encoder. However, generalization ability was limited by the available dataset size at the time. With the introduction of larger scale 3D datasets such
as Objaverse (Deitke al. 2023), and the use of LVLMs for captioning 3D datasets (Liu al. 2023a
Luo et al.] 2024). more recent work (Liu el al.) 2023a Xue et al. 2024) trained with larger datasets of
paired text and 3D shapes. OpenShape (Liu et al.) (023a) and ULIP-2 (Xue et al., (024) were among
the first to use such datasets to train point cloud encoder aligned with text using contrastive learning,
TAMM (Zhang et al. 2024) leverages additional adapters to better align with the pre-trained CL.IP models. Uni3D) (Zhou et al. 2024) exploits 21) priors within pre-trained large-scale vision models and
maps point cloud patches to image patches within pre-trained ViTs to enable the scaling of even larger models. VIT-LENS (Lei et al. 2024) follows similar pipeline by training point cloud embedding layer and perceiver model to map to a frozen CLIP model. RECON++ (Qi et al. 2024) similar to RECON (Qi et al. 2023) employs both contrastive and MAE (He al. 2022) reconstruction losses with an additional multi-view image matching cost. These recent works typically rely on frozen text and image encoders, and all used point-cloud encoders. Our work shows that taking embeddings of multi-view images is more efficient and works better with real-world images. Note that Gao et al. (2024) combines point clouds with optional multi-view embeddings from a frozen CLIP encoder, which lacks inter-view context, and does not support arbitrary views or poses, resulting in lower performance at the same number of views while also requiring point cloud.
Multi-view images for text-3D representations. Multi-view images have long been used to represent 3D shapes (Bradski & Grossberg 1994] Hamdi et al. [021]] Shen et al., 2003 Su et al. 2015) Wei et al. 2020), with recent work (Lin et al. 2024, Song et al.) 2023) using multi-view images for shape representation by aggregating CLIP features. While these works also use CLIP with multi-view images, our focus is whether using trainable multi-view shape encoder as drop-in- replacement for the point-cloud encoder in contrastive settings such as OpenShape will result in better shape representations for downstream tasks. These works do not focus improved representation learning, but on how to use prompting to select better views or view aggregation and improve shape classification. There is also line of work (Huang al. 2023b Zhang et al.. 2022a Zhuu et all 2023) that uses multi-view depth maps from point cloud to adapt CLIP models for 3D. However, the focus of these methods is still on point clouds and the models show limited performance on large-scale datasets (Liu et al. 2023a Xue et al. 2024).
3 METHOD
Our Duoduo CLIP learns a shape representation by using contrastive learning to encode multi-view images into pre- aeiigli text-image space. Our contrastive learning framework is similar to previous works (Liu et al.] 2023a] Xue et al. [2023] 2024, Zhang et al.; 2024 Zhou et al., 2024), with a shape encoder F. and frozen image E. and text ET encoders from a pre-trained CLIP model. The key
3

---

Published as a conference paper at ICLR 2025
Synthetic/ / Real
X6
Attn
LxkeTokes
Average
Tokens
(Lx M) x Tokens
1"M Frames
Tokeos
Figure 3: Our multi-view encoder takes variable number [1, M] of images as input and outputs a single-shape embedding. The first few layers of the ViT model are frozen (blue) and attention operates on the individual frames in parallel. For the latter layers, the attention layers are trainable (orange) and modified to attend over all M views. The embeddings for each frame are averaged to get the final embedding. Note that only major components of the model are depicted.
difference is that we replace the point cloud shape encoder with one that uses multi-view images. Fig. Ta illustrates this difference, while Fig. 3 shows the details of our multi-view shape encoder.
To perform training, the dataset is organized into triplets of (S,, IG, T,). where S is the shape representation, are rendered images of the 3D object and T is the corresponding text description. Embeddings are calculated for each modality l = E (S.). f = (T:) and s = =F.(T.). The
asymmetric contrastive loss (Zhang et al. 2022b)
between any two modalities is:
expl f//
10 log
EN
exp (ff, f21/T
where a, b is any two modalities, 7 is the temperature and N is the batch size. Then, the full loss is:
1 M (1.- + 1f + 15- + 11
LCON
4N
Two important properties of our shape encoder for multi- view images include being able to handle arbitrary number of views as well as being permutation invariant to ensure the flexibility for the input images. Encoding of any number of frames with the model into a single embedding well permutation invariant operation between multi-views are discussed in more detail below. Note that we also do not make assumptions about the poses of the input image, making our method pose-free.
Multi-view shape encoder. For our multi-view shape encoder, we use rendered images of the object. Note that the input shape and image representations are the same, i.e. S, I, (ri,,..,,'i where ry is a rendered frame, and yn is the number of multi-views. E. and ES share the same network architecture, with the main difference between E' and E is that E is frozen, while we train E.5. Weights for E are initialized from E'. Empirically, we found it was useful to keep the contrastive losses (5-+/ + I!- We suspect that by keeping E frozen and the contrastive loss with the original image embeddings, our ES can learn a good representation that is close to the text embeddings, but does not drift much away from the initial image embeddings.
For both ES and E we average the extracted embeddings for all m views to produce a single latent representation for object: f E (S,) L E (iT(S,) where ViT (Dosovitskiy al. 2021) is the backbone encoder. Here we abuse the notation a bit, since our method allows intercttin between views in the model, the index operation gets the corresponding embedding from the j-th frame's ' [CLS] token. We initialize Es initially from E'. During training, we randomly sample 1 to M views for each step so that the model is robust to a variable number of views.
Multi-view attention. We incorporate information flow between views by modifying the self- attention layers to attend over tokens across multiple views similar to recent multi-view aware methods like MVDream (Shi et al. 2023). The original self-attention (Vaswani et al., 2017) is given by Attention(Q, K, V) = saft QK* VoL VV, where Q, K, V RLxk k I. is the token length for single image after patchify operations, and k is the channel dimension. For cross-view attention we aggregate token features across m multi-views (MV) so that QV KM VMV e R(Lxm) xk and the Attention(QMy KM LMV) can be calculated without changing the parameters of the original
4

---

Published as a conference paper at ICLR 2025
Figure 4: Attention maps are extracted from the first trainable MVA layer, capturing token-wise attention across all M views. A specific token (e.g., the plate) is queried retrieve the corresponding row from the attention map, and its magnitudes are mapped onto the original images for visualization. The model has learned geometric correspondences of 3D shapes across different multi-views.
model. After attention, the views are separated and processed parallel in other parts of the model. This mechanism helps the model reason between different views of the object and aggregate the most important information across all tokens. In Fig. we visualize the attention across views corresponding to selected token (see App. 1 for details). From the figure, we see the multi-view attention layers learn capture correspondence between object parts from different views.
Trainable layers. To reduce the memory needed for fine-tuning, we only train portion of the self-attention layers while keeping the MLP layers after attention fixed. Since the non-trainable self-attention layers have only been trained to see patches of single image, we only enable the multi-view attention layers for the unfrozen attention layers (see Fig. 3). Additionally, we find the choice of freezing the MLPs, helps in preventing overfitting and makes the model generalize better on unseen datasets (see App. A.2.2
4 EXPERIMENTS
To compare our Duoduo CLIP to prior work, we conduct experiments showing shape classification on collection of 3D assets (on Objaverse-LVIS in Sec. 4.2.1) real-world multi-view images (on MVImgNet in Sec. 4.2.2). and more fine-grained text-to-shape retrieval (Sec. (.3). We also show image-to-shapp retrieval results for out-of-distributin images (App. A.4.I) and concept mixing (App. A.4.2 where we retrieve shapes that maximizes similarity to two input images. We first deseribe the experimental setup including implementation details and the datasets
4.1 EXPERIMENTAL SETUP
Implementation details. All of our models are trained with 16-bit mixed precision and batch size of 1600 for 80 epochs. We training step we randomly sample 1 to multi-views for a batch of objects. Ablations regarding the number of multi-views sampled during training can be found in App. A.2.1 The pretrained CLIP model used for initialization as well as the contrastive target is the ViT-B/32 CLIP model with checkpoint laion2b_s$3 Beedb____b_____ _3_34__649666 __99 from the open source implementation OpenCLIP (Ilharco et al. 2021). See App. A.2.3 we re-implemented OpenShape with 16-bit mixed precision training without projection layers for faster training in some baselines, indicated by OpenShapet.
learning rate of 5e 5 with cosine annealing. At each
for for a discussion of how initialization affects the performance. Additionally,
3D Datasets. We follow OpenShape (Liu et 2023a) and train our model on combination of four datasets consisting of Objaverse (Deitke et al. 2023). ABO (Collins et al. 2022). ShapeNet (Chang ctt al., 2015) and 3D-FUTURE (Fu et al. 2021). Text for the shapes consists of captions produced by image-captioning models and filtered text descriptions from metadata of the 3D objects crawled from the web (see Liu et al. (2023a) for details). In total, the combined dataset contains ~874k shapes, with 46k shapes within the LVIS subset of Objaverse which are used for evaluation. As in OpenShape (Liu et al. (023a), we consider two variations of the data: 1) Ensembled which contains all shapes, and Enxembled (no LVIS) which excludes shapes from the LVIS subset.
5

---

Published as a conference paper at ICLR 2025
55
52 57
52 M4.1 54
54.03
06
55.24
80
81 92.26
26
Top Ace
03.32
83:43
Objaverse: LVis Accurssy
50-
Duoduo CLIP
11,22
71.5672
28
75.32 75
Top
76:44
76.91
77
-05
Inferee Time (s/Rt)
0.4
45.
44.g0
43 20
43.47
64.50
43
93.79
40-
39
50
OpenCLIP BigG-14
6.38
0.3
-5-
33 34
35:13
50 50
52 37
53
54
Top Acc
5653
55.06
0.2
31.53
OpenCLIP 8:32
50
30-
44.
0.1
5
10
12
5
10 12
Frames
Frames
Figure 5: Left. We compare the Top accuracies on Objaverse-LVIS for our model (Duoduo CLIP) and two pretrained CLIP models (zero-shot) for different number of frames. B-32 is the model we initialize from and train with and BigG-14 14 is the model used by OpenShape. The dashed lines show the difference between using I and 12 frames. Right. We show the Topl, Top3 and Top5 accuracies for our model for to 12 frames. Gray dashed lines show the inference time(s) for iteration and batch size 200 at different number of frames.
Renderings. We use images from 1ero123 Liu et al. 2023b), which provides renderings of most shapes from Objaverse from 12 random views using spherical sampling. The shapes not included in Zero 23 (a few remaining objects in Objaverse and the other 3 datasets) are rendered using their Blender script to match the shapes in OpenShape. We render 12 views for all these objects. Additional details and effects of view selection can be found in App. App.AA1..2
Real-world data. Multi-view images. We leverage MVImgNet eYu al. 2023). dataset comprising multi-view images for 220k real-world objects across 238 categories. To focus on objects with sufficient views, filter those with least 12 views, evenly sampled, resulting in 1900 objects. For each view, we generate captions using BLIP-2 (Li et al., 2023). To facilitate comparisons with point cloud-based methods, we use the MVPNet subset, which includes ~87k objects with densely reconstructed point clouds. After preprocessing and filtering, we retain ~66k objects for the training split and ~66 for the validation set, spanning 180 categories. Importantly, the training and validation objects are disjoint but originate from the broader MVImgNet dataset. Reconstructed 3D data. To further evaluate our approach, we conduct experiments using objects from real-world scans. For instance, ScanObjectNN (Uy et al.] 2019) provides point clouds derived from objects in SceneNN (Hua et al.) 2016) and ScanNet (Dai et al. 2017). We evaluate on the test set, which contains 583 shapes across 15 categories, rendering 12 mieti-view images directly from the point clouds (see App. I for details). Additionally, we use the ScanNet validation set, consisting of 3825 objects across 17 classes, perform classification tasks. For this, we crop multi-view images from the raw captured RGB frames in the dataset (details in App. A.3.2
4.2
CLASSIFICATION EXPERIMENTS
4.2. I OBJAVERSE-LVIS
We conduct classification experiments on Objaverse-LVIS investigate the performance of differ- ent variants of Duoduo CLIP and compare our model to prior work. As default settings for our experiments in this section, we use 12 multi-views multi-view attention of the latter 6 layers of the model is trained (total 12 layers). We run evaluations with 3 different seeds and average the results for experiments. Unless otherwise stated models are trained with the Ensembled dataset. For classification, multi-view shape embedding with class embeddings. As in OpenShape, for each class label, we construct several sentences with the label and then take the average of the sentence embeddings as the class embedding. The final class chosen is the class with the highest average similarity between the class embedding and the shape embedding.
encode an object for evaluation. And only the
use do nearest neighbor search of the
We compare the performance of our model against two CLIP models for different numbers of frames in Fig. 5 and recent state-of-the-art models in Tab. 2 For the CLIP models, we selected VIT with

---

Published as a conference paper at ICLR 2025
Table 2: Classification accuracy on Objaverse-LViS (O-LVIS) and ScanObjectNN trained on En- sembled (no LVIS) and Ensembled. As baselines, the zero-shot CLIP models (ZS B-32 and ZS BigG-14) are not trained on 3D data, whereas the fine-tuned CLIP model (FT B-32) is trained using a single image per object. For each model, we indicate the representation used for computing the embeddings, multi-view (MV) or point-cloud (PC), and the shape encoder (Enc) architecture that is used. Our multi- based Duoduo CLIP is able to outperform point-cloud based approaches with the exception of Uni3D> is a large model with 1B parameters. indicates our reimplementation of OpenShape which is faster and outperforms the original OpenShape. F represents the number of frames/multi-vient used for evaluation.
Pretrain Dataset
Ensembbeed (no LVIS)
Envembled
O-LVIS
O-LVIS
ScanObjectNN Top3
Method
Rep
Enc
Top1
Top3
Top5
Top1
Top3
Top5
Top1
Top5
ZS B-32 (12F)
MV
Avg
35.7
62.1
35.7
62.1
53.9
73.5
81.2
ZS BigG-14 (12F)
MV
Avg
43.5
64.2
71.3
43,5
64.2
71.3
56.7
78.2
85,8
FT B-32 (12F)
MV
Avg
50.1
72.0
79.2
53.0
74.7
81.4
55.1
75.6
83,9
ULIP [Xue et al. 2023
PC
PoinBERR
21.4
46.0
26.8
52.6
51.6
72.5
82.3
OpenShape (Lil of 2023a
PC
PoinBERT
39.1
60.8
68.9
46.8
69.1
77.0
52.2
79.7
88.7
OpenShapel
PC
PointBERT
50.0
72.0
79.2
TAMM (Zhang al. 2024
PC
PoinBERT
42.0
63.6
71.7
50.7
73.2
80.6
55.7
80.7
88.9
MixCon DD Gao et 2024
PC + MV
PointBERT
47.5
69.0
76.2
52.5
74.5
81.2
58.6
80.3
89.2
Uni3D Zhou
2024
PC
3D VII
47.2
68.8
76.1
55.3
82.9
65.3
85.5
92.7
Shapel. LM Qi 2074
PC
RECON+
53.7
75.8
82.0
65.4
84, 1
89.7
VII-LENS (Let al 2024
PC
VII-LENS.
50.1
71.3
78.1
52.0
73.3
79.9
60.1
81.0
90.3
Duoduo CLIP (SF)
MV
MVA
51.3
79.9
54.1
82.4
60.7
82.4
88.5
Duodun CLJIP (12F)
MV
MVA
52.7
74.5
81.3
55.2
77.1
83.4
66.3
85.5
90.2
B-32 (the model that our Duoduo CLIP is initialized with), and BigG-14 (the CLIP teacher model used by OpenShape). Both models are run in a zero-shot manner without training on additional 3D data (the weights are frozen, with embeddings from the image encoder averaged). We also include an additional baseline that mirrors our default model's settings but trained by sampling a single view per object. This setup simulates the single-image training of the original CLIP model, and refer to it as FT B-32 in the tables.
How well does performance scale with number of views? Having more complete view of a 3D object is important for 3D representation learning as parts of the object may be occluded for some views. Thus we investigate how the number of views affects the classification accuracy. From Fig. (left), we see that our model performance scales better as the amount of information (views) increases compared to the zero-shot CLIP models (11.22 for our Duoduo CLIP, and 7.45 for B-32, the model we used for initializing our weights). This suggests that our training strategy of randomly sampling 1 to 6 views during training effective in letting the model learn to leverage the number of views given to it. Although, the accuracy increases as the number of multi-views increases, the inference time also increases (Fig. right). As the accuracy tapers off as we increase the number of views, to avoid additional computational cost during inference, we use 12 views as the default.
How many multi-views to beat a point cloud? Compared with using multi-views, using a point cloud representation offers good coverage of the entire geometry of the 3D object. Thus, we ask the question of how many views does it take to match existing methods on the Objaverse- LVIS classification task. We compare against ULIP (Xue et al., 2023). OpenShape (Liu et al. 2023a), TAMM (Zhang et (024) that use point clouds with PointBERT etu (0226) as the 3D encoder, but different ways of
the 3D encoder. We also compare against more recent methods (Lei) ct al. 2024 Qi et al. 2024 Zhou et al. 2024) that are the current top-performing methods on Objsverse-LVTS. From Tab. on the Ensembled dataset, we are able to beat most previous methods with 5 multi-views except for Uni3D which has a much larger model (1B parameters), When we increase to 12 frames we are on par and even surpass Uni3D. This is also while using much fewer resources as seen in Tab.
Does multi-views generalize better compared to point cloud? From the Ensembled (no LVIS) column of Tab. 2 we see that even the zero-shot BigG-14 14 model used as the teacher model for OpenShape beats it. Suggesting that the CLIP models already generalize well to unseen data, and point cloud methods potentially having worse generalization that is further trained on Ensembled (no LVIS) we surpass point cloud methods by a large margin. Showing that fine-tuning the CLIP model better preserves their generalization ability. Additionally, our model generalizes well on ScanObjectNN (Uy et al.) 2019) and outperforms point cloud methods on both Topl and Top3 accuracies (see App. A.3 for more results on ScanObjectNN and ScanNet).
unseen shapes. With our model
7

---

Published as a conference paper at ICLR 2025
Table 3: MVPNet classification comparison.
Table 4: Text to shape retrieval comparison. Meth- ods are trained on the Tex12Shape CChen et al. 2019
Method
Top
Top
Top
dataset (T2S) or Ensembled.
ZeroShot B-32 (12F)
52.68
70.99
77.22
FT B-32 (12F)
44.43
63.12
70.24
Model
Data
RR@!
RR @5
OpenShapef
10.80
19.62
25.20
Text2Shape Chen ct al. 2019
12S
0.40
2.37
OpenShapef (+MVPNet)
54.59
72.66
78.61
Y2Seq2Seq TTan el al. Parts2Words
2019
T2S
2,93
9.23
Tang el al. 2025
T2S
12.72
32.98
Ours (12F)
49.16
66.96
74.12
TriCol.
2024
T2S
12.22
32.23
Ours (+MVPNet) (1F)
59.23
76.12
81.74
MXM-CLR
Wang
2045
T2S
16.83
39.06
Ours (+MVPNet) (6F)
64.18
80.80
85.92
ZeroShot B-32
Images
27.85
Ours (+MVPNet) (12F)
64.44
81.11
85.97
FT B-32
Images
15.01
32.46
Ours (+MVimgNet) (IF)
61.75
79.07
84.18
OpenShapel
Ens.
10.53
25.94
Ours (+MVImgNet) (6F)
65.70
82.44
87.08
Uni
Ens.
10.78
26.37
Ours (+MVImgNet) (12F)
66.06
82.72
87.21
Duoduo CLIP
Ens.
15.90
34.08
Input
OpenShape (+MVPNet)
Duoduo CLIP (+MVPNet)
Figure 6: Retrieval results for OpenShape and Duoduo CLIP given input images from MVPNet. We perform retrieval with 12 input images, the leftmost column show one of the frames of the query object. For both methods we show the top retrieval results using Objaverse as a retrieval library. Overall, Duoduo CLIP retrieves shapes that are more similar the query image, while OpenShape sometimes retrieves broken geometry (row 3).
4.2.2 MVIMGNET
To investigate how well our Duoduo CLIP works on real-world images, we evaluate on multi-view images from MVImageNet. For these experiments, take our Duoduo CLIP model trained on the Ensembled dataset and add additional training data from MVImgNet. We consider two sets of data from MetmgNet, the MVPNet training split of objects the entire MVImgNet (after preprocessing and filter). For comparison with point cloud based methods, we evaluate on the validation set of MVPNet. As a representative of point cloud based methods, choose OpenShape as it is relatively resource-light compared to other point cloud methods.
Classification results. We report accuracy results MVPNet in Tab. For OpenShape, the top-1 accuracy is very low (only 10.8% top-1) when evaluating on MVPNet directly with the model trained on Ensembled only. We suspect this is due noisy reconstructed point clouds and inconsistent orientations compared to the training data. Note we re-align MVPNet point cloud's 's up direction with Objaverse. After training on MVPNet data, OpenShape's performance improves significantly. For our Duoduo CLIP, even without training on MVImgNet or MVPNet data, our model achieves a top-1 accuracy 49. 16%. However, this is lower than the zero-shot CLIP B-32 model that our model is initialized from, suggesting possible overfitting to synthetic data. When we include training with multi-view images from MVPNet, our method surpasses both OpenShape and the zero-shot CLIP B-32 model even when just using one 1 view. Performance is further boosted when trained on the entire MVIngNet, highlighting the advantages our Duoduo CLIP which uses multi-view images OpenShape's point clouds. By using multi-view images, have better generalization
8

---

Published as a conference paper at ICLR 2025
unseen data due to the priors from initializing with CLIP B-32. We also can train more data, as multi-view images are more readily available compared to point clouds.
Retrieval examples. In Fig. 6 we present example retrieval results for querying Objaverse objects using images from the MVPNet validation set. We compare retrievals using OpenShape (+MVPNet) and Duoduo CLIP (+MVPNet).. For OpenShape, them with the frozen BigG-14 CLIP model, and average the image embeddings to obtain the query embedding
use 12 query images for each object, encode
D This embedding is then matched against a database of point cloud embeddings of Objaverse objects to retrieve the top 3 results. For our method, Duoduo CLIP encodes both the query and Objaverse objects using 12 multi-view images.
From Fig, 6 we see that for basic objects like benches, both methods are able to retrieve plausible objects. For the armchair (row 2), while OpenShape retrieved objects of the same semantic class, our Duoduo CLIP retrieved armchairs that has better matching textures. This is because multi-view images can represent textures better than point clouds. Note that some retrieved objects look similar due to duplicate objects in Objaverse. For the jacket (row 3), the objects returned by OpenShape have broken geometry and are incoherent. We hypothesize this may happen when the sampled points produce embeddings that are by chance similar to the query image. This issue is also present last example (a red and black soccer ball) where the first object retrieved by OpenShape has broken geometry. However, and black dots that resemble the soccer ball. Interestingly, OpenShape is able to retrieve a soccer ball and red ball. This can indicate that point cloud based methods may be more robust to geometric properties of the object compared to the rendered images. However, we find that overall the retrieved objects from our method are more consistent.
method also fails in this case, our model retrieves ladybugs with red wings
4.2.3 MODEL ABLATIONS
Table 5:
Ablation on the number of layers for accuracy Objaverse- LVIS using 12 input frames. Default model highlighted in gray.
Table 6: Multi-view attention (MVA) ablation of accu- racy on Objaverse-LVIS (O-LVIS), MVPNet and ScanOb- jectNN with 12 frames. Default model highlighted in gray.
Method
Layers
O-LVIS
MVPNet
ScanObjectNN
Method Top 1
Top 3
Top 5
-MVA
6
54.61
47.87
58.77
layers
53.77
75.8
82.41
+MVA
6
55.24
49.16
66.32
layers 55.24
77.08
83.43
-MVA
12
55.02
43.75
56,83
12 layers (full) 55.32
77.08
83.49
+MVA
12
55.32
44.42
64.15
Do we need to train all the layers? We conduct an ablation to see if necessary to train all 12 layers. In Tab. [5] we show the accuracy for restricting the training to the last 3 and 6 MVA layers vs the full 12 MVA layers. We see that the number trainable layers can be cut by half with minimal losses to accuracy. while further cuts start to hurt model performance. Training only half the layers reduces the VRAM usage by roughly half and allows us to train on GPUs with less memory. By restricting the training the last 6 MVA layers, 4 A5000s (24 GB) (see Tab. D. Also we find that training less layers also helps preserve the priors of the model and make generalization
shift from training on 4x A40s (48 GB) to
other datasets better seen in Tab.
Does multi-view attention help? We verify the effectiveness of our cross view attention in Tab. 6 +MVA and -MVA denote whether the cross-views attention is enabled. On Objaverse-LVIS, while the +MVA mechanisms help less +0.3 when all 12 layers are trainable, it makes larger impact when only training layers +0.63. Making the +MVA mechanism an effective way to maintain performance while being more efficient compared to training all 12 layers. In addition, we find that the +MVA also helps in making the model generalize better on unseen datasets like MVPNet and ScanObjectNN compared to -M. Inference speeds for the models can be found in App. A.I.I
4.3 FINE GRAINED TEXT-TO-SHAN RETRIEVAL
Previous evaluations focused on classification or image-based retrieval and do not assess how well the methods align ime-ggrined text descriptions with 3D models. To evaluate this aspect, we use the test
For the largest B1gG-14 CLIP model, OpenShape does use any linear projection layer g'

---

Published as a conference paper at ICLR 2025
Uni3D
Duoduo CLIP
cream coloured chair with printed man image and with side arms
A brown chair with round back and thin metal base.
Figure 7: Retrieval using fine-grained text from the Text2Shape dataset. On the left the input text description and we show the top retrieval results for Uni3D and Duoduo CLIP with the ground truth shape highlighted in dotted boxes.
split of Texi2Shape (Chen et al. 2019), which contains 11.4k tables and chairs from ShapeNet and ~7.4k human-captioned descriptions (~5 per shape). We report the shape retrieval rates at and 5,
Retrieval performance. We compare the performance of our model to prior work in Tab. 4 We also include previous state-of-the-art methods that were trained on the Texr2Shape dataset. Overall, compared to other methods trained the Ensembled dataset, performance. The zero-shot performance alone is impressive, surpassing point cloud methods. Notably, we observed significant discrepancy between the zero-shot CLIP model reported by Ruan eet al.] (2024) and our findings, where the LAION checkpoint trained by OpenCLIP significantly outperforms the original OpenAI checkpoint. Lastly, our model approaches the performance of the SOTA method trained directly on Text2Shape, indicating that there is still much to explore regarding more fine- -rained text alignment in large-scale text-to-3D models.
method demonstrates superior
Retrieval examples. We present example retrieval results compared to Uni3D in Fig. 7 Generally, for text descriptions emphasizing complex textures, such as the 'man image" in the first row, our model successfully retrieves the corresponding ground truth 3D model. However, for descriptions focusing on geometric properties, like 'round back' in the second row, point cloud-based methods tend to retrieve shapes that better match the specified criteria.
CONCLUSION
We introduced Duoduo CLIP, which uses multi-view image encoder to represent shapes and align the embedding to pretrained text-image models. We demonstrated that contrastive learning with multi-view images is an efficient and effective way to encode 3D shapes. Our experiments show that using multi-view images our model surpasses current point cloud state-of-the-ard methods, trained in a similar fashion, on both real and synthetic data while using significantly fewer resources.
Multi- view images have limitations. They potentially poorer geometric understanding compared to point clouds. One way to alleviate this is to incorporate additional inputs such as depth or normal images, which provide more geometric cues. Our work is also limited in only exploring CLIP-style contrastive pretraining. Our model can be easily adapted to other contrastive pretraining Cloel et al.) (2022); /Zhai et al. (2023) and SSL methods (Oquab al. 2023 Zhou et al. 2022). This is a promising direction as recent work (Banani et al.) (024) points out the relative weakness of CLIP pretraining on both 3D structure and consistency benchmarks. Another important direction is examine how to increase the data available for training. Even with a relatively small model as ours, training too many layers can lead to severe overfitting. This suggests that data augmentation and incorporating more 3D data would lead to improved generalization and performance.
By advocating for reconsideration of multi-view images for shape representation, we believe our work can convince 3D shape researchers to re-examine when point clouds are useful. Images offer several advantages over point clouds, including better capture of texture and color information, less domain gap with pretrained vision models, and better resource scaling. They also offer greater flexibility for practical use cases in real-world applications such as robotics. Leveraging the texture information captured by multi-view methods like ours complement the geometric understanding of point cloud techniques. A hybrid approach combining both modalities could leverage their respective strengths and adapt to variety of use cases in the future.
10

---

Published as a conference paper at ICLR 2025
Acknowledgements. This work was funded by CIFAR AI Chair, NSERC Discovery grant, and a CFI/BCKDF JELF grant.
REFERENCES
Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proc. of the Conference of the Association for Computational Linguistics (ACL). 2020. URL https
//doi.org//1 485....////......22200
00928
Mohamed El Banani, Amit Raj. Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3D awareness of visual foundation models. Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR), 2024. URL httpt:tps:/dee org/ 10. 18550//rxiiy 2404. 08636
Gary Bradski and Stephen Grossberg. Recognition of 3-D objects from multiple 2-D views by a self-organizing neural architecture. In From Statistics Neural Networks: Theory and Pattern Recognition Applications, pp. 349-375. Springer, 1994.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self- supervspe vision transformers. In Proc. of the International Conference on Computer Vision (ICCV), pp. 9650-9660, 2021. htthst
7/doi. aaboi.ogg/1.. w05.448555....222/. 21C.XX.2210411944 14294]
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. ShapeNet: An information-rich 3D model repository, arXiv preprint 1rxiv:151..03012, 2015. /RLL:
48550/arxiv 1512.11156 03012.
Kevin Chen, Christopher Choy, Manolis Savva, Angel Chang. Thomas Funkhouser, and Silvio Savarese. Text2shape: Generating shapes from natural language by learning joint embeddings. In Proceedings of the Asian Conference on Computer Vision (ACCV), pp. 100-116. Springer, 2019.
URL https: doi www.oogg///.oo 10. 115555//axii1 1803. 08495
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5939-59483 2019. URL
httpp:://dot wwo.rrll....... 48550/855440 arwiv. 1812.1112... 02822
Jasmine Collins, Shubham Goel, Kenan Deng. Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas Yago Vicente, Thomas Dideriksen, Himanshu Arora, et al. ABO: Dataset and benchmarks for real-worid 3D object understanding. In Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR), pp- 21126 21166,,,, 2022. URL htthhttt///ddd /rg/
10. 40550850//4 arxiv.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In Proc. of the Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 5828-5839, 2017. URL
https://dolt org/10 48550/ arXiv. 1702 04405
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3D objects. In Proc. of the Conference Computer Vision and Pattern Recognition (CVPR), 2023. URL / 4tt.///.o.o0010.00//////a/aaa
2212 08051
Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Shanmukha Ramakrishna Vedantam. Hyperbolic image-text representations. In Proc. of the International Conference on Machine Learning (ICML), pp. 7694-7731. PMLR, 2023. URL http:tttss//oo.orgggtttt
48550/ 2w0.xv.v304... 091722
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Proc. of the International Conference on Learning Representations (ICLR), 2021. URL https: //doi. http///...0g/11..455000///////0/ wtw.:/.44550/00......o. 201
11999

---

Published as a conference paper at ICLR 2025
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Proc. of the International Conference on Machine Learning (ICML), 2024. htIhttts:
wrg. A84588555551/. 2.033 03206
Fartash Faghri, David Fleet, Jamie Ryan Kiros, and Sanja Fidler. VSE++: Improving visual-
semantic embeddings with hard negatives. In Proc. of the British Machine Vision Conference (BMVC), 2018. http:ttts///do..ooo
48550/
1707. 05612
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. Advances in neural infor- mation processing systemss, 26, 2013. URL IRt. nips ct/pcc/ppppacc files/ pdd.
Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao. 3D-FUTURE: 3D furniture shape with texture. Internatioonal Journal of Computer Vision (IJCV), 129:3313-3337, 2021. (ttp:t:::: /ttt:p::/doi..ox// 100 48550/ arxiv..iii..i 1.000099333..
Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song. CoWs pasture: Baselines and benchmarks for language-driven zero-shot object navigation. In Proceedings of the IEEF/CVE Conference on Computer Vision and Puttern Recognition, pp-
23171-23181, 2023. URI.httpp /ow.. http://oooooog///// 00000.885550 48550/ arxiv 2203. 10421
Yipeng Gao, Zeyu Wang, Wei-Shi Zheng, Cihang Xie, and Yuyin Zhou Sculpting holistic 3D representation contrastive language-image-30 pre-training. Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22998-22982 230008, 2024. URL https
//doi. /0 48550/ arxiv. 2311. 01734
Sourav Garg, Krishan Rana, Mehdi Hosseinzadeh, Lachlan Mares, Niko Sünderhauf, Feras Dayoub, and Ian Reid. RoboHop: Segment- based topological map representation for open-world visual navigation. In International Conference on Robotics and Automation (ICRA), 2024. URL https 7/doi. org/10 48550/ 2405. 2479.
Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan Rossi, Vishwa Vinay, and Aditya Grover. CyCLIP: Cyclic contrastive language-image pretraining, Advances in Neural Information Processing Sys- tems, 35:6704-6713 2022.
org/ 10 11.11........ 48550/8550/00 arxiv. 2205. 14499
Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. MVTN: Multi-view transformation network for 3D shape recognition. In Proc. of the International Conference on Computer Vision (ICCV), 2021.
48550/ arXiv. 2011 2121411224
Zhizhong Han, Mingyang Shang, Xiyang Wang. Yu-Shen Liu, and Matthias Zwicker. Y2Seq2Seq: Cross-modal representation learning for 3D shape and text by joint reconstruction and prediction of view and word sequences. In AAA/ Conference on Artificial Intelligence. volume 33, pp. 126-133, 2019. URL org 10 48550/555330 arwiv. 1811 02745
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16000-16009, 2022. URL http::::///oi...//.....o../
arXiv. 2111 06377
Binh- Son Hua, Quang-Hice Hieu Pham, Duc Thanh Nguyen, Minh-Khoi Tran, Lap-Fai Yu, and Sai- Kit Yeung. SceneNN: A scene meshes dataset with annotations. In Proc. of the International Conference on 3D Vision (3DV), pp. 921 IEEE, 2016. URL httpt:tt///// pretps://doi. /ttp:///ooooooo/////////
1109/3DV.. 2016 18
Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. In International Conference on Robotics and Automation (ICRA), pp. 10608-10615. IEEE, 2023a. http::tt:///ot http::///ooooo././////// /tt.g//1..855100 118885505aXiiii 2210 05714
Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, Rynson WH Lau, Wanli Ouyang, and Wangmeng Zuo. CLIP2Point: Transfer CLIP point cloud classification with image-depth pre- training. In Proc. of the International Conference on Computer Vision (ICCV), pp. 22157-22166 2023b. URL htl.ttt::: htpp:///do //ooi .erooioorg/oo wtt.///...8.5//aaai.......... 2ttp:///wxwv22.22..0.222 22055
12

---

Published as a conference paper at ICLR 2025
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. OpenCLIP, July 2021. URL https: //dp:::::/do..oo/ ottp1o.rrg10..000 5281/ zenodo 5 5143773
Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Alaa Maalouf, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, Joshua Tenenbaum, Celso Miguel de Mclo, Madhava Krishna, Liam Paull, Florian Shkurti, and Antonio Torralba. ConceptFusion: Open-set multimodal 3D mapping. In Robotics: Science and Systems 48550
(RSS), 2023. URL
0724]
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le. Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Proc. of the International Conference on Machine Learning (ICML), pp. 4904-4916. PMLR, 2021. htthhttp:: .ww.oooorrgoooo htt.://..00/../...//// arxiv http:////wwvv....2...... 2102 05918
Ryan Kiros, Ruslan Salakhutdinov, and Richard $ Zemel. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539, 2014. URL https
//doi. , org. 10. (8550//aXiiy/ 1411...1....1139
Weixian Lei, Yixiao Ge, Jianfeng Zhang, Dylan Sun, Kun Yi, Ying Shan, and Mike Zheng Shou. VIT-lens: Towards omni-modal representations. In Proc. of the Conference Computer Vision and Pattern Recognition (CVPR), 2024. URL https:: https::/ddo..ooo www.//i..oogg..../ 48550044550508851
16081
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proc. of the International Conference on Machine Learning (ICML), pp. 19730-19742. PMLR, 2023. URL https://
doii. 10. 48550/arXiv 2301. 12597
Dongyun Lin, Yi Cheng, Shangbo Mao, Aiyuan Guo, and Yiqun Li. PEVA-net: Prompt-enhanced view aggregation network for zero/few-shot multi-view 3D shape recognition. arXiv preprint arXiv:2404. 19168, 2024.
0tt///10.485050 48550445555111 2404. 19168
Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. OpenShape: Scaling up 3D shape representation towards open-world understanding. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Ad- vances in Neural Information Processing Systems, volume 36, pp. 44860 44879. Curran Associates. files/paper 000011008004477222000000000000e0
Inc., 2023a. URL ht
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image 3D object. In Proc. of the International Conference on Computer Vision (ICCV), pp. 9298-9309, 2023b. URL https:://dda http://66o..og//..4455535
arxii.. 2303. 11328
Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3D captioning with pretrained models. Advances in Neural Information Processing Systems, 36, 2024. URL https:
//doi.org//i1 48550/ 2306. 07279]
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. SLIP: Self-supervision meets language-image pre-training. Pruc. of the European Conference on Computer Vision (ECCV), pp. 529-544. Springer, 2022. URL //do:: http:://doooooo//////o 10 48550/ 221..1112.. 12750
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint (rXiv:1807.03770 2018. URL http:s:://cc...rg//0..666
arxiv.187..
S..iv.1807.. 1rXiv.1807.031 03748
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research (TMLR), 2023. URL httphhtt///eet http://////ooooooooo.ooo/
org
2304 07193
13

---

Published as a conference paper at ICLR 2025
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. In Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 652-660, 2017. URL [ttpss///oc..crr100..8885 arXiv. 1612.005 00593
Zekun Qi, Runpei Dong. Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, and Yi. Contrast with reconstruct: Contrastive 3D representation learning guided by generative pretraining. In Proc. of the International Conference on Muchine Leurming (ICML), pp. 28223-28243, PMLR, 2023.
URL
48550/
(5550/arXiv. www.ia.vv...... 2302. 02318
Zekun Qi, Runpei Dong. Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, and Kaisheng Ma. LLM Universal 3D object understanding for embodied interaction. arXiv preprint arXiv:2402. 2024. URL htttp:://do..orr
48550/
17766
Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. PointNeXt: Revisiting PointNet++ with improved training and scaling strategies. Advances in Neural Information Processing Systems, 15:23192-232000 2022. URL
htttps://doi... owg.o..orgg/..... 48550/ arxiv 2206 04670
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proc. of the International Conference on Ma- chine Learning (ICML), pp. 8748-8763. PMLR, 2021. URLhttps: /tt.i//iiiiaa// http:///....///1.885558855 arXiv. 2103.00020
Yue Ruan, Han Han-Hung Lee, Yiming Zhang, Ke Zhang, and Angel X Chang. TriCoLo: Trimodal contrastive loss for text shape retrieval. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision (WACV), pp. 5815-5825, 2024, URL https: org/
To. 004448555ax1111 2201 07366
Yu-Te Shen, Ding- Yun Chen, Xiao-Pei Tian, and Ming Ouhyoung. 3D model search engine based on lightfield descriptors. In Eurographics (Posters). 2003.
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. MVDream: Multi-view diffusion for 3D generation. In Proc. of the International Conference on Learning Representations (ICLR), 2023. URL htLppttss///...ccmmm
10. 48550/ 2308 16512
Mohit Shridhar, Lueas Manuelli, and Dieter Fox. CLIPort: What and where pathways for robotic manipulation. In Proc. of the Conference on Robot Learning (CoRL), pp. 894-90.. PMLR, 2022.
URL htttt: htt::/oo.ooogoooo 100 48550/ arXiv. 2109. 12098
Dan Song, Xinwei Fu, Weizhi Nie, Wenhui Li, and Anan Liu. MV-CLIP: Multi-view CLIP for zero-shot 3D shape recogmition. arXiv preprint arXiv:231..18402, 2023. URL https://doi
org/10. 00t//10.4555//a220 wtw..xi.....1.1113.
Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller Multi-view convolutional neural networks for 3D shape recognition. Proc. the International Conference on Computer Vision (ICCV), pp. 945-953, 2015. URL httph:tts///e //tp:: http///o..oorog////// wrw.ooog//o.ooo..oo 48550/arXiv. arxiv. 1505 00080
Chuan Tang, Xi Yang, Bojian Wu, Zhizhong Han, and Yi Chang. Parts2words: Learning joint embedding of point clouds and texts by bidirectional matching between parts and words. In Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1884-6893, 2023.
48550//004450///// Aaxii.vie 2107. 01872
Jesse Thomason, Mohit Shridhar, Yonatan Bisk, Chris Paxton, and Luke Zettlemoyer. Language grounding with 3D objects. In Conference Robot Learning, pp. 1691-1701, PMLR, 2022. URL
htttp:://boi.rgg// 48550//4445333//
2107 12514
Mikaela Angelina Uy. Blang--Hiu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model real- world data. In Proc. of the International Conference on Computer Vision (ICCV), pp. 1588-1597,
2019. URL URt.tttss: http::/://.o..oooo0000050///////
4055000..5500 1908 04616
14

---

Published as a conference paper at ICLR 2025
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. URL https: wtt.://......1....555.03.01000.0
10. 48550/arXiv.....
03762
Hongcheng Wang, Andy Guan Hong Chen, Xiaoqi Li, Mingdong Wu, and Hao Dong. Find what you want: learning demand-conditioned object attribute space for demand-driven navigation. Advances in Neural Information Processing Systems, 36, 2024. /tttttss:///oi
arXiv. 2309. 08138
Ye Wang, Bowei liang, Changqing Zou, and Rui Ma. MXM-CLR: A unified framework for contrastive learning of multifold cross modal representations. arXiv preprint arXiv:2303. /0839, 2023. URL
https://do: /ttt://///oo..ooo/00.0005////// http://0...45000/1...232.20
10839
Xin Wei, Ruixuan Yu, and Jian Sun. View-GCN: View-based graph convolutional network for 3D shape analysis. In Proc. of the Conference Computer Vision and Pattern Recognition (CVPR), 2020. 00192
pp. 1850-1859, 2020. URL URL https:/ https://doi..oom org wwt.:111////2222.
Jason Weston, Samy Bengio, and Nicolas Usunier. WSABIE: Scaling up to large vocabulary image annotation. In International Joint Conference on Artificial Intelligecee 2011. URL
https://dl. acm. wtt.m..rgg//o111.5555522222222
2283856
Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. PointContrast: Unsupervised pre-training for 3D point cloud understanding. In Proc. the European Conference on Computer Vision (ECCV), pp. 574-591. Springer, 2020. URL
48550/arXiv. 2007 10985
Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. ULIP: Learning a unified representation of language, images, and point clouds for 3D understanding. In Proc. of the Conference Computer Vision and Pattern Recognition (CVPR), pp. 1179-1189, 2023. URL /tttps://doi..og/110..o arXiv 2212. 05171
Le Xue, Ning Yu, Shu Zhang, Junnan Li, Roberto Martin-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. ULIP-2: Towards scalable multimodal pre-training for 3D understanding, In Proc. of the the Conference on Computer Vision and Pattern Recognition (CVPR), 2024. URL https: /tttps://doi.ogg/
48550/ (55555/aXiiii wrw.v.iiiiv..... 2305. 08275
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-labee space. In Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19163-19173, 2022. URL WRL.hhtps://ddo org/
10. 48550// arwii.ivvv...... 2tt.l..2244003.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. CoCa: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research (TMLR), 2022a. URL
org/ /tceg///...
01917
Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. MVIngNet: A large-scale dataset of multi- view images. In Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9150-9161, 2023.
htLhstt:: / http://ooo...////////// 48550/ arxiv. 2303. 06042
Xumin Yu, Lulu Tang. Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-BERY Pre- training 3D point cloud transformers with masked point modeling. In Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19313-19322, 2022b. (ttttt
7/doi .ww./rgg///.. 100.g//0..0 48550/arXivi 2111..11111.. 14819
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. LIT: Zero-shot transfer with locked-imee -migge text tuning. In Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18123-18133 2022. URL https
485.0/.88555aa/ www..xwww......... 079911
15

---

Published as a conference paper at ICLR 2025
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11975-11986, 2023. http::////a org.////00.g/0 48550/ http5//axxwi......... 10333.5333333
Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. PointCLIP: Point cloud understanding by CLIP. In Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8552-8562 2022a. URL https:
77 //doi.co boi.org/10 485.0..85550/xx1x1.. www.a.iwii.................. 2112.02413
Yuhao Zhang, Hang Jiang. Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Con- trastive learning of medical visual representations from paired images and text. Machine Learning for Healthcare Conference, pp. 2-25, PMLR, 2022b. URL
18550/arXiv.20111 00747
Zhihao Zhang, Shengcao Cao, and Yu-Xiong Wang. TAMM: Triadapter multi-modal learning for 3D shape understanding. In arXiv preprini arXiv:2402.18490, 2024. URL https://dbi.ors
10. ww.4405/0xx...24555 2402 18490.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. iBOT: Image BERT pre-training with online tokenizer. Proc. of the International Conference on Learn- ing Representations (ICLR), 2022. URL http:::/ddo..org//// 48550/arXiv. 2111
07832
Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3D: Exploring unified 3D representation scale. In Proc. of the International Conference on Learning Representations (ICLR), 2024. URL httpps://doi.oto 18.5
2310.06773
Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Ziyao Zeng, Zipeng Qin, Shanghang Zhang, and Peng Gao. PointCLIP v2: Prompting CLIP and GPT for powerful 3D open-world learning. In Proc. of the International Conference on Computer Vision (ICCV), pp. 2639-2650, 2023. URL https: //d:://do: wwwgrrloo.................o..... 10 48550/ arxiv. 2211 11682
16

---

Published as a conference paper at ICLR 2025
A APPENDIX
In this appendix, we provide details of inference speed IA.III. camera sampling (A.1.2). well as ablations of the number of multi-views used during training (A.2.I). MLP fine-tuning (A.2.2). and different initialization of encoders (A.2.3). We also include experiments real-world scanned datasets A.3)). examples of out-of-distribution retrieval (A.4.I). concept mixing (A.4.2). and details
of the attention map visualization (A.5.I).
For the ablation experiments in the appendix unless otherwise mentioned, we follow the same default settings in the main paper. In this default setting, models are trained on the Ensembled dataset and we randomly sample to 6 multi-views during training, with the last 6 attention layers trainable and +MVA enabled using a batch size of 1600.
A.1 INFERENCE SPEED AND CAMERA SAMPLING
A. 1.1.1 INFERENCE SPEED
We show the inference speed between different number of trainable layers and frames for -MVA and +MVA in Tab. [7] We benchmark total run time and time per iteration the entire Objaverse-LVIS (46k shapes). Note that -MVA with 6 layers is not shown, since the run time is the same with -MVA 12 layers during inference since both models have multi-view attention disabled. There is only slight increase in inference time when +MVA is enabled, and it remains efficient for processing a large number of shapes.
Table 7: Inference speed on Objaverse-LVIS with batch size 200. Default model is highlighted in gray.
Table 8:
Different camera sampling settings. Here the elevation is the angle relative to the horizon, with 0 degrees at the horizon, positive angles looking from upward, and negative angles looking from downward.
Method
Layers
Frames
Speed (sfit)
Total Time (8)
+MVA
0.14
32.80
+MVA
0.28
65.52
+MVA
6
12
0.56
130.67
Setting
Radius
Azimuth
Elevation
-MVA
0.14
33,23
Upperl
[1.5, 2.2]
[0, 360)
[0, 60]
-MVA
12
0.28
63,90
ExtendedHHem
[1.5, 2.2)
(60)
[-30. 60)
-MVA
12
12
0.54
126.28
FullSphere (7eml23)
[1.5, 2.2]
[0. 360]
[-90, 90]
+MVA
12
0.15
34.01
+MVA
12
0.29
68.38
+MVA
12
136.10
A.1.2 CAMERA SAMPLING
To verify the effectiveness of our camera sampling strategy, we benchmark different camera settings (see Tab. B. Example renders for each of the different settings as well as the quantitative results can be found in Fig. X The results show clear performance improvement as the elevation range increases, suggesting that capturing the object from more extreme angles also provides more context and information about the shape.
A.2 OTHER ABLATIONS
A.2.1 NUMBER OF MULTI- VIEWS DURING TRAINING
For our default model we choose to sample to 6 multi-views during training as strikes a good balance between memory usage and performance. In Tab. 9 we show the Objaverse-LVIS accuracy over a variation of sampled views seen during training with other settings kept the same as the default model. Except for the model that is trained with only single view ie. [1, 1] which uses -MVA it does not see any multi-views during training.
It can be that the number of multi-views seen during training increases. does its performance when evaluating at 12 frames during inference. However, the memory usage also rises quickly as attention has quadratic growth when the the number of tokens increases in the case of +MVA. So
17

---

Published as a conference paper at ICLR 2025
$4,93
20.24
$3.62
94.51
54,05
54,30
UpperHem
52 5240
59.71
$4,07
$2.51
50.0
Accuracy
ExtendedHem
69
natSphere
FullSphere
43875
Extendidhem
Uppertiem
Number of Frames
Figure 8: Left. We show example renders from each of the different camera settings. Right. We show the Top 1 accuracies of the different camera settings using our default model +MVA (6 layers).
choose to sample [1, 6] views as there an increase of +0.24 on top1 over training with [1, 4] views. However, increasing further [1. 8] frames only results in a +0.09 topl. Also the amount of memory [1, 6] also allows us to stay within the memory limits of an A5000 (24 GB) GPU.
Table 9: Number of views sampled during train- ing on Objaverse-LVIS ablation. Evaluated using 12 frames per shape during validation. The mem- ory usage is reported on a single GPU with 400 batch size. Default model is highlighted in gray.
Table 10: Ablation of also training the MLP lay- ers after attention on Objaverse-LVIS, MVPNet and ScanObjectNN. Evaluated using 12 frames of each shape during validation. Default model is highlighted in gray. Trained with batch size 1400 to in memory.
Num Frames Train
Top1
Top3
Top5
Mem (GB)
[1, 1]
52.97
74.71
81.44 .4
5.14
Method
Layers
O-LVIS
MVPNet
ScanObjectNN
[1,2]
54.47
76.37
82.82
9.12
+MVA
49.16
66.32
[].4]
55.00
76.93
83.36
15.50
+MVA
12
55.32
44.42
64.15
[]
55.24
77.08
83.43
22.18
+MVA MLP
55.96
45.46
59.63
[1.8]
55.33
77.30
83.62
28.92
+MVA MLP
12
55.63
32.68
60.20
A.2.2 MLP FINE-TUNING
Only fine-tuning the attention layers of the model not only makes the model more efficient to train, but also helps to preserve the priors of the original CLIP model. To showcase this we also enable the training of the MLP layers corresponding to the attention layers being tuned in Tab. TO Although find that Objaverse-LViS performance improves when more of the network (MLP) is allowed to be tuned. It results in worse performance the other unseen datasets. Our default model performs the best across the other unseen datasets. Note that although our default model also overfits and performs worse MVPNet compared the zero-shot model as seen in Tab. 3, the performance drop-off is smaller compared to the other models.
A.2.3 DIFFERENT INITIALIZATION
To understand the importance of initializing the multi-view shape encoder from the same model the frozen CLIP encoders we conduct additional experiments by using a different initialization. Namely, we use the same ViT-B/32 architecture, but initialize the shape encoder from the OpenAl (Radford al. 202T) pretrained model and the frozen CLIP model uses the checkpoint laion2b_s.33 b_99 from the open source implementation OpenCLIP (Iharco al. 2021). Note that add additional linear layer for the embeddings of the OpenAl initialized model. The results can be found Tab.
From the results, we in a large drop-off in performance for both Objaverse- LVIS and ScanObjectNN. This shows the importance of using the same initialization for both shape and frozen CLIP encoders. As the embedding space outputted by the models are already close and allows for smoother fine-tuning and prevents potential "overfitting" to the frozen target text and image embeddings. However, expect
that using different initialization from the frozen clip model results
18

---

Published as a conference paper at ICLR 2025
that this problem will diminish with larger datasets, allowing for the use of foundational models for the shape encoder such as the DINO models (Caron et al. 2021 Oquab
powerful 2D
et al. 2023 that also provide better 3D Awareness (Banani al. 2024).
Table 11: Results of using different initializations for the shape encoder. Same Init follows the settings of the main paper which uses the OpenCLIP pretrained model for both the shape and frozen CLIP encoders. OpenAl Init uses the pretrained model from OpenAl for the shape encoder instead. We evaluate on Objaverse-LViS and ScanObjectNN with views. We show results for models with different number of trainable layers.
O-LVIS
ScunObjectNN Top3
Method
Layers
Top1
Top3
Top5
Topl
Top5
OpenAl Init
53.0
75.2
82.1
60.8
78.6
86.6
OpenAl Init
12
53.4
75.7
82.5
56.8
78.9
86.8
Same Init
55.2
77.1
83.4
66.3
85.5
90.2
Same Init
55.3
77.1
83.5
64.2
85.0
90.8
A.3 OTHER DATASETS
A.3.1 SCANOBJECTNN
To evaluate on ScanObjectNN (a point cloud dataset), we have to first obtain the multi-view images for each object. For simplicity, (OBJ_BQ BG) directly as can be seen in Fig. 9 Note that ScanObjectNN objects were obtained from real-world scan datasets that were captured with RGBD cameras. So using the raw frames would be more straightforward and easier to obtain compared to point clouds since it does not require reconstruction. However, we weren't 't able to obtain the mappings the original scenes from the processed ScanObjectNN dataset so opted for this approach instead.
rendered the 12 multi-view images using the point clouds
We show the quantitative results on ScanObjectNN in Tab. [2] It can be seen that the +MVA mechanism outperforms -MVA by large margin in this scenario. And that our default model with 12 frames are able outperform all previous point cloud methods on both top and top accuracies. This is despite the fact that we used rendered point clouds images opposed to the originally captured RGB scene data. This demonstrates the robustness and adaptability of our model to handle wide range of different images.
Table 12: ScanObjeetNN Results. Our +MVA model significantly outperforms zero-shot CLIP and -MVA. We also out- performs prior work using point clouds, in- cluding Uni3D on Top I accuracy.
Method
Frames
Top
Top
Top 5
ZS B-32
37.2
60.3
71.6
7S B-32
6
50.9
72.4
81.2
ZS B-32
12
53.9
73.5
81.2
-MVA
1
41.2
62.8
74.4
-MVA
6
53.7
75.9
84.4
MMA
56.8
78.4
85.5
OpenShape
PC
$2.2
79.7
Recon++
PC
65.4
84.1
89.7
Uni3D
PC
65.3
85.5
92.7
+MVA (6 layers)
1
41.7
64.4
76.4
Figure 9: Example of nulli-view image renders of ScanObjectNN.
+MVA (6 layers)
6
61.7
82.7
89.2
+MVA layers)
66.3
90.2
A.3.2 SCANNET
For ScanNet, we generate the multi-view images using the officially released projections from the 3D segmentations onto the original 2D RGB frames. We then the frames based the projected
19

---

Published as a conference paper at ICLR 2025
Table 13: ScanNet results with cropped video frames evaluated over 12 multi-views. ZS is the zero-shot B-32 model. Ens and Ens + MVT is the default model trained with ensembled dataset and ensembled with MVImgNet datasets respectively.
Method
Avg.
Tabi
Fridg
Toil
Sink
OpenShupe
45.6
66.7
3.2
75.8
83.5
37.7
49.2
47.5
64.9
48.2
1.9
66.1
70.2
1.8
50.0
57.1
45.2
TAMM
49.4
66.7
4.8
83.6
84.5
48.9
57.9
48.2
80.5
61.3
1.9
60.6
83.6
7.0
41.4
56.1
48.4
Uni3D
45.8
3.7
54.9
31.3
39.4
70.1
35.1
27.3
38.7
88.
47.6
zs
69.2
83.1
65.9
90.3
56.3
56.9
13.5
86.2
67.7
93.7
47.0
79.0
64.5
62.0
99.4
78.6
Ens
61.3
83.1
50.0
46.1
81.2
40.5
53.4
39.6
97.0
45.9
16.0
89.2
65.7
86.6
33.3
56.0
98.0
55.1
RissMVi
35.1
63.3
Figure 10: We perform duitt--dist mased retrieval Objaverse LVIS by using manually cropped images from a large indoor scene image soureed from the web to retrieve the top 3 matching objects using our DuoduoCLIP model trained with MVImgNet.
object mask area, selecting the 12 views with the largest areas. Afterward, we crop each object according to its bounding box, centering it within the frame and filling the regions outside the bounding box with white. We compare against recent point cloud based methods in Tab, [3]
It is evident that image-based models significantly outperform point cloud models. Notably, the best-performing model is the zero-shot B-32 CLIP model, highlighting a persistent issue with dataset size and the tendency of these methods overfit. While observe improvements from models trained on both Ensembled and MVImgNet datasets, which include natural images, the gains are still small, suggesting that larger datasets could lead to further performance enhancements.
A.4 ADDITIONAL RESULTS
A.4.1 OUT OF DISTRIBUTION RETRIEVAL
To further assess our model's understanding of 3D objects, we conduct image-based retrieval using tott-f--iitrrbuttinn images. We source indoor scene images from the web and generate additional images using large text-to-image model (Esser et al. 2024).. Objects manually cropped from these scenes (I view each) and processed through our shape encoder to obtain query embeddings. For the retrieval database, compute embeddings for the Objaverse LVIS split using 12 views per
20

---

Published as a conference paper at ICLR 2025
Figure 11: We perform out-of-disti image-based retrieval on Objaverse LVIS by using manually cropped images from a large indoor scene image generated from Stable Diffusion (Esser et al. 2024) to retrieve the top 3 matching objects using our DuoduoCLIP model trained with MVImgNet.
Figure 12: Additional examples of context mixing where we blend concepts from two distinct objects by identifying the Objaverse shape with the embedding that maximizes similarity to both objects.
object. The top 3 matches are returned based on the highest cosine similarity between the query embeddings and the database embeddings. The results for the image sourced from the web are shown in Fig. I0 while those for the image generated by the text-to-image model are presented in Fig. II
Our model successfully retrieves objects that are semantically related to the corresponding cropped portions of the images, demonstrating strong understanding of object categories and their semantic significance. While the retrieved objects may not visually match the cropped images exactly, important note that these in-the-wild images, and identical objects may not exist in the Objaverse LVIS dataset.
A.4.2 CONCEPT MIXING
To explore the latent space learned by our model, we perform concept mixing by identifying the object embedding within Objaverse that maximizes the similarity to both of the two objects being combined. Similar to OpenShape (Liu et al. 2023a), to blend shape with shape b. select the object i with normalized shape embedding f such that i arg max; (min((f) f(), (fe f)))) Note that the shape embedding is extracted by encoding the 12 views of each object. The results are illustrated in 2 and Fig. I2 These examples demonstrate that our model has learned meaningful latent space capable of blending geometric and semantic concepts. For instance, it can combine pumpkin and a carriage into a pumpkin carriage (Fig. rright) or a horse and a wagon into a horse pulling wagon (Fig. left). Interestingly, the model also combine semantic attributes, such
21

---

Published as a conference paper at ICLR 2025
merging dog and engine create four-legged mechanical creature (Fig. 12 top right) blending bird and bus to form a plane (Fig. [2 bottom left), reflecting an understanding of the geometric function of wings and the concept of transport.
A.5 ADDITIONAL DETAILS
A.5.1 ATTENTION MAP VISUALIZATION
For Fig. visualize attention maps (Abnar& & Zuidema 2020) by first selecting area of interest in one of the multi-view images (e.g., the plate in the first example). All 12 multi-view images of the object passed through the model backbone to extract the attention map from the query and key embeddings in the first MVA layer. This produces an attention map of size (M x 49) x (M x 49), where M is the number of multi-views and 49 is the number of tokens per image (excluding the 49 patches). To retrieve attention specific to the area interest (e.g., token 11 (plate) in the first image), we extract the corresponding row in the attention map, resulting in an attention response of size M x 49. This response is overlaid onto the original images, as shown in Fig. 4 These visualizations reveal how the attention response to a token in one image corresponds to regions in other multi-view images, demonstrating the model's s ability to capture geometric relationships across views.
CLS token, with image resolution 224 and ViT patch size 32, giving (224/32)²
22


Reviews:
[{"review_id": "HBZ0XcdDuV", "reviewer": "Reviewer_sfYJ", "summary": "This paper introduces Duoduo CLIP, a model for 3D representation learning that captures object shapes from multi-view images instead of point clouds. By leveraging multi-view images, the model uses 2D priors from pre-trained CLIP models to fine-tune on 3D data. The proposed cross-view attention module aggregates information across different views, and the model is both permutation-invariant and pose-free. Compared to point cloud-based methods, this multi-view approach significantly reduces GPU training requirements and demonstrates improved performance on fine-grained text-to-shape retrieval tasks.", "questions": "1. In Figure 6, the second row describes \"a brown chair with a round back and thin metal base,\" but the ground truth appears to show a dark green or gray chair. It’s unclear whether this discrepancy is due to color shifts introduced during the rendering process or if it reflects a lack of quality control in the dataset.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly written and easy to follow. Extensive experiments are conducted for point cloud classification and text-based retrieval.\n\n2. The proposed method achieves state-of-the-art performance on these tasks while significantly reducing computational costs.", "weaknesses": "1. The zero-shot CLIP baseline is insufficient to demonstrate the proposed model’s advantages, as the test set is specific to 3D renderings, while the original CLIP model is primarily trained on natural images. To provide a fair comparison, the authors should fine-tune the original CLIP model using the same datasets and training strategies as the proposed model, and report the results.\n\n2. Representing 3D shapes with multi-view images is one of the most straightforward approaches and has been used in previous works, such as ULIP2, to align the feature spaces of text, image, and point cloud representations. The purpose of such works is not solely to achieve high classification performance, but to deepen understanding of raw 3D representation beyond using just images. The original CLIP model, even without fine-tuning or special design, already serves as a strong baseline. This paper focuses on multi-view representation and achieves moderate improvements in basic classification and retrieval tasks by training on large-scale images, but this is quiet straight forward and it does not introduce new ideas or insights to the field. The work would be more compelling if the authors explored capabilities *beyond* the scope of those point cloud representations, such as scene-level multi-view data, to better demonstrate the necessity and potential upper bounds of the proposed approach.\n\n3. Have the authors investigated the effect of different view sampling strategies? Unlike point cloud representations, which capture the entire shape, multi-view representations may be influenced by both the number and sampling strategy of views. While the impact of view count is examined in the paper, it would be beneficial if the authors also explored the impact of different sampling strategies, such as front versus back views, or uniformly sampled versus randomly sampled views.\n\n4. Environmental lighting can significantly affect the appearance of 3D renderings by introducing shading, reflections, and slight color variations. It would be useful to conduct experiments examining how these lighting conditions impact the model’s performance.\n\n5. Maybe add some more downstream tasks like image based retrieval and the concept combination as in Figure 1 of OpenShape.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WHkiq4XnWa", "reviewer": "Reviewer_ZsVt", "summary": "The paper proposes a multi-view 3D understanding model based on contrastive learning. The model is a multiview image encoder encoding variable number of input frames and output a feature vector. The model is trained by constrasting against the corresponding clip and language embeddings.", "questions": "- Why is this an important task? What are some example applications of the proposed model? The paper seems to indicate that there are some applications in robotics. I'd like to see some examples of how the proposed technique can be used in robotic tasks and how exactly can current robotics methods can benefit from it.\n- Can the model generalize outside of the training datasets? I didn't see any in-the-wild evaluation in the paper, which I feel is necessary as a metric for evaluation.\n- Isn't it a little cyclic to use features from a frozen CLIP to train a multiview image encoder? Would the encoder benefit from some inter-view and inter-object contrastive losses? Wouldn't that allow the model to discover novel semantic and geometric information which might not have been captured by CLIP and text encoder?\n- Can the learned multiview encoder be used for 3D reconstruction?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written. The figures and well-made and easy to understand. The overall presentation is good.\n- The idea is simple and easy to understand. The architecture proposed might be reused for other multiview image tasks.\n- The quantitative results are pretty strong and convincing.", "weaknesses": "- It's unclear to me why this task is important. The proposed method seems pretty incremental and I don't see a clear insights or surprising findings reviewed by the paper", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uL4YVsxeBP", "reviewer": "Reviewer_p9H4", "summary": "This work presents Douduo CLIP, a technique for 3D representation learning for classification and retrieval that leverages multi-view input images rather than more explicit 3D representations like point clouds or meshes. The method is simple but effective approach to fine-tuning CLIP for 3D understanding, performing CLIP-style contrastive training between multi-view image encodings (a shape representation), single image encodings and text encodings. The multi-view image encoder is initialized from CLIP and fine-tuned with self-attention across all views for some layers of the ViT encoder, while the single image and text encoders are frozen (this allows for more efficient training).\n\nThe proposed Duoduo CLIP outperforms or is competitive with point cloud-based baselines for classification on Objaverse-LVIS and MVPNet, and outperforms CLIP applied zero-shot without any tuning. There are reasonable qualitative image-based retrieval and state-of-the-art zero-shot results (no specialized training for this task) for text-based retrieval.", "questions": "- Clarity: What is the difference between 5F and 12F as described in the tables (e.g. two bottom rows of Table 2)? This notation is not defined, although it's possible to conclude that this might mean number of views?\n- What is the effect of the choice of pre-trained backbone for the multi-view encoder? How much worse would a DINO or randomly initialized model do? It would be good to understand the effect and significance of this choice. \n- What are specific applications in robotics for which multi-view images are available in real time and where improved recognition from multi-view images would have a positive and significant impact?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### Originality\n- It is reasonable to use pre-trained CLIP encoders and fine-tune them for 3D shape understanding from multiple-views. This allows for training on more data and taking advantage of the large-scale pre-trained representation.\n- Attention implemented  across multiple views of a single object is reasonable, and ablation show that it helps.\n### Quality\n- The investigation into the effect of replacing the point-cloud encoder with a multi-view image encoder is thorough, including analysis about training/testing time inference costs, comparisons with non-fine tuned CLIP backbones, and comparisons with baselines on image-based retrieval, text-based retrieval and classification.\n### Clarity\n- The paper is mostly well written and easy to follow with the exception of the experiments section, which could be improved (see comments in weaknesses).\n### Significance\n- For tasks like retrieval and classification it is good to know that point-cloud representations are weaker than multi-view image representations, which is the main takeaway of this work. Point cloud representations find their main applications in robotics, often in object manipulation tasks that do not only require recognition and where real-time multi-view data collection is cumbersome (see comments further comments on significance in weaknesses)", "weaknesses": "### Originality\n- The main technical novelty is replacing a point cloud encoder with a multi-view CLIP image encoder, and tuning this multi-view encoder. While effective, the novelty is minor. It would also be good to understand the significance of what pre-trained representation (CLIP vs. DINO for example) is used to initialize the multi-view encoder\n### Quality\n- Accuracy of evaluation: Both the Objaverse LVIS and the new MVPNet use somewhat automated procedures using off-the-shelf VLMs to create the object labels. Without additional filtering the Objaverse LVIS dataset is quite noisy and this is likely the case for BLIP (what was used for MVPNet) as well. This is also the case for techniques purpose-made for captioning like Luo et al. (NeurIPS 2023). What is the significance of the findings if the ground truth labels themselves are inaccurate and might contain similar mislabeling patters as the CLIP model that is used as the pre-trained representation in this work?\n### Clarity\n- The flow of the writing in the experiment section can be improved, the sentences don't flow into each other, but read as consecutive lists of facts (e.g.L301-314). It's possible to understand with some effort, but a revision would greatly improve the paper.\n### Significance\n- Point clouds are a data modality generally used for tasks that require fine-grained object shape understanding and real-time visual feedback in robotics, like object manipulation. The present work focuses on recognition related tasks, so the significance in robotics (emphasized by the authors) is not entirely clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QAUMnwPohQ", "reviewer": "Reviewer_fQzu", "summary": "The paper introduces a novel method named Duoduo Clip, which addresses limitations in 3D shape representation by using multi-view images over traditional point clouds, improving adaptability and efficiency in 3D shape retrieval. Unlike previous methods constrained by point cloud limitations and high computational demands (e.g., Uni3D), Duoduo CLIP leverages multi-view images to simplify training, generalize better to unseen shapes, and reduce the need for extensive compute resources. This approach allows direct input of images, enabling seamless integration with real-world datasets (e.g., ScanNet, ScanObjectNN) and enhancing real-time applicability in robotics.", "questions": "As posted in weakness, I hope the author could solve these concerns in the rebuttal phase.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Here is the strength of this paper:\n- The authors propose to use rendered multiview images as shape representation instead of point cloud inputs for retrieval task.\n- It proposes a training strategy that requires significantly less computation resources. \n- The author claims SoTA performance in comparison with baseline models.", "weaknesses": "Here is some concerns from me:\n\n**Major:**\n\n- **Novelty:** While this paper proposes a more efficient approach to leverage multiview image for 3D understanding. Using multi-view (MV) attention to establish connections between multiple views for fine-tuning is not a novel idea. This limits the technical contribution. The primary innovation appears to be shifting from single-view images to multi-view images for training, which is incremental rather than groundbreaking. Also, this paper is similar with: `Yipeng Gao et al. Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-training CVPR2024`. Which use point cloud, multi-view image and text to conduct contrastive learning for multi-modality alignment. Author should explain about it.\n\n\n- **Experiment.** I am also concerned about the potential degradation in performance on natural single-view images. The training dataset is smaller compared to the original CLIP model, and some of the data are synthetic. Although the proposed method improves performance on multi-view images, I worry that it may compromise the model’s ability to handle single-view natural images, which could negatively impact its performance in downstream applications.\n\n**Minor:**\n\n1.Line 80:\nA citation is missing for the reference to \"previous methods...\" \n\n2. There is some confusing in Figure 3:\n- So why the first image all focus on the plate when multiple object are present? What is the query?\n- I also question the claim that this figure demonstrates \"geometric correspondences of 3D shapes.\" It could be interpreted as a simple classification heatmap, which does not necessarily indicate the model is aware of 3D structure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)