Duoduo CLIP: Efficient 3D Understanding with Multi-View Images

## Abstract

Abstract We introduce Duoduo CLIP, a model for 3D representation learning that learns shape encodings from multi-view images instead of point-clouds.
The choice of multi-view images allows us to leverage 2D priors from off-the-shelf CLIP models to facilitate fine-tuning with 3D data. Our approach not only shows better generalization compared to existing point cloud methods, but also reduces GPU requirements and training time. In addition, the model is modified with cross-view attention to leverage information across multiple frames of the object which further boosts performance. Notably, our model is permutation invariant to the order of multi-view images while being pose-free. Compared to the current SOTA point cloud method that requires 480 A100 hours to train 1 billion model parameters we only require 57 A5000 hours and 87 million parameters. Multi-view images also provide more flexibility including being able to encode objects with a variable number of images, and performance scales when more views are used. In contrast, point cloud based methods require an entire scan or model of the object. We showcase this flexibility with benchmarks from images of real-world objects. Our model also achieves better performance in more fine-grained text to shape retrieval, demonstrating better text-and-shape alignment than point cloud based models.

## 1 Introduction

Learning representations for 3D shapes is useful for many applications in object generation, shape understanding, grounding and robotics.
Recently, there is also growing interest in aligning 3D representations to text, to enable text-to-3D shape retrieval (Ruan et al., , Tang et al., , Wang et al., ) and generation (Chen et al., ), or as a basis for 3D feature maps that can be queried with text (Huang et al., , Liu et al., , Jatavallabhula et al., ).

**Table 1: Comparison of training time and GPU used for our method and recent point cloud based methods.**
| Method | GPU | Time |
| --- | --- | --- |
| OpenShape Liu et al. () | 1$\times$A100 (80GB) | 300 hr |
| Uni3D Zhou et al. () | 24$\times$A100 (40GB) | 20 hr |
| RECON++ Qi et al. () | 8$\times$A800 (80GB) | 1 day |
| Ours (Full) | 4$\times$A40 (48GB) | 14.3 hr |
| Ours (6 layers) | 4$\times$A5000 (24GB) | 14.3 hr |

Early work in this domain attempted to train such aligned models on limited data (Chen et al., , Ruan et al., , Tang et al., ).
More recent work (Xue et al., , Liu et al., ) leveraged large pre-trained vision-language models (e.g. CLIP (Radford et al., )).
Larger 3D datasets like Objaverse (Deitke et al., ) enabled training of methods that can handle more diverse object categories (Xue et al., , Liu et al., , Qi et al., , Zhou et al., ).
However, more and more GPU resources are required for training such methods.
These works (Xue et al., , Liu et al., , Qi et al., ) predominantly rely on point clouds as their 3D representation.
Point clouds are limited in resolution and have a domain gap with real-world images, and thus these point cloud-based methods can not effectively leverage 2D priors from CLIP.
Uni3D (Zhou et al., ) proposed a training regime to incorporate 2D priors with point clouds as input.
However, their model requires significant scaling (1B parameters) for the best performance.
In this work, we show that our Duoduo CLIP approach which uses multi-view images as the shape representation allows more easily adapting off-the-shelf 2D representation models while enabling more efficient training. As shown in Figure 1(b), our model not only has much stronger generalization on shapes that were not seen during training, but also significantly lowers the compute resources required for training (see Table 1).

Figure: (a) We use multi-view images, instead of point-clouds, to represent the shape.
Refer to caption: x1.png

Multi-view images also offer greater flexibility.
For example, in real world datasets like ScanNet (Dai et al., ) or ScanObjectNN (Uy et al., ) a 3D mesh is first reconstructed from densely captured RGBD data from which 3D point clouds are sampled.
This process introduces errors and artifacts, in contrast to sparse multi-view images which can be directly sub-sampled from the RGB video stream.
Our model takes images directly as input, making it plug-and-play for any method using image features for encoding 3D objects, which is particularly important for real-time robotics applications.
Additionally, previous methods that only train on synthetic objects have limited performance on real-world objects.
Our method can be trained on both real photos and reconstructed objects and outperforms prior work on real-world objects.
Performance can further be boosted with additional multi-view images when point clouds are not available, demonstrating our model’s potential for scaling as more data becomes available.

In summary, our Duoduo CLIP model provides the following:
Simpler model. We average embeddings across views of an object to obtain 3D representations, while modifying the attention layers to span multiple frames of the shape to enable information flow across frames in the model.
Better efficiency. We explore several strategies for efficiently fine-tuning off-the-shelf CLIP models, including training only the attention layers and selectively choosing the number of layers to train without hurting performance.
More flexible representation. We show the benefits of multi-view images as a more flexible 3D representation over point clouds on real-world objects. Furthermore, we demonstrate better text to shape retrieval with more fine-grained descriptions.

## 2 Related Work

There is much work on 3D representation learning using supervised training (Qi et al., , Qian et al., ) or using self-supervised learning (with reconstruction loss (Chen & Zhang, ), contrastive learning (Xie et al., ), masked prediction (Yu et al., ), and others).
Here we focus on work that learns aligned representation spaces between text and either 2D or 3D.

2D-text joint representation. Modeling multiple modalities through a shared representation has been explored through early works by aligning images and text embeddings (Faghri et al., , Frome et al., , Kiros et al., , Weston et al., ). With the introduction of the InfoNCE loss (Oord et al., ), ConVIRT (Zhang et al., ) proposed using contrastive learning with InfoNCE style loss to learn joint text and image embeddings. This was popularized by follow-up work such as CLIP (Radford et al., ) and ALIGN (Jia et al., ), which demonstrated that by scaling up training on a large-scale dataset, the learned embeddings can be useful for many downstream tasks such as classification, retrieval, grounding and generation. Followup work (Jia et al., , Yang et al., , Mu et al., , Zhai et al., , Yu et al., ) has since improved upon the formulation or training. There have also been attempts such as MERU (Desai et al., ) that use a hyperbolic embedding space to better model the hierarchical relations between text and images.

3D-text joint representation. Text2Shape (Chen et al., ) was one of the first works that introduced a captioned text and 3D dataset and used metric learning to learn an aligned space between text and 3D shapes. Follow-up work investigated improvements to the aligned representation for text-to-shape retrieval with triplet loss (Tang et al., ) and contrastive losses (Ruan et al., , Xue et al., , Wang et al., ). Some prior work used of CLIP to align the embeddings (Thomason et al., , Ruan et al., , Xue et al., ). ULIP (Xue et al., ) used frozen CLIP text-image encoders, and trained only the point-cloud encoder.
However, generalization ability was limited by the available dataset size at the time. With the introduction of larger scale 3D datasets such as Objaverse (Deitke et al., ), and the use of LVLMs for captioning 3D datasets (Liu et al., , Luo et al., ), more recent work (Liu et al., , Xue et al., ) trained with larger datasets of paired text and 3D shapes. OpenShape (Liu et al., ) and ULIP-2 (Xue et al., ) were among the first to use such datasets to train a point cloud encoder aligned with text using contrastive learning. TAMM (Zhang et al., ) leverages additional adapters to better align with the pre-trained CLIP models. Uni3D (Zhou et al., ) exploits 2D priors within pre-trained large-scale vision models and maps point cloud patches to image patches within pre-trained ViTs to enable the scaling of even larger models. VIT-LENS (Lei et al., ) follows a similar pipeline by training a point cloud embedding layer and perceiver model to map to a frozen CLIP model. RECON++ (Qi et al., ) similar to RECON (Qi et al., ) employs both contrastive and MAE (He et al., ) reconstruction losses with an additional multi-view image matching cost.
These recent works typically rely on frozen text and image encoders, and all used point-cloud encoders.
Our work shows that taking embeddings of multi-view images is more efficient and works better with real-world images.

Multi-view images for text-3D representations.
Multi-view images have long been used to represent 3D shapes (Bradski & Grossberg, , Shen et al., , Su et al., , Wei et al., , Hamdi et al., ), with recent work (Song et al., , Lin et al., ) using multi-view images for shape representation by aggregating CLIP features. While these works also use CLIP with multi-view images, our focus is whether using a trainable multi-view shape encoder as a drop-in-replacement for the point-cloud encoder in contrastive settings such as OpenShape will result in better shape representations for downstream tasks.
These works do not focus on improved representation learning, but on how to use prompting to select better views or view aggregation and improve shape classification.
There is also a line of work (Zhang et al., , Zhu et al., , Huang et al., ) that uses multi-view depth maps from point cloud to adapt CLIP models for 3D.
However, the focus of these methods is still on point clouds and the models show limited performance on large-scale datasets (Xue et al., , Liu et al., ).

## 3 Method

Figure: Figure 2: Our multi-view encoder takes a variable number $[1,M]$ of images as input and outputs a single shape embedding. The first few layers of the ViT model are frozen (blue) and attention operates on the individual frames in parallel. For the latter layers, the attention layers are trainable (orange) and modified to attend over all $M$ views. The embeddings for each frame are averaged to get the final embedding. Note that only major components of the model are depicted.
Refer to caption: x3.png

Our Duoduo CLIP learns a shape representation by using contrastive learning to encode multi-view images into a pre-aligned text-image space.
Our contrastive learning framework is similar to previous works (Xue et al., ; , Liu et al., , Zhang et al., , Zhou et al., ), with a shape encoder $E^{S}$ and frozen image $E^{I}$ and text $E^{T}$ encoders from a pre-trained CLIP model.
The key difference is that we replace the point cloud shape encoder with one that uses multi-view images. Figure 1(a) illustrates this difference, while Figure 2 shows the details of our multi-view shape encoder.

To perform training, the dataset is organized into triplets of $(S_{i},I_{i},T_{i})$, where $S$ is the shape representation, $I$ are rendered images of the 3D object and $T$ is the corresponding text description. Embeddings are calculated for each modality $f^{S}_{i}=E^{S}(S_{i})$, $f^{I}_{i}=E^{I}(I_{i})$ and $f^{T}_{i}=E^{T}(T_{i})$. The asymmetric contrastive loss (Zhang et al., ) between any two modalities is:

$$ $l^{a\rightarrow b}_{i}=-\log\frac{\exp(\langle f^{a}_{i},f^{b}_{i}\rangle)/ \tau}{\Sigma_{k=1}^{N}\exp(\langle f^{a}_{i},f^{b}_{k}\rangle)/\tau}$ (1) $$

where $a,b$ is any two modalities, $\tau$ is the temperature and $N$ is the batch size. Then, the full loss is:

$$ $L_{CON}=\frac{1}{4N}\Sigma_{i=1}^{N}(l^{S\rightarrow T}_{i}+l^{T\rightarrow S} _{i}+l^{S\rightarrow I}_{i}+l^{I\rightarrow S}_{i})$ (2) $$

Two important properties of our shape encoder for multi-view images includes being able to handle arbitrary number of views as well as being permutation invariant to ensure the flexibility for the input images. Encoding of any number of frames with the model into a single embedding as well as a permutation invariant operation between multi-views are discussed in more detail below. Note that we also do not make assumptions about the poses of the input image, making our method pose-free.

Multi-view shape encoder. For our multi-view shape encoder, we use rendered images of the object. Note that the input shape and image representations are the same, i.e. $S_{i}=I_{i}=\{r_{1},...,r_{m}\}$, where $r_{j}$ is a rendered frame, and $m$ is the number of multi-views. $E^{I}$ and $E^{S}$ share the same network architecture, with the main difference between $E^{I}$ and $E^{S}$ is that $E^{I}$ is frozen, while we train $E^{S}$. Weights for $E^{S}$ are initialized from $E^{I}$. Empirically, we found it was useful to keep the contrastive losses $l^{S\rightarrow I}_{i}+l^{I\rightarrow S}_{i}$. We suspect that by keeping $E^{I}$ frozen and the contrastive loss with the original image embeddings, our $E^{S}$ can learn a good representation that is close to the text embeddings, but does not drift too much away from the initial image embeddings.

For both $E^{S}$ and $E^{I}$ we average the extracted embeddings for all $m$ views to produce a single latent representation for an object: $f^{S}_{i}=E^{S}(S_{i})=\frac{1}{m}\Sigma_{j=1}^{m}\text{ViT}(S_{i})[j]$, where ViT (Dosovitskiy et al., ) is the backbone encoder. Here we abuse the notation a bit, since our method allows interaction between views in the model, the index operation gets the corresponding embedding from the $j$-th frame’s [CLS] token. We initialize $E_{S}$ initially from $E^{I}$. During training, we randomly sample $1$ to $M$ views for each step so that the model is robust to a variable number of views.

Figure: Figure 3: We visualize the interpolated attention map calculated by our MVA layer over the original image. The model has learned geometric correspondences of 3D shapes across different multi-views.
Refer to caption: x4.png

Multi-view attention. We incorporate information flow between views by modifying the self-attention layers to attend over tokens across multiple views similar to recent multi-view aware methods like MVDream (Shi et al., ). The original self-attention (Vaswani et al., ) is given by $\operatorname{Attention}(Q,K,V)=\operatorname{softmax}(\frac{QK^{T}}{\sqrt{d_{
k}}})V$, where $Q,K,V\in R^{L\times k}$, $L$ is the token length for a single image after patchify operations, and $k$ is the channel dimension. For cross-view attention we aggregate token features across $m$ multi-views (MV) so that $Q^{\text{MV}},K^{\text{MV}},V^{\text{MV}}\in R^{(L\times m)\times k}$ and the $\operatorname{Attention}(Q^{\text{MV}},K^{\text{MV}},V^{\text{MV}})$ can be calculated without changing the parameters of the original model. After attention, the views are separated and processed in parallel in other parts of the model. This mechanism helps the model reason between different views of the object and aggregate the most important information across all tokens. From Figure 3, we see the multi-view attention layers learns to capture correspondence between object parts from different views.

Trainable layers.
To reduce the memory needed for fine-tuning, we only train a portion of the self attention layers while keeping the MLP layers after attention fixed. Since the non-trainable self-attention layers have only been trained to see patches of a single image, we only enable the multi-view attention layers for the unfrozen attention layers (see Figure 2). Additionally, we find the choice of freezing the MLPs, help in preventing overfitting and makes the model generalize better on unseen datasets (see Section A.2.2).

Figure: Figure 4: Left. We compare the Top 1 accuracies on Objaverse-LVIS for our model (Duoduo CLIP) and two pretrained CLIP models (zero shot) for different number of frames. B-32 is the model we initialize from and train with and BigG-14 is the model used by OpenShape. The dashed lines show the difference between using 1 and 12 frames. Right. We show the Top1, Top3 and Top5 accuracies for our model for 1 to 12 frames (note the y-axis starts at 40). Gray dashed lines shows the inference time(s) for 1 iteration and batch size 200 at different number of frames.
Refer to caption: x5.png

## 4 Experiments

To compare our Duoduo CLIP to prior work, we conduct experiments showing shape classification on a collection of 3D assets (on Objaverse-LVIS in Section 4.2.1), real-world multi-view images (on MVImgNet in Section 4.2.2), and more fine-grained text-to-shape retrieval (Section 4.3). We first describe the experimental setup including implementation details and the datasets we use.

### 4.1 Experimental setup

Implementation details. All of our models are trained with 16-bit mixed precision and a batch size of $1600$ for 80 epochs. We use a learning rate of $5e-5$ with cosine annealing. At each training step we randomly sample $1$ to $6$ multi-views for a batch of objects. Ablations regarding the number of multi-views sampled during training can be found in Section A.2.1. The pretrained CLIP model used for initialization as well as the contrastive target is the ViT-B/32 CLIP model with checkpoint *laion2b_s34b_b79k* from the open source implementation OpenCLIP (Ilharco et al., ). Additionally, we re-implemented OpenShape with 16-bit mixed precision training without projection layers for faster training in some baselines, indicated by OpenShape$\dagger$.

3D Datasets.
We follow OpenShape (Liu et al., ) and train our model on a combination of four datasets consisting of Objaverse (Deitke et al., ), ABO (Collins et al., ), ShapeNet (Chang et al., ) and 3D-FUTURE (Fu et al., ). Text for the shapes consists of captions produced by image-captioning models and filtered text descriptions from metadata of the 3D objects crawled from the web (see Liu et al. () for details). In total, the combined dataset contains $\sim$874k shapes, with 46k shapes within the LVIS subset of Objaverse which are used for evaluation. As in OpenShape (Liu et al., ), we consider two variations of the data: 1) Ensembled which contains all shapes, and 2) Ensembled (no LVIS) which excludes shapes from the LVIS subset.

Renderings.
We use images from Zero123 (Liu et al., ), which provides renderings of most shapes from Objaverse from $12$ random views using spherical sampling. The shapes not included in Zero123 (a few remaining objects in Objaverse and the other 3 datasets) are rendered using their Blender (^1^11[https://www.blender.org/](https://www.blender.org/)) script to match the shapes in OpenShape.
We render $12$ views for all these objects. Additional details and effects of view selection can be found in Section A.1.2.

Real-world data.
Multi-view images. MVImgNet (Yu et al., ) consists of multi-view images for 220k real-world objects over 238 categories. We first filter objects that has at least $12$ multi-views to obtain $\sim$190k remaining objects, where $12$ views are evenly sampled. BLIP-2 (Li et al., ) is then used to produce captions for each of the views. To compare with point cloud based methods, we use the MVPNet subset containing $\sim$87k objects with densely reconstructed point clouds. After the preprocessing and filtering above, we obtain from MVPNet, $\sim$66k objects for the train split and $\sim$16k objects for the validation set, which has 180 categories. Note that the objects in the training and validation splits for MVPNet are disjoint but are both included in MVImgNet. Reconstructed 3D data. We additionally conduct experiments on objects obtained from real world scans. This includes ScanObjectNN (Uy et al., ) which contains point clouds from objects in SceneNN (Hua et al., ) and ScanNet (Dai et al., ). We use the test set for evaluation which contains 583 shapes and 15 categories. As the dataset only contains point clouds, we render 12 multi-view images from the point clouds directly (more details in Section A.3.1). Furthermore, we conduct classification using 3825 objects with 17 classes from the val set of ScanNet (Dai et al., ) using cropped multi-view images from the raw captured RGB frames in the dataset (more details in Section A.3.2).

### 4.2 Classification Experiments

**Table 2: Classification accuracy on Objaverse-LVIS (O-LVIS) and ScanObjectNN trained on Ensembled (no LVIS) and Ensembled. The zero shot CLIP (B-32 and BigG-14) models are not trained on 3D data and serve as baselines. For each model, we indicate the representation used for computing the embeddings, multi-view (MV) or point-cloud (PC), and the shape encoder (Enc) architecture that is used. Our multi-view based Duoduo CLIP is able to outperform point-cloud based approaches with the exception of Uni3D is a large model with 1B parameters. $\dagger$ indicates our reimplementation of OpenShape which is faster and outperforms the original OpenShape.**
| Pretrain Dataset | Ensembled (no LVIS) | Ensembled |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  | O-LVIS | O-LVIS | ScanObjectNN |  |  |  |  |  |  |
| Method | Rep | Enc | Top1 | Top3 | Top5 | Top1 | Top3 | Top5 | Top1 | Top3 | Top5 |
| ZS B-32 (12F) | MV | Avg | 35.7 | 54.8 | 62.1 | 35.7 | 54.8 | 62.1 | 53.9 | 73.5 | 81.2 |
| ZS BigG-14 (12F) | MV | Avg | 43.5 | 64.2 | 71.3 | 43.5 | 64.2 | 71.3 | 56.7 | 78.2 | 85.8 |
| ULIP Xue et al. () | PC | PointBERT | 21.4 | 38.1 | 46.0 | 26.8 | 44.8 | 52.6 | 51.6 | 72.5 | 82.3 |
| OpenShape Liu et al. () | PC | PointBERT | 39.1 | 60.8 | 68.9 | 46.8 | 69.1 | 77.0 | 52.2 | 79.7 | 88.7 |
| OpenShape$\dagger$ | PC | PointBERT | – | – | – | 50.0 | 72.0 | 79.2 | - | - | - |
| TAMM Zhang et al. () | PC | PointBERT | 42.0 | 63.6 | 71.7 | 50.7 | 73.2 | 80.6 | 55.7 | 80.7 | 88.9 |
| Uni3D Zhou et al. () | PC | 3D VIT | 47.2 | 68.8 | 76.1 | 55.3 | 76.7 | 82.9 | 65.3 | 85.5 | 92.7 |
| ShapeLLM Qi et al. () | PC | RECON++ | – | – | – | 53.7 | 75.8 | 82.0 | 65.4 | 84.1 | 89.7 |
| VIT-LENS Lei et al. () | PC | VIT-LENSG | 50.1 | 71.3 | 78.1 | 52.0 | 73.3 | 79.9 | 60.1 | 81.0 | 90.3 |
| Duoduo CLIP (5F) | MV | MVA | 51.3 | 73.1 | 79.9 | 54.1 | 76.0 | 82.4 | 60.7 | 82.4 | 88.5 |
| Duoduo CLIP (12F) | MV | MVA | 52.7 | 74.5 | 81.3 | 55.2 | 77.1 | 83.4 | 66.3 | 85.5 | 90.2 |

#### 4.2.1 Objaverse-LVIS

We conduct classification experiments on Objaverse-LVIS to investigate the performance of different variants of Duoduo CLIP and compare our model to prior work.
As default settings for our experiments in this section, we use $12$ multi-views to encode an object for evaluation. And only the multi-view attention of the latter 6 layers of the model is trained (total 12 layers). We run evaluations with 3 different seeds and average the results for experiments.
Unless otherwise stated models are trained with the Ensembled dataset. For classification, we use do nearest neighbor search of the multi-view shape embedding with class embeddings. As in OpenShape, for each class label, we construct several sentences with the label and then take the average of the sentence embeddings as the class embedding. The final class chosen is the class with the highest average similarity between the class embedding and the shape embedding.

We compare the performance of our model against two CLIP models for different numbers of frames in Figure 4, and recent state-of-the-art models in Table 2. For the CLIP models, we selected VIT with B-32 (the model that our Duoduo CLIP is initialized with), and BigG-14 (the CLIP teacher model used by OpenShape). Both models are run in a zero-shot manner without training on additional 3D data (the weights are frozen, with embeddings from the image encoder averaged).

How well does performance scale with number of views?
Having a more complete view of a 3D object is important for 3D representation learning as parts of the object may be occluded for some views. Thus we investigate how the number of views affects the classification accuracy.
From Figure 4 (left), we see that our model performance scales better as the amount of information (views) increases compared to the zero-shot CLIP models ($11.22$ for our Duoduo CLIP, and $7.45$ for B-32, the model we used for initializing our weights).
This suggests that our training strategy of randomly sampling 1 to 6 views during training is effective in letting the model learn to leverage the number of views given to it. Although, the accuracy increases as the number of multi-views increases, the inference time also increases (Figure 4 right). As the accuracy tapers off as we increase the number of views, to avoid additional computational cost during inference, we use 12 views as the default.

How many multi-views to beat a point cloud?
Compared with using multi-views, using a point cloud representation offers good coverage of the entire geometry of the 3D object. Thus, we ask the question of how many views does it take to match existing methods on the Objaverse-LVIS classification task.
We compare against ULIP (Xue et al., ), OpenShape (Liu et al., ), TAMM (Zhang et al., ) that use point clouds with PointBERT (Yu et al., ) as the 3D encoder, but different ways of fine-tuning the 3D encoder. We also compare against more recent methods (Zhou et al., , Qi et al., , Lei et al., ) that are the current top-performing methods on Objaverse-LVIS.
From Table 2, on the Ensembled dataset, we are able to beat most previous methods with 5 multi-views except for Uni3D which has a much larger model (1B parameters). When we increase to 12 frames we are on par and even surpass Uni3D. This is also while using much fewer resources as seen in Table 1.

Does multi-views generalize better compared to point cloud? From the Ensembled (no LVIS) column of Table 2, we see that even the zero-shot BigG-14 model used as the teacher model for OpenShape beats it. Suggesting that the CLIP models already generalize well to unseen data, and point cloud methods potentially having worse generalization on unseen shapes. With our model that is further trained on Ensembled (no LVIS) we surpass point cloud methods by a large margin. Showing that fine-tuning the CLIP model better preserves their generalization ability. Additionally, our model generalizes well on ScanObjectNN (Uy et al., ) and outperforms point cloud methods on both Top1 and Top3 accuracies (see Section A.3 for more results on ScanObjectNN and ScanNet).

**Table 3: MVPNet classification comparison.**
| Method | Top 1 | Top 3 | Top 5 |
| --- | --- | --- | --- |
| ZeroShot B-32 (12F) | 52.68 | 70.99 | 77.22 |
| OpenShape$\dagger$ | 10.80 | 19.62 | 25.20 |
| OpenShape$\dagger$ (+MVPNet) | 54.59 | 72.66 | 78.61 |
| Ours (12F) | 49.16 | 66.96 | 74.12 |
| Ours (+MVPNet) (1F) | 59.23 | 76.12 | 81.74 |
| Ours (+MVPNet) (6F) | 64.18 | 80.80 | 85.92 |
| Ours (+MVPNet) (12F) | 64.44 | 81.11 | 85.97 |
| Ours (+MVImgNet) (1F) | 61.75 | 79.07 | 84.18 |
| Ours (+MVImgNet) (6F) | 65.70 | 82.44 | 87.08 |
| Ours (+MVImgNet) (12F) | 66.06 | 82.72 | 87.21 |

#### 4.2.2 MVImgNet

To investigate how well our Duoduo CLIP works on real-world images, we evaluate on multi-view images from MVImageNet. For these experiments, we take our Duoduo CLIP model trained on the Ensembled dataset and add additional training data from MVImgNet. We consider two sets of data from MVImgNet, the MVPNet training split of objects or the entire MVImgNet (after our preprocessing and filter). For comparison with point cloud based methods, we evaluate on the validation set of MVPNet. As a representative of point cloud based methods, we choose OpenShape as it is relatively resource-light compared to other point cloud methods.

Classification results.
We report accuracy results on MVPNet in Table 4. For OpenShape, the top-1 accuracy is very low (only $10.8\%$ top-1) when evaluating on MVPNet directly with the model trained on Ensembled only. We suspect this is due to noisy reconstructed point clouds and inconsistent orientations compared to the training data. Note we re-align MVPNet point cloud’s up direction with Objaverse. After training on MVPNet data, OpenShape’s performance improves significantly. For our Duoduo CLIP, even without training on MVImgNet or MVPNet data, our model achieves a top-1 accuracy of $49.16\%$. However, this is lower than the zero-shot CLIP B-32 model that our model is initialized from, suggesting possible overfitting to synthetic data. When we include training with multi-view images from MVPNet, our method surpasses both OpenShape and the zero-shot CLIP B-32 model even when just using one 1 view. Performance is further boosted when trained on the entire MVImgNet, highlighting the advantages of our Duoduo CLIP which uses multi-view images over OpenShape’s point clouds. By using multi-view images, we have better generalization on unseen data due to the priors from initializing with CLIP B-32. We also can train on more data, as multi-view images are more readily available compared to point clouds.

Figure: Figure 5: Retrieval results for OpenShape and Duoduo CLIP given input images from MVPNet. We perform retrieval with 12 input images, on the left most column we show one of the frames of the query object. For both methods we show the top 3 retrieval results using Objaverse as a retrieval library. Overall, Duoduo CLIP retrieves shapes that are more similar to the query image, while OpenShape sometimes retrieves broken geometry (row 3).
Refer to caption: x6.png

Retrieval examples.
In Figure 5, we present example retrieval results for querying Objaverse objects using images from the MVPNet validation set. We compare retrievals using OpenShape (+MVPNet) and Duoduo CLIP (+MVPNet). For OpenShape, we use 12 query images for each object, encode them with the frozen BigG-14 CLIP model, and average the image embeddings to obtain the query embedding.(^2^22For the largest BigG-14 CLIP model, OpenShape does not use any linear projection layer $g^{I}$). This embedding is then matched against a database of point cloud embeddings of Objaverse objects to retrieve the top 3 results. For our method, Duoduo CLIP encodes both the query and Objaverse objects using 12 multi-view images.

From Figure 5, we see that for basic objects like benches, both methods are able to retrieve plausible objects. For the armchair (row 2), while OpenShape retrieved objects of the same semantic class, our Duoduo CLIP retrieved armchairs that has better matching textures. This is because multi-view images can represent textures better than point clouds. Note that some retrieved objects look similar due to duplicate objects in Objaverse. For the jacket (row 3),
the objects returned by OpenShape have broken geometry and are incoherent.
We hypothesize this may happen when the sampled points produce embeddings that are by chance similar to the query image.
This issue is also present last example (a red and black soccer ball) where the first object retrieved by OpenShape has broken geometry. However, our method also fails in this case, as our model retrieves ladybugs with red wings and black dots that resemble the soccer ball. Interestingly, OpenShape is able to retrieve a soccer ball and a red ball. This can indicate that point cloud based methods may be more robust to geometric properties of the object compared to the rendered images. However, we find that overall the retrieved objects from our method is more consistent.

#### 4.2.3 Model Ablations

**Table 5: Ablation on the number of layers for accuracy on Objaverse-LVIS using 12 input frames. Default model highlighted in gray.**
| Method | Top 1 | Top 3 | Top 5 |
| --- | --- | --- | --- |
| 3 layers | 53.77 | 75.8 | 82.41 |
| 6 layers | 55.24 | 77.08 | 83.43 |
| 12 layers (full) | 55.32 | 77.08 | 83.49 |

Do we need to train all the layers?
We conduct an ablation to see if it is necessary to train all 12 layers. In Table 6, we show the accuracy for restricting the training to the last 3 and 6 MVA layers vs the full 12 MVA layers.
We see that the number of trainable layers can be cut by half with minimal losses to accuracy, while further cuts start to hurt model performance. Training only half the layers reduces the VRAM usage by roughly half and allows us to train on GPUs with less memory. By restricting the training to the last 6 MVA layers, we can shift from training on 4$\times$A40s (48 GB) to 4$\times$A5000s (24 GB) (see Table 1). Also we find that training less layers also helps preserve the priors of the model and make generalization on other datasets better as seen in Table 6.

Does multi-view attention help? We verify the effectiveness of our cross view attention in Table 6.
With the difference between +MVA and -MVA being whether the cross-views attention is enabled. On Objaverse-LVIS, while the +MVA mechanisms helps less +0.3 when all 12 layers are trainable, it makes a larger impact when only training 6 layers +0.63. Making the +MVA mechanism an effective way to maintain performance while being more efficient compared to training all 12 layers. In addition, we find that the +MVA also helps in making the model generalize better on unseen datasets like MVPNet and ScanObjectNN compared to -MVA. Inference speeds for the models can be found in Section A.1.1.

### 4.3 Fine grained text-to-shape retrieval

Previous evaluations focused on classification or image-based retrieval and do not assess how well the methods align fine-grained text descriptions with 3D models. To evaluate this aspect, we use the test split of Text2Shape (Chen et al., ), which contains $\sim$1.4k tables and chairs from ShapeNet and $\sim$7.4k human-captioned descriptions ($\sim$5 per shape). We report the shape retrieval rates at 1 and 5.

Retrieval performance.
We compare the performance of our model to prior work in Table 4. We also include previous state-of-the-art methods that were trained on the Text2Shape dataset. Overall, compared to other methods trained on the Ensembled dataset, our method demonstrates superior performance. The zero-shot performance alone is impressive, surpassing point cloud methods. Notably, we observed a significant discrepancy between the zero-shot CLIP model reported by Ruan et al. () and our findings, where the LAION checkpoint trained by OpenCLIP significantly outperforms the original OpenAI checkpoint. Lastly, our model approaches the performance of the SOTA method trained directly on Text2Shape, indicating that there is still much to explore regarding more fine-grained text alignment in large-scale text-to-3D models.

Figure: Figure 6: Retrieval using fine-grained text from the Text2Shape dataset. On the left is the input text description and we show the top 3 retrieval results for Uni3D and Duoduo CLIP with the ground truth shape highlighted in green.
Refer to caption: x7.png

Retrieval examples.
We present example retrieval results compared to Uni3D in Figure 6. Generally, for text descriptions emphasizing complex textures, such as the ‘man image’ in the first row, our model successfully retrieves the corresponding ground truth 3D model. However, for descriptions focusing on geometric properties, like ‘round back’ in the second row, point cloud-based methods tend to retrieve shapes that better match the specified criteria.

## 5 Conclusion

We introduced Duoduo CLIP, which uses a multi-view image encoder to represent shapes and align the embedding to pretrained text-image models.
We demonstrated that contrastive learning with multi-view images is an efficient and effective way to encode 3D shapes.
Our experiments show that by using multi-view images, our model surpasses current point cloud based state-of-the-art methods, trained in a similar fashion, on both real and synthetic datasets while using significantly fewer resources.

Using multi-view images has limitations, and is potentially poorer at geometric understanding compared to point clouds.
One way to alleviate this is to incorporate additional inputs such as depth or normal images, which provide more geometric cues.
Our work is also limited in only exploring CLIP-style contrastive pretraining. Our model can be easily adapted to other contrastive pretraining Goel et al. (), Zhai et al. () and SSL methods (Zhou et al., , Oquab et al., ).
This is a promising direction as recent work (Banani et al., ) points out the relative weakness of CLIP pretraining on both 3D structure and consistency benchmarks.
Another important direction is to examine how to increase the data available for training.
Even with a relatively small model as ours, training too many layers can lead to severe overfitting.
This suggests that data augmentation and incorporating more 3D data would lead to improved generalization and performance.

By advocating for a reconsideration of multi-view images for shape representation, we believe our work can convince 3D shape researchers to re-examine when point clouds are useful.
Images offer several advantages over point clouds, including better capture of texture and color information, less domain gap with pretrained vision models, and better resource scaling.
They also offer greater flexibility for practical use cases in real-world applications such as robotics.
Leveraging the texture information captured by multi-view methods like ours can complement the geometric understanding of point cloud techniques.
A hybrid approach combining both modalities could leverage their respective strengths and adapt to a variety of use cases in the future.

Acknowledgements.
This work was funded by a CIFAR AI Chair, a NSERC Discovery grant, and a CFI/BCKDF JELF grant.

## Appendix A Appendix

For the ablation experiments in the appendix unless otherwise mentioned, we follow the same default settings as in the main paper. Where models are trained on the Ensembled dataset and we randomly sample 1 to 6 multi-views during training, with the last 6 attention layers trainable and +MVA enabled using a batch size of 1600.

### A.1 Inference speed and camera sampling

#### A.1.1 Inference Speed

We show the inference speed between different number of trainable layers and frames for -MVA and +MVA in Table 8. We benchmark total run time and time per iteration on the entire Objaverse-LVIS (46k shapes). Note that -MVA with 6 layers is not shown, since the run time is the same with -MVA 12 layers during inference since both models have multi-view attention disabled. There is only a slight increase in inference time when +MVA is enabled, and it remains efficient for processing a large number of shapes.

**Table 7: Inference speed on Objaverse-LVIS with batch size 200. Default model is highlighted in gray.**
| Method | Layers | Frames | Speed (s/it) | Total Time (s) |
| --- | --- | --- | --- | --- |
| +MVA | 6 | 3 | 0.14 | 32.80 |
| +MVA | 6 | 6 | 0.28 | 65.52 |
| +MVA | 6 | 12 | 0.56 | 130.67 |
| -MVA | 12 | 3 | 0.14 | 33.23 |
| -MVA | 12 | 6 | 0.28 | 63.90 |
| -MVA | 12 | 12 | 0.54 | 126.28 |
| +MVA | 12 | 3 | 0.15 | 34.01 |
| +MVA | 12 | 6 | 0.29 | 68.38 |
| +MVA | 12 | 12 | 0.59 | 136.10 |

#### A.1.2 Camera Sampling

To verify the effectiveness of our camera sampling strategy, we benchmark different camera settings as seen in Table 8. Example renders for each of the different settings as well as the quantitative results can be found in Figure 7. The results show a clear performance improvement as the elevation range increases, suggesting that capturing the object from more extreme angles also provide more context and information about the shape.

Figure: Figure 7: Left. We show example renders from each of the different camera settings. Right. We show the Top 1 accuracies of the different camera settings using our default model +MVA (6 layers).
Refer to caption: x8.png

### A.2 Other Ablations

#### A.2.1 Number of Multi-Views during Training

For our default model we choose to sample 1 to 6 multi-views during training as it strikes a good balance between memory usage and performance. In Table 10 we show the Objaverse-LVIS accuracy over a variation of sampled views seen during training with other settings kept the same as the default model. Except for the model that is trained with only a single view ie. [1, 1] which uses -MVA as it does not see any multi-views during training.

It can be seen that as the number of multi-views seen during training increases, so does it’s performance when evaluating at 12 frames during inference. However, the memory usage also rises quickly as attention has quadratic growth when the the number of tokens increases in the case of +MVA. So we choose to sample [1, 6] views as there is an increase of +0.24 on top1 over training with [1, 4] views. However, increasing further to [1, 8] frames only results in a +0.09 top1. Also the amount of memory [1, 6] also allows us to stay within the memory limits of a A5000 (24 GB) GPU.

**Table 9: Number of views sampled during training on Objaverse-LVIS ablation. Evaluated using 12 frames per shape during validation. The memory usage is reported on a single GPU with 400 batch size. Default model is highlighted in gray.**
| Num Frames Train | Top1 | Top3 | Top5 | Mem (GB) |
| --- | --- | --- | --- | --- |
| [1, 1] | 52.97 | 74.71 | 81.44 | 5.14 |
| [1, 2] | 54.47 | 76.37 | 82.82 | 9.12 |
| [1, 4] | 55.00 | 76.93 | 83.36 | 15.50 |
| [1, 6] | 55.24 | 77.08 | 83.43 | 22.18 |
| [1, 8] | 55.33 | 77.30 | 83.62 | 28.92 |

#### A.2.2 MLP Fine-Tuning

Only fine-tuning the attention layers of the model not only makes the model more efficient to train, but also helps to preserve the priors of the original CLIP model. To showcase this we also enable the training of the MLP layers correponding to the attention layers being tuned in Table 10. Although we find that Objaverse-LVIS performance improves when more of the network (MLP) is allowed to be tuned. It results in worse performance on the other unseen datasets. Our default model performs the best across the other unseen datasets. Note that although our default model also overfits and performs worse on MVPNet compared to the zero shot model as seen in Table 4, the performance drop-off is smaller compared to the other models.

### A.3 Other Datasets

#### A.3.1 ScanObjectNN

To evaluate on ScanObjectNN (a point cloud dataset), we have to first obtain the multi-view images for each object. For simplicity, we rendered the 12 multi-view images using the point clouds (OBJ_BG) directly as can be seen in Figure 8. Note that ScanObjectNN objects were obtained from real world scan datasets that were captured with RGBD cameras. So using the raw frames would be more straight forward and easier to obtain compared to point clouds since it does not require reconstruction. However, we weren’t able to obtain the mappings to the original scenes from the processed ScanObjectNN dataset so opted for this approach instead.

We show the the quantitaive results on ScanObjectNN in Figure 8. It can be seen that the +MVA mechanism outperforms -MVA by a large margin in this scenario. And that our default model with 12 frames are able to out perform all previous point cloud methods on both top 1 and top 3 accuracies. This is despite the fact that we used rendered point clouds images as opposed to the originally captured RGB scene data. This demonstrates the robustness and adaptability of our model to handle a wide range of different images.

Figure: Figure 8: Example of multi-view image renders of ScanObjectNN.
Refer to caption: x9.png

#### A.3.2 ScanNet

For ScanNet, we generate the multi-view images using the officially released projections from the 3D segmentations onto the original 2D RGB frames. We then sort the frames based on the projected object mask area, selecting the 12 views with the largest areas. Afterward, we crop each object according to its bounding box, centering it within the frame and filling the regions outside the bounding box with white. We compare against recent point cloud based methods in Table 12.

It is evident that image-based models significantly outperform point cloud models. Notably, the best-performing model is the zero-shot B-32 CLIP model, highlighting a persistent issue with dataset size and the tendency of these methods to overfit. While we observe improvements from models trained on both Ensembled and MVImgNet datasets, which include natural images, the gains are still small, suggesting that larger datasets could lead to further performance enhancements.

**Table 12: ScanNet results with cropped video frames evaluated over 12 multi-views. ZS is the zero-shot B-32 model. Ens and Ens + MVI is the default model trained with ensembled dataset and ensembled with MVImgNet datasets respectively.**
| Method | Avg. | Bed | Cab | Chair | Sofa | Tabl | Door | Wind | Bksf | Pic | Cntr | Desk | Curt | Fridg | Bath | Showr | Toil | Sink |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| OpenShape | 45.6 | 66.7 | 3.2 | 75.8 | 83.5 | 37.7 | 49.2 | 47.5 | 64.9 | 48.2 | 1.9 | 66.1 | 70.2 | 1.8 | 50.0 | 57.1 | 45.2 | 7.1 |
| TAMM | 49.4 | 66.7 | 4.8 | 83.6 | 84.5 | 48.9 | 57.9 | 48.2 | 80.5 | 61.3 | 1.9 | 60.6 | 83.6 | 7.0 | 41.4 | 56.1 | 48.4 | 3.6 |
| Uni3D | 45.8 | 58.5 | 3.7 | 78.8 | 83.7 | 54.9 | 31.3 | 39.4 | 70.1 | 35.1 | 1.9 | 27.3 | 94.2 | 13.8 | 38.7 | 10.7 | 88.1 | 47.6 |
| ZS | 69.2 | 83.1 | 58.2 | 65.9 | 90.3 | 56.3 | 56.9 | 43.5 | 86.2 | 67.7 | 44.2 | 93.7 | 47.0 | 79.0 | 64.5 | 62.0 | 99.4 | 78.6 |
| Ens | 61.3 | 83.1 | 50.0 | 46.1 | 87.2 | 40.5 | 53.4 | 39.6 | 97.0 | 45.9 | 16.0 | 89.2 | 65.7 | 86.6 | 33.3 | 56.0 | 98.0 | 55.1 |
| Ens+MVI | 63.3 | 87.2 | 51.3 | 56.5 | 88.5 | 35.1 | 63.3 | 49.5 | 98.7 | 48.8 | 25.0 | 92.9 | 55.1 | 77.8 | 26.9 | 54.8 | 98.9 | 65.0 |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. In Figure 6, the second row describes "a brown chair with a round back and thin metal base," but the ground truth appears to show a dark green or gray chair. It’s unclear whether this discrepancy is due to color shifts introduced during the rendering process or if it reflects a lack of quality control in the dataset.
soundness: 2
strengths: 1. The paper is clearly written and easy to follow. Extensive experiments are conducted for point cloud classification and text-based retrieval.

2. The proposed method achieves state-of-the-art performance on these tasks while significantly reducing computational costs.
summary: This paper introduces Duoduo CLIP, a model for 3D representation learning that captures object shapes from multi-view images instead of point clouds. By leveraging multi-view images, the model uses 2D priors from pre-trained CLIP models to fine-tune on 3D data. The proposed cross-view attention module aggregates information across different views, and the model is both permutation-invariant and pose-free. Compared to point cloud-based methods, this multi-view approach significantly reduces GPU training requirements and demonstrates improved performance on fine-grained text-to-shape retrieval tasks.
weaknesses: 1. The zero-shot CLIP baseline is insufficient to demonstrate the proposed model’s advantages, as the test set is specific to 3D renderings, while the original CLIP model is primarily trained on natural images. To provide a fair comparison, the authors should fine-tune the original CLIP model using the same datasets and training strategies as the proposed model, and report the results.

2. Representing 3D shapes with multi-view images is one of the most straightforward approaches and has been used in previous works, such as ULIP2, to align the feature spaces of text, image, and point cloud representations. The purpose of such works is not solely to achieve high classification performance, but to deepen understanding of raw 3D representation beyond using just images. The original CLIP model, even without fine-tuning or special design, already serves as a strong baseline. This paper focuses on multi-view representation and achieves moderate improvements in basic classification and retrieval tasks by training on large-scale images, but this is quiet straight forward and it does not introduce new ideas or insights to the field. The work would be more compelling if the authors explored capabilities *beyond* the scope of those point cloud representations, such as scene-level multi-view data, to better demonstrate the necessity and potential upper bounds of the proposed approach.

3. Have the authors investigated the effect of different view sampling strategies? Unlike point cloud representations, which capture the entire shape, multi-view representations may be influenced by both the number and sampling strategy of views. While the impact of view count is examined in the paper, it would be beneficial if the authors also explored the impact of different sampling strategies, such as front versus back views, or uniformly sampled versus randomly sampled views.

4. Environmental lighting can significantly affect the appearance of 3D renderings by introducing shading, reflections, and slight color variations. It would be useful to conduct experiments examining how these lighting conditions impact the model’s performance.

5. Maybe add some more downstream tasks like image based retrieval and the concept combination as in Figure 1 of OpenShape.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: - Why is this an important task? What are some example applications of the proposed model? The paper seems to indicate that there are some applications in robotics. I'd like to see some examples of how the proposed technique can be used in robotic tasks and how exactly can current robotics methods can benefit from it.
- Can the model generalize outside of the training datasets? I didn't see any in-the-wild evaluation in the paper, which I feel is necessary as a metric for evaluation.
- Isn't it a little cyclic to use features from a frozen CLIP to train a multiview image encoder? Would the encoder benefit from some inter-view and inter-object contrastive losses? Wouldn't that allow the model to discover novel semantic and geometric information which might not have been captured by CLIP and text encoder?
- Can the learned multiview encoder be used for 3D reconstruction?
soundness: 4
strengths: - The paper is well-written. The figures and well-made and easy to understand. The overall presentation is good.
- The idea is simple and easy to understand. The architecture proposed might be reused for other multiview image tasks.
- The quantitative results are pretty strong and convincing.
summary: The paper proposes a multi-view 3D understanding model based on contrastive learning. The model is a multiview image encoder encoding variable number of input frames and output a feature vector. The model is trained by constrasting against the corresponding clip and language embeddings.
weaknesses: - It's unclear to me why this task is important. The proposed method seems pretty incremental and I don't see a clear insights or surprising findings reviewed by the paper

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Clarity: What is the difference between 5F and 12F as described in the tables (e.g. two bottom rows of Table 2)? This notation is not defined, although it's possible to conclude that this might mean number of views?
- What is the effect of the choice of pre-trained backbone for the multi-view encoder? How much worse would a DINO or randomly initialized model do? It would be good to understand the effect and significance of this choice. 
- What are specific applications in robotics for which multi-view images are available in real time and where improved recognition from multi-view images would have a positive and significant impact?
soundness: 3
strengths: ### Originality
- It is reasonable to use pre-trained CLIP encoders and fine-tune them for 3D shape understanding from multiple-views. This allows for training on more data and taking advantage of the large-scale pre-trained representation.
- Attention implemented  across multiple views of a single object is reasonable, and ablation show that it helps.
### Quality
- The investigation into the effect of replacing the point-cloud encoder with a multi-view image encoder is thorough, including analysis about training/testing time inference costs, comparisons with non-fine tuned CLIP backbones, and comparisons with baselines on image-based retrieval, text-based retrieval and classification.
### Clarity
- The paper is mostly well written and easy to follow with the exception of the experiments section, which could be improved (see comments in weaknesses).
### Significance
- For tasks like retrieval and classification it is good to know that point-cloud representations are weaker than multi-view image representations, which is the main takeaway of this work. Point cloud representations find their main applications in robotics, often in object manipulation tasks that do not only require recognition and where real-time multi-view data collection is cumbersome (see comments further comments on significance in weaknesses)
summary: This work presents Douduo CLIP, a technique for 3D representation learning for classification and retrieval that leverages multi-view input images rather than more explicit 3D representations like point clouds or meshes. The method is simple but effective approach to fine-tuning CLIP for 3D understanding, performing CLIP-style contrastive training between multi-view image encodings (a shape representation), single image encodings and text encodings. The multi-view image encoder is initialized from CLIP and fine-tuned with self-attention across all views for some layers of the ViT encoder, while the single image and text encoders are frozen (this allows for more efficient training).

The proposed Duoduo CLIP outperforms or is competitive with point cloud-based baselines for classification on Objaverse-LVIS and MVPNet, and outperforms CLIP applied zero-shot without any tuning. There are reasonable qualitative image-based retrieval and state-of-the-art zero-shot results (no specialized training for this task) for text-based retrieval.
weaknesses: ### Originality
- The main technical novelty is replacing a point cloud encoder with a multi-view CLIP image encoder, and tuning this multi-view encoder. While effective, the novelty is minor. It would also be good to understand the significance of what pre-trained representation (CLIP vs. DINO for example) is used to initialize the multi-view encoder
### Quality
- Accuracy of evaluation: Both the Objaverse LVIS and the new MVPNet use somewhat automated procedures using off-the-shelf VLMs to create the object labels. Without additional filtering the Objaverse LVIS dataset is quite noisy and this is likely the case for BLIP (what was used for MVPNet) as well. This is also the case for techniques purpose-made for captioning like Luo et al. (NeurIPS 2023). What is the significance of the findings if the ground truth labels themselves are inaccurate and might contain similar mislabeling patters as the CLIP model that is used as the pre-trained representation in this work?
### Clarity
- The flow of the writing in the experiment section can be improved, the sentences don't flow into each other, but read as consecutive lists of facts (e.g.L301-314). It's possible to understand with some effort, but a revision would greatly improve the paper.
### Significance
- Point clouds are a data modality generally used for tasks that require fine-grained object shape understanding and real-time visual feedback in robotics, like object manipulation. The present work focuses on recognition related tasks, so the significance in robotics (emphasized by the authors) is not entirely clear.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: As posted in weakness, I hope the author could solve these concerns in the rebuttal phase.
soundness: 3
strengths: Here is the strength of this paper:
- The authors propose to use rendered multiview images as shape representation instead of point cloud inputs for retrieval task.
- It proposes a training strategy that requires significantly less computation resources. 
- The author claims SoTA performance in comparison with baseline models.
summary: The paper introduces a novel method named Duoduo Clip, which addresses limitations in 3D shape representation by using multi-view images over traditional point clouds, improving adaptability and efficiency in 3D shape retrieval. Unlike previous methods constrained by point cloud limitations and high computational demands (e.g., Uni3D), Duoduo CLIP leverages multi-view images to simplify training, generalize better to unseen shapes, and reduce the need for extensive compute resources. This approach allows direct input of images, enabling seamless integration with real-world datasets (e.g., ScanNet, ScanObjectNN) and enhancing real-time applicability in robotics.
weaknesses: Here is some concerns from me:

**Major:**

- **Novelty:** While this paper proposes a more efficient approach to leverage multiview image for 3D understanding. Using multi-view (MV) attention to establish connections between multiple views for fine-tuning is not a novel idea. This limits the technical contribution. The primary innovation appears to be shifting from single-view images to multi-view images for training, which is incremental rather than groundbreaking. Also, this paper is similar with: `Yipeng Gao et al. Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-training CVPR2024`. Which use point cloud, multi-view image and text to conduct contrastive learning for multi-modality alignment. Author should explain about it.


- **Experiment.** I am also concerned about the potential degradation in performance on natural single-view images. The training dataset is smaller compared to the original CLIP model, and some of the data are synthetic. Although the proposed method improves performance on multi-view images, I worry that it may compromise the model’s ability to handle single-view natural images, which could negatively impact its performance in downstream applications.

**Minor:**

1.Line 80:
A citation is missing for the reference to "previous methods..." 

2. There is some confusing in Figure 3:
- So why the first image all focus on the plate when multiple object are present? What is the query?
- I also question the claim that this figure demonstrates "geometric correspondences of 3D shapes." It could be interpreted as a simple classification heatmap, which does not necessarily indicate the model is aware of 3D structure.

Meta-review: The authors introduce a method to obtain shape embeddings for shape retrieval from multi-view images of the object. It is demonstrated that the method clearly outperforms previous retrieval methods in retrieval quality and efficiency. The paper received positively leaning reviews, slightly above the accept score, with one reviewer questioning the purpose of the solved task.
After taking a closer look at the paper I decided to agree with the majority and recommend to accept the paper.

Decision: Accept (Poster)