OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding
Authors: Minghua Liu
Ruoxi Shi
Kaiming Kuang
Yinhao Zhu
Xuanlin Li
Shizhong Han
Hong Cai
Fatih Porikli
Hao Su
Venue: NeurIPS 2023 poster (2023)
Keywords: 3d
shape understanding
open-world understanding
zero-shot 3D classification
vision-language model

Abstract: We introduce OpenShape, a method for learning multi-modal joint representations of text, image, and point clouds. We adopt the commonly used multi-modal contrastive learning framework for representation alignment, but with a specific focus on scaling up 3D representations to enable open-world 3D shape understanding. To achieve this, we scale up training data by ensembling multiple 3D datasets and propose several strategies to automatically filter and enrich noisy text descriptions. We also explore and compare strategies for scaling 3D backbone networks and introduce a novel hard negative mining module for more efficient training. We evaluate OpenShape on zero-shot 3D classification benchmarks and demonstrate its superior capabilities for open-world recognition. Specifically, OpenShape achieves a zero-shot accuracy of 46.8% on the 1,156-category Objaverse-LVIS benchmark, compared to less than 10% for existing methods. OpenShape also achieves an accuracy of 85.3% on ModelNet40, outperforming previous zero-shot baseline methods by 20% and performing on par with some fully-supervised methods. Furthermore, we show that our learned embeddings encode a wide range of visual and semantic concepts (e.g., subcategories, color, shape, style) and facilitate fine-grained text-3D and image-3D interactions. Due to their alignment with CLIP embeddings, our learned shape representations can also be integrated with off-the-shelf CLIP-based models for various applications, such as point cloud captioning and point cloud-conditioned image generation.

OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding
Minghua Liu
Ruoxi Shi Shizhong Han?
Kaiming Kuang Hong Cai Hao Su'
Yinhao Zhu Fatih Porikli
Xuanlin Li!
UC San Diego
2 Shanghai Jiao Tong University
Qualcomm Al Research
Project Website: httpp::/ccola997.gtthhbb htt.thhSii///ppSShapppp
Abstract
We introduce OpenShape, a method for learning multi-modal joint representations text, image, and point clouds. We adopt the commonly used multi-modal con- trastive learning framework for representation alignment, but with a specific focus on scaling up 3D representations to enable open-worke 3D shape understanding. To achieve this, we scale up training data by ensembling multiple 3D datasets and propose several strategies automatically filter and enrich noisy text descriptions. We also explore and compare strategies for scaling 3D backbone networks and introduce a novel hard negative mining module for more efficient training. We evaluate OpenShape on zero-shot 3D classification benchmarks and demonstrate its superior capabilities for open-world recognition. Specifically, OpenShape achieves zero-shot accuracy of 46.8% the 1,156-category Objaverse-LVIS benchmark, compared less than 10% for existing methods. OpenShape also achieves an accu- racy of 85.3% on ModelNet40, outperforming previous zero-shot baseline methods by 20% and performing on par with some sully-supervised methods. Furthermore, show that our learned embeddings encode wide range of visual and semantic concepts (e.g., subcategories, color, shape, style) and facilitate fine-grained text- 3D and image-3D interactions. Due their alignment with CLIP embeddings, our learned shape representations can also be integrated with off-the-shelf CLIP- based models for various applications, such point cloud captioning and point cloud-conditioned image generation.
Introduction
3D shape understanding has recently garnered surge of interest driven by the growing demands in real-world applications, such as augmented/virtus. reality, autonomous driving, and robotics. Despite significant advancements 3D recognition and analysis, existing data-driven approaches are still greatly limited by the scale of 3D training datasets and tend to exhibit poor generalization when facing unseen shape categories, hindering the deployment of existing models real-world applications.
Note that 3D shapes and 2D images can be casily linked through rendering, and the dataset scale issue of 2D images has been remarkably addressed, as shown in recent works such as CLIP (55). Therefore.
many recent studies aim to utilize trained 2D image-language models [55] 55 to assist 3D tasks, such as 3D generation
22
and 3D scene-level segmentation 18
"Equal Contribution
Qualcomm AI Research is initiative of Qualcomm Technologies, Inc.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Recw
QLPPPPadd
us Ietalled
Gondiago
92
Motohennt (%6
1op1 Top5
3
Recom
cass
(aeria)
Figure I: Left: sero-shot shape classification the Objaverse-LVP (1,156 categories) and Model- Net40 datasets. OpenShape outperforms previous methods by a large margin. We exclude shapes in Objaverse-LVIS during training, and also retrain ULIP 78 for fair comparison. Right: Our shape representations encode a broad range of semantic and visual concepts. We input two 3D shapes and use their shape embeddings to retrieve the top three shapes whose embeddings are simultaneously closest to both inputs. See Section. 4.4 for more details.
ensembled training shapes
Regarding 3D shape-level understanding, a straightforward idea is project 3D data the 2D domain through rendering and use CLIP to analyze the 2D images, thereby enabling zero-shot 3D shape classification [86] [8). However, these methods suffer from occlusion and information loss during projection, and unnecessary latency due to point cloud rendering and multiple CLIP inferences.
To overcome the limitations caused by projection, it is necessary to train a 3D-native model by distilling knowledge from pretrained 2D models. However, training a 3D-native model requires set of 3D shapes, and the amount of knowledge that can be distilled is determined by the size of the 3D dataset. For example, ULIP 78] aims learn a joint representation space between language, 2D images, and 3D shapes, but uses a small-scale 3D dataset ShapeNetCore [8] for knowledge distillation. Specifically, ULIP fixes the 2D CLIP text and image encoders and trains a dedicated 3D-native point cloud encoder to extract 3D shape representations. The 3D encoder strives to align the 3D shape embedding space with the CLIP image and language embedding spaces by utilizing contrastive learning across all three modalities. However, since ULIP is only trained on 52K shapes of 55 object categories, still struggles with out-of-distribution shape categories and fails to demonstrate an impressive open-woneee world understanding of 3D shapes.
In this work, we propose novel method called OpenShape, which follows a similar paradigm as ULIP but aims to achieve a more generalized and scalable joint representation space encompassing language, 2D images, and 3D shapes. Our focus mainly lies on scaling up representation learning and addressing corresponding challenges. In OpenShape, we emphasize four key factors during the training process: (a) data scale: we significantly increase the scale of 3D training data by combining four public 3D shape datasets, resulting in 876k 3D shapes covering much more diverse categories; (b) text quality: the 3D shapes from our main dataset, Objaverse [12, is dominated with inaccurate or uninformative text descriptions. Given the data scale, we propose three strategies to automatically filter and enrich the text descriptions; (c) 3D backbone scaling: since most existing 3D backbones target small datasets, find that it's important but non-trivial to scale up the 3D backbones; and (d) data resampling: since the ensembled dataset is highly unbalanced, we utilize hard negative mining to improve the model's discriminative ability.
We first evaluate OpenShape on the zero-shot 3D shape classification task. As shown in Figure [ OpenShape outperforms previous zero-shot approaches on the ModelNet40 dataset by at least 20% Moreover, ShenS excels at handling long-tail categories. On challenging Objaverse-LVIS dataset, which contains 1,156 categories, OpenShape achieves a 46.8% accuracy, significantly surpassing previous methods. Notably, this performance gap remains even when ULIP retrained on ensembled datasets, highlighting the superiority of our text enrichment and training strategies.

---

Besides zero-shot classification, we present demos that showcase the wide range of visual and semantic concepts learned by OpenShape. For example, in Figure we take two 3D shapes as input and use their OpenShape embeddings to retrieve the top three shapes whose embeddings are simultaneously closest to both inputs from our ensembled dataset. The retrieved shapes exhibit an interesting combination of the semantic and geometric elements from both input shapes. Furthermore, since we align our 3D shape embedding space with the CLIP language and image embedding space, we demonstrate that OpenShape embeddings can be easily integrated with other CLiP-based models to perform mross-modality tasks such as point cloud captioning and point cloud-conditioned image generation.
2
Related Work
2.1 CLIP for 3D Learning
Image-language models like CLIP have achieved remarkable performance through large-scale image-
4 56 36] As these models excel at capturing rich visual concepts and possess impressive zero-shot capabilities, they have been applied various 3D vision tasks. For instance, numerous recent works utilize CLIP to facilitate zero-shot text-to-3D generation [22] 26
text pretraining 55 29
35
84
45 63 33
typically through CLIP-guided per-scene optimization. From recognition perspective, some works focus scene-level representation, aiming to leverage CI IIIP priors for zero-shot 3D segmentation or detection in both indoor [18 24 77
32 28
82 23 (0)
31 and outdoor scenes [ 21). Meanwhile, another line of work focuses on shape-lever understanding.
targeting zero-shot shape classification [86 88 53
78 19] and part segmentation [39 [ There are
two primary working paradigms for these methods. The first [66] 88 24 involves using images as medium representation, projecting 3D point clouds into 2D and employing 2D CLIP for inference. However, these methods typically suffer from occlusion and information loss during projection, along with unnecessary latency due to point cloud rendering and multiple 2D CLIP inferences. The second paradigm involves training a 3D-native encoder attempting distill or fuse CLIP features into 3D representations. Our paper follows this paradigm.
2.2
3D Shape Representation Learning
Various works have studied self-supervised pretraining for point clouds by designing pretext tasks [I5]
[ 66) such self-reconstructioon 13 [22, masked auto-encoding [48] 80 20), distortion
reconstruction [64] 44 68), normal estimation (57]. and contrastive learning [8 62 76.. These tasks
enhance models' shape representations and improve their performance downstream applications, although they do not involve multimodal semantic alignments during pretraining.
Recently, some works [53
78 19, exemplified by ULIP 778. have explored learning multimodal joint
representations for 3D shapes. They train 3D-native shape encoders by aligning 3D shape embeddings with CLIP's language and/or image embeddings through multimodal contrastive learning. Works like ReCon (33 further combines cross-modal contrastive learning with masked auto-encoding for added enhancement. While these methods allow for zero-shot 3D classification through the computation of 3D-text similarity, the amount of distilled knowledge and their model capability are heavily limited by the small-scale training datasets used. Our work follows this paradigm but aims to learn more generalizable and scalable representations to enable open-world 3D shape understanding.
3 Method
We propose a novel method, OpenShape, for learning generalizable and scalable multi-modal joint representation between language, 2D images, and 3D shapes, as shown in Figure [2] We first introduce the multi-modal contrastive learning framework we used for aligning representations of
three modalities in Section 3.1 We then elaborate how we create our training sets and enrich our text
data in Sections and 3.3 Section (.44 we present how we scale up our 3D backbone models.
Finally, we propose hard negative mining strategy to enhance contrastive learning in Section 3.5
3

---

Obyeverse (2688.80 583
Shapphat Skj
original texts
GPT4
fitered
Reouder
Text-to-30 (Retrieval)
Image-to-3D
(Retrieval)
Imago Capton
captions
Zero-Shof
Cassilication
OD-FUTURE
116
ABO 18.k
rendorings
Imege Retrieval
relrieved
Probe
3D-10- (Captioning)
3DHo-Image (Generation)
Hand Negaliva
Miriin
(a) Ensemble Datasets
(b) Text Filtering & Enrichment
Cross-Modal Alignment (c) Cross-
(d) Cross-Modal Applications
Figure 2: (a) We ensemble four public 3D shape datasets, resulting in 876k shapes that encompass diverse categories and concepts. (b) We propose three strategies automatically filter and enrich the noisy texts in the original datasets. (c) We train a 3D point cloud encoder to align the 3D shape embedding space with the CLIP's text and image embedding spaces. We perform cross-modal contrastive learning with scaled 3D backbones and hard negative mining. (d) OpenShape embeddings can be easily integrated with other CLIP-based models, enabling various cross coos-tty tasks.
3.1 Multi-Modal Representation Alignment
We aim to learn 3D shape representations that are aligned with pretrained CLIP embedding spaces of language and image. As shown in 2 we train a 3D native encoder f that takes a 3D point cloud as input and extracts 3D shape feature. Following previous works [53] 78 19]. such as ULIP 78,, we utilize multi-modal contrastive learning for representation alignment. Since CLIP is pretrained on a much larger scale data, we freeze both its text encoder f and its image encoder f during feature alignment to preserve CLIP's feature priors and avoid model collapse. Specifically, given a sampled batch of triplets {(P,, Tú }.)), where P: denotes point cloud of a 3D shape, T; and I; denote corresponding text and image, the contrastive loss is calculated as:
log exp(l? K./T)) log Σ, explh! k//T) log exp(h? be kT/TT Σ, expth" 15/7) log E, exp(h! ((7T)) expp(p?
exp(h? [T)
explh! AT/T)
4n
where is the number of shapes in a batch; is learnable temperature; h, = (P bi = 9" (J" (T:))//l² (s² (T:)),, and h =g' g (1:))// (F'(L)) denote normalized projected features of P.. T., and L,, where g" and g are two learnable linear projections. Since fo and f are frozen, we extract all f (T:) and (T. before training and cache them for acceleration. most of our experiments, we utilize OpenCLIP ViT-bigG-14 [25] as the pretrained CLIP model.
3.2 Ensembling 3D Datasets
Since the scale and diversity of training triplets play crucial role in learning scalable shape represen- tations, we ensemble four currently-largest public 3D datasets for training as shown in Figure 2 (a), re- sulting in 876k training shapes. Among these four datasets, ShapeNetCore (8. 3D-FUTURE [16] and
ABO 11 are three popular datasets used by prior works. They contain human-verified high-quality 3D shapes, but only cover limited number of shapes and dozens of categories. The Objaverse [12 dataset is a more recent dataset, containing many more 3D shapes and covering significantly more diverse categories. However, shapes Objaverse are mainly uploaded by web users and not verified by experts, and thus have uneven quality and exhibit highly unbalanced distributions, necessitating further processing.
To create triplets for training, for each shape, we sample 10,000 points from the mesh surface and interpolate the point colors according to the mesh textures. We also render 12 color images from the preset camera poses that uniformly cover the whole shape. For datasets providing thumbnails, we include them as part of image candidates, since they typically capture the shape from better camera view. For the Objaverse dataset, we use the model name as the raw text for each shape. For other datasets, we utilize provided metadata to create raw texts (see supplementary for details). During each pretraining iteration, we randomly sample one rendered image or thumbnail for each shape, and apply standard augmentation the point clouds (78
3.3 Text Filtering and Enrichment
We find that only applying contrastive learning between 3D shapes and 2D images is insufficient to fuel zero-shot 3D classification, even when training on large-scale datasets. We conjecture that this is

---

azure: black dog sitingg on blue.. sitting
azure: "a hhs plastt chat on pink staped object"
namte: "DOG A- 1ol6 free
"Blrck 7 Faltuatio in loct while Labraser puppy Viny! Mural"
name: Tuk 09 Odt 2018 17:12:57
"arriciiaa"
other props done
azure: pair stsempune pooplees" bil:: Saloampurk gorplls 3d moder"
azure: up
bis: "cllo max f wello /rece
homewart syz detaliigg GPT4: Remove
Steampunk Gogglerr
"Steampunk Gogales Mace from hand
GPT4; Remove
"Milk AetMtx (RedWhhte
Figure 3: Text Filtering & Enrichment Examples In each example, the left section features the thumbnail, model name, and GPT-4 filtering results. The upper right section shows image captions from two captioning models, while the lower right section displays retrieved images and their corresponding texts.
caused by the inherent domain gap in CLIP's language and image embedding spaces, which is also observed by previous studies [37 70). Consequently, 3D-text alignment is not guaranteed even if we obtain good 3D-image alignments via contrastive learning. Therefore, we need explicitly align 3D shapes with text. Along this process, to facilitate better 3D-text alignment, we introduce 3 techniques improve the text quality: filtering, captioning, and image retrieval, as shown Figure (b).
Filtering. As shown in Figure 3 the 3D shapes from our main dataset, Objaverse, dominated with noisy text descriptions ("names") uploaded by web users. Many of the problematic texts can be identified from the text itself without seeing the corresponding 3D shape. We thus leverage a powerful large language model, GPT-4 [47). filter out inaccurate find that GPT-4 excels at recognizing irrelevant contents, such as timestamps, pure model numbers, incomprehensible descriptions, random filenames (e.g., new project), and random characters. Through GPT-4, we filter out about 30% of raw user texts. Note that we only filter the texts, and still keep all shapes for training. More details, such as the prompts used, presented in the supplementary.
uninformative text descriptions. We
Captioning. utilize BLIP 34] and the Azure cognition services to caption the 2D thumbnails (if' present, images rendered from fixed frontal view) of the 3D models, obtaining two texts for each shape. As shown in Figure 3 the captioning models can usually produce meaningful and descriptive captions that either enhance user-uploaded texts or replace low-quality ones. We also notice that the two caption models complement each other, leading to better performance.
Image Retrieval. In addition to image captioning, we also perform image retrieval to obtain additional descriptions of 3D models. We retrieve k-NN images of shape renderings from the LAION-5B dataset (65] using the CLIPP ViT-L retrieval index [6) We then take the captions of the k-NN images as the retrieved texts for our 3D models. Compared with captioning model generations, retrieved texts cover wider range of text styles. They can also include more fine-grained semantics than both the user texts and the generated captions (e.g.. "Labrador" in Figure 3)
In each iteration of pretraining, for each shape, we first randomly sample a text source category among the raw text (if unfiltered), the captions, and the retrieved texts. We then select text candidate from the selected category. We also apply the template- based prompt engineering technique used in ULIP (78 to both training texts and test-time category names. Specifically, we extend word or phrase to collection of templated simple sentences and take their average embedding.
3.4 Scaling Up 3D Point Cloud Backbones
Previous works on 3D point cloud learning have primarily focused on smaller-scale datasets like ShapeNet. These techniques may not be directly applicable our larger-scale ensembled dataset and need be scaled up accordingly. We find that different 3D backbones may exhibit distinct behavior and scalability when trained on datasets with varying sizes. Specifically, compare six popular
5

---

Table I: Comparison of different 3D backbones before scal- ing up their parameters. Models are trained on ShapeNet g ensembled dataset excluding Objaverse-LVIS 122
Zero-shot classification performance are evaluated on Model- Net40 773 and Objaverse-LVIS 12]
Train on ShapeNet
Train Ens-no-LVVSS
D00W
Model
AParam
|NNe40
O.IVIS
MNer40
O-IVIS
25
PointNer
5
1.3M
67.0
9.3
74.9
24.4
DGCNN
2,3M
67.8
9.0
74.2
24.8
15M
PointMLP
42
9.3M
73.5
12.9
82.9
36.6
PuintNeXi
S4
2.8M
72.6
12.2
81.6
33.8
PoinBERT
5.1M
70.3
10.8
84.5
37.0
Figure 4: Accuracy on Objaverse-
SparseConv
10
5.3M
70.7
10.6
78.8
31.7
LVIS 12 when scaling up the pa-
sid. dev.
2.3
1.4
3,9
5.1
rameters of different models.
backbones trained on ShapeNet or our ensembled dataset by evaluating their zero-shot classification performance on ModelNet40 [75) and Objaverse-LVIS datasets (for now, these backbones are trained with their original configurations and without scaling up model sizes). Objaverse-LVIS is subset of Objaverse dataset with human-verified category labels. With 1,156 categories, it serves as a suitable dataset for evaluating zero-shot long-tail classification, and we exclude all shapes of Objaverse- LVIS from this experiment. Results are shown in Table We find that when trained on ShapeNet, all backbones share similar performances. However, when trained on our ensembled dataset, the performance gap between backbones increases significantly. This suggests that while the original versions of these backbones share similar number of parameters, some may have been saturated when trained on small datasets, while others do not.
We also explore the performance and scalability of these backbones when scaling up the model sizes and training on our ensembled dataset, Please refer to the supplementary for details on how we scale up each model. As shown in Figure + we observe that all 3D backbones benefit significantly from model scaling. However, traditional backbones without shrinking hierarchical structure, such as DGCNN and PointNet, require operating completely on dense points or modeling the relationships (e.g., through kNN) between dense points. As result, they become more time-consuming and memory-inte seetory-iteesivve when sealed up compared to more modern backbones. We therefore select PointBERT [8]) Transformerrbasee and SparseConv [10] (convolution- based) as our 3D backbones for the remaining experiments, as they exhibit strong performance and scalability.
3.5 Hard Negative Mining
Our ensembled dataset exhibits high degree of class imbalance. Certain common categories, such as building, may occupy tens of thousands of shapes, while many other categories, such as walrus and wallet, are underrepresented with only few dozen or even fewer shapes. Consequently, when randomly constructing batches, it is unlikely that shapes from two confusing categories (e.g.. apples and cherries) will be contrasted within the same batch. Inspired by some previous works 58 we propose an offline hard negative mining strategy for improving the training efficiency and performance. Specifically, the first round of training, we train our model with random batches until is about to converge. We then compute the kNN for each shape in the learned 3D embedding space. In the second round training, for each iteration, we randomly select seed shapes and then obtain m neighbors from the kNN results of each seed shape, resulting 8 shapes per batch. In this way, confusing pairs are more likely to be selected in a single batch. However, this may also introduce false negative pairs (c.g., two apples) into contrastive learning. To mitigate this issue, leverage image and text embeddings to filter out pairs sharing similar texts when calculating the contrastive loss. Specifically, for two shapes i and selected from the same seed shape, if hg hi + > hς h/, where 1 and / are text and image embeddings, and ă a small threshold, we believe that the text embeddings of and are very close to each other, and we remove j from i's negative examples when calculating contrastive loss. By employing this strategy to construct batches, we observe faster and better model learning.

---

Table 2: Zero-shot classification Objaverse-LVIS 12), ModelNet40 75 and ScanObjectNN 77
training shape
Objaverse
ModelNe140
ScanObject
Method
source
Top1
Top3
Top5
TTop1
Top3
Top5
Top1
Top3
Top5
PoinsCLIPP 86
2D inferences, no 3D training
1.9
4.1
5.8
19.3
28.6
34,8
10.5
20.8
30.6
PoinCT.F IP v2
4.7
9.5
12.9
77.9
85.0
ReCon
1.1
2.7
3.7
61.2
73.9
78.1
42.3
62.5
75.6
CG3D
P
5.0
9.5
11.6
48.7
60.7
66.5
42.5
57.3
60.8
CLIP2Point 24
2.7
5.8
7.9
49.5
71.3
81.2
25.5
44.6
59.4
CTJI-PPimBERR (Official)
ShapeNet
6.2
13.6
79.0
84.4
51.5
OpenShope-SparecCan
11.6
21.8
27.1
72.9
87.2
93.0
52.7
72.7
83.6
OpenShage-PointEER
10.8
20.2
25.0
70.3
86.9
91.3
51.3
69.4
78.4
LLIP-PoinBBRRT (Retrained)
21.4
38.1
46.0
71.4
84.4
89.2
46.0
66.1
76.4
OpenShape SpurseConv
Ensembled
37.0
58.4
95.0
97.5
54.9
PointBRp
LVIS)
39.1
60.8
68.9
85.3
96.2
97.4
47.2
72.4
84.7
ULIP PoinBERT (Retrained)
26.8
44.8
52.6
75.1
88.1
93.2
51.6
72.5
82.3
OpenShape-Spareee
Ensembled
43.4
64.8
72.4
83.4
95.6
97.8
56.7
78.9
88.6
OpenShape PoinBERT
96.5
98.0
52.2
Experiments
4.1 Zero-Shot Shape Classification
We evaluate the zero-shot classification performances of our models on three benchmarks: the traditional ModelNer40 [75] and ScanObjectNN 711, as well as new benchmark, Objaverse- LVIS [1]. ModelNet40 and ScanObjacetNN consist of 40 and 15 common categories, respectively. Objaverse-LVIS is an annotated subset of Objaverse [12 and comprises 46,832 shapes among 1,156
LVIS 17 categories. With much larger base of classes than other benchmarks, Objaverse-LVIS presents challenging long-tailed distribution, making it better reflection models' performance in open-world scenarios. We compare OpenShape with existing zero-shot approaches, including PointCLIP (86). Poi2tCLIPv2 [88. ReCon [53] CG3D 19, CLIP2Point (24), and ULIP (78). Among them, PointCLIP 86 and PointCLIPv2 88 project point clouds into 2D images and directly utilize 2D CLIP for inference, while other methods leverage the CLIP embedding spaces for alignment and require 3D shapes for training. We report results on these basclines using their released checkpoints. To better analyze the source of performance gains, we also retrain the baseline ULIP 78 on our ensembled shape dataset, but we use the original texts in the four constituent datasets along with the official codebase without backbone scaling. We train OpenShape and ULIP on three different sets of training shapes: "Ensembled" denotes using all shapes from the four datasets; "Ensembled (no LVIS)" is the same but excludes all shapes from the Objavserse-LVIS subset; "ShapeNet" only includes shapes from the ShapeNet [8] dataset. Note that even when LVIS shapes are included in the training shapes (i.e., the "Ensembled" dataset), their test-time category labels are probably not included in their training texts. Please refer to the supplementary for more training and evaluation details.
Table 2] shows the results. We observe that OpenShape consistently outperforms prior approaches, when trained only on ShapeNet. When models trained they receive a significant performance boost. In this case, OpenShape still surpasses retrained ULIP by a significant margin, demonstrating the advantages of our text enrichment, backbone scaling, and other training strategies. Specifically, OpenShape greatly improves the classification accuracy on the long tail categories in Objaverse-LLVI from dull < 10% to 46.8%, outperforming the retrained ULIP by about 20 points and reaching a decent top-5 accuracy of 77.0%. These results demonstrate OpenShape's 's capability to recognize open-world world objects effectively. As for ModeINet40. OpenShape achieves 85.3% accuracy, surpassing previous methods by substantial margin of at least 20 percent. OpenShape also achieves impressive top-3 and top-5 accuracies of 96.5% and 98. 08.0% To the best of our knowledge, this is the first time zero-shot methods have matched the performance of a fully- supervised 3D learning method on ModelNet40, where OpenShape outperforms suple-super 3D ShapeNets [75] and VoxNet 43). In addition, on ScanObjectNN, which contains challenging real scans with noise and occlusion, OpenShape exhibits decent sim-to-real transfer capabilities. To contextualize, SeenShapp-SpprrCCom achieves 56.7% zero-shot accuracy on ScanObjectNN without specific sim-to-real training, which surpasses 52.7% reported by SKPConv [74), a recent method specially designed for sim-to-real transfer in point cloud classification tasks.
larger-scale ensembled dataset,
7

---

PointCLIP V2 Objaverse-LVIS
ULIP-Officil
ULlr-Retrained
ModellNet4O
ScanObjectNN
Top-1 Acc. (%)
50
90
80
40
60
20
80
50
10
50
40
30
1
16
0
2
4
8
16
012
2
4
16
of labeled training samples per class
Figure 5: Few-shot linear probing Objaverse-LVIS
12
ModelNet40
75
and ScanOb-
jectNN 70
We report the average performance over 10 random seeds.
Table Ablation study. Top zero- shot accuracies on ModelNet40 (75 and
PoinBERT
40
81
SpareeCoriv
Objaverse-LVIS 12 are shown.
45
35
77
Mosie 1 Acc.
O-LVIS Acc. (16)
Variant
O-LVIS
MNe140
O-LVIS Acc.
30
40
No Objaverse shapes
13.9
75.5
Only Objaverse shapes
41.6
79.2
No backbone up
78.7
20
35
No caption & retrieval
37.0
82.9
No text fillering
41.4
82.9
15
61
No point 1gb, only xyz
83.6
de
28
600
75%
100%
30
Base
Cap.
Ratr.
Full
No text contras, learning
23.3
67.4
No image contras. learning
41.0
81.0
Full
42.0
83.1
Figure 6: Ablation study on using different ratios train- ing data.
Figure 7:
Ablation study
on different text enrichment strategies.
Full hard mining
4.2 Few-Shot Linear Probing
In the literature, linear probing a common way to assess the representation learning capabilities of model. To perform linear probing, gather and freeze the representation vectors from all samples in dataset. Subsequently, we train linear classifier using these fixed vectors and few-shot class labels. We evaluate the accuracy of the linear classifier three benchmarks: Objaverse- -aaie [12]. ModelNet40 (75), and ScanObjectNN [71]. Figure Figuree5summmrizze summarizes the performance of OpenShape in comparison with ULIP 778 (official release and retrained versions) and PointCLIPv2 (88). On the most challenging Objaverse-LLVIS benchmark, OpenShape outperforms all other methods by large margin. Notably, zero-shot OpenShape beats few-shot linear probes of other methods. On ModelNet40 and ScanOhjectNN, we do not see a large performance margin between OpenShape and retrained ULIP. We hypothesize that for few-shot ModelNet40, the error is dominated by in-category sample bias rather than the representation quality; while for ScanObjectNN, the domain gap plays major role. Since both ShenShape and retrained ULIP are exposed to the same source domain of training objects, their few-shot out-of-domain generalization performances tend to be similar.
4.3 Ablation Study
We perform various ablations by training a scaled version of SparseConv [10 on the ensembled dataset and then evaluate it on the Objaverse-LVIS [1] and ModelNet40 (75) zero-shot classification benchmarks, unless otherwise specified. The results are shown in Table and Figures 6 and
Data and Model Scaling. We investigate the impact of training data by ablating (1) without or with only Objaverse shapes (Tab. 33 and (2) with different ratios of our ensembled dataset (Fig. (6). We observe that training with 1% of our ensembled dataset (about 8.8k shapes) achieves similar or better zero-shot performance than training without Objaverse shapes (about 77.1k shapes), indicating that the diversity of training data is sometimes more crucial than the scale. In addition, we compare the performances between scaled-up and non-scaled-up backbones. From Tab. we demonstrate that model scaling plays essential role when training large-scale ensembled dataset (also Fig, 4

---

Figure 8: 3D shape retrieval from image (left, mid) and point cloud (right).
"shark"
"swordtish"
"dolphin"
'goldfish"
hhggh heels"
"boots"
"slippers"
"sneakers"
"white conical mug"
"green cubie mug"
Bnd J PHARELART
"blue spherical mug'
GET
"orange cylinder mug"
"tki mug"
"viking mug"
"animal-shaped mug" "travel mug"
Figure 9: Text-input 3D shape retrieval. In each row, we show input texts on the left and two retrieved shapes for each text on the right. OpenShape embedding encodes a wide range of visual and semantic concepts and enables (a) retrieval of fine-grained subcategories (first two rows), and (b) control of attributes (e.g., color, shape, style) and their combinations (last two rows).
Text Filtering and Enrichment. As shown in Tab. both text filtering and text enrichment are beneficial for performance. We also investigate the specific text enrichment strategies to use for the SparseConv and PointBERT backbones. In Fig. [ observe that both image captioning and text retrieval are helpful, and including both yield the best results. Notably, PointBERT improves more than 10 points from text enrichment, highlighting the significance of enhancing text quality.
Other Aspects. We also conduct additional ablation studies on color information, contrastive loss components, and our hard-negative mining strategy in Tab. 3 We observe that OpenShape performs well with only zyz coordinates as input and no RGB color. While 3D-image contrastive loss is also helpful, observe that 3D shape-text alignment plays very essential role for model zero-shot generalization, which necessitates our text filtering and text enrichment strategies that significantly enhance text quality. Lastly, by employing our hard negative mining strategy, OpenShape effectively addresses the issue of unbalanced data distribution, leading to further improvements in performance.
4.4 Cross-Modal Applications
Multi-modal 3D Shape Retrieval. Through OpenShape multi-modal representations, can index and retrieve 3D shapes from images, texts, or point clouds. In this section, we retrieve 3D shapes from ensembled dataset by calculating the cosine similarity between input embedding(s) and 3D shape embeddings and performing KNN. As shown in Figure OpenShape is capable of retrieving visually or semantically similar shapes from single image or point cloud input. OpenShape embeddings encode a wide range of visual and semantic concepts. In Figure 99] we show that OpenShape supports retrieving 3D shapes from detailed descriptions, which include fine-grained subcategories, attributes, and their combinations. Note that these input texts are typically not present in the raw texts of the retrieved shapes, indicating that OpenShape effectively learns generalizable concepts across shapes. In Figure [ we provide a demo which takes two 3D shapes as inputs and retrieves the shapes that are simultaneously closest to both inputs. This is achieved by finding arg max, min(hf" 1i. hn hh,",, where hy and hy denote normalized shape embeddings of the two input shapes. We can see that the retrieved shapes integrate visual or semantic elements in an interesting manner, highlighting the rich concepts and priors encoded in OpenShape embeddings.

---

2.
"In large desert"
Captions:
style 1920s. 2. The ladder to the second floor. Goldlish the see- photo it day.
wonds"
(a)
Figure 10: (a) Point cloud captioning. (b) Point cloud-conditioned image generation. Our learned 3D shape embeddings can be integrated with off-the-shelf pretrained CLIP-based models (e.g., captioning and image generation models) to support various cross moss-modal applications.
Shape-Con Condi-Condi Multimodal Generation. As OpenShape's 3D shape representations are aligned with CLIP's image and text embedding spaces, they can serve as inputs into other CLIP-based models to facilitate various multimodal generation applications. For example, our 3D shape embeddings into ClipCap [46], an off-the-shelf image captioning model, along with perform point cloud captioning and point couu-coonditoone image generation (optional text prompt supported) without extra training or finetuning. Qualitative results shown in Figure 10 Please refer to the supplementary for more results and details.
show that by feeding
Stable unCLIP [56]. text-to-image diffusion model,
Limitation and Conclusion
We introduce OpenShape, novel approach for learning scalable and generalizable multi-modal joint representations for 3D shapes. OpenShape representations effectively capture a wide range of semantic and visual concepts, enabling superior capabilities for open- open-world 3D shape recognition. By aligming OpenShape with CLIP's embedding space, our shape embeddings can be integrated with off-the-shelf CLIP-based models for various cross-modality applications. Moving forward, there are several directions worth further exploration: during training, this still quite limited compared the counterparts. We hope that our work inspires future investments in more resources to build even more powerful 3D representations. (b) Part-level information. Our current shape representations mainly focus on global semantic and visual features, and it would be beneficial to add more part-level supervision during training. (c) Sim-to-real domain gap. Our model mainly trained on synthetic data, and it's challenging but crucial to explore explicit designs for reducing the domain gap with real- world shapes.
(a) More 3D data. While we utilized 876k 3D shapes
Acknowledgments
This work is supported in part by gifts from Qualcomm.
10

---

References
[1] Ahmed Abdelreheem, Ivan Skorokhodov, Maks Ovsjanikov, and Peter Wonka. Satr: Zero-shot semantic segmentation of 3d shapes. arXiv preprini AXiv:2304.049900 2023.
[2] Idan Achituve, Haggai Maron, and Gal Chechik. Self-supervised learning for domain adaptation on point clouds. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pages 123-133, 2021.
[3] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning repre- sentations and generative models for 3d point clouds. In International conference on machine learning, pages 40-49. PMLR, 2018.
[4] Jean-Baptiste Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204. 12220.//1990 /4198, 2022.
[5] Shivangi Aneja, Justus Thies, Angela Dai, and Matthias Nießner. Clipface: Text-guided editing of textured 3d morphable models. arXiv preprint crXiv:2212.00 0146, 2022.
[6] Romain Beaumont. Clip retrieval: Easily compute clip embeddings and build a clip retrieval system with them. bottps://githut www.
wwt.//oom/mmmmmom rtm1504/11110/0/ elip-retrieee
2022.
[7] Zehranaz Canfes, M Furkan Atasoy, Alara Dirik, and Pinar Yanardag. Text and image guided
3d avatar generation and manipulation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4421-4431, 2023.
[8] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hamrahan, Qixing Huang, Zimo Li, Silvio Savarese. Manolis Savva, Shuran Song. Hao Su, et al. Shapenet: An information- rich 3d model repository. arXiv preprint 1r12v:1512.00 03012, 2015.
[9]
Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. arXiv preprint urXiv:23010049) 2023.
[10] Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings the IEEE Conference Computer Vision and Puttern Recognition, pages 3075-3084, 2019.
[11] Jasmine Collins, Shubham Goel, Kenan Deng. Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, et al. Abo: Dataset and benchmarks for real-worid 3d object understanding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 21 126 21136, 2022.
[12] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. arXiv preprint arXiv: /
[13] Haowen Deng, Tolga Birdal, and Slobodan Ilie. Ppf-foldnet: Unsupervised learning of rotation invariant 3d local descriptors. In Proceedings of the European conference on computer vision (ECCV), pages
2018.
[14) Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Language- driven open-vocabulary 3d scene understanding. arXiv preprint arXiv:221 1633/2, 2022.
[15] Benjamin Eckart, Wentao Yuan, Chao Liu, and Jan Kautz. Self-supervised learning 3d point clouds by learning discrete generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8248 8257, 2021.
[16] Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao. 3d-future: 3d furniture shape with texture. International Journal of Computer Vision, 129:3313-332 2021.
[17] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern 2019.
recognition, pages
[18] Huy Ha and Shuran Song. Semantic abstraction: Open-world 3d scene understanding from 2d vision-language models. In Conference on Robot Learning, 2022.
[19] Deepti Hegde, Jeya Maria Jose Valanarasu, and Vishal M Patel. Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition. arXiv preprint arXiv:2303. / 2023. [20] Georg Hess, Johan Jaxing, Elias Svensson, David Hagerman, Christoffer Petersson, and Lennart Svensson. Masked autoencoder for self-supervised pre-training on lidar point clouds. In Proceedings of the IEFF/CVF Winter Conference on Applications of Computer Vision, pages 350-359, 2023.
11

---

[21]
Georg Hess, Adam Tonderski, Christoffer Petersson, Lennart Svensson, and Kalle Aström. Lidarclip or: How i learned to talk to point clouds. arXiv preprint (rXiv:2212.06858, 2022.
[22]
Fangzhou Hong. Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang. and Ziwei Liu. Avatarclip: Zero-shot text-driven generation and animation of 3d avatars. arXiv preprint arXiv:2205.0853: 2022.
[23] Rui Huang, Xuran Pan, Henry Zheng, Haojun Jiang, Zhifeng Xie, Shiji Song, and Gao Huang. Joint representation learning for text and 3d point cloud. arXiv preprint N.iv:2301.075555 2023.
[24] Tianyu Huang, Bowen Dong, Yunhan Yang. Xiaoshui Huang, Rynson WH Lau, Wanli Ouyang, and Wangmeng Zuo. Clip2point: Transfer clip to point cloud classification with image-depth pre-training. arXiv preprint arXiv:2210.01055, 2022.
[25] Gabriel Ihaarco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongscok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. If you use this software, please cite as below.
[26] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 867-876, 2022.
[27] Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, et al. Conceptfusion: Open-set multimodal 3d mapping. arXiv preprint arXiv:2302 0.244.111 2023.
[28] Nikolay Jetchev. Clipmatrix: Text-controlled creation of 3d textured meshes. arXiv preprint arXiv:2 10::2/00./// 12922, 2021.
[29] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- laiinnla representation learning with noisy text supervision. In International Conference on Machine Learning, pages
4904-4916. PMLR, 2021.
[30] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances in Neural Information Processing Systems, 33:21798-21
2020.
[31] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. arXiv preprini arXiv:2303.0953 09003, 2023.
[32] Nasir Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. Text to mesh without 3d supervision using limit subdivision. arXiv preprint arXiv:2203. 13333, 2022.
[33] Han-Hung Lee and Angel X Chang. Understanding pure clip guidance for voxel grid nerf models. arXiv preprini arXiv:2209. /5/72, 2022.
[34] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-
image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR, 2022.
[35] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10965-10975, 2022.
[36] Xuanlin Li. Yunhao Fang. Minghua Liu, Zhan Ling, Zhuowen Tu, and Hao Su. Distilling large vision- languae model with
generalizability. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9492-2503, 2023.
[37] Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung. and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 15:17612-17655. 2022.
[38] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Zexiang Xu, Hao Su, et al. One-2-3-45:
Any single image to 3d mesh in 45 seconds without per-shape optimization. arXiv preprint arXiv:2306. 16928, 2023.
[39] Minghua Liu, Yinhao Zhu, Hong Cai,, Shizhong Han, Zhan Ling, Fatih Porikli, and Hao Su. Partslip: Low-shot part segmentation for 3d point clouds via pretrained image-language models. arXiv preprint 2rX.v:2212.01.55, 2022.
[40] Zhengzhe Liu, Peng Dai, Ruihui Li, Xiaojuan Qi, and Wiig Fu. Iss: Image as stetting stone for text-guided 3d shape generation. arXiv preprint 22XY::22090044455 2022.
12

---

[41] Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. Open-vocabulag point-cloud object detection without 3d annotation. arXiv preprini 00X7082304.00788 2023.
[42] Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu, Rethinking network design and local geometry in point cloud: A simple residual mlp framework. arXiv preprint arXiv:2202.07123, 2022.
[43] Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for real- time object recognition. In 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 922-928. IEEE, 2015.
[44] Benedikt Mersch, Xieyuanli Chen, Jens Behley, and Cyrill Stachniss. Self-supervised point cloud prediction using 3d apatio-temporal convolutional networks. In Conference on Robot Learning, pages 1444-144 1454. PMLR, 2022.
[45] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text- driven neural stylization for meshes. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages I 13492-13502, 2022.
[46] Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning.
Xiviv preprint arXiv:2111.00 1.09734. 2021.
[47] OpenAI. Gpt-4 technical report, 2023.
[48] Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part I1, pages 604-621. Springer, 2022.
[49] Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. arXiv preprint arXiv:2211. 15654, 2022.
[50] Omid Poursaeed, Tianxing Jiang, Han Qiao, Nayun Xu, and Vladimir G Kim. Self-supervised learning of point clouds via orientation estimation. In 2020 International Conference on 3D Vision (3DV), pages 1018-1028. IEEE, 2020.
[51] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[52] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical
feature learning on point sets in a metric space. Advances in neural information processing
systems, 30, 2017.
[53] Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, and Li
Yi. Contrast with reconstruct: Contrastive 3d representation learning guided by generative 2023.
pretraining. arXiv preprint arXiv:2302.02::
[54] Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet+++ with improved training and scaling strategies. N2Kv::200600667000 2022.
[55] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-878788 0763. PMLR, 2021.
[56] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06/22 2022.
[57] Yongming Rao, Jiwen Lu, and Jie Zhou Global-local bidirectional reasoning for unsupervised representation learning of 3d point clouds. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 5376-5385, 2020.
[58] Joshua Robinson, Ching-Yao Yao Chuang. Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.0 04592, 2020.
[59] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[60] David Rozenberszki, Or Litany, and Angela Dai. sanguagge-rroudee indoor 3d semantic segmentation in the wild. arXiv preprint 2xXv:2204.00761 2022.
[61] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.
13

---

Photorealistic text-to-imae diffusion models with deep language understanding, arXiv preprint arXiv:2205. 14487, 2022.
[62] Aditya Sanghi. Info3d: Representation learning on 3d objects using mutual information
maximization and contrastive learning. In Computer Vision-ECCT 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX 16, pages 626-642. Springer, 2020.
[63] Aditya Sanghi, Hang Chu, Joseph G I Lambourne, Ye Wang, Chin-Yi Cheng, Marco Fumero, and Kamal Rahimi Malckshan. Clip-forge: Towards zero-shot text-to-shape generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18603-18613, 2022.
[64] Jonathan Sauder and Bjarne Sievers. Self-supervised deep learning on point clouds by recon- structing space. Advances in Neural Information Processing Systems, 32, 2019.
[65] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. arXiv preprint wttiv:2200.084000 2022.
[66] Charu Sharma and Manohar Kaul. Self-supervised few-shot learning on point clouds. Advances in Neural Information Processing Systems, 33:7212-7221, 2020.
[67] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero 13+++; a single image to consistent multi-view diffusion base model, 2023.
[68] Chao Sun, Zhedong Zheng, Xiaohan Wang, Mingliang Xu, and Yi Yang. Point cloud pre-training by mixing and disentangling, arXiv e-prints, pages arXiv-2109, 2021.
[69] Ali Thabet, Humam Alwassel, and Bernard Ghanem. Self-supervised learning of local features in 3d point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 938-939, 2020.
[70] Vishaal Udandarao. Understanding and fixing the modality gap in vision-language models. Master's thesis, University of Cambridge, 2022.
[71] Mikaela Angelina Uy Quang-Hien Pham, Binh-Son Hua, Thanh Nguyen, and Sii-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In Proceedings of the IEEF/CVF international conference on computer vision,
pages 1588-1597, 2019
[72] Hanchen Wang, Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt J Kusner. Unsupervised point cloud pre-training via occlusion completion. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9782-9792 2021.
[73] Yue Wang. Yongbin Sun, Ziwei Liu, Sanjay E Sarma. Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning on point clouds. Acm Transactions On Graphics
(tog), 1-15: 2019.
[74] Jean-Baptiste Weibel, Timothy Patten, and Markus Vineze. Sim2real 3d object classification using spherical kernel point convolution and a deep center voting scheme. arXiv preprint
arXiv:2/03.06134 2021.
[75] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015.
[76] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding, In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings, Part III 16, pages 574-591. Springer, 2020.
[77] Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao. Dream3d: Zero-shot text-to-333 3d synthesis using 3d shape prior and text-to-image diffusion models. arXiv preprint arXiv:2212. 1. 2022.
[78] Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Marthn, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning unified representation of language, image and point cloud for 3d understanding. arXiv preprint arXiv:22/2.05117, 2022.
[79] Jihan Yang, Runyu Ding, Zhe Wang, and Xiaojuan Qi. Regionple: Regional point-language contrastive learning for open-world 3d scene understanding. arXiv preprint 2:Xiv:230400096 2023.
14

---

[80] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the Computer Vision and Pattern Recognition, pages 19313-19322, 2022. [81] Xumin Yu, Lulu Tang. Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
IEEE/CVF Conference
[82]
Yihan Zeng. Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit-Yan Yeung. Zhen Yang. Xiaodan Liang. and Hang Xu. Clip 2: Contrastive language-inage-pnnn pretraining from real-world point cloud data. arXiv preprint arXiv:2303. /2417, 2023.
[83] Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Yan saarrage--mag--ppp
Yeung. Zhen Yang. Xiaodan Liang, and Hang Xu. Clip2: Contrastive
pretraining from real-world point cloud data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15244-15221
2023.
[84] Haotian Zhang. Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang. Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language understanding. arXiv preprint arXiv:2206. 220.:222200008366, 2022.
[85] Junbo Zhang, Runpei Dong, and Kaisheng Ma. Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip. arXiv preprint ar arXiv:2333.0477
2023.
[86] Renrui Zhang, Ziyu Guo, Wei Zhang. Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8552-8562, 2022.
[87] Zaiwei Zhang. Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of 3d features on any point-cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10252-10263, 2021.
[88] Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyao Zeng, Shanghang Zhang, and Peng Gao. Point- clip v2: Adapting clip for powerful 3d open-world learning. arXiv preprint arXiv:2211.11682 2022.
15

---

A
More Examples of Multi-Modal 3D Shape Retrieval
In Figures
11 and
12
we showease more examples of multi-modal 3D shape retrieval.
VILY
Figure 11: Image-input 3D shape retrieval. In each triplet, we present the input image and two 3D shapes retrieved using OpenShape embeddings from the Objaverse [2 dataset. Input images are from unsplash. com.
Figure 12: Point cloud-input 3D shape retrieval. In each triplet, present the input point cloud and two 3D shapes retrieved using OpenShape embeddings from the Objaverse 12 dataset.
B More Examples of Shape-Conditioned Multimodal Generation
In Figure 13 and Figure 14 we showcase more examples of point cloud captioning and point cloud-conditioned image generation.
C Details on Raw Text Generation and Filtering
C.1 Raw Text Generation
We leverage the metadata from the four datasets generate the raw texts. Although the original datasets may contain numerous attributes for each shape, we carefully choose the most informative ones compose the text, ensuring its quality and relevance.
Objaverse:W. We utilize the name associated with each shape to serve as the text.
ShapeNetCore: For each shape, we generate three types of texts: (a) the name, (b) the category name (with a total of 55 categories), and (c) the concatenation the sub-category names (with total of 336 sub-categories), separated by commas.
16

---

Burning candle the plate with gold background. chair for patient. mug of collee morning.
dragon flying the sky.
The pen toal drawing. light the darkness. close up
the track.
butterfly picture of a pocket wetch. The 1-shirt the logo. the glasses and plates.
Sports equipment for the gym.
Book: the story of
What is name of this scooter?
Figure 13: Point cloud captioning. In each row, we show the input point clouds on the left and the generated captions the right.
FRE SIRE SUPR.
Figure 14: Point coud-condittonee image generation. Each row shows three examples (input point clouds and generated images).
3DFuture:
For each shape, we generate two types of texts: (a) the category, and (b) the concatena- tion of category, style, theme, and material. separated by commas.
A ABO: For each shape, we generate two types of texts: (a) the item_ same_ and (b) the product_tyyy type.
In this way, generate
raw texts for each shape.
17

---

C.2 Raw Text Filtering
am analyzing 3D dataset with various text descriptions for the 3D models. However, many of these texts are inaccurate or uninformative, and therefore, not suitable as descriptions for 3D models. need your help to identify such incorrect texts. Specifically, if a text primarily consists of irrelevant or unin- formative content, such timestamps, model numbers, incomprehensible descriptions, random filenames (e.g.. "my project"), random characters, etc., (or phrase)
please respond with "N". If text contains clear that could potentially describe a 33 object, please respond with "Y". You will find list of texts below, and each line contains three-digit IID and associated text. For each text, please respond with "Y" or "N", following the ID number (e.g., "001 Y" or "002 N"). Please evaluate all 256 texts.
000 New project ( 19)
000 N
001 3Deceberr Chemestry
001
002 Fake Brand Soda Can
002 Y
003 Spartan Shild
003
004 Apple3d
004 Y
005 Landmine
005
006 FaunveinB-S
006 N
007 FIGURA
007
008 Sphero Blue
008 Y
009 Sofa
009
010 Maddox
010 N
011 A3 Complete
011
012 Suspension Bridge
012 Y
013 Maung
013
014 Captain-ameriin
014 Y
015 sphorb4
015 N
We employ GPT-4 [7 to filter out uninformative raw texts. To accomplish this, we divide all the raw texts into batches, each containing 256 entries, and process each batch independently using GPT-4. Here is an example illustrating the prompt we used and the corresponding response generated by GPT-4.
Afterwards, we combine all the responses to create the final filtering results, effectively removing approximately 30% of the raw texts.
D Details on the Backbone Scaling Experiment
In Figure 4 of the main paper, we investigate the performance and scalability of various backbones when scaling up their model sizes. For this experiment, we employ default resolution of 10,000 points for input point clouds, batch size of 200, and conduct the experiment on a single A A100 GPU. In general, instructions are given in the original paper of backbone, we scale up the model as instructed. Otherwise, we scale up the model by expanding width or depth (i.e., stacking blocks or layers). Specifically, we scale up each backbone as follow:
The scaling parameters are shown in Table We scaled PoinBERT to 72. IM parameters beyond the 32.3M version reported in Figure 4 of the main paper. However, this scale, the model dramatically overfits on the training data and performs worse on all benchmarks than the 32.3M version.
PointBERT 81
The smallest version (5.3M parameters) of the model is adapted from FinKowskiFCNN model by adjusting the width of the final convolution and linear layers. The remaining three models are adaptations of MinkowskiResNet, each varying the number of basic ResNet blocks used. See Table 5 for the specific scaling parameters.
CoarseConv 10
18

---

Table 4: Hyperparameters for scaling up PointBERT 81
# Parameters
Layers
Width
Heads
MLP Dim
# Patches
Patch Embed Dim
5.1M
256
4
1024
64
96
13.3M
512
8
1024
64
128
32.3M
12
512
8
1536
384
256
72.1M
12
768
12
2304
512
256
Table 5: Hyperparameters for scaling up SparseConv [0]
# Parameters
Convolution Layers
# Linear Layers
5.3M
29.0M
18
33.7M
41.3M
42
PointNeXt is proposed as scalable version of PointNet++ [52]. and includes S/B/L/XI, variants in the original paper. We simply adopt these official configurations.
PointNeXt [54]
DGCNN [73] and PointNet [51] For these two backbones without hierarchical structure, increase the width of each layer proportionally to scale up to 4xPointNet and 2xDGCNN before we hit the GPU memory limit. As the models operate completely dense points, is impractical to the default 10k-point resolution. We thus reduce the input resolution for the two backbones, resulting in 1k points for DGCNN and 4k points for PointNet.
E Details on Training and Evaluation
Training Details We freeze the CLIP text and image encoders and train the 3D encoder and two projection heads on our ensembled dataset using the cross-modal contrastive loss. We train the model on a single A100 GPU with batch size of 200. Since we precache the text and image CLIP embeddings of all shapes, the training is greatly accelerated and takes about 300 A100 hours for convergence. We utilize an exponential learning rate schedule, and employ an range test to find initial learning rate. For 32.3M version of Poin we utilize learning rate of 5e 4; for 7M version of PointBERT, we utilize learning rate of 4e - 4; and for other models, we utilize learning rate of 1e 3. For hard-nege vergtivge mining, the number of seed shapes is set 40, and the number of neighbors m is set to 5 per shape, and the threshold is set to 0.1.
we con- ducted experiments to unfreeze and finetune the CLIP text encoder for a single epoch. However, the results obtained did not demonstrate any noticeable improvement on the benchmarks. Moreover, we observed that finetuning the CLIP text encoder could potentially undermine the generalization capabilities of CLIP and hinder the integration of OpenShape embeddings into existing CLIIP-based models. As result, we choose to freeze the CLIP encoders throughout the entire training process.
Fine-tuning CLIP Text and Image Encoders? After training OpenShape-PoiniBel
Evaluation Details We evaluated all baselines using their publicly released pretrained checkpoints. Additionally, we retrained ULIP 78 on our ensembled training shapes using their official code base and backbone networks. Note that the retrained ULIP model utilized the original raw texts from the four datasets during training (prompt engineering is also applied), rather than our filtered and enriched texts. For ModelNet40 (75). the evaluation is conducted on the test split with 2,468 shapes. Regarding ScanObjectNN 771. we follow ULIP 78 evaluate on the OBJ ONLY version, which contains 581 test shapes. For Objaverse- LVIS [1]. the input is 10,000 sampled points with point colors. For ModelNet40 75). the input 10,000 sampled points without color. For ScanObjectNN [71,, we utilize the official 2,048 points without color as input. All methods use the same input during evaluation. The forward inference time on A100 GPU for a 10,000-point point cloud is approximately 0.9ms for OpenShape-Sparreeonm and 3.8ms for OpenShape- PonnSShappPPPiBBRRT...
19

---

Details on Shape-Conditioned Multimodal Generation
Point Cloud Captioning CLIPCap 46 utillzes 10-token prefix generated from CLIP image embeddings to enable GPT-2 for captioning. In order to align with the off-the-shee CLIPCap model, we trained a variant of OpenShape- PointBBe that employs CLIP ViT-B/32 embeddings instead of OpenCLIP ViT-G/14 used in other experiments. Consequently, we directly input the point cloud encoding, withou normalization, into CLIPCap for captioning.
Point Cloud Conditioned Image Generation We take the Stable Diffusion v2.1 unCLIP model [56] for image generation and replace the CLIP image condition encoder with our OpenShape encoder to perform image generation conditioned on point clouds (and optionally text prompts). The unCLIP model takes CLIP ViT-L/14 embeddings without normalization as input. To match the embedding space, we trained variant of OpenShapee-PonnBBIP.
with CLIP ViT-L/14 embeddings. Additionally, we noticed a significant mismatching of scales (L2-norm of embedding vectors) be- tween ViT-L/14 image embeddings and OpenShape embeddings. To mitigate this issue, perform re-normalization on OpenShape embeddings I2-norm of /768, which is our observed mean Lo-norm of ViT-L/14 image embeddings. We use 50 diffusion steps. The guidance scale can be tuned freely.
20


Reviews:
[{"review_id": "RtXryQz64D", "reviewer": "Reviewer_Q47N", "summary": "OpenShape explores scaling-up strategy for learning joint representations of texts, image, and 3D point clouds. It proposes to construct larger-scale 3D datasets, filter and enrich paired texts for pre-training, which achieves SOTA results on zero-shot 3D classification and retrieval benchmarks.", "questions": "None", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The zero-shot performance of OpenShape is impressive, surpassing existing SOTA methods with large margins. This demonstrates the significance of the method.\n\n2. The retrieval visualizations are interesting and clearly illustrate the embedding ability of OpenShape.\n\n3. The authors re-train exiting methods under new settings, which is good for a fair comparison.", "weaknesses": "1. OpenShape utilizes GPT-4 for textual-level processing, which however is expensive to access. How about the performance using a more affordable GPT-3 (like PointCLIP V2) ?\n\n2. I'm curious about how OpenShape can be incorporated with non-parametric 3D network Point-NN (or its parametric derivative Point-PN) for zero-shot learning?\n\nStarting from Non-Parametric Networks for 3D Point Cloud Analysis, CVPR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and valuable suggestions. We will revise our paper based on your feedback. Here are our responses to your comments:\n\n**GPT-4 is expensive to access**\nWe also tried GPT-3 in our experiments but found it less effective at reasoning and finishing the filtering task. In our experiments, the average cost of filtering 1000 shapes using GPT-4 is about $0.5, which is still competitive with manual filtering.\n\n**Incorporate with Point-NN or Point-PN?**\nThanks for pointing this out. Incorporating the parametric version Point-PN into our framework seems possible and interesting. Since Point-PN has only a small number of learnable parameters, it may be more effective when scaling up the model size and enabling large batch sizes. The non-parametric version Point-NN may not be very suitable for our scaled-up experiments since it requires creating a large-scale feature memory for all training shapes, and comparing with such a huge feature memory may be less efficient during inference. Also, it’s not straightforward to enable zero-shot classification (arbitrary texts) using Point-NN, since the point embeddings are required to align with CLIP text embeddings.\n"}, {"review_id": "TU6g1rOpfS", "reviewer": "Reviewer_Mx8V", "summary": "This work learns a multi-modal representation among text, images, and point clouds based on a scaling 3D dataset. The results demonstrate remarkable performance in point cloud zero-shot classification, retrieval, and captioning tasks. In order to construct this dataset, the authors combined four commonly used 3D datasets while filtering and enhancing text prompts. Additionally, they introduced an offline hard negative mining strategy to improve the efficiency of the training process.", "questions": "- The main contribution of this work is the datasets ensemble. Will the authors release the collected dataset and pre-trained 3D representation, similar to ULIP?\n\n- How will multiple object combinations and scene models be addressed in the Objaverse dataset? Will such data be ignored? According to Table 2, pre-training on the assembled dataset has a negative impact on the ScanObjectNN dataset, indicating overfitting in the Objaverse dataset and a degradation of the transferability between the simulation and real domains.\n\n- Section 3.4 appears to be an experimental setup and should be placed in the experiments and analysis chapter.\n\n- Lack of related work: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data ", "limitations": "No needed", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper collects 0.8M text-image-point cloud triplets from four popular 3D datasets and does text cleaning and enrichment to get high-quality prompts for each object.\n2. This work proposes a hard negative mining strategy to improve joint representation training efficiency and performance.\n3. The trained representation shows surprising performance on zero/few-shot classification.", "weaknesses": "1. The major problem is an insufficient novelty compared with ULIP. The pertaining method, triplets construction and downstream tasks are very similar to ULIP.  The primary distinguishing point of this work is the text prompt cleaning and enrichment flowchart, as well as the offline hard negative mining strategy employed. However, the latter strategy lacks an ablation study to provide supporting evidence.\n\n2. The motivation behind this work is not clearly articulated. While the aim is to develop a scaling-up 3D representation, the purpose of this representation is unclear. Unlike ULIP, which focuses on enhancing current 3D backbones through developing a new pipeline, this work attempts to scale up text-to-3D pairs through text augmentation that is still limited by the insufficient number of 3D objects. \n\n3. There is a lack of exploration into the downstream applications of pre-trained representation to counterparts in 2D. This work is more suitable for submission to the dataset and benchmark track.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and valuable suggestions. We will revise our paper based on your feedback. Here are our responses to your comments:\n\n**The difference between the proposed method and ULIP** \n\nWhile we follow the multi-modal contrastive learning framework for representation alignment as used in many previous works (not just ULIP), our objective is to learn a useful 3D representation to enable open-world 3D shape understanding by scaling up the pre-training. Here are some main differences:\nULIP mainly experiments on small-scale datasets (e.g., ShapeNet), while we aim to pretrain on a large-scale ensembled dataset and address the arised challenges. The experiments show that even if ULIP is trained on our ensembled dataset, it’s still less effective than our method.\nULIP utilizes the original meta information (e.g., model category and name) of 3D dataset for training, while the meta information may be missing or noisy when scaling up the dataset. We thus propose several strategies to filter and enrich text descriptions for the 3D models, which is proved to be effective and critical.\nULIP utilizes the off-the-shelf 3D backbones with default settings. We explore the influences and scaling laws of 3D backbones when pretraining on a large-scale dataset, which is proven to be critical when learning a scalable representation.\nWe propose a hard negative mining module to address the severe class imbalance issue of the ensembled 3D dataset. The module is optional and can bring additional performance gain (as ablated in Table 3). Moreover, it can enable more efficient pretraining. We observe a ~2x speed up when using hard negative mining.\nIn addition to the supreme performance over ULIP (both original version and retrained version) on zero/few shot classification, we showcase a lot of cross-modal applications enabled by our powerful representations, which are missing from the ULIP paper. The applications include: (a) image and point cloud input 3D shape retrieval (Figures 1, 8, S1, S2), (b) text input 3D shape retrieval (Figure 9), which supports fine-grained subcategories, attributes and their combinations; (c) point cloud captioning (Figure 10 and S3); (d) point cloud conditioned image generation (Figure 10 and S4). We kindly refer the reviewers to the supplementary material for more exciting results. Note that the ULIP paper only shows some simple cases of image-based retrieval (coarse category-level), and does not demonstrate the capability of open-world understanding and supporting a wide range of downstream applications.\n\n**Purpose of this representation**\n\nOur goal is not to enhance a specific 3D backbone as in ULIP. Instead, we aim to learn a powerful multi-modal representation like in 2D CLIP that could enable open-world 3D shape understanding and directly benefit various downstream applications. As a result, we don’t focus on a specific 3D backbone and explore the scaling law of 3D backbones as well. In the paper, we demonstrate the open-world capability of our method through a wide range of cross modal applications.  \n\n**Still limited by the insufficient number of 3D objects**\n\nAs more and more large-scale 3D datasets become available (e.g., Objaverse-XL has 10M 3D shapes), we believe that 3D representation learning may no longer be limited by the scale of 3D shapes, and open-world 3D shape understanding is more likely to be realized than ever before. As a result, learning a powerful representation and addressing the challenges arose during the scaling up (the goal of this paper) may become more important and receive more attention. In addition to the large-scale 3D datasets, leveraging and distilling the 2D pre-trained models may also provide clues for this goal.\n\n**Lack of exploration into the downstream applications**\n\nIn addition to the supreme performance on zero/few shot classification, we showcase a lot of cross-modal applications enabled by our pre-trained powerful representations. The example applications include: (a) image and point cloud input 3D shape retrieval (Figures 1, 8, S1, S2), (b) text input 3D shape retrieval (Figure 9), which supports fine-grained subcategories, attributes and their combinations; (c) point cloud captioning (Figure 10 and S3); (d) point cloud conditioned image generation (Figure 10 and S4). We kindly refer the reviewers to the supplementary material for more exciting results, which demonstrates that our shape representations encode a broad range of semantic and visual concepts, and enable open-world understanding.\n\n**Code release**\n\nYes, we will release our code, processed training data, pretrained 3D representations, and demo!\n\n**Scene understanding**\n\nThe goal of this paper is single object understanding. However, we believe our work can be a good building block for scene-level 3D understanding, since 3D scene data is much less compared to 3D object data (e.g. 10M shapes). It is interesting to explore how our work can be extended to open-world scene understanding, which we leave as future work.\n\n**Degradation on scanned objects**\n\nYes, we also observe the performance degradation on the ScanObjectNN dataset mainly due to the sim-to-real domain gap. As mentioned in the “Discussion and Conclusions”, exploring explicit designs (e.g. data augmentation) to reduce the domain gap with scanned objects is challenging but critical and we leave it as future work.\n\n**Lack of related work**\n\nThanks for pointing this out. We will add the paper in our revision.\n"}, {"review_id": "VfP8yi2YHD", "reviewer": "Reviewer_9Jeu", "summary": "In this paper, the authors propose a joint learning framework for multi-modal representations among text, image, and point clouds. Specifically, the authors fix the pretrained CLIP language and image encoder, and align the point cloud representation by the proposed multi-modal representation alignment technique. Then, to enlarge the pretraining dataset, the authors introduce text filtering and enrichment technique to annotate point clouds without descriptions. The proposed OpenShape is pretrained on an large-scale ensembled dataset, and the experimental results demonstrate that OpenShape has promising zero-shot classification ability and shape retrieval ability. ", "questions": "See the weakness part.", "limitations": "The authors do not state the limitation of this method. Instead, the authors present the future direction of the proposed OpenShape, i.e., part-level information and synthetic-to-realistic domain gap. These directions can be seen as the limitation of this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method fully leverages the large-scale point cloud datasets to align the point cloud backbone with frozen pretrained CLIP encoders, thus obtaining open-world point could recognition and shape retrieval ability. \n\n2. The proposed text filtering and enrichment technique is effective to preprocess large-scale point cloud datasets. \n\n3. The overall writing is polished.\n\n4. The ablation study is extensive and critical.", "weaknesses": "My major concern is the fairness in the experiments (i.e., table 2). As the authors stated in section 3.1, the chosen vision-language encoder is OpenCLIP ViT-G-14, which is an extremely large backbone and obtains much better feature representation ability. Therefore, the authors should list the point cloud backbone and pretrained vision-languge encoders concretely in the table, and make fair comparisons if possible. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and valuable suggestions. We will revise our paper based on your feedback. Here are our responses to your comments:\n\n**Vision-language encoder**\n\nThanks for your suggestion. We will add the details of the point cloud backbone and pretrained vision-language encoders concretely in the table as shown in Table 1 of the PDF file (see main rebuttal).\n\nTo evaluate the impact of various pretrained visual-language encoders, we compared CLIP and OpenCLIP encoders on the Objaverse and ShapeNet datasets. As shown in Table 2 of the PDF file, while we do observe performance differences between different visual-language encoders, the gap is insignificant. Note that due to the time limit, we cannot train new models on the Objaverse (or ensembled) dataset at this time. We thus report the models that we have during our previous experiments.\n"}, {"review_id": "RrE63Wvw49", "reviewer": "Reviewer_quLe", "summary": "The paper studies the problem of multi-modal learning of text, image and shapes. Shapes are represented by pointclouds and the learning is driven by the standard contrastive loss. The main technical difference of this work to prior ones lies in the scale of data and corresponding training strategies. The authors train their model on a much larger scale of data which pools Objaverse, ShapeNet, 3D-Future and ABO. To handle the noise in the text (mainly from the less-curated Objaverse), the authors use several large pretrained models to clean up the text and make it better aligned with the shapes. Hard-mining is also leveraged to make the model train better under class imbalance. With these techniques which make the model scales better with data size, the proposed model achieves the new SOTA performance across several relevant tasks and datasets.", "questions": "Please see the weaknesses for my main questions and concerns. Overall I like the paper as it presents useful findings for the community and the model's performance is strong. But I think more analyses and discussions are necessary to solidify some experiments and the overall claim.\n\nAnother relatively minor question about the result:\nIn Table 2, the retrained ULIP achieves 26.8% top 1 while OpenShape achieves 43.4%. The authors attribute this to \"text enrichment, backbone scaling, and other training strategies\" (L243). However, it is shown in the ablation that removing text enrichment leads to 6.4% drop and hard mining leads to 1.4% drop. Does this mean the main improvement over ULIP comes from the larger number of parameters/better backbones as the overall drop is ~17%?", "limitations": "The limitations and failure cases are not discussed and should be included.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The model in this paper achieves significantly better results than previous SOTA in zero-shot shape classification. It also demonstrates great qualitative performance in shape retrieval, shape interpolation, shape captioning and has good potential in other cross-modal applications as well. Such model will be practically very useful.\n2. The benchmark and the findings in the paper about different models' scaling performance are valuable information to the community and future research along this direction.\n3. The paper is well-written and easy to follow.", "weaknesses": "1. The evaluation of the zero-shot shape classification on Objaverse-LVIS might not actually be zero-shot. Based on the description in L235, even the \"Ensembled (no LVIS)\" can include LVIS categories, because seemingly the author only excludes the exact evaluation samples from the training set. Ideally, all shapes from the evaluation *categories* should be removed from the training set. This makes the real zero-shot generalization performance of the model potentially lower than reported and the overall comparison on this test set less informative. Besides, ModelNet40 and ScanObjectNN can also have overlapping categories with the training set. (If the authors think category overlapping does not violate their definition of zero-shot, it should be clearly stated so / \"zero-shot\" should be clearly defined at the beginning.)\n\n2. The linear probe results show the proposed model performs similarly to ULIP-retrained on ModelNet and ScanObjectNN. The authors try to explain it with in-category sample bias (what does this mean?) and domain gap dominance, without providing any evidence for these hypotheses. My concern here is that, given that OpenShape performs much better than ULIP in shape classification (requires both shape and language rep) but not linear probe (only requires shape rep), the actual underlying reason could be that the shape representation is of similar quality when tested on out-of-domain, and the real difference between these models lies in the text representation. I believe it would be quite beneficial to provide more analyses on this result, as this is central to the \"shape representation learning\" story of the paper. For example, such analyses could be visualizing and comparing the latent structure of the shape representation; or simply performing shape-latent based NN query and evaluate the distance between queried shapes; or measuring the shape latent's alignment with image latent, to name a few. \n\n3. The limitation of the model is not discussed at all. This is a very important aspect for readers to thoroughly understand the contribution of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and valuable suggestions. We will revise our paper based on your feedback. Here are our responses to your comments:\n\n**Definition of \"zero-shot\"**\n\nThanks for pointing this out. We agree that the current \"zero-shot\" evaluation does not exclude all shapes in the evaluation categories from training. On the one hand, completely filtering out shapes in the evaluation categories is technically tricky, and most recent \"zero-shot\" 3D understanding papers have not done so. On the other hand, we feel that such an evaluation is not necessary. In the 2D CLIP paper, they also don't have such explicit filtering in their \"zero-shot\" evaluation. As more large-scale 3D datasets become available (e.g., Objaverse-XL has 10M shapes), the training set should cover most object categories. As a result, we believe that a more proper definition of \"zero-shot\" is the generalization to unseen datasets or evaluation instances. We will follow your suggestions to define that at the beginning clearly.\n\n**Similar linear prob performances on ModelNet40 and ScanObjectNN**\n\nIt's hard to figure out the actual reasons for similar linear prob performances on ModelNet40 and ScanObjectNN. We have some hypotheses:\n\nModelNet40 is relatively easy, and both representations are already good enough. Some confusing shapes mainly cause the remaining error. Many ModelNet40 shapes have very simple geometry and are normalized to a unit scale without color information. As a result, some shapes may have multiple confusing categories (e.g., TV stand, dresser, nightstand). The most effective solution to further reduce the error rate may become including more training shapes and overfitting the subtle bias.\n\nThe performance of ScanObjectNN is mainly dominated by the domain gap (e.g., noise from scanned objects). Since both models don't have explicit designs for reducing the domain gap, they share similar performance.\n\n\"In-category sample bias\" means that when the number of labeled training samples is small, the performance may be greatly affected by the sampled training shapes instead of the quality of shape representation. This is also the reason the few-shot performance varies across different runs, and we report the mean performance over ten runs.\n\n**Additional analyses on “shape representation”**\n\nTo analyze the learned shape representations, we added three additional small experiments as suggested by the reviewers:\n\n(a) Shap-latent-based NN retrieval: we utilize shape embedding to retrieve the nearest shapes from the Objaverse dataset. As shown in Figure 1 (see the PDF in the main rebuttal), our retrieved shapes capture more fine-grained geometry details.\n\n(b) Image shape alignment accuracy: we randomly divide the Objaverse-LVIS shapes into batches (196 shapes per batch) and calculate the alignment accuracy between the image (thumbnail of the shapes) and shape embeddings. As shown in Table 3(a), our method achieves a higher alignment accuracy.\n\n(c) We utilize real-world images and bounding boxes in the 2D LVIS dataset to create an image dataset comprising ~3k cropped images from 584 categories. Each resulting image (from a bounding box) is cropped and resized to 224x224 and is associated with a unique category name. For each image, we find the nearest shapes using shape embeddings. If the retrieved top 10/100 shapes of an image contain the corresponding category, we regard the retrieval as a success. We then report the success rate as R@10 and R@100 in Table 3(b). As shown in the table, our method achieves a higher success rate.\n\n**”Limitations are not discussed?”**\n\nWe want to clarify that we have discussed the limitations, including limited part-level understanding, sim-to-real gap, etc., in Section 5, “Discussion and Conclusion”. We will move the discussion to a new section, Limitations, in our revision.\n\n**The source of performance improvement**\n\nYes, we agree that scaling up the 3D backbone may play a critical role in the performance gain (~10 percent). However, other design choices, such as text filtering and enrichment, are also important. For example, without text filtering and enrichment, PointBERT will drop 10.5 percent, as shown in Figure 7 of the main paper.\n"}, {"review_id": "WOJQgePHfR", "reviewer": "Reviewer_zuqY", "summary": "The paper proposes a new system for learning a joint embedding space for text, images, and 3D shapes (point clouds). It starts from pre-trained CLIP embeddings for text and images, and learns one more embedding function for 3D shapes in a self-supervised manner, using a contrastive loss. Several additional improvements are proposed, including combining several datasets, cleaning some of those including via automated re-labelling of Objavrese, and optimisation of the 3D embedding neural network architecture. Because of these improvements, the new embedding performs significantly better than prior works on strand tasks such as \"zero-shot\" classification.\n\n", "questions": "* Filtering bad positive samples in contrastive the loss using the existing text and image embeddings (line 213) is simple but also likely useful, however it is not clear to me if this idea is ablated in Table 3. Can you clarify?\n\n* Why SparseConv and PointBERT are so different in performance depending on the test dataset? Is there a recognisable pattern?\n\n", "limitations": "The paper does address some limitations, particularly in the discussion and conclusions where they are implicitly recast as \"future work\".\n\nThere is no discussion of ethics, although this does not seem to be an issue for this paper (notoriously, though, dataset like ShapeNet had issues with copyright, and I don't believe ObjaVerse is necessarily immune from such controversies either).\n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The paper contributes sound incremental work on learning multi-modal embedding functions for text, images and 3D shapes. Various engineering dimensions are considered (model scaling, data diversity, scaling and quality, etc) and some insights are provided on how each of these can be improved, at least compared to current baselines. I expect these findings to provide useful guidelines for future work in this area.\n\n* The paper does contain some interesting suggestions on how to improve datasets such as Objaverse to train multi-modal embeddings. The scheme where captions are assessed via GPT-4 and then replaced using BLIP if needed is useful.\n\n", "weaknesses": "* The paper is mostly about good engineering, but there isn't a lot of very deep technical innovations, or qualitatively surprising findings. The main finding, in fact, is that this is an area where scaling is still very limited (mostly due to the lack of suitable training data), and that, thus, scaling is where most of the low-hanging fruits can be had. This is a good but not very surprising message.\n\n* The model is trained on synthetic datasets and looses much of its edge when applied to real data, compared to prior models like ULIP that are otherwise suboptimal when tested on synthetic data (e.g., Figure 5, right panel).\n\n* Likewise, there isn't a very clear dominance of the two proposed backbones, SparseConv and PontBert: their performance swings significantly depending on the testing data in Table 2.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and valuable suggestions. We will revise our paper based on your feedback. Here are our responses to your comments:\n\n**Limited qualitatively surprising findings**\n\nWe kindly refer the reviewer to the supplementary for more exciting results, where we showcase a lot of cross-modal applications enabled by our powerful representations, including (a) image and point cloud input 3D shape retrieval (Figures 1, 8, S1, S2), (b) text input 3D shape retrieval (Figure 9), which supports fine-grained subcategories, attributes, and their combinations; (c) point cloud captioning (Figure 10 and S3); (d) point cloud conditioned image generation (Figure 10 and S4). These applications demonstrate a broad range of semantic and visual concepts encoded in our representations and the capability of open-world 3D shape understanding, which has not been shown in previous works.\n\nWith more and more large-scale 3D datasets becoming available (e.g., Objaverse-XL has 10M shapes), we believe that learning a powerful 3D representation and addressing the challenges that arose during the scaling up (the goal of this paper) may become more important and receive more attention. Open-world 3D shape understanding is more likely to be realized than ever before.\n\n**Lose much of its performance edge when applied to scanned data**\n\nWe agree that the performance gap between our method and some baseline methods is small on ScanObjectNN. This is because the performance of ScanObjectNN is mainly dominated by the domain gap (e.g., noise from scanned objects). Since both models don’t have explicit designs for reducing the domain gap, they share similar performance. As mentioned in the “Conclusion and Discussion”, exploring explicit designs (e.g., data augmentation) to reduce the domain gap with scanned objects is challenging but critical, and we leave it as future work.\n\n**The performance of PointBERT and SparseConv**\n\nWe did not expect a significant performance advantage between the two backbones. Overall, PointBERT consistently has a slight advantage on clean data (Objaverse-LVIS and ModelNet40), as shown in Table 2 of the main paper. Their ranking changes only when it comes to some ScanObjectNN evaluation settings. We conjecture that the convolution-based network architecture (e.g., SparseConv) is more robust than transformer-based network architecture (e.g., PointBERT) and has better generalization ability to the noise and incompleteness of real scanned objects.\n\n**Filtering bad negative pairs**\n\nThe idea of using existing text and image embeddings to filter false negative pairs is important in hard negative mining since we utilize KNN shapes to build batches, and it’s highly likely to encounter false pairs in a batch (e.g., two apples in a batch). Without the filtering, the performance of hard negative mining will drop significantly in our experiments. However, the filtering is not necessary in our first-stage normal training, where batches are randomly constructed. Our experiments show that the filtering doesn’t affect the performance a lot for random batches since it’s less likely to encounter two apples in a batch."}]

Author rebuttal:
Dear Reviewers,

Thank you for dedicating your time to review our paper and for offering insightful feedback. We deeply appreciate your efforts to help enhance the quality of our research. We are also pleased to note that all reviewers were supportive of our work:

(a) Acknowledge our remarkable performance on zero-shot classification (Q47N, Mx8V, 9Jeu, quLe, zuqY) and shape retrieval (Q47N, Mx8V, 9Jeu, quLe).

(b) Praise our extensive and critical analysis and ablation studies, which provide valuable insights to the community and future research along this direction (9Jeu, quLe, zuqY).

(c) Find our applications are interesting and great, which clearly illustrate the embedding ability (Q47N), and the pretrained models can be practically very useful (quLe).

(d) Find our paper well-written and easy to follow (9Jeu, quLe).


Meta-review: This submission received unanimously positive reviews from the reviewers, who liked the formulation and performance. The AC agrees and recommends acceptance.  The authors are encouraged to incorporate the feedback in the camera-ready version.
Decision: Accept (poster)