Title: OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding
Venue: NeurIPS 2023
Accepted: yes
Keywords: 3d

shape understanding

open-world understanding

zero-shot 3D classification

vision-language model
Abstract: We introduce OpenShape, a method for learning multi-modal joint representations of text, image, and point clouds. We adopt the commonly used multi-modal contrastive learning framework for representation alignment, but with a specific focus on scaling up 3D representations to enable open-world 3D shape understanding. To achieve this, we scale up training data by ensembling multiple 3D datasets and propose several strategies to automatically filter and enrich noisy text descriptions. We also explore and compare strategies for scaling 3D backbone networks and introduce a novel hard negative mining module for more efficient training. We evaluate OpenShape on zero-shot 3D classification benchmarks and demonstrate its superior capabilities for open-world recognition. Specifically, OpenShape achieves a zero-shot accuracy of 46.8% on the 1,156-category Objaverse-LVIS benchmark, compared to less than 10% for existing methods. OpenShape also achieves an accuracy of 85.3% on ModelNet40, outperforming previous zero-shot baseline methods by 20% and performing on par with some fully-supervised methods. Furthermore, we show that our learned embeddings encode a wide range of visual and semantic concepts (e.g., subcategories, color, shape, style) and facilitate fine-grained text-3D and image-3D interactions. Due to their alignment with CLIP embeddings, our learned shape representations can also be integrated with off-the-shelf CLIP-based models for various applications, such as point cloud captioning and point cloud-conditioned image generation.

OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding

Abstract

We introduce OpenShape, a method for learning multi-modal joint representations of text, image, and point clouds. We adopt the commonly used multi-modal contrastive learning framework for representation alignment, but with a specific focus on scaling up 3D representations to enable open-world 3D shape understanding. To achieve this, we scale up training data by ensembling multiple 3D datasets and propose several strategies to automatically filter and enrich noisy text descriptions. We also explore and compare strategies for scaling 3D backbone networks and introduce a novel hard negative mining module for more efficient training. We evaluate OpenShape on zero-shot 3D classification benchmarks and demonstrate its superior capabilities for open-world recognition. Specifically, OpenShape achieves a zero-shot accuracy of 46 . 8% on the 1,156-category Objaverse-LVIS benchmark, compared to less than 10% for existing methods. OpenShape also achieves an accuracy of 85 . 3% on ModelNet40, outperforming previous zero-shot baseline methods by 20% and performing on par with some fully-supervised methods. Furthermore, we show that our learned embeddings encode a wide range of visual and semantic concepts (e.g., subcategories, color, shape, style) and facilitate fine-grained text3D and image-3D interactions. Due to their alignment with CLIP embeddings, our learned shape representations can also be integrated with off-the-shelf CLIPbased models for various applications, such as point cloud captioning and point cloud-conditioned image generation.

1 Introduction

3D shape understanding has recently garnered a surge of interest driven by the growing demands in real-world applications, such as augmented/virtual reality, autonomous driving, and robotics. Despite significant advancements in 3D recognition and analysis, existing data-driven approaches are still greatly limited by the scale of 3D training datasets and tend to exhibit poor generalization when facing unseen shape categories, hindering the deployment of existing models in real-world applications.

Note that 3D shapes and 2D images can be easily linked through rendering, and the dataset scale issue of 2D images has been remarkably addressed, as shown in recent works such as CLIP [55]. Therefore, many recent studies aim to utilize pre-trained 2D image-language models [55, 59] to assist 3D tasks, such as 3D generation [22, 26, 45, 63, 33, 7, 38, 67] and 3D scene-level segmentation [18, 27, 14, 79,

+

+

+

+

Top1

O-LVIS Acc. (%)

60

40

20

0

PointCLIP

PointCLIPv

ReCon

CG3D

CLIP2Point

ULIP (official)

ULIP (retrained)

OpenShape

100

ModelNet40 Acc. (%)

8

Top1

Top5

4

20

1

ReCon

CG3D

PointCLIP

PointCLIP v2

CLIP2Point

ULIP(official)

OpenShape

ULIP (retrained)

41, 49]. Regarding 3D shape-level understanding, a straightforward idea is to project 3D data to the 2D domain through rendering and use CLIP to analyze the 2D images, thereby enabling zero-shot 3D shape classification [86, 88]. However, these methods suffer from occlusion and information loss during projection, and unnecessary latency due to point cloud rendering and multiple CLIP inferences.

To overcome the limitations caused by projection, it is necessary to train a 3D-native model by distilling knowledge from pretrained 2D models. However, training a 3D-native model requires a set of 3D shapes, and the amount of knowledge that can be distilled is determined by the size of the 3D dataset. For example, ULIP [78] aims to learn a joint representation space between language, 2D images, and 3D shapes, but uses a small-scale 3D dataset ShapeNetCore [8] for knowledge distillation. Specifically, ULIP fixes the 2D CLIP text and image encoders and trains a dedicated 3D-native point cloud encoder to extract 3D shape representations. The 3D encoder strives to align the 3D shape embedding space with the CLIP image and language embedding spaces by utilizing contrastive learning across all three modalities. However, since ULIP is only trained on 52K shapes of 55 object categories, it still struggles with out-of-distribution shape categories and fails to demonstrate an impressive open-world understanding of 3D shapes.

In this work, we propose a novel method called OpenShape, which follows a similar paradigm as ULIP but aims to achieve a more generalized and scalable joint representation space encompassing language, 2D images, and 3D shapes. Our focus mainly lies on scaling up representation learning and addressing corresponding challenges. In OpenShape, we emphasize four key factors during the training process: (a) data scale: we significantly increase the scale of 3D training data by combining four public 3D shape datasets, resulting in 876k 3D shapes covering much more diverse categories; (b) text quality: the 3D shapes from our main dataset, Objaverse [12], is dominated with inaccurate or uninformative text descriptions. Given the data scale, we propose three strategies to automatically filter and enrich the text descriptions; (c) 3D backbone scaling: since most existing 3D backbones target small datasets, we find that it's important but non-trivial to scale up the 3D backbones; and (d) data resampling: since the ensembled dataset is highly unbalanced, we utilize hard negative mining to improve the model's discriminative ability.

We first evaluate OpenShape on the zero-shot 3D shape classification task. As shown in Figure 1, OpenShape outperforms previous zero-shot approaches on the ModelNet40 dataset by at least 20%. Moreover, OpenShape excels at handling long-tail categories. On the challenging Objaverse-LVIS dataset, which contains 1,156 categories, OpenShape achieves a 46.8% accuracy, significantly surpassing previous methods. Notably, this performance gap remains even when ULIP is retrained on our ensembled datasets, highlighting the superiority of our text enrichment and training strategies.

Besides zero-shot classification, we present demos that showcase the wide range of visual and semantic concepts learned by OpenShape. For example, in Figure 1-right, we take two 3D shapes as input and use their OpenShape embeddings to retrieve the top three shapes whose embeddings are simultaneously closest to both inputs from our ensembled dataset. The retrieved shapes exhibit an interesting combination of the semantic and geometric elements from both input shapes. Furthermore, since we align our 3D shape embedding space with the CLIP language and image embedding space, we demonstrate that OpenShape embeddings can be easily integrated with other CLIP-based models to perform cross-modality tasks such as point cloud captioning and point cloud-conditioned image generation.

2 Related Work

2.1 CLIP for 3D Learning

Image-language models like CLIP have achieved remarkable performance through large-scale imagetext pretraining [55, 29, 35, 84, 4, 56, 61, 36]. As these models excel at capturing rich visual concepts and possess impressive zero-shot capabilities, they have been applied to various 3D vision tasks. For instance, numerous recent works utilize CLIP to facilitate zero-shot text-to-3D generation [22, 26, 45, 63, 33, 7, 32, 5, 28, 77, 40, 83], typically through CLIP-guided per-scene optimization. From a recognition perspective, some works focus on scene-level representation, aiming to leverage CLIP priors for zero-shot 3D segmentation or detection in both indoor [18, 27, 14, 79, 41, 49, 82, 23, 60, 85, 31] and outdoor scenes [9, 21]. Meanwhile, another line of work focuses on shape-level understanding, targeting zero-shot shape classification [86, 88, 53, 78, 19] and part segmentation [39, 1]. There are two primary working paradigms for these methods. The first [86, 88, 24] involves using images as a medium representation, projecting 3D point clouds into 2D and employing 2D CLIP for inference. However, these methods typically suffer from occlusion and information loss during projection, along with unnecessary latency due to point cloud rendering and multiple 2D CLIP inferences. The second paradigm involves training a 3D-native encoder attempting to distill or fuse CLIP features into 3D representations. Our paper follows this paradigm.

2.2 3D Shape Representation Learning

Various works have studied self-supervised pretraining for point clouds by designing pretext tasks [15, 69, 50, 2, 66] such as self-reconstruction [57, 13, 3, 72], masked auto-encoding [48, 80, 20], distortion reconstruction [64, 44, 68], normal estimation [57], and contrastive learning [87, 62, 76]. These tasks enhance models' shape representations and improve their performance on downstream applications, although they do not involve multimodal semantic alignments during pretraining.

Recently, some works [53, 78, 19], exemplified by ULIP [78], have explored learning multimodal joint representations for 3D shapes. They train 3D-native shape encoders by aligning 3D shape embeddings with CLIP's language and/or image embeddings through multimodal contrastive learning. Works like ReCon [53] further combines cross-modal contrastive learning with masked auto-encoding for added enhancement. While these methods allow for zero-shot 3D classification through the computation of 3D-text similarity, the amount of distilled knowledge and their model capability are heavily limited by the small-scale training datasets used. Our work follows this paradigm but aims to learn more generalizable and scalable representations to enable open-world 3D shape understanding.

3 Method

We propose a novel method, OpenShape , for learning generalizable and scalable multi-modal joint representation between language, 2D images, and 3D shapes, as shown in Figure 2. We first introduce the multi-modal contrastive learning framework we used for aligning representations of three modalities in Section 3.1. We then elaborate how we create our training sets and enrich our text data in Sections 3.2 and 3.3. In Section 3.4, we present how we scale up our 3D backbone models. Finally, we propose a hard negative mining strategy to enhance contrastive learning in Section 3.5.

catalog images

high-resolution geometry & texture

physical y-based renderings

ShapeNet

(52.5k)

3D-FUTURE

(16.6k)

ABO

(8.0k)

(a) Ensemble Datasets

(b) Text Filtering & Enrichment

Zero-Shot

Classification

Text-to-3D

(Retrieval)

3D-to-Text

(Captioning)

Image-to-3D

(Retrieval)

3D-to-Image

(Generation)

(c) Cross-Modal Alignment

(d) Cross-Modal Applications

Objaverse

(798.8k)

original

texts

filtered

texts

GPT4

2D

renderings

Image

Caption

Image

Retrieval

captions

retrieved

texts

Enriched Texts

XXL

Hard Negative

Mining

Text

Encoder

Image

Encoder

PointCloud

Encoder

3.1 Multi-Modal Representation Alignment

We aim to learn 3D shape representations that are aligned with pretrained CLIP embedding spaces of language and image. As shown in Figure 2 (c), we train a 3D native encoder f P that takes a 3D point cloud as input and extracts 3D shape feature. Following previous works [53, 78, 19], such as ULIP [78], we utilize multi-modal contrastive learning for representation alignment. Since CLIP is pretrained on a much larger scale data, we freeze both its text encoder f T and its image encoder f I during feature alignment to preserve CLIP's feature priors and avoid model collapse. Specifically, given a sampled batch of triplets { ( P i , T i , I i ) } , where P i denotes a point cloud of a 3D shape, T i and I i denote corresponding text and image, the contrastive loss is calculated as:

(1)

where n is the number of shapes in a batch; τ is a learnable temperature; h P i = f P ( P i ) / | f P ( P i ) | , h T i = g T ( f T ( T i )) / | g T ( f T ( T i )) | , and h I i = g I ( f I ( I i )) / | g I ( f I ( I i )) | denote normalized projected features of P i , T i , and I i , where g T and g I are two learnable linear projections. Since f T and f I are frozen, we extract all f T ( T i ) and f I ( I i ) before training and cache them for acceleration. In most of our experiments, we utilize OpenCLIP ViT-bigG-14 [25] as the pretrained CLIP model.

3.2 Ensembling 3D Datasets

Since the scale and diversity of training triplets play a crucial role in learning scalable shape representations, we ensemble four currently-largest public 3D datasets for training as shown in Figure 2 (a), resulting in 876k training shapes. Among these four datasets, ShapeNetCore [8], 3D-FUTURE [16] and ABO [11] are three popular datasets used by prior works. They contain human-verified high-quality 3D shapes, but only cover a limited number of shapes and dozens of categories. The Objaverse [12] dataset is a more recent dataset, containing many more 3D shapes and covering significantly more diverse categories. However, shapes in Objaverse are mainly uploaded by web users and not verified by experts, and thus have uneven quality and exhibit highly unbalanced distributions, necessitating further processing.

To create triplets for training, for each shape, we sample 10,000 points from the mesh surface and interpolate the point colors according to the mesh textures. We also render 12 color images from the preset camera poses that uniformly cover the whole shape. For datasets providing thumbnails, we include them as part of image candidates, since they typically capture the shape from a better camera view. For the Objaverse dataset, we use the model name as the raw text for each shape. For other datasets, we utilize provided metadata to create raw texts (see supplementary for details). During each pretraining iteration, we randomly sample one rendered image or thumbnail for each shape, and apply standard augmentation to the point clouds [78].

3.3 Text Filtering and Enrichment

We find that only applying contrastive learning between 3D shapes and 2D images is insufficient to fuel zero-shot 3D classification, even when training on large-scale datasets. We conjecture that this is

name: 'homework xyz detailing'

GPT4: Remove

'

Steampunk Goggles

by MonoFlow on …'

'

Steampunk Goggles

Made from hand ...'

azure: 'a pair of

steampunk goggles

'

blip: '

steampunk goggles

3d model'

name: 'Tue, 09 Oct 2018 17:12:39'

GPT4: Remove

'

armchair

'

'some of the other props done Chair10'

azure: 'a

blue plastic chair

on a pink ...'

blip: 'a 3d model of a blue shaped object'

name: 'DOG A - 1of6 - for free …'

GPT4: Keep

'Black

Labrador

in front of a white …'

'Black

Labrador

puppy Vinyl Wall Mural'

azure: 'a black dog sitting on a blue...'

blip: 'a black dog sitting on a blue...'

name: 'untitled'

GPT4: Remove

'

Nike AirMax 1 (Red/White)

'

azure: 'a close up of a shoe'

blip: '

nike air max 1 - white / red

'

'

nike air max red and white

'

T

1

caused by the inherent domain gap in CLIP's language and image embedding spaces, which is also observed by previous studies [37, 70]. Consequently, 3D-text alignment is not guaranteed even if we obtain good 3D-image alignments via contrastive learning. Therefore, we need to explicitly align 3D shapes with text. Along this process, to facilitate better 3D-text alignment, we introduce 3 techniques to improve the text quality: filtering, captioning, and image retrieval, as shown in Figure 2 (b).

Filtering. As shown in Figure 3, the 3D shapes from our main dataset, Objaverse, is dominated with noisy text descriptions ('names') uploaded by web users. Many of the problematic texts can be identified from the text itself without seeing the corresponding 3D shape. We thus leverage a powerful large language model, GPT-4 [47], to filter out inaccurate or uninformative text descriptions. We find that GPT-4 excels at recognizing irrelevant contents, such as timestamps, pure model numbers, incomprehensible descriptions, random filenames (e.g., new project), and random characters. Through GPT-4, we filter out about 30% of raw user texts. Note that we only filter the texts, and still keep all shapes for training. More details, such as the prompts we used, are presented in the supplementary.

Captioning. We utilize BLIP [34] and the Azure cognition services to caption the 2D thumbnails (if present, or images rendered from a fixed frontal view) of the 3D models, obtaining two texts for each shape. As shown in Figure 3, the captioning models can usually produce meaningful and descriptive captions that either enhance user-uploaded texts or replace low-quality ones. We also notice that the two caption models complement each other, leading to better performance.

Image Retrieval. In addition to image captioning, we also perform image retrieval to obtain additional descriptions of 3D models. We retrieve k-NN images of shape renderings from the LAION-5B dataset [65] using the CLIP ViT-L retrieval index [6]. We then take the captions of the k-NN images as the retrieved texts for our 3D models. Compared with captioning model generations, retrieved texts cover a wider range of text styles. They can also include more fine-grained semantics than both the user texts and the generated captions (e.g., 'Labrador' in Figure 3).

In each iteration of pretraining, for each shape, we first randomly sample a text source category among the raw text (if unfiltered), the captions, and the retrieved texts. We then select a text candidate from the selected category. We also apply the template-based prompt engineering technique used in ULIP [78] to both training texts and test-time category names. Specifically, we extend a word or a phrase to a collection of templated simple sentences and take their average embedding.

3.4 Scaling Up 3D Point Cloud Backbones

Previous works on 3D point cloud learning have primarily focused on smaller-scale datasets like ShapeNet. These techniques may not be directly applicable to our larger-scale ensembled dataset and need to be scaled up accordingly. We find that different 3D backbones may exhibit distinct behavior and scalability when trained on datasets with varying sizes. Specifically, we compare six popular





























backbones trained on ShapeNet or our ensembled dataset by evaluating their zero-shot classification performance on ModelNet40 [75] and Objaverse-LVIS datasets (for now, these backbones are trained with their original configurations and without scaling up model sizes). Objaverse-LVIS is a subset of Objaverse dataset with human-verified category labels. With 1,156 categories, it serves as a suitable dataset for evaluating zero-shot long-tail classification, and we exclude all shapes of ObjaverseLVIS from this experiment. Results are shown in Table 1. We find that when trained on ShapeNet, all backbones share similar performances. However, when trained on our ensembled dataset, the performance gap between backbones increases significantly. This suggests that while the original versions of these backbones share a similar number of parameters, some may have been saturated when trained on small datasets, while others do not.

We also explore the performance and scalability of these backbones when scaling up the model sizes and training on our ensembled dataset. Please refer to the supplementary for details on how we scale up each model. As shown in Figure 4, we observe that all 3D backbones benefit significantly from model scaling. However, traditional backbones without a shrinking hierarchical structure, such as DGCNN and PointNet, require operating completely on dense points or modeling the relationships (e.g., through kNN) between dense points. As a result, they become more time-consuming and memory-intensive when scaled up compared to more modern backbones. We therefore select PointBERT [81] (Transformer-based) and SparseConv [10] (convolution-based) as our 3D backbones for the remaining experiments, as they exhibit strong performance and scalability.

3.5 Hard Negative Mining

Our ensembled dataset exhibits a high degree of class imbalance. Certain common categories, such as building, may occupy tens of thousands of shapes, while many other categories, such as walrus and wallet, are underrepresented with only a few dozen or even fewer shapes. Consequently, when randomly constructing batches, it is unlikely that shapes from two confusing categories (e.g., apples and cherries) will be contrasted within the same batch. Inspired by some previous works [58, 30], we propose an offline hard negative mining strategy for improving the training efficiency and performance. Specifically, in the first round of training, we train our model with random batches until it is about to converge. We then compute the kNN for each shape in the learned 3D embedding space. In the second round of training, for each iteration, we randomly select s seed shapes and then obtain m neighbors from the kNN results of each seed shape, resulting s × m shapes per batch. In this way, confusing pairs are more likely to be selected in a single batch. However, this may also introduce false negative pairs (e.g., two apples) into contrastive learning. To mitigate this issue, we leverage image and text embeddings to filter out pairs sharing similar texts when calculating the contrastive loss. Specifically, for two shapes i and j selected from the same seed shape, if h T j · h I i + δ > h T i · h I i , where h T and h I are text and image embeddings, and δ is a small threshold, we believe that the text embeddings of i and j are very close to each other, and we remove j from i 's negative examples when calculating contrastive loss. By employing this strategy to construct batches, we observe faster and better model learning.

4 Experiments

4.1 Zero-Shot Shape Classification

We evaluate the zero-shot classification performances of our models on three benchmarks: the traditional ModelNet40 [75] and ScanObjectNN [71], as well as a new benchmark, ObjaverseLVIS [12]. ModelNet40 and ScanObjacetNN consist of 40 and 15 common categories, respectively. Objaverse-LVIS is an annotated subset of Objaverse [12] and comprises 46,832 shapes among 1,156 LVIS [17] categories. With a much larger base of classes than other benchmarks, Objaverse-LVIS presents a challenging long-tailed distribution, making it a better reflection on models' performance in open-world scenarios. We compare OpenShape with existing zero-shot approaches, including PointCLIP [86], PointCLIPv2 [88], ReCon [53], CG3D [19], CLIP2Point [24], and ULIP [78]. Among them, PointCLIP [86] and PointCLIPv2 [88] project point clouds into 2D images and directly utilize 2D CLIP for inference, while other methods leverage the CLIP embedding spaces for alignment and require 3D shapes for training. We report results on these baselines using their released checkpoints. To better analyze the source of our performance gains, we also retrain the baseline ULIP [78] on our ensembled shape dataset, but we use the original texts in the four constituent datasets along with the official codebase without backbone scaling. We train OpenShape and ULIP on three different sets of training shapes: ' Ensembled ' denotes using all shapes from the four datasets; ' Ensembled (no LVIS) ' is the same but excludes all shapes from the Objavserse-LVIS subset; ' ShapeNet ' only includes shapes from the ShapeNet [8] dataset. Note that even when LVIS shapes are included in the training shapes (i.e., the 'Ensembled' dataset), their test-time category labels are probably not included in their training texts. Please refer to the supplementary for more training and evaluation details.

Table 2 shows the results. We observe that OpenShape consistently outperforms prior approaches, even when trained only on ShapeNet. When models are trained on our larger-scale ensembled dataset, they receive a significant performance boost. In this case, OpenShape still surpasses retrained ULIP by a significant margin, demonstrating the advantages of our text enrichment, backbone scaling, and other training strategies. Specifically, OpenShape greatly improves the classification accuracy on the long tail categories in Objaverse-LVIS from a dull < 10% to 46 . 8 % , outperforming the retrained ULIP by about 20 points and reaching a decent top5 accuracy of 77 . 0 % . These results demonstrate OpenShape's capability to recognize open-world objects effectively. As for ModelNet40, OpenShape achieves a 85 . 3 % accuracy, surpassing previous methods by a substantial margin of at least 20 percent. OpenShape also achieves impressive top-3 and top-5 accuracies of 96 . 5 % and 98 . 0 % . To the best of our knowledge, this is the first time zero-shot methods have matched the performance of a fullysupervised 3D learning method on ModelNet40, where OpenShape outperforms fully-supervised 3D ShapeNets [75] and VoxNet [43]. In addition, on ScanObjectNN, which contains challenging real scans with noise and occlusion, OpenShape exhibits decent sim-to-real transfer capabilities. To contextualize, OpenShape-SparseConv achieves 56.7 % zero-shot accuracy on ScanObjectNN without specific sim-to-real training, which surpasses 52 . 7% reported by SKPConv [74], a recent method specially designed for sim-to-real transfer in point cloud classification tasks.

Objaverse-LVIS

ModelNet40

ScanObjectNN

10

20

30

40

50

50

60

70

80

90

30

40

50

60

70

80

PointCLIP V2

ULIP-Official

ULIP-Retrained

OpenShape-PointBERT

0 1 2

4

8

16

0 1 2

4

8

16

0 1 2

4

8

16

# of labeled training samples per class

Top-1 Acc. (%)

4.2 Few-Shot Linear Probing

In the literature, linear probing is a common way to assess the representation learning capabilities of a model. To perform linear probing, we gather and freeze the representation vectors from all samples in a dataset. Subsequently, we train a linear classifier using these fixed vectors and few-shot class labels. We evaluate the accuracy of the linear classifier on three benchmarks: Objaverse-LVIS [12], ModelNet40 [75], and ScanObjectNN [71]. Figure 5 summarizes the performance of OpenShape in comparison with ULIP [78] (official release and our retrained versions) and PointCLIPv2 [88]. On the most challenging Objaverse-LVIS benchmark, OpenShape outperforms all other methods by a large margin. Notably, zero-shot OpenShape beats few-shot linear probes of other methods. On ModelNet40 and ScanObjectNN, we do not see a large performance margin between OpenShape and retrained ULIP. We hypothesize that for few-shot ModelNet40, the error is dominated by in-category sample bias rather than the representation quality; while for ScanObjectNN, the domain gap plays a major role. Since both OpenShape and retrained ULIP are exposed to the same source domain of training objects, their few-shot out-of-domain generalization performances tend to be similar.

4.3 Ablation Study

We perform various ablations by training a scaled version of SparseConv [10] on the ensembled dataset and then evaluate it on the Objaverse-LVIS [12] and ModelNet40 [75] zero-shot classification benchmarks, unless otherwise specified. The results are shown in Table 3 and Figures 6 and 7.

Data and Model Scaling. We investigate the impact of training data by ablating (1) without or with only Objaverse shapes (Tab. 3) and (2) with different ratios of our ensembled dataset (Fig. 6). We observe that training with 1% of our ensembled dataset (about 8 . 8 k shapes) achieves similar or better zero-shot performance than training without Objaverse shapes (about 77 . 1 k shapes), indicating that the diversity of training data is sometimes more crucial than the scale. In addition, we compare the performances between scaled-up and non-scaled-up backbones. From Tab. 3, we demonstrate that model scaling plays an essential role when training on our large-scale ensembled dataset (also Fig. 4).





































































icalmug"

bicmug"

J

ricalmug"

Lweebly.com

GET

nug"

C

mug"

apedmug"

mug"

Text Filtering and Enrichment. As shown in Tab. 3, both text filtering and text enrichment are beneficial for performance. We also investigate the specific text enrichment strategies to use for the SparseConv and PointBERT backbones. In Fig. 7, we observe that both image captioning and text retrieval are helpful, and including both yield the best results. Notably, PointBERT improves more than 10 points from text enrichment, highlighting the significance of enhancing text quality.

Other Aspects. We also conduct additional ablation studies on color information, contrastive loss components, and our hard-negative mining strategy in Tab. 3. We observe that OpenShape performs well with only xyz coordinates as input and no RGB color. While 3D-image contrastive loss is also helpful, we observe that 3D shape-text alignment plays a very essential role for model zero-shot generalization, which necessitates our text filtering and text enrichment strategies that significantly enhance text quality. Lastly, by employing our hard negative mining strategy, OpenShape effectively addresses the issue of unbalanced data distribution, leading to further improvements in performance.

4.4 Cross-Modal Applications

Multi-modal 3D Shape Retrieval. Through OpenShape multi-modal representations, we can index and retrieve 3D shapes from images, texts, or point clouds. In this section, we retrieve 3D shapes from our ensembled dataset by calculating the cosine similarity between input embedding(s) and 3D shape embeddings and performing kNN. As shown in Figure 8, OpenShape is capable of retrieving visually or semantically similar shapes from a single image or point cloud input. OpenShape embeddings encode a wide range of visual and semantic concepts. In Figure 9, we show that OpenShape supports retrieving 3D shapes from detailed text descriptions, which include fine-grained subcategories, attributes, and their combinations. Note that these input texts are typically not present in the raw texts of the retrieved shapes, indicating that OpenShape effectively learns generalizable concepts across shapes. In Figure 1, we provide a demo which takes two 3D shapes as inputs and retrieves the shapes that are simultaneously closest to both inputs. This is achieved by finding arg max i min( h P i · h P a , h P i · h P b ) , where h P a and h P b denote normalized shape embeddings of the two input shapes. We can see that the retrieved shapes integrate visual or semantic elements in an interesting manner, highlighting the rich concepts and priors encoded in OpenShape embeddings.

+

'in a large desert'

+

'in the woods'

1.

2.

3.

4.

1. The chair in the style of the 1920s.

2. The ladder to the second floor.

3. Goldfish in the sea  - photo #.

4. The car of the day.

(a)

(b)

Shape-Conditioned Multimodal Generation. As OpenShape's 3D shape representations are aligned with CLIP's image and text embedding spaces, they can serve as inputs into other CLIP-based models to facilitate various multimodal generation applications. For example, we show that by feeding our 3D shape embeddings into ClipCap [46], an off-the-shelf image captioning model, along with Stable unCLIP [56], a text-to-image diffusion model, we can perform point cloud captioning and point cloud-conditioned image generation (optional text prompt supported) without extra training or finetuning. Qualitative results are shown in Figure 10. Please refer to the supplementary for more results and details.

5 Limitation and Conclusion

We introduce OpenShape, a novel approach for learning scalable and generalizable multi-modal joint representations for 3D shapes. OpenShape representations effectively capture a wide range of semantic and visual concepts, enabling superior capabilities for open-world 3D shape recognition. By aligning OpenShape with CLIP's embedding space, our shape embeddings can be integrated with off-the-shelf CLIP-based models for various cross-modality applications. Moving forward, there are several directions worth further exploration: (a) More 3D data. While we utilized 876k 3D shapes during training, this is still quite limited compared to the 2D counterparts. We hope that our work inspires future investments in more resources to build even more powerful 3D representations. (b) Part-level information. Our current shape representations mainly focus on global semantic and visual features, and it would be beneficial to add more part-level supervision during training. (c) Sim-to-real domain gap. Our model is mainly trained on synthetic data, and it's challenging but crucial to explore explicit designs for reducing the domain gap with real-world shapes.

Acknowledgments

This work is supported in part by gifts from Qualcomm.

References

Ahmed Abdelreheem, Ivan Skorokhodov, Maks Ovsjanikov, and Peter Wonka. Satr: Zero-shot semantic segmentation of 3d shapes. arXiv preprint arXiv:2304.04909 , 2023.

Idan Achituve, Haggai Maron, and Gal Chechik. Self-supervised learning for domain adaptation on point clouds. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 123-133, 2021.

Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In International conference on machine learning , pages 40-49. PMLR, 2018.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198 , 2022.

Shivangi Aneja, Justus Thies, Angela Dai, and Matthias Nießner. Clipface: Text-guided editing of textured 3d morphable models. arXiv preprint arXiv:2212.01406 , 2022.

Romain Beaumont. Clip retrieval: Easily compute clip embeddings and build a clip retrieval system with them. https://github.com/rom1504/clip-retrieval , 2022.

Zehranaz Canfes, M Furkan Atasoy, Alara Dirik, and Pinar Yanardag. Text and image guided 3d avatar generation and manipulation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 4421-4431, 2023.

Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 , 2015.

Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. arXiv preprint arXiv:2301.04926 , 2023.

Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3075-3084, 2019.

Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, et al. Abo: Dataset and benchmarks for real-world 3d object understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21126-21136, 2022.

Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. arXiv preprint arXiv:2212.08051 , 2022.

Haowen Deng, Tolga Birdal, and Slobodan Ilic. Ppf-foldnet: Unsupervised learning of rotation invariant 3d local descriptors. In Proceedings of the European conference on computer vision (ECCV) , pages 602-618, 2018.

Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Languagedriven open-vocabulary 3d scene understanding. arXiv preprint arXiv:2211.16312 , 2022.

Benjamin Eckart, Wentao Yuan, Chao Liu, and Jan Kautz. Self-supervised learning on 3d point clouds by learning discrete generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8248-8257, 2021.

Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao. 3d-future: 3d furniture shape with texture. International Journal of Computer Vision , 129:3313-3337, 2021.

Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5356-5364, 2019.

Huy Ha and Shuran Song. Semantic abstraction: Open-world 3d scene understanding from 2d vision-language models. In Conference on Robot Learning , 2022.

Deepti Hegde, Jeya Maria Jose Valanarasu, and Vishal M Patel. Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition. arXiv preprint arXiv:2303.11313 , 2023.

Georg Hess, Johan Jaxing, Elias Svensson, David Hagerman, Christoffer Petersson, and Lennart Svensson. Masked autoencoder for self-supervised pre-training on lidar point clouds. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 350-359, 2023.

Georg Hess, Adam Tonderski, Christoffer Petersson, Lennart Svensson, and Kalle Åström. Lidarclip or: How i learned to talk to point clouds. arXiv preprint arXiv:2212.06858 , 2022.

Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu. Avatarclip: Zero-shot text-driven generation and animation of 3d avatars. arXiv preprint arXiv:2205.08535 , 2022.

Rui Huang, Xuran Pan, Henry Zheng, Haojun Jiang, Zhifeng Xie, Shiji Song, and Gao Huang. Joint representation learning for text and 3d point cloud. arXiv preprint arXiv:2301.07584 , 2023.

Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, Rynson WH Lau, Wanli Ouyang, and Wangmeng Zuo. Clip2point: Transfer clip to point cloud classification with image-depth pre-training. arXiv preprint arXiv:2210.01055 , 2022.

Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. If you use this software, please cite it as below.

Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 867-876, 2022.

Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, et al. Conceptfusion: Open-set multimodal 3d mapping. arXiv preprint arXiv:2302.07241 , 2023.

Nikolay Jetchev. Clipmatrix: Text-controlled creation of 3d textured meshes. arXiv preprint arXiv:2109.12922 , 2021.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, YunHsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning , pages 4904-4916. PMLR, 2021.

Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances in Neural Information Processing Systems , 33:21798-21809, 2020.

Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. arXiv preprint arXiv:2303.09553 , 2023.

Nasir Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. Text to mesh without 3d supervision using limit subdivision. arXiv preprint arXiv:2203.13333 , 2022.

Han-Hung Lee and Angel X Chang. Understanding pure clip guidance for voxel grid nerf models. arXiv preprint arXiv:2209.15172 , 2022.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping languageimage pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning , pages 12888-12900. PMLR, 2022.

Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10965-10975, 2022.

Xuanlin Li, Yunhao Fang, Minghua Liu, Zhan Ling, Zhuowen Tu, and Hao Su. Distilling large vision-language model with out-of-distribution generalizability. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2492-2503, 2023.

Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems , 35:17612-17625, 2022.

Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Zexiang Xu, Hao Su, et al. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. arXiv preprint arXiv:2306.16928 , 2023.

Minghua Liu, Yinhao Zhu, Hong Cai, Shizhong Han, Zhan Ling, Fatih Porikli, and Hao Su. Partslip: Low-shot part segmentation for 3d point clouds via pretrained image-language models. arXiv preprint arXiv:2212.01558 , 2022.

Zhengzhe Liu, Peng Dai, Ruihui Li, Xiaojuan Qi, and Chi-Wing Fu. Iss: Image as stetting stone for text-guided 3d shape generation. arXiv preprint arXiv:2209.04145 , 2022.

Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. Open-vocabulary point-cloud object detection without 3d annotation. arXiv preprint arXiv:2304.00788 , 2023.

Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local geometry in point cloud: A simple residual mlp framework. arXiv preprint arXiv:2202.07123 , 2022.

Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for realtime object recognition. In 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS) , pages 922-928. IEEE, 2015.

Benedikt Mersch, Xieyuanli Chen, Jens Behley, and Cyrill Stachniss. Self-supervised point cloud prediction using 3d spatio-temporal convolutional networks. In Conference on Robot Learning , pages 1444-1454. PMLR, 2022.

Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Textdriven neural stylization for meshes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13492-13502, 2022.

Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734 , 2021.

OpenAI. Gpt-4 technical report, 2023.

Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part II , pages 604-621. Springer, 2022.

Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. arXiv preprint arXiv:2211.15654 , 2022.

Omid Poursaeed, Tianxing Jiang, Han Qiao, Nayun Xu, and Vladimir G Kim. Self-supervised learning of point clouds via orientation estimation. In 2020 International Conference on 3D Vision (3DV) , pages 1018-1028. IEEE, 2020.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 652-660, 2017.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems , 30, 2017.

Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining. arXiv preprint arXiv:2302.02318 , 2023.

Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. arXiv:2206.04670 , 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Yongming Rao, Jiwen Lu, and Jie Zhou. Global-local bidirectional reasoning for unsupervised representation learning of 3d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5376-5385, 2020.

Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592 , 2020.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

David Rozenberszki, Or Litany, and Angela Dai. Language-grounded indoor 3d semantic segmentation in the wild. arXiv preprint arXiv:2204.07761 , 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.

Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487 , 2022.

Aditya Sanghi. Info3d: Representation learning on 3d objects using mutual information maximization and contrastive learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX 16 , pages 626-642. Springer, 2020.

Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Yi Cheng, Marco Fumero, and Kamal Rahimi Malekshan. Clip-forge: Towards zero-shot text-to-shape generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18603-18613, 2022.

Jonathan Sauder and Bjarne Sievers. Self-supervised deep learning on point clouds by reconstructing space. Advances in Neural Information Processing Systems , 32, 2019.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 , 2022.

Charu Sharma and Manohar Kaul. Self-supervised few-shot learning on point clouds. Advances in Neural Information Processing Systems , 33:7212-7221, 2020.

Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++: a single image to consistent multi-view diffusion base model, 2023.

Chao Sun, Zhedong Zheng, Xiaohan Wang, Mingliang Xu, and Yi Yang. Point cloud pre-training by mixing and disentangling. arXiv e-prints , pages arXiv-2109, 2021.

Ali Thabet, Humam Alwassel, and Bernard Ghanem. Self-supervised learning of local features in 3d point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , pages 938-939, 2020.

Vishaal Udandarao. Understanding and fixing the modality gap in vision-language models. Master's thesis, University of Cambridge, 2022.

Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1588-1597, 2019.

Hanchen Wang, Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt J Kusner. Unsupervised point cloud pre-training via occlusion completion. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9782-9792, 2021.

Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog) , 38(5):1-12, 2019.

Jean-Baptiste Weibel, Timothy Patten, and Markus Vincze. Sim2real 3d object classification using spherical kernel point convolution and a deep center voting scheme. arXiv preprint arXiv:2103.06134 , 2021.

Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1912-1920, 2015.

Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16 , pages 574-591. Springer, 2020.

Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao. Dream3d: Zero-shot text-to-3d synthesis using 3d shape prior and text-to-image diffusion models. arXiv preprint arXiv:2212.14704 , 2022.

Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning unified representation of language, image and point cloud for 3d understanding. arXiv preprint arXiv:2212.05171 , 2022.

Jihan Yang, Runyu Ding, Zhe Wang, and Xiaojuan Qi. Regionplc: Regional point-language contrastive learning for open-world 3d scene understanding. arXiv preprint arXiv:2304.00962 , 2023.

Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19313-19322, 2022.

Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit-Yan Yeung, Zhen Yang, Xiaodan Liang, and Hang Xu. Clipˆ 2: Contrastive language-image-point pretraining from real-world point cloud data. arXiv preprint arXiv:2303.12417 , 2023.

Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit-Yan Yeung, Zhen Yang, Xiaodan Liang, and Hang Xu. Clip2: Contrastive language-image-point pretraining from real-world point cloud data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15244-15253, 2023.

Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language understanding. arXiv preprint arXiv:2206.05836 , 2022.

Junbo Zhang, Runpei Dong, and Kaisheng Ma. Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip. arXiv preprint arXiv:2303.04748 , 2023.

Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8552-8562, 2022.

Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of 3d features on any point-cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10252-10263, 2021.

Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyao Zeng, Shanghang Zhang, and Peng Gao. Pointclip v2: Adapting clip for powerful 3d open-world learning. arXiv preprint arXiv:2211.11682 , 2022.

A More Examples of Multi-Modal 3D Shape Retrieval

In Figures 11 and 12, we showcase more examples of multi-modal 3D shape retrieval.

B More Examples of Shape-Conditioned Multimodal Generation

In Figure 13 and Figure 14, we showcase more examples of point cloud captioning and point cloud-conditioned image generation.

C Details on Raw Text Generation and Filtering

C.1 Raw Text Generation

We leverage the metadata from the four datasets to generate the raw texts. Although the original datasets may contain numerous attributes for each shape, we carefully choose the most informative ones to compose the text, ensuring its quality and relevance.

Objaverse :We utilize the name associated with each shape to serve as the text.

ShapeNetCore : For each shape, we generate three types of texts: (a) the name , (b) the category name (with a total of 55 categories), and (c) the concatenation of the sub-category names (with a total of 336 sub-categories), separated by commas.

Burningcandleintheplatewithgoldbackground.

A chairfor a patient.

Amugofcoffeeforthemorning.

A dragon flying in the sky.

The newrelease of the track.

A pen is a toolfor drawing.

A light in the darkness.

A close up of a fan.

A butterfly in the wild.

A pictureof a pocket watch.

The newmen'st-shirt with thelogo.

Set of theglasses and plates.

Apictureofatractor.

Sportsequipmentforthegym.

Book:the story of person.

PELIGISIB

SUDPER

BESTSUAYWNGHE

FRESIRE SUPR.

3DFuture : For each shape, we generate two types of texts: (a) the category , and (b) the concatenation of category , style , theme , and material , separated by commas.

ABO : For each shape, we generate two types of texts: (a) the item_name , and (b) the product_type . In this way, we generate one or more raw texts for each shape.

C.2 Raw Text Filtering

I am analyzing a 3D dataset with various text descriptions for the 3D models.

However, many of these texts are inaccurate or uninformative, and therefore,

not suitable as descriptions for 3D models. I need your help to identify such

incorrect texts. Specifically, if a text primarily consists of irrelevant or unin-

formative content, such as timestamps, model numbers, incomprehensible

descriptions, random filenames (e.g., "my project"), random characters, etc.,

please respond with "N". If a text contains a clear noun (or noun phrase)

that could potentially describe a 3D object, please respond with "Y". You

will find a list of texts below, and each line contains a three-digit ID and

associated text. For each text, please respond with "Y" or "N", following the

ID number (e.g., "001 Y" or "002 N"). Please evaluate all 256 texts.

000 New project ( 19 )

001 3December - Chemestry

002 Fake Brand Soda Can

003 Spartan Shild

004 Apple3d

005 Landmine

006 FaunveinB-S

007 FIGURA 5

009 Sofa

010 Maddox

011 A3 Complete

012 Suspension Bridge

014 Captain-americas-shield

015 sphorb4

......

000 N

001 Y

002 Y

003 Y

004 Y

005 Y

006 N

007 N

008 Y

009 Y

010 N

011 N

012 Y

013 N

014 Y

015 N

......

We employ GPT-4 [47] to filter out uninformative raw texts. To accomplish this, we divide all the raw texts into batches, each containing 256 entries, and process each batch independently using GPT-4. Here is an example illustrating the prompt we used and the corresponding response generated by GPT-4.

Afterwards, we combine all the responses to create the final filtering results, effectively removing approximately 30% of the raw texts.

D Details on the Backbone Scaling Experiment

In Figure 4 of the main paper, we investigate the performance and scalability of various backbones when scaling up their model sizes. For this experiment, we employ a default resolution of 10,000 points for input point clouds, a batch size of 200, and conduct the experiment on a single A100 GPU. In general, if instructions are given in the original paper of a backbone, we scale up the model as instructed. Otherwise, we scale up the model by expanding width or depth (i.e., stacking blocks or layers). Specifically, we scale up each backbone as follow:

PointBERT [81] The scaling parameters are shown in Table 4. We scaled PointBERT to 72.1M parameters beyond the 32.3M version reported in Figure 4 of the main paper. However, at this scale, the model dramatically overfits on the training data and performs worse on all benchmarks than the 32.3M version.

SparseConv [10] The smallest version (5.3M parameters) of the model is adapted from the MinkowskiFCNN model by adjusting the width of the final convolution and linear layers. The remaining three models are adaptations of MinkowskiResNet, each varying in the number of basic ResNet blocks used. See Table 5 for the specific scaling parameters.

PointNeXt [54] PointNeXt is proposed as a scalable version of PointNet++ [52], and includes S/B/L/XL variants in the original paper. We simply adopt these official configurations.

DGCNN [73] and PointNet [51] For these two backbones without a hierarchical structure, we increase the width of each layer proportionally to scale up to 4xPointNet and 2xDGCNN before we hit the GPU memory limit. As the models operate completely on dense points, it is impractical to use the default 10k-point resolution. We thus reduce the input resolution for the two backbones, resulting in 1k points for DGCNN and 4k points for PointNet.

E Details on Training and Evaluation

Training Details We freeze the CLIP text and image encoders and train the 3D encoder and two projection heads on our ensembled dataset using the cross-modal contrastive loss. We train the model on a single A100 GPU with a batch size of 200. Since we precache the text and image CLIP embeddings of all shapes, the training is greatly accelerated and takes about 300 A100 hours for convergence. We utilize an exponential learning rate schedule, and employ an range test to find the initial learning rate. For 32.3M version of PointBERT, we utilize a learning rate of 5 e -4 ; for 72.1M version of PointBERT, we utilize a learning rate of 4 e -4 ; and for other models, we utilize a learning rate of 1 e -3 . For hard-negative mining, the number of seed shapes s is set to 40, and the number of neighbors m is set to 5 per shape, and the threshold δ is set to 0.1.

Fine-tuning CLIP Text and Image Encoders? After training OpenShape-PointBERT, we conducted experiments to unfreeze and finetune the CLIP text encoder for a single epoch. However, the results obtained did not demonstrate any noticeable improvement on the benchmarks. Moreover, we observed that finetuning the CLIP text encoder could potentially undermine the generalization capabilities of CLIP and hinder the integration of OpenShape embeddings into existing CLIP-based models. As a result, we choose to freeze the CLIP encoders throughout the entire training process.

Evaluation Details We evaluated all baselines using their publicly released pretrained checkpoints. Additionally, we retrained ULIP [78] on our ensembled training shapes using their official code base and backbone networks. Note that the retrained ULIP model utilized the original raw texts from the four datasets during training (prompt engineering is also applied), rather than our filtered and enriched texts. For ModelNet40 [75], the evaluation is conducted on the test split with 2,468 shapes. Regarding ScanObjectNN [71], we follow ULIP [78] to evaluate on the OBJ_ONLY version, which contains 581 test shapes. For Objaverse-LVIS [12], the input is 10,000 sampled points with point colors. For ModelNet40 [75], the input is 10,000 sampled points without color. For ScanObjectNN [71], we utilize the official 2,048 points without color as input. All methods use the same input during evaluation. The forward inference time on an A100 GPU for a 10,000-point point cloud is approximately 0.9ms for OpenShape-SparseConv and 3.8ms for OpenShape-PointBERT.

F Details on Shape-Conditioned Multimodal Generation

Point Cloud Captioning CLIPCap [46] utilizes a 10-token prefix generated from CLIP image embeddings to enable GPT-2 for captioning. In order to align with the off-the-shelf CLIPCap model, we trained a variant of OpenShape-PointBERT that employs CLIP ViT-B/32 embeddings instead of OpenCLIP ViT-G/14 used in other experiments. Consequently, we directly input the point cloud encoding, without normalization , into CLIPCap for captioning.

Point Cloud Conditioned Image Generation We take the Stable Diffusion v2.1 unCLIP model [56] for image generation and replace the CLIP image condition encoder with our OpenShape encoder to perform image generation conditioned on point clouds (and optionally text prompts). The unCLIP model takes CLIP ViT-L/14 embeddings without normalization as input. To match the embedding space, we trained a variant of OpenShape-PointBERT with CLIP ViT-L/14 embeddings. Additionally, we noticed a significant mismatching of scales ( L 2 -norm of embedding vectors) between ViT-L/14 image embeddings and OpenShape embeddings. To mitigate this issue, we perform a re-normalization on OpenShape embeddings to a L 2 -norm of 1 2 √ 768 , which is our observed mean L 2 -norm of ViT-L/14 image embeddings. We use 50 diffusion steps. The guidance scale can be tuned freely.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new system for learning a joint embedding space for text, images, and 3D shapes (point clouds). It starts from pre-trained CLIP embeddings for text and images, and learns one more embedding function for 3D shapes in a self-supervised manner, using a contrastive loss. Several additional improvements are proposed, including combining several datasets, cleaning some of those including via automated re-labelling of Objavrese, and optimisation of the 3D embedding neural network architecture. Because of these improvements, the new embedding performs significantly better than prior works on strand tasks such as \"zero-shot\" classification.\n\nSTRENGTHS:\n* The paper contributes sound incremental work on learning multi-modal embedding functions for text, images and 3D shapes. Various engineering dimensions are considered (model scaling, data diversity, scaling and quality, etc) and some insights are provided on how each of these can be improved, at least compared to current baselines. I expect these findings to provide useful guidelines for future work in this area.\n\n* The paper does contain some interesting suggestions on how to improve datasets such as Objaverse to train multi-modal embeddings. The scheme where captions are assessed via GPT-4 and then replaced using BLIP if needed is useful.\n\nWEAKNESSES:\n* The paper is mostly about good engineering, but there isn't a lot of very deep technical innovations, or qualitatively surprising findings. The main finding, in fact, is that this is an area where scaling is still very limited (mostly due to the lack of suitable training data), and that, thus, scaling is where most of the low-hanging fruits can be had. This is a good but not very surprising message.\n\n* The model is trained on synthetic datasets and looses much of its edge when applied to real data, compared to prior models like ULIP that are otherwise suboptimal when tested on synthetic data (e.g., Figure 5, right panel).\n\n* Likewise, there isn't a very clear dominance of the two proposed backbones, SparseConv and PontBert: their performance swings significantly depending on the testing data in Table 2.\n\nQUESTIONS:\n* Filtering bad positive samples in contrastive the loss using the existing text and image embeddings (line 213) is simple but also likely useful, however it is not clear to me if this idea is ablated in Table 3. Can you clarify?\n\n* Why SparseConv and PointBERT are so different in performance depending on the test dataset? Is there a recognisable pattern?\n\nLIMITATIONS:\nThe paper does address some limitations, particularly in the discussion and conclusions where they are implicitly recast as \"future work\".\n\nThere is no discussion of ethics, although this does not seem to be an issue for this paper (notoriously, though, dataset like ShapeNet had issues with copyright, and I don't believe ObjaVerse is necessarily immune from such controversies either).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of multi-modal learning of text, image and shapes. Shapes are represented by pointclouds and the learning is driven by the standard contrastive loss. The main technical difference of this work to prior ones lies in the scale of data and corresponding training strategies. The authors train their model on a much larger scale of data which pools Objaverse, ShapeNet, 3D-Future and ABO. To handle the noise in the text (mainly from the less-curated Objaverse), the authors use several large pretrained models to clean up the text and make it better aligned with the shapes. Hard-mining is also leveraged to make the model train better under class imbalance. With these techniques which make the model scales better with data size, the proposed model achieves the new SOTA performance across several relevant tasks and datasets.\n\nSTRENGTHS:\n1. The model in this paper achieves significantly better results than previous SOTA in zero-shot shape classification. It also demonstrates great qualitative performance in shape retrieval, shape interpolation, shape captioning and has good potential in other cross-modal applications as well. Such model will be practically very useful.\n2. The benchmark and the findings in the paper about different models' scaling performance are valuable information to the community and future research along this direction.\n3. The paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. The evaluation of the zero-shot shape classification on Objaverse-LVIS might not actually be zero-shot. Based on the description in L235, even the \"Ensembled (no LVIS)\" can include LVIS categories, because seemingly the author only excludes the exact evaluation samples from the training set. Ideally, all shapes from the evaluation *categories* should be removed from the training set. This makes the real zero-shot generalization performance of the model potentially lower than reported and the overall comparison on this test set less informative. Besides, ModelNet40 and ScanObjectNN can also have overlapping categories with the training set. (If the authors think category overlapping does not violate their definition of zero-shot, it should be clearly stated so / \"zero-shot\" should be clearly defined at the beginning.)\n\n2. The linear probe results show the proposed model performs similarly to ULIP-retrained on ModelNet and ScanObjectNN. The authors try to explain it with in-category sample bias (what does this mean?) and domain gap dominance, without providing any evidence for these hypotheses. My concern here is that, given that OpenShape performs much better than ULIP in shape classification (requires both shape and language rep) but not linear probe (only requires shape rep), the actual underlying reason could be that the shape representation is of similar quality when tested on out-of-domain, and the real difference between these models lies in the text representation. I believe it would be quite beneficial to provide more analyses on this result, as this is central to the \"shape representation learning\" story of the paper. For example, such analyses could be visualizing and comparing the latent structure of the shape representation; or simply performing shape-latent based NN query and evaluate the distance between queried shapes; or measuring the shape latent's alignment with image latent, to name a few. \n\n3. The limitation of the model is not discussed at all. This is a very important aspect for readers to thoroughly understand the contribution of this paper.\n\nQUESTIONS:\nPlease see the weaknesses for my main questions and concerns. Overall I like the paper as it presents useful findings for the community and the model's performance is strong. But I think more analyses and discussions are necessary to solidify some experiments and the overall claim.\n\nAnother relatively minor question about the result:\nIn Table 2, the retrained ULIP achieves 26.8% top 1 while OpenShape achieves 43.4%. The authors attribute this to \"text enrichment, backbone scaling, and other training strategies\" (L243). However, it is shown in the ablation that removing text enrichment leads to 6.4% drop and hard mining leads to 1.4% drop. Does this mean the main improvement over ULIP comes from the larger number of parameters/better backbones as the overall drop is ~17%?\n\nLIMITATIONS:\nThe limitations and failure cases are not discussed and should be included.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a joint learning framework for multi-modal representations among text, image, and point clouds. Specifically, the authors fix the pretrained CLIP language and image encoder, and align the point cloud representation by the proposed multi-modal representation alignment technique. Then, to enlarge the pretraining dataset, the authors introduce text filtering and enrichment technique to annotate point clouds without descriptions. The proposed OpenShape is pretrained on an large-scale ensembled dataset, and the experimental results demonstrate that OpenShape has promising zero-shot classification ability and shape retrieval ability.\n\nSTRENGTHS:\n1. The proposed method fully leverages the large-scale point cloud datasets to align the point cloud backbone with frozen pretrained CLIP encoders, thus obtaining open-world point could recognition and shape retrieval ability. \n\n2. The proposed text filtering and enrichment technique is effective to preprocess large-scale point cloud datasets. \n\n3. The overall writing is polished.\n\n4. The ablation study is extensive and critical.\n\nWEAKNESSES:\nMy major concern is the fairness in the experiments (i.e., table 2). As the authors stated in section 3.1, the chosen vision-language encoder is OpenCLIP ViT-G-14, which is an extremely large backbone and obtains much better feature representation ability. Therefore, the authors should list the point cloud backbone and pretrained vision-languge encoders concretely in the table, and make fair comparisons if possible.\n\nQUESTIONS:\nSee the weakness part.\n\nLIMITATIONS:\nThe authors do not state the limitation of this method. Instead, the authors present the future direction of the proposed OpenShape, i.e., part-level information and synthetic-to-realistic domain gap. These directions can be seen as the limitation of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work learns a multi-modal representation among text, images, and point clouds based on a scaling 3D dataset. The results demonstrate remarkable performance in point cloud zero-shot classification, retrieval, and captioning tasks. In order to construct this dataset, the authors combined four commonly used 3D datasets while filtering and enhancing text prompts. Additionally, they introduced an offline hard negative mining strategy to improve the efficiency of the training process.\n\nSTRENGTHS:\n1. This paper collects 0.8M text-image-point cloud triplets from four popular 3D datasets and does text cleaning and enrichment to get high-quality prompts for each object.\n2. This work proposes a hard negative mining strategy to improve joint representation training efficiency and performance.\n3. The trained representation shows surprising performance on zero/few-shot classification.\n\nWEAKNESSES:\n1. The major problem is an insufficient novelty compared with ULIP. The pertaining method, triplets construction and downstream tasks are very similar to ULIP.  The primary distinguishing point of this work is the text prompt cleaning and enrichment flowchart, as well as the offline hard negative mining strategy employed. However, the latter strategy lacks an ablation study to provide supporting evidence.\n\n2. The motivation behind this work is not clearly articulated. While the aim is to develop a scaling-up 3D representation, the purpose of this representation is unclear. Unlike ULIP, which focuses on enhancing current 3D backbones through developing a new pipeline, this work attempts to scale up text-to-3D pairs through text augmentation that is still limited by the insufficient number of 3D objects. \n\n3. There is a lack of exploration into the downstream applications of pre-trained representation to counterparts in 2D. This work is more suitable for submission to the dataset and benchmark track.\n\nQUESTIONS:\n- The main contribution of this work is the datasets ensemble. Will the authors release the collected dataset and pre-trained 3D representation, similar to ULIP?\n\n- How will multiple object combinations and scene models be addressed in the Objaverse dataset? Will such data be ignored? According to Table 2, pre-training on the assembled dataset has a negative impact on the ScanObjectNN dataset, indicating overfitting in the Objaverse dataset and a degradation of the transferability between the simulation and real domains.\n\n- Section 3.4 appears to be an experimental setup and should be placed in the experiments and analysis chapter.\n\n- Lack of related work: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data\n\nLIMITATIONS:\nNo needed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

comments: SUMMARY:
OpenShape explores scaling-up strategy for learning joint representations of texts, image, and 3D point clouds. It proposes to construct larger-scale 3D datasets, filter and enrich paired texts for pre-training, which achieves SOTA results on zero-shot 3D classification and retrieval benchmarks.

STRENGTHS:
1. The zero-shot performance of OpenShape is impressive, surpassing existing SOTA methods with large margins. This demonstrates the significance of the method.

2. The retrieval visualizations are interesting and clearly illustrate the embedding ability of OpenShape.

3. The authors re-train exiting methods under new settings, which is good for a fair comparison.

WEAKNESSES:
1. OpenShape utilizes GPT-4 for textual-level processing, which however is expensive to access. How about the performance using a more affordable GPT-3 (like PointCLIP V2) ?

2. I'm curious about how OpenShape can be incorporated with non-parametric 3D network Point-NN (or its parametric derivative Point-PN) for zero-shot learning?

Starting from Non-Parametric Networks for 3D Point Cloud Analysis, CVPR 2023

QUESTIONS:
None

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08