SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ruoxi  $\mathbf{Sh}\mathbf{i}^{2*}$
Shizhong Han<sup>3</sup>
Kaiming Kuang $^{1*}$
Hong Cai<sup>3</sup>
Yinhao Zhu
Fatih Porikli<sup>3</sup>
Xuanlin Li
$^{2}$  Shanghai Jiao Tong University
$^{3}$  Qualcomm AI Research†
Project Website: https://colin97.github.io/OpenShape/
We introduce OpenShape, a method for learning multi-modal joint representations of text, image, and point clouds. We adopt the commonly used multi-modal contrastive learning framework for representation alignment, but with a specific focus on scaling up 3D representations to enable open-world 3D shape understanding. To achieve this, we scale up training data by assembling multiple 3D datasets and propose several strategies to automatically filter and enrich noisy text descriptions. We also explore and compare strategies for scaling 3D backbone networks and introduce a novel hard negative mining module for more efficient training. We evaluate OpenShape on zero-shot 3D classification benchmarks and demonstrate its superior capabilities for open-world recognition. Specifically, OpenShape achieves a zero-shot accuracy of  $46.8\%$  on the 1,156-category Objaverse-LVIS benchmark, compared to less than  $10\%$  for existing methods. OpenShape also achieves an accuracy of  $85.3\%$  on ModelNet40, outperforming previous zero-shot baseline methods by  $20\%$  and performing on par with some fully-supervised methods. Furthermore, we show that our learned embeddings encode a wide range of visual and semantic concepts (e.g., subcategories, color, shape, style) and facilitate fine-grained text-3D and image-3D interactions. Due to their alignment with CLIP embeddings, our learned shape representations can also be integrated with off-the-shelf CLIP-based models for various applications, such as point cloud captioning and point cloud-conditioned image generation.
3D shape understanding has recently garnered a surge of interest driven by the growing demands in real-world applications, such as augmented/virtual reality, autonomous driving, and robotics. Despite significant advancements in 3D recognition and analysis, existing data-driven approaches are still greatly limited by the scale of 3D training datasets and tend to exhibit poor generalization when facing unseen shape categories, hindering the deployment of existing models in real-world applications.
Note that 3D shapes and 2D images can be easily linked through rendering, and the dataset scale issue of 2D images has been remarkably addressed, as shown in recent works such as CLIP [55]. Therefore, many recent studies aim to utilize pre-trained 2D image-language models [55, 59] to assist 3D tasks, such as 3D generation [22, 26, 45, 63, 33, 7, 38, 67] and 3D scene-level segmentation [18, 27, 14, 79,
Figure 1: Left: Zero-shot shape classification on the Objaverse-LVIS (1,156 categories) and ModelNet40 datasets. OpenShape outperforms previous methods by a large margin. We exclude shapes in Objverse-LVIS during training, and we also retrain ULIP [78] on our ensembled training shapes for fair comparison. Right: Our shape representations encode a broad range of semantic and visual concepts. We input two 3D shapes and use their shape embeddings to retrieve the top three shapes whose embeddings are simultaneously closest to both inputs. See Section. 4.4 for more details.
41, 49]. Regarding 3D shape-level understanding, a straightforward idea is to project 3D data to the 2D domain through rendering and use CLIP to analyze the 2D images, thereby enabling zero-shot 3D shape classification [86, 88]. However, these methods suffer from occlusion and information loss during projection, and unnecessary latency due to point cloud rendering and multiple CLIP inferences.
To overcome the limitations caused by projection, it is necessary to train a 3D-native model by distilling knowledge from pretrained 2D models. However, training a 3D-native model requires a set of 3D shapes, and the amount of knowledge that can be distilled is determined by the size of the 3D dataset. For example, ULIP [78] aims to learn a joint representation space between language, 2D images, and 3D shapes, but uses a small-scale 3D dataset ShapeNetCore [8] for knowledge distillation. Specifically, ULIP fixes the 2D CLIP text and image encoders and trains a dedicated 3D-native point cloud encoder to extract 3D shape representations. The 3D encoder strives to align the 3D shape embedding space with the CLIP image and language embedding spaces by utilizing contrastive learning across all three modalities. However, since ULIP is only trained on 52K shapes of 55 object categories, it still struggles with out-of-distribution shape categories and fails to demonstrate an impressive open-world understanding of 3D shapes.
In this work, we propose a novel method called OpenShape, which follows a similar paradigm as ULIP but aims to achieve a more generalized and scalable joint representation space encompassing language, 2D images, and 3D shapes. Our focus mainly lies on scaling up representation learning and addressing corresponding challenges. In OpenShape, we emphasize four key factors during the training process: (a) data scale: we significantly increase the scale of 3D training data by combining four public 3D shape datasets, resulting in 876k 3D shapes covering much more diverse categories; (b) text quality: the 3D shapes from our main dataset, Objaverse [12], is dominated with inaccurate or uninformative text descriptions. Given the data scale, we propose three strategies to automatically filter and enrich the text descriptions; (c) 3D backbone scaling: since most existing 3D backbones target small datasets, we find that it's important but non-trivial to scale up the 3D backbones; and (d) data resampling: since the ensembled dataset is highly unbalanced, we utilize hard negative mining to improve the model's discriminative ability.
We first evaluate OpenShape on the zero-shot 3D shape classification task. As shown in Figure 1, OpenShape outperforms previous zero-shot approaches on the ModelNet40 dataset by at least  $20\%$ . Moreover, OpenShape excels at handling long-tail categories. On the challenging Objaverse-LVIS dataset, which contains 1,156 categories, OpenShape achieves a  $46.8\%$  accuracy, significantly surpassing previous methods. Notably, this performance gap remains even when ULIP is retrained on our ensembled datasets, highlighting the superiority of our text enrichment and training strategies.
Besides zero-shot classification, we present demos that showcase the wide range of visual and semantic concepts learned by OpenShape. For example, in Figure 1-right, we take two 3D shapes as input and use their OpenShape embeddings to retrieve the top three shapes whose embeddings are simultaneously closest to both inputs from our ensembled dataset. The retrieved shapes exhibit an interesting combination of the semantic and geometric elements from both input shapes. Furthermore, since we align our 3D shape embedding space with the CLIP language and image embedding space, we demonstrate that OpenShape embeddings can be easily integrated with other CLIP-based models to perform cross-modality tasks such as point cloud captioning and point cloud-conditioned image generation.
2.1 CLIP for 3D Learning
Image-language models like CLIP have achieved remarkable performance through large-scale image-text pretraining [55, 29, 35, 84, 4, 56, 61, 36]. As these models excel at capturing rich visual concepts and possess impressive zero-shot capabilities, they have been applied to various 3D vision tasks. For instance, numerous recent works utilize CLIP to facilitate zero-shot text-to-3D generation [22, 26, 45, 63, 33, 7, 32, 5, 28, 77, 40, 83], typically through CLIP-guided per-scene optimization. From a recognition perspective, some works focus on scene-level representation, aiming to leverage CLIP priors for zero-shot 3D segmentation or detection in both indoor [18, 27, 14, 79, 41, 49, 82, 23, 60, 85, 31] and outdoor scenes [9, 21]. Meanwhile, another line of work focuses on shape-level understanding, targeting zero-shot shape classification [86, 88, 53, 78, 19] and part segmentation [39, 1]. There are two primary working paradigms for these methods. The first [86, 88, 24] involves using images as a medium representation, projecting 3D point clouds into 2D and employing 2D CLIP for inference. However, these methods typically suffer from occlusion and information loss during projection, along with unnecessary latency due to point cloud rendering and multiple 2D CLIP inferences. The second paradigm involves training a 3D-native encoder attempting to distill or fuse CLIP features into 3D representations. Our paper follows this paradigm.
2.2 3D Shape Representation Learning
Various works have studied self-supervised pretraining for point clouds by designing pretext tasks [15, 69, 50, 2, 66] such as self-reconstruction [57, 13, 3, 72], masked auto-encoding [48, 80, 20], distortion reconstruction [64, 44, 68], normal estimation [57], and contrastive learning [87, 62, 76]. These tasks enhance models' shape representations and improve their performance on downstream applications, although they do not involve multimodal semantic alignments during pretraining.
Recently, some works [53, 78, 19], exemplified by ULIP [78], have explored learning multimodal joint representations for 3D shapes. They train 3D-native shape encoders by aligning 3D shape embeddings with CLIP's language and/or image embeddings through multimodal contrastive learning. Works like ReCon [53] further combines cross-modal contrastive learning with masked auto-encoding for added enhancement. While these methods allow for zero-shot 3D classification through the computation of 3D-text similarity, the amount of distilled knowledge and their model capability are heavily limited by the small-scale training datasets used. Our work follows this paradigm but aims to learn more generalizable and scalable representations to enable open-world 3D shape understanding.
We propose a novel method, OpenShape, for learning generalizable and scalable multi-modal joint representation between language, 2D images, and 3D shapes, as shown in Figure 2. We first introduce the multi-modal contrastive learning framework we used for aligning representations of three modalities in Section 3.1. We then elaborate how we create our training sets and enrich our text data in Sections 3.2 and 3.3. In Section 3.4, we present how we scale up our 3D backbone models. Finally, we propose a hard negative mining strategy to enhance contrastive learning in Section 3.5.
(a) Ensemble Datasets
(b) Text Filtering & Enrichment
(c) Cross-Modal Alignment
Figure 2: (a) We ensemble four public 3D shape datasets, resulting in 876k shapes that encompass diverse categories and concepts. (b) We propose three strategies to automatically filter and enrich the noisy texts in the original datasets. (c) We train a 3D point cloud encoder to align the 3D shape embedding space with the CLIP's text and image embedding spaces. We perform cross-modal contrastive learning with scaled 3D backbones and hard negative mining. (d) OpenShape embeddings can be easily integrated with other CLIP-based models, enabling various cross-modality tasks.
(d) Cross-Modal Applications
3.1 Multi-Modal Representation Alignment
We aim to learn 3D shape representations that are aligned with pretrained CLIP embedding spaces of language and image. As shown in Figure 2 (c), we train a 3D native encoder  $f^{P}$  that takes a 3D point cloud as input and extracts 3D shape feature. Following previous works [53, 78, 19], such as ULIP [78], we utilize multi-modal contrastive learning for representation alignment. Since CLIP is pretrained on a much larger scale data, we freeze both its text encoder  $f^{T}$  and its image encoder  $f^{I}$  during feature alignment to preserve CLIP's feature priors and avoid model collapse. Specifically, given a sampled batch of triplets  $\{(P_{i}, T_{i}, I_{i})\}$ , where  $P_{i}$  denotes a point cloud of a 3D shape,  $T_{i}$  and  $I_{i}$  denote corresponding text and image, the contrastive loss is calculated as:
$$
- \frac {1}{4 n} \sum_ {i} \left(\log \frac {\exp \left(h _ {i} ^ {P} \cdot h _ {i} ^ {T} / \tau\right)}{\sum_ {j} \exp \left(h _ {i} ^ {P} \cdot h _ {j} ^ {T} / \tau\right)} + \log \frac {\exp \left(h _ {i} ^ {T} \cdot h _ {i} ^ {P} / \tau\right)}{\sum_ {j} \exp \left(h _ {i} ^ {T} \cdot h _ {j} ^ {P} / \tau\right)} + \log \frac {\exp \left(h _ {i} ^ {P} \cdot h _ {i} ^ {I} / \tau\right)}{\sum_ {j} \exp \left(h _ {i} ^ {P} \cdot h _ {j} ^ {I} / \tau\right)} + \log \frac {\exp \left(h _ {i} ^ {I} \cdot h _ {i} ^ {P} / \tau\right)}{\sum_ {j} \exp \left(h _ {i} ^ {I} \cdot h _ {j} ^ {P} / \tau\right)}\right)
$$
where  $n$  is the number of shapes in a batch;  $\tau$  is a learnable temperature;  $h_i^P = f^P (P_i) / |f^P (P_i)|$ ,  $h_i^T = g^T (f^T (T_i)) / |g^T (f^T (T_i))|$ , and  $h_i^I = g^I (f^I (I_i)) / |g^I (f^I (I_i))|$  denote normalized projected features of  $P_{i}$ ,  $T_{i}$ , and  $I_{i}$ , where  $g^{T}$  and  $g^{I}$  are two learnable linear projections. Since  $f^{T}$  and  $f^{I}$  are frozen, we extract all  $f^{T}(T_{i})$  and  $f^{I}(I_{i})$  before training and cache them for acceleration. In most of our experiments, we utilize OpenCLIP ViT-bigG-14 [25] as the pretrained CLIP model.
3.2 Ensembling 3D Datasets
Since the scale and diversity of training triplets play a crucial role in learning scalable shape representations, we ensemble four currently-largest public 3D datasets for training as shown in Figure 2 (a), resulting in 876k training shapes. Among these four datasets, ShapeNetCore [8], 3D-FUTURE [16] and ABO [11] are three popular datasets used by prior works. They contain human-verified high-quality 3D shapes, but only cover a limited number of shapes and dozens of categories. The Objaverse [12] dataset is a more recent dataset, containing many more 3D shapes and covering significantly more diverse categories. However, shapes in Objaverse are mainly uploaded by web users and not verified by experts, and thus have uneven quality and exhibit highly unbalanced distributions, necessitating further processing.
To create triplets for training, for each shape, we sample 10,000 points from the mesh surface and interpolate the point colors according to the mesh textures. We also render 12 color images from the preset camera poses that uniformly cover the whole shape. For datasets providing thumbnails, we include them as part of image candidates, since they typically capture the shape from a better camera view. For the Objaverse dataset, we use the model name as the raw text for each shape. For other datasets, we utilize provided metadata to create raw texts (see supplementary for details). During each pretraining iteration, we randomly sample one rendered image or thumbnail for each shape, and apply standard augmentation to the point clouds [78].
3.3 Text Filtering and Enrichment
We find that only applying contrastive learning between 3D shapes and 2D images is insufficient to fuel zero-shot 3D classification, even when training on large-scale datasets. We conjecture that this is
name:"DOG A - 1of6 - for free .. GPT4: Keep
azure: "a black dog sitting on a blue..."  
blip: "a black dog sitting on a blue..."
"Black Labrador in front of a white."  
Black Labrador puppy Vinyl Wall Mural
ing on a blue..."   
ng on a blue..."
name: "Tue, 09 Oct 2018 17:12:39"  
GPT4: Remove
azure:"a blue plastic chair on a pink .. blip:"a 3d model of a blue shaped object
"armchair"
"some of the other props done Chair10"
name:"homework xyz detailing"  
GPT4: Remove
Figure 3: Text Filtering & Enrichment Examples In each example, the left section features the thumbnail, model name, and GPT-4 filtering results. The upper right section shows image captions from two captioning models, while the lower right section displays retrieved images and their corresponding texts.
azure: "a pair of steampunk goggles"  
blip: "steampunk goggles 3d model"
"Steampunk Goggles by MonoFlow on ..."
"Steampunk Goggles Made from hand ..."
by MonoFlow on ...
name:"untitled"
GPT4: Remove
azure: “ a close up of a shoe”blip : nike air max 1 - white
"pike air max red and white"
"Nike AirMax 1 (Red/White)"
caused by the inherent domain gap in CLIP's language and image embedding spaces, which is also observed by previous studies [37, 70]. Consequently, 3D-text alignment is not guaranteed even if we obtain good 3D-image alignments via contrastive learning. Therefore, we need to explicitly align 3D shapes with text. Along this process, to facilitate better 3D-text alignment, we introduce 3 techniques to improve the text quality: filtering, captioning, and image retrieval, as shown in Figure 2 (b).
Filtering. As shown in Figure 3, the 3D shapes from our main dataset, Objaverse, is dominated with noisy text descriptions ("names") uploaded by web users. Many of the problematic texts can be identified from the text itself without seeing the corresponding 3D shape. We thus leverage a powerful large language model, GPT-4 [47], to filter out inaccurate or uninformative text descriptions. We find that GPT-4 excels at recognizing irrelevant contents, such as timestamps, pure model numbers, incomprehensible descriptions, random filenames (e.g., new project), and random characters. Through GPT-4, we filter out about  $30\%$  of raw user texts. Note that we only filter the texts, and still keep all shapes for training. More details, such as the prompts we used, are presented in the supplementary.
Captioning. We utilize BLIP [34] and the Azure cognition services to caption the 2D thumbnails (if present, or images rendered from a fixed frontal view) of the 3D models, obtaining two texts for each shape. As shown in Figure 3, the captioning models can usually produce meaningful and descriptive captions that either enhance user-uploaded texts or replace low-quality ones. We also notice that the two caption models complement each other, leading to better performance.
Image Retrieval. In addition to image captioning, we also perform image retrieval to obtain additional descriptions of 3D models. We retrieve k-NN images of shape renderings from the LAION-5B dataset [65] using the CLIP ViT-L retrieval index [6]. We then take the captions of the k-NN images as the retrieved texts for our 3D models. Compared with captioning model generations, retrieved texts cover a wider range of text styles. They can also include more fine-grained semantics than both the user texts and the generated captions (e.g., "Labrador" in Figure 3).
In each iteration of pretraining, for each shape, we first randomly sample a text source category among the raw text (if unfiltered), the captions, and the retrieved texts. We then select a text candidate from the selected category. We also apply the template-based prompt engineering technique used in ULIP [78] to both training texts and test-time category names. Specifically, we extend a word or a phrase to a collection of templated simple sentences and take their average embedding.
3.4 Scaling Up 3D Point Cloud Backbones
Previous works on 3D point cloud learning have primarily focused on smaller-scale datasets like ShapeNet. These techniques may not be directly applicable to our larger-scale ensembled dataset and need to be scaled up accordingly. We find that different 3D backbones may exhibit distinct behavior and scalability when trained on datasets with varying sizes. Specifically, we compare six popular
Table 1: Comparison of different 3D backbones before scaling up their parameters. Models are trained on ShapeNet [8] or our ensembled dataset excluding Objaverse-LVIS [12]. Zero-shot classification performance are evaluated on ModelNet40 [75] and Objaverse-LVIS [12].
Model	#Param.	Train on ShapeNet [8]	Train on Ens-no-LVIS
MNet40	O-LVIS	MNet40	O-LVIS
PointNet [51]	1.3M	67.0	9.3	74.9	24.4
DGCNN [73]	2.3M	67.8	9.0	74.2	24.8
PointMLP [42]	9.3M	73.5	12.9	82.9	36.6
PointNeXt [54]	2.8M	72.6	12.2	81.6	33.8
PointBERT [81]	5.1M	70.3	10.8	84.5	37.0
SparseConv [10]	5.3M	70.7	10.6	78.8	31.7
std. dev.	2.3	1.4	3.9	5.1
Figure 4: Accuracy on ObjaverseLVIS [12] when scaling up the parameters of different models.
backbones trained on ShapeNet or our ensembled dataset by evaluating their zero-shot classification performance on ModelNet40 [75] and Objverse-LVIS datasets (for now, these backbones are trained with their original configurations and without scaling up model sizes). Objverse-LVIS is a subset of Objverse dataset with human-verified category labels. With 1,156 categories, it serves as a suitable dataset for evaluating zero-shot long-tail classification, and we exclude all shapes of Objverse-LVIS from this experiment. Results are shown in Table 1. We find that when trained on ShapeNet, all backbones share similar performances. However, when trained on our ensembled dataset, the performance gap between backbones increases significantly. This suggests that while the original versions of these backbones share a similar number of parameters, some may have been saturated when trained on small datasets, while others do not.
We also explore the performance and scalability of these backbones when scaling up the model sizes and training on our ensembled dataset. Please refer to the supplementary for details on how we scale up each model. As shown in Figure 4, we observe that all 3D backbones benefit significantly from model scaling. However, traditional backbones without a shrinking hierarchical structure, such as DGCNN and PointNet, require operating completely on dense points or modeling the relationships (e.g., through kNN) between dense points. As a result, they become more time-consuming and memory-intensive when scaled up compared to more modern backbones. We therefore select PointBERT [81] (Transformer-based) and SparseConv [10] (convolution-based) as our 3D backbones for the remaining experiments, as they exhibit strong performance and scalability.
3.5 Hard Negative Mining
Our ensembled dataset exhibits a high degree of class imbalance. Certain common categories, such as building, may occupy tens of thousands of shapes, while many other categories, such as walrus and wallet, are underrepresented with only a few dozen or even fewer shapes. Consequently, when randomly constructing batches, it is unlikely that shapes from two confusing categories (e.g., apples and cherries) will be contrasted within the same batch. Inspired by some previous works [58, 30], we propose an offline hard negative mining strategy for improving the training efficiency and performance. Specifically, in the first round of training, we train our model with random batches until it is about to converge. We then compute the kNN for each shape in the learned 3D embedding space. In the second round of training, for each iteration, we randomly select  $s$  seed shapes and then obtain  $m$  neighbors from the kNN results of each seed shape, resulting  $s \times m$  shapes per batch. In this way, confusing pairs are more likely to be selected in a single batch. However, this may also introduce false negative pairs (e.g., two apples) into contrastive learning. To mitigate this issue, we leverage image and text embeddings to filter out pairs sharing similar texts when calculating the contrastive loss. Specifically, for two shapes  $i$  and  $j$  selected from the same seed shape, if  $h_j^T \cdot h_i^I + \delta > h_i^T \cdot h_i^I$ , where  $h^T$  and  $h^I$  are text and image embeddings, and  $\delta$  is a small threshold, we believe that the text embeddings of  $i$  and  $j$  are very close to each other, and we remove  $j$  from  $i$ 's negative examples when calculating contrastive loss. By employing this strategy to construct batches, we observe faster and better model learning.