SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Zero-shot classification on Objverse-LVIS [12], ModelNet40 [75], and ScanObjectNN [70].
Method	training shape source	Objaverse-LVIS [12]	ModelNet40 [75]	ScanObjectNN [71]
Top1	Top3	Top5	Top1	Top3	Top5	Top1	Top3	Top5
PointCLIP [86]	2D inferences, no 3D training	1.9	4.1	5.8	19.3	28.6	34.8	10.5	20.8	30.6
PointCLIP v2 [88]	4.7	9.5	12.9	63.6	77.9	85.0	42.2	63.3	74.5
ReCon [53]	ShapeNet	1.1	2.7	3.7	61.2	73.9	78.1	42.3	62.5	75.6
CG3D [19]	5.0	9.5	11.6	48.7	60.7	66.5	42.5	57.3	60.8
CLIP2Point [24]	2.7	5.8	7.9	49.5	71.3	81.2	25.5	44.6	59.4
ULIP-PointBERT (Official) [78]	6.2	13.6	17.9	60.4	79.0	84.4	51.5	71.1	80.2
OpenShape-SparseConv	11.6	21.8	27.1	72.9	87.2	93.0	52.7	72.7	83.6
OpenShape-PointBERT	10.8	20.2	25.0	70.3	86.9	91.3	51.3	69.4	78.4
ULIP-PointBERT (Retrained)	Ensembled (no LVIS)	21.4	38.1	46.0	71.4	84.4	89.2	46.0	66.1	76.4
OpenShape-SparseConv	37.0	58.4	66.9	82.6	95.0	97.5	54.9	76.8	87.0
OpenShape-PointBERT	39.1	60.8	68.9	85.3	96.2	97.4	47.2	72.4	84.7
ULIP-PointBERT (Retrained)	Ensembled	26.8	44.8	52.6	75.1	88.1	93.2	51.6	72.5	82.3
OpenShape-SparseConv	43.4	64.8	72.4	83.4	95.6	97.8	56.7	78.9	88.6
OpenShape-PointBERT	46.8	69.1	77.0	84.4	96.5	98.0	52.2	79.7	88.7
4.1 Zero-Shot Shape Classification
We evaluate the zero-shot classification performances of our models on three benchmarks: the traditional ModelNet40 [75] and ScanObjectNN [71], as well as a new benchmark, Objaverse-LVIS [12]. ModelNet40 and ScanObjacetNN consist of 40 and 15 common categories, respectively. Objverse-LVIS is an annotated subset of Objverse [12] and comprises 46,832 shapes among 1,156 LVIS [17] categories. With a much larger base of classes than other benchmarks, Objverse-LVIS presents a challenging long-tailed distribution, making it a better reflection on models' performance in open-world scenarios. We compare OpenShape with existing zero-shot approaches, including PointCLIP [86], PointCLIPv2 [88], ReCon [53], CG3D [19], CLIP2Point [24], and ULIP [78]. Among them, PointCLIP [86] and PointCLIPv2 [88] project point clouds into 2D images and directly utilize 2D CLIP for inference, while other methods leverage the CLIP embedding spaces for alignment and require 3D shapes for training. We report results on these baselines using their released checkpoints. To better analyze the source of our performance gains, we also retrain the baseline ULIP [78] on our ensembled shape dataset, but we use the original texts in the four constituent datasets along with the official codebase without backbone scaling. We train OpenShape and ULIP on three different sets of training shapes: "Ensembled" denotes using all shapes from the four datasets; "Ensembled (no LVIS)" is the same but excludes all shapes from the Objaverse-LVIS subset; "ShapeNet" only includes shapes from the ShapeNet [8] dataset. Note that even when LVIS shapes are included in the training shapes (i.e., the "Ensembled" dataset), their test-time category labels are probably not included in their training texts. Please refer to the supplementary for more training and evaluation details.
Table 2 shows the results. We observe that OpenShape consistently outperforms prior approaches, even when trained only on ShapeNet. When models are trained on our larger-scale ensembled dataset, they receive a significant performance boost. In this case, OpenShape still surpasses retrained ULIP by a significant margin, demonstrating the advantages of our text enrichment, backbone scaling, and other training strategies. Specifically, OpenShape greatly improves the classification accuracy on the long tail categories in Objaverse-LVIS from a dull  $< 10\%$  to  $46.8\%$ , outperforming the retrained ULIP by about 20 points and reaching a decent top-5 accuracy of  $77.0\%$ . These results demonstrate OpenShape's capability to recognize open-world objects effectively. As for ModelNet40, OpenShape achieves a  $85.3\%$  accuracy, surpassing previous methods by a substantial margin of at least 20 percent. OpenShape also achieves impressive top-3 and top-5 accuracies of  $96.5\%$  and  $98.0\%$ . To the best of our knowledge, this is the first time zero-shot methods have matched the performance of a fully-supervised 3D learning method on ModelNet40, where OpenShape outperforms fully-supervised 3D ShapeNets [75] and VoxNet [43]. In addition, on ScanObjectNN, which contains challenging real scans with noise and occlusion, OpenShape exhibits decent sim-to-real transfer capabilities. To contextualize, OpenShape-SparseConv achieves  $56.7\%$  zero-shot accuracy on ScanObjectNN without specific sim-to-real training, which surpasses  $52.7\%$  reported by SKPCov [74], a recent method specially designed for sim-to-real transfer in point cloud classification tasks.
Figure 5: Few-shot linear probing on Objaverse-LVIS [12], ModelNet40 [75], and ScanObjectNN [70]. We report the average performance over 10 random seeds.
Table 3: Ablation study. Top 1 zero-shot accuracies on ModelNet40 [75] and Objverse-LVIS [12] are shown.
Variant	O-LVIS	MNet40
No Objverse shapes	13.9	75.5
Only Objverse shapes	41.6	79.2
No backbone scale up	31.7	78.7
No caption & retrieval	37.0	82.9
No text filtering	41.4	82.9
No point rgb, only xyz	39.6	83.6
No text contras. learning	23.3	67.4
No image contras. learning	41.0	81.0
Full	42.0	83.1
Full + hard mining	43.4	83.4
Figure 7: Ablation study on different text enrichment strategies.
Figure 6: Ablation study on using different ratios of training data.
4.2 Few-Shot Linear Probing
In the literature, linear probing is a common way to assess the representation learning capabilities of a model. To perform linear probing, we gather and freeze the representation vectors from all samples in a dataset. Subsequently, we train a linear classifier using these fixed vectors and few-shot class labels. We evaluate the accuracy of the linear classifier on three benchmarks: Objaverse-LVIS [12], ModelNet40 [75], and ScanObjectNN [71]. Figure 5 summarizes the performance of OpenShape in comparison with ULIP [78] (official release and our retrained versions) and PointCLIPv2 [88]. On the most challenging Objverse-LVIS benchmark, OpenShape outperforms all other methods by a large margin. Notably, zero-shot OpenShape beats few-shot linear probes of other methods. On ModelNet40 and ScanObjectNN, we do not see a large performance margin between OpenShape and retrained ULIP. We hypothesize that for few-shot ModelNet40, the error is dominated by in-category sample bias rather than the representation quality; while for ScanObjectNN, the domain gap plays a major role. Since both OpenShape and retrained ULIP are exposed to the same source domain of training objects, their few-shot out-of-domain generalization performances tend to be similar.
We perform various ablations by training a scaled version of SparseConv [10] on the ensembled dataset and then evaluate it on the Objverse-LVIS [12] and ModelNet40 [75] zero-shot classification benchmarks, unless otherwise specified. The results are shown in Table 3 and Figures 6 and 7.
Data and Model Scaling. We investigate the impact of training data by ablating (1) without or with only Objverse shapes (Tab. 3) and (2) with different ratios of our ensembled dataset (Fig. 6). We observe that training with  $1\%$  of our ensembled dataset (about 8.8k shapes) achieves similar or better zero-shot performance than training without Objverse shapes (about 77.1k shapes), indicating that the diversity of training data is sometimes more crucial than the scale. In addition, we compare the performances between scaled-up and non-scaled-up backbones. From Tab. 3, we demonstrate that model scaling plays an essential role when training on our large-scale ensembled dataset (also Fig. 4).
Figure 8: 3D shape retrieval from image (left, mid) and point cloud (right).
Figure 9: Text-input 3D shape retrieval. In each row, we show input texts on the left and two retrieved shapes for each text on the right. OpenShape embedding encodes a wide range of visual and semantic concepts and enables (a) retrieval of fine-grained subcategories (first two rows), and (b) control of attributes (e.g., color, shape, style) and their combinations (last two rows).
Text Filtering and Enrichment. As shown in Tab. 3, both text filtering and text enrichment are beneficial for performance. We also investigate the specific text enrichment strategies to use for the SparseConv and PointBERT backbones. In Fig. 7, we observe that both image captioning and text retrieval are helpful, and including both yield the best results. Notably, PointBERT improves more than 10 points from text enrichment, highlighting the significance of enhancing text quality.
Other Aspects. We also conduct additional ablation studies on color information, contrastive loss components, and our hard-negative mining strategy in Tab. 3. We observe that OpenShape performs well with only  $xyz$  coordinates as input and no RGB color. While 3D-image contrastive loss is also helpful, we observe that 3D shape-text alignment plays a very essential role for model zero-shot generalization, which necessitates our text filtering and text enrichment strategies that significantly enhance text quality. Lastly, by employing our hard negative mining strategy, OpenShape effectively addresses the issue of unbalanced data distribution, leading to further improvements in performance.
4.4 Cross-Modal Applications
Multi-modal 3D Shape Retrieval. Through OpenShape multi-modal representations, we can index and retrieve 3D shapes from images, texts, or point clouds. In this section, we retrieve 3D shapes from our ensembled dataset by calculating the cosine similarity between input embedding(s) and 3D shape embeddings and performing kNN. As shown in Figure 8, OpenShape is capable of retrieving visually or semantically similar shapes from a single image or point cloud input. OpenShape embeddings encode a wide range of visual and semantic concepts. In Figure 9, we show that OpenShape supports retrieving 3D shapes from detailed text descriptions, which include fine-grained subcategories, attributes, and their combinations. Note that these input texts are typically not present in the raw texts of the retrieved shapes, indicating that OpenShape effectively learns generalizable concepts across shapes. In Figure 1, we provide a demo which takes two 3D shapes as inputs and retrieves the shapes that are simultaneously closest to both inputs. This is achieved by finding  $\arg \max_{i}\min (h_{i}^{P}\cdot h_{a}^{P},h_{i}^{P}\cdot h_{b}^{P})$ , where  $h_a^P$  and  $h_b^P$  denote normalized shape embeddings of the two input shapes. We can see that the retrieved shapes integrate visual or semantic elements in an interesting manner, highlighting the rich concepts and priors encoded in OpenShape embeddings.
Figure 10: (a) Point cloud captioning. (b) Point cloud-conditioned image generation. Our learned 3D shape embeddings can be integrated with off-the-shelf pretrained CLIP-based models (e.g., captioning and image generation models) to support various cross-modal applications.
Captions:
1. The chair in the style of the 1920s.
2. The ladder to the second floor.
3. Goldfish in the sea- photo#.
4. The car of the day.
Shape-Conditioned Multimodal Generation. As OpenShape's 3D shape representations are aligned with CLIP's image and text embedding spaces, they can serve as inputs into other CLIP-based models to facilitate various multimodal generation applications. For example, we show that by feeding our 3D shape embeddings into ClipCap [46], an off-the-shelf image captioning model, along with Stable unCLIP [56], a text-to-image diffusion model, we can perform point cloud captioning and point cloud-conditioned image generation (optional text prompt supported) without extra training or finetuning. Qualitative results are shown in Figure 10. Please refer to the supplementary for more results and details.
We introduce OpenShape, a novel approach for learning scalable and generalizable multi-modal joint representations for 3D shapes. OpenShape representations effectively capture a wide range of semantic and visual concepts, enabling superior capabilities for open-world 3D shape recognition. By aligning OpenShape with CLIP's embedding space, our shape embeddings can be integrated with off-the-shelf CLIP-based models for various cross-modality applications. Moving forward, there are several directions worth further exploration: (a) More 3D data. While we utilized 876k 3D shapes during training, this is still quite limited compared to the 2D counterparts. We hope that our work inspires future investments in more resources to build even more powerful 3D representations. (b) Part-level information. Our current shape representations mainly focus on global semantic and visual features, and it would be beneficial to add more part-level supervision during training. (c) Sim-to-real domain gap. Our model is mainly trained on synthetic data, and it's challenging but crucial to explore explicit designs for reducing the domain gap with real-world shapes.
This work is supported in part by gifts from Qualcomm.
[80] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19313-19322, 2022.
[81] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[82] Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit-Yan Yeung, Zhen Yang, Xiaodan Liang, and Hang Xu. Clip^2: Contrastive language-image-point pretraining from real-world point cloud data. arXiv preprint arXiv:2303.12417, 2023.
[83] Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit-Yan Yeung, Zhen Yang, Xiaodan Liang, and Hang Xu. Clip2: Contrastive language-image-point pretraining from real-world point cloud data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15244-15253, 2023.
[84] Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language understanding. arXiv preprint arXiv:2206.05836, 2022.
[85] Junbo Zhang, Runpei Dong, and Kaisheng Ma. Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip. arXiv preprint arXiv:2303.04748, 2023.
[86] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8552-8562, 2022.
[87] Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of 3d features on any point-cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10252-10263, 2021.
[88] Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyao Zeng, Shanghang Zhang, and Peng Gao. Pointclip v2: Adapting clip for powerful 3d open-world learning. arXiv preprint arXiv:2211.11682, 2022.
A More Examples of Multi-Modal 3D Shape Retrieval
In Figures 11 and 12, we showcase more examples of multi-modal 3D shape retrieval.
Figure 11: Image-input 3D shape retrieval. In each triplet, we present the input image and two 3D shapes retrieved using OpenShape embeddings from the Objaverse [12] dataset. Input images are from unsplash.com.
Figure 12: Point cloud-input 3D shape retrieval. In each triplet, we present the input point cloud and two 3D shapes retrieved using OpenShape embeddings from the Objaverse [12] dataset.
B More Examples of Shape-Conditioned Multimodal Generation
In Figure 13 and Figure 14, we showcase more examples of point cloud captioning and point cloud-conditioned image generation.
C Details on Raw Text Generation and Filtering
C.1 Raw Text Generation
We leverage the metadata from the four datasets to generate the raw texts. Although the original datasets may contain numerous attributes for each shape, we carefully choose the most informative ones to compose the text, ensuring its quality and relevance.
Objaverse: We utilize the name associated with each shape to serve as the text.
ShapeNetCore: For each shape, we generate three types of texts: (a) the name, (b) the category name (with a total of 55 categories), and (c) the concatenation of the sub-category names (with a total of 336 sub-categories), separated by commas.
Figure 13: Point cloud captioning. In each row, we show the input point clouds on the left and the generated captions on the right.
Figure 14: Point cloud-conditioned image generation. Each row shows three examples (input point clouds and generated images).
3DFuture: For each shape, we generate two types of texts: (a) the category, and (b) the concatenation of category, style, theme, and material, separated by commas.
ABO: For each shape, we generate two types of texts: (a) the item_name, and (b) the product_type. In this way, we generate one or more raw texts for each shape.
C.2 Raw Text Filtering
I am analyzing a 3D dataset with various text descriptions for the 3D models. However, many of these texts are inaccurate or uninformative, and therefore, not suitable as descriptions for 3D models. I need your help to identify such incorrect texts. Specifically, if a text primarily consists of irrelevant or uninformative content, such as timestamps, model numbers, incomprehensible descriptions, random filenames (e.g., "my project"), random characters, etc., please respond with "N". If a text contains a clear noun (or noun phrase) that could potentially describe a 3D object, please respond with "Y". You will find a list of texts below, and each line contains a three-digit ID and associated text. For each text, please respond with "Y" or "N", following the ID number (e.g., "001 Y" or "002 N"). Please evaluate all 256 texts.
000 New project (19)	000 N
001 3December - Chemistry	001 Y
002 Fake Brand Soda Can	002 Y
003 Spartan Shild	003 Y
004 Apple3d	004 Y
005 Landmine	005 Y
006 FaunveinB-S	006 N
007 FIGURA 5	007 N
008 Sphero Blue	008 Y
009 Sofa	009 Y
010 Maddox	010 N
011 A3 Complete	011 N
012 Suspension Bridge	012 Y
013 Maung	013 N
014 Captain-americas-shield	014 Y
015 sphorb4	015 N
......	......
We employ GPT-4 [47] to filter out uninformative raw texts. To accomplish this, we divide all the raw texts into batches, each containing 256 entries, and process each batch independently using GPT-4. Here is an example illustrating the prompt we used and the corresponding response generated by GPT-4.
Afterwards, we combine all the responses to create the final filtering results, effectively removing approximately  $30\%$  of the raw texts.
D Details on the Backbone Scaling Experiment
In Figure 4 of the main paper, we investigate the performance and scalability of various backbones when scaling up their model sizes. For this experiment, we employ a default resolution of 10,000 points for input point clouds, a batch size of 200, and conduct the experiment on a single A100 GPU. In general, if instructions are given in the original paper of a backbone, we scale up the model as instructed. Otherwise, we scale up the model by expanding width or depth (i.e., stacking blocks or layers). Specifically, we scale up each backbone as follow:
PointBERT [81] The scaling parameters are shown in Table 4. We scaled PointBERT to 72.1M parameters beyond the 32.3M version reported in Figure 4 of the main paper. However, at this scale, the model dramatically overfits on the training data and performs worse on all benchmarks than the 32.3M version.
SparseConv [10] The smallest version (5.3M parameters) of the model is adapted from the MinkowskiFCNN model by adjusting the width of the final convolution and linear layers. The remaining three models are adaptations of MinkowskiResNet, each varying in the number of basic ResNet blocks used. See Table 5 for the specific scaling parameters.
Table 4: Hyperparameters for scaling up PointBERT [81].
# Parameters	# Layers	Width	# Heads	MLP Dim	# Patches	Patch Embed Dim
5.1M	6	256	4	1024	64	96
13.3M	6	512	8	1024	64	128
32.3M	12	512	8	1536	384	256
72.1M	12	768	12	2304	512	256
Table 5: Hyperparameters for scaling up SparseConv [10].
# Parameters	# Convolution Layers	# Linear Layers
5.3M	7	4
29.0M	18	3
33.7M	26	3
41.3M	42	3
PointNeXt [54] PointNeXt is proposed as a scalable version of PointNet++ [52], and includes S/B/L/XL variants in the original paper. We simply adopt these official configurations.
DGCNN [73] and PointNet [51] For these two backbones without a hierarchical structure, we increase the width of each layer proportionally to scale up to 4xPointNet and 2xDGCNN before we hit the GPU memory limit. As the models operate completely on dense points, it is impractical to use the default 10k-point resolution. We thus reduce the input resolution for the two backbones, resulting in 1k points for DGCNN and 4k points for PointNet.
E Details on Training and Evaluation
Training Details We freeze the CLIP text and image encoders and train the 3D encoder and two projection heads on our ensembled dataset using the cross-modal contrastive loss. We train the model on a single A100 GPU with a batch size of 200. Since we precache the text and image CLIP embeddings of all shapes, the training is greatly accelerated and takes about 300 A100 hours for convergence. We utilize an exponential learning rate schedule, and employ an average test to find the initial learning rate. For 32.3M version of PointBERT, we utilize a learning rate of  $5e - 4$ ; for 72.1M version of PointBERT, we utilize a learning rate of  $4e - 4$ ; and for other models, we utilize a learning rate of  $1e - 3$ . For hard-negative mining, the number of seed shapes  $s$  is set to 40, and the number of neighbors  $m$  is set to 5 per shape, and the threshold  $\delta$  is set to 0.1.
Fine-tuning CLIP Text and Image Encoders? After training OpenShape-PointBERT, we conducted experiments to unfreeze and finetune the CLIP text encoder for a single epoch. However, the results obtained did not demonstrate any noticeable improvement on the benchmarks. Moreover, we observed that finetuning the CLIP text encoder could potentially undermine the generalization capabilities of CLIP and hinder the integration of OpenShape embeddings into existing CLIP-based models. As a result, we choose to freeze the CLIP encoders throughout the entire training process.
Evaluation Details We evaluated all baselines using their publicly released pretrained checkpoints. Additionally, we retrained ULIP [78] on our ensembled training shapes using their official code base and backbone networks. Note that the retrained ULIP model utilized the original raw texts from the four datasets during training (prompt engineering is also applied), rather than our filtered and enriched texts. For ModelNet40 [75], the evaluation is conducted on the test split with 2,468 shapes. Regarding ScanObjectNN [71], we follow ULIP [78] to evaluate on the OBJ_ONLY version, which contains 581 test shapes. For Objaverse-LVIS [12], the input is 10,000 sampled points with point colors. For ModelNet40 [75], the input is 10,000 sampled points without color. For ScanObjectNN [71], we utilize the official 2,048 points without color as input. All methods use the same input during evaluation. The forward inference time on an A100 GPU for a 10,000-point point cloud is approximately 0.9ms for OpenShape-SparseConv and 3.8ms for OpenShape-PointBERT.
F Details on Shape-Conditioned Multimodal Generation
Point Cloud Captioning CLIPCap [46] utilizes a 10-token prefix generated from CLIP image embeddings to enable GPT-2 for captioning. In order to align with the off-the-shelf CLIPCap model, we trained a variant of OpenShape-PointBERT that employs CLIP ViT-B/32 embeddings instead of OpenCLIP ViT-G/14 used in other experiments. Consequently, we directly input the point cloud encoding, without normalization, into CLIPCap for captioning.