SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: Downstream 3D object classification results on the ScanObjectNN and ModelNet40 datasets. The inference model parameters #P (M), FLOPS #F (G), and overall accuracy (\%) are reported.
Method	#P	#F	ScanObjectNN	ModelNet40
OBJ_BG	OBJ_ONLY	PB_T50_RS	1k P	8k P
Supervised Learning Only
PointNet [66]	3.5	0.5	73.3	79.2	68.0	89.2	90.8
PointNet++ [67]	1.5	1.7	82.3	84.3	77.9	90.7	91.9
DGCNN [89]	1.8	2.4	82.8	86.2	78.1	92.9	-
PointCNN [43]	0.6	-	86.1	85.5	78.5	92.2	-
PCT [27]	2.88	2.3	-	-	-	93.2	-
PointMLP [53]	12.6	31.4	-	-	85.4±0.3	94.5	-
PointNeXt [69]	1.4	3.6	-	-	87.7±0.4	94.0	-
with Self-Supervised Representation Learning
Transformer [84]	22.1	4.8	83.04	84.06	79.11	91.4	91.8
Point-BERT [98]	22.1	4.8	87.43	88.12	83.07	93.2	93.8
Point-MAE [60]	22.1	4.8	90.02	88.29	85.18	93.8	94.0
Point-M2AE [102]	14.8	3.6	91.22	88.81	86.43	94.0	-
VPP w/o vot.	22.1	4.8	92.77	91.56	88.65	93.8	94.0
VPP w/ vot.	22.1	4.8	93.11	91.91	89.28	94.1	94.3
with Pretrained Cross-Modal Teacher Representation Learning
ACT [19]	22.1	4.8	93.29	91.91	88.21	93.7	94.0
I2P-MAE [103]	12.9	3.6	94.15	91.57	90.11	94.1	-
ReCon [68]	43.6	5.3	95.18	93.29	90.63	94.5	94.7
ScanObjectNN [83] and ModelNet [92] are currently the two most challenging 3D object datasets, which are obtained through real-world sampling and synthesis, respectively. We show the evaluation of 3D shape classification in Table 3. It can be seen that: (i) Compared to any supervised or self-supervised method that only uses point clouds for training, VPP achieves the best generalization performance. (ii) Notably, VPP outperforms Point-MAE by  $+4.0\%$  and  $+0.3\%$  accuracy on the most challenging PB_T50_RS and ModelNet40 benchmark.
3.4 Diversity & Specific Text-Conditional Generation
We show the diverse qualitative results of VPP on text-conditional 3D generation in Fig. 8 (a). It can be observed that VPP can generate a broad category of shapes with rich diversity while remaining faithful to the provided text descriptions. Meanwhile, we present the qualitative results of VPP on more specific text-conditional 3D generation in Fig. 8 (b). Notably, VPP can generate high-fidelity shapes that react well to very specific text descriptions, like a cup chair, a round table with four legs and an aircraft during World War II, etc.
(a) Diversity Text-Conditional Generation
(b) Specific Text-Conditional Generation
Figure 8: (a) Diversity qualitative results of VPP on text conditioned 3D generation. (b) Qualitative results of VPP on more specific text-conditioned 3D generation. VPP can generate a broad category of shapes with rich diversity and high fidelity while remaining faithful to the provided text descriptions. Besides, VPP can react to very specific text descriptions, like a cup chair.
Figure 9: Partial inputs point clouds completion results of VPP. Our model is capable of generating diverse completion samples.
3.5 Partial Completion
We present the point cloud partial completion experiments in Fig. 9. By employing a block mask on the original point clouds, VPP can generate partial samples, which illustrates the partial completion capability of VPP. Besides, the generated samples exhibit diversity, further demonstrating the diverse performance of VPP.
Figure 10: ShapeNet training data retrieval ablation experiment.
3.6 ShapeNet Retrieval Experiment
We conduct the retrieval evaluation on samples generated by VPP from the ShapeNet dataset. The results are shown in Fig. 10. It can be observed that there are no completely identical samples, proving the great generation ability of VPP is not caused by overfitting the ShapeNet dataset. The VPP results more likely come from the understanding and integration of the learned shape knowledge.
Conditional 3D generation with 2D images or text has witnessed rapid development in recent years. One line of works focuses on solving the ill-posed image-conditioned 3D generation (i.e., single-view reconstruction), where impressive results have been achieved [10, 23, 57, 88]. By introducing a text-shape paired dataset based on ShapeNet [4] that includes chair and table categories, [7] pioneered another line of single-category text-conditioned shape generation. Facilitated with this human-crawled 3D data, ShapeCrafter [24] further enables text-conditioned shape edition in a recursive fashion. To leverage easily rendered 2D images rather than human-crafted languages, several works propose to train rendered image-to-shape generation [9, 51, 76, 77] which enables text-conditioned generation through pretrained large vision-language (VL) models, i.e., CLIP [70]. Besides rendered images, Point-E [59] and Shape-E [38] propose to use real-world text-image-shape triplets from a large-scale in-house dataset, where images are used as the representation bridge.
However, 3D data is significantly lacking and expensive to collect [19]. DreamFields [37] first achieves text-only training by using NeRF [56], where rendered 2D images are weakly supervised to align with text inputs using CLIP. Following this direction, DreamFusion [65] incorporates the distillation loss of a diffusion model [74]. Dream3D [94] improves the knowledge prior by providing the prompt shapes initialization. Though impressive results are obtained, these approaches also introduce significant time costs because of the case-by-case NeRF training during inference. More recently, TAPS3D aligns rendered images and text prompts based on DMTet [79] to enable text-conditional generation, but its generalizability is limited to only four categories. 3DGen [28] achieves high-quality generation through the utilization of Triplane-based diffusion models [3, 80] and the Objverse dataset [14]. Zero-1-to-3 [46] employs relative camera viewpoints as conditions for diffusion modeling. Based on this, One-2-3-45 [45] achieves rapid open-vocabulary mesh generation through multi-view synthesis. Fantasy3D [8] leverages DMTet to employ differentiable rendering of SDF as a substitute for NeRF, enabling a more direct mesh generation. SJC [87] and ProlificDreamer [90] achieve astonishing results by proposing the variational diffusion distillation.
In this paper, we present VPP, a novel Voxel-Point Progressive Representation method that achieves efficient and universal conditional 3D generation. We explore the suitability of different 3D geometric representations and design a VQGAN module tailored for 3D generation. Notably, VPP is capable of generating high-quality 8K point clouds in less than 0.2 seconds using a single RTX 2080Ti GPU. Extensive experiments demonstrate that VPP exhibits excellent performance in conditional generation, editing, completion, and pretraining.
This research was supported by the National Key R&D Program of China (2022YFB2804103), the Key Research and Development Program of Shaanxi (2021ZDLGY01-05), the National Natural Science Foundation of China (31970972), and the Institute for Interdisciplinary Information Core Technology (IIISCT).
[96] Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Trans. Graph., 35(6):1-12, 2016. 17
[97] Li Yi, Hao Su, Xingwen Guo, and Leonidas J. Guibas. Syncspeccnn: Synchronized spectral CNN for 3d shape segmentation. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2017. 17
[98] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2022. 3, 8, 17, 19
[99] Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, and Karsten Kreis. LION: latent point diffusion models for 3d shape generation. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2022. 2
[100] Junbo Zhang, Runpei Dong, and Kaisheng Ma. Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pp. 2048–2059, October 2023. 17
[101] Junbo Zhang, Guofan Fan, Guanghan Wang, Zhengyuan Su, Kaisheng Ma, and Li Yi. Language-assisted 3d feature learning for semantic scene understanding. In AAAI Conf. Artif. Intell. (AAAI), 2023. 17
[102] Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li. Point-m2AE: Multi-scale masked autoencoders for hierarchical point cloud pre-training. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2022. 8, 19
[103] Renrui Zhang, Liuhui Wang, Yu Qiao, Peng Gao, and Hongsheng Li. Learning 3d representations from 2d pre-trained models via image-to-point masked autoencoders. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2023. 8, 19
[104] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan L. Yuille, and Tao Kong. ibot: Image BERT pre-training with online tokenizer. In Int. Conf. Learn. Represent. (ICLR), 2022. 4
3D Representation Learning includes point-based [66, 67], voxel-based [55], and multiview-based methods [29, 81], etc. Due to the sparse but geometry-informative representation, point-based methods [20, 69] have become mainstream approaches in object classification [83, 92]. Voxel-based CNN methods [15, 97] provide dense representation and translation invariance, achieving outstanding performance in object detection [12] and segmentation [1, 96]. Furthermore, due to the vigorous development of attention mechanisms [84], 3D Transformers [20, 49, 54] have also brought about effective representations for downstream tasks. Recently, 3D self-supervised representation learning has been widely studied. PointContrast [93] leverages contrastive learning across different views to acquire discriminative 3D scene representations. Point-BERT [98] and Point-MAE [60] first introduce masked modeling pretraining into 3D. ACT [19] pioneers cross-modal geometry understanding via 2D/language foundation models. RECON [68] further proposes to unify generative and contrastive learning. Facilitated by foundation vision-language models like CLIP [70], another line of works are proposed towards open-world 3D representation learning [16, 22, 64, 95, 100, 101].
Training Details We use ShapeNetCore from ShapeNet [4] as the training dataset. ShapeNet is a clean set of 3D CAD models with rich annotations, including 51K unique 3D models from 55 common object categories. For the acquisition of multi-modal data, we follow ReCon [68] for multi-view rendering and utilize BLIP [41] based on rendered images to obtain textual data. Table 4 shows the training hyperparameters and model architecture information of each part of VPP.
Table 4: Training recipes for 3D VQGAN, Voxel Generator, Grid Smoother and Point Upsampler.
Config	3D VQGAN	Voxel Generator	Grid Smoother	Point Upsampler
Training Parameters
Optimizer	Adam	AdamW	AdamW	AdamW
Learning rate	1e-4	1e-3	1e-3	1e-3
Weight decay	1e-4	5e-2	5e-2	5e-2
Training epochs	100	100	100	300
Warmup epochs	-	5	5	10
Learning rate scheduler	cosine	cosine	cosine	cosine
Batch size	32	128	128	128
Drop path rate	-	0.1	0.1	0.1
Input point size	8192	8192	8192	1024
Model Architecture
Backbone	CNN	Transformer	Transformer	Transformer
Layers	6	12	4	6
Hidden size	256	256	64	384
Heads	-	6	4	6
Voxel resolution	24/32	24/32	24/32	24/32
Point patch size	-	-	-	32
GPU device	NVIDIA A100	NVIDIA A100	NVIDIA A100	NVIDIA A100
Downstream Tasks Details Following Point-E [59], we use pre-trained PointNet++ as a classifier in all ACC, FID, and IS evaluations to extract the features and calculate the accuracy of generated point clouds. In point cloud generation and editing, we employ 8 or 4 steps for a parallel generation. The generation task utilizes initial voxel codebooks composed entirely of [MASK] tokens. In editing, we extract VQGAN features from the original point cloud to initialize the voxel codebooks. As for the transfer classification task on ScanObjectNN [83] and ModelNet40 [92], we fully follow the previous work [19, 60] configuration and trained 300 epochs with the AdamW optimizer, and used the voting strategy in testing.
Figure 11: The training overview of 3D VQGAN. We introduce the occupancy rate loss to have a better reconstruction of the voxel.
Figure 12: Reconstruction results of 3D VQGAN. Our model is capable of flawlessly reconstructing the original voxel, and can even rectify some noise.
B.2 Implementation Details of 3D VQGAN
We show the detailed training overview of 3D VQGAN in Fig. 11. Following the training recipe of VQGAN [21], we use L1 loss to supervise the reconstruction of the voxel and feed the reconstructed voxel into the discriminator to ensure the generated authenticity by GAN loss. Besides the L1 loss and GAN loss, we also introduce the occupancy rate loss to make the occupancy rate of the reconstructed voxel grid similar to the ground truth so as to obtain a better reconstruction of the voxel. Fig. 12 illustrates the reconstructed results of 3D VQGAN, showcasing its impressive capabilities in the domain of 3D voxel reconstruction. As depicted in the figure, the model exhibits remarkable noise rectification capabilities. It not only reconstructs the voxels faithfully but also manages to rectify certain imperfections and artifacts present in the input data.
We conduct more experiments to further demonstrate the generation quality and universality of VPP. Including diversity & specific text-conditional generation, few-shot transfer classification, and ablation studies.
C.1 Novel Categories Point Clouds Generation
Besides the generation of common categories, we also conduct novel categories generation experiments to show the generalization performance of VPP in Fig. 13. Based on the learning of common categories, VPP can generalize to generate more novel category shapes, which are visually reasonable, such as a car boat.
(a) common categories
Figure 13: Novel categories point clouds generation.
(b) novel categories
Table 5: Downstream few-shot results on ModelNet40. Overall accuracy  $(\%)$  w/o voting is reported.
Method	5-way	10-way
10-shot	20-shot	10-shot	20-shot
Supervised Learning Only
DGCNN [89]	31.6 ± 2.8	40.8 ± 4.6	19.9 ± 2.1	16.9 ± 1.5
OcCo [86]	90.6 ± 2.8	92.5 ± 1.9	82.9 ± 1.3	86.5 ± 2.2
with Self-Supervised Representation Learning
Transformer [84]	87.8 ± 5.2	93.3 ± 4.3	84.6 ± 5.5	89.4 ± 6.3
OcCo [86]	94.0 ± 3.6	95.9 ± 2.3	89.4 ± 5.1	92.4 ± 4.6
Point-BERT [98]	94.6 ± 3.1	96.3 ± 2.7	91.0 ± 5.4	92.7 ± 5.1
MaskPoint [44]	95.0 ± 3.7	97.2 ± 1.7	91.4 ± 4.0	93.4 ± 3.5
Point-MAE [60]	96.3 ± 2.5	97.8 ± 1.8	92.6 ± 4.1	95.0 ± 3.0
Point-M2AE [102]	96.8 ± 1.8	98.3 ± 1.4	92.3 ± 4.5	95.0 ± 3.0
VPP (ours)	96.9 ± 1.9	98.3 ± 1.5	93.0 ± 4.0	95.4 ± 3.1
with Pretrained Cross-Modal Teacher Representation Learning
ACT [19]	96.8 ± 2.3	98.0 ± 1.4	93.3 ± 4.0	95.6 ± 2.8
I2P-MAE [103]	97.0 ± 1.8	98.3 ± 1.3	92.6 ± 5.0	95.5 ± 3.0
ReCon [68]	97.3 ± 1.9	98.9 ± 1.2	93.3 ± 3.9	95.8 ± 3.0
C.2 Few-Shot Transfer Classification
We conduct few-shot experiments on the ModelNet40 [92] dataset, and the results are shown in Table 5. In the self-supervised benchmark without the use of additional modality data, VPP achieves excellent performance compared to previous works.
Training Hyper Parameter Fig. 15 (a-b) shows the ablation study of the image-text features ratio and Gaussian noise strength. It can be observed that either too large or too small text-image feature ratio and noise are not conducive to the quality and diversity of 3D generation.
Inference Parameters To explore the parameter dependencies of the Mask Voxel Transformer in the inference process, we study the effects of temperature and iteration steps. The results are shown in Fig. 15 (c-d). It can be seen that moderate temperature achieves optimal generation classification accuracy. Higher temperature promotes model diversity, while with the increase of iteration steps, the model's FID initially decreases and then increases.
Backbone Choice Fig. 14 shows the selection of the CLIP model backbones during VPP training. Both ViT-B/32 and ViT/L14 achieve excellent accuracy and diversity.
Figure 14: Ablation study on CLIP backbone choices.
(a) Image-Text Feature Ratio
(b) Gaussian Noise Strength
Figure 15: Ablation study on image-text feature ratio, Gaussian noise strength, inference temperature, and inference steps.
(c) Central Radiative Temperature
(d) Iteration Steps
D Limitations & Future Works
Despite the substantial advantages of our model in terms of generation speed and applicability, there exists a considerable gap in generation quality compared to NeRF-based models [37, 65, 87]. To address this limitation, we intend to explore the utilization of larger models and more extensive datasets for training, e.g., Objaverse [13, 14]. Furthermore, multi-modal large language models with combined comprehension and generation abilities [18] have advanced rapidly. Incorporating VPP's 3D generation capabilities into such models represents a potential avenue for future research.
VPP enables the rapid generation of high-quality 3D models in a fraction of a second. This has wide-ranging applications in fields such as gaming, virtual reality, augmented reality, and digital media production. These advances can lead to more immersive and visually stunning user experiences in entertainment and educational content. Furthermore, as a generative model, VPP may produce deceptive and malicious content and potentially impact associated employment opportunities.