SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pre-training setups: The input point clouds are obtained by sampling 1024 points from each raw point cloud. Afterward, each point cloud is partitioned into 64 point patches, with each patch consisting of 32 points. The PointGPT model is pre-trained for 300 epochs using an AdamW optimizer [31] with a batch size of 128, an initial learning rate of 0.001, and a weight decay of 0.05. Additionally, based on our empirical results, cosine learning rate decay [30] is employed.
To demonstrate the performance of our method on different downstream tasks, we conduct experiments involving object classification on real-world and clean object datasets, few-shot learning, and part segmentation. The performance of PointGPT is evaluated using three different model capacities: PointGPT-S, which is pre-trained on the ShapeNet dataset without post-pre-training; and PointGPT-B
Table 1: Classification results on ScanObjectNN and ModelNet40 datasets. All results are expressed as percentages. Specifically, three variants are evaluated on the ScanObjectNN dataset. Additionally, The accuracy obtained on the ModelNet40 dataset is reported for both 1k and 8k points. The symbols  $\bullet$  and  $\bullet$  denote larger pre-training dataset and post-pre-training stage, respectively.
Methods	Reference	ScanObjectNN	ModelNet40
OBJ_BG	OBJ_ONLY	PB_T50_RS	1k P	8k P
Supervised Learning Only
PointNet [40]	CVPR 2017	73.3	79.2	68.0	89.2	90.8
DGCNN [58]	TOG 2019	82.8	86.2	78.1	92.9	-
PointCNN [25]	Neurips 2018	86.1	85.5	78.5	92.2	-
GBNet [45]	TMM 2021	-	-	81.0	93.8	-
MVTN [18]	ICCV 2021	92.6	92.3	82.8	93.8	-
PointMLP [32]	ICLR 2022	-	-	85.4	94.5	-
PointNeXt [43]	Neurips 2022	-	-	87.7	94.0	-
P2P-RN101 [59]	Neurips 2022	-	-	87.4	93.1	-
P2P-HorNet [59]	Neurips 2022	-	-	89.3	94.0	-
with Self-Supervised Representation Learning
Point-BERT [66]	CVPR 2022	87.4	88.1	83.1	93.2	93.8
MaskPoint [26]	ECCV 2022	89.3	88.1	84.3	93.8	-
Point-MAE [39]	ECCV 2022	90.0	88.2	85.2	93.8	94.0
Point-M2AE [68]	Neurips 2022	91.2	88.8	86.4	94.0	-
PointGPT-S	-	91.6	90.0	86.9	94.0	94.2
PointGPT-B	-	95.8	95.2	91.9	94.4	94.6
PointGPT-L	-	97.2	96.6	93.4	94.7	94.9
Methods using cross-modal information and teacher models
ACT [12]	ICLR 2023	93.3	91.9	88.2	93.7	94.0
PointMLP+ULIP [62]	CVPR 2023	-	-	89.4	94.5	94.7
ReCon [42]	ICML 2023	95.4	93.6	91.3	94.5	94.7
and PointGPT-L, which undergo both pre-training and post-pre-training stages on the collected hybrid datasets. The impact of post-pre-training is further investigated and discussed in the appendix.
Object classification on a real-world dataset: The performance of the proposed method on a real-world dataset is an important indicator of its practical applicability. Therefore, the pre-trained models are transferred to the ScanObjectNN dataset [54], which contains approximately 15,000 objects extracted from real-world indoor scans. The experiments are conducted under three different settings, OBJ-BG, OBJ-ONLY, and PB-T50-RS. The results are presented in Table 1, our PointGPT-S, which has similar capacities and training data to previous methods like Point-MAE, outperforms other single-modal SSL methods. Furthermore, even when compared to Recon and ULIP, which utilize cross-modal information and teacher models, our scaled PointGPT-B model achieves superior performance, and PointGPT-L achieves accuracy improvements of at least  $1.8\%$ .
Object classification on a clean objects dataset: The pre-trained models are evaluated on the ModelNet40 dataset [60], which includes 12,311 clean 3D CAD models, covering 40 categories. To conduct fair comparisons, the standard voting method [28] is used during testing, and the input point clouds exclusively contain coordinate information, without additional normal information provided. The experimental results are presented in Table 1, our PointGPT-S surpasses other single-modal SSL methods. Even in comparison with Recon and ULIP, our PointGPT-L achieves superior performance.
Few-shot learning: To intuitively demonstrate the generalization ability of our method, few-shot learning experiments are conducted on the ModelNet40 dataset without the post-pre-training stage. Following the protocols of previous studies [39; 66; 68], the few-shot learning experiments consist of four distinct tests, employing  $w$ -way,  $s$ -shot setting. Specifically,  $w \in \{5, 10\}$  represents the number of randomly selected classes, and  $s \in \{10, 20\}$  denotes the number of randomly sampled objects for each selected class. Each test is conducted with 10 independent trials. The results, as shown in Table 2, indicate that our method outperforms other methods in all tests, particularly in the 10-shot tests. This demonstrates the ability of PointGPT in acquiring knowledge for new tasks, even under the constraints of limited training data.
Table 2: Few-shot classification results on the ModelNet40 dataset. In each experimental setting, 10 independent trials are conducted, and the mean accuracy  $(\%)$  is reported with its standard deviation. Symbol  $\bullet$  denotes larger pre-training dataset, and the post-pre-training stage  $\bullet$  is excluded.
Methods	Reference	5-way	10-way
10-shot	20-shot	10-shot	20-shot
DGCNN [58]	TOG 2019	31.6±2.8	40.8±4.6	19.9±2.1	16.9±1.5
OcCo [56]	ICCV 2021	90.6±2.8	92.5±1.9	82.9±1.3	86.5±2.2
with Self-Supervised Representation Learning
Point-BERT [66]	CVPR 2022	94.6±3.1	96.3±2.7	91.0±5.4	92.7±5.1
MaskPoint [26]	ECCV 2022	95.0±3.7	97.2±1.7	91.4±4.0	93.4±3.5
Point-MAE [39]	ECCV 2022	96.3±2.5	97.8±1.8	92.6±4.1	95.0±3.0
Point-M2AE [68]	Neurips 2022	96.8±1.8	98.3±1.4	92.3±4.5	95.0±3.0
PointGPT-S	-	96.8±2.0	98.6±1.1	92.6±4.6	95.2±3.4
PointGPT-B •	-	97.5±2.0	98.8±1.0	93.5±4.0	95.8±3.0
PointGPT-L •	-	98.0±1.9	99.0±1.0	94.1±3.3	96.1±2.8
Methods using cross-modal information and teacher models
ACT [12]	ICLR 2023	96.8±2.3	98.0±1.4	93.3±4.0	95.6±2.8
ReCon [42]	ICML 2023	97.3±1.9	98.9±1.2	93.3±3.9	95.8±3.0
Table 3: Part segmentation results on the ShapeNetPart dataset. The mean intersection over union (mIoU) is reported across all classes (Cls.) and all instances (Inst.). Symbols  $\bullet$  and  $\bullet$  denote larger pre-training dataset and post-pre-training stage, respectively.
Methods	Reference	Cls. mIoU(%)	Inst. mIoU(%)
PointNet [40]	CVPR 2017	80.4	83.7
PointNet++ [41]	Neurips 2017	81.9	85.1
DGCNN [58]	TOG 2019	82.3	85.2
PointMLP [32]	ICLR 2022	84.6	86.1
with Self-Supervised Representation Learning
PointContrast [61]	ECCV 2020	-	85.1
CrossPoint [2]	CVPR 2022	-	85.5
Point-BERT [66]	CVPR 2022	84.1	85.6
Point-MAE [39]	ECCV 2022	-	86.1
PointGPT-S	-	84.1	86.2
PointGPT-B •	-	84.5	86.5
PointGPT-L •	-	84.8	86.6
Methods using cross-modal information and teacher models
ACT [12]	ICLR 2023	84.7	86.1
ReCon [42]	ICML 2023	84.8	86.4
Part segmentation: The representation learning capability of our method is evaluated on the ShapeNetPart [64] dataset, which consists of 16881 objects across 16 categories. The point clouds are sampled into 2048 points, and the segmentation head [39] concatenates the extracted features from layers  $\frac{1\times td}{3}$ ,  $\frac{2\times td}{3}$ , and  $\frac{3\times td}{3}$  of the extractor transformer blocks, with  $td$  representing the depth of the extractor. Subsequently, average pooling, max pooling, and upsampling are utilized to generate features for each point and an MLP is applied for label prediction. The experimental results, displayed in Table 3, demonstrate the superior performance of our PointGPT-L compared to all other methods.
Comprehensive ablation studies are conducted to investigate the fundamental designs of our PointGPT model. The impacts of these designs are evaluated by reporting the accuracy achieved by fine-tuning the model on the ModelNet40 dataset for object classification. To provide an intuitive representation of the results, ablation studies are conducted using the PointGPT-S model, which is pre-trained on the ShapeNet dataset and directly fine-tuned on the target dataset without post-pre-training.
Table 4: Ablation experiments with PointGPT-S pertaining on the ShapeNet dataset. The fine-tuned accuracy (%) achieved without post-pre-training is reported. Default settings are marked in gray.
(a) Generator depth.
Blocks	Acc.
0	93.85 %
2	94.08 %
4	94.21 %
6	94.24 %
(d) Generation loss.
Loss	Acc.
CD l1	93.66%
CD l2	94.13%
CD l1+l2	94.21%
(b) Generation targets.
Targets	Acc.
Coordinates	94.21 %
FPFH	94.13 %
PointNet	94.31 %
DGCNN	94.35 %
(e) Relative direction prompts.
Case	Acc.
None	93.69%
Absolute position	94.06%
Relative direction	94.21%
(c) Generation during fine-tuning.
Coef.	Acc.
0	94.01%
1	94.15%
3	94.21%
5	94.05%
(f) Dual masking strategy.
Ratio	Acc.	Ratio	Acc.
0	93.68%	5	94.01%
1	93.70%	7	94.21%
3	93.85%	9	93.66%
Table 5: Classification results on ScanObjectNN and ModelNet40 datasets. The symbols  $\bullet$  and  $\bullet$  denote larger pre-training dataset and post-pre-training stage, respectively.
Methods	ScanObjectNN	ModelNet40	Params
OBJ_BG	OBJ_ONLY	PB_T50_RS	1k P	8k P
with different point sorting methods
PointGPT-S w/ KD-Tree	89.0	88.6	81.3	93.2	93.4	19.5M
PointGPT-S w/ Hilbert	89.2	89.0	84.3	93.4	93.6	19.5M
PointGPT-S	91.6	90.0	86.9	94.0	94.2	19.5M
with larger pre-training datasets and post-pre-training stage
Point-MAE-B • [39]	94.2	93.9	90.2	94.2	94.4	120.1M
Point-M2AE-B • [68]	95.2	94.3	91.2	94.3	94.5	77.5M
PointGPT-B •	95.8	95.2	91.9	94.4	94.6	82.1M
Generator: Table 4(a) investigates the effect of varying the generator depth. The results demonstrate that the extractor-generator architecture facilitates the learning of strong semantic representations, particularly when combined with a deep generator, resulting in improved performance overall. However, due to the computational complexity associated with the deep generator, a depth of 4 is selected as the default setting for the generator.
Generation targets: The generation objective is essential for enabling the model to learn the intrinsic characteristics of the given data. Table 4(b) exhibits four distinct generation targets, which can be categorized into two groups: one-stage targets that can be directly obtained, including point coordinates and FPFH [48], and two-stage targets that are extracted by a trained deep network, including PointNet [40] and the DGCNN [58]. The experimental results indicate that the use of handcrafted FPFH features leads to underperformance, which may be attributed to the overfitting of low-level geometric features. The variants with two-stage targets outperform the variant with point coordinate targets. However, the pre-training and inference processes of the teacher model inevitably incur an additional computational cost.
Generation task in the fine-tuning stage: The generation task is included as an auxiliary objective during the fine-tuning stage. Table 4(c) presents the obtained results when varying the coefficient  $\lambda$  of the generation loss in the fine-tuning loss. The results signify that this auxiliary objective serves as a regularization term and improves the generalization ability of supervised models. Furthermore, the results suggest that as the coefficient increases, the accuracy achieved in the classification task exhibits an increasing trend followed by a decreasing trend, reaching its highest value when  $\lambda = 3$ .
Generation loss: Table 4(d) presents the performance of variants using different generation loss functions, including the  $l1$ -form CD loss, the  $l2$ -form CD loss, and the combination of both the  $l1$ - and  $l2$ -forms CD loss. The results demonstrate that the combination of the  $l1$ - and  $l2$ -forms achieves
superior performance. We analyze that the  $l2$ -form is more effective in guiding the network toward convergence and the  $l1$ -form has better sparsity, thus the combination of both forms is more effective.
Relative direction prompts: The effect of utilizing relative direction prompts is analyzed in Table 4(e). The variant utilizing relative direction prompts outperforms the variants using absolute positional encoding and excluding positional encoding. We hypothesize that this improvement stems from the ability of relative direction prompts to prevent the model from overfitting to the patch order, thus enhancing the performance of PointGPT in downstream tasks.
Dual masking strategy: We conduct an analysis on the impact of the dual masking strategy and search for the proper mask ratio, as shown in Table 4(f). Decreasing the mask ratio to 0 corresponds to employing the vanilla masking strategy. The results indicate that both excessive and insufficient masking ratios lead to a decline in performance. The experimental results demonstrate that the dual masking strategy is effective in promoting beneficial representation learning and enhancing the generalization ability of the pre-trained model.
Point sorting methods: We analyze various point sorting methods, conducting ablation studies on the ScanObjectNN and ModelNet40 datasets using PointGPT-S models. Performance comparisons with the widely used KD-tree sorting [15] and Hilbert code sorting [20] methods are summarized in Table 5. The results indicate the substantial superiority of Morton code sorting. This observed effectiveness can be attributed to Morton code sorting's proficient preservation of the adjacency relationship between points, where adjacent points in one-dimensional space are often proximate to each other in three-dimensional space.
Comparisons using larger pre-training dataset and post-pre-training: Experiments are conducted by re-training comparison methods using the ViT-B configuration with collected larger datasets and post-pre-training stage. For a fair comparison, we focus on re-training well-performing single-modal self-supervised methods, Point-MAE, and Point-M2AE. The results, presented in Table 5, illustrate the superior performance of our PointGPT over comparison methods, utilizing comparable training parameters and the same training data.
In this paper, we present PointGPT, a novel approach that extends the GPT concept to point clouds, addressing the challenges associated with disorder properties, information density differences, and gaps between the generation and downstream tasks. Unlike recently proposed self-supervised masked point modeling approaches, our method avoids overall object shape leakage, attaining improved generalization ability. Additionally, we explore a high-capacity model training process and collect hybrid datasets for pre-training and post-pre-training. The effectiveness and strong generalization capabilities of our approach are verified on various tasks, indicating that our PointGPT outperforms other single-modal methods with similar model capacities. Furthermore, our scaled models achieve SOTA performance on various downstream tasks, without the need for cross-modal information and teacher models. Despite the promising performance exhibited by PointGPT, the data and model scales explored for PointGPT remain several orders of magnitude smaller than those in NLP [5; 10] and image processing [67; 29] domains. Our aspiration is that our research can stimulate further exploration in this direction and narrow the gaps between point clouds and these domains.
This work was also supported in part by the National Key RD Program of China under Grant 2022ZD0118101, Natural Science Foundation of China NO. 62202014, and Shenzhen Basic Research Program JCYJ20220813151736001.
[1] Panos Achlioptas, Olga Diamanti, Ioannis Mitlagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In International conference on machine learning, pages 40-49. PMLR, 2018.
[2] Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Rodrigo. Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud