SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Guangyan Chen<sup>1</sup> Meiling Wang<sup>1</sup> Yi Yang<sup>1</sup> Kai Yu<sup>1</sup> Li Yuan<sup>2*</sup> Yufeng Yue<sup>1*</sup>
$^{1}$  Beijing Institute of Technology  $^{2}$  Peking University
Large language models (LLMs) based on the generative pre-training transformer (GPT) [46] have demonstrated remarkable effectiveness across a diverse range of downstream tasks. Inspired by the advancements of the GPT, we present PointGPT, a novel approach that extends the concept of GPT to point clouds, addressing the challenges associated with disorder properties, low information density, and task gaps. Specifically, a point cloud auto-regressive generation task is proposed to pretrain transformer models. Our method partitions the input point cloud into multiple point patches and arranges them in an ordered sequence based on their spatial proximity. Then, an extractor-generator based transformer decoder [27], with a dual masking strategy, learns latent representations conditioned on the preceding point patches, aiming to predict the next one in an auto-regressive manner. To explore scalability and enhance performance, a larger pre-training dataset is collected. Additionally, a subsequent post-pre-training stage is introduced, incorporating a labeled hybrid dataset. Our scalable approach allows for learning high-capacity models that generalize well, achieving state-of-the-art performance on various downstream tasks. In particular, our approach achieves classification accuracies of  $94.9\%$  on the ModelNet40 dataset and  $93.4\%$  on the ScanObjectNN dataset, outperforming all other transformer models. Furthermore, our method also attains new state-of-the-art accuracies on all four few-shot learning benchmarks. Codes are available at https://github.com/CGuangyan-BIT/PointGPT.
Point clouds are becoming widely adopted data structures in various application areas, such as autonomous driving and robotics, emphasizing the importance of acquiring informative and comprehensive 3D representations. However, current 3D-centric approaches [9; 40; 41; 58; 32] typically necessitate fully-supervised training from scratch, which entails labor-intensive human annotations. In natural language processing (NLP) and image analysis domains, self-supervised learning (SSL) [11; 19; 47; 5; 46] has emerged as a promising approach for acquiring latent representations without relying on annotations. Among these methods, the generative pre-training transformer (GPT) [46] has been particularly effective at learning representative features [7], where the task is to predict data in an auto-regressive manner. Due to its remarkable performance, we naturally ask the question: can the GPT be adapted to point clouds and serve as an effective 3D representation learner?
To answer this question, we exploit the GPT scheme for point cloud understanding. However, it is challenging to employ GPT on point clouds due to the following reasons: (I) Disorder properties.
*Yufeng Yue and Li Yuan contributed equally to this study as co-corresponding authors. This work was supported by the National Natural Science Foundation of China under Grant No. NSFC 62003039, 62233002, 92370203, 61973034, the National Key RD Program of China (2022ZD0118), the CAST program under Grant No. YESS20200126.
(a) Explanation
(b) Auto-regressive Generation
Figure 1: Illustration of our PointGPT. The transformer decoder is pre-trained to predict point patches in an auto-regressive manner. Such a design enables our method to predict patches without dedicated specifications and avoids positional information leakage, leading to improved generalization ability.
In contrast with the sequential arrangement of words in a sentence, a point cloud is a structure that lacks inherent order. To address this issue, point patches are arranged based on a geometric ordering, namely the Morton-order curve [36], which introduces sequential properties and preserves the local structures. (II) Information density differences. Languages are characterized by high information richness, therefore, the auto-regressive prediction task requires advanced language understanding. On the contrary, point clouds are natural signals with heavy redundancy, thereby the prediction task can be accomplished even without holistic comprehension. To address this disparity, a dual masking strategy is proposed, which additionally masks attending tokens for each token. This strategy effectively reduces redundancy and provides a challenging task that demands comprehensive understanding. (III) Gaps between generation and downstream tasks. Even though the model with a dual masking strategy exhibits sophisticated comprehension, the generation task primarily involves predicting individual points, which may result in the learned latent representations with a lower semantic level than downstream tasks. To mitigate this challenge, an extractor-generator architecture is introduced to the transformer decoder [27], such that the generation task is facilitated through the generator, thus enhancing the semantic level of the latent representations learned by the extractor.
Based on the above analysis, we propose a novel SSL framework for point clouds, called PointGPT. Specifically, our method partitions the input point cloud into multiple irregular point patches, which are subsequently organized using the Morton curve. Then, an extractor-generator based transformer decoder, with a dual masking strategy, processes the point patch sequences to learn latent representations conditioned on the unmasked preceding contents, and predicts the next point patches in an auto-regressive manner. Unlike recently developed masked point modeling approaches [66; 39; 68] that rely on positional information to specify reconstruction regions, resulting in the leakage of the overall object shape, our concise design, as illustrated in Figure 1, effectively circumvents the leakage of positional information and yields an enhanced generalization ability. Consequently, our PointGPT surpasses other single-modal SSL methods with comparable model sizes.
Inspired by the promising performance exhibited by our PointGPT, we endeavor to investigate its scaling property and push its performance limit further. However, a significant challenge arises due to the limited scale of the existing public point cloud datasets compared to NLP and images. This dataset size disparity introduces potential overfitting concerns. To alleviate this and fully unleash the power of PointGPT, a larger pre-training dataset is collected by mixing various point cloud datasets, such as ShapeNet [6] and S3DIS [3]. Moreover, a subsequent post-pre-training stage [57] is introduced, which involves performing supervised learning on the collected labeled dataset, enabling PointGPT to incorporate semantic information from multiple sources. Within this framework, our scaled models achieve state-of-the-art (SOTA) performance on various downstream tasks. In object classification tasks, our PointGPT achieves  $94.9\%$  accuracy on the ModelNet40 dataset and  $93.4\%$  accuracy on the ScanObjectNN dataset, outperforming all other transformer models. In few-shot learning tasks, our method also attains new SOTA performance on all four benchmarks.
Our main contributions can be summarized as follows: (I) A novel GPT scheme, termed PointGPT, is proposed for point cloud SSL. PointGPT leverages a point cloud auto-regressive generation task while mitigating positional information leakage, outperforming other single-modal SSL methods. (II) A dual masking strategy is proposed to create an effective generation task, and an extractor-generator transformer architecture is introduced to enhance the semantic level of the learned representations. These designs boost the performance of PointGPT on downstream tasks. (III) A post-pre-training stage is introduced, and larger datasets are collected to facilitate the training of high-capacity models. With PointGPT, our scaled models achieve SOTA performance on various downstream tasks.
2.1 Self-supervised Learning for NLP and Image Processing
Self-supervised learning has attracted significant attention in recent years, especially in the fields of NLP and image processing, owing to its ability to learn useful representations without labeled data. The core idea of SSL is to design a pretext task to learn the distribution of the given data, obtaining beneficial features for the subsequent supervised modeling tasks [23; 13]. Contrastive learning [8; 16; 65; 38; 34; 44; 51] has been a popular discriminative self-supervised approach in both NLP and image processing, with the goal of grouping similar samples closer and diverse samples further apart. However, generative SSL methods [4; 19; 7; 11; 49; 46] have recently achieved more competitive performance. BERT [11] utilizes a bidirectional transformer to process the randomly masked text and reconstruct the original context. ELMo [49] adopts bidirectional LSTM [21] and generates subsequent words from left to right given representations of the previous contents. The GPT [46] also utilizes the auto-regressive prediction approach, but it employs a unidirectional transformer architecture, and the model is fine-tuned by updating all pre-trained parameters. In the computer vision field, BEiT [4] and MAE [19] randomly mask input patches, and pre-train models to recover the masked patches in the pixel space. Image-GPT [7] trains a sequence transformer to auto-regressively predict pixels without incorporating knowledge concerning the 2D input structure, exhibiting promising representation learning capabilities after pre-training.
2.2 Self-supervised Learning for Point Cloud
The success of SSL in NLP and image processing has motivated researchers to develop SSL frameworks for point cloud representation learning. Among these methods, the contrastive methods [61; 69; 37; 22; 63] have been extensively investigated. DepthContrast [69] constructs augmented depth maps and performs an instance discrimination task for the extracted global features. Similarly, MVIF [22] introduces cross-modal and cross-view invariance constraints to achieve self-supervised modal- and view-invariant feature learning. Another line of work [42; 12; 62] is proposed to integrate cross-modal information and leverage knowledge transferred from language or image models for 3D learning. ACT [12] employs cross-modal auto-encoders as teacher models to acquire knowledge from other modalities. Different from these methods, our work attempts to learn the intrinsic properties of point clouds without relying on cross-modal information and teacher models. Most relevant to our work are generative methods [24; 1; 50; 39; 66; 68; 33], especially recently proposed masked point modeling methods [39; 66; 68; 33]. Point-MAE extends the MAE by randomly masking point patches and reconstructing masked regions. Point-M2AE additionally utilizes a hierarchical transformer architecture and designs a corresponding masking strategy. However, masked point modeling methods still suffer from overall object shape leakage, which limits their ability to effectively generalize to downstream tasks. In this paper, we exploit the auto-regressive pre-training for point clouds and address unique challenges associated with the properties of point clouds. Our concise design avoids positional information leakage, thereby enhancing the generalization ability.
3 PointGPT
Given a point cloud  $\mathbf{X} = \{x_{1}, x_{2}, \dots, x_{M}\} \subseteq \mathbb{R}^{3}$ , the overall pipeline of PointGPT during pre-training is illustrated in Fig. 2. The point cloud sequencer module is utilized to construct an ordered sequence of point patches. This is achieved by dividing the point cloud into irregular patches and arranging them in Morton order. The resulting sequence is then fed into the extractor to learn latent representations, and the generator predicts the subsequent point patches in an auto-regressive manner. After the
Figure 2: Overall architecture of our PointGPT. (a) The input point cloud is divided into multiple point patches, which are then sorted and arranged in an ordered sequence. (b) An extractor-generator based transformer decoder is employed along with a dual masking strategy for the auto-regressively prediction of the point patches. In this example, the additional mask of the dual masking strategy is applied to the same group of random tokens for better illustration purposes.
pre-training stage, the generator is discarded, and the extractor without the use of a dual masking strategy, leverages the learned latent representations for downstream tasks.
3.1 Point Cloud Sequencer
In the field of NLP, the GPT approach benefits from an easily accessible language vocabulary and the inherent ordered properties of words. In contrast, the point cloud domain lacks a predefined vocabulary, and a point cloud is a sparse structure that exhibits a characteristic of the disorder. To overcome these challenges and obtain an ordered point cloud sequence with each component unit capturing rich geometric information, a three-stage process consisting of point patch partitioning, sorting, and embedding is employed.
Point patch partitioning: Taking the inherent sparsity and disorder properties of point clouds into account, the input point clouds are processed by farthest point sampling (FPS) and the K-nearest neighbors (KNN) algorithms to obtain center points and point patches. Given a point cloud  $\mathbf{X}$  with  $M$  points, we initially sample  $n$  center points  $\mathbf{C}$  using FPS. Then, the KNN algorithm is utilized to construct  $n$  point patches  $\mathbf{P}$  by selecting the  $k$  nearest points from  $\mathbf{X}$  for each center point. In summary, the partitioning procedure is formulated as:
$$
\boldsymbol {C} = \operatorname {F P S} (\boldsymbol {X}), \quad \boldsymbol {C} \in \mathbb {R} ^ {n \times 3};
$$
$$
\boldsymbol {P} = \operatorname {K N N} (\boldsymbol {C}, \boldsymbol {X}), \quad \boldsymbol {P} \in \mathbb {R} ^ {n \times k \times 3}. \tag {1}
$$
Sorting: To address the inherent disorder properties of point clouds, the obtained point patches are organized into a coherent sequence based on their center points. Concretely, the coordinates of the center points are encoded into one-dimensional space using Morton code [36], followed by sorting to determine the order  $\mathcal{O}$  of these center points. The point patches are then arranged in the same order. The sorted center points  $C^s$  and sorted point patches  $P^s$  are obtained as follows:
$$
\mathcal {O} = \operatorname {a r g m a x} (\operatorname {M o r t o n C o d e} (\boldsymbol {C})), \quad \mathcal {O} \in \mathbb {R} ^ {n \times 1};
$$
$$
\boldsymbol {C} ^ {s}, \boldsymbol {P} ^ {s} = \boldsymbol {C} [ \mathcal {O} ], \boldsymbol {P} [ \mathcal {O} ], \quad \boldsymbol {C} ^ {s} \in \mathbb {R} ^ {n \times 3}, \boldsymbol {P} ^ {s} \in \mathbb {R} ^ {n \times k \times 3}. \tag {2}
$$
**Embedding:** Following Point-MAE [39], a PointNet [40] network is employed to extract rich geometric information for each point patch. To facilitate training convergence, the normalized coordinates of each point are utilized with respect to its center point. Specifically, the sorted point patches  $P^s$  are embedded into  $D$ -dimensional tokens  $T$  as follows:
$$
\boldsymbol {T} = \operatorname {P o i n t N e t} \left(\boldsymbol {P} ^ {s}\right), \quad \boldsymbol {T} \in \mathbb {R} ^ {n \times D}. \tag {3}
$$
3.2 Transformer Decoder with a Dual Masking Strategy
A straightforward extension of the GPT [46] to point clouds can be achieved by utilizing the vanilla transformer decoder to auto-regressively predict point patches, followed by fine-tuning all pre-trained parameters for downstream tasks. Nevertheless, this approach suffers from low-level semantics due to the limited information density of point clouds and the gaps between generation and downstream tasks. To address this issue, a dual masking strategy is proposed to facilitate comprehensive understanding of point clouds. Additionally, an extractor-generator transformer architecture is introduced, where the generator is more specialized for the generation task and is discarded after pre-training, enhancing the semantic level of the latent representations that are learned by the extractor.
Dual masking strategy: The vanilla masking strategy in the transformer decoder enables each token to receive information from all the preceding point tokens. To further encourage the learning of useful representations, the dual masking strategy is proposed, which additionally masks a proportion of the attending preceding tokens of each token during pre-training. The resulting dual mask  $M^{d}$  is illustrated in Fig. 2(b), the self-attention process with a dual masking strategy can be represented as:
$$
\operatorname {S e l f A t t e n t i o n} (\boldsymbol {T}) = \operatorname {s o f t m a x} \left(\frac {\boldsymbol {Q} \boldsymbol {K} ^ {T}}{\sqrt {D}} - (1 - \boldsymbol {M} ^ {d}) \cdot \infty\right) \boldsymbol {V}. \tag {4}
$$
where  $Q, K, V$  are  $\pmb{T}$  encoded with different weights for the  $D$  channels. The masked locations in  $\pmb{M}^{d}$  are set to 0, while the unmasked locations are set to 1.
Extractor-generator: Our extractor is composed entirely of transformer decoder blocks with a dual masking strategy, obtaining latent representations  $\mathcal{T}$ , where each point token only attends to the unmasked preceding tokens. Considering that point patches are represented in normalized coordinates, and global structures of point clouds are essential for point cloud understanding, sinusoidal positional encodings [55] (PE) are utilized to map the coordinates of the sorted center points  $C^s$  to the absolute positional encoding (APE). Positional encodings are added to every transformer block to provide location information and incorporate global structural information.
The generator architecture is similar to the extractor architecture but contains fewer transformer blocks. It takes the extracted tokens  $\mathcal{T}$  as input and generates point tokens  $T^g$  for the following prediction head. However, the patch order may be affected by the center point sampling process, inducing ambiguity when predicting the subsequent patches. This hinders the model from effectively learning meaningful point cloud representations. To address this issue, the directions relative to the subsequent point patches are provided in the generator, serving as prompts without revealing the locations of the masked patches and the overall object shapes of point clouds. The relative direction prompts  $RDP$  are formulated as:
$$
R D P _ {i} = \mathrm {P E} \left(\left(C _ {i + 1} ^ {s} - C _ {i} ^ {s}\right) / \left\| C _ {i + 1} ^ {s} - C _ {i} ^ {s} \right\| _ {2}\right), i \in \{1, \dots , n ^ {\prime} \}, R D P \in \mathbb {R} ^ {n ^ {\prime} \times D}, \tag {5}
$$
where  $n' = n - 1$ . In summary, the procedure in the extractor-generator architecture is formulated as:
$$
\mathcal {T} = \operatorname {E x t r a c t o r} (\boldsymbol {T} + A P E), \quad \mathcal {T} \in \mathbb {R} ^ {n \times D}; \tag {6}
$$
$$
\boldsymbol {T} ^ {g} = \operatorname {G e n e r a t o r} \left(\mathcal {T} _ {1: n ^ {\prime}} + R D P\right), \quad \boldsymbol {T} ^ {g} \in \mathbb {R} ^ {n ^ {\prime} \times D}.
$$
Prediction head. The prediction head is utilized to predict the subsequent point patches in the coordinate space. It consists of a two-layer MLP with two fully connected (FC) layers and rectified linear unit (ReLU) activation. The prediction head projects tokens  $T^g$  to vectors, where the number of output channels equals the total number of coordinates in a patch. Then, these vectors are reshaped to construct the predicted point patches  $P^{pd}$ :
$$
\boldsymbol {P} ^ {p d} = \operatorname {R e s h a p e} (\mathrm {M L P} (\boldsymbol {T} ^ {g}), \quad \boldsymbol {P} ^ {p d} \in \mathbb {R} ^ {n ^ {\prime} \times k \times 3}. \tag {7}
$$
3.3 Generation Target
The generation target for each point patch is to predict the coordinates of the points within the subsequent point patches. Given the predicted point patches  $P^{pd}$ , as well as the ground-truth point patches  $P^{gt}$ , which correspond to the last  $n'$  patches among the sorted point patches  $P^s$ , the generation loss  $\mathcal{L}^g$  is formulated using the  $l_1$ -form and  $l_2$ -form of the Chamfer distance (CD) [14], denoted as  $\mathcal{L}_1^g$  and  $\mathcal{L}_2^g$ , respectively. Specifically, the generation loss is computed as  $\mathcal{L}^g = \mathcal{L}_1^g + \mathcal{L}_2^g$ .
The  $l_{n}$ -form CD loss  $\mathcal{L}_n^g$ , with  $n \in \{1, 2\}$ , is defined as:
$$
\mathcal {L} _ {n} ^ {g} = \frac {1}{| \boldsymbol {P} ^ {p d} |} \sum_ {a \in \boldsymbol {P} ^ {p d}} \min  _ {b \in \boldsymbol {P} ^ {g t}} \| a - b \| _ {n} ^ {n} + \frac {1}{| \boldsymbol {P} ^ {g t} |} \sum_ {b \in \boldsymbol {P} ^ {g t}} \min  _ {a \in \boldsymbol {P} ^ {p d}} \| a - b \| _ {n} ^ {n}, \tag {8}
$$
where  $|\pmb{P}|$  is the cardinality of the set  $\pmb{P}$  and  $\|a - b\|_n$  represents the  $L_n$  distance between  $a$  and  $b$ .
We additionally find that incorporating the generation task into the fine-tuning process as an auxiliary objective can accelerate training convergence and improve the generalization ability of supervised models. This approach yields enhanced performance on downstream tasks, which is in line with the GPT [46]. Specifically, we optimize the following objective during the fine-tuning stage:  $\mathcal{L}^f = \mathcal{L}^d + \lambda \times \mathcal{L}^g$ , where  $\mathcal{L}^d$  represents the loss for the downstream task,  $\mathcal{L}^g$  represents the generation loss as previously defined, and the parameter  $\lambda$  balances the contribution of each loss term.
3.4 Post-Pre-training
Current point cloud SSL methods directly fine-tune pre-trained models on the target dataset, which may result in potential overfitting due to the limited semantic supervision information [57]. To alleviate this issue and facilitate training of high-capacity models, we adopt the intermediate fine-tuning strategy [57; 4; 29] and introduce a post-pre-training stage for PointGPT. In this stage, a labeled hybrid dataset is leveraged (Sec. 4.1), which collects and aligns multiple point cloud datasets with labels. By conducting supervised training on this dataset, semantic information is effectively incorporated from diverse sources. Subsequently, fine-tuning is performed on the target dataset to transfer the learned general semantics to task-specific knowledge.
This section begins by presenting the implementation and our pre-training setups. Subsequently, the effectiveness of our pre-trained models is evaluated across a range of downstream tasks. Finally, ablation studies are conducted to analyze the main properties of our PointGPT.
4.1 Implementation and Pre-training Setups
Models: Following previous studies [39; 66], PointGPT is trained employing the ViT-S configuration [67] for the extractor module, referred to as PointGPT-S. Additionally, we investigate the high-capacity models by scaling the extractor to the ViT-B and ViT-L configurations, denoted as PointGPT-B and PointGPT-L, respectively. More details can be found in the appendix.
Data: PointGPT-S is pre-trained on the ShapeNet [6] dataset without subsequent post-pre-training. This is in line with the previous SSL methods [39; 66; 68; 26] to allow for a direct comparison with these prior approaches. ShapeNet contains over 50,000 unique 3D models across 55 object categories. Additionally, two datasets are collected to support the training of high-capacity PointGPT models (PointGPT-B and PointGPT-L): (I) an unlabeled hybrid dataset (UHD) for self-supervised pre-training, which collects point clouds from various datasets [52; 35; 6; 53; 3; 60; 17], such as ShapeNet [6], S3DIS [3] for indoor scenes, and Semantic3D [17] for outdoor scenes, etc. In total, the UHD contains approximately 300K point clouds; (II) a labeled hybrid dataset (LHD) for supervised post-pre-training, which aligns the label semantics of different datasets [52; 35; 6; 53; 3; 60], with 87 categories and approximately 200K point clouds in total. Further details are provided in the appendix.