PointGPT: Auto-regressively Generative Pre-training from Point Clouds
Authors: Guangyan Chen
Meiling Wang
Yi Yang
Kai Yu
Li Yuan
Yufeng Yue
Venue: NeurIPS 2023 poster (2023)
Keywords: Generative Pre-training Transformer; GPT; Auto-regressively Generative Pre-training; Self-supervised Learning; Point clouds

Abstract: Large language models (LLMs) based on the generative pre-training transformer (GPT) have demonstrated remarkable effectiveness across a diverse range of downstream tasks. Inspired by the advancements of the GPT, we present PointGPT, a novel approach that extends the concept of GPT to point clouds, addressing the challenges associated with disorder properties, low information density, and task gaps. Specifically, a point cloud auto-regressive generation task is proposed to pre-train transformer models. Our method partitions the input point cloud into multiple point patches and arranges them in an ordered sequence based on their spatial proximity. Then, an extractor-generator based transformer decode, with a dual masking strategy, learns latent representations conditioned on the preceding point patches, aiming to predict the next one in an auto-regressive manner. To explore scalability and enhance performance, a larger pre-training dataset is collected. Additionally, a subsequent post-pre-training stage is introduced, incorporating a labeled hybrid dataset. Our scalable approach allows for learning high-capacity models that generalize well, achieving state-of-the-art performance on various downstream tasks. In particular, our approach achieves classification accuracies of 94.9% on the ModelNet40 dataset and 93.4% on the ScanObjectNN dataset, outperforming all other transformer models. Furthermore, our method also attains new state-of-the-art accuracies on all four few-shot learning benchmarks. Codes are available at https://github.com/CGuangyan-BIT/PointGPT.

PointGPT: Auto-regressively Generative Pre-training from Point Clouds
Guangyan Chen
Meiling Wang'
Yi Yang
Kai Yu!
Li Yuan
Yufeng Yue!
Beijing Institute of Technology
Peking University
Abstract
Large language models (LLMs) based on the generative pre-training transformer (GPT) [46] have demonstrated remarkable effectiveness across diverse range of downstream tasks. Inspired by the advancements of the GPT, we present PointGPT, a novel approach that extends the concept of GPT to point clouds, addressing the challenges associated with disorder properties, low information density, and task gaps. Specifically, a point cloud auto-regressive generation task proposed to pre- train transformer models. Our method partitions the input point cloud into multiple point patches and arranges them in an ordered sequence based on their spatial proximity. Then, extractor- gereettorrggeeattorr based transformer decoder (27], with a dual masking strategy, learns latent representations conditioned on the preceding point patches, aiming predict the next in auto-regressive manner. To explore scalability and enhance performance, a larger pre-training dataset collected. Additionally, subsequent post-pre-training stage is introduced, incorporating labeled hybrid dataset. Our scalable approach allows for learning high-capacity models that generalize well, achieving state-of-the-art performance various downstream tasks. In particular, our approach achieves classification accuracies 94.9% on the ModelNet40 dataset and 93.4% on the ScanObjectNN dataset, outperforming all other transformer models. Furthermore, our method also attains new state-of-the-e accuracies all four feo-shot learning benchmarks. Codes are available at htt.t::::/ / ccm:/CGuangyann-tt//Po
1 Introduction
Point clouds are becoming widely adopted data structures in various application areas, such as autonomous driving and robotics, emphasizing the importance of acquiring informative and compre-
hensive 3D representations. However, current 3D-centric approaches [9 40 41 [58] 32] typically necessitate sully-supervised training from scratch, which entails labor-intensive human annotations. In natural language processing (NI.P and image analysis domains, self-supervised learning (SSL.)
19 47 5 46 has emerged promising approach for acquiring latent representations without relying on annotations. Among these methods, the generative pre-training transformer (GPT) [46) has been particularly effective at learning representative features (7], where the task is to predict data in an auto-regressive manner. Due to its remarkable performance, we naturally ask the question: can the GPT be adapted to point clouds and serve as effective 3D representation learner?
To answer this question, we exploit the GPT scheme for point cloud understanding. However, is challenging to employ GPT on point clouds due to the following reasons: (I) Disorder properties.
'Yufeng Yue and Li Yuan contributed equally to this study co-corresponding authors. This work was supported by the National Natural Science Foundation China under Grant No. NSFC 62003039, 62233002, 92370203, 61973034, the National Key RD Program of China (2022ZD0118) 18), the CAST program under Grant No. YESS20200126.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

TIT IV
The cat chases
Point cloud
Language
Trausformer decoder
The cat chases
(a) Explanation
(b) Auto-regressive Generation
Figure 1: Illustration of our PointGPT. The transformer decoder is pre-trained predict point patches in an auto-regressive manner. Such design enables our method to predict patches without dedicated specifications and avoids positional information leakage, leading improved generalization ability.
In contrast with the sequential arrangement of words in a sentence, a point cloud is a structure that lacks inherent order. To address this issue, point patches are arranged based on geometric ordering, namely the Morton-order curve [3). which introduces sequential properties and preserves the local structures. (II) Information density differences. Languages characterized by high information richness, therefore, the auto-regressive prediction task requires advanced language understanding. On the contrary, point clouds are natural signals with heavy redundancy, thereby the prediction task be accomplished even without holistic comprehension. To address this disparity, dual masking strategy proposed, which additionally masks attending tokens for each token. This strategy effectively reduces redundancy and provides challenging task that demands comprehensive understanding. (III) Gaps between generation and downstream tasks. Even though the model with dual masking strategy exhibits sophisticated comprehension, the generation task primarily involves predicting individual points, which may result in the learned latent representations with lower semantic level than downstream tasks. To mitigate this challenge, an Etraccor--geneator architecture is introduced the transformer decoder (27], such that the generation task is facilitated through the generator, thus enhancing the semantic level of the latent representations learned by the extractor.
Based on the above analysis, we propose novel SSL framework for point clouds, called PointGPT. Specifically, our method partitions the input point cloud into multiple irregular point patches, which are subsequently organized using the Morton curve. Then, an Extractor-generato based transformer decoder, with dual masking strategy, processes the point patch sequences to learn latent represen- tations conditioned on the unmasked preceding contents, and predicts the next point patches in an auto-regressive manner. Unlike recently developed masked point modeling approaches [66] 39 68 that rely on positional information to specify reconstruction regions, resulting the leakage of the overall object shape, our concise design, as illustrated in Figure ! effectively circumvents the leakage of positional information and yields an enhanced generalization ability. Consequently, our PointGPT surpasses other single-modal SSL methods with comparable model sizes.
Inspired by the promising performance exhibited by our PointGPT, endeavor investigate its scaling property and push its performance limit further. However, significant challenge arises due to the limited scale of the existing public point cloud datasets compared to NLP and images. This dataset size disparity introduces potential overfitting concerns. To alleviate this and fully unleash the power of PointGPT, a larger pre-training dataset is collected by mixing various point cloud datasets, such as ShapeNet [6] and S3DIS 3 Moreover, subsequent post- post-pre-training stage [57] is introduced, which involves performing supervised learning on the collected labeled dataset, enabling PointGPT to incorporate semantic information from multiple sources. Within this framework, our scaled models achieve state-of-the- (SOTA) performance on various downstream tasks. object classification tasks, our PointGPT achieves 94.9% accuracy on the ModelNet40 dataset and 93.4% accuracy on the ScanObjectNN dataset, outperforming all other transformer models. few-shot learning tasks, our method also attains new SOTA performance all four benchmarks.

---

Our main contributions can be summarized as follows: (I) A novel GPT scheme, termed PPi proposed for point cloud SSL. PointGPT leverages a point cloud auto-regressive generation task while mitigating positional information leakage, outperforming other single-modal SSL. methods. (II) dual masking strategy is proposed create an effective generation task, and an extractor extractor-generatoo
transformer architecture is introduced to enhance the semantic level of the learned representations.
These designs boost the performance of PointGPT on downstream tasks. (III) A post-pre-traiinng stage is introduced, and larger datasets are collected to facilitate the training of high-capacity models. With PointGPT, our scaled models achieve SOTA performance on various downstream tasks.
2 Related Work
2.1 Self-supervised Learning for NLP and Image Processing
-elf-suue learning has attracted significant attention in recent years, especially in the fields of NLP and image processing, owing to ability learn useful representations without labeled data. The core idea of SSL is to design a pretext task to learn the distribution of the given data,
obtaining beneficial features for the subsequent supervised modeling tasks (33 [3]. Contrastive learning [8] 16
44 51 has been a popular discriminative self-supervised approach in both NI.P and image processing, with the goal of grouping similar samples closer and diverse samples further apart. However, generative SSL. methods
65|
38
34
4 [1 [9] 46 have recently achieved more competitive performance. BERT [[]] utilizes a bidirectional transformer to process the randomly masked text and reconstruct the original context. ELMo [49] adopts bidirectional LSTM (21] and generates subsequent words from left to right given representations of the previous contents. The GPT 46) also utilizes the auto-regressive prediction approach, but it employs unidirectional transformer architecture, and the model is fine-tume une-tuned by updating all pre-trained parameters. In the computer vision field, BEiT [4] and MAE [19] randomly mask input patches, and pre-train models to recover the masked patches in the pixel space. Image-GPT [7] trains a sequence transformer to auto-regressively predict pixels without incorporating knowledge concerning the 2D input structure, exhibiting promising representation learning capabilities after pre-training.
2.2
Self-supervised Learning for Point Cloud
The success of SSL in NLP and image processing has motivated researchers to develop SSL frame- works for point cloud representation learning. Among these methods, the contrastive methods 37 22 have been extensively investigated. DepthContrast [69] constructs augmented
61 69
depth maps and performs an instance discrimination task for the extracted global features. Similarly, MVIF [22] introduces cross-modal and cross-view invariance constraints achieve self-supervised modal- and view-invariant feature learning. Another line of work [42 12 62] is proposed to integrate cross-modal information and leverage knowledge transferred from language or image models for 3D learning. ACT [12] employs moosl auto-encoders as teacher models to acquire knowledge from other modalities. Different from these methods, our work attempts to learn the intrinsic properties of point clouds without relying on cross modal information and teacher models. Most relevant to our work are generative methods (4 50 39 66 68 [R], especially recently proposed masked point modeling methods (39 66
68 33 MAi extends the MAE by randomly masking point patches
and reconstructing masked regions. Point-M2AH additionally utilizes a hierarchical transformer architecture and designs a corresponding masking strategy. However, masked point modeling methods still suffer from overall object shape leakage, which limits their ability to effectively generalize to downstream tasks. In this paper, we exploit the auto-regressive pre-training for point clouds and address unique challenges associated with the properties of point clouds. Our concise design avoids positional information leakage, thereby enhancing the generalization ability.
3 PointGPT
Given point cloud X x= (2,, CR3 the overall pipeline of PointGPT during pre-training illustrated in Fig. 2 The point cloud sequencer module is utilized to construct an ordered sequence of point patches. This is achieved by dividing the point cloud into irregular patches and arranging them in Morton order. The resulting sequence is then fed into the extractor to learn latent representations, and the generator predicts the subsequent point patches in auto-regressive manner. After the
3

---

III
Positional eacoding
Extracted token
Transfrmer docnder
FPS
Sort
Vanilla mask
Generator
Replaced
Relative
Extractor
Position
Embeddiing
Directtinn
Prompt
APE
Dual mask
(a) Sorting and embedding
(b) Auto-regresive pre-training
Figure 2: Overall architecture of our PointGPT. (a) The input point cloud is divided into multiple point patches, which are then sorted and arranged in an ordered sequence. (b) An extractor gereertorggeerraoor based transformer decoder is employed along with a dual masking strategy for the rugr-segryssyvely prediction of the point patches. In this example, the additional mask of the dual masking strategy is applied to the same group of random tokens for better illustration purposes.
pre-training stage, the generator is discarded, and the extractor without the of a dual masking strategy, leverages the learned latent representations for downstream tasks.
3.1 Point Cloud Sequencer
In the field of NLP, the GPT approach benefits from an easily accessible language vocabulary and the inherent ordered properties of words. In contrast, the point cloud domain lacks a predefined vocabulary, and point cloud is sparse structure that exhibits a characteristic of the disorder. To overcome these challenges and obtain ordered point cloud sequence with each component unit capturing rich geometric information, a three-stage process consisting of point patch partitioning, sorting, and embedding employed.
Point patch partitioning: Taking the inherent sparsity and disorder properties point clouds into account, the input point clouds are processed by farthest point sampling (FPS) and the K-nearest neighbors (KNN) algorithms to obtain center points and point patches. Given a point cloud X with M points, initially sample center points C' using FPS. Then, the KNN algorithm is utilized to construct n point patches P by selecting the & nearest points from X for each center point. In summary, the partitioning procedure is formulated as:
C FPS(X), CER
P KNN(C, X), P
To address the inherent disorder properties of point clouds, the obtained point patches are organized into coherent sequence based on their center points. Concretely, the coordinates of the center points are encoded into one- diee-mmnniionll space using Morton code [3),, followed by sorting to determine the order O of these center points. The point patches are then arranged in the same order. The sorted center points C and sorted point patches P are obtained as follows:
Sorting:
O
OE Bxx1
C = P
C' E BXX P pee Raxkxx
PointNet [40] network is employed to extract rich geometric information for each point patch. To facilitate training convergence, the normalized coordinates of each point are utilized with respect to its center point. Specifically, the sorted point patches Ps are embedded into D-dimensional tokens T as follows:
Embedding:
Following Point-MAE
39
= PointNet(PP) (P'), T D

---

3.2 Transformer Decoder with a Dual Masking Strategy
straightforward extension the GPT 46 to point clouds can be achieved by utilizing the vanilla transformer decoder to luto-eegressyvely predict point patches, followed by fine-tuning all pre-trained parameters for downstream tasks. Nevertheless, this approach suffers from low-level semantics due to the limited information density of point clouds and the gaps between generation and downstream tasks. To address this issue, a dual masking strategy is proposed to facilitate comprehensive understanding of point clouds. Additionally, an extractor -ernertor transformer architecture is introduced, where the generator is more specialized for the generation task and is discarded after pre-training, enhancing the semantic level of the latent representations that learned by the extractor.
Dual masking strategy: The vanilla masking strategy in the transformer decoder enables each token to receive information from all the preceding point tokens. To further encourage the learning of useful representations, the dual masking strategy is proposed, which additionally masks a proportion of the attending preceding tokens of each token during pre- training. The resulting dual mask Md is illustrated in Fig. b), the self-attention process with a dual masking strategy can be represented as:
QKT - (1-M)) V.
SelEAttention(T) softmax
VD
where Q, K, V are T encoded with different weights for the D channels. The masked locations in Md are set to O, while the unmasked locations are set to 1.
iitractor-generatte Our extractor is composed entirely of transformer decoder blocks with dual masking strategy, obtaining latent representations T, where each point token only attends to the unmasked preceding tokens. Considering that point patches are represented in normalized coordinates, and global structures of point clouds are essential for point cloud understanding, sinusoidal positional encodings [55] (PE) are utilized map the coordinates of the sorted center points C* to the absolute positional encoding (APE). Positional encodings are added to every transformer block to provide location information and incorporate global structural information.
The generator architecture similar to the extractor architecture but contains fewer transformer blocks. It takes the extracted tokens T as input and generates point tokens T for the following prediction head. However, the patch order may be affected by the center point sampling process, inducing ambiguity when predicting the subsequent patches. This hinders the model from effectively learning meaningful point cloud representations. To address this issue, the directions relative to the subsequent point patches are provided in the generator, serving prompts without revealing the locations of the masked patches and the overall object shapes of point clouds. The relative direction prompts RDP are formulated as:
RDP: PE((C CT)/1 - C2112), (1...,n'), RDP E R''XD
(5)
where n' = n- In summary, the procedure in the extractor-generatoor architecture is formulated as:
T = Extractor + APE), TERxD,, T Cenerator(TT.w RDP),, T
Prediction head. The prediction head is utilized to predict the subsequent point patches in the coordinate space. It consists of two-layer MLP with two fully connected (FC) layers and rectified linear unit (ReLU) activation. The prediction head projects tokens T9 to vectors, where the number of output channels equals the total number of coordinates in patch. Then, these vectors are reshaped to construct the predicted point patches pad,
pnd Reshape(MLP(TT) prd E xx'x
(7)
3.3 Generation Target
The generation target for each point patch is to predict the coordinates of the points within the subsequent point patches. Given the predicted point patches Ppd, as well as the ground-truth point patches pot, which correspond to the last n' patches among the sorted point patches Pe, the generation loss L formulated using the forrrmm and 12-form the Chamfer distance (CD) 14 denoted as Ci and C2, respectively. Specifically, the generation loss is computed as L C C..
5

---

The I, form CD loss L% with n E [1, is defined as:
Σ min ||a - bll Pot be aeped
min ||a
bbll:
(8)
ae Ppd Pg
where P| the cardinality of the set P and ||a - bl. represents the L, distance between a and
We additionally find that incorporating the generation task into the fine-tuning process an auxiliary objective can accelerate training convergence and improve the generalization ability of supervised models. This approach yields enhanced performance on downstream tasks, which is in line with the GPT [46]. Specifically, we optimize the following objective during the fine-tuning stage: L = Ld + À x C where L represents the loss for the downstream task, _ represents the generation loss as previously defined, and the parameter À balances the contribution of each loss term.
3.4 Post-Pre-training
Current point cloud SSL methods directly fine-tune pre-trained models on the target dataset, which may result in potential overfitting due to the limited semantic supervision information [57]. To alleviate this issue and facilitate training of high-capacity models, we adopt the intermediate fine- tuning strategy [57 4 29 and introduce labeled hybrid dataset is leveraged (Sec. [4I]. which collects and aligns multiple point cloud datasets with labels. By conducting supervised training on this dataset, semantic information is effectively incorporated from diverse sources. Subsequently, fine-tuning is performed on the target dataset to transfer the learned general semantics to task-specific knowledge.
post-pre-trainin stage for PointGPT. In this stage,
4 Experiments
This section begins by presenting the implementation and our pre-training setups. Subsequently, the effectiveness of our pre-trained models is evaluated across a range of downstream tasks. Finally, ablation studies conducted to analyze the main properties of our PointGPT.
4.1
Implementation and Pre-training Setups
Models:
Following previous studies 39 66). PoinGGPT is trained employing the ViT-S configuration 667 for the extractor module, referred as PoinIGPT-S. Additionally, we investigate the high-capacity models by scaling the extractor to the ViT-B and ViT-L configurations, denoted as PointGPT-B and PointGPT-L, respectively. More details can be found in the appendix.
Data: PointGPT-S pre-trained on the ShapeNet 6 dataset without subsequent pess--re-training This is in line with the previous SSL methods [39 66 68 26] allow for direct comparison with these prior approaches. ShapeNet contains over 50,000 unique 3D models across 55 object categories. Additionally, two datasets are collected to support the training of high-capacity PointGPT models (PointGPT-B and PointGPT-L): (I) an unlabeled hybrid dataset (UHD) for self-supervised
60 17] such as ShapeNet [6). S3DIS 3 for indoor scenes, and Semantic3D [17 for outdoor scenes, etc. In total, the UHD contains approximately 300K point clouds; (II) a labeled hybrid dataset (LHD) for supervised post-pre-training, which aligns the label semantics of different datasets 552, 35 53 3 60), with 87 categories and approximately 200K point clouds in total. Further details are provided in the appendix.
pre-training, which collects point clouds from various datasets 22 35
Pre-training setups: The input point clouds are obtained by sampling 1024 points from each raw point cloud. Afterward, each point cloud is partitioned into 64 point patches, with each patch consisting of 32 points. The PointGPT model is pre-trained for 300 epochs using an AdamW optimizer (31) with a batch size of 128, initial learning rate of 0.001, and weight decay of 0.05. Additionally, based on our empirical results, cosine learning rate decay (30 is employed.
4.2 Downstream Tasks
To demonstrate the performance of our method on different downstream tasks, we conduct experi- ments involving object classification real-world and clean object datasets, few-shot learning, and part segmentation. The performance of PointGPT evaluated using three different model capacities: PointGPT-S, which is pre-trained on the ShapeNet dataset without post- post-pre-trainin: and PointGPT-B

---

Table Classification results on ScanObjectNN and ModelNet4O datasets. All results are expressed as percentages. Specifically, three variants are evaluated on the ScanObjectNN dataset. Additionally,
The accuracy obtained on ModelNet40 dataset reported both 1K and points. The symbols and denote larger pre-training dataset and post-pre-training stage, respectively.
ScanObjectNN
ModelNer40 1k
Methods
Reference
OBJ BG
OBJ ONLY
PB TS_T5O_S
8
Supervised Learning Only
PointNet 40
CVPR 2017
73.3
79.2
68.0
89.2
90.8
DGCNN [58]
TOG 2019
82.8
86.2
78.1
92.9
PointCNN 25
Neurips 2018
86.1
85.5
78.5
92.2
GBNet 45
TMM 2021
81.0
93.8
MVTN 18
ICCV 2021
92.6
92.3
82.8
93.8
PointMLP
52
ICLR 2022
85.4
94.5
PointNeXt
43
Neurips 2022
87.7
94.0
P2P-RN101
159
Neurips 2022
87.4
93.1
P2P-HorNet 59
Neurips 2022
89.3
94.0
with Self-Supervised Representation Learning
Point-BERT [66]
CVPR 2022
87.4
88.1
83.1
93.2
93.8
MaskPoint [26
ECCV 2022
89.3
88.1
84.3
93.8
Point-MAE 139
ECCV 2022
90.0
88.2
85.2
93.8
94.0
Point-M2AE [68
Neurips 2022
91.2
88.8
86.4
94.0
PointGPT-S
91.6
90.0
86.9
94.0
94.2
PointGPT-BB
95.8
95.2
91.9
94.4
94.6
PointGPT-L.
97.2
96.6
93.4
94.7
94.9
Methods using cross-modal information and teacher models
ACT 12
ICLR 2023
93.3
91.9
88.2
93.7
94.0
PointML++ULIT 662
CVPR 2023
89.4
94.5
94.7
ReCon
ICML 2023
95.4
93.6
91.3
94.5
94.7
and PointGPT-L, which undergo both pre-training and post-pre- post-prettraining stages on the collected hybrid datasets. The impact of post-pre- tee---reerriinngg is further investigated and discussed in the appendix.
Object classification on real-world dataset: The performance of the proposed method on real-world dataset is an important indicator of its practical applicability. Therefore, the pre-trained models are transferred the ScanObjeNtNN dataset [54], which contains approximately 15,000 objects extracted from real-world indoor scans. The experiments are conducted under three different settings, OBJ-BG, OBJ-ONLY, and PB-T5O-RS. The results are presented in Table our PointGPT-S, which has similar capacities and training data to previous methods like Point-MAE, outperforms other single-mi modal SSL methods. Furthermore, even when compared to Recon and ULIP, which utilize cross-modal information and teacher models, scaled PointGPT-B model achieves superior performance, and PointGPT-L. achieves accuracy improvements of at least 1.8%.
Object classification on clean objects dataset: The pre-trained models are evaluated on the ModelNet40 dataset [60] which includes 12,311 clean 3D CAD models, covering 40 categories. To conduct fair comparisons, the standard voting method (28 is used during testing, and the input point clouds exclusively contain coordinate information, without additional normal information provided. The experimental results are presented in Table 1 our PointGPT-S surpasses other single-modal SSL methods. Even comparison with Recon and ULIP, our PointGPT-L achieves superior performance.
Few-shot learning: To intuitively demonstrate the generalization ability of our method, few-shot learning experiments conducted on the ModelNet40 dataset without the post-pre-training stage. Following the protocols of previous studies [39] 66 68). the few-shot learning experiments consist of four distinct tests, employing w-way, s-shot setting. Specifically, w E (5, 10) represents the number of randomly selected classes, and E {10,20) denotes the number of randomly sampled objects for each selected class. Each test is conducted with independent trials. The results, as shown in Table 2 indicate that our method outperforms other methods in all tests, particularly in the 10-shot tests. This demonstrates the ability of PointGPT in acquiring knowledge for new tasks, even under the constraints of limited training data.

---

Table Few-shot classification results on the ModelNet40 dataset. In each experimental setting, 10 independent trials are conducted, and the mean accuracy (%) is reported with its standard deviation. Symbol denotes larger pre-training dataset, and the
stage excluded.
5-way
10-way
Methods
Reference
10-shot
20-shot
10-shot
20-shot
DGCNN [58]
TOG 2019
31.6.2.88
40.814.66
19.9=2.1
16.9
OcCo 56
ICCV 2021
90.6:12.8
9..
82.9=1.3
86.5
with Self- Supp-iuperiiedd Representation Learning
Point-BERT
CVPR 2022
94.6+t.1
96.3+22.7
91.0=#.4
92.7=5.1
MaskPoint 26
ECCV 2022
95.0=t3.7
97.2+1.7
91.4+4.0
93.4=3.5
Point-MAE 39
ECCV 2022
96.3+:2.5
97.8+
92.6+4.11
95.0+3.00
Point-M2AF 68
Neurips 2022
96.8++1.8
98.3+114
92.3+4.55
95.0+3.0
PointGPT-S
96.8++22.0
98.6+1.1
92.6+4.66
95.2++34
PointGPT-BB
97.5+22.0
98.8111.0
93.514.0
95,8=3.0
PointGPT-L.
98.0:11.9
99.0111.0
94.113.3
96.1=288
Methods using cross-modal information and teacher models
ACT 12
ICLR 2023
96.8+t..3
98.0+11.4
93.3
95.64128
ReCon 42
ICML 2023
97.3+11
98.9
93.3+3.9
95.8
Table 3: Part segmentation results on the ShapeNetPart dataset. The mean intersection over union (mloU) is reported across all classes (Cls.) and all instances (Inst.). Symbols and denote larger pre-training dataset and post-pre-training stage, respectively.
Methods
Reference
Cls. mIoU(%)
Inst. mIoU(%)
PointNet [40]
CVPR 2017
80.4
83.7
PointNet++
Neurips 2017
81.9
85.1
DGCNN [58]
TOG 2019
82.3
85.2
PointMLP
ICLR 2022
84.6
86.
with Self- Sele-Suue Reprexentation Learning
PointContrast [1
ECCV 2020
85.1
CrossPoint
CVPR 2022
85.5
Point-BERT 66
CVPR 2022
84.1
85.6
Point-MAE 339
ECCV 2022
86.1
PointGPT-S
84.1
86.2
PointGPT-B
84.5
86.5
PointGPT-L
84.8
86.6
Methods using cross-modal information and teacher models
ACT 212
ICI. 2023
84.7
86.1
ReCon
ICML 2023
84.8
86.4
Part segmentation: The representation learning capability of our method is evaluated the ShapeNetPart (64] dataset, which consists 16881 objects across 16 categories. The point clouds are sampled into 2048 points, and the segmentation head (39 concatenates the extracted features 1xtd 2xtd and 3xtd of the extractor transformer blocks, with td representing the depth 3
from layers 3 3 of the extractor. Subsequently, average pooling, max pooling, and upsampling utilized generate features for each point and an MLP is applied for label prediction. The experimental results, displayed in Table demonstrate the superior performance of our PointGPT-L compared to all other methods.
4.3 Ablation Studies
Comprehensive ablation studies are conducted to investigate the fundamental designs of our PointGPT model. The impacts of these designs are evaluated by reporting the accuracy achieved by fine-tuning the model the ModelNet40 dataset for object classification. To provide an intuitive representation of the results, ablation studies are conducted using the PointGPT-S model, which is pre-trained on the ShapeNet dataset and directly fine-tuned on the target dataset without post-pre-traininee training..

---

Table Ablation experiments with PointGPT-S pertaining on the ShapeNet dataset. The fine-tuned accuracy (%) achieved without post-pre- treetrrainng is reported. Default settings are marked in gray
(a) Generator depth.
(b) Generation targets,
(c) Generation during fine mee-tuunng..
Blocks
Acc.
Targets
Acc.
Coef.
Acc.
0
93.85 %
Coordinates
94.21 %
0
94.01 %
94.08 %
FPFH
94.13 %
1
94.15% %
4
94.21 %
PointNet DGCNN
94.31 %
3
94.21 I %
94.24
94.35 %
5
94.05% %
(d) Generation loss.
(e) Relative direction prompts.
(1) Dual masking strategy.
Loss
Acc.
Case
Acc.
Ratio
Acc.
Ratio
Acc.
CD /1
93.66%
None
93.69%
0
93.68% %
5
94.01 %
CD
94. 93%
Absolute position
94.06%
93. 70%
7
94.21%
CD 71+12
94.21%
Relative direction 94.21 %
3
93.85%
9
93.66%
Table 5:
Classification results on ScanObjectNN and ModelNet40 datasets. The symbols and denote larger pre-training dataset and post- pre-trainit stage, respectively.
ScanObjectNN
ModelNet40
Methods
Params
OBJ BG OBJ ONLY PB T50 RS
Ik
8k
with different point sorting methods
PoinGGTTS w/ KD-Tree
89.0
88.6
81.3
93.2
93.4
19.5M
PoinGGPTSS w/ Hilbert
89.2
89.0
84.3
93.4
93.6
19.5M
PointGPT-S
91.6
90.0
86.9
94.0
94.2
19.5M
with larger pre-training datasets and post pre pre--preettttt trainini stage
Point-MAE-B
[39]
94.2
93.9
90.2
94.2
94.4
120.IM
Point-M2AE-B
168
95.2
94.3
91.2
94.3
94.5
77.5M
PointGPT-B
95.8
95.2
91.9
94.4
94.6
82.1M
Generator: Table 4 a) investigates the effect of varying the generator depth. The results demonstrate that the extractor extractor-generator architecture facilitates the learning of strong semantic representations, particularly when combined with deep generator, resulting in improved performance overall. However, due to the computational complexity associated with the deep generator, depth of is selected as the default setting for the generator.
The generation objective is essential for enabling the model to learn the intrinsic characteristics of the given data. Table 4[b exhibits four distinct generation targets, which can be categorized into two groups: one-stage targets that can be directly obtained, including point coordinates and FPFH [48). and two-stage targets that extracted by trained deep network, including PointNet 40 and the DGCNN [58]. The experimental results indicate that the use of handcrafted FPFH features leads underperformance, which may be attributed to the overfitting of low-level geometric features. The variants with two-stage targets outperform the variant with point coordinate targets. However, the pre-training and inference processes of the teacher model inevitably incur an additional computational cost.
Generation targets:
Generation task in the fine-tuning stage: The generation task is included during the fine-tuning stage. Table( presents the obtained results when varying the coefficient of the generation loss in the fine-tuning loss. The results signify that this auxiliary objective serves as a regularization term and improves the generalization ability of supervised models. Furthermore, the results suggest that the coefficient increases, the accuracy achieved in the classification task exhibits an increasing trend followed by a decreasing trend, reaching its highest value when À =
auxiliary objective
Generation loss: Table 4 d) presents the performance of variants using different generation loss functions, including the TT-form CD loss, the 12-form CD loss, and the combination of both the /1- and (2-forms CD loss. The results demonstrate that the combination of the (1- and /2- 22-forms achieves

---

superior performance. We analyze that the 12-form is more effective in guiding the network toward convergence and the 11-form has better sparsity, thus the combination both forms is more effective.
Relative direction prompts: The effect of utilizing relative direction prompts is analyzed in Table Ae). The variant utilizing relative direction prompts outperforms the variants using absolute positional encoding and excluding positional encoding. We hypothesize that this improvement stems from the ability of relative direction prompts prevent the model from overfitting the patch order, thus enhancing the performance of PointGPT in downstream tasks.
Dual masking strategy: We conduct analysis on the impact of the dual masking strategy and search for the proper mask ratio, as shown in Table ff). Decreasing the mask ratio to 0 corresponds to employing the vanilla masking strategy. The results indicate that both excessive and insufficient masking ratios lead to a decline in performance. The experimental results demonstrate that the dual masking strategy is effective in promoting beneficial representation learning and enhancing the generalization ability of the pre-trained model.
Point sorting methods: We analyze various point sorting methods, conducting ablation studies on the ScanObjectNN and ModelNet40 datasets using PointGPT-S models. Performance comparisons with the widely used KD-tree sorting [15] and Hilbert code sorting [20] methods are summarized in 5 The results indicate the substantial superiority of Morton code sorting. This observed effectiveness can be attributed to Morton code sorting's s proficient preservation of the adjacency relationship between points, where adjacent points in one-dimensional space are often proximate to each other in three-dimensional space.
Comparisons using larger pre-training dataset and post-pre-training Experiments are conducted by re-training comparison methods using the ViT-B configuration with collected larger datasets and pos-pre-trainin stage. For a fair comparison, we focus on re-training well-performing single-modal self-supervised methods, Point-MAE, and Point-M2AE. The results, presented in Table illustrate the superior performance of our PointGPT over comparison methods, utilizing comparable training parameters and the same training data.
Conclusion
In this paper, we present PointGPT, a novel approach that extends the GPT concept to point clouds. addressing the challenges associated with disorder properties, information density differences, and gaps between the generation and downstream tasks. Unlike recently proposed self-supervised masked point modeling approaches, our method avoids overall object shape leakage, attaining improved generalization ability. Additionally, we explore high-capacity model training process and collect hybrid datasets for pre-training and post-pre- post-pre-training. The effectiveness and strong generalization capabilities of our approach are verified on various tasks, indicating that our PointGPT outperforms other single-modal methods with similar model capacities. Furthermore, our scaled models achieve SOTA performance on various downstream tasks, without the need for cross-modal information and teacher models. Despite the promising performance exhibited by PointGPT, the data and model scales explored for PointGPT remain several orders of magnitude smaller than those in NLP [O] and image processing 667 [2] domains. Our aspiration is that our research can stimulate further exploration in this direction and narrow the gaps between point clouds and these domains.
Acknowledgements
This work was also supported in part by the National Key RD Program of China under Grant 2022ZD01 18101. Natural Science Foundation of China NO. 62202014, and Shenzhen Basic Research Program JCY2002208111517260000
References
[1] Panos Achlioptas, Olga Diamanti, loannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In International conference on machine learning, pages 40-49. PMLR, 2018
[2] Mohamed Atham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Rodrigo. Crosspoint: Self-supervised cross- moosl contrastive learning for 3d point cloud
10

---

understanding. In Proceedings of the IEEE/CVF Conference on Compuder Vision and Puttern Recognition, pages 9902-9912, 2022.
[3] Iro Armeni, Ozan Scner, Amir R Zamir, Helen Jiang, loannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE conference on computer
vision and pattern recognition, pages 1534-1543, 2016.
[4] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv
preprint arXiv:2106. 08254. 2021.
[5] Tom B Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.
arXiv preprint ar Xr0s 14165, 2020.
[6] Angel X Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: /512.030/ 2015.
Mark Chen. Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun. David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691-1703.
PMLR, 2020.
[8] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages
1597-16077 PMLR, 2020.
[9] Christopher Choy. Young Gwak, and Silvio Savarese. 4d
convnets: Minkowski convolutional neural networks. In Proceedings of the IREF/CVF conference computer vision and pattern recognition, pages 3075-3084, 2019.
[10] Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR 005, volume 1, pages 886-893. Icce, 2005.
[11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv: 1800004805, 2018.
[12] Runpei Dong, Zekun Qi. Linfeng Zhang, Junbo Zhang. Jianjian Sun, Zheng Ge, Li Yi, and Kaisheng Ma. Autoencoders as mross-modal teachers: Can pretrained 2d image transformers help 3d representation 2022.
learning? arXiv preprint arXiv:2212.08//
[13] Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. Why does unsupervised pre-training help deep learning? In Proceedings the thirteenth international conference on artificial intelligence and statistics, pages 201. JMLR Workshop and Conference Proceedings. 2010.
[14] Haoqiang Fan, Hao Su, and Leonidas Guibas. point set generation network for 3d abject reconstruction from a single image. In Proceedings of the IEFE conference on computer vision and pattern recognition,
pages 605-61 3, 2017.
[15] Matheus Gadelha, Rui Wang, and Subhransu Maji. Multiresolution tree networks for 3d point cloud processing. In Proceedings of the European Conference on Computer Vision (ECCV), pages 103-116 2018.
[16] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-
144, 2020.
[17] Timo Hackel, Nikolay Savinov, Lubor Ladicky, Jan D Wegner, Konrad Schindler, and Mare Pollefeys. Semantic3d. net: A new large-scalc point cloud classification benchmark. arXiv preprint arxiv / 1704.03847,
2017.
[18] Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. Mvtn: Multi-view transformation network for 3d shape recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
1- 2021.
[19] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16000-16009, 2022.
[20] David Hilbert and David Hilbert. Über die stetige abbildung einer linic auf ein flächenstück. Dritter Band: Analysis- Grundlagen der Mathematik Physik Verschiedenex: Nebst Einer Lebensgeschichte, pages 1-2, 1935.
[21] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()): 1780, 1997.
[22] Longlong Jing, Yucheng Chen, Ling Zhang, Mingyi Hc, and Yingli Tian. Self-suporvised modal and view invariant feature learning. arXiv preprint arXiv:2005.1 14169, 2020.
[23] Julia A Lasserre, Christopher M Bishop, and Thomas P Minka. Principled hybrids of generative and discriminative models. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR' 06), volume pages 87- 94, IEEE, 2006.
[24] Jiaxin Li. Ben M Chen, and Gim Hee Lee. So-net: Self-organizing network for point cloud analysis. In
Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9397-9406, 2018. [25] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolution on
x- t-ransffrmed points, Advances in neural information processing systems, 31:820-830 2018
[26] Haotian Liu, Mu Cai, and Yong Jae Lee. Masked discrimination for self-supervised learning on point clouds. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27. 27,
11

---

2022, Proceedings, Part II. pages 657-675. Springer, 2022.
[27] Peter J Liu, Mohammad Salch, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. arXiv preprint 1rXiv:1801.10 2018. [28] Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan. Relation-shape convolutional neural network for point cloud analysis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition. pages 8895-8904, 2019.
129] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang. Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on compuder vision and pattern recogmition, pages 1 12009-12019. 2022.
[30] Loshchilov and F Hutter. Stochastic gradient descent with warm restarts. In Proceedings of the 5th Int.
Conf. Learning Representations, pages 1-16, 2016.
[31] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.
arXiv preprint
Nif::1711.05001.1 2017.
[32] Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local geometry
in point cloud: A simple residual mip framework. arXiv preprint arXiv:2202.0711 2022.
[33] Chen Min, Dawei Zhao, Liang Xiao, Yiming Nie, and Bin Dai. Voxel-mae: Masked autoencoders for
pre-training large-scale point clouds. arXiv preprint arXiv:2206.0990M 2022.
[34] Ishan Misra and Laurens van der Maaten, Self-superviser learning of pretext-invariant representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6707-6717,
2020.
[35] Kaichun Mo. Shilin Zhu, Angll X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large-scal benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 909-918.
2019.
[36] Guy M Morton, A computer oriented gendetic data base and new technique in file sequencing. 1966.
[37] KL Navaneet, Ansu Mathew, Shashank Kashyap. Wei-Chih Hung, Varun Jampani, and R Venkatesh Babu. From image collections to point clouds with sulf-super shape and pose networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1132-1140, 2020.
[38] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive
coding, arXiv preprint arXiv:1807.000 0307.03748, 2018.
[39] Yatian Pang. Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders
for point cloud self-supervised learning. arXiv preprint arXiv:2203.0666 2022.
[40] Charles Qi. Hao Su. Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d
classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern
recognition. pages 652-660, 2017.
[41] Charles R Qi Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical feature learning on
point sets in metric space. arXiv preprint arXiy: 1706.02063 2017.
[42] Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma. and Li Yi. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining, arXiv preprint arXiv:2302.0233 2023.
[43] Guocheng Qian. Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet+++ with improved training and scaling strategies. Advances in Neural Information Processing Systems, 35:23192 23204, 2022.
[44] Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang. Huisheng Wang. Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 6964-6974, 2021.
[45] Shi Qiu, Saeed Anwar, and Nick Barnes. Geometric back-projection network for point cloud classification.
IEEE Transactions on Multimedia, 24:1943-1955 2021.
[46] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
[47] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[48] Radu Bogdan Rusu, Nico Blodow, and Michael Beetz. Fast point feature histograms (fpsh) for 3d registration. In 2009 JEEE international conference on robotics and automation, pages 3212-3217. IEBE, 2009.
[49] Justyna Sarzynska- Wawer, Aleksander Wawer, Aleksandra Pawlak, Julia Szymanowska, Izabela Stefaniak, Michal Jarkiewicz, and Lukasz Okruszek. Detecting formal thought disorder hy deep contextualized word representations. Psychiatry Research, 304:1 14135, 2021.
[50] Jonathan Sauder and Bjarne Sievers. Self-supervised deep learning on point clouds by reconstructing space. Advances in Neural Information Processing Systems, 32. 2019.
[51] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine. and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pages 1.44 IEEE, 2018
[52] Shuran Song. Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: rgb-d scene understanding benchmark suite. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 567-576, 2015.
12

---

[53] Mikaela Angelina Uy. Quang-Hiiuu Pham. Binh-Son Hua, Duc Thanh Nguyen, and Sai-Kit Yeung. Revisit- ing point cloud classification: new benchmark dataset and classification model on real-world data. In International Conference on Computer Vixion (ICCV), 2019.
[54] Mikaela Angelina Uy. Quang-Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In Proceedings of the IEEF/CVE international conference on computer vision, pages 1588-1597, 2019.
[55] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998 0008-600, 2017.
[56] Hanchen Wang, Qi Liu. Xiangyu Yue, Joan Lasenby, and Matt J Kusner. Unsupervised point cloud pre-training via occlusion completion. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9782 9792, 2021.
[57] Limin Wang. Bingkun Huang, Zhiyu Zhao, Zhan Tong. Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. arXiv preprint arXiv:2303. 16727, 2023.
[58] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon.
Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog), 38(5);1-22, 2019, Ziyi Wang. Xumin Yu, Yongming Rao, Jie Zhou, and Jiwen Lu. P2p: Tuning pre-trained image models for point cloud analysis with point-to-pixel prompting. arXiv preprint arXiv:2208.0288
[59]
2022.
[60] Zhirong Wu, Shuran Song. Aditya Khosla, Fisher Yu, Linguang Zhang. Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 1912-1920, 2015.
[61] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part III 16, pages 574-591. Springer, 2020.
[62] Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martin Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles. and Silvio Savarese. Ulip: Learning unified representation of language, image and point cloud for 3d understanding. arXiv preprint arXiv:2212. 05111, 2022.
[63] Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
206-215. 2018.
[64]
Li Yi. Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections.
ACM Transactions on Graphics (ToG), 35(6):1-12, 2016.
[65]
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Proceedings of the AAAI conference artificial intelligence. volume 31, 2017. Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang. Jie Zhou, and Jiwen Lu. Point- bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19313-19322, 2022.
[66]
[67]
Xiaohua Zhai, Alexander Kolesnikov. Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12104- 12113, 2022.
[68] Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li. Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training. arXiv preprint 2r5 14401. 2022.
[69] Zaiwei Zhang. Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of 3d features any point-cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10252-10263, 2021.
13


Reviews:
[{"review_id": "sOLFSoZDwi", "reviewer": "Reviewer_EUQH", "summary": "This paper proposed a point clouds pretraining method named PointGPT, which extends the generative pretraining approach of NLP to point clouds. With point patch partitioning and sorting, point embeddings are feed into a transformer decoder for autoregressive prediction. Besides, a dual masking strategy is proposed to enhance the learned representation. PointGPT is evaluated on several point cloud classifiaction, few-shot classification and part segmentation datasets with promising results.", "questions": "See weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Extend generative pretraining into point clonds. pointGPT let us to rethink the feasibility of generative pretraining on point cloud tasks.\n- A complete framework with point sequencer and dual masking strategy. Re-arranging point clouds into a sequence of tokens like natural text and RGB images is challenging. PointGPT gives one possible solution using Morton code. The dual masking strategy appears to be a combination of masked modeling and autoregressive pretraining.\n- Promising results on a set of datasets including classfication, few-shot classification and part seggmentation.", "weaknesses": "- Is it reasonable to use Morton code to sort unordered point clouds? As shown in Fig.2, PointGPT forces the current point $i$ to predict the coordinates of a point sorted by Morton code. However, the point $i$ should be able to predict any point adjacent to its 3D space, rather than a specified point. So what are the advantages of using Morton code sorting as the prediction target compared to conventional left-to-right and top-to-bottom prediction?\n- The real gain of PointGPT (PointGPT-S) compared with the previous masked modeling methods (Point-M2AE) is not significant. Although PointGPT-B/L show an improvement on these tasks, they use more parameters and more training data.\n- The overall contribution is limited. PointGPT is inspired by generative pretraining in NLP, but it doesn't show us how effective the generative pretraining approach is on point cloud tasks compared to previous methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**@Q1 - Morton code sorting**. \n\n**(1) Morton code sorting effectively preserves the adjacency relationships between points**, allowing points that are close in three-dimensional space to maintain adjacency after sorting. However, the left-to-right and top-to-bottom (`L2R&T2B`) sorting method struggles to achieve this, due to the sparsity of point clouds. Furthermore, **(2)** to **prevent each point patch from predicting a fixed patch**, data augmentations, like rotation and translation, are applied to introduce variations in the order of the sorted patches. Consequently, each patch is tasked with predicting different patches under various transformations, necessitating accurate predictions within its local neighborhood. Notably, **(3)** experimental findings illustrate that **Morton sorting achieves improved accuracy**. To intuitively validate the effectiveness of Morton sorting, we perform additional experiments employing the PointGPT-S model on ScanObjectNN (OBJ_BG, OBJ_ONLY, PB_T50_RS) and ModelNet40 (1k points) datasets. The performance is compared with `L2R&T2B`, as well as the widely adopted `KD-tree` sorting [1] and `Hilbert` code sorting [2] methods. The results indicate that Morton code sorting outperforms alternative sorting methods.\n\n|Methods|OBJ_BG|OBJ_ONLY|PB_T50_RS|ModelNet40|\n|-|-|-|-|-|\n|L2R&T2B|88.8|88.4|80.8|93.2|\n|KD-tree|89.0|88.6|81.3|93.2|\n|Hilbert|89.2|89.0|84.3|93.4|\n|Ours|91.6|90.0|86.9|94.0|\n\nDetails: We implement L2R&T2B by dividing the point cloud into multiple 0.06x0.06x0.06 grids [3] and sorting the points according to their grid coordinates, with a primary sorting based on the x-axis, followed by the y-axis, and ultimately the z-axis. \n\n[1] Gadelha, et al. \"Multiresolution tree networks for 3d point cloud processing.\" Proceedings of the European Conference on Computer Vision (ECCV). 2018.\n\n[2] Hilbert, et al. \"Über die stetige Abbildung einer Linie auf ein Flächenstück.\" Dritter Band: Analysis· Grundlagen der Mathematik· Physik Verschiedenes: Nebst Einer Lebensgeschichte (1935): 1-2.\n\n[3] Thomas, et al. \"Kpconv: Flexible and deformable convolution for point clouds.\" Proceedings of the IEEE/CVF international conference on computer vision. 2019.\n\n**@Q2 - Comparison with Point-M2AE**. \n\n**(1) PointGPT-S attains an improved accuracy-time balance**. PointGPT adopts a more concise design compared to Point-M2AE, which incorporates hierarchical architecture and a specialized multi-scale masking strategy involving backtracking. Consequently, **PointGPT demonstrates a remarkable 50% reduction in pre-training time**, which holds significant practical value, saving considerable wall-clock time when scaling up datasets and model parameters. Furthermore, PointGPT-S surpasses Point-M2AE-S across all tasks, despite Point-M2AE benefiting from multi-scale features. Importantly, PointGPT is not confined to small-scale models, we explore the potential of high-capacity models. \n\n|Methods|Params|Flops|Pre-training time|\n|-|-|-|-|\n|Point-M2AE-S|15.3M|4.0G|32.5 hours|\n|PointGPT-S|19.5M|2.2G|15.8 hours|\n\n**(2)** The additional experiments demonstrate that **the improvements introduced by PointGPT-B/L are closely linked with the designs of PointGPT**, which effectively address the information leakage and enhance the generalization ability. Specifically, experiments are performed to re-train Point-M2AE with ViT-B configurations using UHD and LHD datasets (`Point-M2AE-B`). The results are presented below, encompassing ScanObjectNN (OBJ_BG, OBJ_ONLY, PB_T50_RS), ModelNet40 (1k points), number of parameters (Params), and total pre-training time, demonstrating **the superior performance of PointGPT over Point-M2AE models, even under comparable training parameters and identical training data**.\n\n|Methods|OBJ_BG|OBJ_ONLY|PB_T50_RS|ModelNet40|Params|Pre-training time|\n|-|-|-|-|-|-|-|\n|Point-M2AE-B|95.2|94.3|91.2|94.3|77.5M|156 hours|\n|PointGPT-B|95.8|95.2|91.9|94.4|82.1M|75 hours|\n\n**@Q3 - Contributions**. \n\nRecent pre-training methods for point clouds also belong to the generative pre-training approach, such as Point-MAE. However, a foundational distinction lies in their adoption of BERT-style masked modeling pre-training approaches, contrasting with our exploration of GPT-style auto-regressively generative pre-training methods.\n\n**Compared to preceding methods, PointGPT effectively addresses the information leakage**. Previous methods rely on introducing positional information to specify the regions for prediction, which leads to a significant and widespread issue of information leakage, limiting the efficacy of the pre-training process. To overcome this challenge, PointGPT utilizes an auto-regressive pattern and the relative direction prompt to specify prediction patches. This approach obviates the need for explicitly utilizing positional information, effectively addressing information leakage, and enhancing the generalization ability, as demonstrated in @Q2. \n\n**Our contributions also encompass (1) the first attempt of GPT on point clouds and (2) the exploration of high-capacity model training** within the point cloud domain. (1) PointGPT explores the point cloud sorting methods to manage the disordered nature of point clouds. Additionally, we propose a dual masking strategy and an extractor-generator architecture to overcome the challenges associated with information density differences and gaps between the generation and downstream tasks. (2) To fully unleash the power of PointGPT, we collect a larger pre-training dataset. Moreover, a subsequent post-pre-training phase is introduced alongside a labeled hybrid dataset, facilitating the integration of semantic information from various sources into the models.\n\n|(1) First attempt of GPT on point clouds|(2) High-capacity model training|\n|-|--|\n|point cloud sorting|post-pre-training stage|\n|dual-masking strategy|unlabeled hybrid dataset|\n|extractor-generator architecture|labeled hybrid dataset|\n|relative direction prompt||"}, {"review_id": "24IBhU7Nd8", "reviewer": "Reviewer_5U9K", "summary": "This paper propose an auto-regressively generative pre-training paradigm for point cloud feature encoding. By incorporating GPT, the disorder and low information density properties of point clouds are addressed. Besides GPT, a dual masking strategy is proposed to improve the pre-training performance. The proposed models achieves SOTA o downstream tasks.", "questions": "I think this is an excellent work. But I still have some questions:\n\n1.\tWhy PointGPT outperforms PointMAE and PointM2AE?  I don’t understand what is ``overall object shape leakage`` in L109. In L163, PointGPT also utilizes position information to extract global structural information.\n\n2.\tThe dual masking strategy seems that further introduce MAE to PointGPT. What’s the performance when using the masking strategy only and removing the GPT loss?\n\n3.\tThe training of PointGPT-S is aligned to previous pretraining methods and PointGPT-S only presents minor improvement compared with Point-M2AE. The PointGPT-B and PointGPT-L are trained on larger datasets but the competitive methods Point-M2AE and Point-MAE are only trained on ShapeNet. Please provide an ablation study such as Point-M2AE+UHD+LHD to show that the large performance gain doesn’t trivially come from the larger training dataset.\n", "limitations": "The Morton-order curve may not be the optimal order.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.\tA novel GPT-style point cloud pre-training framework is proposed.\n\n2.\tArranging point patches via Morton-order curve is effective.\n\n3.\tThe performance of the PointGPT-L is extraordinary, significantly surpassing previous methods.\n", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**@Q1 - Overall object shape leakage**. \n\n**(1) The overall object shape leakage is attributed to the positional encoding leakage of masked regions**. Point cloud data is constituted by the spatial positions of individual points. However, previous methods rely on introducing positional encoding into mask tokens to specify prediction regions, leading to the positional encoding of masked regions being leaked. Consequently, the model can effortlessly infer the overall shape. **(2)** In contrast to prior methods, **PointGPT entirely masks the positional encoding of masked regions**. To achieve this, PointGPT eliminates the need for mask tokens and their associated positional encoding by utilizing the auto-regressive prediction pattern and the relative direction prompt. Consequently, while positional encoding remains employed, our method ensures the complete masking of information from masked regions.\n\n**@Q2 - Using the masking strategy only and removing the GPT loss**. \n\nThanks for your valuable feedback! We directly train the PointGPT-S model on the target datasets and maintain the dual masking strategy (`PointGPT-S-D`), as the removal of GPT loss would render pre-training infeasible. The experiments are performed on the ScanObjectNN (OBJ_BG, OBJ_ONLY, PB_T50_RS) and ModelNet40 (1k points) datasets, and the results are shown below. \n\n|Methods|OBJ_BG|OBJ_ONLY|PB_T50_RS|ModelNet40|\n|-|-|-|-|-|\n|PointGPT-S-D|86.2|86.1|79.8|92.3|\n|Ours|91.6|90.0|86.9|94.0|\n\nWe believe that your question aims to investigate the effectiveness of the dual masking strategy and the GPT loss. Therefore, we additionally conducted the following experiments focusing on these aspects.\n\n**(1) Dual masking strategy experiments**. Experiments are conducted by varying the masking ratio on the ModelNet40 dataset, the results reveal that utilizing the dual masking strategy with an appropriate masking ratio significantly boosts accuracy. \n\n|Ratio|0%|10%|30%|50%|70%|90%|\n|-|-|-|-|-|-|-|\n|Acc.|93.68|93.70|93.85|94.01|94.21|93.66|\n\n**(2) GPT loss experiments**. We conducted two supplementary experiments. **(i) Removal of GPT loss** (`PointGPT-S-R`). The PointGPT-S model is directly trained on the target datasets without the pre-training stage and masking strategy, serving as the baseline for the PointGPT-S framework. The results reveal the significant accuracy enhancement achieved by the PointGPT pre-training stage. **(ii) Replacement of GPT pre-training with MAE pre-training** (`PointGPT-S-M`). The auto-regressive pre-training of PointGPT is replaced with Point-MAE's masking and reconstruction pre-training, while utilizing the dual masking strategy. The findings suggest that under the application of the dual masking strategy, PointGPT's pre-training method achieves superior generalization ability. \n\n|Methods|OBJ_BG|OBJ_ONLY|PB_T50_RS|ModelNet40|\n|-|-|-|-|-|\n|PointGPT-S-R|86.7|87.1|79.8|92.3|\n|PointGPT-S-M|89.6|88.3|81.7|92.5|\n|Ours|91.6|90.0|86.9|94.0|\n\n**@Q3 - Comparison with Point-M2AE**. \n\n**(1) PointGPT-S achieves an enhanced accuracy-time trade-off**. It adopts a more concise design compared to Point-M2AE, which incorporates hierarchical architecture and a specialized multi-scale masking strategy involving backtracking. Consequently, **PointGPT-S achieves a remarkable 50% reduction in pre-training time**. This achievement holds substantial practical significance, conserving considerable wall-clock time, particularly when dealing with larger datasets and model parameters. Furthermore, PointGPT-S outperforms Point-M2AE-S across all tasks, despite Point-M2AE benefiting from multi-scale features.\n\n|Methods|Params|Flops|Pre-training time|\n|-|-|-|-|\n|Point-M2AE-S|15.3M|4.0G|32.5 hours|\n|PointGPT-S|19.5M|2.2G|15.8 hours|\n\n**(2)** Additionally, to demonstrate **the superior performance of PointGPT with high-capacity models**. Experiments are conducted by re-training Point-MAE and Point-M2AE using the ViT-B configuration with UHD and LHD datasets (`Point-MAE-B` and `Point-M2AE-B`). The results are depicted below, encompassing ScanObjectNN (OBJ_BG, OBJ_ONLY, PB_T50_RS) and ModelNet40 (1k points). These results demonstrate that **PointGPT outperforms comparison methods with comparative training parameters and identical training data**, and the improvements introduced by high-capacity PointGPT models are closely linked with the designs of PointGPT.\n\n\n|Methods|OBJ_BG|OBJ_ONLY|PB_T50_RS|ModelNet40|Params|Pre-training time|\n|-|-|-|-|-|-|-|\n|Point-MAE-B|94.2|93.9|90.2|94.2|120.1M|73 hours|\n|Point-M2AE-B|95.2|94.3|91.2|94.3|77.5M|156 hours|\n|PointGPT-B|95.8|95.2|91.9|94.4|82.1M|75 hours|\n\n**@Q4 - Morton code sorting**.\n\nWe advocate for Morton code sorting as the appropriate solution. **(1) Morton code sorting effectively preserves the adjacency relationship** between points, with adjacent points in one-dimensional space often being proximate to each other in three-dimensional space. Moreover, **(2) experimental results demonstrate the improved accuracy achieved by Morton sorting**. Ablation studies are performed on ScanObjectNN (OBJ_BG, OBJ_ONLY, PB_T50_RS) and ModelNet40 (1k points) datasets, using the PointGPT-S models. The performance is compared with the popular `KD-tree` sorting [1] and `Hilbert` code sorting [2] methods, indicating that Morton code sorting significantly outperforms other sorting methods. **In the future, we will strive to explore better sorting methods.**\n\n|Methods|OBJ_BG|OBJ_ONLY|PB_T50_RS|ModelNet40|\n|-|-|-|-|-|\n|KD-tree|89.0|88.6|81.3|93.2|\n|Hilbert|89.2|89.0|84.3|93.4|\n|Ours|91.6|90.0|86.9|94.0|\n\n[1] Gadelha, et al. \"Multiresolution tree networks for 3d point cloud processing.\" Proceedings of the European Conference on Computer Vision (ECCV). 2018.\n\n[2] Hilbert, et al. \"Über die stetige Abbildung einer Linie auf ein Flächenstück.\" Dritter Band: Analysis· Grundlagen der Mathematik· Physik Verschiedenes: Nebst Einer Lebensgeschichte (1935): 1-2."}, {"review_id": "7JTg6uhNcp", "reviewer": "Reviewer_4dwU", "summary": "This paper proposes PointGPT, a new self-supervised learning strategy for 3D representation learning. PointGPT follows the success of autoregressive pre-training paradigm in NLP and adapts it into 3D point clouds. With larger pre-training dataset and a post-pre-training stage, PointGPT achieves SOTA performance on different benchmarks.", "questions": "I think it's an interesting paper. Expect the author rebuttal to solve my concerns for increasing the rating.", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It's good to introduce GPT-style pre-training into 3D tasks. Although this idea is straightforward, this paper is the first attempt to my best knowledge.\n\n2. The motivation of GPT pre-training in 3D is also reasonable and well clarified (avoid shape information leakage).\n\n3. The writing, equations, and figures are easy to follow.", "weaknesses": "1. The experiment tables should incorporate the comparison of learnable parameters during training. It's very important to know the parameters of 3D pre-training to better judge the contribution.\n\n2. How about other methods with larger pre-training dataset and post-pre-training? This is also a necessary ablation study to the paper.\n\n3. It's better to cite this related paper in 'Methods using cross-modal information and teacher models':\nLearning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoders, CVPR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**@Q1 - Learnable parameters**. \n\nThanks for your suggestions! We will provide the number of learnable parameters (Params) in the revised article. For a fair comparison, we do not consider methods that utilize cross-modal information and teacher models, and mainly present the number of model parameters for well-performing single-modal self-supervised models here. \n\n|Models|Point-BERT|Point-MAE|Point-M2AE|PointGPT-S|PointGPT-B|PointGPT-L|\n|-|-|-|-|-|-|-|\n|Params|25.5M|29.0M|15.3M|19.5M|82.1M|242.2M|\n\n**@Q2 - Larger pre-training dataset and post-pre-training**.\n\n**PointGPT outperforms comparison methods with comparative training parameters and the same training data**. Specifically, experiments are conducted by re-training Point-MAE and Point-M2AE using the ViT-B configuration with collected larger datasets and post-pre-training stage. For a fair comparison, we focus on re-training well-performing single-modal self-supervised methods, Point-MAE, and Point-M2AE (`Point-MAE-B` and `Point-M2AE-B`). The results, encompassing ScanObjectNN (OBJ_BG, OBJ_ONLY, PB_T50_RS), ModelNet40 (1k points), number of parameters (Params), and total pre-training time, are presented below, which demonstrate the superior performance with high-capacity models. Additionally, we observe that Point-M2AE requires twice the pre-training time compared to our PointGPT, mainly due to its hierarchical architecture and dedicated multi-scale masking strategy.\n\n|Methods|OBJ_BG|OBJ_ONLY|PB_T50_RS|ModelNet40|Params|Pre-training time|\n|-|-|-|-|-|-|-|\n|Point-MAE-B|94.2|93.9|90.2|94.2|120.1M|73 hours|\n|Point-M2AE-B|95.2|94.3|91.2|94.3|77.5M|156 hours|\n|PointGPT-B|95.8|95.2|91.9|94.4|82.1M|75 hours|\n\n**@Q3 - Related work**. \n\nThanks for your suggestions! We will cite this related paper 'Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoders'. Additionally, we will include more recent articles to enhance the coverage of our related work."}, {"review_id": "a0iCJyOJp2", "reviewer": "Reviewer_LJmd", "summary": "This paper investigates self-supervised point cloud learning by introducing the GPT concept (e.g., point order) to the masked modeling framework. Specifically, the clustered point patches are arranged into ordered sequences based on spatial proximity. Then, the masked patches can be predicted without leaking their position (e.g., patch centers) in an auto-regressive manner. Experiments are conducted on object tasks.", "questions": "See the weakness", "limitations": "Non", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. In my view, the most attractive point is the enlarged training set, which brings significant performance improvement over the competitors. (cf. Tab 1). This justifies the importance of the pretraining dataset for point clouds.\n2. Significant performance on object tasks (e.g., shape classification), mainly due to the enlarged pertaining dataset.\n3. The introduced relative direction prompt via point order is also reasonable, which avoids the position leak of patch reconstruction.", "weaknesses": "1. Although the relative direction prompt via point order is highlighted, it brings marginal performance improvement (cf. Tab 4 (e)). However, the enlarged pretrained dataset contributes the most performance improvement and is understated. Therefore, in my view, a paper revision should be conducted.\n2. The proposed dual masking strategy sets some attention values to zero, which is similar to the popular attention dropout (cf. attention is all your need). Please clarify their differences.\n3. Besides the object-level tasks, more practical downstream tasks (e.g., detection and segmentation with scene input) are expected to be evaluated, especially when the scene datasets are used in the pretraining stage in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**@Q1 - The effectiveness of PointGPT's designs**. \n\n**(1) Relative direction prompt is able to enhance the generalization ability at a negligible cost**. Ablation experiments are conducted in a **high-capacity model training scenario**, where superior pre-training generalization is required. We re-train PointGPT-B using absolute positional encoding (`PointGPT-B-A`) on ScanObjectNN (OBJ_BG, OBJ_ONLY, PB_T50_RS) and ModelNet40 (1k points) datasets, and report the accuracy, the number of parameters (Params), and total pre-training time (Time). The results indicate that adopting the relative direction prompt leads to noticeable performance improvement at a negligible cost. We will incorporate the ablation experiments and results in the revised version to clarify the effectiveness of the relative direction prompt. \n\n|Methods|OBJ_BG|OBJ_ONLY|PB_T50_RS|ModelNet40|Params|Pre-training time|\n-|-|-|-|-|-|-|\n|PointGPT-B-A|95.2|94.5|91.3|94.2|82.1M|75 hours|\n|PointGPT-B|95.8|95.2|91.9|94.4|82.1M|75 hours|\n\n**(2)** Additionally, we conducted more experiments and verify that **the improvements introduced by PointGPT-B/L are closely associated with the designs of PointGPT**, which effectively address the information leakage and enhance the generalization ability. Specifically, additional experiments are conducted by re-training the well-performing methods Point-MAE and Point-M2AE using the ViT-B configuration with UHD and LHD datasets (`Point-MAE-B` and `Point-M2AE-B`). Experimental results exhibit the **significant performance improvement achieved by PointGPT under the same utilization of large datasets and model parameters**, despite Point-M2AE benefiting from the multi-scale features. For a fair comparison, we exclude methods using cross-modal information and teacher models. \n\n**(3) In the revised article, we will emphasize the importance of both PointGPT's designs and the training datasets**. This includes emphasizing the role of large datasets in improving accuracy in the Abstract and Introduction, as well as visually showcasing the performance gains achieved through enlarging the dataset size in the paper.\n\n|Methods|OBJ_BG|OBJ_ONLY|PB_T50_RS|ModelNet40|Params|Pre-training time|\n|-|-|-|-|-|-|-|\n|Point-MAE-B|94.2|93.9|90.2|94.2|120.1M|73 hours|\n|Point-M2AE-B|95.2|94.3|91.2|94.3|77.5M|156 hours|\n|PointGPT-B|95.8|95.2|91.9|94.4|82.1M|75 hours|\n\n\n**@Q2 - Differences between the masking strategy and attention dropout**. \n\n**(1) The effect on information leakage**. Although both methods involve the exclusion of random regions, the regions masked by the masking strategy remain consistent across different attention layers, thereby preventing information leakage from masked regions. In contrast, attention dropout varies across different layers, allowing the information dropped out in one layer to still be learned in other layers, resulting in information leakage.\n\n**(2) The impact on attention weight computation**. The masking strategy involves setting `the elements of the mask matrix` $M^{d}$ to 0. This results in the corresponding position being assigned a value of $-\\infty$ during the softmax calculation, entirely eliminating the influence of masked regions on attention weight computation. In contrast, attention dropout simply sets the `obtained attention weights` to 0, allowing dropped regions to retain their impact on the calculation of attention weights for visible regions.\n\n**(3) Masking strategy outperforms attention dropout significantly**. Ablation studies are conducted to directly exhibit the effect of masking strategy and attention dropout. We re-train the PointGPT-S model with the attention dropout (`Dropout`), and the accuracy on the ScanObjectNN (OBJ_BG, OBJ_ONLY, PB_T50_RS) and ModelNet40 (1k points) datasets is reported below. The results demonstrate the superior performance achieved by the masking strategy in comparison to the attention dropout. \n\n|Methods|OBJ_BG|OBJ_ONLY|PB_T50_RS|ModelNet40|\n|-|-|-|-|-|\n|Dropout|89.1|87.9|84.3|93.1|\n|Ours|91.6|90.0|86.9|94.0|\n\n**@Q3 - Practical downstream tasks**. \n\nThanks for your suggestions! We evaluate the performance of PointGPT in the **object detection task on the nuScenes dataset** and observe its adaptability to this task with minor adjustments for handling large-scale point clouds. Specifically, object detection experiments are conducted on the nuScenes dataset. PointGPT utilizes 12 attention blocks, each equipped with 8 heads, 128 input channels, and 256 hidden channels. Pre-training and fine-tuning of PointGPT are performed using the nuScenes dataset, excluding the post-pre-training stage. We compare PointGPT with several well-performing methods and report the major official metrics, mean Average Precision (mAP), and nuScenes detection score (NDS).\n\n|Methods|Reference|mAP|NDS|\n|-|-|-|-|\n|VISTA [1]|CVPR 2022|63.0|69.8|\n|Focals Conv [2]|CVPR 2022|63.8|70.0|\n|Transfusion [3]|CVPR 2022|65.5|70.2|\n|PointGPT||66.8|71.3|\n\nThe specific modifications are as follows: (1) Voxelization: Replacing KNN-based grouping with voxelization, considering each voxel as a point patch. (2) Shift window attention: Confining the receptive field of the attention within a window and employing the window shift operation. (3) Point sampling: Randomly sampling up to K points within each voxel as the reconstruction targets.\n\n[1] Deng, Shengheng, et al. \"Vista: Boosting 3d object detection via dual cross-view spatial attention.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[2] Chen, Yukang, et al. \"Focal sparse convolutional networks for 3d object detection.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[3] Bai, Xuyang, et al. \"Transfusion: Robust lidar-camera fusion for 3d object detection with transformers.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022."}]

Author rebuttal:
We thank all reviewers for their thoughtful feedback! We are pleased to find that reviewers 5U9K and 4dwU appreciate PointGPT as interesting and excellent work. Moreover, reviewers 4dwU and LJmd consider our motivation to be reasonable and well-explained, effectively mitigating information leakage. Furthermore, reviewers 4dwU and EUQH appreciate the novelty of our method as the first attempt of auto-regressive pre-training on point cloud tasks, prompting the rethinking of this approach in such tasks. We are delighted that all reviewers acknowledge the significant improvements and promising performance achieved by our approach. We have carefully considered all questions, concerns, and comments provided by reviewers and addressed all of them appropriately. We provide detailed responses to each review separately and believe that our responses address all of the reviewers' concerns.

Meta-review: This paper proposed a point clouds pretraining method named PointGPT, which extends the generative pretraining approach of NLP to point clouds. With point patch partitioning and sorting, point embeddings are fed into a transformer decoder for autoregressive prediction. In addition, a dual masking strategy is proposed to enhance the learned representation. PointGPT is evaluated on multiple datasets and show good performance.

The authors are recommended to include all added experimental results in the rebuttal to EUQH, 5U9K, LJmd, 4dwU.
Decision: Accept (poster)