PCP-MAE: Learning to Predict Centers for Point Masked Autoencoders
Authors: Xiangdong Zhang
Shaofeng Zhang
Junchi Yan
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Self-supervised Learning
Point Cloud
Transfer Learning
Masked Autoencoders
Primary area: machine_vision

Abstract: Masked autoencoder has been widely explored in point cloud self-supervised learning, whereby the point cloud is generally divided into visible and masked parts. These methods typically include an encoder accepting visible patches (normalized) and corresponding patch centers (position) as input, with the decoder accepting the output of the encoder and the centers (position) of the masked parts to reconstruct each point in the masked patches. Then, the pre-trained encoders are used for downstream tasks. In this paper, we show a motivating empirical result that when directly feeding the centers of masked patches to the decoder without information from the encoder, it still reconstructs well. In other words, the centers of patches are important and the reconstruction objective does not necessarily rely on representations of the encoder, thus preventing the encoder from learning semantic representations. Based on this key observation, we propose a simple yet effective method, $i.e.$, learning to \textbf{P}redict \textbf{C}enters for \textbf{P}oint \textbf{M}asked \textbf{A}uto\textbf{E}ncoders (\textbf{PCP-MAE}) which guides the model to learn to predict the significant centers and use the predicted centers to replace the directly provided centers. Specifically, we propose a Predicting Center Module (PCM) that shares parameters with the original encoder with extra cross-attention to predict centers. Our method is of high pre-training efficiency compared to other alternatives and achieves great improvement over Point-MAE, particularly surpassing it by \textbf{5.50\% on OBJ-BG, 6.03\% on OBJ-ONLY, and 5.17\% on PB-T50-RS} for 3D object classification on the ScanObjectNN dataset. The code is available at \url{https://github.com/aHapBean/PCP-MAE}.

PCP-MAE: Learning to Predict Centers for Point Masked Autoencoders
Xiangdong Zhang", Shaofeng Zhang", Junchi Yan Dept. of CSE & School of AI & MoE Key Lab of Al, Shanghai Jiao Tong University (zhangxiangdong, sherrylone, yanjunchi/@sjtuu aeu..uu.uuuu...
Abstract
Masked autoencoder has been widely explored in point cloud self-supervised learn- ing, whereby the point cloud is generally divided into visible and masked parts. These methods typically include an encoder accepting visible patches (normalized) and corresponding patch centers (position) as input, with the decoder accepting the output of the encoder and the centers (position) of the masked parts to reconstruct each point in the masked patches. Then, the pre-trained encoders are used for down- stream tasks. In this paper, we show motivating empirical result that when directly feeding the centers of masked patches to the decoder without information from the encoder, it still reconstructs well. In other words, the centers of patches are impor- tant and the reconstruction objective does not necessarily rely on representations of the encoder, thus preventing the encoder from learning semantic representations. Based on this key observation, we propose a simple yet effective method, i.e., learning to Predict Centers for Point Masked AutoEncoders (PCP-MAE) which guides the model to learn to predict the significant centers and use the predicted cen- ters to replace the directly provided centers. Specifically, Center Module (PCM) that shares parameters with the original encoder with extra cross-attention predict centers. Our method is of high pre-trate pre-training efficiency compared to other alternatives and achieves great improvement over Toint-MAË. particularly surpassing it by 5.50% on OBJ-BG, 6.03% OBJ-ONLY, and 5.17% % on PB-T50-RS for 3D object classification on the ScanObjectNN dataset. The code is available at https: //gi /////ttuuub com/ aHHppBean/PCP-MA.
propose a Predicting
Introduction
Point clouds are a widely used representation of 3-D objects, offering a rich expression of their geometric information. This versatility has led to their broad adoption across various application scenarios, including autonomous driving [28). robotics [10] 330, and the metaverse [32]. In the early developing stage of point cloud understanding, there are lots of related work proposed to enhance the ability of networks for understanding point cloud 13 18 26
39 where the networks
typically need fully-supervised training from scratch. However, point cloud data difficult to annotate compared to the data in 2-D vision and NL.P owing to the complexity of the toughness of discriminating them. leads to a phenomenon called data desert [9] in 3-D which refrains the development of these sully-supervised methods. Recently, numerous self-supervised learning (SSL) methods 47 24 9 277 were proposed for point cloud understanding to alleviate the negative effect brought by data desert because self self-supervised learning methods can utilize unlabeled data effectively by performing designed pretext task (e.g., reconstruction- based or contrastive based)) for pre- pre-training and the learned semantic representation benefits the performance of downstream tasks.
"Fqual contribution. =Corresponding author. This work supported by Shanghai Municipal Science and Technology Major Project under Grant SHZDDX00000
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Masked autoencoders (MAEs) [16 represent prominent and robust framework within self- supervised learning (SSL), which demonstrates remarkable scalability and superior performance. The architecture of MAF is distinctly asymmetric, featuring an encoder and a decoder. The typical process involves dividing an image into patches, which are then selectively masked prior to encoding. The encoder only accepts visible patches as input along with their positional embedding, í.e. embedding of patch indices. The decoder receives both the latent representations of the visible patches and the tokens representing the masked patches, along with their corresponding positional embeddings. Following MAE in 2-D, Point-MAE 24 and its variants 48 14 |27] are proposed, which divide point cloud into patches where points in patches undergo normalization, achieved by subtracting the coordinates corresponding centers and apply the recnsstrcottttiii paradigm in 3-D domain.
However, different from the indices of image patches (the positional em- beddings in 2-D) are fixed for all in- put images [66], position embedding
Prodica
in point cloud is calculated using the coordinates of the centers belonging to patches, where the coordinates of the centers usually contain rich geo- metric and semantic information of point clouds. Thus we ask: Should the centers (i.e., positional embed- ding) for masked patches be directly given when performing masked re- construction in the point clouds field just like the way in 2-D vision? We answer this question with an interesting phenomenon observed through experimental results in Fig. I We first set the mask ratio to 100% in Point-MAE for pre-training on ShapeNet K i.c., the encoder removed to verify whether the point cloud can be reconstructed by only feeding the masked positional embeddings of the center points to the decoder after training. Intriguingly, the reconstruction results are pretty well, which are shown in Fig. ! This is irreproducible for 2-D MAE since when masking 100% patches and only providing the positional embedding of the masked patches (i.e., 2-D indices of patches), 2-D MAE cannot identify what the images are (as all the images with the same resolution share the same positional embeddings).
Images
Point Clonds
Figure I:
Illustrations of MAE reconstruction results for 2-D> MAE and Point-MAE when masking ratio equals to 100%.
The gap between 2-D and point cloud data reveals that the coordinates of the centers (i.c. positional embeddings) are essential in the point cloud field, meaning that the decoder can even abandon the output of the encoder and still reconstruct well. Thus, the reconstructing objective may make the encoder unable to learn semantic features, and we argue that the answer to the previous question is: The centers for masked patches can not be directly provided and should be utilized in smarter and more proper way. To this end, we propose a method, which learns to Predict Centers for Point Masked AutoEnco,rrs, termed as PCP-MAE. While existing MAE-based methods [24 27 48 51]
overlook the importance of centers during pre- pre-trainin and directly leak the masked centers for the decoder, our PCP-MAE learns more semantic representation by performing point cloud reconstruction under the auxiliary of learning to predict centers, using predicted centers replace real masked centers.
The overview of the proposed PCP-MAE is illustrated in Fig. 2 Specifically, we propose custom pre-training task that forces the encoder to not only learn representations for visible patches but also predict the centers of the masked ones. It is achieved by feeding both visible and masked patches to the encoder. Visible patches (with centers) perform self-attention in each block as in Point-MAE, while masked patches (without centers) leverage a cross-attention mechanism simultancously to acquire center information from themselves and the visible ones. Additionally, a dedicated objective introduced to minimize the difference between the predicted masked centers and the ground truth values (positional embeddings). Furthermore, the predicted positional embeddings of the masked centers are used to replace the real positional embeddings, which will be fed into the decoder. This allows the network to not only encode visible centers effectively but also learn the inter-relationships between visible and masked centers. Essentially, the encoder learns to infer the missing information, leading to a more robust representation and improved performance in downstream tasks achieved through fine-tuning. The main contributions include:
1) Center-aware objective makes the pre-training trivial. We identify the difference between 2-D (image) and 3-D (point cloud), i.e., positional embedding (PE) in 2-D represents indices of patches,

---

Table 1: Comparisons between our PCP-MAE and existing Singlc/Cross-Modde MAE-based methods in terms of method features, pre-training efficiency, and performance on standard SSL benchmarks.
Methodl
Masked Centers
Single/Crus-
Pre-tmined Model
Pre thetrrining elficinccy
Performace
Leokage
Modal
Needed
4 Parms
GFLOPS
Time (h)
ScanOtjeciNN
ModelNNeti0
Poin: MAE 21
Single
29.0 (bwseline)
2.0 (buselii)))))
8.7 (buseline)
85.18 (buseline)
93:2 (baseline)
Poin-MMAEE 50
Single
X
15.3 (0.53x)
(12()
19.1 (2.2%)
80.43 (71.55)
93.4 (10.2)
Fuin-FEMAEE [5
Single
(1,5())43))
4.4x222))
32.2 (3.x)
90.22 (15.44)
9100
ACT [
Closs
135,5 (4,77)
34,8 (33,5))
88.21 (73.13)
93.2 (10.0)
17P-MAF 5D
Cross
74,9 (2.58x)
14.6 (7.3x)
42.6 (4.9x)
90.1 (74.9))
93.7 (10
ReCon
Cross
140,9 (4.85%)
18.2 (9.1x)
44.4 (5,11)
90.63 (13.45)
(10.9)
PCP-MAE
Single
29.5
12.3(114
(15.17)
94.0 (10.8)
while PE in point cloud provides coordinates of patch centers, where the indices usually provide much less information than coordinates. As shown in Fig. I point clouds are well reconstructed by the decoder under 100% masking ratio, indicating that the decoder does not necessarily rely on the representations learned by the encoder. This phenomenon does not hold for 2-D MAE, suggesting that existing MAE-based methods in point cloud adopt trivial reconstruction pre-training task.
2) A new center-unaware pretraining method. We propose PCP-MAE, a framework that not only performs point cloud reconstruction but also avoids direct leakage of semantically rich centers by guiding the network to learn to predict them, which is much more challenging than previous cooonstrcttionoollybbiiee -based methods, and foreing the model to learn more semantic representations. Besides, the Predicting Center Module (PCM) is introduced to predict the positional embedding of the centers, which shares parameters with the encoder to prevent an increase in parameter count.
3) Significant improvement over Point-MAE and high pre-training efficiency. Compared to other MAE-based methods, our method only introduces a few parameters increase and affordable extra pre- training time cost, remaining highly efficient for pre-training as shown in Tab. [ Notably, extensive experiments demonstrate the effectiveness and the efficiency of our PCP-MAE over other MAE-based methods by improving Point-MAE. Specifically, our method surpasses baseline Point-MAE by large margins of 5.50%, 6.03%, and 5. 5.17% across three ScanObjectNN variants, respectively.
2 Related Work
Self-super learning for Point Cloud. Inspired by the recent successes of Self- Self-Supervised Learning (SSL) in NLP and 2-D vision, an increasing amount of research has been conducted to explore the potential and strengths of SSL in the realm of 3-D vision (24 47
41
Existing 3-D
SSL methods can be mainly divided into two categories: contrastive learning 41 17
29 [9] and
51. PointContrast 41 pioneers the contrastive learning in 3-D by borrowing the idea of contrastive learning in 2-D and performing point-level invariant mapping learning on two transformed views of the given point cloud. CrossPoint 1 advances this approach by engaging in intra-modal learning while also introducing an auxiliary cross-modal contrastive objective that facilitates the learning of transferable 3-D point cloud representations through 3D- 2D correspondence. Generative learning in 3-D typically trains autoencoders to learn semantic latent representation during pre-training with a pre-designed task, i.e.. reconstructing the original input. Inspired by BERT, POiR [47] firstly devises a Masked Point Modeling (MPM) task to pre-train point cloud Transformers auxiliary by point cloud Tokenizer with discrete Variational AutoEncoder (dVAE). Masked Autoencoders (MAE) [16), which mask random patches from the input image and reconstruct the missing patches at the pixel level, have demonstrated substantial potential and established a significant milestone in the field of SSL.
generative learning 47 24 50
MAE-based methods for point cloud. Following the masked-reconstruccie paradigm established by MAE in vision, Point-MAR 24 is proposed, which pioneers designing a neat and efficient architecture entirely built upon standard Transformer blocks for point cloud understanding. Motivated by the success of Point-MAE, numerous works based on it are proposed (50) 9 ing notable improvement compared to Point-MAE. On the basis of MAin Point-M2AE [50] tailors the encoder and decoder into pyramid architectures, which allows for progressive modeling of spatial geometries, facilitating the capture of both intricate details and overarching semantics of 3D shapes. ACT (9) as an MAE--ased method, acquires knowledge from other modalities by adopting cross-modal auto-encoders as teachers. ReCon 22 harmoniously combines the generative framework (MAF-based) and contrastive framework to share the merits of them. Joint-MAE [14 enables better cross-modal interaction by constructing two hierarchical 2-D-3-D embedding modules. 12P-MAE (51) introduce 2-D guided masking strategy and 2-D semantic reconstruction apart from
show-
3

---

Erooder
Thundernaa
CD&C, Loss
FPS
Normalization Masking Embed.
Decoder
Shared Weights
Tousdoure Bleck
PCM
Tranfoene R
D
D
Visible
Uncodod Visible
Projector
Masked Patch Token
Leamable Mask Token
Predicting Centers
Lg Loss
Pusitional Embesiting
Predicted Masked PE
Point Cloud Reconstruction
Figure 2: Overview of the proposed PCP-MAR. After patch division, the centers and normalized patches divided into visible and masked parts, with center coordinates embedded into positional embedding (PE) and patches embedded into tokens (emheddings). The encoder accepts visible tokens and PE input, performing self-attention. Simultaneously, the weight-shared PCM (Predicting Center Module) performs cross-attention (masked tokens as query and visible along with masked tokens as key and value) acquire knowledge to predict the positional embeddings of the masked patches. CD-L2 refers to the l2 Chamfer Distance loss function [1]
point cloud reconstruction. Point-FEMAA consisting of a global branch and a local branch to capture latent semantic features, with extra modules Local Enhancement Modules introduced. We find almost all existing MAE-based methods focus on structural improvement, employing the same reconstruction objective as Point-MAE. However, as shown in Fig. [ we claim again that this reconstruction objective not suitable to directly adopt from 2-D to point clouds.
48 extends global random masking two branches
3 PCP-MAE
3.1 Patches generation and masking
Patches generation. Patch-based learning paradigm [47 24 27 has been proven to be more effective than directly consuming the whole point cloud [25] 26 empirically. We adopt the patch- based paradigm and divide the input point cloud into point patches via Farthest Point Sampling (FPS) and K-Nearest Neighborhood (KNN) algorithm following previous work (24] Specifically, given point cloud with p points X Px3 FPS is first applied to sample centers C from points. Then, we adopt KNN to select & nearest points corresponding centers to consist n point patches P:
C FPS(X), P KNN(X, C), C E RRx P E Buxxx3
(1)
Coordinates of points in the point patch are normalized with the corresponding center, thus enabling better separation between patches and centers.
Masking. With a pre-defined masking ratio m, we apply global random patch masking to point patches. The masked patches are denoted as P.. E [R [[mn]xxx3 and the visible patches are denoted as P. (1-mm))]] xkx3, where [.] and [.] represent the floor function and the ceiling function, respectively. For simplicity, will omit these two symbols in the subsequent text. The corresponding centers are denoted C. E gmmxx and C, g (11mm)xx3
Embedding, P is embedded via a lightweight PointNet (25 yield encoded tokens E, following Point-BERT 47 which can be expressed as follow, where D is the hidden dimension of the networks.
E,- PointNet(PP),, E, RoXD
We calculate fixed positional embedding for each center using the popular form 16 detailed in Eq. (3), which we call sin-cos positional embedding (sin-cos PE). For each triplet of coordinates
(z, y, ) for the center:

---

x
PEg-
sin
008
sin
D/6
e
2x
p/o
2x D/0
I
I
COS
2 D7C
sin
cos
where PEx E 1x/3 Then we concatenate PE PEy and PE₃ to obtain PE E RIXD Then feed the obtained sin-cos PE into learnalee MLP called PEM (Positional Embedding Module) to yield the final PE, which is different from the operation Point-MAE [24]. On the one hand, the sin-cos PE enables more sparse positional information compared to 3-dimension coordinate and thus is more suitable as the predicting target of our proposed task just as illustrated in Sec. 4.3 On the other hand, applying the sin-cos positional embedding to both visible and masked patches can better align the usage of known visible PE with the predicted masked PE. The process can be expressed as:
PEgin-cos sin-cos PE(C.),
PE COS E ( (I-)
PE m
Ppein-cos cos = sin-cos PE(C,,,,
PE pieein-cos E R m
(5)
PE, PPMMPPPPP (PE B c0s),
PE E (1-mjxD
(6)
PE (PE sin-cos).
PE E m
3.2 Encoder
We employ an encoder that consists of standard Transformer blocks. Given the encoded visible tokens E,, masked tokens Em and visible positional embeddings PE,, the encoder performs olffaattention encode E, while performing cross-attentin utilizing E,,, input to acquire information of masked centers from itself and the visible representations. Note the self-attention and cross-attention share the parameters of Transformer blocks to avoid increasing parameters.
Visible tokens encoding. The latent representation the visible parts can be obtained by inputting Eg, PE, the encoder which utilizes self-attention mechanisms, The process can be formulated as:
Tp Eacoder(E., PE.)
Specifically, the self-attention in i-th Transformer block of the encoder can be expressed as:
QuK,
T, Attn(QuQQu.... K., V.) SSofMMax
V
where Q. T:-IWo. K. T. WK and V. T.-1Wy and the WQ. WK. Wy are learnable parameters. Note Tn E, PE,
Modules for learning to predict centers iee, PCM. The PCM shares parameters with the encoder and accepts only masked patches F., as input. Apart from the shared parameters of Transformer blocks, additional MLP for projecting latent predicted PE to masked center embeddings E is introduced into PCM. Based on the encoded masked patches, the PCM aims to recover the centers of masked patches by acquiring knowledge from both the masked and visible representations. This can be straightforwardly understood in form where local patterns (without telling where they are) and visible global patterns are given, and the task is acquiring positions of local patterns from the visible patterns based on the known local patterns. The cross-attention mechanisms are performed:
PEprred PCM(Em. E., PE.))
(10)
Specifically, for the i-th Transformer block in PCM, the cross-attention can be formulated as:
Qm Kotm
PE m
= Attn(Qm, Kutm, Votm) = Sof
V Vutm
(11)
VD
where Qm = PE m WQ. K,, = PE I and V.. = PE Ww and the Wq. WK are shared parameters. We concatenate Q and Qm to obtain Qu+m and it's also the case for Ko4m and V mtt Note PEO is the input of PCM, i.e., PE E
5

---

3.3 Decoder
The decoder also consists of Transformer blocks and is used for point cloud masked recon- struction. The decoder in existing MAE-based methods such as Point-MAE [24), ReCon [27]. Point-FEMAE 48 typically take T.,, PE.. PE.. as input along with learnable mask tokens [M] E R can be expressed as Hm Decoder([M] PE., Tu, PE.) where the centers for masked patches are directly provided. However, the centers are extremely important in point cloud as shown in Fig. Therefore, instead of doing that, we use the PE to replace the PE.. which means we use what the network learns but doesn't 't leak the ground truth of positional embeddings to the decoder. The process can be written as:
m Decoder(M PEpred T,, PE.))
(12)
where sg(+) means the stop- gradient operation and Hm E R Stop-gradien is applied to prevent the decoder from finding reconstruction shortcuts by back-propagating through the PEgre that modifies the weights of PCM, as illustrated in the ablation study Sec. [4.3] Finally. following Point-MAE, we adopt a projection head consisting of one MLP layer to predict the coordinates of normalized masked points in the input point cloud:
pred Reshape([inear(HL,, P pred mmnxkxx
(13)
3.4 Objective function
The objective function L consists of two parts, i.e,, recovering the centers of masked patches and the points in every masked patch. The L can be written as L TP + RRecon 1) is a scaling factor.
Learn to predict centers. We compute the loss for predicting centers using the lz loss function:
LPc
PEpped
PE,.I
(14)
mnD
The information of centers is removed when generating point cloud patches through normalization. thus posing no possibility for PCM to find shortcuts to recover the masked centers PEm based on Em. E, and PE.. And recovering the PE.. is non-trivial because it needs the PCM to encode masked patches well and demands the visible representation T, possessing not only intra-informant but also inter-information between itself and the unseen masked ones. In other words, minimizing the LPC enables the visible representation T. to contain enough information to infer the distribution of masked centers which is infeasible in the original Point-MAR.
Point cloud reconstruction. The reconstruction loss Litecon is computed using the l2 Chamfer Distance loss function [1I. written as:
Liecon
min la
bl2
Σ
min ||a
bll2
(15)
P Ppred aEP pred
P
aCP Pred
PEP
Experiments
We conduct experiments including object classification, few-shot learning and segmentation to demonstrate the superior performance of our method over Point-MAE. With slightly longer pre- training time than Point-MAE, PCP-MAE achieves much higher performance than it and even surpasses other more complex methods, achieving tftte-of-the-art in some tasks, e.g., ScanObjectNN classification (OBJ-BG; OBJ-ONLY) and ModelNet40 few-shot (5-way, 10/20-shot; 10-way, 20-shot). Ablation studies are conducted to illustrate the properties of our proposed PCP-MAE.
4.1 Pre-training setups
Model architecture. The backbone of our pre-trained PCP-MAE consists of standard Transformer blocks where the encoder has 12 Transformer blocks and the decoder has 4, aligned with Point-
MAE 24). The hidden dimension of Transformer blocks is 384 and the number of heads is 6.
Pre-training dataset. PCP-MAE is pre-trained on ShapeNet B which consists of about 51,300 clean 3-D models, covering 55 common object categories.
Experiment details. For each input point cloud, we first apply scale and translate operations, followed by rotation for pre-training data augmentation. After sampling 1024 points via Farthest

---

Table 2: Classification accuracy (%) on ScanObjectNN and ModelNet40. The parameters of inference models #P (M) are reported. Three variants are evaluated on ScanOhjectNN and the accuracies obtained on ModelNet40 are reported for both without and with voting.
ScanObeeNNN
ModelNetiO
Methods
References
#P
Input
OBI BG
ORI ONLY
PR T50 RS
Input
we Vate
w/ Vote
Superiied Loarning Only
PointNet 251
CVPR 2017
3.5
1k Points
73.3
79.2
68.0
Ik Points
89.2
PointNet-x
R6
NerulPS 2017
1.5
1k Points
82.3
84.3
77.9
Ik Points
90.7
DGCNN
1.8
Points
78.1
Points
92.9
SimpleView
[4
ICML, 2021
6 Images
80,5:20.3
6 Images
95.9
MVTN [13
ICCV 2021
11.2
20 Images
92.6
92.3
82.8
12 Images
93.8
PointML.P
2022
Points
85.4+0.3
Points
94.5
STR 49
ICASSP 2023
20 Images
87.8
20 Images
93.9
P2P-HorNet
56
NeroIPS 2022
195.8
40 Images
89.3
40 Images
94.0
with Single-ffutol Self-Supenised Teurning
Point-BEKT
+x
CVPR 2022
22.1
Ik Points
87.43
88.12
83.07
Ik Points
92.7
93.2
MaskPoint
20
ECCV 2022
2k Points
89.30
88.10
1k Points
93.8
Point-MAE 24
ECCV 2022
22.1
2k Points
90.02
88.29
85.18
Ik Points
93.2
93.8
Point-M2N M2AE
50
NeurlPS 2022
15.3
2k Points
91.22
88.81
86.43
Ik Points
93.4
94.0
PointGPT 4
NerulPS 2023
19.5
2k Points
91.60
90.00
86.90
Ik Points
94,0
FEMA
4R
AAAI 2024
27.4
2k Points
95.18
93,29
90.22
Ik Points
94.0
94.5
PCP-MAE
Points
95.52
94.0
94.2
Improve (over Poinr-MAE)
+5.50
+6.03
+5.17
-0.8
+0.4
with Cross Al odal Self Supervised Learning
ACT 9
ICLR 2023
22.1
2k Points
93.29
91.91
88.21
Ik Points
93,2
93.7
Joint-MAE
14
IICAI 2023
2k Points
90.94
88.86
86.07
Ik Points
94.0
51
CVPR
Points
93.7
TAP X8
ICCV 2023
22.1
2k Points
90.36
89.50
85.67
ReCon 22
ICML 2023
43.6
2k Points
95.18
93.63
90.63
1k Points
94.1
94.5
Point Sampling (FPS) from the input point cloud, it is divided into 64 point patches, each containing 32 points, selected using FPS and K-Nearest Neighbors (KNN). The PCP-MAE is pre-trained for 300 epochs using an Adam W optimizer (22) with batch size of 128. The initial learning rate is set 0.0005, with a weight decay of 0.05. A cosine learning rate decay scheduler [2]) is utilized. Check the detailed experimental settings in Appendix A
4.2 Fine-tuming on downstream tasks
3-D object classification on real-world dataset. We showcase the transferability of our models by evaluating them on real-world 3D object dataset. Specifically, we transfer our pre-trained model ScanObjectNN 33 for classification, a dataset renowned for its classification complexity,
encompassing around 15,000 real-world objects spanning 15 diverse categories. Our experimental evaluations encompass three variants: OBJ-BG, OBJ-ONLY, and PB-T50-RS. Rotation is applied for data augmentation during training following (27 48). The Tab. 2 presents the results, showing the superior performance of our PCP-MAF, which obtains great performance not only over Point-MAE but also over other more complex MAE-based methods and achieves state-of-the-an aht OBJ-BG and OBJ-ONLY. Specifically, PCP-MAE outperforms Point-MAE by 5.50% 6.03 %. 5.17 % on three variants and performs competitively or better than the leading cross-modal method ReCon 277
3-D object classification on a dataset with clean objects. We assess the efficacy our pre-trained model for object classification on the ModelNet40 dataset [0] ModelNet40 comprises 12,311 meticulously crafted 3-D CAD models, representing 40 distinct object categories. Standard random scaling and random translation are applied for data augmentation during training. Notably, the input point clouds exclusively contain coordinate information, without supplementary normal information. The results are presented in Tab. 2 including results with and without voting. Improvements in PCP-MAE over Point-MAE can be observed, 0.8% and 0.4% for w/o and w/ voting respectively.
Few-shot learning. We conduct few-shot learning experiments on the ModelNet40 dataset, following established protocols 24 3T. The experiments are structured "w-way, s-shot" and consist of four components, where w E (5. 10). representing the number of randomly selected classes, and E (10, 20). indicating the number randomly selected samples for each w. Each component undergoes 10 independent trials. The reported results include the mean accuracy and standard deviation, as detailed Tab. 3 Results show that with limited downstream finetuning data, our PCP-MAE exhibits outstanding generalization ability among existing single-modal and cross-modal methods, achieving SOTA in 5-way, 10-shot; 5-way, 20shot; 10-way, 20-shot experiments. Specifically, 1.1%, 1.3%, 0.9%, 0.9% performance gains over Point-MAE on four settings are obtained.
7

---

Table 3: Few-shot classification results on ModelNet40, We perform ten separate trials for each experimental setting and the mean accuracy (%) and standard deviation are reported.
w-way
10-way
Methods
10-shot
20-shot
10-shot
20-shot
Supervised Learning Only
PointNet 25
52.0+-3.8
57.8+
46.6+4.33
35.2++4.8
DGCNN 37
31.6++2.8
40.8+4.66
19.9+2.1
16.9+1.5
OcCo
35
90.6+
92.5++19
82.9+1.3
86.5+2.2
with Single-Moda Self-Superoised Representation Learning
Point-BERT 44
94.6+3.1
96.3++2.7
91.0+5.44
92.7++5.1
MaskPoint 20
95.0+3.7
97.2+1.7
91.4+4.0
93.443.5
Point-MAF 224
96,3+2255
97.8+1.8
92.6+4.1
95.0+3.0
Point-M2A1 [50
96,8+188
9.,3+1144
92.3+4.55
95.0++.0
PointGPT 4
96,1+2.0
98.61111
92.61:466
95,2
Point-FEMAE
48
97.2++1.9
98.6+1.3
94.0+3..3
95.8+2288
PCP-MAE
2.3
99.1+
93.5+3.7
95.9+2.7
Improve (over Point-MAF)
+1.1
+1.3
+0.9
+0.9
with Crose-Modal SSpp-rrrd Representation Learning
ACT 9
96,8++2.3
98.0=1.4
93.34-4.0
95.6++2.8
Joint-MAR 11
96.7++2.2
97.9+1.9
92.6+3.7
95.1+
12P-MAR 51
97.0+1.8
98.3+1.3
92.6+5.0
95.5+3.0
TAP 38
97.3+118
9.9
93.1+2.6
9.0
ReCon 27
97.3+1.9
9.29
93.3+:399
95,8+3.0
Object part segmentation. We conduct part segmentation experiments on the ShapeNetPart 46 to validate the effectiveness of our PCP-MAE method. ShapeNetPart is used to evaluate the learning capacity of models toward knowledge of detailed shape semanties within 3D objects. It comprises 16,881 objects across 16 categories. In alignment with prior research 24. we sample 2048 points from each input point cloud and partition them into 128-point patches. As indicated in Tab. our PCP-MAR yields comparable performance over peer methods and outperforms Point-MAF by a large margin. Particularly, PCP-MAE achieves 84.9% Cls.mloU and improves Point-MAE by 0.7%.
3-D scene segmentation. Scene segmentation is a challenging task, especially in large-scale 3-D scenes, as it demonstrates the ability of models comprehend contextual semantics and intricate local geometric relationships. We conduct 3-D scene segmentation on the S3DIS dataset (2], which provides densely annotated semantic labels for point clouds. The results can be observed in Tab, PCP-MAE shows better ability under scene segmentation scenarios and outperforms Point-MAN by 1.1% and 0.5% in mAce and mloU, respectively.
4.3 Ablation studies
We conducted several experiments demonstrate the efficacy of PCP-MAE, with results three variants of ScanObjectNN (%) reported, maintaining experimental settings consistent with Sec. 4.2 PCP-MAE is pre-trained on ShapeNet before fine-tuning on ScanObjectNN.
Main components the PCP-MAE. Compared to Point-MAE which only performs point cloud reconstruction, our PCP-MAE also adopts reconstruction as a pre-training target and differs from Point-MAE mainly in two aspects: 1) Adding PCM which shares parameters with the encoder to guide the encoder to learning to predict centers for masked patches. 2) Instead of directly providing the ground truth PEm. replace it with the predicted results P P.ppred To validate the effectiveness of these two operations in PCP-MAE, we conduct experiments by varying the utilization of PCM and the predicted PE on the performance our PCP-MAE. The Tab. presents the results, showing that only adopting predicting centers pertaining target outperforms the trom-scrateh baseline which means the task using PCM proposed by us is meaningful as an isolated pre- pre-trainin training task. Results also indicate that while guiding encoder learning to predict centers improves the performance (w/ PCM), replacing PE,, with PEpred further enhances the performance.
Predicting Targets. The centers can expressed in two forms: coordinates with a 3-dimensional representation (x. z) and positional embedding (PE) with a D-dimensional representation obtained by embedding the coordinates, where / refers to the dimensionality of the model. Therefore, there at least two available targets for center prediction. We claim that the PE form is more suitable
8

---

Table 4: Segmentation Results on ShapeNetPaat and S3DIS Area 5: Mcan intersection over union for all classes Cls. (%) and all instances Inst.mloU (%) for Part Segmentation; Mean accuracy mAcc (%) and IoU mloU (%) for Semantic Segmentation.
Part Seg.
Semantic Seg.
Methods
Cls.mloU
InstmloU
mAcc
mloU
PointNet
25
80.4
83.7
49.0
41.1
PointNet+
26
81.9
85.1
67.1
53.5
DGCNN
37
82.3
85.2
PointMLP
23
84.6
86.1
with Single-Modal Modal Sel fSppevvve Representation Learning
Transformer
34
83.4
84.7
68.6
60.0
CrossPoint
85.5
Point-BERE
47
84.1
85.6
MaskPoint
20
84.4
86.0
Point-MAIF [2
84,2
86.1
69.9
60.8
PPin 4
84.1
86.2
Point-FEMAE
84.9
86.3
PCP-MAE
84.9
86.1
71.0
61.3
Improve (over Point-MAE)
+0.7
+0.0
+1,1
+0.5
with Croas-Moda odal Sel f-S Suerrvised Representation Learning
ACT 9
84.7
86.1
71.1
61.2
ReCon 27
84.8
86.4
Table 5: Effects of the main components in the proposed PCP-MAE include the pre-training targets (point cloud reconstruction, learning predict centers with PCM) and the operation that replaces the PE. with predicted centers PEpred
The default setting is marked blue.
reconstruction
w/ PCM
using PE!..!
OBJ_BG
OBI_ONLY
PB _S__SS_RSS
X
X
90.01
(scratch)
88.64 (scratch)
83.93 (scratch)
X
X
92.42
92.42
88.13
X
/
/
91.73
92.42
88.37
X
92.94
92.42
88,65
X
/
90.01
88.66 64.64
84.73
94.32
93.11
89.38
95.52
94.32
90.35
Table 6: Effects of predicting targets.
Table 7: Effects of stop gradientt operation.
predicting targets
OBJ_BG
[UJ_ONLY
PB_150_RS
stop-gradient OBJ_BG
OBJ SeJ_ONLY
PB PB_T50_RG
coordinates
94.32
92.25
88.68
92.42
91.56
87.36
95.52
94.32
90.35
in this situation because it provides sparser positional information compared to the 3-dimensional coordinates, and it represents high-dimenn -ing--immnsional semantic space that is casier and clearer for the model learn. The experiment results reported in Tab. Susubttatee this claim.
noo-grraieent operation. To prevent the decoder from finding shortcuts during pre-training, we apply stop-gradient operation sg(-) the predicted centers before feeding them into the decoder. This stops the back pockppropagttion gradients to the PCM, ensuring that any reduction in reconstruction loss does not falsely adjust the predicted centers PE as the reconstruction loss only contains information for points in patches but semantic information for the centers. Experimental results, shown in Tab. 7 indicate a significant performance drop when the stop-gradient operation omitted.
Conclusion
In this paper, we first identify the disparities positional embeddings between 2-D vision and 3-D point cloud, finding that direct leakage of masked centers makes the pre-training in Point-MAF and existing MAE-based methods trivial as shown Fig. T To address this issue, we introduce PCP-MAE, which adds novel objective guide the encoder to predict positional embeddings for

---

the masked centers, enabling the encoder to learn much more semantic representations. A Predicting Center Module which uses cross- coss-attention and shares parameters with the encoder is proposed to achieve this. Exhaustive experiments on SSL. benchmarks showcase the superior performance of our method compared to Point-MAF, setting new SOTA results across various tasks.
References
[1] M. Afham, I. Dissanayake, D. Dissanayake, A. Dharmasiri, K. Thilakarathna, and R. Rodrigo. Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9902-9912, 2022.
[2] I. Armeni, O. Sener, A. R. Zamir, H. Jiang, I. Brilakis, M. Fischer, and S. Savarese. 3d semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1534-1543, 2016. 2016.
[3] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, al. Shapenet: An information-rich 3d model repository. arXiv preprini 1rXiv:1512.03013 2015.
[4] G. Chen, M. Wang, Y. Yang, K. Yu, L. Yuan, and Y. Yue. Pointgpt: Auto-regressively generative pre-training from point clouds. Advances in Neural Information Processing Systems, 36, 2024.
[5] Y. Chen, J. Bu, and J. Yan. Unveiling the matthew effect across channels: Assessing layer width sufficiency via weight norm variance. In NeurIPS, 2024.
[6] Y. Chen, Q. Ren, and J. Yan. Rethinking and improving robustness of convolutional neural networks: shapley value-based approach in frequency domain. In NeurlPS, 2022.
[7] Y. Chen and J. Yan. What rotary position embedding can tell us: Identifying query and key weights corresponding to basic syntactic or high-level semantic information. In NewrlPS, 2024.
[8] Y. Chen, Z. Zhou, and J. Yan. Going beyond neural network feature similarity: The network feature complexity and its interpretation using category theory. In ICLR, 2024.
[9] R. Dong. Z. Qi, L. Zhang, J. Zhang, J. Sun, Z. Ge. L. Yi, and K. Ma. Autoencoders as cross- modal teachers: Can pretrained 2d image transformers help 3d representation learning? arXiv preprint 222.::2222000320. 2022.
[10] I. Enebuse, M. Foo, B. S. K. K. Ibrahim, H. Ahmed, F. Supmak, and O. S. Eyobu. A comparative review of hand-eye calibration techniques for vision guided robots. IEEE Access, 9: 13143- 113155, 2021.
[11] H. Fan, H. Su. and L. J. Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 605-613, 2017.
[12] A. Goyal, H. Law, B. Liu, A. Newell, and J. Deng. Revisiting point cloud shape classification with a simple and effective bascline. In International Conference on Machine Learning, pages 1809-3820. PMLR, 2021.
[13] M.-H. Guo, J.-X. Cai, Z.N. Liu, T.-J Mu, R. R. Martin, and S.-M. Hu. Pct: Point cloud transformer. Computational Visual Media, 7:187-199, 2021.
[14] Z. Guo, R. Zhang, L. Qiu, X. Li, and P.-A. Heng. Joint-mae: 2d-3d joint masked autoencoders for 3d point cloud pre-training, arXiv preprint arXiv:2302.14007, 2023.
[15] A. Hamdi, S. Giancola, and B. Ghanem. Mvtn: Multi-view transformation network for 3d shape recognition. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages -11,,021.
[16] K. He, X. Chen, S. Xie, Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 16000-16009, 2022.
10

---

[17] S. Huang. Y. Xie, S.-C. Zhu, and Y. Zhu. Spatio-temporal self-supervised representation learning for 3d point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4535-6545, 2021.
[18) Y. Li, R. Bu, M. Sun, W. Wu, X. Di, and B. Chen. Pointenn: Convolution on x-transformed points. Advances in neural information processing systems, 31, 2018.
[19] F. Liu, G. Lin, C.-S. Foo, C. K. Joshi, and J. Lin. Point discriminative learning for data-efficient 3d point cloud analysis. In 2022 International Conference on 3D Vision (3DV), pages 42-51. IEEE, 2022.
[20] H. Liu, M. Cai, and Y. J. Lee. Masked discrimination for self-supervised learning on point clouds. In European Conference on Computer Vision, pages 657-675. Springer, 2022.
[21] I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint arXiv: arXiv:1608.039833 2016.
[22] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint 100000171 2017.
[23] X. Ma, C. Qin, H. You, H. Ran, and Y. Fu. Rethinking network design and local geometry in point cloud: A simple residual mlp framework. arXiv preprint urXiv:2202.07123: 2022.
[24] Y. Pang, W. Wang, F. E. Tay, W. Liu, Y. Tian, and L. Yuan. Masked autoencoders for point cloud supe-supe learning. European conference on computer vision, pages 604 621. Springer, 2022.
[25] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning point for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[26] C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. Advances neural information processing systems, 30, 2017.
[27] 7. Qi, R. Dong, G. Fan, Z. Ge, Zhang, Ma, and L.. Yi. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining. arXiv preprint arXiv:2302.02311 02.30 2023.
[28] R. Qian, X. Lai, and X. Li. 3d object detection for autonomous driving: A survey, Pattern Recognition, 130:108796, 2022.
[29] A. Sanghi. Info3d: Representation learning on 3d objects using mutual information maximiza- tion and contrastive learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX 16, pages 626-642. Springer, 2020.
[30] O. Y. Sergiyenko and V. V. Tyrsa. 3d optical machine vision sensors with intelligent data management for robotic swarm navigation improvement. IEEE Sensors Journal, (1(10):1126) 1 11274, 2020.
[31] C. Sharma and M. Kaul. Self-supervised few-shot learning on point clouds. Advances in Neural Information Processing Systems, 33:7212-7221, 2020.
[32] Q Sun, Y. Xu, Y. Sun, C. Yao, S. A. Lee, and K. Chen. Gn-enn: a point cloud analysis method for metaverse applications. Electronics, 12(2):273, 2023.
[33] M. A. Uy, Q.-H. Pham, B.-S. Hua, T. Nguyen, and S.-K. Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In Proceedings of the IEEE/CVF international conference computer vision, pages 1597, 2019.
[34] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, F.. Kaiser, and Polosukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
11

---

[35] H. Wang, Q. Liu, X. Yue, J. Lasenby, and M. J. Kusner. Unsupervised point cloud pre- training via occlusion completion. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9782-9792, 2021.
[36] L. Wang and K.-J. Yoon. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. IEEE transactions on pattern analysis and machine intelligence, 44(6):3048-300 3068, 2021.
[37] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon. Dynamic graph enn for learning on point clouds. ACM Transactions on Graphics (tog), 38(5):1-12, 2019.
[38] Z. Wang, X. Yu, Y. Rao, J. Zhou, and J. Lu, Take-a-photo: 3d-to-2d generative pre-training of point cloud models. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 5640-5650, 2023.
[39] X. Wu, L. Jiang. P.-.. Wang, Z. Liu, X. Liu, Y. Qiao, W. Ouyang. T. He, and H. Zhao. Point transformer v3: Simpler, faster, stronger. arXiv preprint arXiv:2312.10035, 2023.
[40] 7. Wu, S. Song, A. Khosla, F. Yu, L.. Zhang, X. Tang, and J. Xiao. 3d shapenets: A deep the
representation for volumetric shapes. In Proceedings the IEEE conference on computer vision and pattern recognition. pages 1912-1920, 2015.
[41] S. Xie, J. Gu, D. Guo, C. R. Qi, L. Guibas, and O. Litany. Pointcontrast: Unsupervised pre- training for 3d point cloud understanding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 574-591. Springer, 2020.
[42] X. Ye and J. Ma. Visual place recognition via local affine preserving matching. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 12954-12960. IEEE, 2021.
[43] X. Ye and Ma. Neighborhood manifold preserving matching for visual place recognition. IEEE Transactions on Industrial Informatics, 19(7):8127-8) 2022.
[44] X. Ye, G. Yan, and J. Yan. Towards quantum machine learning for constrained combinatorial optimization: a quantum qap solver. In International Conference on Machine Learning, pages 39903-39912. PMLR, 2023.
[45] X. Ye, G. Yan, and J. Yan. Vqne: Variational quantum network embedding with application to network alignment. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3105-3115 15, 2023.
[46] L. Yi, V. G. Kim, D. Ceylan, I.-C. Shen, M. Yan, H. Su, C. Lu, Q. Huang, A. Sheffer, and L. Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Transactions on Graphics (ToG), 35 2016.
[47] X. Yu, I.. Tang, Y. Rao, T. Huang, Zhou, and Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 19313-19322, 2022.
[48] Y. Zha, H. J., Li, R. Li, Dai, B. Chen, Wang, and S.-T. Xia. Towards compact 3d represen- tations via point feature enhancement masked autoencoders. arXiv preprint arXiv:2312.10776. 2023.
[49] Y. Zha, R. Li, T. Dai, J. Xiong, X. Wang, and S.-T. Xia. Sfr: Semantic-aware feature rendering of point cloud. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[50] R. Zhang, Z. Guo, P. Gao, R. Fang. B. Zhao, D. Wang. Y. Qiao, and H. Li. Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training. Advances in neural information processing systems, 15:27061-270177 2022.
[51] R. Zhang, L. Wang, Y. Qiao, P. Gao, and H. Li. Learning 3d representations from 2d pre-trained models via smage-to-pon to-nont masked autoencoders. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11779-217780 2023.
12

---

[52] S. Zhang, M. Liu, and J. Yan. The diversified ensemble neural network. NeurIPS, 2020.
[53] S. Zhang, M. Liu, J. Yan, H. Zhang, L. Huang, X. Yang, and P. Lu. M-mix: Generating hard negatives via multi-sample mixing for contrastive learning. In SIGKDD, 2022.
[54] Zhang, L. Qiu, Zhu, J. Yan, H. Zhang, R. Zhao, Li, and X. Yang. Align representations with base: A new approach self-supervised learning. In CVPR, 2022.
[55] S. Zhang, F. Zhu, J. Yan, R. Zhao, and X. Yang. Zero-el: Instance and feature decorrelation for negative-free symmetric contrastive learning. In ICIR, 2021.
[56] S. Zhang, F. Zhu, R. Zhao, and J. Yan. Contextual image masking modeling via synergized contrasting without view augmentation for faster and better visual pretraining. In ICLR, 2023.
13

---

Table 8: Training details for pretraining and downstream fine-tuning.
Conlig
ShapeNet
SeanObjectNN
ModelNet
ShapeNetPart
S3DIS
optimizer
AdamW
AdamW
AdamW
AdamW
AdamW
learning rate
5c-4
2e-5
le-5
2c-4
2e-4
weight decay
5c-2
5c-2
5c-2
Sc-2
5o-2
learning rate scheduler
cosine
cosine
cosine
cosine
cosine
training epochs
300
300
300
300
warmup epochs butch size drop path rate number of points number of point patches point patch size augmentation
10
10
10
10
10
128
32
32
16
32
0.2
0.2
1024
2048
1024
2048
2048
64
128
64
128
128
32
32
32
32
32
Scalc&Trans+Rotenn
Rotation
Scale& Trans
GPU device
V100 / RTX 3090
V100 RTX 3090
V100/ RTX 3090
RTX 3090
RIX 3090
Table 9: Different Loss functions. The accuracy (%) on three variants of ScanObjectNN reported.
The default setting is marked in blue
Loss function
OBJ_BG
OBJ_ONLY
PB PB_T5ORR RS
1 Distance
94.49
93.28
89.76
Smooth 4
95.18
92.94
89.48
Cosine Similarity
94.83
93.45
89.52
l2 Distance
95.52
94.32
90.35
Appendix
A Additional Experimental Eetails
Pre-training details. We use ShapeNet (3) as the pretraining dataset. ShapeNet is clean of 3D CAD object models with rich annotations, including 51K unique 3D models from 55 common object categories. The overall pretraining includes 300 epochs, and
cosine learning rate [21] of 5e-4 warming up for 10 epochs. Adam W optimizer [22] is used, and the batch size is 128. We run all experiments with single GPU either using RTX 3090 (24GB) or V100 (32GB). More details are shown in Tab, g
Model details. The model is shown in the Fig. 2 The encoder comprises 12 Transformer blocks with attention heads, while the decoder consists of 4 Transformer blocks with 6 attention heads each. The projector is composed of MLP layers, LayerNorm, ReLU, and another MI.P layer, sequentially. The PCM (Predicting Center Module) also comprises 12 Transformer blocks and shares parameters with the encoder by leveraging the same layers within the Transformer block. They differ in their attention mechanisms, i.e., employing self-attention in the encoder or cross-attention in the PCM and their attention objectives.
B Additional Ablation Study
Masking ratio. We conduct experiments using various masking ratios to determine the most suitable option for our PCP-MAE. The results are illustrated in Fig. and identify a masking ratio of m 0.6 as optimal.
Loss function for predicting centers. The pre-training objective guiding the encoder to predict centers entails a regression task. We explore various regression loss functions to select a suitable loss function for our PCP-MAE. The results reported in Tab. 9 Notably, the L2 Distance loss emerges as the best choice. Consequently, we adopt the L2 Distance loss for center prediction.
Pre-training augmentation. The pre-training augmentation plays important role in increasing the variance of training samples and thus affecting the performance of our PCP-MAF. We experiment with different augmentations for pre-training and validate their performance through fine-tuning.
14

---

PB T50 RS Perforance of Different Masking Ratios
Performance of Different n
90.25
Accuracy
Accuracy
89.75
89.50
89.50
89.25
89.00
88.75
88.75
Masking Ratio
Masking Ratio
Figure 3: Performance of different masking ra- tios in PCP-MAE. The accuracy (%) PB T50 RS variant of ScanObjectNN are re- ported. Masking ratio 0.6 performs the best.
Figure 4: Performance of different v) in objective function L PcCcc Recon- The aceuracy (%) on PB _50 _S variant of ScanObjectNN are reported. 7) 0.1 performs the best.
Table 10: Effects of pre-training augmentations. The accuracy (%) on three variants of ScanObjectNN reported. The default setting is marked in blue
Augmentation
OBJ BG
OBJ ONLY
94.19
92.08
88.06
Jitter
93.63
92.25
88.05
Horizontal Flip
93,45
91.91
88.82
Scale&Translanr
94.14
92.25
88.54
Rotation
94.14
93.11
88.96
Rotation + Scale&Translen
94.83
93.11
90.18
Scale& Translate Rotation
95.52
94.32
90.35
When fine-tuning, adopt Rotation for augmenting ScanObjectNN. The results, presented in Tab. TO show that the combination of augmentations, particularly Scale Translate + Rotation, increases the diversity of training samples and yields the best performance. We adopt the augmentation for the pre-training of our PCP-MAE.
Additional analysis on the data augmentation. We used the Scale& Translate=Rotation augmen- tation for pre-training because this combination benefits our PCP-MAF, differing from previous baselines. Since apply different pre-training augmentations compared prior methods, report the performance of the bascline (Point-MAE (24) under various augmentations Tab. [I for clarity. Additionally, test previous SOTA methods (Point-FEMAE 48) and ReCon 27 with our augmentation strategy and observe a slight performance drop, as shown Tab. 12 These results indicate that the explored augmentation composition benefits methods more than others. This improvement can be attributed to the fact that the combination of augmentations enables a PCP-MAE to learn richer distribution of center information
variety of centers, allowing and become more robust. In contrast, the baseline method Point-MAE benefits only slightly from this augmentation due to relying solely point cloud reconstruction during trainiiinng, and prior SOTA methods like ReCon and Point-FEMAF. do not empirically benefit from it.
Different 7 in the objective function. As discussed in Sec. 3.4 the objective function includes scaling factor to balance the point reconstruction loss Reoon and the center prediction loss LPc. formulated as L = nC + Rtecom As depicted in Fig. the optimal performance is achieved when 7 0.1. Therefore, we adopt this value as the scaling factor for the objective function.
Projector for predicting centers. After obtaining the output of the last transformer block in PCM, is passed through projector comprising MLP layers, ReLU, and LayerNorm to predict masked centers. In contrast, the output of the encoder is directed into decoder, which consists of transformer blocks, for further reconstructing the point cloud. Thus, we conduct experiments to investigate whether additional transformer layers are necessary for PCM to enhance the decoding of the predicted
15

---

Table 11: Performance comparison of Point-MAE with different augmentations. The star (") marks the setting used by the original Point-MAE (24], the dagger (1) by peer SOTA methods such as Point-FEMAE 48
and ReCon 27. and the double dagger (+) by us.
Augmentation
ScanObjectNN
Pre-training
Fine-tuning
OBJ-BG
OBJ-ONLY
PB-T50-RS
Scale& Translate
Scale& Translate
90.02
88.29
85.18
Rotation
Rotation
92.60
91.91
88.42
Scale& Translate+Rotationn
Rotation
92.94
92.25
88.86
Table 12: The performance of previous SOTA methods using explored augmentation.
Method
OBJ-BG OBJ-ONLY
PB-T50-RS
Point-FEMAE (Origin)
95.18
93.29
90.22
Point-FEMAE (Our augmentation)
94.32
92.94
89.38
ReCon (Origin)
95.18
93.63
90.63
ReCon (Our augmentation)
94.49
92.77
89.55
Table 13: Number transformer layers in the projector. The accuracy (%) on three variants of ScanObjectNN is reported. The default setting is marked in blue
Depth
# (M)
OBJ_
OBJ_ONLY
PB_TS0_KE
0
29.05
95.52
94.32
90.35
31.27
94.14
92.94
88.93
2
33.05
94.49
93.45
90.04
3
34.81
93.28
92.25
89.27
4
36.59
94.49
93,80
89.55
representations. The results presented in Tab. 13 indiccate that additional transformer layers do not improve performance, hence we continue to employ a simple projector for decoding the predicted representations obtained by the PCM.
Shared parameters between the encoder and the PCM. We conduct an ablation study compare the effects of sharing parameters versus not sharing parameters between the encoder and the PCM. As shown in Tab. [4] sharing parameters not only significantly reduces the number of parameters but also enhances the performance of the model.
This improvement be attributed to the following reasons: The encoder responsible for encoding visible tokens to obtain semantic representations and is the only component used during fine-tuning. When the encoder and PCM share parameters, updates from the PCM are directly incorporated into the encoder. This allows the encoder (or PCM) to learn more semantic representations and more effectively understand the relationship between visible representations and masked patches, Although the loss for predicting centers back-propagater to the encoder due to the cross- atoss-attetiion between the PCM and encoder, its impact on the encoder is indirect when parameters are not shared. In this case, the parameters of the PCM are useless and wasted in downstream tasks because only the encoder is utilized during fine-tuning. In conclusion, sharing parameters makes the updates the PCM directly influence the encoder, leading to more efficient and effective learning and enabling the encoder to learn more semantic representations.
Concerning the above advantages, we choose to share weights between the encoder and the PCM in our method.
C Additional Visualization Results
After pre-training Point-MAE, with 100% mask ratio, i.e., the encoder is completely discarded during training and only the decoder is utilized, the decoder is able to reconstruct the full point
16

---

Positional Embedding
Positional Embodding
Ground Truth
Reconstuction
Ground Truth
Reconstuction
Figure 5: Additional visualization results of Point-MAE reconstruction results on the ScanObjectNN dataset.
Table 14:
Ablation shared parameters between the encoder and PCM. The accuracy (%) three
variants of ScanObjectNN is reported. The default setting marked in blue
Type
# (M)
BBJ_ONLY
PB T50 RS
shared
29.05
95.52
94.32
90.35
non-shared
50.77
94.66
92.77
89.27
cloud using only masked positional embeddings, shown in Fig. [ We refer to this the "masked center leakage", which demonstrates that the decoder does not necessarily rely on the encoder's representation during the pre-training of Point-MAE.
To extend the validation this phenomenon beyond the ShapeNet dataset 3 which primarily contains objects with clear and well-defined shapes, we conducted experiments on another dataset, ScanObjectNN [33]. This dataset poses a more significant challenge, as it consists real-world scans that include background clutter and occlusions. The visualization results in Fig. [ show that the masked center leakage phenomenon is also present in this more complex dataset, further demonstrating the generality of the phenomenon that identified.
D Limitations and Future Works
PCP-MAE simple and effective method that improves Point-MAE by additionally learning to predict the centers, which significantly boosts performance. However, there are some limitations in PCP-MAE, which may be two-fold. PCP-MAE is single-modal self-supervised method. Neverthe- less, the current dataset of 3-D point clouds is constrained in size due to the challenges associated with collecting point cloud data, which turn limits the wider applicability of our approach. Additionally, generative learning, it does not leverage the benefits of contrastive learning and some other works such
while PCP-MAR capitalizes
7 show great performance by harmoniously combining
generative learning
[56] (33 and contrastive learning [54 (5.55 52).
To address these issues, future works could focus developing multi-modal PCP-MAE, explainable
Al 7 6 5 [ or a hybrid model that effectively combines the strengths of both generative and contrastive learning.
E Broader Impacts
Our PCP-MAF demonstrates a marked improvement over the existing Point-MAE, and establishes new state-of-the-an (SOTA) benchmarks across variety of tasks in point cloud understanding. This
17

---

enhancement is pivotal in advancing technologies reliant on precise spatial recognition 43 42 44 45 45] []. such as autonomous driving, which demands accurate environmental perception for safety and efficiency. Moreover, the versatility of the PCP-MAE extends its utility to other critical applications, including urban planning. where detailed and scalable city modeling is essential, and in augmented reality, enhancing interactive experiences by integrating more accurate virtual information with the real world. While our PCP-MAE marks an advancement in point cloud understanding with broad applications ranging from autonomous driving to urban planning, is also susceptible to potential negative societal impacts. The primary concerns stem from its potential use surveillance systems, where enhanced spatial recognition capabilities could lead privacy infringements.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The Abstract and Introduction sections accurately reflect the paper's contribu- tion and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The discussion on the limitations of work is stated in Appendix D
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Our work is motivated by interesting experiment phenomenon and proposes methods based on this observation, which improves the baseline by a large margin. There are no assumptions and any following proofs.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Comprehensive experimental details necessary for reproducibility. including datasets. model configurations, used optimizers, and learning rates, are thoroughly detailed in Sections Sec. 4.1 Sec. 4.2 and Appendix [A] These details collectively ensure that the main experimental results supporting the paper's claims and conclusions can be independently reproduced.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
20

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have included the link to code implementation in the Abstract and provide detailed instructions in the repository for reproducing the main results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
mublic//uiides/Codeaabmss aosoooooooiiioooocooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https
crajss.cc/uublcc/uuii httpa////ggieee/eeeuuuiiiiiesss
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide comprehensive experimental details, including hyperparameters (such as learning rate, max epochs, batch size, etc.), and the type of optimizer, meticulously outlined Sections Sec. 4.1] Sec. [4.2] and Appendix [ Additionally, we conduct ablation studies to select experiment settings, with key ablations claborated upon in Sections Sec. 4.3 and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report the mean accuracy and standard deviation in Tab
Guidelines:
21

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We primarily executed all experiments relying GPU resources, with details of the specific GPUs used provided in Appendix A and corresponding execution times
reported in Tab. [
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have carefully reviewed the NeurlPS Code of Ethics, and we affirm that the research conducted in our paper adheres to the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
22

---

Answer: [Yes]
Justification: See Appendix B for the broader impacts of our work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: our knowledge, the model proposed in our paper does not pose risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We properly cite the creators of models and datasets mentioned in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
23

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our experiments focus on point cloud data and do not include crowdsourcing or involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The research presented in the paper does not involve human subjects or crowdsoureing experiments.
24

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
25


Reviews:
[{"review_id": "x5wF15rh50", "reviewer": "Reviewer_MoXR", "summary": "The paper proposes a masked autoencoding based self-supervised approach for 3D point clouds. The approach which terms as PCP-MAE, learns to predict centers for Point Masked AutoEncoders. The paper investigate that the coordinates of the centers are essential in the point cloud field and the decoder can even reconstruct the masked patches without the encoder representation of the visible patches. Thus, the paper introduces another pre-training objective which predicts the centers and uses them to replace directly provided centers, leading to improved performance. Moreover, this allows the network to not only encode visible centers effectively but also learn the inter-relationships between visible and masked centers. Finally, the approach is efficient and outperforms Point-MAE on multiple benchmark datasets.", "questions": "Please see the questions and the suggestions in the weakness section.", "limitations": "Yes, the authors have adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Provide an insight on why only providing center is good enough for point cloud reconstruction in masked autoencoding.\n- Propose an additional objective task that learns to predict the center of the masked patches and the use of predicted center of the masked patches in the decoder instead of ground-truth center.\n- Experiments results on ScanObjectNN, ModelNet40 and S3DIS.\n- Extensive ablation studies on different components in PCP-MAE, loss functions, masking ratio and other architecture related modules.", "weaknesses": "- The paper writing can definitely be improved and sometimes it's really to hard to follow the underlying idea behind it.\n- The figure 2 can be misleading and it doesn't correlate with the equation 10. In figure 2, it looks like only the masked tokens are passed to the PCP module but in equation 10 which is given as:$PE_{m}^{pred} = PCM (E_{m}, E_{v}, PE_{v})$, it looks like PCP module uses encoded representation of masked patches, visible patches and positional embeddings of visible patches. Can the authors please clarify about this?\n- In Table 2, PointMAE achieves 90.02 88.29 85.18 on three variants of ScanObjNN but in the ablation study (c.f. Table 5 ) the results on the 4th row ( just reconstruction only ) achieves 92.94 92.42 88.65. Isn't only doing reconstruction equivalent to PointMAE? Does it mean the performance boost comes from the positional encoding?\n- Although approach looks simpler and effective, but it only shows improvement by incorporating the additional details on PointMAE. Compared to other the baselines like Point-FEMAE, it doesn't outperforms them on multiple datasets. Can the authors please provide additional results by incorporating PCP and new positional encoding in multiple MAE approaches for point clouds?\n- Did the authors adapt the same set of augmentations for pre-training baselines?\n- The methods looks very sensitive to $\\eta$ (c.f. Figure 4) \n- Some additional results on indoor datasets like ScanNet would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time, thorough comments, and nice suggestions. We are pleased to clarify your questions one-by-one.\n> Q1. The paper writing should be improved.\n\nThank you for your feedback. We will improve the manuscript's language and structure for better clarity and readability, and these updates will be reflected in the next version.\n> Q2. The figure 2 can be misleading and it doesn't correlate with the equation 10.\n\nThank you for pointing out this misleading detail. We have changed Figure 2 to align with Equation 10. Please check the attached rebuttal PDF.\n> Q3. Isn't only doing reconstruction equivalent to PointMAE? Does it mean the performance boost comes from the positional encoding?\n\nYes and no. They are equivalent except for a minor change in the positional encoding. We experimented on the effect of this behavior when aligning other setups:\nMethod|OBJ-BG|OBJ-ONLY|PB-T50-RS\n-|-|-|-\nPoint-MAE|92.94|92.25|88.86\nReconstruction_only|92.94|92.42|88.65\n\nThe results show that it only slightly influences performance.\n\nThe performance boost mainly comes from the augmentations, i.e., the rotation augmentation explored by [1] for fine-tuning and the Scale & Translate + Rotation pre-training augmentation explored by us. The ablation results of Point-MAE are shown as follows:\nAugmentation||OBJ-BG|OBJ-ONLY|PB-T50-RS\n-|-|-|-|-\nPre-training|Fine-tuning|\nScale&Translate|Scale&Translate $^*$ |90.02|88.29|85.18\nRotation|Rotation $^\\dagger$|92.60|91.91|88.42|\nScale&Translate+Rotation|Rotation $^\\ddagger$|92.94|92.25|88.86\n\n$^*$ Adopted by Point-MAE.\n\n$^\\dagger$ Adopted by other SOTA methods including Point-FEMAE and ReCon.\n\n$^\\ddagger$ Adopted by our PCP-MAE.\n\nNote that Scale&Translate+Rotation pre-training augmentation benefits both Point-MAE and our PCP-MAE but does not benefit other SOTA methods, including Point-FEMAE and ReCon (experiment results are in Q5).\n> Q4.1. Compared to other the baselines like Point-FEMAE, it doesn't outperforms them on multiple datasets.\n\nAlthough our PCP-MAE only marginally outperforms previous SOTA methods like Point-FEMAE and ReCon or performs comparably, **our PCP-MAE significantly exceeds them in efficiency.** The comparisons among these three methods are shown below (using one RTX 3090):\n\nMethod|Pre-training||Fine-tuning|||||\n-|-|-|-|-|-|-|-\n||Params (M)|Time (s/epoch)|Params (M)|OBJ-BG Time (s/epoch)|OBJ-ONLY Time (s/epoch)|PB-T50-RS Time (s/epoch)|ModelNet40 Time (s/epoch)\nPoint-MAE (baseline)|29.0|88|22.1|10|10|49|29\nPoint-FEMAE|41.5 (1.43x)|326 (3.70x)|27.4 (1.24x)|30 (3.00x)|30 (3.00x)|148 (3.02x)|72 (2.48x)\nReCon|140.9 (4.85x)|452 (5.14x)|43.6 (1.97x)|13 (1.30x)|13 (1.30x)|60 (1.22x)|42 (1.45x)\nPCP-MAE (ours)|29.5 (**1.01x**)|120 (**1.36x**)|22.1 (**1.00x**)|10 (**1.00x**)|10 (**1.00x**)|49 (**1.00x**)|30 (**1.01x**)\n> Q4.2. Additional results by incorporating PCP and new positional encoding in multiple MAE approaches\n\nAlthough intuitively feasible, it is important to note that implementing our method into various MAE-based frameworks requires careful consideration due to the different optimization goals of each specific approach.\n\n**We incorporated PCP into MPM (Masked Point Modeling) methods including Point-BERT [4] and Point-FEMAE [5] with additional minor changes to maintain architecture consistency.** The results demonstrate that PCP enhances performance when other setups are aligned:\nMethods|OBJ-BG|OBJ-ONLY|PB-T50-RS\n-|-|-|-|\nPoint-BERT|87.43|88.12|83.07\nPoint-BERT+PCP ($\\eta=2.0$)|89.32|89.84|84.77\nPoint-FEMAE $^\\dagger$|94.32|92.94|89.38\nPoint-FEMAE+PCP ($\\eta=0.1$) $^\\dagger$|95.00|93.45|89.83\n\n$^\\dagger$ Reproduced results adopting our augmentation $i.e.$, Scale&Translate+Rotation for pre-training.\n\nDue to time limits, we couldn't explore different design choices and $\\eta$, which could bring improvement.\n>Q5. Did the authors adopt the same set of augmentations for pre-training baselines?\n\nWe used the Scale&Translate+Rotation augmentation for pre-training because we found that this combination benefits our PCP-MAE and slightly benefits Point-MAE, **which is different from previous baselines (Tab. 10).** This improvement can be attributed to the fact that the combination of augmentations enables a variety of centers, allowing our PCP-MAE to learn a richer distribution of center information and become more robust. **In contrast, previous SOTA methods like ReCon and Point-FEMAE do not benefit from this augmentation empirically.**\n\nThe detailed augmentations are illustrated in Q3. We further provide the results after aligning the augmentations of Point-FEMAE and ReCon with ours for clarity:\nMethod|OBJ-BG|OBJ-ONLY|PB-T50-RS\n-|-|-|-\nPoint-FEMAE (Origin)|95.18|93.29|90.22\nPoint-FEMAE (Our augmentation)|94.32|92.94|89.38\nReCon (Origin)|95.18|93.63|90.63\nReCon (Our augmentation)|94.49|92.77|89.55\n>Q6. The methods looks very sensitive to $\\eta$ (Figure 4). \n\nWe tried values from 0.1 to 1.0 in increments of 0.1 for convenience and the results in Figure 4 show a small $\\eta$ is suitable for our PCP-MAE. When choosing more fine-grained $\\eta$, our approach behaves less sensitively:\n\n$\\eta$|0.02|0.04|0.06|0.08|0.10|0.12|0.14|0.16|0.18\n-|-|-|-|-|-|-|-|-|-\nScanObjectNN (OBJ-BG)|94.14|94.66|94.49|95.35|**95.53**|95.00|94.66|94.49|94.66\nModelNet40 (w/o voting)|93.56|93.48|93.64|93.72|**93.96**|93.84|93.52|93.44|93.52\n>Q7. Some additional results on indoor datasets like ScanNet would be helpful.\n\nWe experimented with the indoor dataset S3DIS [2] (lines 275-279), which provides detailed indoor 3D point cloud of over 6,000 square meters across six buildings. The scene segmentation results are:\nMethods|mAcc|mIoU\n-|-|-\nPoint-MAE|69.9|60.8\nPCP-MAE (ours)|71.0|61.3\n\nFor ScanNet [3], the dataset's different statistics (e.g., 2k input points for ShapeNet vs. 50k for ScanNet) require changes to the model architecture and pre-training setups. Due to time and resource constraints, we couldn't complete this but will update the results in the next paper version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tAzM2loq8F", "reviewer": "Reviewer_nJYR", "summary": "This work examines representation learning of 3D point clouds using masked autoencoding. A known issue with such an approach is that the coordinates of the patch centers leak significant information about the geometry and semantics of the shape being reconstructed, which degrades the representations learned by these approaches. This work proposes to add an extra criterion, whereby the model predicts the positions of the masked patch centers. Pretraining is performed on ShapeNet. This simple change leads to significant improvements over Point-MAE when finetuning on ScanObjectNN and ModelNet40 for 3D classification, 3d scene segmentation, and object part segmentation.", "questions": "* If the authors have the capacity to conduct such an experiment, I am primarily curious to see the effect of the proposed center point prediction task with stronger baselines which provide advances orthogonal to the contributions in this work.", "limitations": "N.A.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**Originality**\n* The proposed point center prediction objective is novel to the best of my knowledge, and requires several subtle innovations on the implementation side to yield improved performance (e.g., choice of sin-cos positional embeddings before MLP projection module, re-using predicting positions in the reconstruction objective with a stop-gradient operation, and the shared network architecture with cross-attention for point center prediction). Ablation experiments demonstrate the importance of each of these factors.\n\n**Clarity**\n* The work is reasonably clear, enough that the reader can understand basic ideas, motivation, and details of the proposed method. Releasing code with the paper will greatly improve reproducibility and impact.\n\n**Quality**\n* The work is of a reasonable quality; experiments and numerical results on standard benchmarks and standard evaluation setups.\n\n**Significance**\n* This work contributes to a large body of work on MAE-based representation learning for 3D point clouds, and provides an interesting and simple approach for addressing a known limitation of these approaches. Improvements over Point-MAE, the most similar baseline, are notable.", "weaknesses": "* Improvements over previous MAE-based methods, such as Point-FEMAE, are quite marginal, and perhaps not statistically significant. While the contributions of other MAE related methods are orthogonal and could be potentially combined with the proposed method, it is unclear if the issue of 3D center point position leakage is still significant when combined with other tools.\n* Numerical results are primarily restricted to end-to-end fine-tuning. Would be curious to see the results of linear or frozen evaluations to assess the quality of the representations, as finetuning from scratch (i.e., random representations) can already achieve much of the topline performance in many cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time, thorough comments, and nice suggestions. Your endorsement of our method and experiments gives us significant encouragement. Here are our clarifications.\n\n> Q1.1. Improvements over previous MAE-based methods, such as Point-FEMAE, are quite marginal, and perhaps not statistically significant.\n\nAlthough our PCP-MAE only marginally outperforms other advanced MAE-based methods such as Point-FEMAE and ReCon, **our approach significantly advances in efficiency,** particularly regarding pre-training and fine-tuning time costs. The efficiency comparisons are shown below (experiments conducted using an empty RTX 3090):\n\nMethod|Pre-training|| Fine-tuning| | | | |\n-|-|-|-|-|-|-|-\n||Params (M)|Time (s/epoch)|Params (M)|OBJ-BG Time (s/epoch)|OBJ-ONLY Time (s/epoch)|PB-T50-RS Time (s/epoch)|ModelNet40 Time (s/epoch)\nPoint-MAE (baseline)|29.0|88|22.1|10|10|49|29\nPoint-FEMAE|41.5 (1.43x)|326 (3.70x)|27.4 (1.24x)|30 (3.00x)|30 (3.00x)|148 (3.02x)|72 (2.48x)\nReCon|140.9 (4.85x)|452 (5.14x)|43.6 (1.97x)|13 (1.30x)|13 (1.30x)|60 (1.22x)|42 (1.45x)\nPCP-MAE (ours)|29.5 (**1.01x**)|120 (**1.36x**)|22.1 (**1.00x**)|10 (**1.00x**)|10 (**1.00x**)|49 (**1.00x**)|30 (**1.01x**)\n\n> Q1.2. While the contributions of other MAE-related methods are orthogonal and could be potentially combined with the proposed method, it is unclear if the issue of 3D center point position leakage is still significant when combined with other tools.\n\nThank you for your good question. First, 3D center point position leakage results from the Point-MAE or exactly the reconstruction objective due to the directly provided centers. Point-MAE remains the core component of MAE-based methods such as Point-FEMAE and ReCon, and point cloud reconstruction continues to be one of the pre-training objectives. **Thus, the issue of position leakage still exists in these methods.** Our approach directly guides the encoder to learn distributions of centers, which helps to mitigate this issue.\n\n**However, the extent of the negative impact caused by position leakage varies in different MAE-related methods.** Take Point-FEMAE [2] as an example. It adds a branch to the pre-training model which prevents the model from learning limited representations. This will possibly enhance the model's understanding to the point cloud better and indirectly enhance its grasp of center distributions. Consequently, the position leakage issue may be alleviated. Although the performance of these methods can reveal the degree of influence posed by position leakage, a quantitative method to analyze this issue is necessary and could be addressed in our future research.\n\n> Q2. Would be curious to see the results of linear or frozen evaluations to assess the quality of the representations.\n\nFollowing ReCon [1], we assess the performance of MLP-Linear, which freezes the pre-trained encoder and only updates the classification head to evaluate the quality of the learned representations. Here are the experiment results:\n\nMLP-Linear|OBJ-BG|OBJ-ONLY|PB-T50-RS|\n-|-|-|-\nPoint-MAE|82.77±0.30|83.23±0.16|74.13±0.21\nACT (Cross-modal)|85.20±0.83|85.84±0.15|76.31±0.26\nPoint-FEMAE|88.98±0.15|89.50±0.22|80.32±0.10\nReCon (Cross-modal)|89.50±0.20|89.72±0.17|81.36±0.14\nPCP-MAE (ours)|89.41±0.13|89.41±0.26|80.63±0.08|\n\nThe results show that our approach greatly outperforms the baseline Point-MAE, marginally surpasses Point-FEMAE, and lags behind the leading cross-modal method ReCon.\n\n> Q3. The effect of the proposed center point prediction task with stronger baselines which provide advanced orthogonal improvement to the contributions in this work.\n\nThank you for this nice suggestion. We tried to incorporate our center point prediction task with Masked Point Modeling (MPM) family methods include Point-BERT [3] and Point-FEMAE [2] among which Point-BERT is parallel to the Point-MAE [4] and Point-FEMAE provide advanced orthogonal improvement based on Point-MAE compared to this work.\n\nAlthough intuitively feasible, it is important to note that implementing our method into various MPM-based frameworks requires careful consideration due to the different optimization goals of each specific approach. **We incorporated PCP (learning predicting masked centers) into MPM methods including Point-BERT and Point-FEMAE with additional minor changes to maintain architecture consistency.** The experiment results show that incorporating our center point predicting task enhances performance when other experiment setups are aligned: \n\nMethods|OBJ-BG|OBJ-ONLY|PB-T50-RS\n-|-|-|-|\nPoint-BERT|87.43|88.12|83.07\nPoint-BERT+PCP ($\\eta=2.0$)|89.32|89.84|84.77|\nPoint-FEMAE $^\\dagger$|94.32|92.94|89.38\nPoint-FEMAE+PCP ($\\eta=0.1$) $^\\dagger$|95.00|93.45|89.83|\n\n$^\\dagger$ Reproduced results adopting our augmentation $i.e.$, Scale&Translate+Rotation for pre-training.\n\nDue to time constraints, exploration of various design choices and different $\\eta$ was not feasible. We believe that further customization could lead to more substantial improvements.\n\n[1] Qi Z, Dong R, Fan G, et al. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining[C], ICML 2023.\n\n[2] Zha Y, Ji H, Li J, et al. Towards compact 3d representations via point feature enhancement masked autoencoders[C], AAAI 2024.\n\n[3] Yu X, Tang L, Rao Y, et al. Point-bert: Pre-training 3d point cloud transformers with masked point modeling[C], CVPR 2022.\n\n[4] Pang Y, Wang W, Tay F E H, et al. Masked autoencoders for point cloud self-supervised learning[C], ECCV 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DbX4IWPGm4", "reviewer": "Reviewer_DT8p", "summary": "The paper studies the point cloud pretraining under the self-supervised learning paradigm. Authors experimentally found that the position embedding in the decoder may decrease the learning ability of the encoder and propose a new method to overcome this issue.", "questions": "pealse refer to the weakness", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is clearly written and easy to follow.\n2. The phenomenon revealed in this paper is new to me and sounds reasonable.\n3. The proposed method seems reasonable and aligns with the core motivation of the paper.", "weaknesses": "1. In line 30, authors mentioned the difficulty in collecting 3D dataset and want to use SSL to solve this problem. However, 3D data is different with 2D images which can be collected easily on the internet. For the experimented dataset like Scanning datasets, the label will be collected during scanning, then why SSL for 3D is important?\n2. According to table 1, Point-FEMAE achieves almost the same performance as the proposed model, thus it is hard to say that the reconstructing objective may make the encoder unable to learn semantic features in Line 70. \n3. The improvements of the proposed method over Point-FEMAE are marginal. \n4. One possible reason behind the phenomenon of the position embedding may be the simplicity of the dataset. Would the phenomenon also hold for lidar datasets?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for providing your detailed feedback. Below, we answer your questions in detail:\n\n> Q1. The importance of SSL for 3D.\n\nUpon revisiting our statement, we find it more appropriate to remove \"collect\" in line 27 of the original paper, which we will update in the next version. Here are clarifications regarding your question.\n\nWe agree with you that collecting 3D data is easy through scanning. However, to our knowledge, most labeled 3D datasets, such as ScanNet [1] and NuScenes [2], which contain vast amounts of 3D data, require laborious annotation, and labels cannot be directly assigned or collected within them.\n\nTake autonomous driving as an example, a data acquisition vehicle can collect more than 200k frames of point clouds within 8 working hours, but a skilled worker can only annotate 100-200 frames per day [3]. Thus, effectively leveraging unlabeled 3D data becomes a critical issue in practical applications. **Self-supervised learning in 3D can effectively leverage large amounts of unlabeled 3D data by designing specific pre-text tasks for pre-training, benefiting downstream tasks.** Therefore, 3D SSL is of significant importance.\n\n> Q2. According to table 1, Point-FEMAE achieves almost the same performance as the proposed model, thus it is hard to say that the reconstructing objective may make the encoder unable to learn semantic features in Line 70.\n\nThank you for your insightful question. It's crucial to focus on the word **\"may\"** in our statement. We have observed that reconstruction does not necessarily depend on the encoder’s representation when positional embeddings (centers) are directly provided, which underscores the significance of these embeddings. Our proposed PCP-MAE model addresses this by guiding the encoder to learn distributions of semantically rich centers, replacing the directly provided centers.\n\n**However, apart from directly guide the encoder to enhance its understanding on the centers to address the found position leakage issue, other approaches could indirectly alleviate it.** Take Point-FEMAE as an example. It adds a branch to the pre-training model which prevents the model from learning limited representations. This will possibly enhance the model's understanding to the point cloud better and indirectly enhance its grasp of center distributions. Consequently, the position leakage issue may be alleviated. That's why we use **may** in our statement. In essence, while the reconstruction objective could impede learning semantic features, it can be alleviated with auxiliary tools.\n\n**Most importantly, our approach is the first to directly address the position leakage issue**. Although Point-FEMAE could also possibly alleviate this issue, it significantly lags behind in efficiency (check the answer to the Q3).\n\n> Q3. Marginal improvement over Point-FEMAE.\n\nOur PCP-MAE and Point-FEMAE are two orthogonal works based on Point-MAE. Point-FEMAE proposes a global branch and a local branch to capture latent semantic features, rather than using just one branch. Our PCP-MAE, on the other hand, guides the model to learn to predict centers based on a motivating observation.\n\nThe reported results show that both PCP-MAE and Point-FEMAE significantly improve the baseline performance, with our PCP-MAE marginally outperforming Point-FEMAE.\n\n**However, when it comes to training efficiency, Point-FEMAE significantly lags behind our approach.** Point-FEMAE not only adds an additional branch but also uses extra Local Enhancement Modules for modeling the local point clouds. This results in a significant increase in both parameter and time costs compared to Point-MAE. The comparisons among these three methods are shown as follows:\n\nMethod|Pre-training||Fine-tuning|||||\n-|-|-|-|-|-|-|-\n||Params (M)|Time (s/epoch)|Params (M)|OBJ-BG Time (s/epoch)|OBJ-ONLY Time (s/epoch)|PB-T50-RS Time (s/epoch)|ModelNet40 Time (s/epoch)\nPoint-MAE (baseline)|29.0|88|22.1|10|10|49|29\nPoint-FEMAE|41.5 (1.43x)|326 (3.70x)|27.4 (1.24x)|30 (3.00x)|30 (3.00x)|148 (3.02x)|72 (2.48x)\nPCP-MAE (ours)|29.5 (**1.01x**)|120 (**1.36x**)|22.1 (**1.00x**)|10 (**1.00x**)|10 (**1.00x**)|49 (**1.00x**)|30 (**1.01x**)\n\n\nPoint-FEMAE retains the Local Enhancement Modules during fine-tuning, which adds an unignorable extra computational burden and thus greatly decreases the speed of model fine-tuning.\n\n> Q4. One possible reason behind the phenomenon of the position embedding may be the simplicity of the dataset. Would the phenomenon also hold for lidar datasets?\n\nThank you for your nice suggestion. In addition to the pre-training ShapeNet dataset, we have experimented with setting the mask ratio to 100% on another dataset called ScanObjectNN [6]. This is a much more challenging dataset, sampled from real-world scans that include background and occlusions, and is built based on two popular real scene mesh datasets, SceneNN [7] and ScanNet [1]. **When only using the decoder (mask 100%) for pre-training, the point cloud still can be reconstructed well through experimenting.** \n\nFrom a statistical perspective, the Chamfer Distance L2 Loss (Eq. 15) is even lower than that of the simpler ShapeNet dataset. The reconstruction losses are as follows:\n\n|Dataset|Epoch 1 Loss|Epoch Last Loss|\n|-|-|-|\n|ShapeNet|0.15823|0.00324|\n|ScanObjectNN|0.44190|0.00250|\n\nWe will add the visualizations in the next version of our paper. Additionally, this phenomenon is likely to hold for lidar datasets, which we will test and update the results in the next version.\n\nPlease check the official comment for the Reference list.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MJU0AVF70Z", "reviewer": "Reviewer_GmbZ", "summary": "The paper proposes a novel self-supervised learning method called PCP-MAE for point cloud understanding. The key innovation of PCP-MAE is that it guides the model to learn to predict the positional embeddings of the centers of masked regions, rather than directly providing the coordinates. This approach encourages the encoder to learn richer semantic representations, leading to performance improvements on downstream tasks such as point cloud classification.", "questions": "Although the author has demonstrated superior improvement in various experiments, this improvement seems to come more from trick's improvements (such as data augmentation) rather than the proposed method itself. Meanwhile, the author lacks a detailed analysis of the generality of the proposed method. Therefore, I reject this article and encourage the author to conduct a more in-depth, thorough, and fair analysis and comparison.", "limitations": "The author has pointed out some issues in the Limitations section, and I hope the author can make more improvements based on the review comments, so that their work becomes truly solid rather than superficial incremental improvements.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.Instead of directly providing the coordinates of the centers of masked regions, PCP-MAE guides the model to learn to predict the positional embeddings of these centers. This idea encourages the encoder to learn richer semantic representations.\n\n2.The authors have conducted a comprehensive set of experiments, comparing PCP-MAE against state-of-the-art methods on a variety of point cloud understanding tasks.", "weaknesses": "1.There are significant concerns about the actual effectiveness of the center point prediction method being discussed. Although the author conducted numerous experiments to demonstrate the superiority of their method, these comparisons with previous methods are unfair. This unfairness is reflected in the following aspects: 1. Although this method has achieved significant improvement, it seems that the main improvement comes from the use of more complex data augmentation tricks during pre training, without a fair comparison with previous methods when using the same tricks. Due to the lack of detailed code provided by the author, based on my personal experience, when using only the center point prediction method with the same settings or tricks, the proposed method does not show significant improvement and may even experience performance degradation. For example, when using the center point of ground truth without center point prediction and with the same settings such as position encoding and data augmentation, I don't think the author's center point prediction will be better than using the ground truth method.\n\n2.On line 69, the author get the conclusion that 'the reconstructing objective may make the encoder unable to learn semantic features'. I would like to know how the author came to this conclusion. Can a randomly initialized token with a 100% mask rate be used to reconstruct the approximate shape of a point cloud using only the center point as the position encoding through an encoder? I don't think so. This phenomenon does not mean that the encoder cannot learn semantic features. Secondly, the author only conducted a qualitative analysis without providing a detailed quantitative comparison. In fact, the CD distance of the point cloud reconstructed by this method is much greater than that of the Point MAE method, indicating that the effect of the point cloud reconstructed by this method is very poor.\n\n3.The author's improvement is incremental and limited, so I do not think that this paper meets the contribution criteria of NeurIPS. The proposed method is only an improvement on Point MAE, in fact, Point MAE is just one method in the field of point cloud self supervised learning.\n\n4.The author did not demonstrate the generality of the proposed method. Assuming that this central prediction method is effective, it should be applicable to all MAE based point cloud self-supervised methods, such as Point BERT, ACT, Point-M2AE, PointFEMAE, I2P-MAE, Recon, etc. The author lacks further analysis of the generality of the proposed ideas.\n\n5.Insufficient Visualizations and Explanations: The paper lacks some key visualizations and explanations that would aid reader understanding. For example, the pipeline (Fig. 2) does not clearly annotate the \"stop gradient\" operation, which is a crucial component of the proposed method. Additional visualizations and explanations would help readers better comprehend the technical details of PCP-MAE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. There may be some misunderstandings on your part regarding certain aspects of our work. We hope that our response will address your concerns. Here is our clarification.\n> Q1.1. Comparisons with previous methods are unfair. It seems that the main improvement comes from the use of more complex data augmentation tricks during pre training, without a fair comparison with previous methods when using the same tricks. \n\nThank you for pointing out the lack of baseline performance using the same augmentation. Firstly, different methods suit different pre-training augmentations. During experimentation, we found our approach works well when using a composition of augmentations, *i.e.*, Scale&Translate+Rotation. This can be attributed to the fact that a composition of augmentations enables a variety of centers, thus allowing our PCP-MAE to learn richer center distribution information and be more robust. While this also slightly benefits the performance of the baseline Point-MAE, **it doesn't benefit the previous SOTA methods such as Point-FEMAE and ReCon.** The performance of Point-MAE under different augmentations is displayed as follows for clarity:\nAugmentation||OBJ-BG|OBJ-ONLY|PB-T50-RS\n-|-|-|-|-\nPre-training|Fine-tuning|\nScale&Translate|Scale&Translate $^*$ |90.02|88.29|85.18\nRotation|Rotation $^\\dagger$ |92.60|91.91|88.42|\nScale&Translate+Rotation|Rotation $^\\ddagger$|92.94|92.25|88.86\n\n$^*$ Adopted by Point-MAE.\n\n$^\\dagger$ Adopted by other SOTA methods include Point-FEMAE and ReCon.\n\n$^\\ddagger$ Adopted by our PCP-MAE.\n\n**When aligning the augmentations of Point-FEMAE and ReCon with ours, there are no performance gains, and they even suffer from a performance drop.** This phenomenon is caused by the features of different methods. The performance is shown as follows:\nMethod|OBJ-BG|OBJ-ONLY|PB-T50-RS\n-|-|-|-\nPoint-FEMAE (Origin)|95.18|93.29|90.22\nPoint-FEMAE (Our augmentation)|94.32|92.94|89.38\nReCon (Origin)|95.18|93.63|90.63\nReCon (Our augmentation)|94.49|92.77|89.55\n\n**Feel free to validate this through their official codebase.** We apologize for the lack of baseline performance in our paper, which we will definitely add in our next version to ensure fairness and clarity. However, it's also important to reclaim that the composition of augmentations benefits our method due to its features and hampers other SOTA methods, which means the main improvement comes from our approach, not only from the explored augmentations.\n> Q1.2. Through your personal experience, you found when using only the center point prediction method with the same settings or tricks, the proposed method does not show significant improvement and may even experience performance degradation.\n\nFirst, when you try our idea in person, please note there remain some subtle designs and pay attention not to implement them incorrectly. These include sin-cos positional embedding followed by MLP projection, the detailed design of the projector, the shared weight encoder with self/cross attention and *etc*.\n\nSecond, regarding the $\\eta$ in line 210, it's important to select an appropriate $\\eta$ to improve the performance. Empirically, a small $\\eta$ is preferred, which still needs to be tuned carefully. So be careful to choose an appropriate $\\eta$ when you try our idea.\n\nMoreover, the sin-cos position embedding doesn't bring performance gain:\nMethod| OBJ-BG | OBJ-ONLY | PB-T50-RS \n-|-|-|-\nPoint-MAE (Orig.)|92.94|92.25|88.86\nPoint-MAE (Sin-cos)|92.94| 92.42 | 88.65\n\nThe pre-training augmentation aligns well with our approach and may hinder other methods. This should be seen not as a trick but as a feature of our PCP-MAE. For more details, please refer to our response to Q1.1\n\n**Regarding the effectiveness of our approach, the ablation results have confirmed its effectiveness when all other experimental setups are aligned:**\nReconstruction|w/PCM|Using $PE_m^{\\text{pred}}$ |OBJ_BG|OBJ_ONLY|PB_T50_RS\n-|-|-|-|-|-\n✓|✗|✗|92.94|92.42|88.65\n✓|✓|✗|94.32|93.11|89.38\n✓|✓|✓|**95.52**|**94.32**|**90.35**\n\n**We have provided our code and the corresponding checkpoints via an anonymous link to the AC.** Hope this will help you to better comprehend our method.\n\n> Q2.1. Can a randomly initialized token with a 100% mask rate be used to reconstruct the approximate shape of a point cloud using only the center point as the position encoding through an encoder?\n\nYes. When only given positional embeddings for a masked patch and a global, randomly initialized **learnable** mask token, the point cloud can be reconstructed only using the decoder after pre-training. The visualization can be seen in Fig. 1. Furthermore, we found that this phenomenon also holds on a more challenging dataset, ScanObjectNN.\n\n> Q2.2. This phenomenon does not mean that the encoder cannot learn semantic features.\n\nThis phenomenon indicates that the decoder does not necessarily rely on the representations of the encoder for reconstruction, which **may** make the encoder unable to learn semantic representations. Our PCP-MAE guides the encoder to learn to predict semantically rich centers to force the learning of richer representations, thus improving performance.\n\n> Q2.3. The quantitative reconstruction analysis between Point-MAE and Mask 100%.\n\nWe use the Reconstruction Chamfer Distance L2 loss to evaluate the quality of reconstruction. The losses for them are shown as follows:\nMethods|Epoch 1 Loss|Epoch Last Loss\n-|-|-|\nPoint-MAE|0.12921|0.00270\nMask 100|0.15823|0.00324\n\nThe Mask 100% exhibits a slightly higher loss than that of the original Point-MAE, but continues to decline and can reconstruct the point cloud after pre-training. Though the encoder slightly benefits point cloud reconstruction, the reconstruction can still be achieved without it. This phenomenon aligns with our core idea that the decoder does not necessarily rely on the representations of the encoder.\n\n**Please check the official comment for the remaining questions.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2lhNHfBdIi", "reviewer": "Reviewer_rD3v", "summary": "This paper makes a novel observation (at least to me) that for mask-based autoencoder paradigm for point cloud self-supervised pretraining, the centers of patches are important and the reconstruction objective does not necessarily rely on representations of the encoder. This is different from the 2-D case on mask-based autoencoding for images. Then the authors introduce a simple yet strong scheme that directly learning to Predict Centers for Point Masked AutoEncoders to prevent the encoder from the failure of semantic information learning. This approach is more efficient than baselines, and achieves state-of-the-art performance on public datasets.", "questions": "Could this approach inspire the development in other areas of selfsupervised learning?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1) The observation is interesting and the motivaiton behind the approach is clear, though the abstract I think needs improvement to make it more logical, crisp, and convincing.\n2) It shows the difference between 2-D and 3-D data for mask-based autoencoding, and the knowledge and evidence shown in this paper is worth publishing (in NeurIPS 2024) at this moment.\n3) The devised approach is cost-effective, and shows strong empirical performance.\n4) The paper is basically clearly written with comprehensive experiments.\n\nI think the paper gives a very important observation that center-aware objective makes the pre-training trivial. It opens up space for improvement for the community as also reflected by their experimental results. And this is the major reason for me to vote for acceptance.", "weaknesses": "The authors may discuss the limitation of their approach (if there was any).\n\nThe abstract can be improved, e.g. the numbers in abstract is vague as there is no specific metric.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time, detailed comments, and valuable suggestions. We are delighted that you recognize the clear motivation, high efficiency, and novelty of our PCP-MAE. Here are our responses to the questions you raised:\n\n> Q1. The limitation of our approach.\n\nWe have discussed the limitations of our approach in Appendix C of the submitted paper. The limitations of our work can be summarized as follows:\n\n1. PCP-MAE is a single-modal self-supervised method. However, the current dataset of 3D point clouds is constrained in size due to the challenges associated with collecting point cloud data, which in turn limits the wider applicability of our approach.\n2. PCP-MAE capitalizes on generative learning but does not leverage the benefits of contrastive learning. Some other works, such as ReCon [1], show great performance by harmoniously combining generative and contrastive learning.\n\nFuture work could focus on developing a multi-modal PCP-MAE or a hybrid model that effectively combines the strengths of both generative and contrastive learning.\n\n> Q2. The abstract can be improved, e.g. the numbers in abstract is vague as there is no specific metric.\n\nThank you for your valuable advice. We have enhanced the abstract by adding a specific metric.\n\nRegarding lines 17-20, we have modified the text from\n- \"Our method is of high pre-training efficiency compared to other alternatives and achieves great improvement over Point-MAE, particularly outperforming it by 5.50%, 6.03%, and 5.17% on three variants of ScanObjectNN.\" \n\nto \n\n- \"Our method is of high pre-training efficiency compared to other alternatives and achieves great improvement over Point-MAE, particularly surpassing it by 5.50% on OBJ-BG, 6.03% on OBJ-ONLY, and 5.17% on PB-T50-RS for 3D object classification on the ScanObjectNN dataset.\"\n\nIf you have any further concerns, feel free to contact us.\n\n> Q3. Could this approach inspire the development in other areas of self-supervised learning?\n\nYes. Our approach offers insights for the broader field of self-supervised learning. Specifically, it challenges the conventional use of positional encodings in the reconstruction process, highlighting the importance of semantic richness in these encodings in the point cloud SSL. While our study focuses on point cloud data in 3D form, the implications extend to other 3D data representations such as meshes and voxels. In these contexts, the usage of positional encodings may need to be reconsidered if they contains case-specific information rather than only case-agnostic information.\n\nFurthermore, our findings illuminate considerations for adapting methods across different domains. For example, when extending the Masked Autoencoder (MAE) approach to 3D from 2D, it is crucial to account for the distinct interpretations of positional encodings in these dimensions. This underscores the need for domain-specific adjustments to ensure the transferability and effectiveness of self-supervised learning techniques.\n\n\n[1] Qi Z, Dong R, Fan G, et al. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining[C], ICML 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their detailed and valuable suggestions and are grateful for the encouraging comments:

1. The paper is clearly written and easy to follow. (DT8p, nJYR, rD3v)
2. The core observation and motivation remain novel and clear. This paper is well-motivated and provides new insights into the differences in positional encodings between 2D MAE and Point-MAE. It opens up new directions for improvement in the field of point cloud self-supervised learning. (MoXR, nJYR, DT8p, rD3v)
3. The proposed method aligns well with the motivation. It is a simple and effective approach that successfully addresses the identified limitations. (MoXR, nJYR, DT8p, rD3v)
4. The experimental results sufficiently demonstrate the effectiveness of the proposed method. (MoXR, nJYR, rD3v)
5. Extensive ablation studies show the effectiveness of different components and designs. (MoXR, nJYR)

Based on the comments, we conclude some noteworthy replies for the reviewers including:

- **[Inappropriate details in the overview figure (Fig. 2). Reviewer MoXR, GmbZ]** We have clarified the details for the reviewers and modified the figure according to the feedback. Please check the attached rebuttal PDF.

- **[Marginal improvement over other SOTA methods. Reviewer MoXR, nJYR, DT8p]** In response, we highlighted the simplicity and efficiency of our method. Compared to other SOTA methods, we require significantly less training time and fewer parameters. Specifically, Point-FEMAE requires 326s per epoch, whereas our PCP-MAE only necessitates 120s per epoch. Our approach is 2.7x faster than the Point-FEMAE.

- **[The generality of our approach. Reviewer MoXR, nJYR, GmbZ]** We have incorporated our core idea into different methods, including Point-BERT and Point-FEMAE. Both demonstrate performance gains after integrating our idea, showcasing the great generality of our approach.

- **[Seemingly unfair comparison and concerns about the effectiveness of our approach. Reviewer GmbZ]** Given that Reviewer GmbZ tested our idea personally, we kindly remind them that there may be some subtle design aspects that should be carefully implemented. The reviewer also believes that all our improvements stem from the different pre-training augmentations rather than the method itself, leading to potentially unfair comparisons with previous SOTA methods. **To address this, our experiments show that other SOTA methods suffer performance drops when using the same augmentation, which we will add to the next version of our paper. Our method benefits from this due to its specific features, which we explain in the rebuttal.** We thank him for reminding us to display the performance of previous methods under the same settings. More importantly, we have confirmed through experiments that the effectiveness of our approach stems from the proposed method itself, rather than from the "data augmentation trick", as demonstrated by testing other methods with the same augmentation. To further enhance the reliability of our approach, we have provided the code and corresponding checkpoints to the ACs via an anonymous link.

- **[Additional experiments. Reviewer MoXR, DT8p]** We have added new experiments based on the comments, including frozen evaluations and validation of the core phenomenon on other datasets.

We sincerely hope that this work provides valuable insights into the field of point cloud self-supervised learning and that the misunderstandings raised by Reviewer GmbZ can be resolved through this rebuttal. Thanks again to all reviewers for their valuable time to help improve our work.

Meta-review: **Summary of the paper:**
The paper identifies a central problem with PointMAE, that is, the position of patch centers that are provided to the decoder leaks information about the 3D structure of the object and therefore a reconstruction of the point cloud is possible without considering any output of the encoder. Then, it suggests an updated mechanism where the encoder also predicts the center of the patches which is then used in the decoder to better encourage the encoder to embed relevant information for the reconstruction. The modified PointMAE is shown to consistently perform better at downstream tasks.

**Summary of the reviews:**
The reviewers find the insight on the different role of patch center embedding for point-clouds and fixed grids interesting, informative and important. They also find the proposed modification relevant. They finally appreciate the improved results over PointMAE and the extensive side/ablation studies.

On the other hand, the reviewers believe the presentation, text, figures, and equations can be improved, and that some of the reported results are confusing. They also asked for the addition of PCP on more recent variants of masking-based SSL on point clouds. 

**Summary of the rebuttal and discussions:**
The authors clarified some confusions about the presentation and the reported numbers. They also argue for the computational efficiency of PCP-MAE over more recent state of the art methods. They further provided additional experiments that (1) show improvements when PCP is added to Point-BERT and Point-FEMAE, and (2) evaluates the method on a new indoor dataset. 

Some reviewers remained unconvinced regarding the comparison to the state of the art and considered most improvements coming from the different augmentations. The authors provided their code anonymously and reviewer GmbZ put a laudable effort in reproducing results, using their code, and with 10 random seeds that makes it quite reliable. 

**Consolidation report:**
Most concerns were satisfactorily refuted. The main outstanding concern is the comparison with state-of-the-art which only makes PCP-MAE comparable to them and not superior. The reproduced results of the reviewer, while slightly lower than the reported ones by the authors, show similar trends, i.e. improved performance over pointMAE but only comparable performance to the state of the art. The authors demonstrate that their method has significantly better computational efficiency.

AC’s additional take: the family of PointMAE is an active field of research both for improvement and application. The problem that the paper finds with PointMAE is fundamental and indisputable. The significance of this finding by itself was more or less undermined by the reviewers. The proposed approach, although can be improved, is simple and sensible and achieves some improvement over the main baseline and is comparable to the state of the art while being significantly more efficient. The identification of a central problem and proposing simple mitigation are considered significant steps forward for the field.

**Recommendation:**
The mere identification of such a central and important issue makes the paper newsworthy for a considerably-sized audience working with point clouds. The improved results over pointMAE and comparable results with sota with much lower computational needs, showcase the potential improvements that working on the main finding can bring about. The AC recommends acceptance.
Decision: Accept (spotlight)