PCP-MAE: Learning to Predict Centers for Point Masked Autoencoders

Authors: xiangdong  zhang
shaofeng  zhang
junchi  yan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 09:44:54

Abstract: Masked autoencoder has been widely explored in point cloud self-supervised learning, whereby the point cloud is generally divided into visible and masked parts. These methods typically include an encoder accepting visible patches (normalized) and corresponding patch centers (position) as input, with the decoder accepting the output of the encoder and the centers (position) of the masked parts to reconstruct each point in the masked patches. Then, the pre-trained encoders are used for downstream tasks. In this paper, we show a motivating empirical result that when directly feeding the centers of masked patches to the decoder without information from the encoder, it still reconstructs well. In other words, the centers of patches are important and the reconstruction objective does not necessarily rely on representations of the encoder, thus preventing the encoder from learning semantic representations. Based on this key observation, we propose a simple yet effective method, $i.e.$, learning to \textbf{P}redict \textbf{C}enters for \textbf{P}oint \textbf{M}asked \textbf{A}uto\textbf{E}ncoders (\textbf{PCP-MAE}) which guides the model to learn to predict the significant centers and use the predicted centers to replace the directly provided centers. Specifically, we propose a Predicting Center Module (PCM) that shares parameters with the original encoder with extra cross-attention to predict centers. Our method is of high pre-training efficiency compared to other alternatives and achieves great improvement over Point-MAE, particularly surpassing it by \textbf{5.50\% on OBJ-BG, 6.03\% on OBJ-ONLY, and 5.17\% on PB-T50-RS} for 3D object classification on the ScanObjectNN dataset. The code is available at \url{https://github.com/aHapBean/PCP-MAE}.

Main paper: 
 introduction: 
 Point clouds are a widely used representation of 3-D objects, offering a rich expression of their geometric information. This versatility has led to their broad adoption across various application scenarios, including autonomous driving [28] , robotics [10, 30] , and the metaverse [32] . In the early developing stage of point cloud understanding, there are lots of related work proposed to enhance the ability of networks for understanding point cloud [13, 18, 25, 26, 39] where the networks typically need fully-supervised training from scratch. However, point cloud data are difficult to annotate compared to the data in 2-D vision and NLP owing to the complexity of the toughness of discriminating them. It leads to a phenomenon called data desert [9] in 3-D which refrains the development of these fully-supervised methods. Recently, numerous self-supervised learning (SSL) methods [47, 24, 9, 27] were proposed for point cloud understanding to alleviate the negative effect brought by data desert because self-supervised learning methods can utilize unlabeled data effectively by performing designed pretext task (e.g., reconstruction-based or contrastive-based) for pre-training and the learned semantic representation benefits the performance of downstream tasks.Masked autoencoders (MAEs) [16] represent a prominent and robust framework within selfsupervised learning (SSL), which demonstrates remarkable scalability and superior performance. The architecture of MAE is distinctly asymmetric, featuring an encoder and a decoder. The typical process involves dividing an image into patches, which are then selectively masked prior to encoding. The encoder only accepts visible patches as input along with their positional embedding, i.e. embedding of patch indices. The decoder receives both the latent representations of the visible patches and the tokens representing the masked patches, along with their corresponding positional embeddings. Following MAE in 2-D, Point-MAE [24] and its variants [48, 51, 14, 27] are proposed, which divide point cloud into patches where points in patches undergo normalization, achieved by subtracting the coordinates of corresponding centers and apply the mask-reconstruction paradigm in 3-D domain. However, different from the indices of image patches (the positional embeddings in 2-D are fixed for all input images [16] ), position embedding in point cloud is calculated using the coordinates of the centers belonging to patches, where the coordinates of the centers usually contain rich geometric and semantic information of point clouds. Thus we ask: Should the centers (i.e., positional embedding) for masked patches be directly given when performing masked reconstruction in the point clouds field just like the way in 2-D vision? We answer this question with an interesting phenomenon observed through experimental results in Fig. 1 : We first set the mask ratio to 100% in Point-MAE for pre-training on ShapeNet [3] , i.e., the encoder is removed to verify whether the point cloud can be reconstructed by only feeding the masked positional embeddings of the center points to the decoder after training. Intriguingly, the reconstruction results are pretty well, which are shown in Fig. 1 . This is irreproducible for 2-D MAE since when masking 100% patches and only providing the positional embedding of the masked patches (i.e., 2-D indices of patches), 2-D MAE cannot identify what the images are (as all the images with the same resolution share the same positional embeddings).The gap between 2-D and point cloud data reveals that the coordinates of the centers (i.e. positional embeddings) are essential in the point cloud field, meaning that the decoder can even abandon the output of the encoder and still reconstruct well. Thus, the reconstructing objective may make the encoder unable to learn semantic features, and we argue that the answer to the previous question is: The centers for masked patches can not be directly provided and should be utilized in a smarter and more proper way. To this end, we propose a method, which learns to Predict Centers for Point Masked AutoEncoders, termed as PCP-MAE. While existing MAE-based methods [24, 9, 27, 48, 51] overlook the importance of centers during pre-training and directly leak the masked centers for the decoder, our PCP-MAE learns more semantic representation by performing point cloud reconstruction under the auxiliary of learning to predict centers, using predicted centers replace real masked centers.The overview of the proposed PCP-MAE is illustrated in Fig. 2 . Specifically, we propose a custom pre-training task that forces the encoder to not only learn representations for visible patches but also predict the centers of the masked ones. It is achieved by feeding both visible and masked patches to the encoder. Visible patches (with centers) perform self-attention in each block as in Point-MAE, while masked patches (without centers) leverage a cross-attention mechanism simultaneously to acquire center information from themselves and the visible ones. Additionally, a dedicated objective is introduced to minimize the difference between the predicted masked centers and the ground truth values (positional embeddings). Furthermore, the predicted positional embeddings of the masked centers are used to replace the real positional embeddings, which will be fed into the decoder. This allows the network to not only encode visible centers effectively but also learn the inter-relationships between visible and masked centers. Essentially, the encoder learns to infer the missing information, leading to a more robust representation and improved performance in downstream tasks achieved through fine-tuning. The main contributions include:1) Center-aware objective makes the pre-training trivial. We identify the difference between 2-D (image) and 3-D (point cloud), i.e., positional embedding (PE) in 2-D represents indices of patches,  
related work: 
 Self-supervised learning for Point Cloud. Inspired by the recent successes of Self-Supervised Learning (SSL) in NLP and 2-D vision, an increasing amount of research has been conducted to explore the potential and strengths of SSL in the realm of 3-D vision [24, 47, 41, 1] . Existing 3-D SSL methods can be mainly divided into two categories: contrastive learning [41, 1, 17, 29, 19] and generative learning [47, 24, 50, 48, 51] . PointContrast [41] pioneers the contrastive learning in 3-D by borrowing the idea of contrastive learning in 2-D and performing point-level invariant mapping learning on two transformed views of the given point cloud. CrossPoint [1] advances this approach by engaging in intra-modal learning while also introducing an auxiliary cross-modal contrastive objective that facilitates the learning of transferable tailors the encoder and decoder into pyramid architectures, which allows for progressive modeling of spatial geometries, facilitating the capture of both intricate details and overarching semantics of 3D shapes. ACT [9] , as an MAE-based method, acquires knowledge from other modalities by adopting cross-modal auto-encoders as teachers. ReCon [27] harmoniously combines the generative framework (MAE-based) and contrastive framework to share the merits of them. Joint-MAE [14] enables better cross-modal interaction by constructing two hierarchical Figure 2 : Overview of the proposed PCP-MAE. After patch division, the centers and normalized patches are divided into visible and masked parts, with center coordinates embedded into positional embedding (PE) and patches embedded into tokens (embeddings). The encoder accepts visible tokens and PE as input, performing self-attention. Simultaneously, the weight-shared PCM (Predicting Center Module) performs cross-attention (masked tokens as query and visible along with masked tokens as key and value) to acquire knowledge to predict the positional embeddings of the masked patches. CD-L 2 refers to the l 2 Chamfer Distance loss function [11] . point cloud reconstruction. Point-FEMAE [48] extends global random masking to two branches consisting of a global branch and a local branch to capture latent semantic features, with extra modules Local Enhancement Modules introduced. We find almost all existing MAE-based methods focus on structural improvement, employing the same reconstruction objective as Point-MAE. However, as shown in Fig. 1 , we claim again that this reconstruction objective is not suitable to directly adopt from 2-D to point clouds. 
pcp-mae: 
 3.1 Patches generation and masking Patches generation. Patch-based learning paradigm [47, 24, 27] has been proven to be more effective than directly consuming the whole point cloud [25, 26] empirically. We adopt the patchbased paradigm and divide the input point cloud into point patches via Farthest Point Sampling (FPS) and K-Nearest Neighborhood (KNN) algorithm following previous work [24] . Specifically, given a point cloud with p points X ∈ R p×3 , FPS is first applied to sample n centers C from p points. Then, we adopt KNN to select k nearest points of corresponding centers to consist n point patches P:EQUATIONCoordinates of points in the point patch are normalized with the corresponding center, thus enabling better separation between patches and centers.Masking. With a pre-defined masking ratio m, we apply global random patch masking to point patches. The masked patches are denoted as P m ∈ R ⌊mn⌋×k×3 and the visible patches are denoted asP v ∈ R ⌈(1-m)n⌉×k×3, where ⌊•⌋ and ⌈•⌉ represent the floor function and the ceiling function, respectively. For simplicity, we will omit these two symbols in the subsequent text. The corresponding centers are denoted asC m ∈ R mn×3 and C v ∈ R (1-m)n×3 .Embedding. P v is embedded via a lightweight PointNet [25] to yield encoded tokens E v following Point-BERT [47] which can be expressed as follow, where D is the hidden dimension of the networks.EQUATIONWe calculate fixed positional embedding for each center using the popular form [16] as detailed in Eq. ( 3), which we call sin-cos positional embedding (sin-cos PE). For each triplet of coordinates (x, y, z) for the center:PE x = sin x e 2× 1 D/6, cos x e 2× where PE x ∈ R 1×D/3 . Then we concatenate PE x , PE y and PE z to obtain PE ∈ R 1×D . Then feed the obtained sin-cos PE into a learnable MLP called PEM (Positional Embedding Module) to yield the final PE, which is different from the operation in Point-MAE [24] . On the one hand, the sin-cos PE enables more sparse positional information compared to 3-dimension coordinate and thus is more suitable as the predicting target of our proposed task just as illustrated in Sec. 4.3. On the other hand, applying the sin-cos positional embedding to both visible and masked patches can better align the usage of known visible PE with the predicted masked PE. The process can be expressed as:EQUATIONEQUATIONEQUATIONWe employ an encoder that consists of standard Transformer blocks. Given the encoded visible tokens E v , masked tokens E m and visible positional embeddings PE v , the encoder performs self-attention to encode E v while performing cross-attention utilizing E m as input to acquire information of masked centers from itself and the visible representations. Note the self-attention and cross-attention share the parameters of Transformer blocks to avoid increasing parameters.Visible tokens encoding. The latent representation of the visible parts can be obtained by inputting E v , PE v to the encoder which utilizes self-attention mechanisms. The process can be formulated as:EQUATIONSpecifically, the self-attention in i-th Transformer block of the encoder can be expressed as:EQUATIONwhereQ v = T i-1 W Q , K v = T i-1 W K and V v = T i-1 W V and the W Q , W K , W V are learnable parameters. Note T 0 = E v + PE v .Modules for learning to predict centers i.e., PCM. The PCM shares parameters with the encoder and accepts only masked patches E m as input. Apart from the shared parameters of Transformer blocks, additional MLP for projecting latent predicted PE to masked center embeddingsPE pred m is introduced into PCM. Based on the encoded masked patches, the PCM aims to recover the centers of masked patches by acquiring knowledge from both the masked and visible representations. This can be straightforwardly understood in a form where local patterns (without telling where they are) and visible global patterns are given, and the task is acquiring positions of local patterns from the visible patterns based on the known local patterns. The cross-attention mechanisms are performed:EQUATIONSpecifically, for the i-th Transformer block in PCM, the cross-attention can be formulated as:EQUATIONwhereQ m = PE i-1 m W Q , K m = PE i-1 m W K and V m = PE i-1m W V and the W Q , W K are shared parameters. We concatenate Q v and Q m to obtain Q v+m and it's also the case for K v+m and V v+m . Note PE 0 m is the input of PCM, i.e., PE 0 m = E m .The decoder also consists of Transformer blocks and is used for point cloud masked reconstruction. The decoder in existing MAE-based methods such as Point-MAE [24] , ReCon [27] , Point-FEMAE [48] typically take T v , PE v , PE m as input along with learnable mask tokens[M] ∈ R mn×D . It can be expressed as H m = Decoder([M], PE m , T v , PE v )where the centers for the masked patches are directly provided. However, the centers are extremely important in point cloud as shown in Fig. 1 . Therefore, instead of doing that, we use the PE pred m to replace the PE m which means we use what the network learns but doesn't leak the ground truth of positional embeddings to the decoder. The process can be written as:EQUATIONwhere sg(•) means the stop-gradient operation and H m ∈ R mn×D . Stop-gradient is applied to prevent the decoder from finding reconstruction shortcuts by back-propagating through the PE pred m that modifies the weights of PCM, as illustrated in the ablation study Sec. 4.3. Finally, following Point-MAE, we adopt a projection head consisting of one MLP layer to predict the coordinates of normalized masked points in the input point cloud:EQUATIONThe objective function L consists of two parts, i.e., recovering the centers of masked patches and the points in every masked patch. The L can be written as L = ηL P C + L Recon . η is a scaling factor.Learn to predict centers. We compute the loss for predicting centers using the l 2 loss function:EQUATIONThe information of centers is removed when generating point cloud patches through normalization, thus posing no possibility for PCM to find shortcuts to recover the masked centers PE m based on E m , E v and PE v . And recovering the PE m is non-trivial because it needs the PCM to encode masked patches well and demands the visible representation T v possessing not only intra-information but also inter-information between itself and the unseen masked ones. In other words, minimizing the L P C enables the visible representation T v to contain enough information to infer the distribution of masked centers which is infeasible in the original Point-MAE.Point cloud reconstruction. The reconstruction loss L Recon is computed using the l 2 Chamfer Distance loss function [11] , written as:L Recon = 1 |P pred | a∈P pred min b∈P ∥a -b∥ 2 2 + 1 |P| b∈P min a∈P pred ∥a -b∥ 2 2 (15)Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: See Appendix E for the broader impacts of our work.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments: 
 We conduct experiments including object classification, few-shot learning and segmentation to demonstrate the superior performance of our method over Point-MAE. With slightly longer pretraining time than Point-MAE, our PCP-MAE achieves much higher performance than it and even surpasses other more complex methods, achieving state-of-the-art in some tasks, e.g., ScanObjectNN classification (OBJ-BG; OBJ-ONLY) and ModelNet40 few-shot (5-way, 10/20-shot; 10-way, 20-shot). Ablation studies are conducted to illustrate the properties of our proposed PCP-MAE. Point Sampling (FPS) from the input point cloud, it is divided into 64 point patches, each containing 32 points, selected using FPS and K-Nearest Neighbors (KNN). The PCP-MAE is pre-trained for 300 epochs using an AdamW optimizer [22] with a batch size of 128. The initial learning rate is set at 0.0005, with a weight decay of 0.05. A cosine learning rate decay scheduler [21] is utilized. Check the detailed experimental settings in Appendix A.3-D object classification on a real-world dataset. We showcase the transferability of our models by evaluating them on a real-world 3D object dataset. Specifically, we transfer our pre-trained model to ScanObjectNN [33] for classification, a dataset renowned for its classification complexity, encompassing around 15,000 real-world objects spanning 15 diverse categories. Our experimental evaluations encompass three variants: OBJ-BG, OBJ-ONLY, and PB-T50-RS. Rotation is applied for data augmentation during training following [27, 48] . The Tab. 2 presents the results, showing the superior performance of our PCP-MAE, which obtains great performance not only over Point-MAE but also over other more complex MAE-based methods and achieves state-of-the-art on OBJ-BG and OBJ-ONLY. Specifically, PCP-MAE outperforms Point-MAE by 5.50%, 6.03%, 5.17% on three variants and performs competitively or better than the leading cross-modal method ReCon [27] . Few-shot learning. We conduct few-shot learning experiments on the ModelNet40 dataset, following established protocols [24, 31] . The experiments are structured as "w-way, s-shot" and consist of four components, where w ∈ {5, 10}, representing the number of randomly selected classes, and s ∈ {10, 20}, indicating the number of randomly selected samples for each w. Each component undergoes 10 independent trials. The reported results include the mean accuracy and standard deviation, as detailed in Tab. 3 . Results show that with limited downstream finetuning data, our PCP-MAE exhibits outstanding generalization ability among existing single-modal and cross-modal methods, achieving SOTA in 5-way, 10-shot; 5-way, 20shot; 10-way, 20-shot experiments. Specifically, 1.1%, 1.3%, 0.9%, 0.9% performance gains over Point-MAE on four settings are obtained. 3-D scene segmentation. Scene segmentation is a challenging task, especially in large-scale 3-D scenes, as it demonstrates the ability of models to comprehend contextual semantics and intricate local geometric relationships. We conduct 3-D scene segmentation on the S3DIS dataset [2] , which provides densely annotated semantic labels for point clouds. The results can be observed in Tab. 4. PCP-MAE shows better ability under scene segmentation scenarios and outperforms Point-MAE by 1.1% and 0.5% in mAcc and mIoU, respectively.We conducted several experiments to demonstrate the efficacy of PCP-MAE, with results on three variants of ScanObjectNN (%) reported, maintaining experimental settings consistent with Sec. 4.2. PCP-MAE is pre-trained on ShapeNet before fine-tuning on ScanObjectNN.Main components in the PCP-MAE. Compared to Point-MAE which only performs point cloud reconstruction, our PCP-MAE also adopts reconstruction as a pre-training target and differs from Point-MAE mainly in two aspects: 1) Adding PCM which shares parameters with the encoder to guide the encoder to learning to predict centers for masked patches. 2) Instead of directly providing the ground truth PE m , we replace it with the predicted results PE pred m . To validate the effectiveness of these two operations in PCP-MAE, we conduct experiments by varying the utilization of PCM and the predicted PE pred m on the performance of our PCP-MAE. The Tab. 5 presents the results, showing that only adopting predicting centers as pertaining target outperforms the from-scratch baseline which means the task using PCM proposed by us is meaningful as an isolated pre-training task. Results also indicate that while guiding encoder learning to predict centers improves the performance (w/ PCM), replacing PE m with PE pred m further enhances the performance.Predicting Targets. The centers can be expressed in two forms: coordinates with a 3-dimensional representation (x, y, z) and positional embedding (PE) with a D-dimensional representation obtained by embedding the coordinates, where D refers to the dimensionality of the model. Therefore, there are at least two available targets for center prediction. We claim that the PE form is more suitable in this situation because it provides sparser positional information compared to the 3-dimensional coordinates, and it represents a high-dimensional semantic space that is easier and clearer for the model to learn. The experiment results reported in Tab. 6 substantiate this claim.Stop-gradient operation. To prevent the decoder from finding shortcuts during pre-training, we apply a stop-gradient operation sg(•) to the predicted centers before feeding them into the decoder. This stops the back-propagation of gradients to the PCM, ensuring that any reduction in reconstruction loss does not falsely adjust the predicted centers PE pred m , as the reconstruction loss only contains information for points in patches but no semantic information for the centers. Experimental results, shown in Tab. 7, indicate a significant performance drop when the stop-gradient operation is omitted. 
conclusion: 
 In this paper, we first identify the disparities in positional embeddings between 2-D vision and 3-D point cloud, finding that direct leakage of masked centers makes the pre-training in Point-MAE and existing MAE-based methods trivial as shown in Fig. 1 . To address this issue, we introduce PCP-MAE, which adds a novel objective to guide the encoder to predict positional embeddings for the masked centers, enabling the encoder to learn much more semantic representations. A Predicting Center Module which uses cross-attention and shares parameters with the encoder is proposed to achieve this. Exhaustive experiments on SSL benchmarks showcase the superior performance of our method compared to Point-MAE, setting new SOTA results across various tasks.  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: To our knowledge, the model proposed in our paper does not pose a risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We properly cite the creators of models and datasets mentioned in our paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Masking ratio. We conduct experiments using various masking ratios to determine the most suitable option for our PCP-MAE. The results are illustrated in Fig. 3 , and we identify a masking ratio of m = 0.6 as optimal.Loss function for predicting centers. The pre-training objective guiding the encoder to predict centers entails a regression task. We explore various regression loss functions to select a suitable loss function for our PCP-MAE. The results are reported in Tab. 9. Notably, the L 2 Distance loss emerges as the best choice. Consequently, we adopt the L 2 Distance loss for center prediction.Pre-training augmentation. The pre-training augmentation plays an important role in increasing the variance of training samples and thus affecting the performance of our PCP-MAE. We experiment with different augmentations for pre-training and validate their performance through fine-tuning. Additional analysis on the data augmentation. We used the Scale&Translate+Rotation augmentation for pre-training because this combination benefits our PCP-MAE, differing from previous baselines. Since we apply different pre-training augmentations compared to prior methods, we report the performance of the baseline (Point-MAE [24] ) under various augmentations in Tab. 11 for clarity. Additionally, we test previous SOTA methods (Point-FEMAE [48] and ReCon [27] ) with our augmentation strategy and observe a slight performance drop, as shown in Tab. 12. These results indicate that the explored augmentation composition benefits our methods more than others. This improvement can be attributed to the fact that the combination of augmentations enables a variety of centers, allowing our PCP-MAE to learn a richer distribution of center information and become more robust. In contrast, the baseline method Point-MAE benefits only slightly from this augmentation due to relying solely on point cloud reconstruction during pre-training, and prior SOTA methods like ReCon and Point-FEMAE do not empirically benefit from it.Different η in the objective function. As discussed in Sec. 3.4, the objective function includes a scaling factor η to balance the point reconstruction loss L Recon and the center prediction loss L P C , formulated as L = ηL P C + L Recon . As depicted in Fig. 4 , the optimal performance is achieved when η = 0.1. Therefore, we adopt this value as the scaling factor for the objective function.Projector for predicting centers. After obtaining the output of the last transformer block in PCM, it is passed through a projector comprising MLP layers, ReLU, and LayerNorm to predict masked centers. In contrast, the output of the encoder is directed into a decoder, which consists of transformer blocks, for further reconstructing the point cloud. Thus, we conduct experiments to investigate whether additional transformer layers are necessary for PCM to enhance the decoding of the predicted Table 11 : Performance comparison of Point-MAE with different augmentations. The star ( * ) marks the setting used by the original Point-MAE [24] , the dagger ( † ) by peer SOTA methods such as Point-FEMAE [48] and ReCon [27] , and the double dagger ( ‡ ) by us. Shared parameters between the encoder and the PCM. We conduct an ablation study to compare the effects of sharing parameters versus not sharing parameters between the encoder and the PCM. As shown in Tab. 14, sharing parameters not only significantly reduces the number of parameters but also enhances the performance of the model.This improvement can be attributed to the following reasons: The encoder is responsible for encoding visible tokens to obtain semantic representations and is the only component used during fine-tuning.When the encoder and PCM share parameters, updates from the PCM are directly incorporated into the encoder. This allows the encoder (or PCM) to learn more semantic representations and more effectively understand the relationship between visible representations and masked patches. Although the loss for predicting centers back-propagates to the encoder due to the cross-attention between the PCM and encoder, its impact on the encoder is indirect when parameters are not shared. In this case, the parameters of the PCM are useless and wasted in downstream tasks because only the encoder is utilized during fine-tuning. In conclusion, sharing parameters makes the updates of the PCM directly influence the encoder, leading to more efficient and effective learning and enabling the encoder to learn more semantic representations.Concerning the above advantages, we choose to share weights between the encoder and the PCM in our method.After pre-training Point-MAE with a 100% mask ratio, i.e.,, the encoder is completely discarded during training and only the decoder is utilized, the decoder is able to reconstruct the full point cloud using only masked positional embeddings, as shown in Fig. 1 . We refer to this as the "masked center leakage", which demonstrates that the decoder does not necessarily rely on the encoder's representation during the pre-training of Point-MAE.To extend the validation of this phenomenon beyond the ShapeNet dataset [3] , which primarily contains objects with clear and well-defined shapes, we conducted experiments on another dataset, ScanObjectNN [33] . This dataset poses a more significant challenge, as it consists of real-world scans that include background clutter and occlusions. The visualization results in Fig. 5 show that the masked center leakage phenomenon is also present in this more complex dataset, further demonstrating the generality of the phenomenon that we identified.PCP-MAE is a simple and effective method that improves Point-MAE by additionally learning to predict the centers, which significantly boosts performance. However, there are some limitations in PCP-MAE, which may be two-fold. PCP-MAE is a single-modal self-supervised method. Nevertheless, the current dataset of 3-D point clouds is constrained in size due to the challenges associated with collecting point cloud data, which in turn limits the wider applicability of our approach. Additionally, while PCP-MAE capitalizes on generative learning, it does not leverage the benefits of contrastive learning and some other works such [27, 7] show great performance by harmoniously combining generative learning [56, 53] and contrastive learning [54, 55, 52] .To address these issues, future works could focus on developing a multi-modal PCP-MAE, explainable AI [7, 6, 5, 8] or a hybrid model that effectively combines the strengths of both generative and contrastive learning.Our PCP-MAE demonstrates a marked improvement over the existing Point-MAE and establishes new state-of-the-art (SOTA) benchmarks across a variety of tasks in point cloud understanding. This enhancement is pivotal in advancing technologies reliant on precise spatial recognition [43, 42, 44, 45, 6] , such as autonomous driving, which demands accurate environmental perception for safety and efficiency. Moreover, the versatility of the PCP-MAE extends its utility to other critical applications, including urban planning, where detailed and scalable 3D city modeling is essential, and in augmented reality, enhancing interactive experiences by integrating more accurate virtual information with the real world. While our PCP-MAE marks an advancement in point cloud understanding with broad applications ranging from autonomous driving to urban planning, it is also susceptible to potential negative societal impacts. The primary concerns stem from its potential use in surveillance systems, where enhanced spatial recognition capabilities could lead to privacy infringements.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: **Summary of the paper:**
The paper identifies a central problem with PointMAE, that is, the position of patch centers that are provided to the decoder leaks information about the 3D structure of the object and therefore a reconstruction of the point cloud is possible without considering any output of the encoder. Then, it suggests an updated mechanism where the encoder also predicts the center of the patches which is then used in the decoder to better encourage the encoder to embed relevant information for the reconstruction. The modified PointMAE is shown to consistently perform better at downstream tasks.

**Summary of the reviews:**
The reviewers find the insight on the different role of patch center embedding for point-clouds and fixed grids interesting, informative and important. They also find the proposed modification relevant. They finally appreciate the improved results over PointMAE and the extensive side/ablation studies.

On the other hand, the reviewers believe the presentation, text, figures, and equations can be improved, and that some of the reported results are confusing. They also asked for the addition of PCP on more recent variants of masking-based SSL on point clouds. 

**Summary of the rebuttal and discussions:**
The authors clarified some confusions about the presentation and the reported numbers. They also argue for the computational efficiency of PCP-MAE over more recent state of the art methods. They further provided additional experiments that (1) show improvements when PCP is added to Point-BERT and Point-FEMAE, and (2) evaluates the method on a new indoor dataset. 

Some reviewers remained unconvinced regarding the comparison to the state of the art and considered most improvements coming from the different augmentations. The authors provided their code anonymously and reviewer GmbZ put a laudable effort in reproducing results, using their code, and with 10 random seeds that makes it quite reliable. 

**Consolidation report:**
Most concerns were satisfactorily refuted. The main outstanding concern is the comparison with state-of-the-art which only makes PCP-MAE comparable to them and not superior. The reproduced results of the reviewer, while slightly lower than the reported ones by the authors, show similar trends, i.e. improved performance over pointMAE but only comparable performance to the state of the art. The authors demonstrate that their method has significantly better computational efficiency.

AC’s additional take: the family of PointMAE is an active field of research both for improvement and application. The problem that the paper finds with PointMAE is fundamental and indisputable. The significance of this finding by itself was more or less undermined by the reviewers. The proposed approach, although can be improved, is simple and sensible and achieves some improvement over the main baseline and is comparable to the state of the art while being significantly more efficient. The identification of a central problem and proposing simple mitigation are considered significant steps forward for the field.

**Recommendation:**
The mere identification of such a central and important issue makes the paper newsworthy for a considerably-sized audience working with point clouds. The improved results over pointMAE and comparable results with sota with much lower computational needs, showcase the potential improvements that working on the main finding can bring about. The AC recommends acceptance.

Reviews: review: paper_summary: paper_summary: The paper proposes a masked autoencoding based self-supervised approach for 3D point clouds. The approach which terms as PCP-MAE, learns to predict centers for Point Masked AutoEncoders. The paper investigate that the coordinates of the centers are essential in the point cloud field and the decoder can even reconstruct the masked patches without the encoder representation of the visible patches. Thus, the paper introduces another pre-training objective which predicts the centers and uses them to replace directly provided centers, leading to improved performance. Moreover, this allows the network to not only encode visible centers effectively but also learn the inter-relationships between visible and masked centers. Finally, the approach is efficient and outperforms Point-MAE on multiple benchmark datasets.
strength_weakness: strength_weakness: - The paper writing can definitely be improved and sometimes it's really to hard to follow the underlying idea behind it.
- The figure 2 can be misleading and it doesn't correlate with the equation 10. In figure 2, it looks like only the masked tokens are passed to the PCP module but in equation 10 which is given as:$PE_{m}^{pred} = PCM (E_{m}, E_{v}, PE_{v})$, it looks like PCP module uses encoded representation of masked patches, visible patches and positional embeddings of visible patches. Can the authors please clarify about this?
- In Table 2, PointMAE achieves 90.02 88.29 85.18 on three variants of ScanObjNN but in the ablation study (c.f. Table 5 ) the results on the 4th row ( just reconstruction only ) achieves 92.94 92.42 88.65. Isn't only doing reconstruction equivalent to PointMAE? Does it mean the performance boost comes from the positional encoding?
- Although approach looks simpler and effective, but it only shows improvement by incorporating the additional details on PointMAE. Compared to other the baselines like Point-FEMAE, it doesn't outperforms them on multiple datasets. Can the authors please provide additional results by incorporating PCP and new positional encoding in multiple MAE approaches for point clouds?
- Did the authors adapt the same set of augmentations for pre-training baselines?
- The methods looks very sensitive to $\eta$ (c.f. Figure 4) 
- Some additional results on indoor datasets like ScanNet would be helpful.
questions: questions: Please see the questions and the suggestions in the weakness section.
limitations: limitations: Yes, the authors have adequately addressed the limitations.
paper_summary: paper_summary: This work examines representation learning of 3D point clouds using masked autoencoding. A known issue with such an approach is that the coordinates of the patch centers leak significant information about the geometry and semantics of the shape being reconstructed, which degrades the representations learned by these approaches. This work proposes to add an extra criterion, whereby the model predicts the positions of the masked patch centers. Pretraining is performed on ShapeNet. This simple change leads to significant improvements over Point-MAE when finetuning on ScanObjectNN and ModelNet40 for 3D classification, 3d scene segmentation, and object part segmentation.
strength_weakness: strength_weakness: * Improvements over previous MAE-based methods, such as Point-FEMAE, are quite marginal, and perhaps not statistically significant. While the contributions of other MAE related methods are orthogonal and could be potentially combined with the proposed method, it is unclear if the issue of 3D center point position leakage is still significant when combined with other tools.
* Numerical results are primarily restricted to end-to-end fine-tuning. Would be curious to see the results of linear or frozen evaluations to assess the quality of the representations, as finetuning from scratch (i.e., random representations) can already achieve much of the topline performance in many cases.
questions: questions: * If the authors have the capacity to conduct such an experiment, I am primarily curious to see the effect of the proposed center point prediction task with stronger baselines which provide advances orthogonal to the contributions in this work.
limitations: limitations: N.A.
paper_summary: paper_summary: The paper studies the point cloud pretraining under the self-supervised learning paradigm. Authors experimentally found that the position embedding in the decoder may decrease the learning ability of the encoder and propose a new method to overcome this issue.
strength_weakness: strength_weakness: 1. In line 30, authors mentioned the difficulty in collecting 3D dataset and want to use SSL to solve this problem. However, 3D data is different with 2D images which can be collected easily on the internet. For the experimented dataset like Scanning datasets, the label will be collected during scanning, then why SSL for 3D is important?
2. According to table 1, Point-FEMAE achieves almost the same performance as the proposed model, thus it is hard to say that the reconstructing objective may make the encoder unable to learn semantic features in Line 70. 
3. The improvements of the proposed method over Point-FEMAE are marginal. 
4. One possible reason behind the phenomenon of the position embedding may be the simplicity of the dataset. Would the phenomenon also hold for lidar datasets?
questions: questions: pealse refer to the weakness
limitations: limitations: N/A
paper_summary: paper_summary: The paper proposes a novel self-supervised learning method called PCP-MAE for point cloud understanding. The key innovation of PCP-MAE is that it guides the model to learn to predict the positional embeddings of the centers of masked regions, rather than directly providing the coordinates. This approach encourages the encoder to learn richer semantic representations, leading to performance improvements on downstream tasks such as point cloud classification.
strength_weakness: strength_weakness: 1.There are significant concerns about the actual effectiveness of the center point prediction method being discussed. Although the author conducted numerous experiments to demonstrate the superiority of their method, these comparisons with previous methods are unfair. This unfairness is reflected in the following aspects: 1. Although this method has achieved significant improvement, it seems that the main improvement comes from the use of more complex data augmentation tricks during pre training, without a fair comparison with previous methods when using the same tricks. Due to the lack of detailed code provided by the author, based on my personal experience, when using only the center point prediction method with the same settings or tricks, the proposed method does not show significant improvement and may even experience performance degradation. For example, when using the center point of ground truth without center point prediction and with the same settings such as position encoding and data augmentation, I don't think the author's center point prediction will be better than using the ground truth method.

2.On line 69, the author get the conclusion that 'the reconstructing objective may make the encoder unable to learn semantic features'. I would like to know how the author came to this conclusion. Can a randomly initialized token with a 100% mask rate be used to reconstruct the approximate shape of a point cloud using only the center point as the position encoding through an encoder? I don't think so. This phenomenon does not mean that the encoder cannot learn semantic features. Secondly, the author only conducted a qualitative analysis without providing a detailed quantitative comparison. In fact, the CD distance of the point cloud reconstructed by this method is much greater than that of the Point MAE method, indicating that the effect of the point cloud reconstructed by this method is very poor.

3.The author's improvement is incremental and limited, so I do not think that this paper meets the contribution criteria of NeurIPS. The proposed method is only an improvement on Point MAE, in fact, Point MAE is just one method in the field of point cloud self supervised learning.

4.The author did not demonstrate the generality of the proposed method. Assuming that this central prediction method is effective, it should be applicable to all MAE based point cloud self-supervised methods, such as Point BERT, ACT, Point-M2AE, PointFEMAE, I2P-MAE, Recon, etc. The author lacks further analysis of the generality of the proposed ideas.

5.Insufficient Visualizations and Explanations: The paper lacks some key visualizations and explanations that would aid reader understanding. For example, the pipeline (Fig. 2) does not clearly annotate the "stop gradient" operation, which is a crucial component of the proposed method. Additional visualizations and explanations would help readers better comprehend the technical details of PCP-MAE.
questions: questions: Although the author has demonstrated superior improvement in various experiments, this improvement seems to come more from trick's improvements (such as data augmentation) rather than the proposed method itself. Meanwhile, the author lacks a detailed analysis of the generality of the proposed method. Therefore, I reject this article and encourage the author to conduct a more in-depth, thorough, and fair analysis and comparison.
limitations: limitations: The author has pointed out some issues in the Limitations section, and I hope the author can make more improvements based on the review comments, so that their work becomes truly solid rather than superficial incremental improvements.
paper_summary: paper_summary: This paper makes a novel observation (at least to me) that for mask-based autoencoder paradigm for point cloud self-supervised pretraining, the centers of patches are important and the reconstruction objective does not necessarily rely on representations of the encoder. This is different from the 2-D case on mask-based autoencoding for images. Then the authors introduce a simple yet strong scheme that directly learning to Predict Centers for Point Masked AutoEncoders to prevent the encoder from the failure of semantic information learning. This approach is more efficient than baselines, and achieves state-of-the-art performance on public datasets.
strength_weakness: strength_weakness: The authors may discuss the limitation of their approach (if there was any).

The abstract can be improved, e.g. the numbers in abstract is vague as there is no specific metric.
questions: questions: Could this approach inspire the development in other areas of selfsupervised learning?
limitations: limitations: N/A

Comments: title: Thanks for the clarifications
Look forward to further reply.
Look forward to your further reply.
Clarification on Method's Effectiveness: Beyond Augmentation Tricks
Response to Authors
Response 3 (Response 1 continued.)
Response 2 (Response 1 continued.)
Response 1
Results reproduction
Anonymous source code and checkpoints are available.
Quick Response to author
Continued rebuttal
Code submission
comment: Dear All,

Based on the author's recommendation, I retrained PCP-MAE from scratch and conducted experiments on three variations of ScanObjectNN using ten different seeds for each. The experimental results are as follows:

**Best result of ten experiments.**

| Methods  |  OBJ-BG |  OBI-ONLY | PB-T50-RS  |
| ------------ | ------------ | ------------ | ------------ |
| Point-MAE  | 93.12  | 92.77  | 89.04  |
| PCP-MAE （Reproduce） | 95.01  | 93.29  | 89.76  |
| PCP-MAE （Author） | 95.53  | 94.32 | 90.35  |
| Recon  |  95.19 | 93.12  | 90.25  |

**Average result of ten experiments.**

| Methods  |  OBJ-BG |  OBI-ONLY | PB-T50-RS  |
| ------------ | ------------ | ------------ | ------------ |
| Point-MAE  | 92.67  | 92.31  | 88.59  |
| PCP-MAE（Reproduce）  | 94.46  | 92.65  | 89.37  |
| PCP-MAE （Author） | 94.56  | 93.35 | 89.57  |
| Recon  |  94.37 | 92.51  | 89.95  |

I only provided the latest reproduction results. My recommendation is borderline (neither accept nor reject), and I fully respect the final decision of the AC.
I appreciate the authors for further clarifications on the approach and the experiments. However, I'm still not fully convinced on the results. For example if we remove the augmentation trick from the approach, the method doesn't outperform ReCon, Point-FEMAE on ScanOBJNN (c.f. Table 10, row 3). As I mentioned before, given the randomness in 3D datasets, I'm more interested in the comparison on other datasets, even on S3DIS, I don't see a significant difference in the results (c.f Table 4).
Dear Reviewer GmbZ:

We would like to express our sincere gratitude for your thorough and insightful review comments. Your feedback has been invaluable in helping us refine and improve our paper significantly.

In our most recent discussion, we have provided the **training logs** for our model. These logs offer a transparent and verifiable record of our method's performance, allowing you to easily confirm that our approach indeed achieves the results reported in the paper. This step towards greater reproducibility is part of our commitment to scientific rigor and openness.

We are particularly interested in understanding whether these newly provided training logs have influenced your perspective on our work.

We value your expertise and judgment, and we are eager to hear your thoughts on this new information. If you have any remaining questions or areas where you feel further clarification is needed, please don't hesitate to let us know.
Dear Reviewer DT8p:

Thanks again for the valuable feedback and constructive suggestions. As the discussion phase is coming to an end, we have provided our understanding and perspective on the significance of self-supervised learning for point clouds in our rebuttal, along with new experimental results. We would like to know if our rebuttal has addressed your concerns, and if you have any further questions that need clarification from us.
Thank you for your response, which gives us the opportunity to clarify that the benefits of our method do not stem from augmentation tricks.

Firstly, we attempted to apply the mentioned augmentation tricks to previous SOTA methods, but this did not yield improvements and even showed a declining trend, as demonstrated below:

Method|OBJ-BG|OBJ-ONLY|PB-T50-RS
-|-|-|-
Point-FEMAE (+ Our augmentation)|94.32|92.94|89.38
ReCon (+Our augmentation)|94.49|92.77|89.55
PCP-MAE (Ours) |  **95.52** | **94.32** | **90.35**

Besides, we believe that different methods are suited to **different hyperparameters** (including augmentation), and it is widely accepted to apply new hyperparameters to the newly proposed methods. Moreover, we have also provided results for the baseline method under **our hyperparameter** settings for **fair comparison**, which show that our method significantly outperforms both the baseline and other state-of-the-art methods:

Methods|OBJ-BG|OBJ-ONLY|PB-T50-RS
-|-|-|-
Point-MAE (baseline) |92.60|91.91|88.42
Point-MAE (+ our aug) |92.94|92.25|88.86
Ours |  **95.52** | **94.32** | **90.35**

Furthermore, our method (PCP) can serve as a **plug-in** module, applicable to a wide range of models. This demonstrates that our approach is not merely a trick, but a method with broad applicability:

Methods|OBJ-BG|BJ-ONLY|PB-T50-RS|
-|-|-|-
Point-BERT|87.43|88.12|83.07
Point-BERT+PCP ($\eta$=2.0)|89.32|89.84|84.77|
Point-FEMAE + (Our Aug)|94.32|92.94|89.38
Point-FEMAE+PCP ($\eta$=0.1) + (Our Aug)| **95.00** | **93.45** | **89.83**

Additionally, we would like to reiterate that one of our contributions is revealing a vulnerability in the Point MPM (Masked Point Modeling) series of methods. Specifically, we showed that it is possible to reconstruct masked point clouds without needing an encoder or decoder. This challenges the assumptions of many researchers and can stimulate deeper reflection in the field.
Thanks for the detailed rebuttal. The authors have addressed some of my concerns regarding the paper diagram and promise to improve the manuscript. I have also read other reviewers comments and rebuttals by the authors and I conclude that although the PCP-MAE shows some improvement compared to other SOTA methods, I agree with Reviewer GmbZ that most the improvement comes from the other tricks like augmentation. Moreover, I would encourage the authors to study the impact of proposed approach in other methods more rigorously. Furthermore, I understand the randomness in current 3D datasets, so it would be important to study the impact of the approaches on datasets like ScanNet. Even on S3DIS dataset, the approach doesn't show any improvement compared to ReCon, Point-FEMAE. So, I will maintain my original score.
> Q4. Our PCP-MAE is with minimal inspiration.

We strongly believe our PCP-MAE is well-motivated and novel. And our approach aligns closely with the found motivation. **This perspective is shared not only by us but also by four other reviewers (Reviewer MoXR, nJYR, DT8p, rD3v) who agree that our method is innovative and sheds new light on the differences in positional encodings between 2D MAE and Point-MAE.** It opens up new directions for improvement in the field of point cloud self-supervised learning. 

To the best of our knowledge, prior to our work, no one had observed that the decoder in 3D SSL does not necessarily rely on the encoder's representation before which is significantly different in the 2D counterpart. This led us to hypothesize that the distinction arises from the differing roles of positional encodings in 3D versus 2D contexts, prompting us to develop a straightforward yet effective approach, PCP-MAE.

Point-MAE has been a foundational model in the 3D SSL landscape, with most leading methods such as ACT, ReCon, I2P-MAE, and Point-FEMAE building upon it. Enhancing this influential model carries significant importance. 

Our method not only highlights a potential drawback in existing MAE-based frameworks but also inspires the development of other 3D data representations, such as meshes and voxels. In these contexts, the application of positional encodings might require reevaluation, particularly if they contain context-specific information, as opposed to merely context-agnostic details.

> Q5. The words that the authors want to say.

Setting aside the fact that our method improves upon Point-MAE with affordable additional computational demands, achieves state-of-the-art in some benchmarks, and enhanced training efficiency compared to other SOTA methods, **it is crucial to recognize that in high-quality research, performance is not the sole criterion. The core idea and its contribution to advancing thought in the field merit greater attention.** A valuable research work should be evaluated not only by its performance metrics and quantitative outcomes but also by its potential to inspire deeper understanding and innovation within the relevant community.

Thus, while you might focus on the marginal numerical improvements our PCP-MAE method offers over the much more complex cross-modal method, ReCon, we encourage you to also consider the deeper insights it provides. Exploring the true significance and advantages of our method will reveal the broader value of our work.

Kind regards, The Authors
> Q2. Considering that the method does not show improvements over the state-of-the-art (ReCon) and is merely an incremental improvement over Point-MAE.

We previously addressed this question in our rebuttal. Initially, our method was motivated by the observation that point cloud reconstruction does not necessarily rely on the encoder's representation. Hence, we introduced the Point Center Masking (PCM) technique to guide the encoder in predicting masked centers effectively.

Our method enhances Point-MAE with reasonable additional computational needs and achieves state-of-the-art (SOTA) performance in some benchmarks. **Although our performance improvements over previous SOTA methods like ReCon are marginal, it is crucial to highlight the distinct advantages of our approach:**

1. **Independence from Pre-trained Models:** Unlike ReCon, which heavily relies on pre-trained models such as the text encoder from CLIP and ViT-B pre-trained on ImageNet, our PCP-MAE is simpler and more concise, as it does not depend on external pre-trained models. This simplicity facilitates easier implementation in real-world scenarios.

2. **Single-modal Data Utilization:** The pre-training of ReCon not only requires point cloud data but also paired images and text, whereas our PCP-MAE exclusively utilizes single-modal data, i.e., point cloud. The need for paired data significantly hampers the scalability of the ReCon model, as larger models require larger amounts of data according to the scaling law. To build datasets larger than ShapeNet, it is necessary to collect extensive amounts of unlabeled point cloud data, render paired images, and manually annotate corresponding text descriptors. The rendering process is time-consuming, and annotation is labor-intensive, contradicting the initial intentions of self-supervised learning. Currently, ReCon uses the labels from point cloud samples as text inputs in the ShapeNet pre-training dataset, which somewhat violates the principles of self-supervised learning. **In contrast, our PCP-MAE can scale easily using only unlabeled point cloud data.**

3. **Enhanced Efficiency:** ReCon introduces greater complexity with more trainable parameters, whereas our PCP-MAE significantly improves efficiency. We've detailed these comparisons in the rebuttal. Here comes the comparison again for clarity:

    Method|Pre-training|| Fine-tuning| | | | |
    -|-|-|-|-|-|-|-
    ||Params (M)|Time (s/epoch)|Params (M)|OBJ-BG Time (s/epoch)|OBJ-ONLY Time (s/epoch)|PB-T50-RS Time (s/epoch)|ModelNet40 Time (s/epoch)
    PCP-MAE (ours)|29.5 (1.00x)|120 (1.00x)|22.1 (1.00x)|10 (1.00x)|10 (1.00x)|49 (1.00x)|30 (1.00x)
    ReCon|140.9 (4.78x)|452 (3.77x)|43.6 (1.97x)|13 (1.30x)|13 (1.30x)|60 (1.22x)|42 (1.40x)

Though the improvements of our PCP-MAE over the cross-modal method ReCon are not statistically significant, our method notably excels in implementation feasibility and training efficiency.

> Q3. Our PCP-MAE is with limited generality.

We demonstrate the generality of our method through demo experiments, including incorporating the proposed PCM into Point-BERT and Point-FEMAE. Without extensive exploration of architecture and the hyperparameter $\eta$, we still achieve improvement over the original methods:

Methods|OBJ-BG|OBJ-ONLY|PB-T50-RS
-|-|-|-|
Point-BERT|87.43|88.12|83.07
Point-BERT+PCP ($\eta=2.0$)|89.32|89.84|84.77
Point-FEMAE $^\dagger$|94.32|92.94|89.38
Point-FEMAE+PCP ($\eta=0.1$) $^\dagger$|95.00|93.45|89.83

$^\dagger$ Reproduced results adopting our augmentation $i.e.$, Scale&Translate+Rotation for pre-training.

**We believe that simply by varying $\eta$, we can achieve further improvements, and additional exploration of the architecture could lead to even higher performance. This also indicates that there is substantial room for further exploration, contradicting the notion of limited generality as you suggested.** We leave the development of a universal PCP incorporation framework to future work.
Dear Reviewer GmbZ,

First, we want to express gratitude for your responsible and thorough review. **We are also pleased to have clarified, to some extent, a previous misunderstanding; specifically, that the improvements observed are attributable to our method, not to a "data augmentation trick."**

Given that there remain some concerns, we would like to address the issues you raised in your comment.
> Q1. The authors have somewhat overstated their results. The authors reported their results in Table 2 as 95.52%, 94.32%, and 90.35%, which differ significantly from the results I reproduced: 94.32%, 93.28%, and 89.94%, respectively. The discrepancies are 1.20%, 1.04%, and 0.41%.

We agree with you that random seeds can significantly impact experimental results. However, **it is important to consider that variations in hardware, such as different machines and GPU types, as well as differing computational environments, also play a crucial role in the performance outcomes.** This is particularly evident in the field of 3D SSL, where such factors are well-known to affect results. That's to say, discrepancies between the machines used for pre-training and those used for fine-tuning can lead to a performance drop. This is exemplified by our observations that directly downloaded checkpoints from Point-FEMAE and ReCon yield results that are consistently lower than the initially reported outcomes:
Methods|OBJ-BG|OBJ-ONLY|PB-T50-RS
-|-|-|-
ReCon (reported)|95.18|93.63|90.63
ReCon (reproduced)|95.00|92.94|90.18
Point-FEMAE (reported)|95.18|93.29|90.22
Point-FEMAE (reproduced)|94.49|92.77|89.73

To address the discrepancies in your reproduced results and the reported results, we suggest the possibility of machine inconsistency between pre-trained machine (ours) and the fine-tuning machine (yours). For a truly fair comparison, we recommend pre-training our model directly rather than using pre-provided checkpoints to prevent the performance drop brought by the different machines (our machine and your machine). 

Additionally, it is quite unexpected that you got such a bad performance with our PCP-MAE. For instance, you reported only a 94.32% accuracy on the OBJ-BG using your setup, whereas we routinely achieve performances exceeding 95% with relative ease. Furthermore, for the OBJ-ONLY, we replicate the methodology Point-MAE used to produce their results, which involves running multiple experiments at checkpoints 300, 275, and 250 (refer to the issues discussed on the official Point-MAE GitHub repository). We achieved 94.15% at ckpt-300 and 94.32% at ckpt-275. Both of these checkpoints easily yield performances above 93.45%, which is higher than the results you obtained.

**We provide the logs for three variants of ScanObjectNN (OBJ-BG, OBJ-ONLY, PB-T50-RS), which we got in April 2024. We have updated these logs at the anonymous code link https://anonymous.4open.science/r/2128-PCP-MAE-529D for you to check. The timestamps in these logs are continuous, indicating that the experiments were conducted continuously and fairly without selectively choosing high-performing logs.** Specifically, there are 10 continuous logs for the OBJ-BG, 6 for the OBJ-ONLY (ckpt-300), 8 for the OBJ-ONLY (ckpt-275), and 100 for the PB-T50-RS. Please review the logs at the provided code link. The specific accuracies from these logs are detailed below:


Benchmark|1|2|3|4|5|6|7|8|9|10
-|-|-|-|-|-|-|-|-|-|-
OBJ-BG (ckpt-300)|**95.53**|94.14|94.83|94.14|94.32|95.35|94.49|94.83|93.45|94.49
OBJ-ONLY (ckpt-275)|93.29|92.94|93.29|92.60|93.12|**94.32**|93.63|93.63|||
OBJ-ONLY (ckpt-300)|93.45|92.77|93.80|93.63|92.59|**94.15**|
PB-T50-RS (ckpt-300)|89.49|**90.35**|88.86|89.55|89.49|89.83|89.63|89.24|89.83|89.38|

The results of this logs can be summarized as: 

Best results: 
Methods|OBJ-BG|OBJ-ONLY||PB-T50-RS
-|-|-|-|-
Point-MAE|93.12|92.77||89.04
PCP-MAE|95.53 (ckpt-300)|94.32 (ckpt-275)|94.15 (ckpt-300)|90.35 (ckpt-300)
Recon|95.19|93.12||90.25

Average results:
|Methods|OBJ-BG|OBJ-ONLY||PB-T50-RS|
|-|-|-|-|-|
Point-MAE|92.67|92.31||88.59
PCP-MAE|94.56 (ckpt-300)|93.35 (ckpt-275)|93.40 (ckpt-300)|89.57 (ckpt-300)
Recon|94.37|92.51||89.95

**The results show that we do not overstate our results.**

We also understand that you directly ran the ReCon pre-training checkpoints on your machine, which may have also introduced architectural inconsistencies, leading to a performance drop, albeit less than ours. Therefore, comparing your results directly with ours might not provide a fair assessment. However, due to the more significant machine and GPU inconsistencies encountered when you use the checkpoints trained on our machine, we recommend that you run the pre-training PCP-MAE yourself and then fine-tune to conduct an entirely fair comparison.

**Finally, we wish to affirm that we are fully responsible for the logs (results) provided and guarantee their reproducibility.** We will definitely release the corresponding fine-tuning checkpoints for each benchmark.
Dear Authors, Reviewers, and AC,

After running the code provided by the authors, I found that the proposed method does indeed show some improvement over the baseline Point-MAE. However, I think the authors have somewhat **overstated their results**. As shown in the following tables, considering that the method does not show improvements over the state-of-the-art and is merely an incremental improvement over Point-MAE with limited generality and minimal inspiration for other research, I am raising my score to a border reject. While I acknowledge that the authors' work has some merit, I consider it to be a gradual contribution and do not believe it meets the NeurIPS acceptance standards.

Specifically, I reran the authors' code on the classification task of ScanObjectNN using the recommended settings. I conducted a relatively fair comparison between PCP-MAE, the baseline Point-MAE, and the state-of-the-art method Recon. I repeated the experiments ten times on three different variants of ScanObjectNN, using ten identical random seeds for each method. I reported the average and highest values across the ten runs to further eliminate the impact of randomness. The experiments show that, while PCP-MAE indeed offers some improvement over the original Point-MAE, it underperforms compared to the state-of-the-art method Recon in both average and highest values.

Additionally, the authors reported their results in Table 2 as 95.52%, 94.32%, and 90.35%, which differ significantly from the results I reproduced: 94.32%, 93.28%, and 89.94%, respectively. The discrepancies are 1.20%, 1.04%, and 0.41%. Although random seeds can significantly impact the results, a fair comparison using ten identical seeds is sufficient to illustrate the issue. Therefore, I believe the authors have overstated their results.

**Best result of ten experiments.**

| Methods  |  OBJ-BG |  OBI-ONLY | PB-T50-RS  |
| ------------ | ------------ | ------------ | ------------ |
| Point-MAE  | 93.12  | 92.77  | 89.04  |
| PCP-MAE  | 94.32  | 93.29  | 89.94  |
| Recon  |  95.19 | 93.12  | 90.25  |

**Average result of ten experiments.**

| Methods  |  OBJ-BG |  OBI-ONLY | PB-T50-RS  |
| ------------ | ------------ | ------------ | ------------ |
| Point-MAE  | 92.67  | 92.31  | 88.59  |
| PCP-MAE  | 93.99  | 92.62  | 89.34  |
| Recon  |  94.37 | 92.51  | 89.95  |
Thank you for your prompt reply and valuable feedback.

As per your suggestion, we have made the comments on the anonymous source code and checkpoints visible to you. You can now access this content at the following link:
https://openreview.net/forum?id=i1xjK5a0X8&noteId=PtuAJIdLVH

If you have any questions or need further clarification while reviewing, please don't hesitate to contact us at your earliest convenience.
I believe you could provide me with your anonymous checkpoints and source code. I hope to verify the effectiveness of your method through a detailed analysis of your code.
> Q3.1. The author's improvement is incremental and limited.

We have demostrated the effectiveness and superior performance of our method through ablations and experiemnts. Hope the rebuttal can help you better understand our method.

> Q3.2. The proposed method is only an improvement on Point-MAE, in fact, Point MAE is just one method in the field of point cloud self supervised learning.

We agree with you that Point-MAE is just one method in 3D SSL field, but it's of significant meaning in this field. It inspires many great works, improving it from different aspects including Point-M2AE, Point-FEMAE, ACT, Joint-MAE, ReCon and etc. Our method is proposed based on a motivating observation to improve the milestone method.  

We acknowledge that Point-MAE represents just one of many approaches within the field of 3D self-supervised learning. However, its contribution cannot be overstated. Point-MAE has served as a foundational model, sparking a wave of innovative research and developments aimed at enhancing its framework. This includes advancements through models like Point-M2AE, Point-FEMAE, ACT, Joint-MAE, and ReCon, among others. Our proposed method builds upon these insights, originating from a critical observation that seeks to further refine this pivotal technique.

> Q4. The generality of the proposed method. And our approach should be applicable to all MAE based point cloud self-supervised methods.

Thank you for proposing a good question to inspire us to explore the generality of our approach. You mensioned that our method should fits all MAE-based method. However, we design our method to improve Point-MAE but not implement an universal framework. Take Point-FEMAE as an example. It address the limited representation issue in the Point-MAE but that doesn't mean it fits all MAE-based method and can improve their performance undoubtedly.

Intuitively, our method fits all MAE-based method, but the concrete design should be carefully considered. Because different MAE-based method may optimize towards different directions, though within the same framework. Directly add PCM (Predicting Center Module) may hamper the consistency.

To explore the generality of our approach, **we incorporated PCM into MAE-based methods including Point-BERT and Point-FEMAE with additional minor changes to maintain architecture consistency.** The experiment results show that incorporating our center point predicting task enhances performance when other experiment setups are aligned: 

Method|OBJ-BG|OBJ-ONLY|PB-T50-RS
-|-|-|-|
Point-BERT|87.43|88.12|83.07
Point-BERT+PCP ($\eta=2.0$)|89.32|89.84|84.77|
Point-FEMAE $^\dagger$|94.32|92.94|89.38
Point-FEMAE+PCP ($\eta=0.1$) $^\dagger$|95.00|93.45|89.83|

$^\dagger$ Reproduced results adopting our augmentation $i.e.$, Scale&Translate+Rotation for pre-training.

Due to time constraints, exploration of various design choices and different $\eta$ was not feasible. We believe that further customization could lead to more substantial improvements. 

> Q5. Insufficient Visualizations and Explanations.

Thank you for your valuable advice. We have added the stop gradient to Fig. 2 according to your suggestion. Please check the polished figure in the attached rebuttal PDF. If you have any further concerns, please contact us.
To address the misunderstandings about the implementation and effectiveness of our approach on the part of Reviewer GmbZ, we provide the anonymous link to our PCP-MAE code to the ACs for clarification. The link is: https://anonymous.4open.science/r/2128-PCP-MAE-529D
Reference

[1] Dong R, Qi Z, Zhang L, et al. Autoencoders as cross-modal teachers: Can pretrained 2d image transformers help 3d representation learning?[J]. arXiv preprint arXiv:2212.08320, 2022.

[2] Armeni I, Sener O, Zamir A R, et al. 3d semantic parsing of large-scale indoor spaces[C], CVPR 2016.

[3] Dai A, Chang A X, Savva M, et al. Scannet: Richly-annotated 3d reconstructions of indoor scenes[C], CVPR 2017.

[4] Yu X, Tang L, Rao Y, et al. Point-bert: Pre-training 3d point cloud transformers with masked point modeling[C], CVPR 2022.

[5] Zha Y, Ji H, Li J, et al. Towards compact 3d representations via point feature enhancement masked autoencoders[C], AAAI 2024.
Note that the pre-training time cost contradicts the statistics in our paper's Table 1, which is reported from Point-FEMAE. It seems Point-FEMAE provides incorrect time efficiency comparison results, which we correct here and will update in the next version of our main paper.

To ensure a fair time comparison, the code for Point-MAE should be modified slightly in two ways:

1. Add "config.dataset.train.others.whole = True" to the training to align Point-FEMAE and our method.
2. Instead of using KNN_CUDA, change it into the knn_point function (refer to the official code of ReCon [4] or Point-FEMAE [5]) which directly uses torch operation to align with Point-FEMAE and our approach. This will significantly increase the training speed.

Reference

[1] Dai A, Chang A X, Savva M, et al. Scannet: Richly-annotated 3d reconstructions of indoor scenes[C], CVPR 2017.

[2] Caesar H, Bankiti V, Lang A H, et al. nuscenes: A multimodal dataset for autonomous driving[C], CVPR 2020.

[3] Mao J, Niu M, Jiang C, et al. One million scenes for autonomous driving: Once dataset[J]. arXiv preprint arXiv:2106.11037, 2021.

[4] Qi Z, Dong R, Fan G, et al. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining[C], ICML 2023.

[5] Zha Y, Ji H, Li J, et al. Towards compact 3d representations via point feature enhancement masked autoencoders[C], AAAI 2024.

[6] Uy M A, Pham Q H, Hua B S, et al. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data[C], ICCV 2019.

[7] Hua B S, Pham Q H, Nguyen D T, et al. Scenenn: A scene meshes dataset with annotations[C]//2016 fourth international conference on 3D vision (3DV). Ieee, 2016: 92-101.