Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper uses MAE to learn the point cloud features without fixed topology. Its multi-scale architecture can capture features at different resolutions for guidance. The pyramid architecture also facilitates the network to extract high-level features, and the self-attention mechanism adopted by this network also controls the focus of the network. The complementary skip connections also better connect the global and local features to guide each other. And it also produces competitive results in experiments.
strengths_and_weaknesses: - Originality: The main idea of the proposed approach is to use the masked autoencoders. To my limited knowledge about it, this is novel and thus the originality is reasonable.
- Quality：While the approach seems reasonable and the experimental results look promising, I have the following concerns(See Questions) about the paper.
- Clarity：This manuscript is clearly written.
- Significance：There are not many works in the field of point cloud feature learning by masked autoencoders, so I think this paper makes a positive contribution.
questions: 1) There are some works in the field that apply masked autoencoders in point clouds except Point-BERT, but I find that the authors miss these works[1][2][3][4], and I want to see more discussion from the authors about some of these out-picked works, and what this paper amounts to in terms of improvements to these precursor works.
2) I hope that the authors can add visual interpretation of the results of the Local Spatial Self-Attention module in their ablation experiments.
3) The authors keeps stressing that their model can pay more attention to fine-grained information, but there are no more descriptions about fine-grained information except \textbf{Figure 2} in \textbf{Appendix C} (which only shows the point cloud images with different scales).
4) Although the authors have achieved good results in the experiments after freezing the encoder for the relevant downstream tasks, have you analyzed the influence of the CD loss you selected for the training of the model's results? Now it seems that the decoder is driven only by CD Loss. Whether to use EMD as loss in the experiments or combine the two as compound loss? Will the results still be as good as now? I'd like to see the quantitative results if possible.
5) Miss the figure tag in \textbf{Appendix C. Line 68}: “As shown in Figure []”.

In the current version, there are some issues as well. I look forward to the response by the authors. For now, I would recommend a borderline accept rating for the paper.

[1]Fu, Kexue, et al. "POS-BERT: Point Cloud One-Stage BERT Pre-Training." arXiv preprint arXiv:2204.00989 (2022).

[2]Liu, Haotian, Mu Cai, and Yong Jae Lee. "Masked Discrimination for Self-Supervised Learning on Point Clouds." arXiv preprint arXiv:2203.11183 (2022).

[3]Pang, Yatian, et al. "Masked autoencoders for point cloud self-supervised learning." arXiv preprint arXiv:2203.06604 (2022).

[4]Pellis, Eugenio, et al. "An Image-Based Deep Learning Workflow for 3D Heritage Point Cloud Semantic Segmentation." International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences-ISPRS Archives 46.2/W1-2022 (2022): 429-434.
limitations: Yes, I don't see any potential negative social impact of this work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposes a multi-scale masked autoencoder for pre-training on point clouds. It designs a pyramid architecture for hierarchical encoder and decoder to progressively model 3D point clouds. A novel multi-scale masking and a back-project strategy are proposed for generating consistent visible regions across scales for point clouds. Extensive experiments demonstrate that the proposed method outperforms other pre-training methods on the linear SVM classification task and downstream tasks.
strengths_and_weaknesses: Strengths:
1. This paper is generally technically sound.
2. The design of the masking and back-project strategy is novel and fits well on the 3D point clouds pretraining task.
3. The extensive experiments verify the effectiveness of the proposed approach.

Weakness:
1. The overall framework seems to be simply a variant of PoinNet++ with Transformer Encoder, especially the idea of the Skip Connection and Point Token Upsampling, hence the framework seems to have insufficient technical innovations.
2. It can be noted that the U-Net like network demonstrated in Figures 2 and 3 have arbitrary multiple scales (at least greater than three). However, the actual experiments have only 2-3 scales and do not give an adequate explanation for the choice of the number of scales.

 


questions: 1. Why does the final experiment use only 2-3 scales and what about more? Please explain further the motivation for the choice of the number of scales.
2. In Sec 3.2 ‘Token Embedding and Merging’, “After that, we obtain the initial point tokens T_1^v for the 1-st stage of the encoder” seems to be conflict with the Figure 2 where the T_1^v is the output of the 1-st stage of the encoder.
3. In Sec 4.2 ‘Shape Classification’, “We follow Point-BERT to use the voting strategy [29] for fair comparison on ModelNet40…”, as the Point-BERT does not seem to mention that it uses a voting strategy and dose not cite the paper [29]. Please explain further the voting strategy.


limitations: NA
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper extends the MAE to deal with irregular 3d point clouds. It proposes a multi-scale u-net like encoder-decoder to learn both fine-grained local shape information and global semantic information. The network is self-supervised trained by masking the input point cloud with multi-scale mask strategy. Experiments are conducted on ModelNet40 with the learned representation combined with SVM, achieving good classification results which is comparable to the SOTA supervised methods. The effectiveness of the proposed method is also validated by fine-tuning on the down-stream tasks such as classification, part segmentation and 3d object detection, and observed improvements over existing methods.
strengths_and_weaknesses: Positives
+: The paper studies an important problem in deep learning and computer vision community. The proposed method is a reasonable extension of MAE to deal with point clouds.
+: Good results were obtained by the proposed method on different tasks.
+: Detailed ablation study to validate the effectiveness of each component.

Negatives
-: It would be better if the authors can also provide the results of using the frozen encoder for other tasks like part segmentation and 3d object detection, so as to show that the conclusion about the learned representation is general enough.
-: As the advantage of self-supervised learning is to harvest from a large amount of unlabeled data, the paper lacks results about using more dataset for pre-training, for instance, using all three kinds of datasets (ShapeNet, ModelNet, ScanNet) together. This is not only helpful for readers to know the potential of the proposed method, but also can be interested to see if the proposed method can deal with different kinds of data together to boost the performance.
-: The influence of different amount of pre-training examples should also be included in the ablation study.
-: For the paper writing, it is unclear about the difficulties of introducing the MAE into 3d point cloud. It seems that there are MAE methods for 2D images, then why not we use it for 3d point, and so, the paper makes it done. Therefore, the contribution of this paper is not significant.

questions: please see the above negative points.
limitations: yes
review_title: null

Meta-review: metareview: This paper proposes, the Point-M2AE, a multi-scale masked autoencoder (MAE) pre-training framework for self-supervised learning of 3D point clouds. This is a generalization of the existing 2D-MAE framework to 3D point cloud domain. The proposed Point-M2AE introduces a U-Net-like transformer and a multi-scale masking strategy to generate consistent visible regions across scales. Extensive experiments are conducted on various downstream tasks to validate the power of the proposed method.  All reviewers think that the current paper presents a novel framework, making an important contribution to point cloud representation. It also has sufficient empirical results to demonstrate the performance of the proposed model. The feedback from the authors also addresses the major concerns of the reviewers. After reading the reviewers’ comments and the authors’ replies, the AC recommends accepting the paper.
confidence: Certain
Decision: Accept