Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training
Authors: Renrui Zhang
Ziyu Guo
Peng Gao
Rongyao Fang
Bin Zhao
Dong Wang
Yu Qiao
Hongsheng Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: Masked autoencoders
self-supervised learning
3D point cloud pre-training

Abstract: Masked Autoencoders (MAE) have shown great potentials in self-supervised pre-training for language and 2D image transformers. However, it still remains an open question on how to exploit masked autoencoding for learning 3D representations of irregular point clouds. In this paper, we propose Point-M2AE, a strong Multi-scale MAE pre-training framework for hierarchical self-supervised learning of 3D point clouds. Unlike the standard transformer in MAE, we modify the encoder and decoder into pyramid architectures to progressively model spatial geometries and capture both fine-grained and high-level semantics of 3D shapes. For the encoder that downsamples point tokens by stages, we design a multi-scale masking strategy to generate consistent visible regions across scales, and adopt a local spatial self-attention mechanism during fine-tuning to focus on neighboring patterns. By multi-scale token propagation, the lightweight decoder gradually upsamples point tokens with complementary skip connections from the encoder, which further promotes the reconstruction from a global-to-local perspective. Extensive experiments demonstrate the state-of-the-art performance of Point-M2AE for 3D representation learning. With a frozen encoder after pre-training, Point-M2AE achieves 92.9% accuracy for linear SVM on ModelNet40, even surpassing some fully trained methods. By fine-tuning on downstream tasks, Point-M2AE achieves 86.43% accuracy on ScanObjectNN, +3.36% to the second-best, and largely benefits the few-shot classification, part segmentation and 3D object detection with the hierarchical pre-training scheme. Code is available at https://github.com/ZrrSkywalker/Point-M2AE.

Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training
Renrui Zhang!??, Ziyu Guo", Rongyao Fang', Bin Zhao?, Dong Wang", Yu Qiao', Hongsheng Li Peng Gao
(UHK-SSnseTTing Joint Laboratory, The Chinese University of Hong Kong. Shanghai AI Laboratory. Centre for Perceptual and Interactive Intelligence Limited
(zhangrenrui, gaopeng @pjlab. .erb.rr..coo. hslißee. cuhk edu.hk
Abstract
Masked Autoencoders (MAE) have shown great potentials in self-supervised pre- training for language and 2D image transformers. However, still remains an open question how exploit masked autoencoding for learning 3D repre- sentations of irregular point clouds. In this paper, we propose Point-MZAE. a strong Multi-scale MAE pre-training framework for hierarchical self-supervised learning of 3D point clouds. Unlike the standard transformer in MAE, we mod- ify the encoder and decoder into pyramid architectures spatial geometries and capture both fine fine-grained and high-level semantics of 3D shapes. For the encoder that downsamples point tokens by stages, we design a multi-scale masking strategy to generate consistent visible regions across scales, and adopt a local spatial self-attention mechanism during fine-tuning to focus neighboring patterns. By multi-scale token propagation, the lightweight de- coder gradually upsamples point tokens with complementary skip connections from the encoder, which further promotes the reconstruction from global-to-local perspective. Extensive experiments demonstrate the state-of-the toee------eerrr-re the-art performance of Point-M2AE for 3D representation learning. With frozen encoder after pre- training, Point-M2AE achieves 92.9% accuracy for linear SVM on ModelNet40, even surpassing some fully trained methods. By fine-tuning downstream tasks, Point-M2AE achieves 86.43% accuracy on ScanObjectNN, +3.36 % to the second- best, and largely benefits the few-shot classification, part segmentation and 3D object detection with the hierarchical pre-training scheme. Code is available at https: / / thub. com www./rrsskyeeee wweeeerrieie M2-2E...........
progressively model
Introduction
Learning represent from unlabeled data without annotations, known self-supervised learning. has attained great success in natural language processing [10, 32, 33, 5], computer vision [19, 7, 8, 18] and multi-moduan modii-mod learning [31, 50, 21]. By pre-training on the large-scat raw data, the networks are endowed with robust representation abilities and can significantly benefit downstream tasks with tnn-tuning. Motivated by masked language modeling [32, 10], MAE [18] and some other methods [46, 53, 3] adopt asymmetric encoder-decoder transformers [13] to apply masked autoencoding for self-supervised learning on 2D images. They represent the input image as multiple local patches, and randomly mask them with high ratio to build the pretext task for reconstruction. Specifically, the encoder aims capturing high-level latent representations from limited visible patches, and the lightweight decoder is forced to reconstruct the RGB values of masked patches on
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

MAE
Plain Decoder
2D Image
Masking
Reconstructed Pixels
Point-M2AF
Skip Connections
Hierarchical Ancoder
Ilierachical Decoder
3D Point Cloud
Multi-scale Masking
Downsorngle
Uppsmple
Reconstructed 30 Coordinates
Figure 1: Comparison of MAE (Top) and our Point-M2AE (Bottom). MAE for 2D image pre- training adopts standard transformer of the plain encoder and decoder, while Point-M2AE introduces hierarchical transformer with skip connections for multi-scale point cloud pre-training.
top. Despite its superiority grid-based 2D images, ask the question: MAE-style masked autoencoding be adapted to irregular point clouds as a powerful 3D representation learner?
To tackle this challenge, we propose Multi-scale Masked autoencoders for learning the hierarchical representations of point clouds via self-supervised pre-training, termed as Point-M2AE. We represent point cloud as set of point tokens depicting different spatial local regions, and inherit MAE's pipeline to first encode visible point tokens and then reconstruct the masked 3D coordinates. Different from 2D images, masked autoencoding for 3D point clouds has three characteristies to be specially considered. Firstly, it is critical to understand the relations between local parts and the overall 3D shapes, which have strong geometric and semantic dependence. As examples, the network can recognize an airplane starting from its wing, or segment the wing's 's part from the airplane's 's global feature. Therefore, regard the standard transformer with the plain encoder and decoder is sub-optimal for capturing such local-global spatial relations in 3D, which directly downsamples the input into low-resolution representation as shown in Figure 1 (Top). We modify both the encoder and decoder into multi-stage hierarchies for progressively encoding multi-scale features of point clouds, constructing an asymmetric U-Net [35] like architecture in Figure 1 (Bottom). Secondly, as our Point-M2AE encodes multi-scale point clouds unlike the single-scale 2D images, the unmasked visible regions are required be both block-wise within one scale and consistent across scales, which are respectively for reserving complete local geometries and ensuring coherent feature learning for the network. For this, we introduce multi-scale masking strategy, which generates random masks at the final scale with high ratio (e.g., 80%), and back pook-proeects the unmasked positions to all preceding scales. Thirdly, to better reconstruct 3D geometries from local-to- rooglobal perspective, we utilize skip connections complement the decoder with fine-grained information from the corresponding stages of the encoder. During fine-tuning on downstream tasks, we also adopt local spatial self-attention mechanism with increasing attention scopes for point tokens at different stages of the encoder, which refocus each token within neighboring detailed structures.
By the multi-scale pre-training, Point-M2AE can encode point clouds from local-to-global hier- archies and then reconstructs the masked coordinates from global-to-local perspectives, which learns powerful 3D representations and performs superior transfer ability. After self- pre-training on ShapeNet [6], Point-M2AE achieves 92.9% classification accuracy for linear SVM on ModelNet40 [44] with the frozen encoder, which surpasses the runner-up CrossPoint [2] by +1 +1.2% and even outperforms some fully supervised methods. By fine-tuning on various downstream tasks, Point-M2AE, achieves 86.43% (+3.36%) accuracy on ScanObjectNN [38] and 94.0% (+0.8%) accuracy on ModelNer40 [44] for shape classification, 86.51 51% (+0.91 %) instance mloU on ShapeNet- Part [48] for part segmentation, and 95.0% (+2.7%) accuracy 10-way 20-shot ModelNet40 for

---

few-shot classification. Our multi-scale masked autoencoding also benefits the 3D object detection on ScanNetV2 [9] by +1.3% 25 and +1.3% APs0. which provides the detection backbone with a hierarchical understanding of the point clouds.
We summarize the contributions of paper follows:
1. We propose Point-M2AE, strong masked autoencoding framework, which conducts hi- erarchical point cloud encoding and reconstruction for better learning multi-scale spatial geometries of 3D shapes.
2. We introduce U-Net like transformer architecture for MAE-style pre-training on point clouds, and adopt multi-scale masking strategy to generate consistent visible regions across scales.
3. Point-M2AE achieves state-of-the-ar performance for transfer learning on various down- stream tasks, which indicates our approach to be powerful representation learner for 3D point clouds.
2 Related Work
Pre-training by Masked Modeling. Compared to contrastive learning methods [19, 7, 8] that learn from inter-sample relations, self-supervised pre-training by masked autoencoding builds the pretext
tasks to predict the masked parts of the input signals. The series of GPT [32, 33, 5] and BERT [11] apply masked modeling to natural language processing and achieve extraordinary performance boost on downstream tasks with fine-tuning. Inspired by this, BETT [4] proposes to match image patches with discrete tokens via dVAE [34] and pre-train a standard vision transformer [13, 49] by masked image modeling. On top of that, MAE [18] directly reconstructs the raw pixel values of masked tokens and performs great efficiency with a high mask ratio. The follow-up works further improve the performance of MAE by momentum encoder [53], contrastive learning [3], and modified reconstruction targets [42]. For self-supervised pre- pre-training on 3D point clouds, the masked autoencoding has not been widely adopted. Similar to BEÎT, POiT [49] utilizes dVAE to map 3D patches to tokens for masked point modeling, but heavily relies on constrastive learning [19], complicated data augmentation, and the costly two-stage pre-training. In contrast, our Point-M2AE is a pure masked autoencoding method of one-stage pre-training, and follows MAE to reconstruct the input signals without dVVE mapping. Different from previous MAE methods adopting standard plain transformer, we propose hierarchical transformer architecture along with the multi-scale masking strategy better learn a strong and generic representation for 3D point clouds.
Self-supervised Learning for Point Clouds. 3DD representation learning without annotations has
been widely studied in recent years. Mainstream methods mainly build the pretext tasks to reconstruct the transformed input point cloud based on the encoded latent vectors, such as rotation [28], defor- mation [1], rearranged parts [36] and occlusion [40]. From another perspective, PointContrast [45] utilizes contrastive learning between features of the same points from different views learn discrimi- native 3D representations. DepthContrast [5] further extends the contrast for depth maps of different augmentations. CrossPoint [2] conducts cross- modalltyy contrastive learning between point clouds and their corresponding rendering images to acquire rich self- supervieed signals. Point-BERG [49] and Point-MAE [27] respectively introduce BERT-style [10] and MAE-style [18] pre-training schemes for 3D point clouds with standard transformer networks and performs competitively on various downstream tasks, but both of them can only encode point clouds with single resolution and ignores the local-global relations between 3D shapes. In this paper, propose Point-M2AE, an MAE-style framework with hierarchical transformer for multi-scale point cloud pre-training. We achieve state-of-the-art downstream performance by learning the multi-scale representation of point clouds.
Method
The overall pipeline of Point-M2AE is shown in Figure 2, where we encode and reconstruct the point cloud by a hierarchical network architecture. In Section 3.1, We first introduce the masking strategy of Point-M2AE with multi-scale representations of point clouds. Then Section 3.2 and Section 3.3, present the details of our encoder and decoder with multi-stage hierarchies.
3

---

TY
Token Ensbboding
1st Encoder
Token Merging
2-nd Stage iinodder
Token Marging
S Stage incoder
Multi-Scale Masking
Skip Connection
Concatrrate
Vistable
Maslind Tokens
Tolen
1 st
Stage
Propon
Decoder
Reconstructed Pa-1
NE
Figure 2: Overall pipeline of Point-M2AE. After the multi-scale masking, we embed point tokens the 1-st scale and feed the visible ones into hierarchical encoder-decoder transformer, which captures both high-level semantics and fine-grained patterns of the point cloud during pre-training.
3.1 Multi-scale Masking
To build U-Net [35] like masked autoencoder for hicrarchical learning, we encode the point cloud by S scales with different number points each scale, and correspondingly modify the standard plain encoder into the S-stage architecture. Following MAE, we embed the point cloud into discrete point tokens and randomly mask them for reconstruction. Importantly, for trregula-ddisrributes points in the multi-scale architecture, the unmasked visible spatial regions are required to be consistent not only within one scale, but also across different scales. This is because the block-wise parts of 3D shapes tend to preserve more complete fine-grained geometries, and the unmasked positions are better to be shared across all scales for coherent feature learning of the encoder. Therefore, as shown in Figure 3, we first construct the S-scale coordinate representations of the input point cloud and back-project the random masks from the final S-th scale to the earlier scales to avoid fragmented visible parts.
denote the input point cloud as P E R Xx and regard it as the 0-th scale. For the i-th scale, 1 < S S, we utilize Furthest Point Sampling (FPS) to downsample the points from the (i - 1)-th scale, which produces seed points P; E R.N.33 for scale of N, points. Then, adopt point and obtain the neighbor indices 4 E R N;
S-scale Representations.
Neighbour (k-NN) to aggregate the neighboring k points for each seed xk By successively downsampling and grouping, we acquire the S-scale representations 1, |= of the imput point cloud, where the number of points N; gradually decreases and the inclusion relations between scales are recorded in Tç.
Back- projeprojg Visible Positions. For seed points Ps at the final S-th scale, we randomly mask them with large proportion (e.g., 80%) and denote the remaining visible points P5 E IR N of Ns points. We then back-project the unmasked positions P3 to ensure the consistent visible regions across scales. For the i-th scale, < i S, we retrieve all the l nearest neighbors of P from the indices T+41 to serve as the visible positions P." and mask the others. By recursively back proje-rojecting, we obtain the visible and masked positions of all S scales, denoted as {P.". pm P..... 15 Ji=1+
where PN E N x3 pm E N. x3 and N; = N + -N.
Hierarchical Encoder
Based on the multi-scale masking, we embed the initial tokens of visible points P for the 1-st scale and them into the hierarchical encoder with stages. Every stage is equipped with stacked encoder blocks, and each block contains a self-attention layer and Feed Forward Network (FFN) of MIPP layers. Between every two consecutive stages, we introduce spatial token merging modules to aggregate adjacent visible tokens and enlarge receptive fields for downsampling the point clouds.

---

Input Point Cloud P
1-st Scale (P4. 43
2-nd Scale (P2.12) 12)
S-th Scale (Ps. 153
FPS, R-NN
FPS, N-NN
PPS, R-NN
Mask the S-th Scale
Back-projest
Bade-prajott is
Figure 3: Multi-scale masking strategy. To obtain a consistent visible regions across scales, we first represent the input point cloud by multi-scale coordinates and generate the random mask the highest one. Then, we back back-project the unmasked visible positions to all earlier scales.
Token Embedding and Merging. Indexed by J1 we utilize mini-PointNet [29] to extract and fuse the features of every seed point from PT E IR NY x3 with its k nearest neighbors. After that, we obtain the initial point tokens TV E R for the 1-st stage of the encoder, which embeds Nf local patterns of the 3D shape. Between the (i 1)-th and i-th stages, Th E NR to acquire the downsampled point tokens for the i-th stage. We utilize MLP layers and max pooling to integrate every k tokens nearest to PT indexed by 4, which outputs Ty RN,,CC Due to our multi-scale masking, the merged Tи² corresponds to the same visible parts of Ty- which enables the consistent feature encoding across different scales. For larger i of deeper stages, set higher feature dimension C, to encode spatial geometries with richer semantics.
S S, merge
Local Spatial Self-Attention. During pre-training, we expect point tokens in the multi-stage encoder to capture global cues for 3D shapes, which benefits the reconstruction of masked parts. However, when fine-tuning on downstream tasks without masked autoencoding, point tokens the shallower stages better mainly focus local information and not be disturbed by long-ran ong-range signals, referring to the inductive bias of 3D locality [30]. Thus, during fine-tuming we modify the original self-attention layer in the encoder with local spatial constraint that only neighboring tokens within a ball query would be available for attention calculation. As the point tokens are downsampled by stages, set increasing radii (r:) of multi-scale ball queries for gradually expanding the attention scopes, which fulfills the local-to-global feature aggregation scheme.
3.3 Hierarchical Decoder
Via the hierarchical encoder, we obtain the encoded visible tokens {2T' /i=1 of all scales. Starting from the highest S-th scale, we assign shared learnable mask token all the masked positions Ps and concatenate them with the visible tokens 77. We denote them as {H," H!T } with coordinates which serve as the input of the hierarchical decoder. We design the decoder to be lightweight with S - 1 stages and only one decoder block for each stage. which enforces the encoder embed more semantics of the point clouds. Each decoder block consists of vanilla self-attention layer and an FFN. We do not apply the local constraint to the attention in the decoder, since a global understanding between visible and mask tokens crucial the reconstruction.
Point Token Upsampling. We upsample the point tokens between stages to progressively recover the fine-gr fine-grained geometries of 3D shapes before reconstruction. We regulate that the j-th stage of -))th stage of the encoder, both of which contain point tokens of the same (S + 1 --') j'h scale with the feature dimension Cs+1-j- Between the (j 1)- th and j-th stage, < < S 1, we upsample the tokens {H H from the coordinates into { [P P via the token propagation module. Specifically, we obtain the & nearest neighbors of each point token in (Hý-1, H indexed by Is+2-j. and recover their neighbors' features by weighted interpolation referring PointNet++ [30], which generates the tokens (H]. HHm) of the j-th stage.
the decoder corresponds to the (S
{P5 (P5+2-3: P (++--)
5

---

Table 1: Linear evaluation on Model- Net40 [44] by SVM. We report different self-supervised learning methods and un- derline the second-best one.
Table 2: Shape classification on ModelNet40 [44].
##ooins' and 'Ace'' denote the number points for training and the overall aceuracy. [S] represents fine- tuning after self-suuervised pre-training.
Method
Acc. (%)
Method
#points
Acc. (%)
3D-GAN [43]
83.3
PointNet [29]
lk
89.2
Latent-GAN [39]
85.7
PointNet++ [30]
lk
90.5
SO-Net [22]
87.3
PointCNN [23]
1k
92.2
FoldingNet [47]
88.4
[S] SO-Net [22]
5k
92.5
MAP-VAE [17]
88.4
DGCNN [41]
1k
92.9
VIP-GAN [16]
90.2
PCT [15]
1k
93.2
Point Transformer [52]
93.7
DGCNN Jiasaw [37]
90.6
DGCNN OcCo [40]
90.7
Transformer [49]
1k
91.4
DGCNN CrossPoint [2]
91.2
[S] Transformer + OcCo [49]
1k
92.1
Transformer + OcCo [49]
89.6
[S] Point-BERT [49]
1k
93.2
Point-BERT [49]
87.4
[S] Point-BERT
4k
93.4
Point-M2AE
92.9
[S] Point-BERT
8k
93.8
Improvement
+1.7
[S] Point-M2AE
1k
94.0
Skip Connections. To further complement the fine-gratn fine-grained geometries, channel-wisely con- catenate the visible tokens HỤ E RBNS=1-,xCs+11p of the decoder with 55+1-5 E R [NS+1-1x08+11
from the corresponding (S+11 1+1 j)-th stage of the encoder via skip connections, and adopt linear projection layer fuse their features. For the mask tokens H we keep them unchanged, since the encoder only contains visible tokens without the masked ones.
Point Reconstruction. After S --1 stages of the decoder, we acquire 1, H5 with co- ordinates P and reconstruct the masked values from the mask tokens H5 Other than predicting values at the O-th scale of the input point cloud P. we reconstruct the coordinates of PT", namely, recovering the masked positions of the 1-st scale P E R.T x3 from the 2-nd scale } of the 1-st scale could well represent the overall 3D
PT E N x3 This is because { P shape and simultancously preserve enough local patterns, which already constructs comparatively challenging pretext task for pre- vre-training. and reconstruct the masked raw points from Pm the extra spatial noises and computational over- head would adversely influence our performance and efficiency. Therefore, for every token in H E N xC2, we reconstruct its & nearest neighbors recorded in by reconstruction head of one linear projection layer and compute the loss by l2 Chamfer Distance [14], formulated as,
we further upsample { 1. into {Hf H
Pm Linear(H where P. 5m CCD ChamferDistance(PP PM,1),
No x
(2)
where Pm and P denote the predicted and ground-truth reconstruction coordinates from the 2-nd scale to the 1-st scale. We only utilize LeD for supervision without contrastive loss to conduct a pure masked autoencoding for self-supervised pre-training.
Experiments
In Section 4.1 and Section 4.2, we introduce the pre-training experiments of Point-M2AE and report the fine-tuning performance on various downstream tasks. We also conduct ablation studies in Section 4.3 to validate the effectiveness of our approach.
4.1 Self-supervised Pre-training
Settings. pre-train our Point-M2AE, on ShapeNet [6] dataset, which contains 57,448 synthetic 3D shapes of 55 categories. We set the stage number S as and construct 3-stage encoder and

---

Table 3: Shape classification on ScanOhjectNN [38]. We report the accuracy (%) on the three splits of ScanObjectNN, [S] represents fine-tuning after self-supervised pre-training.
Method
OBJ-BG
OBJ-ONLY
PB-T50-RS
PointNet [29]
73.3
79.2
68.0
PointNet++ [30]
82.3
84.3
77.9
DGCNN [41]
82.8
86.2
78.1
PointCNN [23]
86.1
85.5
78.5
Transformer [49]
79.86
80.55
77.24
[S] Transformer + OcCo [49]
84.85
85.54
78.79
[S] Point-BERE [49]
87.43
88.12
83.07
[S] Point-M2AE
91.22
88.81
86.43
Improvement
+3.79
+0.69
+3.36
2-stage decoder for hierarchical learning. We adopt 5 blocks in each encoder stage, but only 1 block per stage for the lightweight decoder. For the 3-scale point clouds, set the point numbers and token dimensions respectively as {512, 256, 64) and (96, 192, 384]. We also set different k for the k-NN at different scales, which are {16, 8, 8). We mask the highest scale of point clouds with high ratio of 80% and set 6 heads for all the attention modules. The detailed training settings are in Appendix.
Linear SVM. After pre-training on ShapeNet, we the 3D representation capability of Point- M2AE via linear evaluation on ModelNet40 [44]. We sample 1,024 points from each 3D shape of ModelNet40 and utilize our frozen encoder extract their features. On top of that, we train linear SVM and report the classification accuracy in Table 1. As shown, Point-M2AE achieves the best performance among all existing self-supervised methods for point clouds, and surpasses the second-best CrossPoint [2] by +1.7%. Point-M2AE also exceeds Point-BERT [49] by +5.5%, which is a masked point modeling method with a MoCo loss [19] but adopts a standard transformer and conducts single-scale learning. It is worth noting that even if we freeze all our parameters, Point-M2AE with 92.9% accuracy still outperforms many fully trained methods on ModelNet40, e.g.. 90.5% by PointNet++ [30], 92.8% by DensePoint [24], etc. The experiments fully demonstrate the superior 3D representation capacity our Point-M2AE.
4.2 Downstream Tasks
For fine-tuning on downstream tasks, we discard the hierarchical decoder in pre-training and append different heads onto the hierarchical encoder for different tasks.
Shape Classification. We fine-tune Point-M2AE on two shape classification datasets: the widely adopted ModelNet40 [44] and the challenging ScanObjectNN [38]. For local spatial attention layers. we set the ball queries' radii of 3-scale point clouds as [0.32, 0.64, 1.28). We follow PoiRT-BERT to use the voting strategy [25] for fair comparison on ModelNet40. To handle the noisy spatial structures, we increase & of k--N into {32, 16, 16) for ScanObjectNN to encode local patterns with larger receptive fields. As reported in Table 2, Point-M2AE achieves 94.0% accuracy on ModelNet40 with 1024 points per sample, which surpasses Point-BERT fine-tuned with 1024 points by +0.8% and 8192 points by +0.2%. For ScanObjectNN Table our Point-M2AB outperforms the second- best Point-BERT by significant margin, +3.79% +0.69% and +3.36%, respectively for the three splits, indicating our great advantages under complex circumstances by multi-scale encoding. As ScanObjectNN of real-world scenes has large semantic gap with the pre-trained synthetic ShapeNet, Point-M2AE also exerts strong transfer ability to understand point clouds of another domain.
Part Segmentation. We evaluate Point-M2AE for part segmentation on ShapeNetPart [48], which predicts per-point part labels and requires detailed understanding for local patterns. We adopt an extremely simple segmentation head validate the effectiveness of our pre-training for well capturing both high-level semantics and fine- fine-grained details. By the hicrarchical encoder, obtain
7

---

Table 4: Few-shot classification ModelNet40 [44]. We report the average accuracy (%) and standard deviation (%) of 10 independent experiments.
5 5-way
10-way
Method
10-shot
20-shot
10-shot
20-shot
DGCNN [41]
91.8 + 3.7
93.4 + 3.2
86.3 + 6.2
90,9 + 5,1
[S] DGCNN OcCo [40]
91.9 3.3
93.9 3.1
86.4 : 5.4
91.3 + 4.6
Transformer [49]
87,8 + 5.2
93.3 + 4.3
89.4 5
89.4 + 6.3
[S] Transformer + OcCo [49]
94.0 + 3.6
95.9 + 2.3
92.4 4.6
[S] Point-BERT [49]
94.6 + 3.1
96.3 + 2.7
91.0
92.7 5.1
[S] Point-M2AE
96.8 = 1.8
98.3 + 1.4
92.3 1 4.5
95.0 + 3.0
Improvement
+2.2
+2.0
+1.3
+2.3
Table 5: Part segmentation on ShapeNetPart [48]. 'mloUc' (%) and 'mloU," (%) denote the mean IoU across all part categories and all instances in the dataset, respectively.
Table 6: 3D object detection on Scan- NetV2 [9]. We report the performance (%) of self-supervised learning methods based on VoteNet [12] and 3DETR-m [26].
Method
mloUc
mloUr
Method
25
AP50
PointNet [29]
80.39
83.70
VoteNet [12]
33.5
PointNet++ [30]
81.85
85.10
[S] STRL [20]
59.5
38.4
DGCNN [41]
82.33
85.20
[S] PointContrast [45]
59.2
38.0
[S] DepthContrast [51]
61.3
Transformer [49]
83.42
85.10
[S] Transformer + OcCo [49]
83.42
85.10
3DETR [26]
62.1
37,9
[S] PointBERTT [49]
84.11
85.60
3DETR-m [26]
65.0
47.0
[S] Point-M2AE
84.86
86.51
[S] Point-M2AF
66.3
48.3
Improvement
+0.75
+0.91
Improvement
+1.3
+1.3
3-scale point tokens of (512, 256, 64) points, and perform feature propagation in PointNet++ [30] to independently upsample the tokens into 2048 points of the input point cloud. Then, we concatenate the upsampled 3-scale features for each point and predict the part label by stacked linear projection layers. As reported in Table 4.2, Point-M2AE achieves the best 86.51 511 % instance mIoU with the simple segmentation head, surpassing the second-best Point-BERT by +0.91% %. Note that Point-BERT [49] and other methods [29, 30, 41] adopt hierarchical segmentation heads progressively upsample the point features from intermediate layers, while our head contains no hierarchical structure and only relies on the pre pre-trained encoder to capture the multi-scu scale information of point clouds. The results fully demonstrate the significance of Point-M2AE's multi-scale pre-training to segmentation tasks.
Few-shot Classification. We conduct experiments for few-shot classification on ModelNet40 [44] evaluate the performance of Point-M2AB with limited fine-tuning data. As reported in Table 4.2, Point-M2AE achieves the best performance for all four settings, and surpasses Point-BERT by +2.2%, +2.0% +1.3% and +2.7%, respectively. Our approach also shows smaller deviations than other transformer-basee methods, which indicates Point-M2AE has learned to produce more universal 3D representations for well adapting to downstream tasks under low-data regimes.
3D Object Detection To further evaluate our hierarchical pre- pre-training on 3D object detection, we apply Point-M2AE to serving as the feature backbone on the indoor ScanNet [9] dataset. We select 3DETR-m [26] as our bascline, which consists of 3-block encoder and transformer decoder. Considering the quite different dataset statistics, e.g., 2k input points for Shape [6] and 50k input points for SeanNetV2, we adopt the same encoder architecture with that of 3DETR-1 -m, and keep our hierarchical decoder with skip connections unchanged for self-supervised pre-traine -rainiiiing on VcanNetV2. More details of models and training are in Appendix. As reported in Table 4.2, compared training from scratch, our hierarchical pre-training boosts the performance of 3DETR-m by +1. +1.34% AP25 and learn multi-scale
+1. 1..% APs0- The experiments demonstrate the effectiveness of Point-M2AF point cloud encoding for object detection and its potential to benefit wider range of 3D applications.

---

[NH] Features of Point Cloud
[H] Features of Point Cloud
Input Cloud
[NH] Constructed
[H] Constructed Point Cloud
Point Cloud
Figure 4: Visualization of fine-grained information. We denote the outputs from hierarchical and non-hierarchical architectures as [NH] and [H], respectively. For an input point cloud (Middle), we visualize its extracted features (Left) and reconstruction results (Right).
Table 7: Hierarchical Modules. represents the encoder and decoder with multi- stage hierarchies. "Skip C.' denotes the skip connections.
Table 8: Different Masking Strategy. 'MS Mask' and 'Ratio' denote the multi-scale mask- ing and the mask ratio.
Encoder
Decoder
Skip C.
Acc. (%)
MS Mask
Ratio
Acc. (%)
H
92.9
0.8
92.9
90.7
0.8
88.4
H
V
91.5
/
0.6
92.3
H
v
92.2
/
0.7
92.7
H
H
92.1
/ 0.9
92.5
4.3
Ablation Study
We conduct ablation study by modifying one of the components at time during pre-training and explore the best masking strategy. We report the classification accuracy ModelNet40 [44] by linear SVM to evaluate the pre-trained representations. For downstream tasks, we train the network from scratch validate the significance of our hierarchical pre-training.
Hierarchical Modules. As reported in Table 7, on top of our final solution, Point-M2AE, in the first row, we respectively experiment with removing the hierarchical encoder, hierarchical decoder, and skip connections from our framework. Specifically, we replace our encoder and decoder with 1 1-stage plain architectures similar to MAE, which contains 15 and vanilla transformer blocks, respectively. We observe the absence of multi-stage structures either in encoder or decoder hurts the performance, and the hicrarchical encoder plays a better role than the decoder. Also, the skip connections well benefits the accuracy by providing complementary information for the decoder.
Masking Strategy. In Table 8, we report Point-M2AE with different mask settings. Without the multi-scale masking, we randomly generate masks at each scale, which leads to fragmented visible regions for all scales. With this strategy, the network would 'peek' different parts of the point cloud different stages, which disturbs the representation learning and harms the performance by -4.5% accuracy. For different mask ratios, we find the 80% ratio performs the best to build properly challenging pretext task for self-supervised pre-training.
With and Without Pre-training. We report the performance of Point-M2AE on downstream tasks with and without the pre-training in Ta- ble 9. For 'w/o', we randomly initialize the parameters and train the network from seratch. As shown, the pre-training can largely boost performance on four datasets respectively by +1.5%, +2.5%, +3.8%, and +1.1%, which indicates the superiority and significance of our hierarchical pre-training.
Table 9: With and without the pre-training. 'ModelNet4O-FS'' denotes the few-shot classifica- tion on 10-way 20-shot ModelNet40 [44].
Dataset
w/o (%)
(%)
ModelNet40 [44]
92.5
94.0
ScanObjectNN [38]
83.9
86.4
ModeINet4O-FS [44]
91.2
95.0
ShapeNetPart [48]
85.4
86.5

---

Input
Scale
Z-nd Stale
3-rd Scale
Reconstructed
I-st Scale
2-nd Scale
3-rd Scale
Point Clouds
Representations
Point Clouds
Multi-Scale Masked Point Clouds
Figure 5: Visualization of multi-scale point clouds. each row, we visualize the input point clouds, their multi-scale representations, the reconstructed coordinates, and multi-scale masked point clouds.
5
Visualization
Multi-scale Masking. To ease the understanding of our multi-scale masking strategy, we visualize the input point cloud, the 3-scale representations, the reconstructed point cloud, and 3-scale masked point clouds, respectively in each row of Figure 5. As shown, different scales can represent different levels of geometric details and semanties for point clouds. By the multi-scale masking strategy, we observe the visible positions of masked point clouds are block-wise within one scale and consistent across scales, which is significant for our hierarchical pre-training.
Fine-grained Information. The fine-grained 3D structures, c.g., thin branches of a plant, fingers of human, engines of plane, are significant distinguish similar shapes and can be well encoded by our hierarchical representations. In Figure 4, we compare our Point-M2AE with multi-stage, [H], and single-scale [NH], architectures by visualizing their extracted point features and reconstructed point clouds during trainiiinng In contarst to the single-scale network, the multi-scale one indicates higher feature responses the fine-grat grained structures and reconstructs more accurate spatial details.
Conclusion
We propose Point-M2AF, a multi-scale masked autoencoder for self-supervised pre-training on 3D point clouds. With a hicrarchical architecture, Point-M2AE learns to produce powerful 3D representations by encoding multi-scale point clouds and reconstructing the masked coordinates from a global-to-local upsampling scheme. Extensive experiments have demonstrated the superiority of Point-M2AE to be strong 3D representation learner. For limitations and future work, will focus on applying Point-M2AE for wider 3D applications, e.g.. outdoor and open-world scene understanding. We do not foresee negative social impact from the proposed work.
Acknowledgement. This work is supported by the National Natural Science Foundation China (Grant No. 62206272). Shanghai Committee of Science and Technology (Grant No. 21DZI 100100), Centre for Perceptual and Interactive Intelligence Limited, and the General Research Fund through the Research Grants Council of Hong Kong (Grant No. 14204021, 14207319).
10

---

References
[1] Idan Achituve, Haggai Maron, and Gal Chechik. Self-supe learning for domain adaptation on point clouds. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 123-133, 2021. 3
[2] Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Rodrigo. Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding. arXiv preprini arXiv:2203. 00680, 2022. 3, 6.
[3] Alexii Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language.
arXiv preprini urXiv:2202. 200.00020.3355 2022. 1,3
[4] Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv
preprint arXiv:2/06. 08254, 2021. 3
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33: 33:1877-1901, 2020. 1,
3
[6] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: /5 12.0012, 2015. 2, 6,8
[7] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020. 1,3
[8] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758,
2021. 1,
[9] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotate 3d reconstructions of indoor scenes. In Proceedings of the
IEEE conference computer vision and patlern recognition, pages 5828-5839, 2017. 2017, 3, 8
[10] Jacob Devlin, Ming- Weei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1810:04880
2018. 3.3
[11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 100004805, 2018.
[12] Zhipeng Ding, Xu Han, and Mare Niethammer. Votenet: A deep learning label fusion method for multi-atlas segmentation. In International Conference on Medical Image Computing and
Computer-Assisted Intervention, pages 202-210. Springer, 2019. 8
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11 11999, 2020. 3.3
[14] Haoqiang Fan, Hao Su, and Leonidas! Guibas. A point set generation network for 3d object reconstruction from single image. In Proceedings of the IEEE conference on computer vision
and pattern recognition, pages 605-613, 2017. 6
[15) Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min
Hu. Pet: Point cloud transformer. Computational Visual Media, 7(2):187-19, 2021. 6 [16] Zhizhong Han, Mingyang Shang, Yu-Shen Liu, and Matthias Zwicker. View inter inter-prediction gan: Unsupervised representation learning for 3d shapes by learning global shape memories to support local view predictions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8376-8384, 2019. 6
[17] Zhizhong Han, Xiyang Wang, Yu-Shen Liu, and Matthias Zwicker. Multi-angle point cloud- vae: Unsupervised feature learning for 3d point clouds from multiple angles by joint self- reconstruction and half-to-half prediction. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 10441-10450. IEBE, 2019. 6
[18] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprini arXiv:21 11.2/11107777 2021. 1,
[19] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 9729-9733 2020. 3,7
11

---

[20] Siyuan Huang, Yichen Xie. Song- Chun Zhu, and Yixin Zhu. Spatio-temporal self-supervised representation learning for 3d point clouds. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6535-6545, 2021. 8
[21] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages
4904-4916. PMLR, 2021. 1
[22] Jiaxin Li, Ben M Chen, and Gim Hee Lee. So-net: Self-organizing network for point cloud analysis. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 9397-9406, 2018. 6
[23] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. pointenn: Convolution on x-transformed points. Advances in neural information processing systems,
31:820-830, 2018.00 6,
[24] Yongcheng Liu, Bin Fan, Gaofeng Meng, Jiwen Lu, Shiming Xiang, and Chunhong Pan. Densepoint: Learning densely contextual representation for efficient point cloud processing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5239-5248,
2019.
[25] Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan. Relation-shape convolutional neural network for point cloud analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8895-8904 2019. 7
[26] Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2906-2917, October 2021. 8
[27] Yatian Pang. Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud sepf-suue learning. arXiv preprint arXiv:2203.066044 2022.
3
[28] Omid Poursaeed, Tianxing Jiang. Han Qiao, Nayun Xu, and Vladimir G Kim. Self-supervised learning of point clouds via orientation estimation. In 2020 International Conference on 3D
Vision (3DV), pages 1018-1028. IEEE, 2020.
[29] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer
vision and pattern recognition, pages 652-660, 2017. 6, 7,8
[30] Charles Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical feature learning point sets in a metric space. arXiv preprint arXiv: 10e::1706000 02413, 2017. 5, 6, 7, 8 [31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[32] Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training- 2018. 1,3
[33] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019. 1.3
[34] Jason Tyler Rolfe. Discrete variational autoencoders. arXiv preprint 1609 000,00000,, 2016.
[35] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference Medical image computing and computer-assiisted intervention, pages 234-241. Spriiger 2015. 2,
[36] Jonathan Sauder and Bjarne Sievers. Self-supervised deep learning on point clouds by recon-
structing space. Advances in Neural Information Processing Systems, 32. 2019. 3
[37] Jonathan Sauder and Bjarne Sievers. Self-supervised deep learning on point clouds by recon- structing space. Advances in Neural Information Processing Systems, 32, 2019. 6
[38] Mikaela Angelina Uy, Quang- Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In Proceedings of the IEEE/CVF International Conference on Computer Vision,
pages 1888-1597.222 1519.2019. 2, 7
[39] Diego Valsesia, Giulia Fracastoro, and Enrico Magli. Learning localized representations of point clouds with graph-convolltiinall generative adversarial networks. IEEE Transactions on Multimedia, 23:402 414, 2020. 6
[40] Hanchen Wang. Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt J Kusner. Unsupervised point cloud pre-training via occlusion completion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9782-9792 2021. 3, 6,8
12

---

[41] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning on point clouds. Acm Transactions On Graphics (tog), 38(5):1-12, 2019. 6. 7, 8
[42] Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichten- hofer. Masked feature prediction for self-supervised visual pre-training. arXiv preprint
arXiv:2//2.09/3 2021.
[43] Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning a probabilistic latent space of object shapes via 3d generative-adversaria modeling. Advances in neural information processing systems, 29, 2016.
[44] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 1912-1920, 2015. 2, 6,
7. 7.8.9
[45] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In European conference on computer vision, pages 574-591. Springer, 2020. 3,
[46] Zhenda Xie, Zheng Zhang. Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. arXiv preprint
2021.
[47] Yaoqing Yang. Chen Feng. Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In Pruceedings of the IEEE conference on computer vision and pattern recognition, pages 206-215, 2018. 6
[48] Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Transactions on Graphics (ToG), 35(6):1-12, 2016. 2, 7, 8, 9 [49] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point- bert: Pre-training 3d point cloud transformers with masked point modeling. arXiv preprint 1.X.:22111.14//9 2021. 3, 6, 7,8
[50] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointelip: Point cloud understanding by clip. arXiv preprint
arXiv:21/2.02443 2021.
[51] Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of
3d features on any point-cloud. In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pages 10252-10263, 2021. 3, 8
[52] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer.
In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 16259-
16268, 2021. 6
[53] Jinghao Zhou, Chen Wei, Huiyu Wang. Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832 2021. 1,
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
13

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasig new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material or as URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you're you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "VjMzVDtyE7p", "reviewer": "Reviewer_Ddan", "summary": "This work proposes a multi-scale masked autoencoder for pre-training on point clouds. It designs a pyramid architecture for hierarchical encoder and decoder to progressively model 3D point clouds. A novel multi-scale masking and a back-project strategy are proposed for generating consistent visible regions across scales for point clouds. Extensive experiments demonstrate that the proposed method outperforms other pre-training methods on the linear SVM classification task and downstream tasks.", "questions": "1. Why does the final experiment use only 2-3 scales and what about more? Please explain further the motivation for the choice of the number of scales.\n2. In Sec 3.2 ‘Token Embedding and Merging’, “After that, we obtain the initial point tokens T_1^v for the 1-st stage of the encoder” seems to be conflict with the Figure 2 where the T_1^v is the output of the 1-st stage of the encoder.\n3. In Sec 4.2 ‘Shape Classification’, “We follow Point-BERT to use the voting strategy [29] for fair comparison on ModelNet40…”, as the Point-BERT does not seem to mention that it uses a voting strategy and dose not cite the paper [29]. Please explain further the voting strategy.\n\n", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper is generally technically sound.\n2. The design of the masking and back-project strategy is novel and fits well on the 3D point clouds pretraining task.\n3. The extensive experiments verify the effectiveness of the proposed approach.\n\nWeakness:\n1. The overall framework seems to be simply a variant of PoinNet++ with Transformer Encoder, especially the idea of the Skip Connection and Point Token Upsampling, hence the framework seems to have insufficient technical innovations.\n2. It can be noted that the U-Net like network demonstrated in Figures 2 and 3 have arbitrary multiple scales (at least greater than three). However, the actual experiments have only 2-3 scales and do not give an adequate explanation for the choice of the number of scales.\n\n \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "N28EVxYMt17", "reviewer": "Reviewer_774k", "summary": "This paper uses MAE to learn the point cloud features without fixed topology. Its multi-scale architecture can capture features at different resolutions for guidance. The pyramid architecture also facilitates the network to extract high-level features, and the self-attention mechanism adopted by this network also controls the focus of the network. The complementary skip connections also better connect the global and local features to guide each other. And it also produces competitive results in experiments.", "questions": "1) There are some works in the field that apply masked autoencoders in point clouds except Point-BERT, but I find that the authors miss these works[1][2][3][4], and I want to see more discussion from the authors about some of these out-picked works, and what this paper amounts to in terms of improvements to these precursor works.\n2) I hope that the authors can add visual interpretation of the results of the Local Spatial Self-Attention module in their ablation experiments.\n3) The authors keeps stressing that their model can pay more attention to fine-grained information, but there are no more descriptions about fine-grained information except \\textbf{Figure 2} in \\textbf{Appendix C} (which only shows the point cloud images with different scales).\n4) Although the authors have achieved good results in the experiments after freezing the encoder for the relevant downstream tasks, have you analyzed the influence of the CD loss you selected for the training of the model's results? Now it seems that the decoder is driven only by CD Loss. Whether to use EMD as loss in the experiments or combine the two as compound loss? Will the results still be as good as now? I'd like to see the quantitative results if possible.\n5) Miss the figure tag in \\textbf{Appendix C. Line 68}: “As shown in Figure []”.\n\nIn the current version, there are some issues as well. I look forward to the response by the authors. For now, I would recommend a borderline accept rating for the paper.\n\n[1]Fu, Kexue, et al. \"POS-BERT: Point Cloud One-Stage BERT Pre-Training.\" arXiv preprint arXiv:2204.00989 (2022).\n\n[2]Liu, Haotian, Mu Cai, and Yong Jae Lee. \"Masked Discrimination for Self-Supervised Learning on Point Clouds.\" arXiv preprint arXiv:2203.11183 (2022).\n\n[3]Pang, Yatian, et al. \"Masked autoencoders for point cloud self-supervised learning.\" arXiv preprint arXiv:2203.06604 (2022).\n\n[4]Pellis, Eugenio, et al. \"An Image-Based Deep Learning Workflow for 3D Heritage Point Cloud Semantic Segmentation.\" International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences-ISPRS Archives 46.2/W1-2022 (2022): 429-434.", "limitations": "Yes, I don't see any potential negative social impact of this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Originality: The main idea of the proposed approach is to use the masked autoencoders. To my limited knowledge about it, this is novel and thus the originality is reasonable.\n- Quality：While the approach seems reasonable and the experimental results look promising, I have the following concerns(See Questions) about the paper.\n- Clarity：This manuscript is clearly written.\n- Significance：There are not many works in the field of point cloud feature learning by masked autoencoders, so I think this paper makes a positive contribution.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9jxCa8LrMEA", "reviewer": "Reviewer_3boA", "summary": "This paper extends the MAE to deal with irregular 3d point clouds. It proposes a multi-scale u-net like encoder-decoder to learn both fine-grained local shape information and global semantic information. The network is self-supervised trained by masking the input point cloud with multi-scale mask strategy. Experiments are conducted on ModelNet40 with the learned representation combined with SVM, achieving good classification results which is comparable to the SOTA supervised methods. The effectiveness of the proposed method is also validated by fine-tuning on the down-stream tasks such as classification, part segmentation and 3d object detection, and observed improvements over existing methods.", "questions": "please see the above negative points.", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Positives\n+: The paper studies an important problem in deep learning and computer vision community. The proposed method is a reasonable extension of MAE to deal with point clouds.\n+: Good results were obtained by the proposed method on different tasks.\n+: Detailed ablation study to validate the effectiveness of each component.\n\nNegatives\n-: It would be better if the authors can also provide the results of using the frozen encoder for other tasks like part segmentation and 3d object detection, so as to show that the conclusion about the learned representation is general enough.\n-: As the advantage of self-supervised learning is to harvest from a large amount of unlabeled data, the paper lacks results about using more dataset for pre-training, for instance, using all three kinds of datasets (ShapeNet, ModelNet, ScanNet) together. This is not only helpful for readers to know the potential of the proposed method, but also can be interested to see if the proposed method can deal with different kinds of data together to boost the performance.\n-: The influence of different amount of pre-training examples should also be included in the ablation study.\n-: For the paper writing, it is unclear about the difficulties of introducing the MAE into 3d point cloud. It seems that there are MAE methods for 2D images, then why not we use it for 3d point, and so, the paper makes it done. Therefore, the contribution of this paper is not significant.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept