3D Feature Prediction for Masked-AutoEncoder-Based Point Cloud Pretraining
Authors: Siming Yan
Yuqi Yang
Yu-Xiao Guo
Hao Pan
Peng-Shuai Wang
Xin Tong
Yang Liu
Qixing Huang
Venue: ICLR 2024 poster (2024)
Keywords: point cloud
point cloud self-supervised learning
point cloud pre-training
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose a masked-autoencoder-based method for point-cloud pre-training. The model is enhanced by focusing on intrinsic features using a novel attention-based decoder, showing improved performance.

Abstract: Masked autoencoders (MAE) have recently been introduced to 3D self-supervised pretraining for point clouds due to their great success in NLP and computer vision. Unlike MAEs used in the image domain, where the pretext task is to restore features at the masked pixels, such as colors, the existing 3D MAE works reconstruct the missing geometry only, i.e, the location of the masked points. In contrast to previous studies, we advocate that point location recovery is inessential and restoring intrinsic point features is much superior. To this end, we propose to ignore point position reconstruction and recover high-order features at masked points including surface normals and surface variations, through a novel attention-based decoder which is independent of the encoder design. We validate the effectiveness of our pretext task and decoder design using different encoder structures for 3D training and demonstrate the advantages of our pretrained networks on various point cloud analysis tasks.

Published as a conference paper at ICLR 2024
3D FEATURE PREDICTION FOR MASKED- AUTOENCODER- -BAEED POINT CLOUD PRETRAINING
Siming Yan*t, Yuqi Yang', Yuxiao Guo Hao Pan Peng-Shuai Wang", Xin Tong', Yang Liu', Qixing Huang" The University of Texas at Austin, Microsoff Research Asia Peking University
(s tsiming, huanggx)@cs utexas.edee [wangps]@hotmaillccmm (t- (t-yuqyan, Yuxiao. Guo, haopan, yangliu, xtongjemiccosoot..co
ABSTRACT
Masked autoencoders (MAE) have recently been introduced to 3D self-supervised pretraining for point clouds due to their great success in NLP and computer vision. Unlike MAES used in the image domain, where the pretext task to restore features at the masked pixels, such as colors, the existing 3D MAE works reconstruct the missing geometry only, i.e, the location of the masked points. In contrast to previous studies, advocate that point location recovery is inessential and restoring intrinsic point features is much superior. To this end, we propose to ignore point position reconstruction and recover high-order features masked points including surface normals and surface variations, through a novel attention-basee decoder which is independent of the encoder design. We validate the effectiveness of our pretext task and decoder design using different encoder structures for 3D training and demonstrate the advantages of our pretrained networks on various point cloud analysis tasks. The code is available at https:://gituubbboo www.//imi////////eeeoooommii Yan/MaskFeat3D
INTRODUCTION
Self-supervised pretraining has recently gained much attention. It starts from a pretext task trained large unlabeled data, where the learned representation is fine-tuned on downstream tasks. This approach has shown great success in 2D images (Chen et al., 2020;
medkad points
ingui unmoeked points
low
mask tokets
predicted poins
pradictal Eesturcs
Decoder
Grill et al., 2020
He al. 2020; Bao al.
2022, He et al.
2022] Zhou et al.] 2022
Zhuang et al.) 2021
2019) and natural language
processing (NLP) (Devlin et al. 2019 Brown ct al., 2020). Recently, people started looking into self celf-super pretraining point cloud data due to its importance 3D analysis and robotics applications.
Enceder
Enceder
Astention based Decoder
Standard Point-MAL:
An important self-supervised pretraining paradigm
Figure 1: Comparison of standard Point-MAE and our proposed method. Unlike standard Point-MAE that uses masked points as the prediction target, our method use novel drention-basee decoder to leverage masked points as additional input and infer the corresponding features.
masked signal modeling (MSM), including BERT (Bugliarello et al. 2021). BEiT (Bao al. 2022), and masked autoencoders (MAE) (He et al. 2022). has recently been adopted to 3D domains. MSM has simple setup: mondomly-maskedr input is fed to the encoder, and decoder strives to recover the signal at the masked region. MSM highly scalable and exhibits superior performance in many downstream vision and NLP tasks, outperforming their fully supervised equivalents. Additionally, it does not require extensive augmentation, which is essential and critical to another self-supervised
Part of the work done when interning Microsoft Research Asia.

---

Published as a conference paper at ICLR 2024
pretraining paradigm contrastive learning. In images, a mask refers to a randomly selected portion of the pixels, and the pixel colors or other pixel features in the masked region are to be reconstructed by the decoder.
For 3D point clouds, the PointBRT approach et al. 2022) masks point patches and recovers patch tokens that are pretrained by point cloud Tokenizer. As reconstruction features are associated with patches of points, the learned features at the point level are less competitive. MAE-based pretraining schemes (Pang et al. 2022 Hess et al. 2022 Zhang al. 2022 Liu al. 2022) tackle this problem by point-wise pretext tasks. However, their decoders are designed to recover the positions of the masked points in Cartesian coordinates or occupancy formats (Fig. Ieft). These designs make an intrinsic difference from 2D MSMs, where there is no need to recover masked pixel locations. This key difference makes MSM pay more attention to capturing the irregular and possibly noisy point distribution and ignore the intrinsic surface features associated with points, which are essential for 3D point cloud analysis.
In the presented work, we propose to recover intrinsic point features, i.e., point normals, and surface variations (Pauly et al.) 2002) at masked points, where point normals are first-order surface features and surface variations are related to local curvature properties. We clearly demonstrate that the recovery of high-order surface point features, not point locations, the key to improving 3D MSM performance. Learning to reconstruct high-order geometric features forces the encoder to extract distinctive and representative features robustly that may not be captured by learning to reconstruct point positions alone. Our study justifies the importance of designing signal recovery for 3D MSMs. It aligns 3D MSM learning with MSM development in vision, where feature modeling plays a critical role (Wei al. 2022).
To recover point signals, we design a practical dreettion-basd decoder. This new decoder takes masked points as queries, and stacks several transformer blocks. In each block, self-attention is used to propagate context features over the masked points and cross-attention is applied to fabricate the point features with the encoder's 's output (As shown in Fig. righh and Fig. 2 This design is separable from the encoder design. Therefore, common 3D encoders, such as sparse CNNs, point- based networks, and ransformer-baseee networks, can all be adopted strengthen the pretraining capability. Another benefit of this decoder design is that the masked point positions are only accessible by the decoder, thus avoiding leakage of positional information in the early stage of the network, as suggested by (Pang al. 2022).
We conducted extensive ablation studies to verify the efficacy of our masked feature design and decoder. Substantial improvements over previous approaches and the generalization ability of our pretraining approach are demonstrated on various downstream tasks, including 3D shape classification, 3D shape part segmentation, and 3D object detection. We hope that study can stimulate future research on designing strong basedd 3D backbones.
We summarize the contributions of our paper follows:
We propose novel masked autoencoding method for 3D self-supervised pretraining that predicts intrinsic point features at masked points instead of their positions.
We introduce a unique attention-based decoder that can generate point features without relying on any particular encoder architecture.
Our experiments demonstrate that restoring intrinsic point features is superior to point location recovery in terms of Point cloud MAE, and we achieve state-of-the-ar performance various downstream tasks.
RELATED WORK
Self-supervised pretraining in 3D Self-supervised pretraining is an active research topic in machine learning (Liu et al. (202T). The early adoption of self-supervised pretraining for 3D is to use autoencoders (Yang et al.) 2018 Yan et al. 2022) and generative adversarial networks (Wu) et al. 2016) to learn shape-level features, mainly for shape classification and retrieval tasks. Other self-supervised pretext tasks, such as clustering and registration, are also developed for 3D pretraining. Later, due to the great ability to learn features at both the instance and pixel levels in self self-supervised manner, contrastive learning (Wu et al.) 2018: Grill et al. 2020 He et al., 2020 Brown et al.) 2020
Chen & He 2021 Yan et al. 2023) was introduced into the 3D domains to extract distinctive instance
2

---

Published as a conference paper at ICLR 2024
Black Feature
Decoder
Masked points
Black Feature Pair
Encodor
Campleie lapat
feature
Block Feature Pair
points
Figure 2: The pretraining pipeline of masked 3D feature prediction approach. Given complete input point cloud, first separate into masked points and unmasked points (We cuhe mask here for hetter visualization). We take unmasked points the encoder input and output the block feature pairs. Then the decoder takes the block feature pairs and query points(i.e.. masked points) as the input, and predicts ee-quury--oint features.
and point-wise features for various downstream tasks (Wang et al. 202 Xie et al. 2020 Hou et al., 2021 Zhang al. 2021). However, contrastive learning requires data augmentation heavily to form positive negative pairs for effective feature learning.
Masked signal modeling in 3D Masked signal modeling using ransformer-based architectures for self-supervised learning (SSL) has shown great simplicity and superior performance. PointBERT (Yu et al.
2022) and PointMAE (Pang et al. (022) are two such works that inherit from this idea. PointBERT partitions a point cloud into patches and trains transformer- recover masked patches' tokens. In contrast, PointMAE directly reconstructs point patches without costly tokenizer training. using Chamfer distance as the reconstruction loss. Other works like (Zhang al. 2022 Liu et al. 2022) and (Hess et al., 2022) explore different strategies for point cloud reconstruction or classification with masking. As discussed in Sec. the pretext tasks most previous works focus only masked point locations.
autoencoder to
Signal recovery in masked autoencoders Masked autoencoders for vision pretraining typically use raw color information in masked pixels the target signal (He et al. [022). However, Wei al. (Wei et al., 2022) have found that using alternative image features, such as HOG descriptors, tokenizer features, and features from other unsupervised and supervised pretrained networks, can improve network performance and efficiency. In contrast, existing 3D MAE methods have limited use of point features and struggle with predicting the location masked points. Our approach focuses on feature recovery rather than position prediction, selecting representative 3D local features such as point normals and surface variation (Pauly et al. 2002) as target features to demonstrate their efficacy. Our study allows for leveraging more advanced 3D features in 3D masked autoencoders, while further exploration of other types of 3D features (Laga et al. 2018) is left for future work.
MASKED 3D FEATURE PREDICTION
In this section, we present our masked 3D feature prediction approach for self-supervised point cloud pretraining. Our network design follows the masked autoencoder paradigm: 3D encoder takes a point cloud whose points are randomly masked as input, and decoder is responsible for reconstructing the predefined features at the masked points. The network architecture is depicted in Fig. 2 In the following sections, we first introduce the masking strategy and 3D masked feature modeling in Sec. Sec.B.11 and 3.2, and then present our encoder and decoder design in Sec. 3.c.3.3aamm and Here, the key ingredients of our approach are the design of prediction targets and the decoder, which govern the quality of the learned features.
3.1 3D MASKING
We follow the masking strategy proposed by PointBERT (Yu et al. 2022) to mask out some portions of an input point cloud and feed it to the encoder. Denote the input point cloud as P E I NgNx where N is the number of points. We sample K points using farthest point sampling (FPS). For each sample point, its k-nearest neighbor points form a point patch. For given mask ratio m,, 0 < m, < 1, we randomly select M patches and remove them from the input, where M min( (m, K], K 1).
3

---

Published as a conference paper at ICLR 2024
In the following, the masked points and the remaining points are denoted by PM and Pu. respectively.
3.2 TARGET FEATURE DESIGN
As argued in Sec. | we advocate against using point locations as the reconstructed target. We choose to reconstruct normal and surface variation at each point, which reflect differential surface properties. On the other hand, our decoder design (to be introduced in Sec. (3.4) takes query points as input and output predicted point-wise features. Therefore, the decoder implicitly carries positional information for learning meaningful features through the encoder.
Given point cloud, both point normal and surface variations are defined using local principal component analysis (PCA). We first define a covariance matrix C, over local surface region around p:
point cloud
point normal surface variation
Figure 3: Visualization of point features.. The point
Cr := (xESnE (- - x) dx
normal is color-cnded by the normal vector. The surface variation is color-coded where white indicates low value and red indicates high value.
(1)
(xESNS.(p) 1 dx
where (p) is the local surface region at P. restricted by sphere centered at with radius We set r = 0.1 in our case. The ablation details are shown the supplement.
The normal r2p) at is estimated as the smallest eigenvector of C,. The sign of each normal is computed by using the approach of (Hoppe et al. 1992).
Surface variation (Pauly et al. 2002) at is denoted by (p))), in the following form:
A
(p( An + +2
where A S A2 S Ag are the eigenvalues of C,-. Surface variation is a geometric feature that measures the local derivation at point p in neighborhood of size original and modified versions have been used as robust feature descriptor for a variety of shape analysis and processing tasks, such saliency extraction (Pauly & Gross 2001), curved feature extraction (Pauly et al. 2003), shape segmentation (Huang et al. 2006; Yan et al. 2021). and shape simplification (Pauly et al. 2002).
given surface S Its
In the limit, i.e., when r + 0, (p) is related to the mean curvature (Clarenz et al. 2004). By varying the radii of S,, multiscale surface variation descriptors can be constructed. In our work, we chose only single-scale surface variation for simplicity.
Although both surface normal and surface variation are derived from local PCA, they are complementary to each other in the sense that surface normal carries first-order differential property while surface variation carries second-order differential property due to its relation to mean curvature. We visualize both features in Fig. 33 show more examples in supplement. In Sec. 4 we show that reconstructing surface normal and surface variation leads to better learned features than reconstructing one of them.
Loss function Point normals and surface variations represent first- and second-order surface properties. Their value intervals are also bounded: surface normal has wnit length; surface variation non-negative and not greater than 3. Their value-bounded properties are suitable for easy minimizing the deviation from the prediction to their ground truths, compared to using unbounded features such as curvatures. We denote the point normals and surface variations of PM by Nrr E MMx and VM E a respectively. The loss function for pretraining the masked autoencoders is composed of two terms:
I, = WWss - -N 11::
(3)
L. 1Vs Dar l1:;
(4)
4

---

Published as a conference paper at ICLR 2024
where NM and Vas the predicted versions of Nas and VM, respectively. The total loss function d + AeLy, where Å1 = 22 = in our case.
3.3 ENCODER DESIGN
Unlike most MAE-based approaches that are limited to ViT-based encoders, our approach is not restricted to any specific type of encoder. Common 3D encoders for point clouds are all supported, as long as the encoder outputs set of learned features bind to spatial blocks, where spatial blocks could
be point patches used for ViT-like transformer encoders Yuu et al. 2022 Pang et al. 2022 [ Liu et al.
2022: Zhang et al. 2022), set abstractions used by PointNet+++liik encoders (Qi et al. 201 Qian
al. 2022), coarse voxels used by sparse CNN-based encoders (Wang et al. 2017 Graham et al.
2018
Choy et al. 2019).
In the following, we briefly review these typical encoders and their adaption for our pretraining.
ViT-based encoders These encoders first embed point patches via PointNet Qi et al. 201 7a) then send these patch tokens to a standard transformer that includes several multihead self attention layers and feedforward layers. The transformer outputs the fabricated token features, corresponding to every input point patch. The token feature f, and the patch center c, form a block feature pair B3 (fi, c.), which is needed by our decoder. Here we can call f; block feature and block centroid.
PointNet++-lik encoders In these encoders, the network features are aggregated through a number of set abstraction levels. We take the learned features and the centroids the coarsest set abstractions as block feature pairs.
Sparse CNN-based encoders These encoders apply 3D convolution on sparse voxels from the finest level to the coarsest level. Multiple convolution layers and resblocks are commonly used. We interpolate the coarse voxel features at the centroids of the unmasked patches and use these interpolated features and the patch centroids form our block feature pairs.
As suggested by (Pang et al., 2022), the early leaking of masked point information to the network could jeopardize feature learning. We adopt this suggestion: feed the unmasked points to the encoder only, and leave the masked points to the decoder.
3.4 DECODER DESIGN
Decoder structure We design an attention decoder to restore the target features at masked regions. The decoder takes the block feature pairs B = ( [ from the encoder and query point set Q, i.e. the masked point set Pv. It is composed of stack of transformer blocks, where / = 4 in our case (See Fig. 2). Each block contains self-attention layer and cross-attention layer. The self-attention layer takes the query points and their positional embeddings as input and outputs the per-query point features, denoted by S Then Sin and the encoder block features B are passed into the cross-altention layer, where Sün serves as attention query, the block features serve as attention key and value, and the block centroids are the positional embedding the block features. The output per-point features from the last block go through MLP head to predict the target features at the query points.
Efficacy of self-attention layers At first glance, sufficient to use cross-attention layers only for predicting per-point features. The recent masked discrimination work (Liu et al., 2022) obeys this intuition for its decoder design, no information exchanged between different query points. Instead, we introduce the self-attention layer to propagate information over query points and use multiple attention blocks to strengthen the mutual relationship progressively. We found that our design significantly improves feature learning, verified by our ablation study (See Sec. 4.3
Supporting of various encoders In the above design, the decoder needs block feature pairs only from the encoder, thus having great potential to leverage various encoder structures, not limited to ViT-based transformer structures. This advantage is verified by our experiments (See Sec.
Feature reconstruction versus position reconstruction Note that our decoder and loss design do not explicitly model point positions, which are zero-order surface properties complementary to
5

---

Published as a conference paper at ICLR 2024
surface normals and surface variations. Instead, the decoder predicts feature values at query points. Therefore, the zero-order positional information is already encoded implicitly. This explains why our approach is superior to baseline approaches that reconstruct point positions for feature learning (See Sec. 4.2)
Query point selection Due to the quadratic complexity self-attention, the computational cost for full query set could be much higher. In practice, we can randomly choose point subset from PM as the query set during training. By default, we use masked points as queries.
EXPERIMENT ANALYSIS
We conducted a series of experiments and ablation studies to validate the efficacy and superiority of our masked 3D feature prediction approach, in short MaskFeat3D, for point cloud pretraining.
4.1 EXPERIMENT SETUP
Pretraining dataset We choose ShapeNet (Chang et al. 2015) dataset for our pretraining, following
the practice of PointBERT (Yu et al. 2022) and previous 3D MAE-based approaches (Pang et al. 2ha2g et al. 2022 Liu et al. 2022). ShapeNet (Chang et al. 2015) contains 57 748 synthetic 3D shapes from 55 categories. We sample 50 000 points uniformly on each shape and select 128 nearest points from them for each point in the point cloud for constructing the local region to approximate
surface variation. During pretraining. N 2048 points are randomly sampled to create the point cloud.
Network training We integrated different encoders with our masked 3D feature prediction approach, including the ViT-based transformer used by (Pang et al. 2022), sparse- Caatss--CN-baaeed encoder (Choy er al., 2019), and PointNeXt encoder (Qian et al.] (022) which is an advanced version of PointNet+++ implemented all pretraining models in PyTorch and used AdamW optimizer with 10-1 weight decay. We use PointBERTTs masking strategy for ShapeNet pretraining. We set K = 128 in FPS, 32-nearest points to form the point patch, and the best masking ratio is 60% empirically, The number of transformer blocks in the decoder is 4. The learning rates of the encoder and the decoder are set to 10-3 and 10 10-4, respectively. Standard data augmentation such as rotation, scaling, and translation employed. All models were trained with 300 epochs on eight 16 GB Nvidia V100 GPUs. The total batch size is 64.
Downstream Tasks We choose shape classification and shape part segmentation tasks to validate the efficacy and generalizability of our pretrained networks.
carried ModelNer40 (Wu et al. 2015) and ScanObjectNN (Uy et al. 2019). ModelNer40 is a widely used synthetic dataset that comprises 40 classes and contains 9832 training objects and 2468 objects. is real-world scanned dataset that includes approximately 15 000 actual scanned objects from 15 classes. As the domain gap between ShapeNet and ScanObjectNN is larger than that between ShapeNet and ModelNet40, the evaluation on ScanObjectNN is good measure of the generalizability pretrained networks.
Shape classification: The experiments
two different datasets:
In contrast, ScanObjecth
Shape part segmentation ShapeNetPart Dataset (Yi et al. 2016) contains 16 880 models from 16 shape categories, and each model has 2~6 parts. Following the standard evaluation protocol (Qi et al. 2017b), 2048 points are sampled on each shape. For evaluation, report per-class mean IoU (cls. mloU) and mean IoU averaged over all test instances (ins. mloU).
The training-and-test split of the above tasks follows existing works. For these downstream tasks, we employ the task-specific decoders proposed by PointMAE (Pang al. (002)) and reload the pretrained weights for the encoder. Training details are provided in the supplemental material.
4.2 EFFICACY OF 3D FEATURE PREDICTION
The advantages in learning discriminative features by our masked feature prediction approach are verified by its superior performance in downstream tasks.

---

Published as a conference paper at ICLR 2024
SeanObjectNN OBI-ONLY
ShapeNetPart
ShapeNetParr( (1% labels)
Method
OBJ-BG
PB-TSO-RS
mloU
mloU
ins. mloU
cls. mloU
PointViT et
2022
79.9
80.6
77.2
85.1
83.4
77.6
72.2
PointBERT Y0
22122
87.4
88.1
83,1
85.6
84,1
79.2
73.9
MaskDiscr
20122
89.7
89.3
84.3
86.0
84,4
78.8
72.3
MaskSurle Zhbang et 3l. 2022
91.2
89.2
85.7
86.1
84.4
PointMAB
2022
90.0
88.3
85.2
86.1
79.1
74.4
MaskFeat (PobutV1))
91.7(91.6)
90.0(89.6)
87.7(87.5)
86.3(86.3)
84.9(84.8)
80.0(79.9)
75.1(75.0)
Performance comparison of MAE-based approaches on downstream tasks. All the methods in the first section use the same transformer backbone architecture, PointViT. represents from scratch results and all other methods represent the fine-runing results using pretrained weights. The average result of runs given brackets.
Table I:
ScanObjectNN
ShapeNetPart
Method
OBJ-BG
OBJ-ONLY
PB-T50-RS
ins. mloU
cls. mloU
PointNet Qi al.
20
73.3
79.2
68.0
PointNet+++ al. 7b
82.3
84.3
77.9
85.1
81.9
Poin 2018
86.1
85.5
78.5
86.1
84.6
DGCNN AGCNN/Wangeed el. 20199
86.2
78.1
82.3
MinkowskdNee
2019
84.1
86.1
80.1
85.3
83.2
Poin'TTransfomeet Zhao et al. 202
86,6
83.7
PointMLPPMa et al, Zuzz
88.7
88.2
85.4
86.1
84.6
Stratified Trums former Lai ct al.
2022
86.6
85.1
PointNeXt Qian ul. 2002
91.9
91.0
87.1
84.7
MaskFcat3D (PointVI
91.7(91.6)
90.0(89.6)
87.7(87.5)
86.3(86.3)
84.9(84.8)
MaskFeat3D (MinkowskiNet)
(85.0)
87.0(86.7)
80.8(80.6
85.6(85.5)
83.5(83.5)
MaskFeat3D (PointNeXt)
92.7(92.6)
92.0(91.9)
88.6(88.5)
87.4(87.4)
85.5(85.5)
Table Comparison with supervised methods. The average result 3 runs is given in brackets.
Comparison with MAE-based approaches We compare our approach with other MAE-base approaches that use the same encoder structure. Tab. reports that: (1) the performance of MAE- based methods surpasses their supervised baseline PoinViT; (2) our strategy of reconstructing point features instead of point positions yields significant improvements ScannObjectNN classification, improving overall accuracy the most challenging split, PB-T50-RS, from 85. 7% (MaskSurfel) to 87.7%, and showing consistent improvements on other splits and ShapeNetPart segmentation.
2022) PPoitMMAE (Pang et al. 2022), and MaskDiscr (Liu et al., 2022) on ShapeNetPart segmentation with less labeled data. In this experiment, we randomly select 1% labeled data from each category, and finetune the network with all selected data. The performance is reported in Tab. [ which shows that using our pretrained network leads much better performance than the baseline methods.
We also compare the performance of our approach with PointBERT Yu
Comparison with supervised approaches Compared with state-of-thed of-the-attttttt supervised methods, our approach again achieves superior performance than most cxisting works as seen from Tab. including PointNet++ (Qi et al. 2017b) PooinCCNN (Li al. 2018). DGCNN (Wang al. 2019). MinkowskiNet (Choy et al. 2019), FrintTransformer (Zhao et al. 2021) and PointMLP (Ma et al.
2022). It is only inferior to the approaches that advanced encoder structures such stratified transformer (Lai et al. 2022) and PointNeXt (Qian et al., (022).
Encoder replacement To make more fair comparison, we replaced the PointViT encoder with PointNeXt's encoder, and retrained our pretraining network, denoted as MaskFeat3D (P) From Tab. 2 we can see that our pretraining approach with this enhanced encoder can yield SOTA performance all the downstream tasks, surpassing PointNeXt trained from scratch. We also used MinkowskiNet (Choy et al.) 2019) as our pretraining encoder, the performance gain over MinkowskiNet trained from scratch is +0.7% overall accuracy improvement ScanObjectNN classification, and +0.3% on ShapeNetPart segmentation. Please refer to the supplementary material for details.
Few-shot Classification To perform few-shot classification ModelNet40, we adopt the "K-way N-shot" settings as described in prior work (Wang et al. 2021a Yu et al. 2022 Pang et al. 2022). Specifically, randomly choose K out of the 40 available classes and sample N+20 3D shapes per class, with shapes used for training and 20 for testing. We evaluate the performance of FaskFeat3D under four few-shot settings: 5-way 10-shot, 5-way 20-shot, 10-way 10-shot, and 10-way 20-shot. To
7

---

Published as a conference paper at ICLR 2024
Method
5-way
10-way
10-shot
20-shot
10-shot
20-shot
DGCNN
31.6 + 2.8
40.8 1 4.6
19.9
1 2.1
16.9
+ 1.5
Method
Target Feature
ScanNN
OcCo
92.5
82.9
1.3
CrossPoint
92.5 = 3.0
94.9
2.1
83.6
5.3
87.9
+ 4.2
position only
85.2
Transformer'
$7.8 +7.8 5.2
84. 93.
89.4
6.3
PointMAE
position normal"
position + surface variation
85.7
85.9
OcCo
94.0 3.6
2.3
5.1
4.6
position normal variation"
86.0
PoinBERT
94.6 1. 3.1
06.3.19
91.0
92.7
5.11
MaskDiser
95.0 +: 3.7
97.2
9.44
93.4
4. 3.5
normal
86.5
PointMAE
96.3 = 2.5
9
92.6+11
95.0
3,0
MaskFeatt3
surface variation
87.0
MaskFear3D
97.1
2.1
98.4
3.8
95.7
3.4
normal variation
87.7
Table 3: Few-shot classification on ModeINet40. We report the average accuracy (%) and standard deviation (%) of 10 independent experiments.
Table 4: Ablation study on different features. uses position-index matching [Zhang et al.
2022 for feature loss computation.
mitigate the effects of random sampling, we conduct 10 independent runs for each few-shot setting and report the mean accuracy and standard deviation. Additionally, more ModelNet40 results can be found in the supplementary material.
Overall, the improvements of our approach are consistent across different backbone encoders and datasets.
4.3 ABLATION STUDY
We proceed to present an ablation study to justify various design choices. For simplicity, we choose the shape classification task on ScanObjectNN, where the gaps under different configurations are salient and provide meaningful insights on the pros and cons of various design choices. Due to space constraints, additional ablation studies are available in the supplementary material.
Decoder design The primary question that arises is whether is essential to disregard point position recovery. PointMAE's decoder follows standard ViT-like architecture, utilizing fully connected (FC) layer to directly predict the masked point coordinates. We implemented this decoder to predict our target features. However, since their decoder design does not encode masked point position, cannot solely predict target features without predicting point position. To address this, we follow the approach proposed in (Zhang et al. (2022) and employ position-index matching for feature loss computation. As shown in Tab. [4] even though incorporating point features as the predicting target can enhance performance, the overall performance still significantly lags behind our design. This experiment highlights the significance of both point feature prediction and disregarding point position recovery.
Target feature choice In Tab. [4] the experiment shows that: (1) All combinations of point normal and surface variation can yield significant improvements
existing MAE approaches that recover point positions (cf. Tab. T): (2) using both point normals and surface variations yields the best performance.
first- and second-order differential properties. They are relevant but complementary to each other. Therefore, reconstructing them together forces the encoder to learn more informative features than merely reconstructing one of them.
As discussed in Sec. 3.2 this due to the fact that they correspond
Decoder depth Tab. a varies the number of transformer blocks (decoder depth). A sufficient deep decoder is necessary for feature learning. Increasing the number of blocks from to 4 provides +1.5% improvement ScanObjectNN classification task. The performance drops when increasing the depth further, due to the overfitting issue. Interestingly. we note that --llckk decoder can strongly achieve 85.8% accuracy, which is still higher than the runner-up method (PointMAE).
Data augmentation Tab. b studies three traditional data augmentation methods: rotation, scaling, and translation. Since the standard scaling could change the surface normal and variation, we scale the shape by using the same factor on 3 different axis. The experiments show that rotation and scaling play more important role.
Masking ratio. Tab, 5ie varies the masking ratio of input point cloud, which is another important factor on our approach. When the masking ratio is too large, e.g., 90%, the remaining part contains too limited information, which makes the task too hard complete. When masking ratio is too
8

---

Published as a conference paper at ICLR 2024
Decuder depth
(b)
Data angmentation
(c) Mask ratio
(d) Decoder attention
(c) Query point ratio
blocks
ScamNN
scale
ScamNN
ralio
ScanNN
atiention type
ScanNN
query/mask
ScanNN
85.8
87.0
40%
86.8
cross only
85.7
25%
85.7
86.2
85.9
60%
87.7
crosssell
87.7
50%
86.2
87.7
87.7
90%
86.5
75%
86.6
8
87.5
85.1
100%
87.7
12
87.1
86.7
Table 5: Ablation studies our design choices. Please refer to Sec. 4.3
detailed analysis.
small, e.g., 40%, the task becomes too simple and impedes the feature learning. In our experiments, masking ratio=60% shows the best performance.
Decoder block design We tested whether the self-attention layer in our decoder is essential. By simply removing self-attention layers and using cross-attention layers only, we find that the performance has large drop (-2.0), see Tab. 5td.
Number of query points Finally, we varied the number of query points used by our decoder to see how it affects the network performance. Tab. [ shows that more query points lead to better performance. Here, "query/mask" is the ratio selected query points with respect to the total number of masked points.
4.4 SCENE-LEVEL PRETRAINING EXTENSION
In principle, masked point cloud autoencoders could be scaled to noisy, large-scale point clouds. Additionally, conducted an extension experiment on real-world scene-level data evaluate our approach. Specifically, we pretrained our model on the ScanNet (Dai et al. 2017) dataset and evaluated its performance on 3D object detection task using the ScanNet and SUN RGB-D (Song et al. 2015) dataset. The training details can be found in the supplementary material. In this experiment, we observed that surface normal has minor influence the pretraining, while surface variation remains robust feature. Moreover, we discovered that color signal could be an effective target feature. Hence, we pretrained model with surface variation and color the target features, and then fine-tuned the pretrained encoder on the downstream tasks. As shown in Tab. 6 given that previous studies lack unified network backbone, we selected two of the most common works, VoteNet and Point-M2AE, along with the latest work, CAGroup3D, as the network backbones respectively. And our model exhibits consistent improvements in all the settings, which further proves the generalizability of our approach on noisy, large-scale point clouds. Although the concrete scene-level experiments are not the main focus of this paper, the results indicate that this is promising direction.
ScanNet
SUN RGB-D
Method
Backbone
AP25
AP50
AP25
AP50
STRL
VoteNet
59.5
38.4
58.2
35.0
RandomRooms
VoteNet
61.3
36.2
35.4
PointContrast
VoteNet
59.2
38.0
34.8
DepthContrast
VoteNet
62.1
39.1
60.4
35.4
Point-M2AB
Point-M2.. M2AE
66.3
48.3
MaskFeat3D
VoteNet
63.3
41.0
61.0
36.5
MaskFeat33
Point-M2AE
67.5
50.0
MaskFeat33
CAGroup3D
75.6
62.3
67.2
51.0
Table 6: 3D object detection results.
CONCLUSION
Our study reveals that restoration of masked point location is not essential for 3D MAE training. By predicting geometric features such as surface normals and surface variations at the masked points via our cross-atiention-bet evaluated through extensive experiments and downstream tasks. Moreover, the performance gains remain consistent when using different encoder backbones. We hope that our study can inspire future research in the development of robust MAE-based 3D backbones.
decoder, the performance of 3D MAEs can improved significantly, as
Acknowledgement. Part of this work was done when Siming Yan was a research intern at Microsoft Research Asia. Additionally, would like to acknowledge the gifts from Google, Adobe, Wormpex AI, and support from NSF IIS 1047777777 HDR-1999 1934932, CCF-2019844, and IARPA WRIVA program.

---

Published as a conference paper at ICLR 2024
REFERENCES
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEÎT: BERT pre-training of image transformers. In ICLR, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
Emanuele Bugliarello, Ryan Cotterell, Naoaki Okazaki, and Desmond Elliott. Multimodal pretraining unmasked: A meta-analysis and unified framework of saioo-and-languaae BERTs. Transactions of the Association for Computational Linguistics, 9, 2021.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An information-rich 3D model repository. arxiv::152.003011 2015.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Christopher Choy, Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In CVPR, pp. 3075-3084, 2019.
Ulrich Clarenz, Martin Rumpf, and Alexandru Telea Robust feature detection and local classification for surfaces based on moment analysis. IEEE Trans. Vis. Comput. Graphics, 10(5):516-5244 2004.
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas A. Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In CVPR, pp. 2432-2443, 2017.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In ACL., pp. 4171-4186, Stroudsburg, PA, USA, 2019.
Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 3D semantic segmentation with submanifold sparse convolutional networks. In CVPR, 2018.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, BernardoAvila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. In NeurIPS, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In CVPR, pp. 15979-15988, 2022.
Georg Hess, Johan Jaxing, Elias Svensson, David Hagerman, Christoffer Petersson, and Lennart Svensson. Masked autoencoders for self-supervised learning on automotive point clouds. arXiv:2207.00531, 2022.
Hugues Hoppe, Tony DeRose, Tom Duchamp. John Alan McDonald, and Werner Stuetzle. Surface reconstruction from unorganized points. In SIGGRAPH, pp. 71-78. ACM, 1992.
Hou, Benjamin Graham, Matthias Nießner, and Saining Xie. Exploring data-efficient 3D scene understanding with contrastive scene contexts. In CVPR, 2021.
Qi-Xing Huang, Simon Flöry, Natasha Gelfand, Michael Hofer, and Helmut Pottmann. Reassembling fractured objects by geometric matching. ACM Trans. Graph., 25(3)566-5578 2006.
Hamid Laga, Yulan Guo, Hedi Tabia, Robert B Fisher, and Mohammed Bennamoun. 3D Shape
analysis: fundamentals, theory, and applications. John Wiley & Sons, 2018
10

---

Published as a conference paper at ICLR 2024
Xin Lai, Jianhui Liu, Li Jiang, Liwei Wang, Hengshuang Zhao, Shu Liu, Xiaojuan Qi, and Jiaya Jia. Stratified transformer for 3d point cloud segmentation. CVPR, pp. 8500- 2022.
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. PointCNN: Convolution on X-transformed points. In NeurlPS, pp.
2018.
Haotian Liu, Mu Cai, and Yong Jae Lee. Masked discrimination for self-supervised learning point clouds. In ECCV, 2022.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering, 2021.
Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local geometry in point cloud: simple residual MLP framework. ICLR, 2022.
Yatian Pang, Wenxiao Wang, Francis H. Tay, W. Liu, Yonghong Tian, and Liuliang Yuan. Masked autoencoders for point cloud self-supervised learning, In ECCV, 2022.
Mark Pauly and Markus Gross. Spectral processing of point-sampled geometry. In SIGGRAPH, pp. 379-386, 2001.
Mark Pauly, Markus Gross, and Leif P. Kobbelt. Efficient simplification of point-sampled surfaces. In Proceedings of the Conference on Visualization, pp. 163-170, 2002.
Mark Pauly, Richard Keiser, and Markus Gross. Multi-scale Feature Extraction Point-Sampled Surfaces. Computer Graphics Forum, 2003. ISSN 1467-8659.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. In CVPR, pp. 652-660, 2017a.
Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas. PointNet++: Deep hierarchical feature learning on point sets in a metric space. In NeurlPS, pp. 5105-5 511,,, 14, 2017b.
Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Abed Al Kader Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. PointNeXt: Revisiting PointNet++ with improved training and scaling strategies. In NewrlPS, 2022.
Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 567-576, 2015.
Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Duc Thanh Nguyen, and Sai-Ki Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In ICCV, pp. 119881197. IEEE, 2019.
Hanchen Wang, Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt J Kusner, Unsupervised point cloud pre-training via occlusion completion. In ICCV, pp. 9782-9792 2021a.
Peng-Shune Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun Yu Sun, and Xin Tong. O-CNN: Octree-based convolutional neural networks for 3D shape analysis. ACM Trans. Graph., 36(4), 2017.
Peng-Shuai Wang, Yu-Qi Yang, Qian-Fang Zou, Zhirong Wu, Yang Liu, and Xin Tong. Unsupervised 3D learning for shape analysis via multiresolution instance discrimination. In AAAI, 2021b.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael Bronstein, and Justin M. Solomon. Dynamic graph cnn for learning point clouds. ACM Trans. Graph., 38(5), 2019.
Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In CVPR, pp. 2022.
Jiajun Wu, Chengkai Zhang, Tianfan Xue, William T. Freeman, and Joshua B. Tenenbaum. Learning a probabilistic latent space of object shapes via 3D 2016.
modeling. In NeurIPS,
11

---

Published as a conference paper at ICLR 2024
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3D ShapeNets: A deep representation for volumetric shapes. In CVPR, pp. 1912-1920. IEEE Computer Society, 2015.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018.
Saining Xie, Jiatao Gu, Demi Guo, Charles R. Qi, Leonidas Guibas, and Or Litany. PointContrast: Unsupervised pre-training for 3D point cloud understanding. ECCV, 2020.
Siming Yan, Zhenpei Yang, Chongyang Ma, Haibin Huang, Etienne Vouga, and Qixing Huang.
Hpnet: Deep primitive segmentation using hybrid representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2753-27622 2021.
Siming Yan, Zhenpei Yang, Haoxiang Li, Li Guan, Hao Kang, Gang Hua, and Qixing Huang. Implicit autoencoder for point cloud self-supe representation learning. arXiv preprint arXiv:2201.00785 2022.
Siming Yan, Chen Song, Youkang Kong, and Qixing Huang. Multi-view representation is what you need for point-cloud pre-training. In The Twelfth International Conference on Learning Representations, 2023.
Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. FoldingNet: Point cloud auto-encoder via deep grid deformation. In CVPR, 2018.
Li Yi, Vladimir G. Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. scalable active framework for region annotation 3d shape collections. ACM Trans. Graph., 35(6), 2016.
Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-BERT: Pre- training 3D point cloud transformers with masked point modeling. In CVPR, pp. 19313-19322, June 2022.
Yabin Zhang, Jiehong Lin, Chenhang He, Yongwei Chen, Kui Jia, and Lei Zhang. Masked surfel prediction for self-supervised point cloud learning. MxXiv:2207.03111 2022.
Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of 3D features on any point-cloud. In ICCV, 2021.
Hengshuang Zhao, Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In ICCV. pp. 16269, 2021.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. ICLR, 2022.
Chengxu Zhuang, Siming Yan, Aran Nayebi, and Daniel Yamins. Self-supervised neural network models of higher visual cortex development. In 2019 Conference on Cognitive Computational Neuroscience, pp. 566-569, CCN, 2019.
Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael C Frank, James J DiCarlo, and Daniel LK Yamins. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, 210(3))e0 18, 2021.
12


Reviews:
[{"review_id": "xySBITLeOI", "reviewer": "Reviewer_HyKn", "summary": "The paper proposes a pre-training task for 3D encoders, so, later, can lead to improved performance when fine-tuned on a downstream task. The pre-training objective is the prediction of point-surface properties such as normal or surface variation from masked regions of the input point cloud.", "questions": "I would encourage the authors to include more challenging tasks such as semantic and instance segmentation of 3D scans.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper proposes an alternative to point coordinates prediction on a mask auto-encoder setup. Sampling point coordinates can be difficult for decoder architectures as the ones used by previous works. However, by fixing the point coordinate in the decoder these problems disappear and the task becomes to predict shape properties around the queried point.", "weaknesses": "I like the main idea of the paper, it is well presented and presents a significant improvement over previous works for most of the task. However, my main concern is not only related to this work in particular but to this line of works where they focus on tasks related to single objects. I have been playing around with these datasets for many years already, and I can say that datasets such as classification on ModelNet40, and segmentation on ShapeNet are relatively \"easy\", there is a lot of noise in the annotations, and I believe the improvements presented by current methods is simply overfitting to this specific data set. In other subfields of computer vision, a pre-training paper that is only evaluated on MNIST or CIFAR10 would not be accepted, but for some reason, they do for point clouds. So, I don't find these works convincing since the reported results and architectures usually do not translate to more challenging tasks such as semantic segmentation or instance segmentation on real 3D scans. That being said, this work presents results on the task of object detection of ScanNet and SUN-RGBD, which I believe is the right direction. However, I think more results reported on other tasks such as semantic or instance segmentation should be necessary to determine the quality of the pre-training strategy. Therefore, I will rate this paper marginally below the acceptance but I will be happy to see additional results during the rebuttal phase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZkxefuMaWg", "reviewer": "Reviewer_wYee", "summary": "This paper proposes a self-supervised learning method from point cloud. Typically, this paper addresses the importance of using surface normal and surface variance instead of using point location as proposed by the previous studies. The idea is straightforward and easy-to-understand. The experiments demonstrate that the efficacy of the proposed method. Moreover, the ablation study consistently proves the addressed issue by the authors.", "questions": "Alongside with the addressed weakness, I have one minor question.\n\nQ-1. __Is there any reason that authors did not conduct experiments on the S3DIS dataset using 3D semantic segmentation?\nIf there are some reasonable and meaningful results, I can convince the efficacy of this work. Otherwise, this work could be understood as naive extension.__", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors address the importance of the geometric measurements for the usage of pre-training the network. Typically, using surface normal as surface variation are meaningful in point cloud based understanding. Typically, the authors provide the various experiments such as backbone architectures, loss designs, and downstream task evaluations. I really enjoyed reading this paper.", "weaknesses": "There are some minor things that need to be discussed\n\nW-1. Analysis of 2D/3D masked autoencoders.\n\nIn the manuscript, the authors commented that __\"These designs make an intrinsic difference from 2D MSMs, where there is no need to recover masked pixel locations.\"__\n\nIt is true. I understand the analysis by the authors. When we think of the vanilla MAE, it also takes a masked image as an input and predicts the color information, not its pixel location. However, when we think of the nature of the point cloud, it is sparse, irregular, and unordered. Even, I would say _raw point cloud_ naturally does not involve color information. Accordingly, it is not feasible to extend the concept of the MAE for the 2D image into the MAE for the 3D points. In my opinion, the authors should have written such a clear understanding of MAE for 3D points. \n\nW-2. Details in computing surface normal and surface variance on scene-level experiments.\nWhile the various object-level datasets, such as shapenet, are synthetically created, the real-world points are captured by the sensors. Due to such difference, raw point cloud from the real world naturally involves lots of noise, which could be an issue when computing surface normal using PCA. So I wonder how the authors solve this issue when conducting experiments on Sec. 4-4 in the manuscripts.\n\nW-3. Insightful analysis\nI truly agree that the proposed experiments demonstrate that the surface normal and surface variance are important measurements for self-supervising learning using 3D points. Technically, I also agree with such an observation. However, I wonder why such an approach brings performance improvement. Is there any geometric analysis? Based on the manuscript, this approach can be viewed as a naive extension of the Point-MAE that additionally uses other geometric measurements. \n\nI want to know the author's own analysis of such problem setup and insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cTJSF5NfY6", "reviewer": "Reviewer_cRKn", "summary": "The paper proposes a point cloud pre-training method to improve the downstream tasks’ performances. More specifically, instead of predicting point positions by a masked autoencoder, the authors propose to recover high-order features at masked points including surface normals and surface variations through a novel attention-based decoder. To verify the effectiveness of the method, various point cloud analysis tasks have been tested, and promising results have been achieved.", "questions": "Since the authors claim that it is essential to disregard point position recovery. Hence, how to understand that predicting point positions actually enhances the performances when using PointMAE in Table 4?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is interesting, and the results are promising.\n2. Extensive experiments are conducted with SOTA performances.\n3. The paper is clearly written and well-organized.", "weaknesses": "It seems that the ablation study shown in Table 4 failed to support the idea that it is essential to disregard point position recovery, since at the same time to predict point positions using PointMAE, the decoder architecture is also changed when using MaskFeat3D architecture. To make a fairer comparison, the same decoder architecture from MaskFeat3D should be used to predict point position as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TuUJMtTYsG", "reviewer": "Reviewer_UgQs", "summary": "In this paper, the authors apply the self-supervised pretraining paradigm of masked signal modeling to point cloud pretraining. They propose a novel approach called MaskFeat3D, which focuses on recovering high-order features of masked points rather than their locations. Additionally, they propose an encoder-agnostic attention-based decoder. The effectiveness of the proposed method is evaluated through experiments conducted on the ScanObjectNN dataset for shape classification and the ShapeNetPart dataset for shape part segmentation.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors present evidence that the recovery of high-order point features yields more effective results compared to the recovery of point positions for 3D masked signal modeling.\n- A novel encoder-agnostic attention-based decoder is proposed by the authors to accurately regress the high-order features of masked points.\n- The paper is well-written and provides clear explanations, making it easy to follow.", "weaknesses": "- It appears that this is not the first work in 3D masked signal modeling that focuses on recovering high-order features of masked points. For example, MaskSurfel (Zhang et al.) specifically aims to recover surface normals of points. This similarity with previous work may diminish the novelty of the paper.\n- The results on the ScanObjectNN dataset indicate that Point-MA2E outperforms MaskFeat3D significantly; however, this comparison is not included in the paper.\n- To assess the effectiveness of the decoder, it is recommended to include results obtained by combining the block features and masked queries, and feeding them into self-attention blocks of the same depth, similar to the MAE approach, with points used as the positional embeddings.\n\n\n[1] Zhang et al., Point-MA2E: Masked and Affine Transformed AutoEncoder for Self-supervised Point Cloud Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)