LCM: Locally Constrained Compact Point Cloud Model for Masked Point Modeling
Authors: Yaohua Zha
Naiqi Li
Yanzi Wang
Tao Dai
Hang Guo
Bin Chen
Zhi Wang
Zhihao Ouyang
Shu-Tao Xia
Venue: NeurIPS 2024 poster (2024)
Keywords: Point Cloud
Self-supervised Learning; Masking Point Modeling
Primary area: machine_vision
TL;DR: We propose a Locally constrained Compact point cloud Model (LCM) for masking point modeling to achieve an elegant balance between performance and efficiency.

Abstract: The pre-trained point cloud model based on Masked Point Modeling (MPM) has exhibited substantial improvements across various tasks. However, these models heavily rely on the Transformer, leading to quadratic complexity and limited decoder, hindering their practice application. To address this limitation, we first conduct a comprehensive analysis of existing Transformer-based MPM, emphasizing the idea that redundancy reduction is crucial for point cloud analysis. To this end, we propose a Locally constrained Compact point cloud Model (LCM) consisting of a locally constrained compact encoder and a locally constrained Mamba-based decoder. Our encoder replaces self-attention with our local aggregation layers to achieve an elegant balance between performance and efficiency. Considering the varying information density between masked and unmasked patches in the decoder inputs of MPM, we introduce a locally constrained Mamba-based decoder. This decoder ensures linear complexity while maximizing the perception of point cloud geometry information from unmasked patches with higher information density. Extensive experimental results show that our compact model significantly surpasses existing Transformer-based models in both performance and efficiency, especially our LCM-based Point-MAE model, compared to the Transformer-based model, achieved an improvement of 1.84%, 0.67%, and 0.60% in performance on the three variants of ScanObjectNN while reducing parameters by 88% and computation by 73%. The code is available at https://github.com/zyh16143998882/LCM.

LCM: Locally Constrained Compact Point Cloud Model for Masked Point Modeling
Yaohua Zha 1.
Naiqi Li
Yanzi Wang'
Tao Dai
Hang Guo
Bin Chen'
Zhi Wang'
Zhihao Ouyang² Shu-Tao Xia?
TTinghua Shenzhen International Graduate School, Tsinghua University Institute of Visual Intelligence. Pengcheng Laboratory
CColleee of Computer Science and Software Engineering, Shenzhen University *Harbin Institute of Technology, Shenzhen Bytedannc Inc.
2yh161144998894mma1..comm
Abstract
The pre-trained point cloud model based on Masked Point Modeling (MPM) has ex- hibited substantial improvements across various tasks. However, these models heav- ily rely the Transformer, leading quadratic complexity and limited decoder, hindering their practice application. To address this limitation, we first conduct a comprehensive analysis of existing Transformer-baseee MPM, emphasizing the idea that redundancy reduction is crucial for point cloud analysis. To this end, we pro- pose Locally constrained Compact point cloud Model (LCM) consisting of lo- cally constrained compact encoder and a locally constrained Mamba-based decoder. Our encoder replaces self-attention with our local aggregation layers to achieve an elegant balance between performance and efficiency. Considering the varying information density between masked and unmasked patches in the decoder inputs of MPM, we introduce a locally constrained Mamba-based decoder. This decoder ensures linear complexity while maximizing the perception of point cloud geometry information from unmasked patches with higher information density. Extensive experimental results show that our compact model significantly surpasses existing Transformer-bot based models in both performance and efficiency, especially our LCM- based Point-MAE, model, compared to the Transformer-basee an improvement of 1 1.84%, 0.67%, and 0.60% in average accuracy on the three variants of ScanObjeNNNN while reducing parameters by 88 % and computation by 73%. Code is available at https: // //gi thub. com/ hoom/2yh1111339888///
model, achieved
Introduction
3D point cloud perception, as crucial application of deep learning, has achieved significant success across various areas such as autonomous driving, robotics, and virtual reality. Recently, point cloud self-supervised learning [1, 58, 60], capable of learning universal representations from extensive unlabeled point cloud data, has gained much attention. Among which, masked point modeling (MPM) [8, 37, 60, 62, 65, 66], as an important self-supervised paradigm, has become mainstream in point cloud analysis and has gained immense success across diverse point cloud tasks.
The classical MPM [37, 60, 65], inspired by masked image modeling [2, 22, 59] (MIM), divides point clouds into patches and uses standard Transformer [46] backbone. It randomly masks some patches in the encoder input and combines the unmasked patch tokens with randomly initialized
"Corresponiing author. X daitao.edus edu @ gmail.com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

masked patch tokens in the decoder input. It predicts the geometric coordinates or semantic features of the masked patches from the decoder output tokens, enabling the model to learn universal 3D representations. Despite the significant success, two inherent issues of Transformers still limit their practical deployment.
The first issue is that the Transformer architecture leads to quadratic complexity and huge model sizes. As shown in Figure (a) and (b), MPM methods like Point-MAE [37] based on standard Transformer [46] require 22. IM parameters and complexity exponentially grows with an increase in the length of input patches. However, in practical point cloud applications, models are often deployed on embedded devices such as robots or VR headsets, where strict constraints exist regarding the model's 's size and complexity. In this context, lightweight networks such as PointNet++ [40] are more popular in practical applications due to their lower parameter requirement (only 1.5M) even though they may have inferior performance.
Another issue is that when Transformers [46] are used as decoders in Masked Point Modeling (MPM), their potential to reconstruct masked patches with lower information density is lim- ited. In the decoder input of MPM, randomly initialized masked tokens with lower informa- tion density are typically concatenated with un- masked tokens with higher information density and fed into the Transformer- based decoder. The self-attention layers then learn to process these tokens of varying information density based on loss constraints. However, relying solely on the loss to learn this objective is challenging due to the lack of explicit importance guidance for
1600
Transtormer LCM]Ours]
0(N"))
1420
Overal accrooaa
acouncytii
MogsPoint
1200
LCM]Ours)
89.0
000
600
400
O(N)
10
25
1024
2048
16354
Paramcters/Mt
(a) Model Accuracy and Parameters
(b) Complexity growth curve.
Comparison of our LCM and Trans- former in terms of performance and efficiency. different densities. Additionally, in Section 5.1, we further explain from an information theory perspective that the self otlf-attention mechanism, as a higher-order processing function, can limit the model's 's reconstruction potential.
Figure 1:
To address the above issues, as shown in Figure 2, we first conducted a comprehensive analysis of the effects of different top-K K attention on the performance of the Transformer model, emphasizing the idea that redundancy reduction is crucial for point cloud analysis. To this end, we propose a Locally constrained Compact point cloud Model (LCM), consisting of a locally constrained compact encoder and a locally constrained Mamba-based decoder, to replace the standard Transformer. Specifically, based on the idea of redundancy reduction, our compact encoder replaces self-attention with our local aggregation layers to achieve an elegant balance between performance and efficiency. The local aggregation layer leverages static local geometric constraints to aggregate the most relevant information for each patch token. Since static local geometric constraints only need to be computed once at the beginning and are shared across all layers, it avoids dynamic attention computations in each layer, significantly reducing complexity. Furthermore, it uses only two MLPs for information mapping, greatly redueing the network's 's parameters.
In our decoder design, considering the varying information density between masked and unmasked patches in the inputs of MPM, our decoder introduces the State Space Model (SSM) from Mamba [13, 16, 19,30, 71] to replace self-attention, ensuring linear complexity while maximizing the perception of point cloud geometry information from unmasked patches with higher information density. However, as discussed in Section 5.4, the directly replaced SSM layer exhibits a strong dependence on the order of input patches. Inspired by our compact encoder, to the feedforward neural network of our Mamba- bascdd decoder, proposing the Local Constraints Feedforward Network (LCFFN). This eliminates the need to explicitly consider the sequence order of input in SSM layers because the subsequent LCFFN can adaptively exchange information among geometrically adjacent patches based on their implicit geometric order.
migrate the idea of local constraints
Our LCM a universal point cloud architecture designed based on the characteristics of the point cloud to replace the standard Transformer. It can be trained from scratch or integrated into any existing pretraining strategy to achieve an elegant balance between performance and efficiency. For example, the LCM model pre-trained based on the Point-MAF strategy requires only 2.7M parameters, which is about 10 x efficient compared to the original Transformer with 22, IM. Furthermore, in
2

---

93.6
92.6
89.5
93.4
92.5
89.0
93.2
93.0
88.5
92.8
88.0
92.6
92.2
Top-K
92.4
92.1
87.5
Geametry
92.2
92.0
128(AII)
87.0
10 20 (c) PB-T50-RS
128(AI))
(a) OBJ-BG
(b) OBJ-ONLY
Figure 2: The effect of using top-K attention in feature space and geometric space by the Transformer on the classification performance in ScanObjectNN, all results are the averages of ten repeated experiments.
terms of performance, compared to the Transformer, the LCM shows significant improvements of 1.84%, 0.67%, and 0.60% in average classification accuracy of three variants of ScanObjectNN [44]. Additionally, in the detection task of ScanNetV2 [6], there are also significant improvements of +5.2% Á AP and +6.0% AP50-
We summarize the contributions of our paper as follows: 1) We propose a locally constrained compact encoder, which leverages static local geometric constraints to aggregate the most relevant information for each patch token, achieving an clegant balance between performance and efficiency. 2) We propose locally constrained Mamba-based decoder for masked point modeling, which replaces the self-attention layer with Mamba's SSM layer and introduces locally constrained feedforward neural network to eliminate the explicit dependency of Mamba the input sequence order. 3) Our locally constrained compact encoder and locally constrained Mamba-based decoder together constitute the efficient backbone LCM for masked point modeling. We combine LCM with various pretraining strategies to pre-train efficient models and validate our model's superiority in efficiency and performance across various downstream tasks.
Related Work
Point Cloud Self-supervised Pre-training, Point cloud self-supervised pre-training [47, 54, 55, 58, 60] has achieved remarkable improvement in many point cloud tasks. This approach first applies a pretext task to learn the latent 3D representation and then transfers it to various downstream tasks. PointContrast [58] and CrossPoint [1] initially explored utilizing contrastive learning [36, 43] for learning 3D representations, which achieved some success; however, there were still some shortcomings in capturing fine-grained semantic representations. Recently, masked point modeling methods [37, 60-62] demonstrated significant improvements in learning fine-grained point cloud representations through masking and reconstruction. Many methods [4, 21, 41, 66] have attempted to leverage multimodal knowledge to assist MPM in learning more generalized representations, yielding significant improvements. After obtaining pre-trained point cloud model, many works [18, 64, 67, 70] remain to explore parameter-efficient fine-tuning methods to better adapt these pretrained models variety of downstream tasks. While the pre-trained models mentioned above have achieved tremendous success, they all rely on the Transformer architecture. In this paper, we focus on designing more efficient architecture to replace the Transformer these methods, significantly reducing computational and resource requirements.
Methodology
3.1 Observation of Top-K Attention
Standard Transformer [46] architecture requires computing the cor- relation between each patch with all input patches, resulting quadratic complexity. While this architecture performs well in lan- guage data, its effectiveness explored. Not all points are equally important. As illustrated in Fig- ure 3, the key points for aireraft recognition are mainly distributed on the wings, while for vase recognition, they are primarily located on the bottom the vase. Therefore, directly skipping the attention computation for less important points provides straightforward solution.
High
Importance
point cloud data has been under-
Low
Figure 3: Point heatmap.
3

---

Reconstruction
PC
Par
Compact Decader 2
Broser
On bodding
KNN
OFF
FP8
MASK
Cur
Compact
Encoder g
N4:00
N4560
Desader
broddding
Croadinn
Tm
Macced
Pv
Tikes
EEP
T2
Tokan
Figure 4: The pipeline of our Locally Constrained Compact Model (LCM) with Point-MAE pre- training. Our LCM consists of locally constrained compact encoder and locally constrained Mamba-based decoder.
We first replaced the computation of global attention for all patch tokens with calculations top-K attentions in both feature and geometric space. As shown in Figure 2, our empirical observations indicate that: 1) In self-attention, it is often more effective most important patch tokens rather than using all patch; 2) Compared to using top-K K attention in a dynamic feature space, employing top-K attention in static geometric space yields nearly identical representational capacity and offers the advantage of a smaller K value. Although this naive method of masking out unimportant attention still exhibits quadratic complexity, this redundancy reduction idea not only brings performance improvements but also provides direction for further optimizing computational efficiency.
attention weights based on the top-K
3.2 The Pipeline of Masking Point Modeling with LCM
The overall architecture of Locally constrained Compact Model (L.CM) is shown in Figure 4. The specific process is as follows.
Patching, Masking, and Embedding. Given an input point cloud PC E 4x3 with L points, initially downsample a central point cloud C E Nx3 with N points by farthest point sampling (FPS). Then, perform K-Nearest Neighborhood (KNN) around C to get point patches P R R Nx Kx3
Following this, we randomly mask a portion of C and P. resulting in masked elements CM E R(I-r)Nx33 and PM E R(1-r)NxXXX and unmasked elements Cy E R x3 and Pv E Rr where denotes the unmask ratio. Finally, use MLP-based embedding layer (Embed) and position encoding layer (PE) respectively to extract semantic tokens Eo e I xd and central position embedding Ep E 1 xd for the unmasked patches, where d is the feature dimension.
Encoder. We employ our locally constrained compact encoder T extract features from the unmasked features Eg. It consists of n stacked encoder layers, each layer incorporating a local aggregation layer and feedforward neural network, detailed in Figure 4. For the input feature E,-1 of the i-th layer, after adding its positional embedding EP, it feeds to the i-th encoding layer T, to obtain the feature E, Therefore, the forward process of each encoder layer defined as:
E, = TT + EP), i= 1,.,,,,,
(1)
Decoder. In the decoding phase, although various MPMs have different decoding strategies, they can generally be divided into feature-level or coordinate-level reconstruction, and their decoders mostly rely on the Transformer architecture. Here, we illustrate the decoding process of our locally constrained Mamba-bass based decoder using the coordinate-level reconstruction method Point-MAE [37] as an example.
We first concatenate unmasked tokens E,, Rr-xx before the randomly initialized masked tokens Q E (I-r)N xd to obtain the input To E N xd for the decoder. Then, separately calculate the positional encoding for unmasked patches TP E R xd and masked patches T.P. E N and then concatenate them together to obtain the positional embeddings TP E N shared by all layers of the decoder. Finally, for the input feature T₃- of the i-th decoder layer, after adding their positional embeddings TP, they passed into the i-th decoder layer D; to compute the output

---

Layer Nomaal
Laver Normal
Local MaxPool
Layer
Rec Cal
Down MLP
Up MuP
Layen
Mamba
Local
KNN
Local Aggregation Layer
(LAL)
LCFFN
(a) Structure of the 1-th Compact Encoder Layer.
(b) Structure of the i-h Mamba Decoder Layer.
The structure of i-th locally constrained compact encoder layer (a) and i-th locally constrained Mamba-based decoder layer (b).
Figure 5:
features T, Therefore, the forward process of each decoder layer is defined as:
T; (T TD),
L,... m7,
(2)
Reconstruction. We utilize the features R = Tm [rN decoded by the decoder to perform the 3D reconstruction. We employ multi-layer MLPs construct coordinates reconstruction head H. and our reconstruction target is to recover the relative coordinates RM H(R) of the masked patches. We use the l2 Chamfer Distance [9] (CD) reconstruction loss. Therefore, our loss function C is as follows
L CDD(RMM PM)
3.3 Locally Constrained Compact Encoder
The classical Transformer [46] relies on the self-attention mechanism to perceive long-range corre- lations among all patches globally and has achieved great success in language and image domains. However, there remains uncertainty about whether directly transferring a Transformer- encoder is suitable for point cloud data. Firstly, applications of point clouds more inclined towards practical embedded devices such as robots or VR headsets. The hardware resources these devices limited, imposing higher limits the model size and complexity, and the Transformer-based backbone demands significantly more resources than traditional networks, as illustrated in Table 1. Secondly, extensive research [34, 40, 50] and empirical observation illustrated in Figure also indicate that the perception of local geometry in point cloud data far outweighs the need for global perception. Therefore, the computation of long rangee correlations in self-attentin leads to considerable amount of redundant calculations. To address these practical issues, we propose locally constrained compact encoder.
Our compact encoder consists of n stacked compact encoder layers, each layer comprising local aggregation layer (LAL) and feed-forward network (FFN), shown in Figure (a). For the i-th encoder layer, the output (E;-1) of the preceding layer, added with the positional embedding and normalized by layer normal, initially fed the Local Aggregation Layer (LAL) for aggregating local geometric. Afterward, the result is added to the input residual, passed through layer normaliza- tion, and finally fed into Feed-forward Network (FFN) obtain the ultimate output feature (E₃). This process can be formalized as follows,
E, E, (1 C.)
E, = E, + (Ea) (E4))
where 4.() represents the LAM, (-) and n.(()) represents layer normalization, and f+(-) represents the FFN.
In the local aggregation layer, first the k-nearest neighbors algorithm based on the central coordinates C,, of the features E;- to find the nearest neighbors feature ET E R xd for each token in E;-1. We then replicate each token of E;- times and concatenate them with their corresponding neighbors to obtain E E R x2d Next, Down MLP performs a non-linear mapping on all local neighboring features to capture local geometric information. Subsequently, local max pooling is applied to aggregate all local features for each patch. Finally, Up MIP maps all patches to obtain locally enhanced features E; xd Our LAL consists of only two simple MLP layers,

---

significantly reducing the network's 's parameters. Additionally, since static local geometric constraints only need to be computed once the beginning and are shared across all layers thereafter, avoids dynamic attention computations in each layer, significantly reducing computational requirements. It uses only two MLPs for information mapping, greatly reducing the network's parameters.
3.4 Locally Constrained Mamba-based Decoder
The decoder for mask point modeling needs to recover information about masked patches based on the features E,, extracted from unmasked patches by the encoder. A common approach is to concatenate the features En of unmasked patches before randomly initialized features Q of masked patches as the input the decoder, as shown in Figure 4. However, this point, there is significant difference in information density between features En and Q The Transformer architecture and our local aggregation layer both treat each token in the input as equally important initially, works well when the information density of all tokens is similar. It does not adapt well to cases where there is large difference information density in the input.
To efficiently extract more geometric priors from unmasked features En, we were inspired by the Mamba [13] model in time sequence and proposed using Mamba-basee decoder. This decoder can extract more prior information from the preceding tokens in the sequence based on the input order to aid the learning of subsequent tokens. Initially, we simply replaced the self-attention layer in the original Transformer-basee [46] decoder with the state space model (SSM) layer from Mamba. We also sorted the input sequence based on the order of each patch's 's center point coordinates, creating a naive Mamba-based decoder. Our experiments in Section 8 revealed that although this naive decoder efficient enough, the simple sorting method cannot effectively model the complex spatial geometry of point clouds and leads to strong dependence on the order of input patches.
To ensure that the SSM fully perceives the spatial geometry of point clouds, we further introduced the concept of local constraints from the local aggregation layer into the feedforward neural network layer of our decoder, getting the Local Constraints Feedforward Network (LCFFN). By feeding the tokens outputted by the SSM layer into the LCFFN, the LCFFN can implicitly exchange information between geometrically adjacent patches based on their central coordinates. This eliminates the limitation in the SSM layer where explicit sequential input fails to perceive complex geometry fully. Finally, in Section 5. we also qualitatively explain from an information theory perspective that this Mamba-based architecture has greater reconstruction potential compared to the Transformer.
Our Mamba- hasea-based decoder consists of m stacked decoder layers, each layer comprising a Mamba SSM layer and a local constraints feedforward network (LCFFN), as shown in Figure 5 (b). For the i-th decoder layer, we first add the output (T; of the previous layer with the positional embeddings (TP) and normalize it through layer normalization. Then, we use the Mamba SSM layer (sx(-)) (-) to perceive geometry from unmasked features and predict masked features. Finally, in the LCFFN (.),,, we further perceive shape priors based on the central coordinates of each token from its geometrically adjacent tokens. This process can be formalized as follows:
T. T + (n)
T: = T. + (n,
4 Experiments
4.1 Pre-training
We pre-training our L.CM using five different pretraining strategies: Point-BERT [60], MaskPoint [28], Point-MAE [37], Point-M2AE [65], and ACT [8]. For a fire comparison, we use ShapeNet [3] as
pre-training dataset, encompassing over 50,000 distinct 3D models spanning 55 prevalent object categories. For the hyperparameter settings during the pretraining phase, we used the same settings as previous methods.
4.2 Fine-tuning Downstream Tasks
We assess the performance of our LCM by fine-tuning our models on various downstream tasks, including object classification, scene-level detection, and part segmentation.

---

Table 1: Classification accuracy real-scanned point clouds (ScanObjectNN)) We report the overall accuracy (%) on three variants. "#Params" represents the model's parameters and FLOPs refer to the model's floating point operations. GPT, CL, and MPM respectively refer to pre-trale pre-training strategies based on autoregression, contrastive learning, and masked point modeling. o is the reported results from the original paper. is the result reproduced in downstream settings.
SeanObjectNN
Method
Pretrain Params(M)
(LOPs(G)
OBI-BG
OBI-ONLY
PB-T50-RS
Supervised Learning Only
PointNe [39]
3.5
0.5
73.3
79.2
68.0
PointNett+ [40]
82.3
84.3
PointML.P [32]
12.6
31.4
85.2
Transformer [46]
X
22.1
4.8
86.75
86.92
80.78
PointMarbaa [27]
X
12.3
88.30
87.78
82.48
SFR [63]
87.80
Transformer [46]
22.1
91.95
91.39
86.65
..CM (Ours)
2.7 88%)
1.3(L
92.77(1 0.82)
91.54(1 0.15)
87.75(1 1.10)
Self- Supervised Learning
Point-BERT [60]
MPM
22.1
4.5
87,4 43
88.12
83.07
MaskPoint [28]
MPM
22.1
89.30
88.10
84.30
Point-MAE [37]
MPM
22.1
90.02
88.29
85.18
Point-MAE w/ IDPT [64]
MPM
23.3
7.1
91.22
90.02
84.94
Point-MAR w/ DAPT [70]
MPM
22.7
5.0
90.88
90.19
85.08
Inter-MAE [29]
MPM
22.1
4.8
88.70
89.60
85.40
Point-M2AE [65]
MPM
12.9
7.9
91.22
83.81
86.43
ACT
MPM
22.1
93.29
88.21
PoinIGPT-B [5]
GPT
120.5
36.2
93.60
92.50
89.60
PointMarnba [27]
MPM
12.3
93.29
91.91
88. 17
Point-BERT [60]
MPM
22.1
4.5
92.48
91.60
87.91
MaskPoint [28]
MPM
22.1
92.17
91.69
87.65
Point-MAE [37]
MPM
22.1
92.67
Point-M2AE [65]
MPM
12.9
7.9
93.12
91.22
88.06
ACT [8]
MPM
22.1
4.8
92.08
91.70
87.52
Point-BERT w/ LCM
MPM
3.1 80%)
2.5 (L. 44%)
93.55 (1 1.07)
92.43 (1 0.83)
88.57 (1 0.66)
MaskPoint LCM
MPM
(1 (168
2.5 444%)
93.31 1.14)
91.98 (1 0.29)
87.75 (f 0.
Point-MAF LCM
MPM
88%)
73%%
94.51 1.84)
(1 0.67)
88.87 (1 0.60)
Point-M2AE LCM
MPM
2,5 81%)
(L 15%)
93.83 0.71)
92.41 (1
88.38 (1 0.32)
ACT w/ LCM
MPM 3.1 (4.8))
2.8 2. 42%%
94.13 2.05)
92.66 (1 0.96)
88.57 (1 1.05)
4.2.1 Object Classification
We initially assess the overall classification accuracy of our pre-trained models on both real-scanned (ScanObjectNN [44]) and synthetic (ModelNet40 [57]) datasets. ScanObjectNN is prevalent dataset consisting of approximately 15,000 real-world scanned point cloud samples from 15 categories. These objects represent indoor scenes and often characterized by cluttered backgrounds and occlusions caused by other objects. For the ScanObjectNN dataset, we sample 2048 points for each instance and report results without voting mechanisms. We applied simple scaling and rotation data augmentation previous work [8, 37] in the downstream setting of ScanObjectNN. We reported the results of different models under downstream setting, with marking the results. For the ModelNet40 dataset, due to space limitation, we will further analyze its results in Section 5.4.
To ensure fair comparison, we conducted our experiments following the standard practices in the field (as used in previous work [8, 28, 37, 60, 65]). For each point cloud classification experiment, we used eight different random seeds (0-7) to ensure the robustness and reliability of our results. The performance reported Table represents the average accuracy achieved across these eight trials for each model configuration.
As presented in Table our model has many exciting results. 1) Lighter, faster, and more powerful. When trained from scratch using supervised learning only, our LCM model demonstrates performance improvements 0.82%, 15%, and 1.10% across three variant datasets compared the Transformer architecture. Similarly, after pre-training (e.g., Point-MAE), our model outperformed the standard Transformer by 1.84%, 0.67%, and 0.60% across the three variants of ScanObjectNN dataset. Notably, these improvements are achieved despite 88% reduction in parameters and 73 % reduction FLOPS. This improvement exciting as indicates that our architecture is better suited for point cloud data compared to the standard Transformer. Additionally, due to its extremely high
7

---

Table 2: Object detection results on ScanNetV2. We adopt the average precision with 3D IoU thresholds of 0.25 (2 and 0.5 (AP)n) for the evaluation metries. is our reproduction results, due to the lack of detection code in their paper.
Table 3: Part segmentation results on the ShapeNetPart. The mean IoU across all cate- gories, i.e., mloU, (%), and the mean IoU across all instances, ie., mloU, I (%) are reported.
Methods
Pretrain mloU.
mloU,
AF3s
AP5o
Supervised Learning Only
Superiized Leaminng Only
PointNet++ [40]
x
85.1
DGCNN [50]
x 82.3
85.2
VoteNet [38]
$8.6
33,5
Transformer [46]
83,9
86.0
3DETR [31]dasel)
62.1
37.9
(Ours)
84.6 0.7)
86.3 (.3)
Transformer [46]
LCM (Ours)
x
63.8 (1 3.3)
46.4 (1 5.8)
Self- Seperiiped Learning
Self- Self-Superviied Leording
Transfomme-OCee [48]
CL
83.4
85.1
PointContrad [58]
CL
85,1
PointContrast [58]
CL
58.5
38.0
CrossPoins [1]
CL
85.5
[25]
Point-BEKT [60]
MPM
84,1
85.6
Point-BEE Boiit-BERTTBA [60]
MPM
61.0
38.3
IDPT [64]
MPM
83.8
85.9
PIMAE [4]
MPM
62.6
39.4
MaskPoint [28]
MPM
84.4
86.0
Point-MAR [37]
MPM
59.5
41.2
Point-MAIB [37]
MPM
84.2
86.1
Point-M2AB [65]
MPM
84.7
ACT [8]
MPM
63.8
42.1
PointGPT-S [5]
MPM
84.1
86.2
DeptiContrast [69]
CL
64.0
42.9
PointGPTTB [S]
MPM
84.5
86.4
MaskPoint [28]
42.1
[65]
MPM
84,9
86.5
Poist-BERT [60] w/ LCM
MPM
65.3 (1 4.3)
47.3 9.D)
Point-BERT [60] w/ LCM
MPM
85.0 0.9)
86.5 (1 0.9)
Point-MAB [37] LCM
47.2 (..))
MaskPoint [28] LCM
MPM
85,1
86.6 0.6)
Point-M2AE [65] w/ W/MM
MPM
63.5 3.5)
47 2.6)
Point-MAR [37] w/ L.C
MPM
85,1 0.9)
86.6 (1
ACT w/] LCM
MPM
65.0 1.2)
45.8 3.7)
Point-M2AN [65] w/ W/LCM
MPM
85.0 (1 0.1)
86.5 (-)
MaskPoint [28]
65.3 1.1)
ACT [8] w/LCM
MPM
85.0
86.7
efficiency, it provides strong support for the practical deployment of these pre-trained models. 2) Universal. We have replaced the original Transformer architecture with our LCM model in five different MPM-based pre- pre-training methods. All experimental results are exciting our model achieved universal performance improvements with fewer parameters and computations, highlighting the versatility of our model. In the future, will further adapt to additional pre-training methods.
4.2.2 Object Detection
We further assess the object detection performance of our pre-trained model on the more challeng- ing scene-level point cloud dataset, ScanNetV2 [6], to evaluate our model's s scene understanding capabilities. Following the previous pre-training work [8, 28], we use 3DETR [34] the base- line and only replace the Transformer-based encoder of 3DETR with our pre-trained compact encoder. Subsequently, the entire model is fine-tuned for object detection. In contrast to previous approaches [4, 8, 28], which necessitate pre-train on large-scale scene-level point clouds like Scan- Net, our approach directly utilizes models pre-trained on ShapeNet. This further emphasizes the generalizability of our pre-trained models.
Table 2 showcases our experimental results, our compact model has shown significant improvements in scene-level point cloud data, such as Point-MAE [37] achieving a 5.2% improvement in 2P and 6.0% improvement in A.to compared to the Transformer. This improvement is remarkable. and we believe this is primarily due to the presence a large number background and noise points in the scene-level point cloud. Using local constraint modeling approach effectively filters out unimportant background and noise, allowing the model to focus more on meaningful points.
4.2.3 Part Segmentation
We also assess the performance of LCM in part segmentation using the ShapeNetPart dataset [3]. comprising 16,881 samples across 16 categories. We utilize the same segmentation setting after the pre-trained encoder as in previous works [37, 65] for fair comparison. As shown in Table 3, our LCM-based model also exhibits a clear boost compared to Transformer-based models. These results demonstrate that our model exhibits superior performance in tasks such as part segmentation, which demands more fine-gre fine-grained understanding of point clouds.
8

---

Table 5: Effects of Locally Constrained
Mamba-bas based Decoder.
Table 4: Effects of the Network Structure of the Locally Constrained Compact Encoder.
Decoder
ScanObjectNN
Transformer
88.76
Local
Local
MLPs
FFN
Param(M)
ScanObjectNN
LAL
88.38
A
x
1.4
85.45
Mamba
88.62
v
V
85.74
C
x
1.8
87.77
Transformer w/ LCFFN
88.79
D
2.7
88.06
LAL w/ LCFFN
88.51
Mamba w/ LCFEN
89,35
4.3 Ablation Study
Effects of Locally Constrained Compact Encoder. We explore the performance of locally constrained compact encoder by comparing it with a Transformer-based encoder in classification, detection, and part segmentation. The results from Tables 1, Table and Table 3, obtained solely through supervised learning from scratch, clearly demonstrate the advantages of our LCM encoder over the Transformer-based encoder in terms of performance and efficiency, particularly in detection tasks, with an improvement of up to 6.0% in the Al metric.
This substantial improvement is attributed the compact encoder's focused attention the most crucial information for each point patch, such as local neighborhoods while disregarding unimportant details. This is similar to sedundancy-reduciio compression concept, which is crucial for point cloud analysis, especially in large-scale scene-level point clouds where significant redundancy and noise points often exist. Our local constraint approach enables the model focus critical areas, leading to a combined improvement in efficiency and performance. Moreover, this redundancy-recuucng concept helps our model avoid overfitting the training dataset. We provide detailed explanations of this phenomenon in the Section 5.5.
Effects of the Network Structure of the Locally Constrained Encoder. As shown in Figure 5(a), each layer of our locally constrained compact encoder consists primarily of three parts: a locally constrained unit based on k-NN, MLPs mapping unit composed of Down MLP and Up MLP, and the final FFN layer. We explore the effects of each unit separately. Specifically, we train Encoders with different structures from scratch the ScanObjectNN [44] dataset and test their classification performance. As shown in Table 4, comparing A and B reveals that simple two-layer MLP without local aggregation does substantially improve the network's performance. In contrast, the results of C and D compared to A and B demonstrate a significant performance improvement. This improvement mainly attributed to the introduction of local geometrie perception and aggregation. Comparing the results of C and D, the introduction of FFN brings slight improvement. Therefore, FFN is not indispensable in our compact encoder, but we choose incorporate FFN further perform mapping. These experiments further indicate the necessity of local geometric perception and aggregation for point cloud feature extraction.
Effects of Locally Constrained Mamba-based Decoder. We further compared the impact of different decoder designs during the pre-training phase. Specifically, we compared the vanilla Transformer-bast decoder, our LAL--based decoder, and the vanilla Mamba-based architecture, as well as their performance after incorporating LCFFN, As shown in Table the results indicate that the Vanilla Transformer slightly outperforms the Vanilla Mamba in terms of performance, likely due the limitation imposed by the simple geometric sequential input sequences on the Vanilla Mamba's 's capabilities. After incorporating LCFFN, the Mamba decoder exhibits a significant improvement due the introduction of implicit geometric order. In contrast, the Transformer's improvement slight because the geometric order is already implicitly captured by self self-attention.
4.4 Limitation
Our current model does have limitations in handling dynamic importance perception and long-range dependency modeling. Our design prioritizes efficiency, which can at odds with the increased complexity required for capturing dynamic importance and long-range dependencies. This focus on efficiency led us to simplify the model certain aspects, and as a result, we did not fully integrate mechanisms for dynamic importance perception and long-range dependency modeling in this version

---

of our model. Despite these constraints, the current model has demonstrated significant improvements in performance across various tasks. Nevertheless, we also acknowledge that incorporating dynamic importance perception and long-range dependency modeling could further enhance the model's s capabilities, particularly in more complex scenarios. We are actively exploring methods to address these limitations in future work.
4.5
Conclusion
In this paper, we propose a compact point cloud model, LCM, specifically designed for masked point modeling pre-training, aiming to achieve an elegant balance between performance and efficiency. Based on the idea of redundancy reduction, we propose focusing on the most relevant point patches ignoring unimportant parts in the encoder, and introducing local aggregation layer to replace the vanilla self aeff-attetiio. Considering the varying information density between masked and unmasked patches in the decoder inputs of MPM, we introduce a locally constrained Mamba-base decoder to ensure linear complexity while maximizing the perception of point cloud geometry information from unmasked patches. By conducting extensive experiments across various tasks such classification and detection, we demonstrate that our LCM is a universal model with significant improvements in efficiency and performance compared to traditional Transformer models.
4.6 Acknowledgements
This work is supported in part by the National Key Research and Development Project of China (Grant No. 2023YFF0905002,, the National Natural Science Foundation of China, under Grant (62302309, 62171248), Shenzhen Science and Technology Program (JCY12020081810101) JCY120220118010010033 and the PCNL KEY project (PCL2023AS6-1),
References
[1] Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Rodrigo. Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9902-9912 2022. 3,
[2] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image trans-
formers. arXiv preprin! arXiv:2106. 21611082.4. 2021. 1
[3] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: / 030012, 2015. 6, 8,
[4] Anthony Chen, Kevin Zhang, Renrui Zhang, Zihan Wang. Yuheng Lu, Yandong Guo, and Shanghang Zhang. Pimae: Point cloud and image interactive masked autoencoders for 3d object detection. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern
Recognition (CVPR). pages 5291-5301, Vancouver, Canada, Jun 18-22 2023. 3, 8
[5] Guangyan Chen, Meiling Wang, Yi Yang, Kai Yu, Li Yuan, and Yufeng Yue. Pointgpt: Auto- regressively generative pre-training from point clouds. volume 36, 2024. 7, 8, 18
[6] Angela Dai, Angel X Chang. Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias NieBner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5828-5839 2017. 3, 8, 17,20 20
[7] Tao Dai, Beiliang Wu, Peiyuan Liu, Naiqi Li, Jigang Bao, Yong Jiang, and Shu-Tao Xia. Periodicity decoupling framework for long-term series forecasting. In The Twelfth International
Conference on Learning Representations, 2024. 16
[8] Runpei Dong, Zekun Qi, Linfeng Zhang, Junbo Zhang, Jianjian Sun, Zheng Ge, Li Yi, and Kaisheng Ma. Autoencoders as cross-modal teachers: Can pretrained 2d image transformers help 3d representation learning? Kigali, Rwanda, May 1-5 2023. 1, 3, 6, 7, 8, 17, 18 [9] Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 605-613, Honolulu, Hawaii, USA, July 21-26
2017. 5
[10] Hao Fang, Bin Chen, Xuan Wang, Zhi Wang, and Shu-Tao Xia. Gifd: A generative gradi- ent inversion method with feature domain optimization. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 4967-4976, 2023. 16
10

---

[11] Hao Fang, Jiawei Kong, Wenbo Yu, Bin Chen, Jiawei Li, Shutao Xia, and Ke Xu. One pertur bation enough: On generating universal adversarial perturbations against vision- vision-language pre-training models. arXiv preprint arXiv:2406. 05491, 2024. 16
[12] Hao Fang, Yixiang Qiu, Hongyao Yu, Wenbo Yu, Jiawei Kong, Baoli Chong, Bin Chen, Xuan Wang, and Shu-Tao Xia. Privacy leakage on dnns: A survey of model inversion attacks and
defenses. arXiv preprint arXiv:2402. 04013, 2024. 16
[13] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.
arXiv preprin! arXiv:23 22.22/22227722.. 2023. 2. 6, 15, 16
[14] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory
with optimal polynomial projections. Advances in neural information processing systems,
33:1474-1487, 2020. 16
[15] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. Advances in Neural Information Processing Systems, 35:35971-359833 2022.
[16] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00330 2021. 2, 15, 16
[17] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers.
Advances in neural information processing systems. 34:572-588 2021. 16
[18] Hang Guo, Tao Dai, Yuanchao Bai, Bin Chen, Shu-Tao Xia, and Zexuan Zhu. Adaptir: Parameter efficient multi-task adaptation for pre-trained image restoration models. arXiv preprint 23 12.8881,, 2023. 3
[19] Hang Guo, Jinmin Li, Tao Dai, Zhihao Ouyang, Xudong Ren, and Shu-Tao Xia. Mambair: A simple baseline for image restoration with state-space model. arXiv preprint arXiv:2402.1155
2024.7 2
[20] Meng-Hao Guo, Jun-Xiong Cai, Zheng- -ing Liu, Tai-Jians Mu, Ralph R Martin, and Shi-Min
Hu. Pet: Point cloud transformer. Computational Visual Media, 7()::187-199, 2021. 16
[21] Ziyu Guo. Renrui Zhang, Longtian Qiu. Xianzhi Li, and Pheng Ann Heng. mae: 2d-3d joint masked autoencoders for 3d point cloud pre-training In Proceedings of International Joint Conference on Artificial Intelligence (IJCAI), Macao, China, August 19-25 2023. 3
[22] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000-16009, 2022.
[23] Xinwei He, Silin Cheng. Dingkang Liang, Song Bai, Xi Wang, and Yingying Zhu. Latformer: Locality-aware point-view fusion transformer for 3d shape recognition. Pattern Recognition,
151:1 10413, 2024. 16
[24] David Hilbert and David Hilbert. Über die stetige abbildung einer linie auf ein flächenstück. Dritter Band: Analysis- Grundlagen der Mathematik- Physik Verschiedenes: Nebst Einer Lebensgeschichte, pages 1-2, 1935. 19
[25] Siyuan Huang, Yichen Xie, Song-Chun Zhu, and Yixin Zhu. Spatio-temporal self-supervised representation learning for 3d point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6535-6545, 2021. 8
[26] Md Mohaiminul Islam, Mahmudul Hasan, Kishan Shamsundar Athrey, Tony Braskich, and Gedas Bertasius. Efficient movie scene detection using state-space transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18749-18758, 2023. 16
[27] Dingkang Liang, Xin Zhou, Xinyu Wang, Xingkui Zhu, Wei Xu, Zhikang Zou, Xiaoqing Ye, and Xiang Bai. Pointmamba: A simple state space model for point cloud analysis. arXiv preprint 2rXiv:2402.1077:
2024. 7, 16, 18
[28] Haotian Liu, Mu Cai, and Yong Jae Lee. Masked discrimination for self superviedd learning on point clouds. In Proceedings of the European Conference on Computer Vision (ECCV), pages
657-675, Tel Aviv, Israel, October 23-27 2022. 6, 7, 8, 17, 18
[29] Jiaming Liu, Yue Wu, Maoguo Gong, Zhixiao Liu. Qiguang Miao, and Wenping Ma. Inter- modal masked autoencoder for self-supervised learning on point clouds. IEEE Transactions on Mudtimedia, 2023. 7, 18
[30] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang. Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401. /00666 2024. 2, 16 [31] Jun Ma, Feifei Li. and Bo Wang. U-mamba: Enhancing long-range dependency for biomedical image segmentation. arXiv preprint
240.::2000044225 2024. 16
[32] Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local geometry in point cloud: A simple residual mlp framework. In Proceedings of International
11

---

Conference on Learning Representations (ICLR), page 31, Online, Apr. 25-29 2022. 7, 16, 18 [33] Harsh Mehta, Ankit Gupta. Ashok Cutkosky, and Behnam Neyshabur. Long range language modeling via gated state spaces. arXiv preprint arXiv: 2t0.:2200.11947,,, 13947, 2022. 16
[34] Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-ent transformer model for 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision,
pages 2906-2917, 2021. 5, 8, 16, 17
[35] Eric Nguyen, Karan Goel, Albert Gu. Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. S4nd: Modeling images and videos as multidimensional signals with state
spaces. Advances in neural information processing systems, 35:2846- 2861, 2022. 16
[36] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive
predictive coding, arXiv preprint arXiv: wew.:0077.000 003748 2018. 3
[37] Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud sele-vuper learning. In Proceedings of the European Confer- ence on Computer Vision (ECCV), Tel Aviv, Isracl, October 23-27 2022. 1, 2, 3, 4, 6, 7, 8, 16,
17, 18
[38] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In proceedings the IEEE/CVE International Conference on Computer Vision. pages 9277-9286, 2019. 8
[39] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 652 660, Honolulu, HI, USA, July
21-26 2017. 7, 16, 17, 18
[40] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Proceedings of Advances in Neural Informa- tion Processing Systems (NeurIPS), page 30, Long Beach, CA. USA, Dec. 4-9 2017. 2, 5, 7, 8,
16, 18
[41] Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining. In International Conference on Machine Learning, 2023. 3
[42] Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208. 04933, 2022. 16
[43] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV,
pages 776-794. Springer, 2020. 3
[44] Mikaela Angelina Uy, Quang-Hics Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), pages 1588-1597, Seoul, Korea, Oct 27- Nov 2 2019. 3, 7, 9, 17, 20 [45] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911). 2008 20
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), page 30, Long Beach, CA, USA, Dec. 4-9
2017. 1. 2, 5, 6, 7, 8, 16, 20
[47] Chengyao Wang, Li Jiang, Xiaoyang Wu, Zhuotao Tian, Bohao Peng, Hengshuang Zhao, and Jiaya Jia. Groupcontrast: Semantic-aware self-supervised representation learning for 3d understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4917-4928, 2024. 3
[48] Hanchen Wang. Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt J Kusner. Unsupervised point cloud pre-training via occlusion completion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9782-9792, 2021.
[49] Jue Wang. Wentao Zhu, Pichao Wang. Xiang Yu, Linda Liu, Mohamed Omar, and Ruffay Hamid. Selective structured state-spaces for long-form video understanding. In Proceedings of the IEEFCVE Conference on Computer Vision and Pattern Recognition, pages 6387-6397, 2023. 16
[50] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics
(TOG), 38(5):1-12, 2019. 5, 8. 16, 18
[51] Ziyi Wang, Xumin Yu, Yongming Rao, Jie Zhou, and Jiwen Lu. P2p: Tuning pre-trained image models for point cloud analysis with point-to-pixel prompting. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), New Orleans, Louisiana, USA, December
1-9 2022. 18
12

---

[52] Xiaoyang WU, Li Jiang, Peng- Song-Shuii Wang, Zhijian Liu, Xihui Liu, Yu Qiao. Wanli Ouyang. Tong He, and Hengshuang Zhao. Point transformer v3: Simpler, faster, stronger. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), volume 35, Seattle, USA, Jun 17-21 2024. 16
[53] Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, and Hengshuang Zhao. Point transformer v2: Grouped vector attention and partition-based pooling. In Proceedings of Advances in Neural Information Processing Systems (NeurlPS), volume 35, pages 33330-33342, New Orleans, Louisiana. USA. November 28- December 9 2022. 16
[54] Xiaoyang Wu, Zhuotao Tian, Xin Wen, Bohao Peng. Xihui Liu, Kaicheng Yu, and Hengshuang Zhao. Towards surge-scale 3d representation learning with multi-dataset point prompt training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19551-19562, 2024. 3
[55] Xiaoyang Wu, Xin Wen, Xihui Liu, and Hengshuang Zhao. Masked scene contrast: A scalable framework for unsupervised 3d representation learning. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 9415-9424, 2023. 3
[56] Yue Wu, Jiaming Liu, Maoguo Gong, Peiran Gong, Xiaolong Fan, A Kai Qin, Qiguang Miao, and Wenping Ma. Self-supervised intra-modas and cross-modal contrastive learning for point cloud understanding. IEEE Transactions on Multimedia, 26: 1628, 2023.
[57] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition, pages 1912-1920, 2015. 7,
17.20
[58] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In European conference on computer vision, pages 574-591. Springer, 2020. 3,
[59] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han
Hu. Simmim: A simple framework for masked image modeling. In CVPR, pages 9653-9663,
2022. 1
[60] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19313-
19322, New Orleans, Louisiana, USA, June 21-24 2022. 1, 6, 7, 8, 16, 17, 18
[61] Yaohua Zha, Tao Dai, Yanzi Wang, Hang Guo, Taolin Zhang, Zhihao Ouyang, Chunlin Fan, Bin Chen, Ke Chen, and Shu-Tao Xia. Block-to-scene pre-training for point cloud hybrid-domain masked autoencoders. arXiv preprint arXiv:2410. 09886, 2024.
[62] Yaohua Zha, Huizhen Ji, Jinmin Li, Rongsheng Li. Tao Dai, Bin Chen, Zhi Wang, and Shu-Tao Xia. Towards compact 3d representations via point feature enhancement masked autoencoders. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), VANCOUVER. CANADA, February 20-27 2024. 1, 16
[63] Yaohua Zha, Rongsheng Li, Tao Dai, Jianyu Xiong, Xin Wang, and Shu- Tao Xia. Sfr: Semantic- aware feature rendering of point cloud. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023. 7
[64] Yaohua Zha, Jinpeng Wang, Tao Dai. Bin Chen, Zhi Wang. and Shu-Tao Xia. Instance-aware dynamic prompt tuning for pre-trained point cloud models. In Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), pages 14161-14170, Paris, France,
October 4-6 2023. 3, 7
[65] Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li. Point-m2ae: Multi-scale masked autoencoders for hierarchical point cloud pre-training. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS),
New Orleans, Louisiana, USA, November 28- December 9 2022. 1, 6. 7, 8, 17, 18
[66] Renrui Zhang, Liuhui Wang. Yu Qiao, Peng Gao, and Hongsheng Li. Learning 3d represen- tations from 2d pre-trained models via image-to-pon of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
masked autoencoders. In Proceedings
21769-21780, Vancouver, Canada, Jun 18-22 22 2023. 1.3
[67] Taolin Zhang, Jiawang Bai, Zhihe Lu, Dongze Lian, Genping Wang, Xinchao Wang, and Shu- Tao Xia. cerameter-efficiee and memory- efficient tuning for vision transformer: A disentangled approach. arXiv preprint arXiv:2407. 06964, 2024.
[68] Taolin Zhang, Sunan He, Tao Dai, Zhi Wang, Bin Chen, and Shu-Tao Xia. Dision-language pre-training with object contrastive learning for 3d scene understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7296-7304, 2024. 16
13

---

[69] Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of 3d features on any point-cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10252-10263, October 2021. 8
[70] Xin Zhou, Dingkang Liang, Wei Xu. Xingkui Zhu, Yihan Xu, Zhikang Zou, and Xiang Bai. Dynamic adapter meets prompt tuning: Parameter-efficien transfer learning for point cloud analysis. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14707-14717, 2024. 3,7
[71] Lianghui Zhu, Bencheng Liao, Qian Zhang. Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401 2024. 2, 16
14

---

Appendix
5.1 An Information Theoretic Perspective of Our Mamba-hased Decoder for MPM.
Here, we provide an information- theooreticc perspective for our de- coder design, using mutual information to qualitatively demon- strate that the Mamba-based SSM can perceive more information from unmasked patches to predict masked patches compared to Transformer-basee self-attention. The mutual information between random variables X and Y. ((XYY). measures the amount of in- formation that can be gained about a random variable X from the knowledge about the other random variable Y. Therefore, based on the decoder input's 's different information densities, we can sim- ply divide the input into X1. representing unmasked patches with higher information density. and X2. representing randomly initial- ized masked patches with lower information density. As illustrated in Figure 6, after being processed by the decoder, X1 and X₂ respec- tively yield outputs Y1 for unmasked patches and Y2 for masked patches. We reconstruct the masked points based on Y2.
X 1
X 2
Decoder Φ
Y1
1
Y2
Figure 6: simple illustration of information processing of MPM decoder.
Ideally, Y2 needs to perceive sufficient geometric priors from both X1 and X₂ recover the masked points, more mutual information represents more recovery potential. Therefore, we would like to maximize the mutual information [Y2;; Xx,, X2). In what follows, we demonstrate that the mutual information preserved by our proposed Mamba-based decoder is larger than that of the standard transformer decoder.
Theorem 1. Let Y,M Y and Y Y denote the oulputs of the Mamba-based and Transformer- based decoders respectively, I(Y,M, ;X,, X2) denote the mutual information preserved by the decoder, and I(Y2 X2) denote that of the Transformer-based decoder. We have I(Y, X2) 2 I(Y ;x1;;1,23 X2).
Proof. The first step is to formalize the input-output relation of the two decoding structures. For the Mamba decoder, as defined in [13, 16], the output can be expressed as:
Y CÁBX, + CBX22
-AX + BX2.
For the Transformer decoder, the attention mechanism can be expressed in the following matrix form:
X W W
X W, X- W. W
W.Xii] W.X2
Y
X2 W W
Y
Thus,
Y!" = x, W. W.XI W, + X2 W. WxX2 W,
Compared with the linear relation captured by Y,M, Y models higher-order interactions of the input variables. So for any given Mamba parameters A and B, there exists Transformer parameters WK, Wy, W. and a function g. such that Y M)
As Y is function of YgM, (2), + Y,M Y forms Markov chain. So (Xi, X2) and Y are independent when conditioned on Y.M. f.e.e, p(Y² , X2|Y,M p(Y² [Yg According to the definition of conditional mutual information, this implies
M )p (p(11, X, 17
I(X1, X2; 1 = 0.
On the other hand, by the chain rule of mutual information we have Y,M Y,M, y I(X1, X2; Y, X2 T(X1, X2;
=[; Yz) T(X,, X2; Y,,,7")
Since we already show that X2; | O, and mutual information is non-negative, we have T(X1, X2; T(x Yz) + (X1 X X x(x,,
15

---

5.2 Additional Related Work
Deep Network Architecture for Point Cloud. With the development of deep learning, various deep neural network- models [7, 10, 12, 13, 23, 46, 52, 53, 68] have become the mainstream approach for 3D point cloud analysis. PointNet [39], a pioneer in point cloud analysis, introduced an MLP-based network to address the disorder of point clouds. Subsequently, PointNet+++ [40] further proposed adaptive aggregation of multiscale features on MI.Ps and incorporated local point sets for effective feature learning. DGCNN [50] introduced the graph convolutional networks dynamically computing local graph neighboring nodes to extract geometric information. PointMLP [32] suggested efficient point cloud representation solely relying pure residual MLPs. Recently, many Transformer-basee
models [20, 34, 37, 62], benefiting from attention mechanisms, have achieved notable improvements in point cloud analysis. However, this led to a significant increase in model size, posing considerable challenges for practical applications. PointMamba [27] first attempted to introduce the Mamba architecture based on the state space model to point clouds, but it still has high complexity and parameters. In this paper, we focus on designing more efficient point cloud architectures specific to pre-training models.
State Space Models. State Space Models [14-17, 42] (SSMs) originate from classical control theory and have been introduced into deep learning as the backbone of state space transformations. They combine the parallel training capabilities of CNNs with the fast inference characteristics of RNNs, capturing longerange dependencies in sequences while maintaining linear complexity. The Structured State-Space Sequence model [16] (S4) is a pioneer work for the deep state-space model in modeling the long-range dependency. S5 [42] proposed based on S4 and introduces MIMO SSM and efficient parallel scan. GSS [33] leverages the gating structure in the gated attention unit to reduce the dimension of the state space module. Recently, Mamba [13] with efficient hardware design and selective state space, outperforms Transformers [46] in terms of performance and efficiency. Subsequent works [11, 26, 30, 31, 35, 49, 71] have attempted to introduce Mamba into the visual domain, achieving significant improvements. For example, Vision Mamba [71] and VMamba [30] directly apply Mamba to image processing and design corresponding scanning methods tailored for image data. As for point cloud, PointMamba [27] is the first to introduce Mamba into point cloud analysis, traversing the input sequences from the x. y. and z geometric directions. In this paper, we introduce Mamba into the decoder for masked point modeling and discuss its advantages from an information -hooortiin-thhooreiii perspective. Additionally, we propose a locally constrained feedforward neural network for Mamba block to adaptively exchange information among geometrically adjacent patches based on their implicit geometry.
5.3 Implementation Details
Top-K Attention Settings in Observation. In Figure 2, we replace the global attention computation of all patch tokens in Self-Attention with top-K attention computation in both feature space and geometric space to demonstrate the significant amount of redundant computation in the vanilla Transformer. Specifically, after computing all global attention, we further compute mask matrix. We then add negative infinity to the attention values that need to be masked. After that, we calculate the softmax, where the attention values that were set to negative infinity will become O, ensuring that the sum of the attention values of the unmasked top-K patches equals 1. We compute different top-K values in both feature space and geometric space, and pretrain the corresponding models. Subsequently, we fine-tune these pretrained models on the three variants of ScanObjectNN using the same top- K attention algorithm, evaluating their accuracy on classification tasks. To minimize error, we report the average accuracy over 10 repeated experiments.
Positional Encodings. To complement the 3D spatial information, we apply positional encodings to all encoder and decoder layers. As shown in Figure 4, we first use the Encoder Positional Encoding (EPE) to compute the positional encoding E² for Cv. which is then shared across all layers of the encoder. In the decoding stage, we use the Decoder Positional Encoding (DPE) to calculate the positional encoding for the unmasked Cv and the masked patches CM. These positional encodings are concatenated to form decoder positional encodings TP, which is shared across all layers of the decoder. Following previous work [37, 60], we utilize a twoelayer MLP to encode its corresponding 3D coordinates Cy E Nx3 or CM E (I x3 into d-channel vectors E E R xx or TP RRNxd and element-wisely add them with the token features before feeding into the attention layer.
16

---

Token Embedding. We follow the approach of previous works [37, 60] and use simple Point- Net [39] map the point patches Pv E RT from coordinate space to feature space E" E RRNxxd For Point-BERT [60], MaskPoint [28], Point-M2AF [65], and ACT [8], we use the exact same embedding structures as described in their original papers. For Point-MAR [37], we further simplify the embedding, using only a two-layer MLP with dimensions 3-128-384 as the embedding, which further reduces the parameters and computational complexity, as shown in Table 1.
Object Classification. Due to significant differences in the settings used for downstream fine-tuning tasks of point cloud classification on the ScanObjectNN [44] dataset in previous self-supervised learning methods [8, 28, 37, 60, 65]. such as input point quantity, data augmentation, and the input of the classification task head, we conducted extensive experiments to obtain a performance- friendly downstream fine-tuning setting. Furthermore, we re-evaluated most of the previous methods under our setting, while also conducting a fair comparison between our LCM model and the previous Transformer model under setting. We mark the results of our downstream fine-tuning setting with an in Table 1.
It can be observed that, compared to the results reported in the original paper, the fine-tuning results using our downstream settings have achieved significant performance improvements. For instance, Point-BERT has shown improvements of 5.34%. 3.62%, and 4.99% the three variants of the ScanObjectNN dataset, respectively. This improvement is surprising, indicating that there is further potential to be explored in earlier supervide learning methods such as Point-BERT, Point-MARE etc.
Experiments Compute Resources. Due to the surprisingly lightweight and efficient of our LCM model, we were able to complete the pre-training tasks using just a single 24GB NVIDIA GeForce RTX 3090 GPU. For downstream classification and segmentation tasks, we used a single RTX 3090 GPU for each. For detection tasks, to accelerate training, we utilized four parallel RTX 3090 GPUs.
3D Object Detection. We pre-train and fine-tune Point-MAE for 3D object detection both on ScanNetV2 [6]. In our detection experiments on Sean2erV2, we evaluate our model's understanding of scene-level tasks. Specifically, in the downstream detection fine-tuning experiments, we use 3DETR [34] as the baseline model and replace 3DETR's pre-encoder and encoder with our embed layer and compact encoder, respectively, while keeping all other training settings identical to 3DETR. Unlike many previous methods [8, 28, 65] that require retraining models on ScanNet, we initialize the embed layer and compact encoder with models pre-trained directly on ShapeNet [3]. While this may result in some loss of performance due to the gap between ShapeNet and ScanNet data, it demonstrates the universality of our pre-trained models.
5.4 Additional Experiments
Object Classification on ModelNet40. ModelNet40 [57] is a well-known synthetic point cloud dataset, comprising 12.311 meticulously crafted 3D CAD models distributed across 40 categories. Following previous work [37, 60, 65], for the ModelNet40 dataset, we sample 1024 points for each instance and report overall accuracy with voting mechanisms. In ModelNet40, we no longer differentiate between the results reported in the paper and our results, as we use the exact same downstream fine-tuning settings as previous methods [8, 28, 37, 65]. Table 6 presents our experimental results, and the overall conclusions are consistent with Section 4.2. Our LCM model outperforms the Transformer architecture in terms of both efficiency and performance, indicating the superiority of our model.
Effects of Locally Constrained K Value. We further explore the impact of using different numbers of neighbors K in local constraints on performance and efficiency. K=1 indicates no consideration of neighboring information. As K increases, the consideration of local geometry for each point patch also increases, but so does the computational complexity. We train object classification from scratch on the PB-RS-TSH T50 variant of ScanObjectNN, and Figure presents our ablation results. The area of the circle represents the computational floating -pinnt operations (FLOPs). We found that a smaller K, such as 5, is sufficient to achieve satisfactory results in terms of performance and efficiency. Performance initially increases slowly, but when K exceeds a certain threshold, it tends to decline. This is mainly due to larger K values introducing excessive redundancy, thereby limiting the learning capacity.
17

---

Table 6: Classification accuracy synthetic (ModelNet40) point clouds. In ModelNet40, following
previous work, we report the overall accuracy (%) with voting mechanisms. For a fair comparison, used the same downstream task settings in previous studies.
Method
MPrrams(M) FLOPs(G)
ModelNer40
Supervised Learning Only
PointNet [39]
3.5
0.5
89.2
PointNet++ [40]
1.5
1.7
90.7
DGCNN [50]
1.8
2,4
92.9
PointMLP [32]
31.4
94.5
P2P-HorNet [S1]
195.8
34.6
94.0
Transformer [27]
22.1
4.8
92.3
PointMamba [27]
12.3
92.4
Transformer [27]
22.1
92.3
LCM (Ours)
2.7
93.6
Self Supervised Learning
Point BiittBBRTTT [60]
22.1
CrossNet [56]
1.8
2.4
93.4
Inter- Inter-MAR [29]
22.1
2.3
93.6
MaskPoint [28]
22.1
93.8
Point-MAE [37]
22.1
Point-M2AR [65]
12.8
4.7
94.0
ACT [8]
22.1
2.4
93.7
PointGPT-S [5]
29.2
94.0
PointGPT-B
120.5
18.1
PointMamba [27]
12.3
93.6
Point-BERC w/ LCM
3.1
1.3
93.8
MaskPoint
3.1
1.3
94.1
PointM2AE w/ LCM
2.5
1.7
94.1
ACT w/ LCM
3.1
1.4
93.9
Point-MAE w/ LCM
2.7
0.6
94.2
Figure 7: Effects of locally constrained K value.
88.5
Table Effects of K-NN Space.
88.0
Sruurcy(ScanDajeeree
K
Feature K-NN
Geometry K-NN
1
85.70
85.81
87.0
5
87.65
88.06
10
87.51
88.17
20
87.20
88.20
86.0
Area denotes the FLOPs
3
10
20
40
60
80
Effects of K-NN Space. We further explored the impact of performing K-NN based Euclidean distance in both the feature space and the geometric space of our compact encoder. Geometric K-NN in the geometric space imposes explicit geometric constraints, serving static importance measure that greatly benefits point cloud analysis. Searching for K-NN based on feature Euclidean distance in the feature space be considered simple form of dynamic importance. We analyzed the effect of this approach on point cloud classification from scratch on ScanObjectNN, evaluating geometric K-NN and feature K-NN different values.
As shown Table we found that feature K-NN performed consistently lower than geometric K-NN in almost all cases. This result suggests that the naive idea of assigning dynamic importance to point patches based on Euclidean distance the feature space does not lead to substantial improvements. Efficient computation of dynamic importance for each point patch remains area for further exploration.
Effects of Patch Order and LCFEN for Mamba-based Decoder. The ordering of input patches significantly impacts our Mamba-based Decoder. more effectively illustrate this effect on Mamba's SSM model, analyze the issue from different perspective. Specifically,
Mamba-
18

---

83.5
78
83.0
70
Epoch Timests)
82.5
60
50
82.0
40
39
81.5
1x w/LCFFN
81.0
80.5
H
X
HXYZ XYZH YZHX ZHXY
1x
4x
1x W/LCFFN
Sequence Order (a) The Effect of Sequence Order Mamba Model
Sequence Length
(b) Times of Different Sequence Lengths
Figure 8: Training and testing curves different encoders trained from scratch. We present the training and testing curves for both the classification task on ScanObjectNN and the detection task on Sca All encoders were not pretrained.
based Decoder Encoder to directly extract features from the input point cloud and perform classification on ScanObjectNN, This substitution straightforward, as our Mamba-bases Decoder can also be viewed as Encoder.
We trained our Mamba-based encoder from scratch for the classification task on the PB-RS-TSS variant of ScanObjectNN without using any data augmentation strategies, and we took the average of ten repeated experiments as the final result. We first experimented with a naive Mamba-base decoder using a traditional FFN to illustrate the impact of different sequence orders on the original Mamba. selected four different patch ordering methods: sorting by the center point of the patch along x-axis (X), y-axis (Y), and z-axis (Z), and Hilbert curve [24] ordering (H), as shown by the orange curve in Figure 8 (a). Furthermore, we also conducted experiments with combinations sequences, combining these four orderings "H+X+Y+Z (HXYZ)", "X+Y+Z+H (XYZH)". "Y+Z+H+X (YZHX)", and "Z+H+X+Y (ZHXY)", as shown by the green curve in Figure 8 (a). Finally, based on the single- order sequence, used our proposed LCFFN to demonstrate the performance of Mamba with added implicit geometric constraints, as shown by the yellow curve in Figure 8 (a). The experimental results, illustrated in Figure 8, lead us the following conclusions:
The performance of the Mambu model is greatly influenced by the different orders of impul patches. The orange line represents the results for individual sequences, highlighting that different sequences have a significant impact on the final model performance. example, Y-order achieves highest classification accuracy at 82.34%, while the Hilbert order performs the worst at 80.65%, resulting in a difference of 1.69%.
2) The more combinations of sequences, the better the representation of point cloud geometry, result- ing improved performance, but also increased computational complexity. The green line represents the combinations sequences. While different combinations sequences do affect the final model performance, the impact is relatively minor. This indicates that the Mamba model can compensate for information across different sequences, allowing it to capture nearly complete geometric information each patch. Consequently, this significantly enhances the model's performance. However, this approach leads to significant increase in computational complexity due to the increase in the length of the input sequence, as shown in Figure (b). The processing time for the sequences of the four orders is approximately 3x longer than that of single order.
3) Introducing LCFFN allows for better perception of point cloud geometry through implicit local geometric constraints, thereby mitigating the dependence on sequence order. The yellow line represents the experimental results of using LCFFN to replace FFN for single-order input. It can be observed that the overall classification accuracy is significantly improved, surpassing the combinations sequence in the y-order and showing only slight differences from the combinations sequence in other orders. Moreover, in terms of runtime efficiency, as shown in Figure 8 (b), our single-order + LCFFN method exhibits considerable improvement compared to the combinations sequence, indicating the superiority of our design.
19

---

5.5 Additional Visualization
Effects of the Compact Encoder from the Perspective Overfitting. While our compact encoder has fewer parameters compared to Transformer-base encoders, its performance surpasses that of Transformer- based encoders [46], as analyzed Scction 4.2.1. One significant reason for this lies in the reduced risk of overfitting in downstream tasks due to the redundancy reduction. Given the challenging nature acquiring point cloud data, existing point cloud datasets for downstream tasks often small, such as ScanObjectNN [44] and ModelNet40 [57], each comprising just over 10,000 point clouds, and ScanNetV2 [6] with only 1,000 scenes. These dataset sizes are much smaller than those commonly found in image and language tasks. Therefore, fine-tuning in these size-limited datasets can be more prone to overfitting when considerable redundancy exists in the computation.
We visualize the training and testing curves for different encoders on the classification task in ScanObjectNN and the detection task in ScanNetV2 in Figures and 10. Figure illustrates the classification and detection curves for compact encoder and Transformer-bis based encoder after pretraining. It can be observed that during training, the classification accuracy and AP25 metric of the Transformer-based encoder significantly higher than those of our compact encoder. However, during testing, our compact encoder exhibits superior performance compared to the Transformer encoder. This starkly indicates that the Transformer-bast training set, demonstrating poorer generalization. Conversely, our compact encoder displays stronger generalization capabilities, indicating the superiority of the design of our compact encoder.
encoder tends to overfit the
Compact Encoder w/ Pre. Test Compact Encoder w/ Pre. Train
Transfomeer Encoder w/ Pre. Test Train: Transformer Encoder
Transformer Encoder w/ Pre. Train
Compact Encoder
100
100
90
Accuracy (1%)
80
AP25 (1%)
70
60
50
90
40
30
20
10
80
(a) ScanObjectNN
(b) ScanNetV2
Test: Compact Encoder Transformer Encoder
Figure 9: Training and testing curves of different pre-trained encoders. We present the training and testing curves for both the classification task on ScanObjectNN and the detection task on ScanNerV2. All encoders are pre-trained.
Figure 10 displays the classification and detection curves of our compact encoder and the Transformer- based encoder trained from scratch. In comparison to its counterpart in Figure 9, although shows a slower convergence, the overfitting issue of the Transformer- based encoder still emerges in the late stages training, reaffirming our conclusion. Meanwhile, the phenomenon of slow convergence in Figure 10 is reasonable it is encoder trained from scratch without better initialization.
t-SNE Visualization. We further used t-SNE [45] to visualize the feature distributions extracted by our LCM model and the Transformer. In Figure 11, we visualized the two-dimensional (2D) feature distributions of the two models, pretrained using Point-MAE, when directly transferred to the test set of the ModelNet40 [57] dataset without downstream fine-tuning. In Figure 12, we visualized the 2D feature distributions of the two pre-trained models after fine-tuning the most challenging variant of the ScanObjectNN [44] dataset, PB-RS-T50, using its test set.
In the 2D t-SNE visualizations, instances from the same category tend to be distributed in relatively clear and tight clusters. The compactness of the feature distributions different instances from the same category can be viewed as the model's ability to represent features of the same category. more compact distribution indicates a stronger modeling capability. As shown in Figure 11 and Figure 12, our LCM model achieves more compact feature distributions for instances of the same
20

---

Compact Encoder w/o Pre. Test
Transformer Encoder w/o Pre. Test
Compact Encoder w/o Pre. Train
Transformer Encoder w/o Pre. Train
Train:
Transformer Encoder > Compact Encoder
100
100
90
Accuracy (1%)
80
AP25 (1%)
70
60
90
50
40
30
20
10
80
(a) ScanObjectNN
(b) ScanNetV2
Test: Compact Encoder > Transformer Encoder
Figure 10: Training and testing curves for different encoders trained from scratch. We present the training and testing curves for both the classification task on ScanObjectNN and the detection task on ScanNetV2. All encoders
pretrained.
category compared the Transformer model in most cases, indicating that our LCM model has stronger ability to model the general representations of the same category.
Label
Label 35
LCM (Oura)
Transformer
LCM (Ours)
Transformer
Label0
Lebel
Figure 11: The feature distribution visualization of the pre-trained models on the test set of Mod- elNet40.
Figure 12: The feature distribution visualiza- tion the fine-tuned models on the test set of ScanObjectNN.
5.6 Broader Impacts and Safeguards
Our designed compact point cloud network will greatly facilitates the deployment of existing point cloud pre-training models resource-constraing devices, which would significantly advance existing point cloud applications. However, the prolifcration of more point cloud applications may lead to privacy data leaks, such personal housing layout point cloud leaks, and human feature point cloud leaks. Therefore, we advocate for the implementation of strict security measures during the actual deployment of applications prevent malicious access tampering of data. Below some corresponding measures:
1) Access Control: Implement stringent access control policies to restrict data access authorized users or systems only.
2) Data Encryption:
Utilize robust encryption algorithms to encrypt sensitive point cloud data during transmission and storage, ensuring its security against unauthorized access.
3) Anonymization: Anonymize sensitive information whenever possible to reduce the risk of data leaks. For instance, remove blur identifiable information, retaining only essential data for analysis.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our abstract and introduction illustrate our contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section 4.4.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification:
Our assumptions and proofs are in Section 5.1.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We discuss the experimental information in section 4 and 5.3.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification:
We have provided the complete code, checkpoints, and running instructions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtppcccccc/ccccccccccccccccclct / ssssssoliccyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We specify all the experimental training and test details in Section 4 and 5.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We report the average of the experiment results and therefore do not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We detail the compute resources in Section 5.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https //tt://nneuippp http://wwwww...ppppecc//cccc//ce
Answer: [Yes]
Justification: aspects.
The research of our study complies with the NeurlPS ethical guidelines in all
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: illustrate this impacts in 5.6
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
Justification:
We illustrate some solution in 5.6
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have correctly cited the assets used the paper and adhered to their licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: We will release all newly generated assets from the paper, including code and models, after the paper is accepted.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "5FG1udiY1p", "reviewer": "Reviewer_UfZW", "summary": "This paper proposes LCM, a locally constrained compact point cloud model, to improve the efficiency and performance of point cloud processing tasks. It consists of a locally constrained compact encoder and a locally constrained Mamba-based decoder. A locally constrained compact encoder utilizes the proposed local aggregation layer to replace self-attention to achieve an elegant balance between performance and efficiency. A locally constrained Mamba-based decoder introduces LCFFN after each Mamba SSM layer, maximizing the perceived point cloud geometry from unmasked patches. By focusing on local geometric constraints and leveraging SSM, the model achieves a balance between performance and efficiency.", "questions": "1.The authors have validated the effectiveness of the proposed LCM model in scene data through indoor scene detection tasks. However, is the proposed model also effective for other scene tasks, such as indoor semantic segmentation?\n\n2.In the supplementary materials, the authors used the proposed mamba-based decoder as an encoder to verify the impact of patch order and LCFFN. I have the following two questions: \n\n1) Firstly, the performance listed in Figure 8(a) seems to show a significant disparity compared to the classification performance listed in Figure 2(c) and Table 1. What is the reason for this disparity? \n\n2) Secondly, does this disparity indicate that the effectiveness of the proposed mamba-based decoder is inferior to that of the Transformer and the proposed LCM encoder?", "limitations": "The limitations presented by the authors are reasonable. However, the solutions to these limitations appear to lack specificity. I would like to inquire whether the authors have any concrete solutions regarding \"efficient dynamic importance.\" This is merely to ask if the authors have any reasonable and specific ideas without requiring actual experimental results.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Introducing a locally constrained compact encoder and Mamba-based decoder is a creative solution that improves both performance and efficiency. \n\n2.  The paper offers a thorough explanation of the proposed model, including the local aggregation layers and the integration of state space models, making the methodology clear and reproducible.\n\n3. The paper provides comprehensive experimental results demonstrating the effectiveness of the proposed method across multiple tasks and datasets.", "weaknesses": "1. ***[Static Importance Perception]*** The reliance on static local constraints may limit the model’s ability to dynamically identify and focus on important regions of the point cloud, potentially missing critical information.\n\n2. ***[Long-Range Dependency Modeling]*** While the local constraints improve efficiency, they might not capture long-range dependencies as effectively as self-attention mechanisms, which could be a limitation in some applications.\n\n3. ***[Scene-level semantic segmentation]*** It would be better if the author could provide fine-tuning performance on semantic segmentation with scene-level point cloud datasets like ScanNet or ScanNet++ to make the claim stronger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **@Q1 - Static Importance Perception & Long-Range Dependency Modeling & Limitations.**\n\nThank you for your insightful question. Our current model does have limitations in handling dynamic importance perception and long-range dependency modeling. Our design prioritizes efficiency, which can be at odds with the increased complexity required for capturing dynamic importance and long-range dependencies. This focus on efficiency led us to simplify the model in certain aspects, and as a result, we did not fully integrate mechanisms for dynamic importance perception and long-range dependency modeling in this version of our model.\n\nDespite these constraints, the current model has demonstrated significant improvements in performance across various tasks. Nevertheless, we also acknowledge that incorporating dynamic importance perception and long-range dependency modeling could further enhance the model's capabilities, particularly in more complex scenarios.\n\nWe are actively exploring methods to address these limitations in future work. Specifically, we are investigating the use of approximate nearest neighbor (ANN) algorithms to model dynamic importance and long-range dependencies more efficiently. By leveraging non-exact neighbor queries, we aim to balance the computational cost with the need for more sophisticated modeling, ensuring that we can extend the model's capabilities without compromising on efficiency.\n\n### **@Q2 - The effectiveness of LCM in scene-level semantic segmentation.**\n\nTo demonstrate the generalization capability of our proposed LCM model on scene-level data, we further validated its effectiveness in the scene-level semantic segmentation task. Specifically, we replaced the Transformer encoder used in Point Transformer V3 [1] with our LCM encoder to create our model for semantic segmentation. We trained our LCM model on the three most commonly used indoor scene point cloud datasets: ScanNet [2], ScanNet200 [3], and S3DIS [4], and reported their segmentation results on the validation sets. In all three datasets, we report the mean Intersection over Union (mIoU) percentages and benchmark these results against previous backbones.\n\nAs shown in the table below, our LCM model continues to perform well in semantic segmentation tasks across multiple scene datasets. Notably, in the ScanNet200 dataset, it outperforms the state-of-the-art Point Transformer V3 by 0.9. In the other two datasets, its performance is comparable to the state-of-the-art Point Transformer V3. These results collectively demonstrate the strong generalization capability of our model on scene-level point cloud data.\n\n| Methods  | ScanNet  | ScanNet200  | S3DIS  |\n| :------------ | :------------ | :------------ | :------------ |\n| PointNeXt [5]  | 71.5  | -  | 70.5  |\n| OctFormer [6]  | 75.7  | 32.6  | -  |\n| Point Transformer V1 [7]  | 70.6  | 27.8  | 70.4  |\n| Point Transformer V2 [8]  | 75.4  | 30.2  | 71.6  |\n| Point Transformer V3 [1]  | 77.5  | 35.2  | 73.4  |\n| LCM (Ours)  | 77.6  | 36.1  | 73.4  |\n\n###  **@Q3 - mamba-based decoder.**\n\n**1.Explanation of the Performance Difference.**\n\nThe observed difference primarily arises from the use of different data augmentation strategies during fine-tuning on downstream tasks. In Figure 8(a), all experimental results reflect the classification accuracy on the ScanObjectNN dataset when trained from scratch. Without any data augmentation, our Mamba encoder combined with LCFFN and y-order sorting achieved an accuracy of 83.1%. In contrast, our LCM encoder, trained from scratch using scaling and rotation data augmentation strategies, reached an accuracy of 86.3%. This significant discrepancy is largely attributable to the different data augmentation methods employed. To ensure a fairer comparison and minimize the impact of data augmentation on performance, we further evaluated the networks' performance without using any data augmentation strategies in the table below.\n\n**2.Comparison of the Effectiveness of the Mamba-based Decoder with Transformer and LCM Encoder**\n\nWe further compared the performance of three different encoder structures—standard Transformer, our Mamba+LCFFN, and LCM encoder—on downstream tasks without using any data augmentation. As shown in the table, our LCM encoder and LCM Mamba architecture as encoders significantly outperformed the standard Transformer architecture, demonstrating the effectiveness of these two architectures. Moreover, when comparing the LCM encoder with the LCM Mamba architecture, the LCM encoder proved to be more efficient and effective, largely due to our redundancy reduction strategy. Overall, both the LCM encoder and LCM Mamba architectures are highly efficient network architectures; however, the LCM encoder is better suited for encoder roles, while the LCM Mamba architecture is better suited for decoder roles.\n\n| Methods  | #Params. (M)  | OBJ-BG | OBJ-ONLY  | PB-T50-RS  |\n| :------------ | :------------ | :------------ | :------------ | :------------ |\n| Transformer  | 22.1  | 87.61  | 87.87  | 82.79 | \n| Mamba+LCFFN(y_order)  | 12.7  | 88.79  | 88.98 | 83.09| \n| LCM Encoder  | 2.7  | 89.85  |89.11 | 83.38 | \n\n[1] Wu, et al. “Point Transformer V3: Simpler Faster Stronger.” CVPR, 2024.\n\n[2] Dai, et al. “ScanNet: Richly-annotated 3d reconstructions of indoor scenes.” CVPR, 2017.\n\n[3] Rozenberszki, et al. “Language-grounded indoor 3d semantic segmentation in the wild.” ECCV, 2022.\n\n[4] Armeni, et al. “3d semantic parsing of large-scale indoor spaces.” CVPR, 2016.\n\n[5] Qian, et al. “Pointnext: Revisiting pointnet++ with improved training and scaling strategies.” NeurIPS, 2022.\n\n[6] Peng-Shuai Wang. “Octformer: Octree-based transformers for 3D point clouds.” TOG, 2023.\n\n[7] Zhao, et al. “Point transformer.” ICCV, 2021.\n\n[8] Wu, et al. “Point transformer v2: Grouped vector attention and partition-based pooling.” NeurIPS, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MmsOoxwcQW", "reviewer": "Reviewer_L8Jt", "summary": "To address the issues of quadratic complexity and constrained decoders in existing masked point modeling methods based on Transformers, this paper proposes a locally constrained compact point cloud model. First, to tackle the complexity problem, the paper presents an observational experiment with top-K attention to demonstrate the importance of redundancy reduction in point cloud analysis. Based on this redundancy reduction idea, the paper then introduces a locally constrained compact encoder by replacing the self-attention layer with a local aggregation layer. Finally, to overcome the limited reconstruction potential of the original Transformer decoder, the paper proposes a locally constrained Mamba decoder and demonstrates its superiority through both experimental results and information theory analysis.", "questions": "My concerns and suggestions have already been outlined in the weaknesses section. I hope the author can provide further explanations on these issues.", "limitations": "The author adequately explains the limitations of their work. This static importance of the model may somewhat restrict its actual performance. I hope this limitation can be addressed in future work.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1)\tThe motivation of this paper is strong, and the mentioned efficiency and reconstruction potential issues of Transformer is important in point cloud analysis;\n2)\tThe paper presents a locally constrained compact point cloud model (LCM) in the field of point cloud self-supervised learning. This model design is quite novel.\n3)\tThe proposed model is universal and achieves impressive results, reaching state-of-the-art effects with a minimal number of parameters.\n4)\tThe paper also offers many well-reasoned and insightful explanations for observed phenomena.", "weaknesses": "1)\tIn the model design, as shown in Figure 5, I noticed that the proposed method seems to perform KNN at each layer. In fact, KNN is a computationally intensive operation. Considering practical applications, this operation could significantly increase the actual inference time of the model. I hope the author can provide a reasonable explanation or experimental validation to address this concern.\n2)\tThe results in Table 1 appear to show the highest performance. However, due to the influence of random seeds, this is not sufficient to reflect the true effectiveness of the model. The authors need to explain how these experimental results were selected, and also report the average performance over multiple trials.\n3)\tThere is an error in the citations. Both the Transformer and PointMamba references in Table 1 under \"supervised learning only\" point to the paper \"Pointmamba: A simple state space model for point cloud analysis.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **@Q1 - The computational cost of KNN.**\n\nThank you for your question and for highlighting an important aspect of our model's design. \n\nIn fact, the computational cost of KNN in LCM is very low. While Figure 5 may give the impression that KNN is performed at each layer, our method actually requires computing the KNN only once at the beginning, not at each layer. This design choice significantly reduces the computational cost associated with KNN operations. Specifically, our low cost is due to the following factors:\n\n * **Single KNN Computation:** In our approach, the geometric neighbor properties of each point are computed once at the start of the model using KNN. These properties, encapsulated in the graph indices, are then fixed and remain consistent across all layers of the model. This means that after the initial KNN computation, the graph indices are reused in all subsequent layers, eliminating the need for repeated KNN calculations. By sharing the precomputed graph indices across layers, we maintain the enforcement of local constraints while minimizing computational overhead.\n\n * **Low Cost of Single KNN Computation:** Although KNN has a time complexity of O(n²), the impact on practical applications is mitigated because we only compute it once. Moreover, in common point cloud classification tasks, the number of point patches (n) is typically quite small, ranging from 64 to 128. This relatively small value of n, coupled with a minimal number of nearest neighbors (K), which we set to 5, ensures that the computational cost remains low.\n\nwe conducted experiments on the PB-T50-RS variant using the ScanObjectNN dataset. We compared inference time and classification accuracy under different conditions: using shared random indices (without KNN computation), using shared KNN indices (**Ours**), and using individual KNN indices for each layer. As shown in the table below, compared to the random aggregation method without KNN, using shared KNN indices results in only a 0.005 millisecond increase in inference time while improving accuracy by 1.88%. Additionally, compared to performing independent KNN computations at each layer, our method achieves the same level of accuracy with a 0.015 millisecond improvement in speed. \n\n|   | Inference Times  | ScanObjectNN  |\n| :------------ | :------------ | :------------ |\n| w/ Random K Indices (w/o KNN)  | 0.641 ms  | 87.47  |\n| w/ Shared KNN (Ours)  | 0.646 ms  | 89.35  |\n| w/ Individual KNN  | 0.661 ms  | 89.35  |\n\n\n### **@Q2 - Average performance.**\n\nThank you for your insightful question. We appreciate your concern regarding the influence of random seeds on the reported performance.\n\nTo ensure a fair comparison, we conducted our experiments following the standard practices in the field (as used in Point-BERT[1], MaskPoint[2], Point-MAE[3], Point-M2AE[4], ACT[5]). Specifically, for each point cloud classification experiment, we used eight different random seeds (0-7) to ensure the robustness and reliability of our results. The performance reported in Table 1 represents the highest accuracy achieved across these eight trials for each model configuration.\n\nHowever, to provide a more comprehensive evaluation of the models' true effectiveness, we also calculated the average performance over these eight trials. This average, which accounts for variability due to random seed selection, offers a more reliable assessment of the models' general performance.\n\nAs shown in the three tables below, we report the average classification accuracy of each model on the ScanObjectNN dataset. This mean value, averaged across the eight different random seed runs, demonstrates the stability and consistency of our method. Our LCM model not only achieves the highest performance but also shows superior average performance, consistently outperforming existing transformer-based methods. This analysis confirms the true effectiveness and reliability of our approach.\n\n**(1) ScanObjectNN（OBJ-BG）**\n|   | Point-BERT[1]  | MaskPoint[2]  | Point-MAE[3]  | Point-M2AE[4]  | ACT[5]  |\n| :------------ | :------------ | :------------ | :------------ | :------------ | :------------ |\n| Transformer  |  92.48 |  92.17 |  92.67 | 93.12  | 92.08  |\n| **LCM (Ours)** | 93.55  | 93.31  | 94.51  | 93.83  | 94.13  |\n\n**(2) ScanObjectNN（OBJ-ONLY）**\n|   | Point-BERT[1]  | MaskPoint[2]  | Point-MAE[3]  | Point-M2AE[4]  | ACT[5]  |\n| :------------ | :------------ | :------------ | :------------ | :------------ | :------------ |\n| Transformer  |  91.60 |  91.69 |  92.08 | 91.22  | 91.70  |\n| **LCM (Ours)**| 92.43  | 91.98  | 92.75  | 92.41  | 92.66  |\n\n**(3) ScanObjectNN（PB-T50-RS）**\n|   | Point-BERT[1]  | MaskPoint[2]  | Point-MAE[3]  | Point-M2AE[4]  | ACT[5]  |\n| :------------ | :------------ | :------------ | :------------ | :------------ | :------------ |\n| Transformer  |  87.91 |  87.65 |  88.27 | 88.06  | 87.52  |\n| **LCM (Ours)** | 88.57  | 87.75  | 88.87  | 88.38  | 88.57  |\n\n### **@Q3 - Citation error.**\n\nThank you for your thorough review. We will address this issue in the next version.\n\n[1] Yu, et al. \"Point-bert: Pre-training 3d point cloud transformers with masked point modeling.\" CVPR, 2022.\n\n[2] Liu, et al. \"Masked discrimination for self-supervised learning on point clouds.\" ECCV, 2022.\n\n[3] Pang, et al. “Masked autoencoders for point cloud self-supervised learning.” ECCV, 2022.\n\n[4] Zhang, et al. “Point-m2ae: Multi-scale masked autoencoders for hierarchical point cloud pre-training.” NeurIPS, 2022.\n\n[5] Dong, et al. “Autoencoders as cross-modal teachers: Can pretrained 2d image transformers help 3d representation learning?” ICLR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OsH5PTLedE", "reviewer": "Reviewer_jpku", "summary": "This paper first proposes a locally constrained compact encoder, which leverages static local geometric constraints to aggregate the most relevant information for each patch token, achieving an elegant balance between performance and efficiency. Moreover, this paper also proposes a locally constrained Mamba-based decoder for masked point modeling. The authors verify the effectiveness and efficiency of the proposed model on multiple pre-training strategies and downstream tasks.", "questions": "See the weaknesses.", "limitations": "The authors' description of the limitations of their work is reasonable and effective solutions are given.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The model LCM proposed by the authors is novel and very efficient, achieving leading performance with only 2.7M parameters, which is much lower than existing mainstream models. \n\n2. The authors provide a rational explanation for the model design, i.e., an encoder design based on redundancy reduction and a decoder design based on mutual information maximization. \n\n3. Extensive experiments show the effectiveness of the proposed method.\n\n4. The paper is well-organized. The tables, figures and notations are clear.", "weaknesses": "Some experimental comparisons are insufficient, and certain details are not clearly described.\n1. I noticed that the authors only compared the results of the proposed method with PointGPT-S (NeurIPS 2023) and PointGPT-B. In fact, PointGPT-L offers more powerful performance. Can the author provide a specific explanation for this?\n\n2. The ShapeNet dataset is relatively small, with a limited number of 3D models. Could the author provide results of the proposed method pre-trained on a larger dataset, such as the unlabeled hybrid dataset used in PointGPT (NeurIPS 2023)? This would have a significant impact on demonstrating the generalization ability of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **@Q1 - Comparison with PointGPT-L.**\n\nThank you for your question. We understand the importance of comparing our method with all relevant benchmarks, including PointGPT-L, to provide a comprehensive evaluation.\n\nPointGPT[1], a point cloud pretraining approach published at NeurIPS 2023, proposed extending the generative pretraining techniques from natural language processing (NLP) to point clouds. By partitioning and sorting point patches, the method feeds point embeddings into a transformer decoder for autoregressive prediction. Furthermore, a dual masking strategy was introduced to enhance the learned representations.\n\nThe primary reason we initially compared our LCM model only with PointGPT-S and PointGPT-B, and not with PointGPT-L, is due to a significant difference in model size and computational complexity. PointGPT-L has a substantially larger number of parameters compared to our LCM model, which focuses on achieving high efficiency. Specifically, PointGPT-L's parameter count is significantly higher, leading to greater computational load and resource requirements. In contrast, our LCM model is designed to be lightweight and efficient, with a parameter count that is only a fraction of PointGPT-L's.\n\nTo address this, we have now included a comparison of our LCM model with PointGPT-L. As shown in the table below, while PointGPT-L does achieve higher accuracy, our LCM model provides a highly efficient alternative, using significantly fewer parameters and computational resources. Specifically, our model's parameter count is only 0.75% of PointGPT-L's, and the computational load is just 1.75% of theirs. This comparison highlights the trade-offs between performance and efficiency, with our LCM model achieving approximately a 100-fold improvement in efficiency.\n\nWe hope this explanation clarifies our initial decision and provides a comprehensive understanding of the trade-offs involved.\n\n|   | #Params(M)  | FLOPs(G)  | OBJ-BG  | OBJ-ONLY  | PB-T50-RS  |\n| :------------ | :------------ | :------------ | :------------ | :------------ | :------------ |\n| PointGPT-L  | 360.5  |  74.2 |  95.7 | 94.1  | 91.1  |\n| LCM  | 2.7  | 1.3  |  95.2 | 93.1  |  89.4 |\n\n\n### **@Q2 - Results on the unlabeled hybrid dataset (UHD).**\n\nThank you for your question and suggestion. We acknowledge that the ShapeNet dataset is relatively small and may not fully demonstrate the generalization capabilities of our proposed method.\n\nTo address this concern, we have conducted additional experiments by pre-training our model on a larger dataset, specifically the unlabeled hybrid dataset used in PointGPT[1]. This dataset is significantly larger and more diverse, providing a more robust evaluation of the generalization abilities of our method.\n\nThe results from these additional experiments are presented in the table below. The findings show that our proposed method maintains strong performance and generalization capabilities even when pre-trained on a larger and more diverse dataset. The results underscore the effectiveness of our approach in leveraging large-scale data to learn comprehensive and robust representations.\n\nWe believe these additional experiments will provide a clearer understanding of our method's potential and its applicability to a wider range of 3D models and datasets. Thank you for your valuable feedback, which has helped us to strengthen our evaluation.\n\n|   | #Params(M)  | OBJ-ONLY  | PB-T50-RS  |\n| :------------ | :------------ | :------------ | :------------ |\n| ShapeNet55 |  95.18 | 93.12  | 89.35  |\n| UHD  |  95.53 | 93.63  |  90.04 |\n\n[1] Chen, et al. \"Pointgpt: Auto-regressively generative pre-training from point clouds.\" NeurIPS, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SW1XV93Ema", "reviewer": "Reviewer_y4yX", "summary": "This paper proposes a locally constrained compact point cloud model (LCM), which consists of a locally constrained compact encoder and a locally constrained decoder based on Mamba. The encoder replaces the self-attention layer with a local aggregation layer, thus achieving a perfect balance between performance and efficiency. The locally constrained Mamba-based decoder is introduced considering the different information densities between masked and unmasked patches in the MPM decoder input. Extensive experimental results show that the LCM greatly outperforms existing Transformer-based models in terms of both performance and efficiency.", "questions": "The significance of this work seems to me and whether the authors will choose to make it fully open source? I hope this work can contribute to the development of the field.", "limitations": "No obvious limitations, but some elements require more detailed explanation.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The novelty of this paper is evident in proposing a locally constrained compact point cloud model by revisiting the shortcomings of the commonly used Transformer-based point mask modeling approach. By acting on five classical point cloud self-supervised methods, it revolutionizes the point cloud self-supervised learning technique in terms of methodology and results.\n\nThe theoretical analysis and experimental results in this paper are quite detailed. By localizing and spatially statefulizing the structure of common encoder-decoder architectures, it allows existing self-supervised methods to achieve significant improvements.\n\nThe quality of the presentation in this paper is generally excellent, but some of the illustrations and content still need to be improved for clarity. See “Weaknesses” for more details.\n\nThe method proposed in this paper is aimed at self-supervised learning of 3D point clouds and has considerable application value in practical applications.", "weaknesses": "1. The quality of some figure illustrations in the paper concerns me; The fonts in Figures 4 and 5 are too small and both are in bold, which does not seem to fit the human eye.\n\n2. The elements in Sections 3.3 and 3.4 appear to be improvements over the existing technology and Equation 4-7 should be interpreted differently.\n\n3. Some parts of this paper conflict and it is recommended that the word “section” be capitalized and standardized.\n\n4. The experimental analysis shows that the decrease in the number of parameters of the LCM is extremely obvious, which should be attributed to the Mamba model. It is suggested that the authors analyze the Transformer and Mamba network parameters in detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **@Q1 - Figure quality and font size.**\n\nThank you for your feedback regarding the quality of the figure and formatting in our paper. We appreciate your attention to these details, as they are crucial for a clear presentation. We acknowledge that the fonts in Figures 4 and 5 are too small and bold, which may affect readability. In the revised version of the paper, we will adjust the font size and weight to ensure they are appropriately sized and easy to read. We will also review all figures to ensure consistency and clarity in visual presentation.\n\n### **@Q2 - The elements in Sections 3.3 and 3.4 & Equation 4-7.**\n\nThank you for your suggestion regarding Sections 3.3 and 3.4. In Sections 3.3 and 3.4, we discuss several improvements and modifications to existing technology, which involve the design and implementation of our Locally Constrained Compact Encoder and Locally Constrained Mamba-based Decoder.\n\n * Section 3.3: The design of the Locally Constrained Compact Encoder introduces innovations based on redundancy reduction. Equations (4) and (5) describe the forward propagation process of the encoder's features. These equations should be interpreted in the context of the local constraints and feature aggregation methods employed in our model. Specifically, the function $f_i(⋅)$ reflects a unique implementation that leverages local geometric constraints, which distinguishes our approach from the standard Transformer. We will provide a more detailed explanation of these modifications in next versions.\n\n * Section 3.4: Our Locally Constrained Mamba-based Decoder is a design based on mutual information maximization. Equations (6) and (7) outline the forward propagation process within this decoder. The primary changes we made involve the implementation of the functions  $s_i(⋅)$ and $f_i^l(⋅)$. We will also expand on these modifications in more detail in the next version.\n\nThank you again for your valuable feedback. We will incorporate these clarifications in the revised manuscript.\n\n### **@Q3 - Consistency in terminology.**\n\nWe apologize for any inconsistencies in terminology, particularly with the use of the word “section.” In the revised manuscript, we will capitalize \"Section\" consistently and standardize its usage throughout the paper. This will include careful proofreading to ensure uniformity in all headings, labels, and references. We appreciate your careful review and constructive comments, which help us improve the clarity and professionalism of our work. These revisions will be reflected in the next version of the manuscript.\n\n### **@Q4 - Detailed comparison of network parameters.**\n\nThank you for your insightful feedback. The significant reduction in the number of parameters observed in the LCM model is indeed a crucial aspect of our design, but it is important to clarify that this reduction is primarily due to the Locally Constrained Compact Encoder rather than the Mamba model.\n\nThe Mamba model is employed in our Locally Constrained Mamba-based Decoder and is used specifically during the pretraining phase to enhance the reconstruction process. The parameter reduction achieved by the LCM is mainly attributed to the innovative design of the Locally Constrained Compact Encoder, which replaces the standard self-attention layer in Transformers with a more efficient local neighbor aggregation mechanism and compresses the FFN parameters accordingly.\n\nBased on your suggestion, we conducted a detailed comparison of the parameters between the Transformer Encoder Block and our Locally Constrained Compact Encoder Block. The results are as follows, our LCM Encoder has a high parameter reduction compared to the standard Transformer in both the attention layer and the FFN layer.\n\n```javascript\nTransformer Block 1.773 M{\n    LayerNormal1: 0.001M\n    Self-Attention 0.590 M {\n\t    Q: 0.147 M\n\t    K: 0.147 M\n\t    V: 0.147 M\n\t    Projection: 0.148 M\n    }\n    LayerNormal2: 0.001M\n    FFN: 1.182 M {\n\t    Linear1: 0.59 M\n\t    Linear2: 0.59 M\n    }\n}\n```\n```javascript\nLCM Block 0.188 M{{\n    LayerNormal1: 0.001 M\n    Local Aggregation Layer: 0.113 M {\n\t    Down MLP: 0.074 M\n\t    Up MLP: 0.038 M\n    }\n    LayerNormal2: 0.001 M\n    FFN: 0.074M {\n\t    Linear1: 0.037 M\n\t    Linear2: 0.037 M\n    }\n}\n```\n\n### **@Q5 - Open source.**\n\nOf course! We are dedicated to advancing the point cloud field and plan to make our work fully open source. We are currently organizing the code and checkpoints, and we aim to release them within the next few weeks. By making these resources available, we hope to facilitate further research and experimentation, and contribute to the broader academic and practical applications of our methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their time and their thoughtful comments and questions. We are pleased to find that:

* All reviewers unanimously appreciated the novelty and effectiveness of our work, recognizing it as a creative solution that revolutionizes the point cloud self-supervised learning technique in terms of both methodology and results.
* All reviewers commended the thorough theoretical and experimental explanations we provided for the proposed method.
* All reviewers acknowledge the significant improvements and promising performance achieved by our approach. 
* Reviewer y4yX recognizes the practical application value of the LCM. Designing deployable point cloud algorithms has been our constant pursuit. 

We carefully considered all questions, concerns, and comments provided by reviewers. We attempted our best to address the questions as time allowed. We believe the comments & revisions have made the paper stronger and thank all the reviewers for their help.  We provide detailed responses to each review separately, please find individual responses to your questions below.

Meta-review: This paper presents a novel approach called the Locally Constrained Compact Point Cloud Model (LCM), which features a locally constrained compact encoder and a Mamba-based decoder. By replacing the traditional self-attention mechanism with a local aggregation layer, the proposed model achieves an effective balance between performance and efficiency. The LCM method is supported by comprehensive experiments and rigorous comparisons with state-of-the-art models. All reviewers recommend accepting the paper, and the Area Chair finds no compelling reasons to overturn their recommendations, thus recommending its acceptance.
Decision: Accept (poster)