What We Miss Matters: Learning from the Overlooked in Point Cloud Transformers
Authors: Yi Wang
Jiaze Wang
Ziyu Guo
Renrui Zhang
Donghao Zhou
Guangyong Chen
Anfeng Liu
Pheng-Ann Heng
Venue: NeurIPS 2025 poster ()
Keywords: Point Cloud Understanding
Representation Learning
Primary area: deep_learning

Abstract: Point Cloud Transformers have become a cornerstone in 3D representation for their ability to model long-range dependencies via self-attention. However, these models tend to overemphasize salient regions while neglecting other informative regions, which limits feature diversity and compromises robustness.
To address this challenge, we introduce BlindFormer, a novel contrastive attention learning framework that redefines saliency by explicitly incorporating features typically neglected by the model. The proposed Attentional Blindspot Mining (ABM) suppresses highly attended regions during training, thereby guiding the model to explore its own blind spots. This redirection of attention expands the model’s perceptual field and uncovers richer geometric cues.
To consolidate these overlooked features, BlindFormer employs Blindspot-Aware Joint Optimization (BJO), a joint learning objective that integrates blindspot feature alignment with the original pretext task. BJO enhances feature discrimination while preserving performance on the primary task, leading to more robust and generalizable representations.
We validate BlindFormer on several challenging benchmarks and demonstrate consistent performance gains across multiple Transformer backbones. Notably, it improves Point-MAE by +13.4\% and PointGPT-S by +6.3\% on OBJ-BG under Gaussian noise. These results highlight the importance of mitigating attentional biases in 3D representation learning, revealing BlindFormer’s superior ability to handle perturbations and improve feature discrimination.

What We Miss Matters: Learning from the Overlooked in Point Cloud Transformers
Yi Wang': Jiaze Wang" Ziyu Guo?, Renrui Zhang', Donghao Zhou',
Guangyong Chen", Anfeng Liu', Pheng-Ann Heng*
Central South University The Chinese University of Hong Kong FitX Technology (Hong Kong) Limited ZZhjiaang Lab
Abstract
Point Cloud Transformers have become a cornerstone in 3D representation for their ability to model long-range dependencies via self-attention. However, these models tend to overemphasize salient regions while neglecting other informa- tive regions, which limits feature diversity and compromises robustness. To ad- dress this challenge, we introduce BlindFormer, a novel contrastive attention learning framework that redefines saliency by explicitly incorporating features typically neglected by the model. The proposed Attentional Blindspot Mining (ABM) suppresses highly attended regions during training, thereby guiding the model to explore its own blind spots. This redirection of attention expands the model's perceptual field and uncovers richer geometric cues. To consolidate these overlooked features, BlindFormer employs Blindspot-Awwad Joint Optimization (BJO), joint learning objective that integrates blindspot feature alignment with the original pretext task. BJO enhances feature discrimination while preserving performance on the primary task, leading to more robust and generalizable rep- resentations. We validate BlindFormer on several challenging benchmarks and demonstrate consistent performance gains across multiple Transformer backbones. Notably, it improves Point-MAE by +13.4% and PointGPT-S by +6.3% on OBJ- BG under Gaussian noise. These results highlight the importance of mitigating attentional biases in 3D representation learning, revealing BlindFormer's superior ability handle perturbations and improve feature discrimination. Project page: https://winffred2277..m satthub..//pojeets/iinnooomonmoo htmll
1
Introduction
Point clouds serve as fundamental 3D representation with broad applications in robotics 6 47]
autonomous driving 77. augmented reality (3), and virtual reality [13]. While their versatility highlights the need for robust space understanding, the irregular and sparse nature of point clouds complicates efficient processing.
The recent success Transformer architectures [34 5 65 in point cloud analysis has demonstrated
remarkable capabilities in capturing long-range dependencies through self-attention mechanisms. These models follow a paradigm where the attention weights automatically emphasize salient regions for understanding the point cloud while downplaying less significant areas. Originally designed for natural language, attention mechanism has been successfully adapted for 2D vision. However, unlike natural language 9 images [17], which often contain redundant information such as
'These authors contributed equally to this work. Coorespponingg author: afengliu esu.edu.cn
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

100
MAE
PoinIGPTT
BlindFormme
Point-MAE
OBL-BG Top-1 Accuracy (%)
+5.0
PointGPT
50
BlindFrrmer
Geussian Nose (o-0.00)
Roonion (M[-30 301) (0.5, 1.5)
Scaling
DropPoint (60%)
Lamp
Bookshelf
Radio
Laptop
Bottle
Noliy Enviromments
Figure I: Illustration of Blinersrmer's Advantages. Point-MAE is employed as the backbone of our proposed BlindFormer. Left: BlindFormer emphasizes extracting information from greater number of patches. Right: BlindFormer demonstrates greater robustness than previous methods.
contextual structures and backgrounds, point cloud data are inherently sparse, meaning that each point or region is critical to the overall representation. This scarcity of redundant information implies that Transformer-based models, when neglecting less prominent point patches, may inadvertently overlook essential latent information. This observation leads to pivotal question: Can we design framework that actively mitigates attentional blind spots to improve both robustness and feature discrimination in 3D representation learning?
To answer this question, we re- re-examine the attention weights in Transformer-based point cloud models. As illustrated in Figure I we find that models like Point-MAE (65] and PointGPT (5) primarily rely on a limited set of high-attention patches for analysis. This reliance presents two significant issues: (1) Increased sensitivity perturbations. Over-focusing on high-attention patches makes the models more susceptible to noise and incomplete data, as disturbances in these areas disproportionately affect performance. (2) Inadequate feature discrimination. Ignoring potential information in low-attention regions limits the model's ability to distinguish objects with similar local structures, reducing their ability generalize across distribution shifts.
To address the limitations of attentional bias, we introduce BlindFormer, a contrastive attention learning framework that can be seamlessly integrated into existing point cloud Transformers. Our approach comprises two key components: First, we introduce an Attentional Blindspot Mining (ABM) module, which reshapes the attention distribution by explicitly suppressing salient regions during training. By constructing a masking distribution derived from self-attention scores, ABM selectively occludes patches that contribute most to the global representation. This encourages the model to redirect its focus toward previously overlooked blind spots, thereby learning to capture spatial structures from more subtle and expanding its perceptual field. To reinforce these under- attended features, BlindFormer further incorporates Blindspot- Aware Joint Optimization (BJO), united learning objective that integrates blindspot feature alignment with the original pre-training task. This joint supervision not only retains the model's ' task-specific capabilities but also enhances its ability capture invariant and discriminative features across diverse point clouds.
Extensive experiments demonstrate that our BlindFormer significantly enhances the robustness of Transformer- based models across various noisy environments, including Gaussian noise, rotation, scaling, and point dropout. Specifically, under Gaussian noise perturbation, BlindFormer improves the classification accuracy of Point-MAE by 17.2% in the OBJ-ONLY setting. Similarly, under rotation perturbation, boosts the segmentation performance of PointGPT-S by 3.3% on ShapeNetPart. Furthermore, BlindFormer achieves state-oftthe-ae art performance across various downstream tasks for 3D understanding. Even when competing models are given additional training time, BlindFormer maintains a consistent performance advantage. These results highlight BlindFormer's potential to effectively address the limitations of existing Transformer- based models by comprehensively activating regions and improving feature discrimination capabilities.
Our main contributions can be summarized as follows:
(I) We propose BlindFormer, a novel contrastive attention learning framework for point cloud understanding that explicitly addresses attentional blindspots which enhances the model's ability to capture global structures and significantly improves its robustness and generalization capabilities.

---

(II) We introduce an attentional blindspot mining strategy that dynamically suppresses dominant regions and encourages the model focus on previously neglected parts, promoting the learning of spatial features from broader set of patches rather than over-relying on a salient subset.
(III)
Extensive experimental results demonstrate that BlindFormer can be seamlessly integrated into mainstream transformer architectures and achieve significant improvements across variety of 3D understanding tasks.
Related Works
Self-Supervised Learning for NLP and Image. Self-supervised learning (SSL) has emerged as a
powerful paradigm in natural language processing (NLP) TTI 75 and computer vision 41 14 64
32 40 T (2], enabling models to learn rich representations from unlabeled data. In NLP, BERT
exemplifies this by randomly masking input tokens and training the model to predict them, fostering
deep contextual understanding. ELMo [55] utilizes bidirectional LSTMs to generate contextual- ized word embeddings, while GPT
41 adopts autoregressive approach with unidirectional Transformer to predict the next word, fine-tuning all parameters for specific tasks. Recently, gener- ative SSL methods have begun to outperform contrastive approaches in computer vision. Masked Autoencoders 17 randomly mask image patches and train the model to reconstruct the missing pixels, leading to effective visual representations. BEiT extends this by tokenizing image patches and predicting masked tokens, integrating NLP techniques into vision tasks. Additionally, Image GPT 23 treats images as sequences of pixels and trains Transformer autoregressively predict pixels without explicit spatial structure, demonstrating strong representation learning. This shift towards generative self-supervised learning methods not only demonstrates their ability capture comprehensive data representations and improve performance in NLP and computer vision but also highlights their significant potential in advancing point cloud processing and analysis.
Self-Supervised Learning for Point Cloud. Various methods have been investigated for self- [2
supervised learning point clouds [54] [2 [57 69
58 [56). Many works focused on generative modeling with generative adversarial networks and autoencoders, aiming to reconstruct PointMAE (34]
77,
16
66 [2
inputs using different architectural designs [31
65
46
22
2
55
21
51 43
proposes an effective scheme of masked autoencoders for point cloud self-supervised learning. Point- M2AE 67) further employs a hierarchical transformer architecture and implements a specific masking strategy. PointGPT (5) proposes point cloud auto-regressive generation task to pre-train transformer
models. Moreover, contrastive methods also have been extensively explored [39]6]] 61 62
33 72 60)
19. DepthContrast (72) generates augmented depth maps and conducts instance discrimination on the extracted global features. MVIF 20) employs gross-modal and cross-view invariance constraints to enable self-supervised learning of modal- and view-invariant features. OcCo 49 aims to reconstruct the original point cloud from an occluded version observed in camera views. Some studies focus on integrating cross-modal information, utilizing knowledge from language or image models to 44 38 PointCLIP 688 facilitates the alignment between point clouds encoded by CLIP and corresponding text descriptions, enhancing cross-modal understanding. PointCLIP V2 (74) uses a shape projection module to guide CLIP in generating more realistic depth maps and prompts GPT model to create 3D-specific text for textual encoder input.
enhance 3D learning (37 10 50
3 Methods
The overall framework of BlindFormer is illustrated in Figure First, the Attentional Blindspot Mining module generates a masked point cloud by dynamically suppressing regions with high attention scores. Both masked point cloud and the original input point cloud are then fed into the shared backbone model to obtain the global features of each input. By explicitly aligning the features from these two branches, BlindFormer guides the model better capture and understand information from the overlooked regions, enhancing feature discrimination and generalization. During the pre- training stage, the model is optimized using combination of blindspot feature alignment and the original pre-training loss-such as the reconstruction loss from Point-MAE 34 or the generation
loss from PointGPT [5). After pre-training, we employ the backbone model without the masking strategy, leveraging the learned latent representations for downstream tasks.
3

---

Standard Branch
Feature
Cloud Token
Misk Takes
Point Cloud
3D Embooding
Eeput
Transformer Blocks
Latest Represantations p
Task Task-Spoc.
FPS
Taken Fimbed.
9
KNN
Seloctel Top-K Mask Patches
Dynamic Mask Probubillty
Self-Antemion
Global Feature
Specific Repeesentarion R
CLS
Atfentional
Blindspot
Blindspot Feature Aligument
Original Loss Minimization
Mining
CLS T, T2
Global Feature H*
Spocific Represencation
Masked Point Cloud
3D Embedding
Inpur Takens n
Latens Representations
Task-Spec. Head
Blocks
FPS
Embed.
ge
KNN
Blindspot Branch
Figure 2: Overview of the BlindFormer Framework. BlindFormer consists of two branches that share the same weights: a standard mode branch and a blindspot mode branch. An attentional blindspot mining module generates masked point cloud by selecting less activated patches from the output of the standard mode branch. Both branches process their respective inputs through the shared Transformer blocks obtain latent representations. Finally, blindspot-awart joint optimization is used to align the representations of these two branches.
3.1 Attentional Blindspot Mining
To fully leverage the self-attention mechanism and mitigate the model's reliance small subset of key patches, we propose an Attentional Blindspot Mining (ABM) strategy, which guides the model to focus blindspot regions and enforces in challenging scenarios by dynamically masking salient areas.
comprehensive understanding of the whole structure
Point patch attention. Given point cloud X E RPx3 utilize Farthest Point Sampling (FPS) and K-Nearest Neighbors (KNN) algorithms to identify n center points and their corresponding k nearest neighbors, forming point patches P. We employ the self-attentin mechanism in the transformer architecture to compute the attention weights of point patches relative to the global feature. A new of input tokens E [m+1)xdd consisting of point tokens TP Igax and learnable global feature token T RR Ixd is utilized compute the queries Q ((++1)xd keys K [n++1xxd and values V [(x+1)xd The attention matrix A is subsequently derived from the dot product of queries and keys. Since the first element of the input tokens Ti corresponds to the global feature token, the first row of the attention matrix be interpreted the contribution of each token to the global feature. Considering the output tokens depend on both the attention matrix and the values, we incorporate the norm of V, when determining the significance score of token j. Consequently, the attention matrix and significance score for point patch j are computed as follows:
A = SoSSoffman (QKT) /Vd).
A1j ||V
S, Σ AI, X [[]]
(2)
where i,j 2,..., 1. For multi-head attention layer, compute the significance scores for each head separately and aggregate them by taking the sum over heads.
Dynamic blindspot generation. A straightforward idea is mask the top & patches with the highest significance scores, as they are key to the model's 's understanding of the point cloud. However, fixed masking probability merely shifts the model's attention without engaging a broader set of

---

patches. As the model becomes reliant on new areas of focus, it similarly falls into the trap of limited comprehension of the point cloud. Our primary objective is ensure that high-attention regions have higher likelihood of being suppressed. Therefore, we propose dynamic blindspot generation strategy. Specifically, we construct an updatable base masking probability using the latest self-attention significance scores, prioritizing the masking of patches that currently contribute significantly to the global features. Additionally, Gumbel noise noise[30] is introduced as a perturbation probability, derived from a uniform distribution U|0, 1]. to enhance the variability of the masking probability. Based on this concept, the final dynamic masking probability Pdy is expressed as:
Pdy = log (Softmar(S//t) (S/Tpr)) - log (-logE),
(3)
where Tpro is temperature hyperparameter which controls the sharpness of the base masking probability. A lower temperature (less than 1) results in a sharper distribution, meaning that regions with the highest attention are more likely be masked. Based on the dynamic masking probability, we apply simple Top-K strategy to select the k point patches pmack E RR*x3 to be masked:
pmosk (pp-K(Pay::))
(4)
pblind P pmosk
(5)
The blindspot region phlind contains the previously under-attended patches, which are preserved as input for the model. this manner, regions that previously attract high attention are more likely to be suppressed, promoting a deeper understanding of the whole spatial information for the model.
3.2 Blindspot-Aware Joint Optimization
To effectively consolidate the spatial cues learned from under-attended regions, we introduce Blindspot-Aware Joint Optimization (BJO), bidirectional learning strategy that integrates blindspot feature alignment into the pre-training objective. This optimization encourages the model to learn more discriminative feature representations from blindspot regions while preserving its task-specific learning capabilities and improving its generalization.
Blindspot feature alignment. The dynamically selected blindspot token T and the standard token T are both input into shared- weight model, producing two distinet levels of point cloud latent representations F and F*. Unlike the blindspot input, the complete point cloud retains all original information. Although the masking strategy results in the loss of some regional details, both representations still correspond to the same underlying point cloud entity. To encourage this consistency, we introduce blindspot feature alignment objective that explicitly aligns global features from the masked and standard inputs. This promotes invariance to specific localized patterns and pushes the model to rely more on global context rather than overfitting to salient regions. The blindspot-aware contrastive loss is defined as:
1 Σ (log exp(11) / E, exp(H HT]/Tim)
exp(1I 11/ + log Σ, exp((1F H//TTs))
Leontr - 2a
(6)
where a the number of point clouds in a batch; Tsim a temperature hyperparameter; H. and Hi are the normalized projection features of F. and F.. By omitting the high-attention regions in the masked point clouds, the contrastive objective incentivizes the model to focus on and extract valuable information from less emphasized areas. This process facilitates the learning of more holistic latent representation, thereby improving the model's capacity to effectively differentiate between various point cloud objects.
Contrastive learning enhancement. While contrastive learning is effective at distinguishing features, directly integrating it into existing models can lead to multi-task conflicts. To address this, we propose a phased weighted combination strategy. In the early stage of training, the model focus solely on the original loss achieve strong performance on the primary task. In the subsequent phase, blindspot feature alignment is gradually introduced through a weighted loss combination, enhancing its ability to recognize global invariant features, while the original task continues to guide the learning process. The proposed total loss is formulated as follows:
Liotal Lorigin + XLeontra,
where Lorigin represents the original loss the existing framework; is a weight hyperparameter that controls the contribution of blindspot-aware contrastive learning loss. During the pre-training
5

---

Table I: Robustness object classification. We report the classification accuracy (%) with four
noisy environments: Gaussian noise, rotation, scaling, and droppoint on ScanObjectNN,
Gaussian Noise
Rotation
Scaling
DropPoint
Dataset
Methods
#=0.01
e=0.03
X[-30 30]
Y(-30 30]
Z[-30 30]
(0.5, 1.5)
0.2
0.6
Point-MAE
77.5
47.2
72.1
87.6
72.5
86.2
87.4
84,9
+BlindFormer
81.8
60.6
77.3
90.5
77.3
89.3
90.7
89.7
T Improve
+4.3
+13.4
+5.2
+2.9
+4.8
+3.1
+3.3
+4.8
OBJ-BG
PointGPT-S
78.6
51.5
72.3
89.3
74.0
88.3
90.7
85.0
BlineFormer
81.8
57.8
76.8
91.9
79.2
90.4
91.4
86.1
1 Improve
+3.2
+6.3
+4.5
+2.6
+5.2
+2.1
+0,7
+1.1
Point-MAE
70.9
37.0
75.4
86.7
74.9
84.0
86.6
84.5
-llindFomerr
76.2
78.5
79.7
86.7
88.5
87.6
1 Improve
+5.3
+17.2
+3.1
+1.9
+4.8
+2.7
+1.9
+3.1
OBJ-ONLY
PointGPT-S
71.2
39.4
72.3
89.3
74.5
86.6
89.7
85.9
-BinnFormmr
73.3
41.3
79.9
92.3
81.8
90.0
91.2
87.4
1 Improve
+2.1
+1.9
+7.6
+3.0
+7.3
+3.4
+1.5
+1.5
Point-MAE
66.3
40.6
65.9
83.9
68.8
82.6
83.2
80.0
-lindFFrmer
69.4
43.9
68.4
85.4
70.0
84.5
84.9
82.0
T Improve
+3.1
+3.3
+2.5
+1.5
+1.2
+1,9
+1.7
+2.0
PB-T50-RS
PointGPT-S
63.2
34.9
63.1
85.6
67.0
84.1
86.1
82,0
BliinFormmr
66.5
38.8
63.8
87.0
68.2
85.6
86.7
83.5
t Improve +3.3
+0.7
+1.2
+1.5
+1.5
phase, Point-MAE's original pre-training loss Corigin is equivalent to the reconstruction loss Cre- For PointGPT, Lorigin refers to the generation loss Lge. Therefore, we optimize the joint learning objective with the phased weighted combination strategy, ensuring that the model not only achieves high-quality reconstructions (or generations) but also learns globally consistent feature representations. Through this strategy, BlindFormer exhibits strong potential for adaptability and scalability across wide range of multi-task learning scenarios, ultimately improving the model's overall performance.
Experiments
4.1 Experimental Setup
Datasets. We evaluate BlindFormer on three benchmarks in point cloud analysis. ScanOhjectNN 48
comprises approximately 15,000 real-world 3D objects from 15 categories derived from indoor RGB-
D scans, presenting challenges like background clutter, occlusions, and sensor noise. ModelNet40 [59] a synthetic dataset with 12,311 CAD models across 40 categories, split into 9,843 for training and 2,468 for testing. ShapeNetPart [63] contains 16,881 models across 16 categories, each annotated with part labels totaling 50 classes, enabling evaluation fine- areegga part segmentation.
Backbone models. To evaluate the seamless integration of the proposed method into existing Transoormer-based models for point cloud processing, apply it different backbone architectures, specifically Point-BERT, Point-MAE and PointGPT-S. Experimental results across various tasks indicate that the method is adaptable and enhances the performance of these Transformer architectures, thereby demonstrating its versatility and practical applicability.
Experimental details. Our input point clouds are obtained by sampling 1,024 points from each raw point cloud. Each point cloud then divided into patches with 32 points each. The BlindFormer model is pre-trained for total of 600 epochs: the first 300 epochs focus the original task alone, and the next 300 epochs incorporate both original pre- pre-training and contrastive learning objectives. We the Adam optimizer with an initial learning rate of 0.001, a weight decay of 0.05, and batch size of 128. The learning rate is adjusted using a cosine decay schedule. All experiments are implemented using the Py Porch framework and conducted on four NVIDIA V100 GPUs. More training strategy details and training cost analysis are provided in the Appendix 6.2
4.2 Robustness against point cloud perturbations
To assess the robustness of our BlindFormer framework, we conducted experiments object classification and part segmentation tasks under different noisy environments, including Gaussian noise, rotation, scaling, and point dropout.

---

Table 2: Robustness on part segmentation. We report the mean Intersection over Union (mloU) for all classes on Sarrt
Gaussian Noise
Rotation
Scaling
DropPoint
Methods
o=0.03
c=0.05
X[-30 30]
Y--30 30]
ZJ-30 30]
(0.5, 1.5)
0.6
Point-MAE
74.1
71.1
77.9
80.1
74.0
82.4
77.1
64.8
+BlindFormer
75.4
72.9
80.0
81.6
76.4
83.7
78.3
67.5
Improve +1.3
+1.8
+2.1
+1.5
+2.4
+1.3
+1.2
+2.7
PoinGGPT-S 71.5
68.2
75.9
79.1
74.5
80.6
76.5
65.0
BlinnFFormer 72.3
69.9
79.2
81.0
77.2
82.2
77.5
66.9
Improve +0.8
+1.7
+3.3
+1.9
+2.7
+1.6
+1.0
+1,9
Table 3: Object classification on ScanObjectNN and ModelNet40. We report the Top-1 classifi-
cation accuracy (%) of BlinnFormer with Point-MAE and PointGPT-S backbones respectively. On ScanObjectNN, * denotes using simple rotational augmentation for training. On ModelNet40, * denotes the results obtained by voting.
ScanObjectNN
Methods
Reference
ModelNer40
OBJ-RG
OBJ-ONLY
PR-TS0-RS
Supervised Learning Only
PointNet
CVPR
79.2
PointNct++
36
NeurIPS
82.3
84.3
77.9
90.2
PoinCCNN
NeurIPS
85.5
78.5
91.7
DGCNN 53
TOG 19
82.8
86.2
78.1
92.0
PRANet
8
TIP 21
81.0
92.9
MVTN IS
ICCV 21
82.8
93.8
PointNeXi
39
NeurlPS 22
87.7
92.9
PointMLP
29
ICLR 22
85.4
94.1
RepSurf-U
42
CVPR 22
84.3
93.8
ADS 18
ICCV 23
87.5
94.0
with Self- Representatioon Learning
MaskPoint
26
CVPR
89.3
88.1
84.3
92.6
Point-MZAR 67
NeurIPS
91.2
88.8
86.4
93.4
PointDif
CVPR
93.3
91.9
GPM 24
CVPR
90.2
90.0
84.8
PointMamba
25
NeurlPS
90.7
88.5
84.9
93.6
PCM 2
AAAI 25
88.1
93.4
Point-BERT 65
CVPR
87.4
88.1
83.1
92.7
+BlindFormer
89.5 (+2.1)
88.6 (+0.5)
84.5 (+1.4)
93.1 (+0.4)
Point-MAR
34
ECCV 22
90.0
88.3
85.2
93.2
+BlindFormer
90.9 (+0.9)
88.8 (+0.5)
85.4 (+0.2)
93.7 (+0.5)
PoinGGPT-S
NeurlPS
91.6
90.0
86.9
+BlindFormer
92.3 (+0.7)
91.6 (+1.6)
87.1 (+0.2)
93.5 (+0.2)
Point-MAF*
M
ECCV 22
92.8
91.2
89.0
93.8
+BlindFormer**
93.1 (+0.3)
91.7 (+0.5)
89.2 (+0.2)
94.1 (+0.3)
PoinIGPT-S*
NeurlPS
93.4
92.4
89.2
94.0
+BlindFormer"
94.5 (+1.1)
93.5 (+1.1)
89.9 (+0.7)
94.1 (+0.1)
Robustness to object classification. We conduct extensive experiments on the ScanObjectNN dataset across various noise settings in Table [ Compared state-of-the-an models, including PointMAE and PointGPT-S, our method consistently achieves superior classification accuracy in the OBJ-BG, OBJ-ONLY and PB-T50-RS settings. For instance, under Gaussian noise perturbation with = 0.03 on OBJ-ONLY, BlindFormer surpasses PointMAE by 17.2%. Additionally, under rotation perturbation around the x-axis, BlindFormer outperforms PointGPT-S by 7.6%. These results suggest that the enhanced robustness of BlindFormer originates from its attentional blindspot mining strategy with a joint learning objective. By encouraging attention to under-attended regions, this strategy fosters more comprehensive structural understanding, while blindspot-aware contrastive learning further improves feature discrimination. Consequently, BlindFormer significantly enhances the model's s resilience to noise and transformations.

---

Table 4: Few-shot classification ModelNet40. We Table 5: Part segmentation perfor- report the mean accuracy (%) with standard deviation over mance on the ShapeNetPart dataset. 10 independent experiments.
We report the mean Intersection over Union (mloU) across instances (Ins.) and classes (Cls.).
5-way
10-way
Methods
10-shot
20-shot
10-shot
20-shot
Methods
Ins. mloU Cls.mloU
Supervised Learning Only
Supervised Learning Only
PointNet
52.0+3.8
57.8+4.9
46.624.3
35.2+4.8
PointNet
83.7
80.4
SeiniNet-CrossPovi
90.9+1.9
93.5+4.4
64.644.7
90.2+2.2
PointNet++
85.1
81.9
DGCNN
31.6+2.8
40.8+4.6
19.9+2.1
16.9=1.5
DGCNN
85.2
82.3
DGCNN-CrossPe
92.5+3.0
94.9=2.1
83.625.3
87.9+4.2
with Self SuperSuie Representation Learning
with Self-Supervvied Representation Leurning
Point-BERT
85.6
84.1
GPM
85.8
84.2
MaskPoint
95.0+3.7
97.2*1.7
91.4+4.0
93.4+3.5
PointMamba
86.0
84.4
Point-M2AE
96.8+1.8
98.3#1.4
92.3+4.5
95.0=3.0
Point-BERT
85.6
84.1
+BiindFormer
86,0
84,6
Point-BERT
94.6+3.1
96,3+2.7
91 91.0+5.4
92.745.1
Improve
+0.4
+0.5
+BlindFormer
95.1+2.6
97.4=2.0
91.5+4.7
93.0=3.1
Point-MAE
86.
84,2
Point-MAE
96.342.5
97. 97.8+188
92.6+4.1
95.0=3.0
FBlindFormer
86.4
85.2
+BlindFormer
96.7+2.7
98.2+1.6
92.8+4.0
95.3+3.2
1 Improve
+0.3
+1.0
PoinGGPT-S
86.2
84.1
PoinGGPTSS
96.8+2.0
98.6#1.1
92.6+4.6
95.2+3,4
+BlindFormer
86.7
84.8
+BlindFormer 97.1+2.3
98.8+1.3
93.0+4.0
95.643.0
1 Improve
+0.5
+0.7
Robustness to part segmentation. We assess the robustness of the BlindFormer framework for part segmentation under various noise conditions using the ShapeNetPart dataset, The integration of BlindFormer into the backbone network significantly enhances segmentation performance across different perturbation scenarios. Notably, under rotation perturbation, our approach improves the class mloU of Point-GPT by 3.3%. These findings demonstrate that our contrastive attention learning strategy effectively preserves the model's 's reliable segmentation capability in noisy environments.
4.3 Downstream tasks for 3D understanding
Standard object classification. Table 3 compares our proposed BlindFormer method with existing approaches the ScanObjectNN and ModelNet40 datasets. Our BlindFormer consistently outper- forms these state-of-the-ard methods. Compared to Point-MAE [34], BlindFormer achieves higher accuracies by +0.9% +0.5%, and +0.2% on OBJ-BG, OBJ-ONLY, and PB-T50-RS, respectively. Against PointGPT-S [ BlindFormer attains improvements of +0.7%. +1.6%, and +0.2% on the same splits. Following recent work setting 255. BlindFormer sets new state-of-the- results, achiev- ing up to 94.5% on OBJ-BG, 93.5% on OBJ-ONLY and 89.9% on PB-T50-RS. On ModelNet40, BlindFormer achieves 93.7% accuracy without voting and 94.1% with voting, surpassing previous methods without adding additional parameters. These results demonstrate that BlindFormer promotes a comprehensive understanding and feature discrimination for point cloud, leading to improved model performance in standard object classification.
Few-shot object classification. Compared to previous approaches, BlindFormer consistently achieves higher accuracy on the ModelNet40 dataset under few-shot learning settings, and the results are presented in Table 4 In the 5-way 10-shot task, our method attains an accuracy of 97.1 1% with standard deviation of 2.3%, demonstrating superior generalization with limited labeled data.
Part segmentation. We evaluate the effectiveness of BlindFormer on the part segmentation task, as shown in Table[5] BlindFormer achieves superior performance compared to both traditional supervised models and self-supervised methods. Specifically, method attains instance mloU of 86.4% and a class mloU of 85.2% with Point-MAE. These results confirm the efficacy of BlindFormer in advancing the fate-of-the-art in point cloud segmentation.
4.4 Ablation Studies
We conduct extensive experiments with Point-MAE ScanObjectNN under Gaussian noise to validate the effectiveness of each component. Table summarizes the ablation study on different combinations for the OBJ-BG and OBJ-ONLY settings.

---

Table 6: Ablation studies of components in Foiner We report the overall accuracy (%) on ScanObjectNN with Point-MAE. The settings adopted by BlindFFormrr are marked
(a) Blindspot Mining Strategy and Loss Optimization Function.
Blindspot Mining Strategy
Lorigin
Leantr
OBJ-BG
OBJ-ONLY
NO Mining
47.2
37.0
Random Blindspot Mining
50.5
41.3
Attentional Blindspot Mining
56.9
50.4
Random Blindspot Mining
47.7
38.3
Attentional Blindspot Mining
53.8
47.5
Random Blindspot Mining Attentional Blindspot Mining
52.7
43.4
60.6
54.2
(b) Mask Ratio.
(c) Probability Temperature.
(d) Contrastive Loss Weight.
R
OBJ-BG
OBJ-ONLY
Tpro
OBI-BG
OBJ-ONLY
OBJ-BG
OBJ-ONLY
0.2
53.1
45.6
0.3
59.2
53.1
0.4
59.8
$3.5
0.4
57.8
52.1
0.5
60.6
54.2
0.6
60.6
54.2
0.6
60.6
54.2
0.7
60.1
53.8
0.8
58.2
52.
0.8
45.9
36.8
0.9
58.7
52.9
1.0
57.5
51.8
Blindspot mining strategy. As shown in Table a). we explore how different blindspot mining strategies affect performance. Without Mining, the baseline model achieves 47. 47.2% (OBJ-BG) and 37.0% (OBJ-ONLY) accuracy. The Random Blindspol Mining provides a moderate improvement, indicating that even coarse region perturbations help the model discover underutilized cues. However, the Attentional Blindspot Mining yields the best results, achieving 56.9% (OBJ-BG) and 50.4% (OBJ-ONLY) accuracy with the original loss Lorigin- Similar trends can also be observed when maintaining the same loss optimization function. By masking out high-attention regions, this strategy explicitly forces the model to improve comprehensive understanding and robustness by learning from less salient areas and enhancing its generalization capabilities.
Loss optimization function. We also analyze the effect of blindspot-awarr joint learning in Table Compared the baseline, applying contrastive learning alone with attentional blindspot mining already yields notable improvements (+6.6% on OBJ-BG and +10.5% on OBJ-ONLY). However, contrastive loss alone underperforms relative to the original task loss with attentional mining strategy. This is because Ceontr only focuses on aligning blindspot features across masked views, but lacks the fine-grained semantic supervision that Corigin provides for accurate reconstruction or generation. highlights the role of contrastive loss as a regularizer rather than a substitute. When both objectives are jointly optimized, we observe the best performance (60.6% on OBJ-BG and 54.2% on OBJ- ONLY). The original loss ensures task arik-triented fidelity, while the contrastive objective regularizes the learning process, further encouraging the model extract robust and invariant features across diverse blindspot regions. This synergy confirms that Blindspot-Awart Joint Optimization effectively leverages both local precision and global consistency, resulting superior representation learning.
Effect of hyperparameters. We further explore the effects of varying hyperparameter in BlindFormer. Regarding the masking ratio in Table b). the model achieves optimal performance at 0.6, yielding classification accuracies of 60.6% on OBJ-BG and 54.2% on OBJ-ONLY. However, a higher mask ratio (R 0.8) negatively impacts performance due excessive information loss, highlighting the need for a balance between data complexity and sufficient feature retention. As shown in Table shows the effect of the temperature Tpro in dynamic masking. Setting Tpro 0.5 achieves the best accuracy, as it enables selective masking of high-attention regions while preserving dynamic adaptability. Similarly, the contrastive loss weight in Table ([ crucial, with À = 0.6 providing the best trade-off between the original and contrastive loss components. This optimal weighting enhances feature discrimination and generalization while preserving task-specific accuracy.
Conclusion
In this work, we present BlindFormer, a robust self-supervised framework for point cloud understand- ing. By introducing an attentional blindspot mining strategy, BlindFormer identifies and dynamically masks high-attention regions, encouraging the model to explore under- antendd and informative

---

areas. Building upon this, the blindspot-awand joint optimization scheme effectively combines the strengths of task-speeific and contrastive objectives further enhance blindspot representation learn- ing. Extensive experiments challenging benchmarks validate the effectiveness of our approach, including object classification, few-shot learning, and part segmentation. These experimental results demonstrate that BlindFormer not only improves performance under standard conditions, but also exhibits strong resistance to perturbations. Our findings point toward the importance of attention regularization in 3D learning and provide foundation for more resilient and adaptive models.
Acknowledgments
This work was supported in part by the Joint Funds of the National Natural Science Foundation of China under Grant U24A20248. The work described in this paper was supported in part by the Research Grants Council of the Hong Kong Special Administrative Region, China, under Project T45-401/22-N and under Project CUHK 14200824.
References
[1] Mohamed Abdelfattah, Mariam Hassan, and Alexandre Alahi. Maskelr: Attention-guided contrastive learning for robust action representation Icarning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 18678-18687, 2024.
[2] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In International conference on machine learning, pages 40-49. PMLR, 2018.
[3] Fabio Arema, Mario Collotta, Giovanni Pau, and Francesco Termine. An overview of augmented reality. Computers, 11(2)288, 2022.
[4] Hangbo Bao, Li Dong. Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2 08254, 2021.
[5] Guangyan Chen, Meiling Wang, Yi Yang. Kai Yu. Li Yuan, and Yufeng Yue. Pointept: Auto-regressivel generative pre-training from point clouds. Advances in Neural Information Processing Systems, 36, 2024.
[6] Siheng Chen, Baoan Liu, Chen Feng, Carlos Yallespi-Gonzaler and Carl Wellington. 3d point cloud processing and learning for autonomous driving: Impacting map creation, localization, and perception. IEFE Signal Processing Magazine, 38( 1)68-66, 2020.
[7] Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1907-1915, 2017. 2017.
[8] Silin Cheng. Xiwu Chen, Xinwei He, Zhe Liu, and Xiang Bai. Pra-net: Point relation- aware network for 3d point cloud analysis. IEEE Transactions Image Processing, 30:4436-444 2021.
[9] Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1x0i::1100.0490000 2018.
[10] Runpei Dong, Zekun Qi. Linfeng Zhang, Junbo Zhang. Jianjian Sun, Zheng Ge, Li Yi, and Kaisheng Ma. Autoencoders as cross-modal teachers: Can pretrained 2d image transformers help 3d representation learning? arXiv preprint arXiv:22/2.00:22 2022.
[11] Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. Why does unsupervised pre-training help deep learning? In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 201 208. JMLR Workshop and Conference Proceedings. 2010.
[12] Tuo Feng, Wenguan Wang. Ruijie Quan, and Yi Yang. Shape2scene: 3d scene representation learning through pre-training shape data. arXiv preprint arXiv:2407. 10200, 2024.
[13] Daniel Garrido, Rui Rodrigues, A Augusto Sousa, Joao Jacob, and Daniel Castro Silva. Point cloud inter- action and manipulation in virtual reality. In 2021 5th International Conference on Artificial Intelligence and Virtual Reality (AIVR), pages 15-20, 2021.
[14] Ian Goodfellow, Jean Souget-Abadic Mehdi Mirza, Bing Xu, David Warde Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139 144, 2020.
10

---

[15] Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. Mvtn: Multi-view transformation network for 3d shape recognition. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 1-11. 2021.
[16] Xu Han, Yuan Tang, Zhaoxuan Wang, and Xianzhi Li. Mamba3d: Enhancing local features for 3d point cloud analysis via state space model, arXiv preprint arXiv:2404. 14966, 2024.
[17] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 16000-16009, 2022.
[18] Cheng- Yao Hong, Yu- Ying Chou, and Tyng-Luh Liu. Attention discriminant sampling for point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14429-14440, 2023.
[19] Tianyu Huang, Bowen Dong, Yunhan Yang. Xiaoshui Huang, Rynson WH Lau, Wanli Ouyang. and Wangmeng Zuo. Clip2point: Transfer clip to point cloud classification with image-depth pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22157-22167, 2023,
[20] Longlong Jing, Yucheng Chen, Ling Zhang, Mingyi He, and Yingli Tian. Self-supervised modal and view invariant feature learning. arXiv preprint arXiv:2005. 14169, 2020.
[21] Sihyeon Kim, Sanghyeok Lee, Dasol Hwang. Jaewon Lee, Seong Jae Hwang, and Hyunwoo J Kim. Point cloud augmentation with weighted local transformations. In Proceedings of the IEEE/CVF international conference compuder vision, pages 548-557, 2021.
[22] Jiaxin Li, Ben M Chen, and Gim Hee Lee. So-net: Self-organizing network for point cloud analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9397-9406, 2018.
[23] Yangyan Li, Rui Bu, Mingehao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenm: Convolution on t-transformed points. Advances in neural information processing systems, 31, 2018.
[24] Zhe Li, Zhangyang Gao, Cheng Tan, Bocheng Ren, Laurence T Yang. and Stan Z Li. General point model pretraining with autoencoding and autoregressive. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 20954-20964, 2024.
[25] Dingkang Liang, Xin Zhou, Wei Xu, Xingkui Zhu, Zhikang Zou, Xiaoqing Ye, Xiao Tan, and Xiang Bai. Pointmamba: A simple state space model for point cloud analysis. In Advances Neural Information Processing Systems, 2024.
[26] Haotian Liu, Mu Cai, and Yong Jac Lec. Masked discrimination for self superviedd learning on point clouds. In European Conference on Computer Vision, pages 657-675. Springer, 2022.
[27] Jiuming Liu, Ruiji Yu, Yian Wang. Yu Zheng, Tianchen Deng, Weicai Ye, and Hesheng Wang. Point mamba: A novel point cloud backbone based state space model with octree-based ordering strategy. arXiv preprint arXiv:2403.0 064677 2024.
[28] Luigi Tommaso Luppino, Michael Kampffmeyer, Filippo Maria Bianchi, Gabriele Moser. Sebastiano Bruno Serpico, Robert Jenssen, and Stian Normann Anfinsen. Deep image translation with an affinity- bassd change prior for unsupervised multimodal change detection. IEEE Transactions on Geoscience and Remote Sensing, 60:1 22, 2021.
[29] Xu Ma. Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local geometry in point cloud: A simple residual mip framework. arXiv preprint arXiv:2202.07123, 2022.
[30] Chris Maddison, Andriy Maih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint 1rXiv:161 1.00712,, 2016.
[31] Chen Min, Dawei Zhao, Liang Xiao, Yiming Nie, and Bin Dai. Voxel-mae: Masked autoencnders for pre-training large-scale point clouds, arXiv preprint arXiv:2206.009000 3, 2022.
[32] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext- invariatt representations. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 6707-6717, 2020.
[33] KL Navancet, Ansu Mathew, Shashank Kashyap. Wei-Chi Wei-Chih Hung, Varun Jampani, and R Venkatesh Babu. From image collections to point clouds with self supervied shape and pose networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1132 1140, 2020.
11

---

[34] Yatian Pang, Wenxiao Wang. Francis EH Tay. Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. In European conference on computer vision, pages 604-621. Springer, 2022.
[35] Charles Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning point for 3d classification and segmentation. Proceedings of the IEFE conference on computer vision and pattern recognition, pages 652-660, 2017.
[36] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnct++: Deep hicrarchical feature learning point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[37] Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining. In International Conference on Machine Learning, pages 28223-208 28243. PMLR, 2023.
[38] Zekun Qi Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, and Kaisheng Ma. Shapellm: Universal object understanding embodied interaction. European Conference on Computer Vision, pages 214-238. Springer, 2025.
[39] Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. Advances in neural information processing systems, 35:23192-23222 2022.
[40] Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang. Huisheng Wang. Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 6964-6974, 2021.
[41] Alec Radford. Improving language understanding by generative pre-training. 2018.
[42] Haoxi Ran, Jun Liu, and Chengjie Wang. Surface representation for point clouds. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 18942-18952, 2022.
[43] Jiawei Ren, Lingdong Kong, Liang Pan, and Ziwei Liu. Pointcloud-c: Benchmarking and analyzing point cloud perception robustness under coruptiinns. preprint, 3, 2022.
[44] Ayumu Saito and Jiju Poovvancheri. Point-jepa: A joint embedding predictive architecture for self- supervised learning on point cloud. arXiv preprint arXiv:2404. 16432, 2024.
[45] Justyna Sarzynska-Ware Aleksander Wawer, Aleksandra Pawlak, Julia Szymanowska, Izabela Stefaniak, Michal Jarkiewiez, and Lukasz Okruszek. Detecting formal thought disorder by deep contextualized word representations. Psychiatry Research, 904:114135, 2021.
[46] Jonathan Sauder and Bjarne Sievers. Self-supervised deep learning on point clouds by reconstructing space. Advances in Neural Information Processing Systems, 32, 2019.
[47] Desney S Tan, George G Robertson, and Mary Czerwinski. Exploring 3d navigation: combining speed- coupled flying with orbiting. In Proceedings of the SIGCHI conference Human factors in computing systems, pages 418 425, 2001.
[48] Mikacla Angelina Uy, Quang- Hicu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Ycung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In Proceedings of the IEEF/CVF international conference on computer vision, pages 1588-1597, 2019.
[49] Hanchen Wang. Qi Liu. Xiangyu Yue, Joan Lasenby, and Matt J Kusner. Unsupervised point cloud pre-training via occlusion completion. In Proceedings of the IEFE/CVF international conference on computer vision, pages 9782-9792, 2021.
[50] Jiaze Wang. Yi Wang, Ziyu Guo, Renrui Zhang, Donghao Zhou, Guangyong Chen, Anfeng Liu, and Pheng-Ann Heng. Mm-mixing: Multi-modal mixing alignment for 3d understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 7744 7752, 2025.
[51] Jie Wang, Lihe Ding, Tingfa Xu, Shaocong Dong, Xinli Xu, Long Bai, and Jianan Li. Sample-adaptive augmentation for point cloud recognition against real-world corruptions. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pages 14330-14339, 2023.
[52] Yi Wang, Jiaze Wang. Jinpeng Li. Zixu Zhao, Guangyong Chen, Anfeng Liu, and Pheng Ann Heng. Pointpatchmix: Point cloud mixing with patch scoring. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5686-5694, 2024.
12

---

[53] Yue Wang. Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein. and Justin M Solomon. Dynamic graph for learning on point clouds. ACM Transactions Graphics (log), 38(5):1-12, 2019.
[54] Zicheng Wang, Zhenghao Chen, Yiming Wu, Zhen Zhao, Luping Zhou, and Dong Xu. Pointramba: A hyhrid transformer-mamba framework for point cloud analysis. arXiv preprint arXiv:2405. 2024.
[55] Ziyi Wang, Xumin Yu, Yongming Rao, Jie Zhou, and Jiwen Lu. P2p: Tuning pre-trained image models for point cloud analysis with point-to-pixel prompting. Advances in neural information processing systems, 34:14388-14 14402, 2022.
[56] Xiaoyang Wu, Daniel DeTone, Duncan Frost. Tianwei Shen, Chris Xie, Nan Yang. Jakob Engel. Richard Newcombe, Hengshuang Zhao, and Julian Straub. Sonata: Self-supervised learning of reliable point representations. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 22193 22293-22204, 2025.
[57] Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao. Wanli Ouyang. Tong He. and Hengshuang Zhao. Point transformer v3: Simpler faster stronger. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4840-4851, 2024.
[58] Xiaoyang Wu, Xin Wen, Xihui Liu, and Hengshuang Zhao. Masked seene contrast: A scalable framework for unsupervised 3d representation learning. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 9415 9424, 2023.
[59] Zhirong Wu, Shuran Song. Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015.
[60] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre- ve--raining for 3d point cloud understanding, In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part III 16, pages 574-59) Springer, 2020.
[61] Le Xue, Mingfei Gao, Chen Xing, Roberto Martin- Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pages 1179 1189, 2023.
[62] Le Xue, Ning Yu, Shu Zhang, Artemis Panagopoulou, Junnan Li, Roberto Martin-Martin Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, et al. Ulip-2: Towards scalable multimodal pre-training for 3d understanding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 27091 122001, 2024.
[63] Li Yi. Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Transactions on Graphics (ToG), 35(6):1-12, 2016.
[64] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
[65] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattemn recognition, pages 193322, 2022.
[66] Yaohua Zha, Huizhen Ji, Jinmin Li, Rongsheng Li, Tao Dai, Bin Chen, Zhi Wang, and Shu-Tao Xia. Towards compact 3d representations via point feature enhancement masked autoencoders. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 6962-6970, 2024.
[67] Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li. Point-m2ae: multi-scal scale masked autoencoders for hicrarchical point cloud pre-training. Advances in neural information processing systems, 35:27061-27077 2022.
[68] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao. Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointelip: Point cloud understanding by clip. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 8552-8562 2022.
[69] Renrui Zhang, Liuhui Wang. Yu Qiao, Peng Gao, and Hongsheng Li. Learning 3d representations from 2d pre-trained models via amage-to-poing masked autoencoders. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 21769-211 21780,17000000 2023.
13

---

[70] Tao Zhang, Xiangtai Li. Haobo Yuan, Shunping Ji, and Shuicheng Yan. Point could mamba: Point cloud learning via state space model. arXiv preprint urXiv:2403.00762, 2024.
[71] Tao 7hang, Haobo Yuan, Lu Qi, Jiangning Zhang. Qianyu Zhou, Shunping Ji, Shuicheng Yan, and Xiangtai Li. Point cloud mamha: Point cloud learning via state space model. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pages 10121-10130, 2025.
[72] Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of 3d features any point-cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10252-10263, 2021.
[73] Xiao Zheng, Xiaoshui Huang, Guofeng Mei, Yuenan Hou, Zhaoyang Lyu, Bo Dai, Wanli Ouyang, and Yongshun Gong. Point cloud pre-training with diftusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22935 22945, 2024.
[74] Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Ziyao Zeng, Zipeng Qin, Shanghang Zhang, and Peng Gao. Pointclip v2: Prompting clip and gpt for powerful 3d worn-world learning. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 2639-2650, 2023.
[75] Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng. Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 291 1-2921, 2023.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have a Limatation Analysis in the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
14

---

The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper does not produce any new theorems, but application of existing theories.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper fully explains the composition and parameter settings the frame- work and provides relevant code in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (c.g., in the case
15

---

of a large language model), releasing of a model checkpoint, or other means that are
appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide relevant code in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. //nips.cc/pul cc/ppbblicgguidess Siaeeee//ooSSSeeeeeeeeee ssionnolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Relevant information can be found in the experimental section of the paper and the configuration file of the code.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We conduct experiments only once and report the accuracy the best model following previous work, and it would be too computationally expensive conduct the pre-training multiple times.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provide sufficient information on the computer resources in the experimental section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpps://euuriss.cee. http:///wrppp.ccccaaaaaaccccaaaa
Answer: [Yes]
17

---

Justification:
Our research complies with the NeurIPS ethical guidelines in all respects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential impacts in the conclusion section of paper and the Futuer Work in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
18

---

12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original paper that produced the datasets used in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwwpsswipppsseeeeeeeeeeeee thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new assets introduced in the paper are well documented and the documen- tation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
19

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
20

---

Appendix
6.1 Preliminary
Transformer-based self-supervised learning. Given a point cloud X E Rex3, we utilize Farthest Point Sampling (FPS) and K-Nearest Neighbors (KNN) algorithms identify n center points C and their corresponding & nearest neighbors, forming n point patches P. Following the previous methods 5) each point patch normalized to integrate local information. A lightweight token embedding module, implemented via PointNet, subsequently transforms these normalized local patches into trainable point tokens T. These point tokens, together with positional embeddings, are input into the transformer blocks to produce latent representations F. For different tasks, these latent representations are input into task-specific heads, where they are transformed into specific representations adapted to the task. The learning pipeline based on the Transformer architecture is as follows:
F Transforment
(8)
Had Spec. (F).
(9)
For Point-MAE, Headrask-SSppe denotes the reconstruction head. For PointGPT, Head Spec. denotes the prediction head.
6.2 Training Strategy with Cost Analysis
Given that BlindFormer determines mask patches based on the attention weights of the backbone network, suggest two strategies for obtaining these attention weights. The first strategy initializes the network with random attention and applies the attentional blindspot mining for adaptive attention refinement during subsequent training. Following standard protocol, the model undergoes pre- training for 300 epochs. This approach does not incur any additional training overhead. The second strategy, by contrast, employs attention learned from the standard branch for initialization, aiming dynamically adjust the model's dependencies in a targeted manner. This method necessitates 300 epochs of pre-training in the standard branch, followed by another 300 epochs in the dual branch, resulting in total of 600 epochs.
Furthermore, introduce BlindFormer during the fine-tuning phase of downstream tasks to further evaluate the scalability and effectivvneess our approach. Two strategies are employed here as well: one leverages the pre-trained attention for initialization, while the other requires additional 300 epochs training to obtain attention learned from the standard branch for initialization. should be noted that the dual-branch training framework results in twofold increase in the data processed per batch. Consequently, we introduce comparative baseline with a doubled batch size to ensure fair evaluation.
Table 7: Training cost analysis. We report the classification accuracy (%) on ScanObjectNN.
Pre-Train Pre-Training Epoch
Finetune Epoch
DataSet
Methods
300
600
300
600
Point-MAE
90.0
90.2
90.0
90.2
Formnr
90.5
90.5
90.5
90.9
1 Improve
+0.5
+0.3
+0.5
+0.7
OBJ-BG
PointGPT-S 91.6
91.7
91.6
91.9
+BlindFormer
91.9
91.9
92.1
92.3
1 Improve
+0.3
+0.2
+0.5
+0.4
Point-MAE 88.3
88.5
88.3
88.5
+BlindFormer
88.8
89.8
89,2
88.8
1 Improve
+0.5
+1.3
+0,9
+0.3
OBI-ONLY
PointGPT-S 90.0
90.2
90.2
90.2
BllinFFormer
90.5
91.4
90,9
91.6
Improve
+0.5
+1.2
+0.7
+1.4
21

---

100
Point-MAF
Point-MAE
Pout-MAE-BlindFomm
100
PomiMAA-BlindFemms
PointGP
PoinGPT
OBI-BG Classfication Top-1 Accuracy (%)
90
PoinGGP BennGPT-BindFFrmmet
ORI-ONLY Classification Top-1 Accurscy (%)
PoinCGP BlindFormer
80
80
70
60
50
5.0
Noise Stundard Deviation a
Noise Standard Deviation
Figure 3: Gaussian noise analysis on ScanObjeetNN, While the performance of existing methods decmidrules sharply with increasing Gaussian noise, this issue is mitigated by incorporating Blind- Former. Notably, when Point-MAE is used as the backbone network, our BlindFormer significantly enhances its robustness, resulting in minimal accuracy degradation.
Our experimental results, presented in Table 7 demonstrate the inherent advantages of our Blind- Former over existing approaches (such Point-MAE and PoinGGPT-S under the same training time, training batch size and training phase. For Point-MAE, with 300 pre-training epochs or 300 fine-tuning epochs, BlindFormmr achieves accuracy of 90.5% on the OBJ-BG dataset, surpassing Point-MAE's 90.0% by a margin 0.5%. This improvement persists when both methods are trained for 600 epochs during the fine-tuning phase, with BlindFormer reaching 90.9% accuracy compared to Point-MAE's 90.2%. Similarly, when evaluating against PointGPT-S, BlindFormer continues to exhibit superior performance. With both models trained for 300 fine-tuning epochs OBJ-ONLY, BlindFormer attains an accuracy of 90.9% compared to PoinGPT-S's 90.2%. Even when the train- ing epochs extended 600, BlindFormer maintains its advantage, achieving 91.6% accuracy, outperforming PointGPT-S by 1.4%. On the OBJ-BG dataset, a similar pattern is observed, where BlindFormer consistently outperforms PointGPT-S regardless of training duration.
The superior performance of BlindFormer across various datasets, training epochs, and application phases validates the efficacy of our framework. It demonstrates the performance gains of BlindFormer are not a consequence longer training times but are a direct result of designed framework contribu- crrns-namely. the attentional blindspot mining strategy with blindspot-aware joint optimization. By focusing on under-attender regions and enhancing feature discrimination, BlindFormer effectively captures both global and local features, leading to enhanced robustness and generalization.
6.3 Robutness Analysis
Robustness to noise corruptions. We further evaluate the robustness of method against existing approaches under Gaussian noise conditions using the OBJ-BG and OBJ-ONLY subsets of the ScanObjectNN dataset. To simulate noisy point clouds, we add Gaussian noise X (O00 o²) to all points, incrementally increasing the noise level by varying from 0 to 0.05 with step size 0.005. As illustrated in Figure while the accuracy all methods decmidrules as the noise standard deviation increases, BlindFormer exhibits slower performance degradation, demonstrating its superior ability to handle noisy point clouds. Notably, BlindFormer significantly improves the robustness of the Point-MAE backbone and outperforms baseline methods such as Point-MAE and PointGPT, particularly under extreme noise conditions (o = 0.05). This improvement can attributed to our attentional blindspot mining strategy, which encourages the model to focus on under-attended regions, thereby enhancing its capacity to capture comprehensive global structural information from point clouds. By not solely relying salient local features, BlindFormer mitigates sensitivity to
22

---

Early Stage I Feature Distribution
Transitional Fenture Distribution
Final Feature Distriburion
Figure 4: Feature distribution visualization on ModelNet40. Top: An overview the evolution of feature distributions across all 40 classes. Bottom: Detailed depiction of the evolution of feature distributions for selected typical classes.
noise-induced perturbations. Additionally, the integration of contrastive learning with the original task further refines feature discrimination, enabling the model distinguish subtle variations in data even under noisy conditions. The consistently strong performance across both the OBJ-BG and OBJ-ONLY datasets underscores the versatility and reliability of BlindFormer in diverse settings.
the standard setting 21 43 51, we conduct additional experiments to evaluate the robustness of BlindFormer against local corruptions, shown in Table 8 We consider two types local corruptions: (1) LocalDrop, which drops C' local clusters, and (2) LocalAdd, which adds C local clusters. Each cluster consists of K nearest points from randomly selected cluster center point. We used 100 in both cases.
Robustness to local corruptions. Following
By integrating BlindFormer, both Point- MAE and PointGPT-S demonstrate sig- nificant performance improvements under local corruptions. Specifically, under se- vere interference (C=9), BlindFormer en- hances Point-MAE by 5.2% for LocalDrop and 5.3% for LocalAdd, while improving PointGPT-S by 4. % for LocalDrop and 6.3% for LocalAdd. We attribute this im- provement to BlindFormer's ability to ac- tivate global structural attention, which not only enhances feature discrimination but also enables the model to learn more robust representations against local perturbations.
Table 8: Robustness to local corruptions ScanOb- jectNN.
LocalDrop
LocalAdd
Methods
C=3
C=6
C=9
C=3
C=6
C=9
Point-MAE +BlindFormer
87.9
82.1
83,5
81.4
90.4
88.7
87.3
88.8
86.4
82.6
Improve
+2.5
+5.2
+5.3
+5.0
+5.3
PoinGPT-S
88.5
86.6
84.0
87.7
84.5
79.9
FlinnForrmrr
91.7
90.7
Improve
+3.2
+3.2
+4.1
+3.0
+4.5
+6.3
real-world applications. 3D data often affected by noise from sensor inaccuracies and environ- mental factors, making BlindFormer's robustness various corruptions especially valuable. The performance under such conditions demonstrates its practicality for tasks where data quality is uncer- tain, underscoring the effectiveness of BlindFormer and its advantage existing Transformer-basee methods.
6.4 Feature Distribution Evaluation
Figure illustrates the evolution of the feature distribution using t-SNE during the fine-tuning of Fommer with Point-MAE as the backbone, on the ModelNet40 dataset. In the early stage feature distribution, the feature space is highly scattered with overlapping clusters, indicating that the backbone has not yet learned to effectively discriminate between different classes. As the backbone starts to align global representations from standard branch and blindspot branch based on attention- driven blindspot mining, the transitional feature distribution shows a notable improvement, with clusters becoming more distinct. However, there still remains some inter-class overlap.
23

---

Input
Point-MAE
PointGPT
BlindFormer
Input
Point-MAE
PointGPT
BlindFormer
Figure 5: Attention visualization Former with Point-MAE and PointGPT, Patches with high attention are closer to red, while patches with low attention are closer to blue. Point-MAE is employed as the backbone of our proposed BlindFormer.
In the final feature distribution, the clusters are well-separated and compact, reflecting highly discriminative feature space. The backbone has successfully learned to distinguish between different classes with a high degree of accuracy. The representative clusters at the bottom of each visualization further emphasize this progression, showing clear transition from mixed and overlapping clusters in the early stages to well-defines and isolated clusters the final stage. These visualizations highlight the effectiveness of the FlindFormer, demonstrating a clear trajectory of improvement in feature discrimination, culminating in a robust and well-defined feature space.
6.5 Qualitative Analysis
As shown in Figure 5 we visualize the classification heatmaps generated by Point-MAE, PointGPT, and our proposed BlindFormer, revealing significant distinctions in how these model attend to various regions of the point clouds. BlindFormer exhibits more balanced and comprehensive activation across both prominent and under-represented areas of the input data. Notably, on 2,468 ModelNet samples, Point-MAE exhibits highly concentrated attention distribution, with top 10% patches capturing 61. 61.2% of total attention weights. This excessive reliance small subset of high-attention regions limits the model's s ability to generalize and adapt to perturbations. In contrast, BlindFormer demonstrates more balanced attention distribution, requiring 36% of patches to reach the same cumulative attention weight. This redistribution indicates that BlindFormer effectively engages under-attended regions, reinforcing its ability to capture global structural information.
The observation aligns with our motivation the introduction-xxittiie Transformer-be based models tend to prioritize a limited set of salient regions, potentially neglecting latent information crucial for robust feature learning. By integrating our attentional blindspot mining strategy, BlindFormer compels the model to explore less prominent areas, leading to more comprehensive feature extraction. Furthermore, blindspot-aware contrastive learning refines feature discrimination, ensuring a more stable and generalized understanding of the point cloud. The richer and more evenly distributed activations BlindFormer's heatmaps further substantiate its ability to mitigate the limitations of prior methods, improving both robustness and generalization in 3D understanding.
6.6 Domain Adaptation Discussion
Although our work does not explicitly target domain adaptation, we find that BlindFormer improves generalization under input perturbations, which is conceptually aligned with the goals of domain generalization. For example, under Gaussian noise, categories with weaker original attention show significant accuracy gains (e.g., "display" from 38.10% 76. 19%, "sofa" from 54.76% to 80.95%). Additionally, BlindFormer better distinguishes between morphologically similar categories (e.g.. average accuracy of "desk" and "table" improves from 50.19% to 74.26%), demonstrating its robustness against subtle inter-class ambiguities.

---

These preliminary observations suggest that BlindFormer enables the model to handle challenging or less typical samples more effectively. This behavior naturally enhances resistance distribution shifts, as it reduces dependence on domain-specifee or spurious correlations. We believe such robustness makes BlindFormer promising candidate for future exploration in out-of-distribution and domain-adaptive settings.
6.7 Limatation Analysis
Despite the significant improvements achieved by BlindFormer, there are still areas that offer op- portunities for further enhancement. For example, while our method has been validated on specific datasets, applying it broader range of datasets could further demonstrate its generalizability and robustness. Additionally, although we have shown that BlindFormer integrates seamlessly with certain Transformer- based architectures, exploring its compatibility with an even wider variety of models could highlight its versatility even more. These considerations open avenues for future research to build upon our work and continue advancing the field of point cloud analysis.
6.8 Future Works
While the proposed BlindFormer framework has shown significant improvements in point cloud analysis tasks, there are several promising directions for future research further enhance its capabilities and applications.
One potential avenue is the integration of multi-modal data sources enrich point cloud repre- sentations. By incorporating complementary information from modalities such as images, textual descriptions, LiDAR intensity values, the model can leverage cross-modal correlations learn more comprehensive and robust feature embeddings. This multi-modal fusion could enhance the model's ' ability understand complex scenes and improve performance in several tasks. Another direction is the exploration of hierarchical or multi-scale feature learning within the BlindFormer framework. By capturing features various spatial resolutions, the model can better represent both local geometric details and global structural contexts. This enhancement could be particularly beneficial for handling large-scale point clouds or scenes with significant variations in point den- sities. Additionally, optimizing the computational efficiency of the attentional blindspot mining and blindspot-award aware joint optimization is an important consideration for real-wonf applications. Investigating lightweight architectures or efficient training strategies could make the model more suitable for deployment in resource-constrained environments, such as mobile robots or embedded systems used in autonomous driving. Lastly, applying the BlindFormer approach to other types of data representations, such as meshes or voxels, could broaden its applicability across different domains in 3D data processing. Exploring transfer learning techniques between these representations may also provide insights into shared structures and features among various 3D data forms.
By pursuing these future research directions, we aim to further advance the capabilities of Blind- Former, contributing to the development of more robust, efficient, and versatile models for point cloud analysis. These enhancements have the potential to impact wide range of applications, including robotics, augmented reality, virtual reality, and autonomous navigation, by enabling more accurate and comprehensive understanding of complex 3D environments.
25


Reviews:
[{"review_id": "VmRtG0HpM7", "reviewer": "Reviewer_yhBh", "summary": "In this paper, the authors propose BlindFormer, which addresses a critical limitation in point cloud Transformers: their tendency to overfocus on salient regions while neglecting less prominent areas. The framework introduces two core innovations:\n\n1. Attentional Blindspot Mining (ABM): Dynamically suppresses high-attention regions during training, forcing the model to explore \"blind spots\" to capture richer geometric cues.\n\n2. Blindspot-Aware Joint Optimization (BJO): Integrates contrastive learning with the original pretext task, enhancing feature discrimination without sacrificing primary task performance.\n\nEvaluated on benchmarks like ScanObjectNN, ModelNet40, and ShapeNetPart, BlindFormer consistently boosts robustness against noise (Gaussian, rotation, scaling, point dropout) and improves downstream task performance.", "questions": "Have you ever tried the method on other tasks?", "limitations": "yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. This paper identifies and mitigates attentional bias in point cloud Transformers, a previously underexplored issue critical for 3D understanding.\n2. The proposed method, though utilizing existing tools such as topk selection and contrastive learning, is still a novel solution, when previous methods usually add loss on the attention matrix.\n3. The method shows significant improvement on three datasets, and is compatible with multiple backbones (Point-MAE, PointGPT-S, Point-BERT) without architectural changes.\n4. Code is provided and I believe they will open-source them.\n\nWeakness:\n\n1. The blind spot problem can happen in all transformer backend architectures, especially for the common pre-training-then-finetuning paradigm. The authors should apply their method to other tasks to get more impact.\n2. In Equation 6, there is a $H_i^m$. Through the context, I guess that it is the feature of the masked tokens. It is better to explain it clearly in the text. Moreover, since there are two terms in Equation 6, their effect should be ablated in the experiment.\n3. When using contrastive loss, we usually want to see the improvement is from positive pairs or both positive and negative pairs. It would be good to provide such ablation study.", "was_revised": true, "final_justification": "The authors solved my concerns about the weakness of the paper. I tend to keep my rating as accept.\nMoreover, I think the method has potential on other task despite point cloud, such as LLM, MAE or any kinds of task with pretraining-finetuning paradigm.", "rebuttal": "We thank the reviewer for the positive evaluation and the valuable feedback. We believe that we have fully addressed your concerns and questions, and will incorporate all points mentioned below in the final version. We would be happy to address any further questions you might have.\n***\n> **Weakness** 1 & **Question** 1: Evaluation on other tasks.\n\nWe have evaluated ``object detection`` on ScanNetV2 by replacing 3DETR's encoder with our pre-trained encoder. Although there is a performance gap with methods dedicated to scene understanding, BlindFormer enhances scene comprehension of baseline through dynamic blindspot mining.\n\nTable A. 3D object detection on ScanNet v2.\n| Method | $AP_{25}$ | $AP_{50}$ |  \n| ---- | ---- | ---- | \n| Point-MAE | 63.2 | 42.8 |  \n| +BlindFormer | 64.1 | 43.5 |\n***\n> **Weakness** 2: Symbol explanation and effect.\n\nWe appreciate the reviewer’s suggestion and agree that the notation could be clearer. $H^m_i$ in Eq. 6 is indeed a typo and should be $H^b_i$, representing the blindspot branch feature. \n\nThe two terms in Eq. 6 enforce bidirectional alignment: blindspot to standard and standard to blindspot. Regarding the two terms, we experimented with removing each of them, and the results are reported in the Table B. This bidirectional alignment design ``prevents representation collapse``, ensuring that the standard branch learns from under-attended cues while the blindspot branch remains grounded in the original representation, resulting in richer and more robust features.\n\nTable B. Ablation studies of contrastive loss.\n| Method | OBJ-BG | OBJ-ONLY | \n| ---- | ---- | ---- | \n| Baseline | 47.2 | 37.0 | \n| Blindspot to standard alignment | 59.8 | 53.8 |  \n| Standard to blindspot alignment | 58.9 | 52.7 |\n| Bidirectional alignment | 60.6 | 54.2 |\n***\n> **Weakness** 3: Positive and negative pairs.\n\nWe thank the reviewer for the suggestion. Our contrastive loss in Eq. 6 uses both positive and negative pairs, where positive pairs enforce alignment between blindspot and standard branch features, and negative pairs ensure discrimination from other samples. Following your kind suggestion, we conducted an ablation in Table C comparing the use of positive pairs only versus both positive and negative pairs. The results confirms that ``both positive and negative pairs`` are essential for robust blindspot-aware representation learning.\n\nTable C. Ablation studies of positive and negative pairs.\n| Method | OBJ-BG | OBJ-ONLY | \n| ---- | ---- | ---- | \n| Baseline | 47.2 | 37.0 | \n| Positive pairs only | 57.2 | 51.4 |  \n| Positive and negative pairs | 60.6 | 54.2 |"}, {"review_id": "YBbW4XuUXr", "reviewer": "Reviewer_fgb3", "summary": "The paper proposes an attentional blindspot mining strategy, which enhances attention to overlooked regions in the point transformer. It also designs a contrastive loss combined with the original loss. The weight of the contrastive loss increases gradually during training. Experiments on various datasets and backbone models demonstrate that the attentional blindspot mining could improve the results.", "questions": "$L_{origin}$ is from the original method, e.g., Point-MAE. In ablation studies, how to compute the random blindspot mining and attentional blindspot mining for $L_{origin}$?\n\nCan the strategy of suppressing salient regions be applied to other transformer models, such as image transformers or natural languages?\n\nSuppressing salient regions contradicts the conventional transformer model. How to understand the phenomenon? Are there other strategies to improve transformer?", "limitations": "The limitation is not properly addressed, which should be added. Are there any cases whose accuracy decreases?", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe attentional blindspot mining enhances the focus on regions overlooked in standard transformer. It seems that the blindspot mining enhances could rectify transformer bias.\n\nThe contrastive learning combines previous transformer loss and the proposed blindspot feature, which improves the conventional point transformer model.\n\nWeaknesses:\n\nThough improvements are significant for individual examples, they are small for most examples. I consider the 17.2% increase is an exception, because the original 37%  accuracy is rather low.", "was_revised": true, "final_justification": "I appreciate the authors' rebuttal. Most of my concerns are addressed. I choose to keep my score.", "rebuttal": "We thank the reviewer for the positive evaluation and the valuable feedback. We believe that we have fully addressed your concerns and questions, and will incorporate all points mentioned below in the final version. We would be happy to address any further questions you might have.\n***\n> **Weakness** 1: Performance improvement.\n\nWe acknowledge that the +17.2% gain on OBJ-ONLY stems from a challenging baseline scenario, but this does not diminish its significance. The low baseline highlights the brittleness of standard Transformers under noise, where BlindFormer offers ``a substantial robustness improvement.`` Especially, we achieve these gains without architectural modifications or additional inference parameters. BlindFormer demonstrates consistent gains across all datasets and tasks, even when baseline performance is already strong, showing that the approach is not limited to extreme cases. This consistency across diverse conditions underpins the method’s generalizability and practical relevance.\n***\n> **Question** 1: The original loss in ablation studies.\n\nIn our ablation studies, $L_{origin}$ corresponds to loss in the original pre-training method (e.g., the reconstruction loss for Point-MAE). To evaluate the effect of blindspot mining strategies, we incorporate both Random Blindspot Mining (RBM) and Attentional Blindspot Mining (ABM) into BlindFormer’s dual-branch framework. RBM randomly masks a fixed proportion of points to simulate blindspot discovery without attention guidance. In contrast, ABM leverages the attention distribution to suppress salient regions to encourage learning from blindspot cues. They both ensure that the model needs to rely on the features of the mined blindspot region to reconstruct the complete point cloud, and calculate the original loss through $l_2$ Chamfer distance.\n***\n> **Question** 2: Extension of our approach.\n\nWe believe that the strategy of blindspot mining can be generalized to other Transformer-based models, but its impact depends on domain characteristics. Due to the lack of redundant information, over-focusing on a few salient regions is particularly harmful in point clouds. This motivates the emergence of BlindFormer, which reallocates attention to explore blindspot and capture more diverse geometric clues.\n\nIn domains such as images or natural language, when redundant information (irrelevant background) significantly deviates from the subject, ``high attention to core semantics is often beneficial.`` When they are somewhat related, occasionally suppressing these regions during training may encourage the model to exploit complementary context. Overall, our method still plays the role of attention regularization. By dynamically mining blindspot regions, the model can better exploit complementary information, thereby improving robustness to occlusion and noisy inputs.\n***\n> **Question** 3: Phenomenon understanding and strategy improvement.\n\n(1) BlindFormer complements rather than opposes Standard Transformers. Standard self-attention often over-concentrates on a few dominant cues, which can lead to overfitting and reduced robustness, especially for point clouds with limited redundancy. By dynamically masking salient regions, BlindFormer redistributes attention, encouraging integration of complementary under-attended cues for a more balanced representation. Especially, it does not completely negate the original attention bias - ``a more robust attention distribution`` is obtained while preserving the original task and representation contrast.\n\n(2) Other strategies to improve the Transformer. Prior work explores techniques such as attention dropout[1], label pruning[2], and loss-based regularization[3]. BlindFormer offers a principled, data-driven alternative by explicitly leveraging the model’s own attention distribution to identify and learn from blindspots, yielding more generalizable and robust point cloud representations.\n\n[1] Wu, Qiangqiang, et al. \"Dropmae: Masked autoencoders with spatial-attention dropout for tracking tasks.\" CVPR, 2023.\n\n[2] Park, Dongmin, et al. \"Robust data pruning under label noise via maximizing re-labeling accuracy.\" NeurIPS, 2023.\n\n[3] Zhai, Shuangfei, et al. \"Stabilizing transformer training by preventing attention entropy collapse.\" ICML, 2023.\n***\n> **Limitation** 1: Description of limitations.\n\nWe appreciate your suggestions on limitations. As mentioned in lines 624-631, exploration in more tasks and architectures will further demonstrate the generalizability of our method.\n\nFurthermore, we carefully checked all experiments and found that BlindFormer may not improve every category equally. For example, in OBJ-BG classification under Gaussian noise, although overall accuracy improved, a few categories with already highly effective original attention experienced slight drops (e.g., “chair” from 91.03% to 87.18%, “table” from 87.04% to 85.19%), likely due to attention redistribution reducing reliance on original salient cues. However, BlindFormer enables stronger generalization and robustness, leading to significant gains in categories where the original attention struggled (e.g., “bin” from 47.50% to 80.00%, “door” from 7.14% to 57.14%) and overall improved performance."}, {"review_id": "Dm6TUPubzJ", "reviewer": "Reviewer_oJo8", "summary": "The paper introduces BlindFormer, a masking strategy for blind spots in 3D point clouds to enhance the focus on non-salient regions, thus improve generalization after self-supervised pre-training. A self supervised auxiliary objective induces the model to focus on regions of the point cloud that would be otherwise ignored. A wide range of experiments are conducted to show its performance.", "questions": "1. Authors state that over-attending to certain areas of the point cloud make transformers more sensitive to noise or missing points in that area. But are we sure that they would still attend these regions if noise or missing points occurred? This is considering that attention uses all the information around instead of focusing on points independently.\n\n2. In Eq. 3, what happens if the mask corresponds only to the strength of the attention scores? Is the Uniform component really improving the diversity of the masking? \n\n3. What is H^m_i in Eq. 6? Is it an intermediate projection to contrast H^s_i? If so, this loss approximate InfoNCE. \n\n4. From lines 183 to 189, the authors mention that in the early stages of training, the model only focuses on the main task (original loss function), while the subsequent phase focuses on the blindspot feature alignment. This means that the term \\lambda in Eq. 7 will be modulated during training? This is not clear in the paper. An explanation comes in the experimental details, but then the \\lambda will not be used?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed ideas are interesting and correlate well with the motivations.\n2. BlindFormer is evaluated on two main tasks, and a few other downstream tasks. This, combined with using different datasets and pre-training strategies, makes the experimental part of the paper sound and rigorous. \n\nWeaknesses:\n\n1. The motivation is interesting, but there is a sense of lack of evidence or sustain that the negative effects of not attending to ignored regions really affect. \n2. The empirical results show small improvements in object classification, specially with respect to supervised alternatives (Table 3 w.r.t. ADS). The same happens against standard SSL methods in Tables 4 and 5, considering that standard deviation is relatively large.This issue puts the significance of the method into question.\n4. Minor writing errors need to be resolved. For example, line 193 \"we optimizes\".", "was_revised": true, "final_justification": "The authors replied to my concerns, and the paper proposes a interesting perspective for the pre-training of 3D models. I increase my score to borderline accept, hoping to be put into consideration in contrast with the other reviewers' final scores.", "rebuttal": "We thank the reviewer for the time and the valuable feedback. We believe that we have fully addressed your concerns and will incorporate the points mentioned below into the final version. We would be happy to address any further questions you might have.\n***\n> **Weakness** 1 & **Question** 1: Empirical evidence in support of motivation.\n\nWe appreciate the reviewer's concern and agree that empirical evidence is crucial. Our evidence is threefold:\n\n(1) Qualitative analysis of attention distribution. Baseline models exhibit highly concentrated attention, with a few points consistently dominating (lines 609-615). BlindFormer decentralizes attention by revisiting blindspot regions. Rather than discarding the original attention, our blindspot mining and joint optimization refine it to obtain ``a broader and more balanced distribution.``\n\n(2) Quantitative analysis of performance under perturbations. ``Standard Transformers show clear robustness issues in a variety of noisy environments.`` For example, on OBJ-BG with Gaussian noise $\\sigma$=0.03, Point-MAE achieves only 47.2% accuracy, while BlindFormer improves it by 13.4%.   This shows that the standard transformer's inherent attention bias exacerbates performance degradation when some regions are noisy or partially missing. Thanks to blindspot mining, BlindFormer encourages the model to exploit a wider range of information cues to solve this problem.\n\n(3) Theoretical justification. Attention adapts gradually and excessive focus leads to sparse features, reducing redundancy and making models brittle. BlindFormer acts as a regularizer, encouraging ``richer and more balanced perceptual representations``, thereby improving generalization and robustness.\n***\n> **Weakness** 2: Performance improvement.\n\n(1) Consistent improvements across settings. Although individual improvements may seem small, BlindFormer shows consistent improvements over supervised learning and self-supervised backbone models across ``all benchmarks and tasks``, demonstrating its generalizability rather than being tuned for a specific dataset.\n\n(2) Robustness under perturbations. The improvements are more significant under challenging conditions. For example, under Gaussian noise ($\\sigma$=0.03), BlindFormer improves the classification performance of Point-MAE on OBJ-ONLY by 17.2%; under rotation perturbations, it improves the segmentation performance of PointGPT-S on ShapeNetPart by 3.3%, demonstrating its value ``in real-world noisy environments.``\n***\n> **Weakness** 3: Minor writing errors.\n\nThank you for your careful reading and kind feedback. We have re-reviewed the entire paper and corrected some minor writing errors to ensure clarity and accuracy in the final version.\n***\n> **Question** 2: The role of perturbation probability.\n\nWe would like to clarify the role of dynamic blindspot generation. If masking relies solely on attention scores, it repeatedly targets the same high-attention regions, reducing diversity and causing over-regularization. The perturbation probability adds randomness, ensuring that less-salient regions also have a chance to be masked. This promotes exploration of a wider range of cues. The probability temperature in Table 6(c) in the paper is essentially a balance between the effects of attention probability and perturbation probability. To make it more explicit, the ablation results after removing the perturbation probability are shown in Table A. The consistent drop in performance under this configuration confirms the importance of the perturbation variable in improving generalization and robustness.\n\nTable A. Ablation studies of perturbation probability.\n| Method | OBJ-BG | OBJ-ONLY | \n| ---- | ---- | ---- | \n| Baseline | 47.2 | 37.0 | \n| ABM w/o perturbation | 59.0 | 52.6 |  \n| ABM w/ perturbation | 60.6 | 54.2 | \n***\n> **Question** 3: Symbol explanation.\n\nWe apologize for the typo in Eq. 6 — $H^m_i$ should be $H^b_i$, representing the blindspot branch feature. The contrastive loss is a symmetric InfoNCE-style objective between the blindspot ($H^b_i$) and standard representations ($H^s_i$). Its purpose is to align representations learned from complementary attention distributions, encouraging the model to explore blindspot regions while maintaining consistency with the standard view. Thank you for pointing this out, and we will double-check the writing of the paper in the final version.\n***\n> **Question** 4: Contrastive loss weight.\n\nWe would like to explain in detail your doubts about contrastive loss weight. $\\lambda$ in Eq. 7 is modulated during training following a two-phase schedule. In the early stage, $\\lambda$ is set to 0 so that the model focuses on the original loss $L_{origin}$, ensuring stable representation learning. As training progresses, $\\lambda$ is set to 0.6 to incorporate the blindspot-aware contrastive loss $L_{contra}$, enabling the model to align blindspot and standard features. The specific values of $\\lambda$ are detailed in the ablation studies Table 6(d), where we show that optimal weighting leads to the trade-off between task performance and blindspot feature alignment."}, {"review_id": "H7on9ElJlE", "reviewer": "Reviewer_mAUX", "summary": "This manuscript proposes a novel contrastive learning framework for object-level point cloud transformers, where the core idea is to suppress the salient attention regions, forcing the model to expand its perceptual field and therefore increasing the robustness. Specifically, the patches with high attention scores in the last transformer layer during the first forward pass are occluded during the second pass. Results of the two passes are forced to be similar through contrastive learning. Experiments demonstrate consistent improvements on 3 different backbones.", "questions": "1.\tA fair baseline: As discussed in weakness 2, your framework doubles the training computation per data sample, which is equivalent to doubling the batch size. IMHO the proper baselines are the baseline models with twice the batch size. It is important to separate your contribution apart from the increment that is brought by simply using a greater batch size, possibly through some experiments.\n\n2.\tAdditional comparison methods: How does your framework compare with other occlusion-oriented pretraining methods, such as MSC [2] and Sonata[3]? Note that this question is optional and omitting this question does not negatively affect my final scores.\n\n3.\tCost analysis: Can you provide detailed VRAM and time analysis compared to baselines?\n\n4.\tDesign choice questionable: As discussed in weakness 3, the claims from L143 to L147 is not well supported. Can you make a mathematical derivation of the properties of your dynamic blindspot generation strategy?\n\nOverall, the proposed method is original and novel, but not without problems such as cost analysis, fairness, and design choices. The strengths outweigh the weaknesses, which grants a borderline accept. Although I do hope this paper can be made better through drawing experiences of other fields (weakness 4), this will be out of scope for this paper. I would keep the right to adjust my rating according to the author response.\n\n[2] Xiaoyang Wu, Xin Wen, Xihui Liu, and Hengshuang Zhao. Masked scene contrast: A scalable framework for unsupervised 3d representation learning. In CVPR, 2023.\n\n[3] Xiaoyang Wu, Daniel DeTone, Duncan Frost, Tianwei Shen, Chris Xie, Nan Yang, Jakob Engel, Richard Newcombe, Hengshuang Zhao, and Julian Straub. Sonata: Self-supervised learning of reliable point representations. arXiv preprint arXiv:2503.16429, 2025 (Accepted in CVPR 25)", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe paper is clearly written and well structured. The core idea is exceptionally novel and provides valuable insight to existing methods.\n\n2.\tThe core finding is very valuable. The authors discover that over-reliance on several high attention patches affects model robustness. This has highlighted the importance of analyzing attention score distribution on point cloud transformers, which is not very popular in the point cloud transformer community to the best of my knowledge, compared to the wide adoption of such analysis in VLM community.\n\n3.\tThe findings are solid and well backed up by ablation and extensive experiments on various benchmarks.\n\nWeaknesses:\n\n1.\tCost analysis is almost absent. From my understanding, the pipeline doubles the training computation cost compared to baseline models due to the two-pass training pipeline. When adopting the learned attention initialization strategy (Appendix L524), a complete additional training is also required. While the authors provide the performance superiority in the so-called cost analysis of Tab. 7, no actual VRAM costs or time costs are mentioned.\n\n2.\tDue to the same reason as W1, the baselines are not completely fair. Your framework doubles the training computation per data sample, which is equivalent to doubling the batch size. A fair baseline with double the batch size is needed. See question 1 for details.\n\n3.\tEq. 3 design is weired (this is not a major weakness). Eq. 3 simplifies to $p_{dy}=log(-softmax(S/\\tau) / log(\\epsilon))$, where $\\epsilon \\sim U[0,1]$. This means that every softmax significance score with range $(0, 1)$ is randomly scaled by a factor $ -1/log(\\epsilon)$ which is of range $(0,+\\infty)$. Therefore, the effect of noise is significantly higher than original softmax significance scores. I do not know whether such high level of randomness is vital for your design, does this mean that your attention-guided masking strategy is actually similar to random masking? \n\n4.\tDesigns can be drawn from other communities (this is not a major weakness, more like a piece of advice). In the ViT community, such occasional bloated attention scores are called attention artifacts, and their effect has been determined as the rendezvous point to aggregate global information, since their local geometry is the least important and can be charged for other purposes with little loss. All tokens will have high attention scores with these tokens. Furthermore, adding a `register token’ [1] can restore the geometrical discriminativeness of attention maps. Maybe their experience can be applied to solve your problem.\n\n[1] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309.16588, 2023.", "was_revised": true, "final_justification": "My major concerns have been adequately solved in the discussion period. While Q1 remains, it is minor and does not affect the final rating, and I have suggested the authors include such experiments in the revision. After checking with other reviewers' opinions, I would remain the current rating on this paper (4, borderline accept).", "rebuttal": "We thank the reviewer for the positive evaluation and the valuable feedback. We believe that we have fully addressed your concerns and questions, and will incorporate all points mentioned below in the final version. We would be happy to address any further questions you might have.\n***\n> **Weakness** 1 & **Question** 3: Detailed cost analysis.\n\nWe appreciate the reviewer’s concerns regarding computational cost and have conducted a more detailed analysis.\n\n(1) Learned Attention Initialization. Acquiring learned attention indeed requires additional training (Appendix Line 524). However, for fairness, we apply the same training configuration to the baseline. The 600-epoch results in Table 7 all use learned attention initialization (The first 300 rounds are used for initial attention learning). Regardless of whether the attention initialization is random or learned, BlindFormer ``consistently outperforms the baseline`` under identical training schedules, demonstrating that the gains stem from blindspot mining rather than longer training.\n\n(2) VRAM usage. Using gpustat to monitor GPU memory at the same batch size (bs=32), BlindFormer occupies 11,119 MB, compared to 11,043 MB for Point-MAE, a marginal increase of about 0.7%. This includes ``all runtime allocations`` (model parameters, optimizer states, activation buffers, and CUDA memory fragmentation), not just model weights.\n\n(3) Training time. Due to the dual-branch design, BlindFormer incurs approximately 2× training overhead. For instance, on OBJ-BG classification training, one epoch takes 43s versus 22s for Point-MAE. Crucially, this additional cost is confined to training and ``does not affect inference speed``, making the method practical for deployment. For a fairer comparison, we explain this in detail in the next weakness.\n***\n> **Weakness** 2 & **Question** 1: Fair comparison of data samples.\n\nWe thank the reviewer for raising this concern. To further clarify, we conduct additional experiments on Point-MAE and PointGPT-S with double batch size under the same training schedule, as shown in Table A. The results on OBJ-BG show only marginal gains (+0.2% on Point-MAE and +0.3% on PointGPT-S), below the improvements obtained by BlindFormer (+0.9% and +0.7%, respectively). Similar results are observed on OBJ-ONLY. This confirms that BlindFormer’s performance boost arises from its blindspot mining and feature alignment mechanisms, rather than increased batch size. Moreover, BlindFormer’s inference cost remains identical to the baseline, as the blindspot branch is only used in training.\n\nTable A. Performance with the same data sample size.\n| Method | OBJ-BG | OBJ-ONLY | \n| ---- | ---- | ---- | \n| Point-MAE w/o 2x data | 90.0 | 88.3 | \n| Point-MAE w/ 2x data | 90.2 | 88.5 |  \n| +BlindFormer | 90.9 | 88.8 | \n| PointGPT-S w/o 2x data | 91.6 | 90.0 |  \n| PointGPT-S w/ 2x data | 91.9 | 90.2 |  \n| +BlindFormer | 92.3 | 91.6 | \n***\n> **Weakness** 3 & **Question** 4: Design choice of perturbation probability.\n\nWe appreciate your simplification of our formula and would like to clarify the role of the perturbation term.\n\n(1) Derivation of probability distribution. Your series of derivations are reasonable. Please allow us to continue the derivation from the perspective of probability. For a random number $\\epsilon \\sim U[0, 1]$, the cumulative distribution function $F_Y(y)=P(-log\\epsilon \\le y) = P(\\epsilon \\ge e^{-y}) = 1- P(\\epsilon \\le e^{-y}) = 1- e^{-y}$. In other words, although the range of $-log\\epsilon$ is theoretically $(0, +\\infty)$, ``it is highly concentrated``. (e.g. $P(-log\\epsilon \\le 3)\\approx0.95$, $P(-log\\epsilon \\le 7)\\approx0.999$). In practice, the attention score is still the dominant variable, and the perturbation only slightly alters mask selection to promote diversity.\n\n(2) Additional theoretical explanation. The perturbation term $-log(-log\\epsilon)$ is called ``Gumbel noise``, which is a random variable sampled from the Gumbel distribution. It is widely used in reinforcement learning and generative models to introduce randomness. Given our core contribution is not an improvement based on this, more detailed mathematical theory and related work can be referred to [1, 2]. Thank you again for your concern. We will add relevant literature to the final version to facilitate readers' understanding.\n\n[1] Maddison, et al. \"The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables.\" ICLR, 2017.\n\n[2] Oberst, Michael, and David Sontag. \"Counterfactual off-policy evaluation with gumbel-max structural causal models.\" ICML, 2019.\n***\n> **Weakness** 4: Design inspiration from the ViT community.\n\nWe thank the reviewer for their valuable suggestions. As you mentioned, in the ViT community, adding registration markers has been shown to mitigate the loss of geometric discriminability. BlindFormer approaches this issue from a different perspective, encouraging the model to explore blindspots in sparse point clouds. Ultimately, they both achieve the common goal of providing smoother feature maps and attention maps for downstream visual processing. We agree that integrating these ideas will further improve our framework. We will consider studying registration markers as a promising extension in future work.\n***\n> **Question** 2: Additional comparison methods.\n\nWe thank the reviewer for pointing out related occlusion-oriented pretraining works. MSC and Sonata focus on improving pretraining efficiency and mitigating the geometric shortcut problem, respectively. Both methods combine reconstruction and contrastive objectives, demonstrating excellent performance in 3D scene understanding. BlindFormer specifically addresses attentional blindspots by dynamically reallocating attention to under-explored regions, improving robustness to occlusion and sparsity. We view our blindspot mining as complementary rather than competitive and will include a discussion of MSC and Sonata to the related work section in the final revision.\n***\n> **Paper Formatting Concerns**: Formula optimization.\n\nThank you for your suggestions for revisions to the formulas and symbols. We have re-reviewed the entire paper and corrected some minor writing errors to ensure clarity and accuracy in the final version."}]

Author rebuttal:

Meta-review: The authors did a great job in rebuttal and discussion period, addressing most of the concerns by reviewers. 

Reviewrs think the paper introduces an interesting perspective for the pre-training of 3D models. 

The paper should be revised based upon incorporating the comments, added experiments/analysis, especially method clarity and fairness etc.
Decision: Accept (poster)