Title: What We Miss Matters: Learning from the Overlooked in Point Cloud Transformers
Venue: NeurIPS 2025
Accepted: yes
Keywords: Point Cloud Understanding

Representation Learning
Abstract: Point Cloud Transformers have become a cornerstone in 3D representation for their ability to model long-range dependencies via self-attention. However, these models tend to overemphasize salient regions while neglecting other informative regions, which limits feature diversity and compromises robustness.
To address this challenge, we introduce BlindFormer, a novel contrastive attention learning framework that redefines saliency by explicitly incorporating features typically neglected by the model. The proposed Attentional Blindspot Mining (ABM) suppresses highly attended regions during training, thereby guiding the model to explore its own blind spots. This redirection of attention expands the model’s perceptual field and uncovers richer geometric cues.
To consolidate these overlooked features, BlindFormer employs Blindspot-Aware Joint Optimization (BJO), a joint learning objective that integrates blindspot feature alignment with the original pretext task. BJO enhances feature discrimination while preserving performance on the primary task, leading to more robust and generalizable representations.
We validate BlindFormer on several challenging benchmarks and demonstrate consistent performance gains across multiple Transformer backbones. Notably, it improves Point-MAE by +13.4\% and PointGPT-S by +6.3\% on OBJ-BG under Gaussian noise. These results highlight the importance of mitigating attentional biases in 3D representation learning, revealing BlindFormer’s superior ability to handle perturbations and improve feature discrimination.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript proposes a novel contrastive learning framework for object-level point cloud transformers, where the core idea is to suppress the salient attention regions, forcing the model to expand its perceptual field and therefore increasing the robustness. Specifically, the patches with high attention scores in the last transformer layer during the first forward pass are occluded during the second pass. Results of the two passes are forced to be similar through contrastive learning. Experiments demonstrate consistent improvements on 3 different backbones.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThe paper is clearly written and well structured. The core idea is exceptionally novel and provides valuable insight to existing methods.\n\n2.\tThe core finding is very valuable. The authors discover that over-reliance on several high attention patches affects model robustness. This has highlighted the importance of analyzing attention score distribution on point cloud transformers, which is not very popular in the point cloud transformer community to the best of my knowledge, compared to the wide adoption of such analysis in VLM community.\n\n3.\tThe findings are solid and well backed up by ablation and extensive experiments on various benchmarks.\n\nWeaknesses:\n\n1.\tCost analysis is almost absent. From my understanding, the pipeline doubles the training computation cost compared to baseline models due to the two-pass training pipeline. When adopting the learned attention initialization strategy (Appendix L524), a complete additional training is also required. While the authors provide the performance superiority in the so-called cost analysis of Tab. 7, no actual VRAM costs or time costs are mentioned.\n\n2.\tDue to the same reason as W1, the baselines are not completely fair. Your framework doubles the training computation per data sample, which is equivalent to doubling the batch size. A fair baseline with double the batch size is needed. See question 1 for details.\n\n3.\tEq. 3 design is weired (this is not a major weakness). Eq. 3 simplifies to $p_{dy}=log(-softmax(S/\\tau) / log(\\epsilon))$, where $\\epsilon \\sim U[0,1]$. This means that every softmax significance score with range $(0, 1)$ is randomly scaled by a factor $ -1/log(\\epsilon)$ which is of range $(0,+\\infty)$. Therefore, the effect of noise is significantly higher than original softmax significance scores. I do not know whether such high level of randomness is vital for your design, does this mean that your attention-guided masking strategy is actually similar to random masking? \n\n4.\tDesigns can be drawn from other communities (this is not a major weakness, more like a piece of advice). In the ViT community, such occasional bloated attention scores are called attention artifacts, and their effect has been determined as the rendezvous point to aggregate global information, since their local geometry is the least important and can be charged for other purposes with little loss. All tokens will have high attention scores with these tokens. Furthermore, adding a `register token’ [1] can restore the geometrical discriminativeness of attention maps. Maybe their experience can be applied to solve your problem.\n\n[1] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309.16588, 2023.\n\nQUESTIONS:\n1.\tA fair baseline: As discussed in weakness 2, your framework doubles the training computation per data sample, which is equivalent to doubling the batch size. IMHO the proper baselines are the baseline models with twice the batch size. It is important to separate your contribution apart from the increment that is brought by simply using a greater batch size, possibly through some experiments.\n\n2.\tAdditional comparison methods: How does your framework compare with other occlusion-oriented pretraining methods, such as MSC [2] and Sonata[3]? Note that this question is optional and omitting this question does not negatively affect my final scores.\n\n3.\tCost analysis: Can you provide detailed VRAM and time analysis compared to baselines?\n\n4.\tDesign choice questionable: As discussed in weakness 3, the claims from L143 to L147 is not well supported. Can you make a mathematical derivation of the properties of your dynamic blindspot generation strategy?\n\nOverall, the proposed method is original and novel, but not without problems such as cost analysis, fairness, and design choices. The strengths outweigh the weaknesses, which grants a borderline accept. Although I do hope this paper can be made better through drawing experiences of other fields (weakness 4), this will be out of scope for this paper. I would keep the right to adjust my rating according to the author response.\n\n[2] Xiaoyang Wu, Xin Wen, Xihui Liu, and Hengshuang Zhao. Masked scene contrast: A scalable framework for unsupervised 3d representation learning. In CVPR, 2023.\n\n[3] Xiaoyang Wu, Daniel DeTone, Duncan Frost, Tianwei Shen, Chris Xie, Nan Yang, Jakob Engel, Richard Newcombe, Hengshuang Zhao, and Julian Straub. Sonata: Self-supervised learning of reliable point representations. arXiv preprint arXiv:2503.16429, 2025 (Accepted in CVPR 25)\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces BlindFormer, a masking strategy for blind spots in 3D point clouds to enhance the focus on non-salient regions, thus improve generalization after self-supervised pre-training. A self supervised auxiliary objective induces the model to focus on regions of the point cloud that would be otherwise ignored. A wide range of experiments are conducted to show its performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The proposed ideas are interesting and correlate well with the motivations.\n2. BlindFormer is evaluated on two main tasks, and a few other downstream tasks. This, combined with using different datasets and pre-training strategies, makes the experimental part of the paper sound and rigorous. \n\nWeaknesses:\n\n1. The motivation is interesting, but there is a sense of lack of evidence or sustain that the negative effects of not attending to ignored regions really affect. \n2. The empirical results show small improvements in object classification, specially with respect to supervised alternatives (Table 3 w.r.t. ADS). The same happens against standard SSL methods in Tables 4 and 5, considering that standard deviation is relatively large.This issue puts the significance of the method into question.\n4. Minor writing errors need to be resolved. For example, line 193 \"we optimizes\".\n\nQUESTIONS:\n1. Authors state that over-attending to certain areas of the point cloud make transformers more sensitive to noise or missing points in that area. But are we sure that they would still attend these regions if noise or missing points occurred? This is considering that attention uses all the information around instead of focusing on points independently.\n\n2. In Eq. 3, what happens if the mask corresponds only to the strength of the attention scores? Is the Uniform component really improving the diversity of the masking? \n\n3. What is H^m_i in Eq. 6? Is it an intermediate projection to contrast H^s_i? If so, this loss approximate InfoNCE. \n\n4. From lines 183 to 189, the authors mention that in the early stages of training, the model only focuses on the main task (original loss function), while the subsequent phase focuses on the blindspot feature alignment. This means that the term \\lambda in Eq. 7 will be modulated during training? This is not clear in the paper. An explanation comes in the experimental details, but then the \\lambda will not be used?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-20"}

comments: SUMMARY:
The paper proposes an attentional blindspot mining strategy, which enhances attention to overlooked regions in the point transformer. It also designs a contrastive loss combined with the original loss. The weight of the contrastive loss increases gradually during training. Experiments on various datasets and backbone models demonstrate that the attentional blindspot mining could improve the results.

STRENGTHS AND WEAKNESSES:
Strengths:

The attentional blindspot mining enhances the focus on regions overlooked in standard transformer. It seems that the blindspot mining enhances could rectify transformer bias.

The contrastive learning combines previous transformer loss and the proposed blindspot feature, which improves the conventional point transformer model.

Weaknesses:

Though improvements are significant for individual examples, they are small for most examples. I consider the 17.2% increase is an exception, because the original 37%  accuracy is rather low.

QUESTIONS:
$L_{origin}$ is from the original method, e.g., Point-MAE. In ablation studies, how to compute the random blindspot mining and attentional blindspot mining for $L_{origin}$?

Can the strategy of suppressing salient regions be applied to other transformer models, such as image transformers or natural languages?

Suppressing salient regions contradicts the conventional transformer model. How to understand the phenomenon? Are there other strategies to improve transformer?

LIMITATIONS:
The limitation is not properly addressed, which should be added. Are there any cases whose accuracy decreases?
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-22

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose BlindFormer, which addresses a critical limitation in point cloud Transformers: their tendency to overfocus on salient regions while neglecting less prominent areas. The framework introduces two core innovations:\n\n1. Attentional Blindspot Mining (ABM): Dynamically suppresses high-attention regions during training, forcing the model to explore \"blind spots\" to capture richer geometric cues.\n\n2. Blindspot-Aware Joint Optimization (BJO): Integrates contrastive learning with the original pretext task, enhancing feature discrimination without sacrificing primary task performance.\n\nEvaluated on benchmarks like ScanObjectNN, ModelNet40, and ShapeNetPart, BlindFormer consistently boosts robustness against noise (Gaussian, rotation, scaling, point dropout) and improves downstream task performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. This paper identifies and mitigates attentional bias in point cloud Transformers, a previously underexplored issue critical for 3D understanding.\n2. The proposed method, though utilizing existing tools such as topk selection and contrastive learning, is still a novel solution, when previous methods usually add loss on the attention matrix.\n3. The method shows significant improvement on three datasets, and is compatible with multiple backbones (Point-MAE, PointGPT-S, Point-BERT) without architectural changes.\n4. Code is provided and I believe they will open-source them.\n\nWeakness:\n\n1. The blind spot problem can happen in all transformer backend architectures, especially for the common pre-training-then-finetuning paradigm. The authors should apply their method to other tasks to get more impact.\n2. In Equation 6, there is a $H_i^m$. Through the context, I guess that it is the feature of the masked tokens. It is better to explain it clearly in the text. Moreover, since there are two terms in Equation 6, their effect should be ablated in the experiment.\n3. When using contrastive loss, we usually want to see the improvement is from positive pairs or both positive and negative pairs. It would be good to provide such ablation study.\n\nQUESTIONS:\nHave you ever tried the method on other tasks?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}