Title: How Many Tokens Do 3D Point Cloud Transformer Architectures Really Need?
Venue: NeurIPS 2025
Accepted: yes
Keywords: point cloud representation

token merging

transformer models
Abstract: Recent advances in 3D point cloud transformers have led to state-of-the-art results in tasks such as semantic segmentation and reconstruction. However, these models typically rely on dense token representations, incurring high computational and memory costs during training and inference. In this work, we present the finding that tokens are remarkably redundant, leading to substantial inefficiency. We introduce \textbf{GitMerge3D}, a \textbf{g}lobally \textbf{i}nformed graph \textbf{t}oken \textbf{merging} method that can reduce the token count by up to 90–95\% while maintaining competitive performance. This finding challenges the prevailing assumption that more tokens inherently yield better performance and highlights that many current models are over-tokenized and under-optimized for scalability. We validate our method across multiple 3D vision tasks and show consistent improvements in computational efficiency. This work is the first to assess redundancy in large-scale 3D transformer models, providing insights into the development of more efficient 3D foundation architectures. Our code and checkpoints are publicly available at \href{https://gitmerge3d.github.io/}{https://gitmerge3d.github.io}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors observe that existing 3D point cloud transformers rely on excessive redundant tokens, yet only a small critical subset is essential for performance. Merging tokens based on spatial importance reduces usage by 90-95% with minimal accuracy loss, enabling efficient scalability across multiple vision tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Timely and impactful:\n- The paper tackles a highly relevant issue—scalability and inefficiency in transformer-based 3D models.\n- It challenges the implicit assumption that “more tokens = better performance,” backed by robust empirical evidence.\n\n2. Methodological novelty:\n- The proposed global-informed energy score for adaptive merging is well motivated and exploits domain-specific spatial structure, improving upon generic token reduction strategies.\n\n3. Strong empirical validation:\n- Experiments span three core 3D tasks with multiple datasets (ScanNet, S3DIS, ShapeNet, Objaverse, GSO, Waymo), showing generality.\n- Ablations on merging strategies, thresholds, and feature types (Q, K, V) are interesting.\n\n4.\tPractical implications:\n- The method operates inference-time without retraining, and can even outperform the baseline with minimal fine-tuning.\n- Clear gains in FLOPs and memory make the work valuable for real-world deployment (e.g., mobile robotics, AR/VR).\n\n5.\tComprehensive related work and discussion:\n- Covers point cloud, transformer compression, and downsampling literature in a balanced way.\n- Acknowledges open problems (e.g., automatic merging rate optimization, extension to meshes or real-time streams).\n\nWeaknesses: \n- While this work is primarily motivated by empirical observations, it lacks a rigorous theoretical analysis regarding the inherent sparsity of tokens in Transformer-based architectures. \n\n- The proposed Aggressive Merging Method demonstrates only marginal technical differentiation from the prior approach [1], thereby constraining the innovative contribution of this work.\n\t\n- The methodological pipeline presented in Figure 4 lacks sufficient detail to clearly elucidate the functional relationships between components (a), (b), and (c). Furthermore, the figure contains labeling inaccuracies, particularly in the misassignment of r⁺ and r in (a), and the incorrect color coding of source (red) and destination (blue) points in (c). Upon closer examination, the merging process appears fundamentally similar to the down-sampling strategy employed in RandLA-Net [2], raising questions about the claimed technical novelty of this aspect of the proposed method.\n\t\n- The manuscript fails to provide a formal definition for term ‘E’ introduced in line 191, which impedes proper understanding of the subsequent mathematical derivation.\n\t\n- The ablation study fails to evaluate the impact of parameter K in Figure 4 (c). The current approach of uniformly dividing each patch into K spatial bins may adversely affect the model’s ability to capture local geometric structures in the SA Merge operation. \n\t\n- While I greatly appreciate the thorough experiments and validations presented in the supplementary materials, the proposed method has only been validated on the PTv3 framework. This limited scope makes it difficult to assess the generalizability of the approach. To strengthen the claims, the authors should demonstrate its effectiveness across additional baselines or diverse benchmarks.\n\n[1] Bolya D, Fu C Y, Dai X, et al. Token merging: Your vit but faster[J]. arXiv pre\n[2] Hu Q, Yang B, Xie L, et al. Randla-net: Efficient semantic segmentation of large-scale point clouds[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 11108-11117.\n\nQUESTIONS:\n(1) Could the authors provide a theoretical analysis or formal justification for the observed token sparsity in their Transformer-based framework? Specifically, what underlying principles or properties of point cloud data lead to this sparsity, and how might this connect to existing theoretical work on token efficiency in attention mechanisms?\n\n(2) Could the authors explicitly clarify the key technical distinctions between their Aggressive Merging Method and the prior approach in [1]?\n\n(3) Explicitly contrast their merging process with RandLA-Net [2], highlighting both algorithmic similarities and any substantively novel aspects?\n\n(4) How does the choice of parameter K in the Figure 4 (c) influence the model’s ability to preserve local geometric structures during the SA Merge operation? Could the authors provide an ablation study analyzing the sensitivity of performance to different K values, particularly given that uniform partitioning may not optimally adapt to varying local point densities?\n\n(5) Could the authors discuss potential challenges or limitations in extending the proposed method to other frameworks beyond PTv3?\n\n[1] Bolya D, Fu C Y, Dai X, et al. Token merging: Your vit but faster[J]. arXiv pre\n[2] Hu Q, Yang B, Xie L, et al. Randla-net: Efficient semantic segmentation of large-scale point clouds[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 11108-11117.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper puts forward an interesting point of view: only 5%-10% of the tokens are needed to represent the original complex and redundant token sequence without significantly affecting the performance. Based on this idea, the paper proposes a specific token merging strategy for 3D transformer, which integrates local geometric structure with attention saliency to evaluate the importance of voxels, thereby achieving significant token reduction while maintaining minimal accuracy loss. Finally, a large number of experiments verify the effectiveness of the proposed token merging strategy in 3D semantic segmentation, reconstruction and detection tasks. It is worth mentioning that the proposed method can even surpass the baseline through a fine-tuning strategy, after removing 90%-95% of the tokens.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n1. This paper proposes a specific token merging strategy for 3D transformer, which evaluates the importance of voxels by integrating local geometric structure with attention saliency, thereby achieving significant token reduction while maintaining minimal accuracy loss.\n2. This paper conducts a large number of experiments to verify the effectiveness of the proposed token merging strategy in 3D semantic segmentation, reconstruction and detection tasks. It is worth mentioning that the proposed method can even surpass the baseline by fine-tuning the strategy while removing 90%-95% of the tokens.\n\n**Weaknesses**:\n1. The method is too simple. The proposed token merging method does not involve any parameter updates, and the method is too simple.\n2. Lack of more experimental verification. The method proposed in this paper should be plug-and-play. In addition to PTv3, it can also be adapted to other transformer-based models in theory. The author should do more experiments to verify the feasibility and applicability of this method.\n3. The icon title and caption are wrong. Table 2 should actually be Figure 6.\n\nQUESTIONS:\nIn addition to the weaknesses mentioned, please solve the following confusions:\n1. Can other token merging strategies, such as ToMe, ALGM, PiToMe, etc., improve their compression rate so that GFLOPS remains at a level comparable to that in this article, and then evaluate their performance? For example, Figure 2.\n2. In Table 1, why did the performance decrease after fine-tuning on the S3DIS dataset? This result should be discussed specifically.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of token reduction in 3D point cloud transformers by introducing a token merging strategy aimed at significantly lowering computational cost and memory footprint without sacrificing performance. The proposed method builds upon the Point Transformer V3 architecture and incorporates a domain-specific token merging mechanism at each attention block. Central to this approach is the computation of a globally-informed energy score for each point (token) in the input point cloud. This score aggregates the cosine similarity between a token and all patch centroids, effectively capturing its alignment with the global structure. Based on these energy scores, an adaptive token merging strategy is applied inspired by the ToMe framework: tokens within partitions exhibiting lower average energy, indicating weaker alignment with the global structure, are merged more aggressively, while those with higher energy are preserved, controlled by a tunable partition energy threshold. The method is evaluated on two key tasks, indoor semantic segmentation and 3D reconstruction, demonstrating its effectiveness in maintaining performance while significantly improving efficiency.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The paper provides a systematic analysis of prior token merging methods and offers a clear comparison with the proposed approach.\n- A thorough ablation study supports the rationale behind key design choices. \n- The proposed method consistently outperforms alternative token merging strategies across all evaluated tasks and matches the accuracy of the original backbone network, while significantly reducing both computational and memory costs.\n\n**Weaknesses:**\n- While the overall analysis is solid, some key aspects remain insufficiently explored. First, prior work such as ToMe suggests that token merging is most effective when applied between the attention and MLP layers within each block, and that an alternating strategy for selecting source and destination tokens in the bipartite graph yields better results. In contrast, as shown in Figure 4, the proposed method applies merging before the attention mechanism, and appears to use a random split of tokens within each patch to form the source and destination sets. This deviation from established findings is not clearly justified.\n- Second, the logic behind the globally-informed energy score raises concerns. According to lines 194–196 and 199–200, tokens with lower energy, i.e., those that are spatially dissimilar to the overall structure, are aggressively merged, while tokens with higher similarity are preserved. However, within each partition, the merging process appears to follow a different logic: similar tokens are merged, while dissimilar ones are retained. This inconsistency suggests that the adaptive merging strategy may be conceptually reversed. Arguably, partitions with tokens that are aligned with the global structure (and thus potentially redundant) should be merged more aggressively, while partitions with spatially distinct tokens should be preserved to maintain diversity.\n- There are structural and editorial inconsistencies throughout the paper. The supplementary material unnecessarily duplicates the entire main paper. Figure 5 presents results for ScanNet200 and S3DIS, yet its caption references additional datasets (nuScenes, Waymo, ScanNet) that are not shown. The second sentence of Table 2’s caption (which is in fact a figure) mistakenly describes Table 1. Figures 1 (main paper) and 10 (supplementary) are duplicates. Additionally, Section 7 of the main paper heavily references analyses from Section B of the supplementary material, which the reader may not have encountered yet, disrupting the flow and self-containment of the discussion.\n\nQUESTIONS:\n- The adaptive merging procedure follows a predefined merging ratio $r$, which determines the number of similar edges to retain in the bipartite graph between source and destination tokens. However, for certain partitions, a different ratio  $r^+>>r$ is used. What is the actual value of this higher ratio, and how is it selected?\n- While the discussion on local versus global energy scores in Appendix D is informative, it would be beneficial to include a quantitative comparison to better support the analysis.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper adapts and improves token merging techniques from 2D to 3D, demonstrating that 3D point cloud Transformers are heavily over-tokenized. The proposed method is plug-and-play, motivated by the inefficiencies caused by excessive point tokens in both training and inference. However, the merging decisions are based on a fixed heuristic energy score and are only evaluated on indoor benchmarks, raising concerns about the learnability and robustness of the approach in large-scale LiDAR or dynamic scenes.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The motivation is valuable, as state-of-the-art 3D scene point cloud methods often require a large number of point tokens to achieve strong performance on both local and global tasks.\n\n2. The modifications proposed by the authors to the PTv3 framework are effective, including the design of a globally structure-aware energy definition, adaptive merging rates, and spatially uniform dst.\n\n**Weaknesses**\n1. The use of heuristic energy scores lacks learnability. AdaPT [1], applied in 3D scenes as well, adopts Gumbel-Softmax to enable differentiable token selection and allows dynamic adjustment under computational budgets. However, the current method adopts only two fixed merging ratios, which are neither elegant nor truly dynamic.\n\n2. Several relevant works on token merging or pruning in point clouds have not been discussed, analyzed, or compared, such as YOGO [2], SuperPoint [3], and Token Halting [4]. The current baselines are primarily adapted from 2D methods and directly applied to 3D point clouds, resulting in insufficient baseline comparisons.\n\n3. If further experiments could be conducted in additional domains, such as LiDAR point clouds and object point clouds, it would significantly enhance the impact and credibility of this work.\n\n\n**Lots of Typos**\n1. Line 121      meodels  ->  models\n2. Line 165      Sonana and S3DIS  ->  Sonata\n3. Line 169      GLOPs  ->  GFLOPs\n4. Line 209      differnt dataset  ->  different\n5. Line 240      Furthest Point Sampling  ->  Farthest Point Sampling\n6. Figure 5       nuScenese  ->  nuScenes\n7. Figure 5       Wamo  ->  Waymo\n\n\n\n\n[1] Adaptive Point Transformer\n\n[2] You Only Group Once: Efficient Point-Cloud Processing with Token Representation and Relation Inference Module\n\n[3] Efficient 3D Semantic Segmentation with Superpoint Transformer\n\n[4] Efficient Transformer-based 3D Object Detection with Dynamic Token Halting\n\nQUESTIONS:\nsee weakness\n\nLIMITATIONS:\nsee weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis research highlights the importance of bringing the idea of \"patchify\" to point cloud transformers, similar to how it's used in image transformers. The motivation is clear: there are just too many tokens to process, which holds back progress in point cloud processing. But applying \"patchify\" to sparse point clouds (without losing too much information) isn't as straightforward as it is with images. This paper tackles that challenge and proposes a method that strikes a solid balance between accuracy and token count.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n**[Good Motivation / Direction]** \nOverall, I like this paper’s starting point and think it's doing some of the right things for our community. We've already put a lot of effort into improving the efficiency and scalability of basic point cloud operators. But no matter how far we push those boundaries, the most straightforward way to reduce cost is to process fewer points. That’s why we need the concept of \"patchifying\", similar to what’s been done in image processing. In that sense, the proposed \"merging\" techniques make a lot of sense.\n\n**[Performance]** \nThe accuracy from the merged point cloud reported by the paper is quite acceptable. Compared with the compression ratio, I think it would be helpful to some applications in the robotic learning or autonomous driving area. And also, it is good to extend the downstream tasks from perception to reconstruction.\n\n## Weakness\n**[Efficiency Analysis of Patchifying Itself]**\nI understand there's no doubt that processing a patchified point cloud is more efficient. However, I'm also curious about the efficiency of the patchifying algorithm itself. I think the paper would be stronger if it included some latency benchmarks comparing the different patchifying methods discussed.\n\n**[Absence of ScanNet Semantic Segmentation]**\nI noticed the paper includes semantic results on ScanNet200, so it feels a bit odd that ScanNet itself isn’t included (especially since the main difference is just the label mapping). It would be helpful to at least discuss performance on ScanNet, even if the results aren’t great.\n\n**[Please Polish the TeX (Minor)]**\nI’d suggest the authors double-check the consistency of terminology used throughout the paper. For example, I noticed that Point Transformer V3 is referred to in multiple ways, such as PTv3, PvT-3, and Point Transformer v3. It would be better to stick to a single, consistent naming convention.\n\nQUESTIONS:\nSee the discussion in the Weakness section. I initially suggested a `borderline accept` and am open to adjusting my score based on the authors’ rebuttal.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}