How Many Tokens Do 3D Point Cloud Transformer Architectures Really Need?
Authors: Tuan Anh Tran
Duy Minh Ho Nguyen
Hoai-Chau Tran
Michael Barz
Khoa D Doan
Roger Wattenhofer
Vien Anh Ngo
Mathias Niepert
Daniel Sonntag
Paul Swoboda
Venue: NeurIPS 2025 poster ()
Keywords: point cloud representation
token merging
transformer models
Primary area: deep_learning
TL;DR: optimizing 3D point cloud transformer model for large-scale processing

Abstract: Recent advances in 3D point cloud transformers have led to state-of-the-art results in tasks such as semantic segmentation and reconstruction. However, these models typically rely on dense token representations, incurring high computational and memory costs during training and inference. In this work, we present the finding that tokens are remarkably redundant, leading to substantial inefficiency. We introduce \textbf{GitMerge3D}, a \textbf{g}lobally \textbf{i}nformed graph \textbf{t}oken \textbf{merging} method that can reduce the token count by up to 90–95\% while maintaining competitive performance. This finding challenges the prevailing assumption that more tokens inherently yield better performance and highlights that many current models are over-tokenized and under-optimized for scalability. We validate our method across multiple 3D vision tasks and show consistent improvements in computational efficiency. This work is the first to assess redundancy in large-scale 3D transformer models, providing insights into the development of more efficient 3D foundation architectures. Our code and checkpoints are publicly available at \href{https://gitmerge3d.github.io/}{https://gitmerge3d.github.io}.

How Many Tokens Do 3D Point Cloud Transformer Architectures Really Need?
Tuan Anh Tran's Duy M. H. Nguyen 122.3 Hoai-Chau Tran Michael Barz's
Khoa D. Doan Roger Wattenhofer Ngo Anh Vien Mathias Niepert?: 2.3
Daniel Sonntag 1.8 Paul Swoboda"
GGrman Research Centre for Artificial Intelligence (DFKI)
Max Planck Research School for Intelligent Systems (IMPRS-IS)
3 University of Stuttgart, VinUni-lllinois Smart Health Center, VinUniversity
'College of Engineering & Computer Science, VinUniversity, "ETH Zurich, Vinnoobotics, University of Oldenburg. HHeinich Heine University Düsseldorf
(tuan.tran, ho minh duy.nguyen| } @dfkiide, paul.swoboda@hhuu.d
Abstract
Recent advances in 3D point cloud transformers have led to state-of-the-art re- sults in tasks such semantic segmentation and reconstruction. However, these models typically rely on dense token representations, incurring high computational and memory costs during training and inference. In this work, present the finding that tokens are remarkably redundant, leading to substantial inefficiency. We introduce GitMerge3 a globally informed graph token merging method that can reduce the token count by up to 90-95% while maintaining competitive performance. This finding challenges the prevailing assumption that more to- kens inherently yield better performance and highlights that many current models are over-tokenized and under-optimized for scalability. We validate our method across multiple 3D vision tasks and show consistent improvements in computa- tional efficiency. This work the first assess redundancy in large-scale 3D transformer models, providing insights into the development of more efficient 3D foundation architectures. Our code and checkpoints are publicly available https//ggimmeereedd.iuubb...
1 Introduction
The rise of transformer-basee architectures has significantly advanced the field of 3D point cloud
understanding (35] 101 28), particularly in tasks such as semantic segmentation [47 87 97 46),
object detection 25 36
(2), and reconstruction 44
11
success 19 and 2D vision 220 9 54 42),
80
14]. Building upon
of attention mechanisms in natural language processing 82
early works like Point Transformer (P'V-1) 1) [122] introduced self-attention tailored to the irregular and unordered nature of point clouds. This line of research evolved through subsequent versions, PTv-2 [93] and the more scalable PTv3 [92], which incorporated enhancements such as local-global feature fusion, serialized points ordering and optimized positional encoding. Among these, PTv3 has emerged as a particularly powerful backbone, capable of capturing complex spatial dependencies and scaling effectively to large-scale scenes. Leveraging large-scall scale attention mechanisms based on a ID serialized ordering of 3D points and a hierarchical architecture, it enables rich geometric reasoning and excels in dense 3D semantic segmentation tasks (27 91], where fine-grained boundary understanding is essential. PTV3 also serves as a strong encoder in advanced 3D reconstruction pipelines, such as SplatFormer [14] LSM (27), and excels in 3D object detection
[1 84 61 by
effectively capturing both local and global features. These strengths make PTV3 a key enabler of pirh-performance and generalizable solutions in modern 3D vision.
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

While PTv3 has emerged as a cornerstone for high-performance and generalizable 3D vision models, our analysis reveals surprising inefficiency the core of its attention mechanism. Despite archi- tectural advancements aimed at improving scalability, such as replacing computationally expensive K-nearest neighhor operations (28% of inference time) with lightweight 1D serialized neighbor mappings and removing image-relative positional encodings (26% of inference steps) to efficiently expand the receptive field from 16 to 1024 points, PTv3 still dramatically overutilizes tokens during self-attention. Strikingly, we show that preserving only 5-10% of the most spatially informative tokens sufficient to maintain nearly identical performance across diverse 3D tasks. This challenges the prevailing belief that dense tokenization is essential for transformer performance in 3D domains [3, 102 56 45 To our knowledge, this is the first work to reveal and systematically analyze the redundancy in token usage, pointing significant opportunities for improving the efficiency of point cloud transformers without compromising accuracy (Figure 1
85.0
FT-3 Boxike
Senate-Our
Senaa thteaan))
200
PTv3
82.5
P1x3+Oua
OFLOP.
175
80.0
78.9
Senata
39.20)
77.6
780
77.8
77
150
77.5
SunaterOurs
77.
715
125
GFLOPs
76.
Consumpiion
100
PTv3
187 SOPLAIP
72.5
75
PTv3iCurs-
1080TEO0
Ell effidins
50
67.5
35
30
34
Ex eleoerfoorr
0.5
0.6
0.7
0.8
0.9
0.95
100
1:5
Merge Rate
Computational
Figure 1: We compare the original PTv3 and Sonata with our proposed token merging method (PTv3 Ours and Sonata Ours) in terms of FLOPs and memory consumption. Despite merging up 90% of tokens, our method applied to PTV3 achieves a 5.3 x reduction in FLOPs (from 107.5 GFLOPs to 19.9 GFLOPs) and 6.4 x reduction in memory usage (from 10.12 GB to 1.6 GB), with minimal performance degradation. Notably, the model maintains comparable accuracy when fine-tuned by updating only the MLPs before and after the attention layer for just 10% of the original training epochs, while requiring significantly less computation per epoch during fine-tuning.
In particular, we conducted a comprehensive set of experiments by integrating several prominent token reduction techniques, originally developed for vision transformers, into the PTv3 framework. These included Token Merging (ToMe) (5 Token Pruning [99 103 41, ALGM [66], and PiToMe 81]. We inserted these methods before each attention layer to merge or prune varying proportions of tokens (from 10% to 50%) during inference, followed by an unmerging step to restore the token structure for compatibility with subsequent MLP layers. Remarkably, across the standard PTv3 and its advanced variants, such as PTv3 Sonata [91] and Splatformer [14), observed negligible accuracy degradation even under substantial token reduction. This surprising resilience held consistently across range of benchmarks, including 3D semantic segmentation (ScanNet [16). S3DIS (2), nuScenes g ) novel view object reconstruction (ShapeNet [0). Objaverse (18), ) and object detection (Waymo while significantly reducing both FLOPS and memory consumption.
Encouraged by this observation, we hypothesized that a domain domain-specifie merging mechanism, one that explicitly accounts for spatial locality and attention relevance in 3D point clouds, could unlock even greater efficiency. To this end, we developed a novel 3D-aware token merging strategy capable of merging up to 99% of tokens without retraining. The results are striking: even with 95% of tokens removed, our method maintains competitive performance while pushing computational and memory efficiency new bounds (see Figure 1). Furthermore, with just 10% of the original training schedule allocated for fine-tuning, the model not only fully recovers its baseline performance but even surpasses it in some cases, e.g., in ScanNet S3DIS datasets, highlighting the practical potential of our method for scalable and efficient real-world deployment.
In summary, we make the following contributions:

---

Through a systematic study, we uncover surprising degree of token redundancy in state- of-the-art point cloud transformers, showing that up to 90-95% of tokens can be removed without significant performance drop. This challenges the common assumption that dense tokenization is essential for 3D transformer effectiveness.
We propose a 31>-specific token merging strategy that integrates local geometric structure and attention saliency to estimate voxel importance, enabling aggressive token reduction with minimal accuracy degradation.
We validate the proposed token merging across 3D semantic segmentation, reconstruction, and detection tasks, achieving substantial efficiency gains and, in some cases, surpassing the baseline performance with minimal fine-tuning. We expect these findings will pave the way for future research toward lightweight and scalable transformer architectures for 3D point cloud processing.
2 Related Work
3D Point Cloud Architectures. 3D point cloud understanding has evolved through multiple architectural paradigms. Early approaches include projection-basee methods [13 48 49 76), which project point clouds onto 2D image planes for processing with standard CNNs, and voxel-based methods [63 [75 15 32 88), which discretize space into regular grids to apply 3D convolutions. While effective, these techniques often suffer from resolution loss, high memory consumption, or limited geometric expressiveness. To address these challenges, point-based methods such as PointNet [67] PointNet++ [68), and the more recent PointMLP 600 directly operate on raw point still sets, preserving tine-grained spatial structure. However, their reliance on local operations limit global context modeling. This has led to a growing shift toward transformer-based architectures that better capture long-range dependencies in 3D data. The Point Transformer (PTv) family, spanning PTv-1 [102], PTv-2 (93] to the more scalable PTV3 [92]. adapts attention mechanisms to unordered point sets and has become state-of-the-ar art backbone for 3D tasks such as semantic segmentation [47 87 97 46). object detection [25 36 [5 922. and reconstruction [44] 11 80 [4]. Building on
PTv3's success, recent variants like PTv3 Sonata [91], pretrained on 140K point clouds for improved
generalization, and Splatformer [4] tailored for robust 3D novel view synthesis, further demonstrate the versatility and dominance of transformer- based models in modern 3D vision pipelines.
In contrast to prior works, our study takes a step back to ask a fundamental question: "Is PTv3 already efficient in how it uses tokens?" Our findings reveal that the model can be significantly compressed, preserving only a fraction of tokens while maintaining comparable accuracy, opening a new direction for building lightweight and memory-efficient 3D point cloud transformers.
Token Redundancy and Sparsity in Transformers. To improve transformer efficiency, prior work has explored range of strategies, including approximating attention via hashing [17 (3), low-rank factorization [52 26), or sparsity [71 [74, as well as head pruning [64] 29 and domain- specific modules [54
89]. While effective, many of these methods require retraining or extensive fine-tuning from scratch, limiting their practicality. In contrast, token reduction techniques such as token pruning [99 103
[55
85 41 and token merging 7 81] aim to accelerate inference by reducing the number of tokens processed, often with minimal accuracy degradation. Notably, methods like ToMe [5] and its variants (6 666 leverage bipartite soft matching to efficiently merge similar tokens, though they may suffer from heuristic decisions sensitivity to token distributions. Other approaches
employ clustering 4 62 or graph-basee based methods [85 96 81 to merge tokens more systematically, but these often introduce computational overhead that counters their intended efficiency. In this work, we systematically adapt several general-purpose token reduction methods to PTTY3 and its advanced variants, uncovering that these models preserve performance even after substantial token reduction during inference. Motivated by this insight, we introduce a 3D-specific token merging strategy that incorporates spatial structure and densities among regions in 3D scenes, which enables up to 99% token reduction while achieving substantial efficiency gains with minimal or no performance loss, pointing to a promising direction for scalable 3D transformer design.
3D Point Cloud Compression and Efficiency. There is line of work that focuses on designing efficient architectures for 3D point clouds, such as MinkUNet [15). Sparse Point Transformer
79 78 86). PTv3 [92), and others 47 30 39,, which significantly reduce computation through
3

---

architectural innovations. However, these models typically require training from scratch, which limits their adaptability and prevents seamless integration with pre- are-trainee models. In parallel, various off-the-shelf techniques have been proposed to reduce the size of 3DD point clouds before feeding them into neural networks, including Random Sampling [37
58 10O), Farthest Point Sampling (FPS)
[24 68 effective at reducing input size, they are typically rule-based and insensitive to the underlying task or feature importance, leading to suboptimal performance in downstream applications. In contrast, our token merging strategy operates at the feature level and is computationally flexible, which allows dynamic token compression during inference without retraining and integrates seamlessly with existing architectures like PTv3. Furthermore, we consistently outperform traditional downsampling methods in both efficiency and performance across segmentation, reconstruction, and detection tasks.
[1 (50), and VoxelGrid Downsampling (70, 59 83 While these methods are simple and
3
Analyzing Token Redundancy in 3D Transformers
3.1 Point Transformer v3 architecture
PTv3 introduces simplified and efficient framework for 3D point cloud processing by replacing KNN-based grouping with ID serialization strategy, where points are ordered via space- curves to preserve spatial locality. The model follows U-Net-style encoder-decoder architecture with skip connections, enabling hierarchical feature learning (Figure 3
At each resolution, the serialized sequence is partitioned into disjoint local groups, and self-attention is applied independently each group capture local context. PTv3 evenly divides the input point set X' (21, NN into K' disjoint subsets (partitions) (P1, P ".....,Pk) such that UK=1 Ps- X. PinP, Ø for * j. and [Pa] 1024. Self-attention is then applied independently within each partition to capture local geometric structure.
The attention for each point is computed only over the points in its own partition P(i), formulated as:
q. kj
Attn( = Σ softmax, (()
V
where q(z;), k, = k(T,), and v. = v(z;) denote the query, key, and value projections of the respective points. The summation is restricted 2j E P(i). ensuring attention confined to the local context defined by the partition. While this attention mechanism offers strong representational capacity, it becomes prohibitively slow and computationally expensive with O(N2 complexity, especially when processing point clouds containing millions of points.
3.2 Token Merging Formulation
To address this limitation, token merging 1
87 12 66/ is introduced reduce the number of tokens
participating in the attention computation. Each original token is mapped to a merged representation via a function f : I; - T. inducing a transformation of the attention partition P(i) - P(i), where [P()) [P()). Attention is then computed over the merged tokens as:
f(a.) f(kk) f(v3), Vd
Att((((z:))
Σ
softmax,
(x)
The function ff merges token features according to a learned token-level mapping. Unlike existing token merging methods [1 12 66). which are designed for classification and operate on the merged tokens throughout subsequent layers, our approach targets dense 3D point cloud processing. This requires restoring the token features to their original resolution. To achieve this, an unmerging function f-1 approximating original attention layer required: Attn(r;)) & f-1 (Attn(f(zz))
Token Merging. Token Merging (ToMe) [ defines the merge function f((x;), r) by partitioning the set of tokens (Ix) into source (src) and destination (dst) sets, and assigning the most similar tokens from src to tokens in dst. Each merged token is computed by averaging the features of its assigned source tokens along with the corresponding destination token.
Besides ToMe, we also evaluate several existing token merging methods, including PiToMe and ALGM, two 3D segmentation benchmarks: ScanNet and ScannNer200, using PTv3 and PTv3

---

Sonata. For each method, we apply different token merging ratios and report both GFLOPs and mloU.
As shown in Fig. D and Fig. we observe that model performance remains stable, i.e., mloU when up 50% of tokens merged. Meanwhile, GLOPSs are reduced significantly, demonstrating the redundancy in token representations. This suggests that aggressive token reduction is feasible for 3D data.
drops only slightly
However, because existing methods are designed primarily for image classification and are not optimized for the characteristies of 3D point clouds. Their merging strategies are generic and do not exploit spatial locality or the density variation in 3D scenes. Moreover, most of them do not support feature recovery, which is crucial for dense segmentation. This motivates to propose new token merging method specifically designed for 3D) point clouds. Our approach allows higher merging ratios while preserving fine-grained information necessary for accurate segmentation. It integrates a learned merging and unmerging mechanism to reduce attention cost while maintaining per-point predictions.
4 Aggressive Merging Methods for Point Clouds Processing
Input Scene
Feature - Before Merge
Feature - After Merge
Difference in Predictions
Figure 2: Observation: After merging 90% of the tokens each attention layer, the change in PCA visualization of feature representation (3rd image) is minimal compared to the original feature (2nd image). Most the predictions remain unchanged after merging, with red indicating the areas where predictions differ. This leads us to conclude that there is high redundancy in the point cloud processing model.
Empirical Observations. As shown in Figure when applying spatial- preserviiee token merging at extreme rates (e.g., 90% and 95%), the majority of points retain their original predictions. Addition- ally, PCA visualizations of token features reveal that, in high-resolution layers of PTv3, features well-separated by object, indicating strong spatial and semantic consistency.
Adaptive Merging via Global-Informed Energy Score. We observe that aggressive merging (e.g., 000% can still retain performance if the merging ratio is adapted to the information content each partition. Inspired by the energy score in existing token merging methods 81
5
66
12]
we propose
a global-informed energy score to guide adaptive merging decisions.
We define bipartite graph G (V,E), where the vertex set is V (x;) U (P)). Here, Pj denotes the centroid of partition Pj. computed as: P3 P. Emee, Zg. The edge set is defined as
(x,, Pj)}, forming directed bipartite graph from each token T3 to all partition centroids P,- For each token Iy we define its outgoing neighbors as N²x)) = (P, (x,, P,) E).
The energy score E(r:) is then computed as the mean cosine similarity between æ, and all connected centroids:
E(x;) = [N(z;)]
Σ PyeA(a.)
cos(a;, P,).
(1)
This score reflects how globally aligned a token with all partition centroids. Tokens with lower energy (i.e., more aligned with global structure) are considered less informative and can be merged more aggressively, whereas high-energy tokens (i.e., less aligned with global structure) are preserved to retain critical information.
5

---

Adaptive Merging by Energy. Using the above formulation, we define the importance score of partition as the mean energy its tokens:
[(P)
P E())
If E(P) > т, we apply moderate merging f(P, r): otherwise, we apply aggressive merging f(P, r+), where r+t >> r. This branching mechanism enables us to aggressively reduce redundancy and significantly improve computational efficiency, while still supporting batch training and preserving performance on off-the-shelf evaluation. Here is a common threshold we effectively used for all datasets and tasks. We present in Figure our proposed algorithm with further insights in Sec. Appendix.
RiP)
Mage
Sclf Alt
MIP
Mege
High
Paker
Mingel
Paic
Cerooid PT
Conneiés
Fanrgy
Sart:
Compone Paach Eaergy Score B(P,)
Mis.)
Step devide euch biss (5K,)
Step 2: Bandonily select from B.
Step Marge ES.
spotial
=
Figure 3: For each Point Transformer layer, we compute token energy scores and propagate them to patches using globally informed graph over the local self-attention. a) These patch-level scores guide adaptive merging, retaining more information for high-energy patches. e) Each patch is divided into evenly sized bins, and destination tokens are randomly selected within these bins to enable spatially aware merging.
5
Experimental Results
5.1 Downstream Tasks and Baseline Setup
We evaluate our method on three 3D tasks: 3D Semantic Segmentation, 3D Reconstruction, and
Object Detection. For semantic segmentation, we test our approach Sonata [9 and PTv3 [92] across four datasets: ScanNe1200 (72), ScanNet (16), S3DIS 22] and NuScenes [8]. For 3D re-
construction, we evaluate our method using SplatFormer 14 on three datasets: ShapeNet [TO] ObjectVerss [18), and GSO [22] We present results on the evaluation sets of indoor semantic segmentation datasets, while test set results and an additional object detection task are assessed using the state-of-the oottte-rrttttttt Emngagge-guided object detector SpatialI.M [6]
In addition recent token merging methods 81 66
incorporate point cloud downsampling techniques to reduce input complexity. Random Token Drop [37] randomly discards a subset of points, offering fast but coarse reduction. Farthest Point Sampling (FPS) [21 95 selects points that are maximally distant from each other to preserve geometric coverage. VoxeIGrid Downsampling [70] (9) partitions space into voxels and retains one representative point per voxel, ensuring spatial regularity. Final predictions are upsampled in the last stage.
5.2
3D Point Cloud Semantic Segmentation
Indoor Segmentation: We evaluate our method using GFLOPs and mloU on Sonata and PTv33 (Fig 4). Even without fine-tuning, approach shows minimal segmentation drop. Finetuning

---

with only 10% of the original training epochs, our merging strategy significantly outperforms others in efficiency. 80% merging for high-energy and 97% for low-energy branches (K=32), performance remains unaffected. Table [ also shows that our method outperforms traditional point cloud downsampling techniques by preserving more latent information, leading better results.
As shown qualitatively in Tab. 2 even with up to 95% token merging, feature representations remain largely unchanged, indicating high redundancy and supporting our aggressive merging strategy. We attribute this to the nature of point cloud data, which is both sparse in 3D space and fine-grained. Our finding suggests a more efficient way process such data.
PTy3- ScanNet
Sonata ScanNet
855
85
mttli (%6)
60
80
13
75
PiToMe
(Iactuaed)
70
Oun (fanttao
Our
ALU
PTv
Onrx
ALLOM
SomsteFT
65
15
20
100
30
6
10
100
150
200
250
300
350
PTv3 scanne1200
Sonata scannet200
0.8
mlltt (%6
0.5
30
30
Tabie
PYlil
PTv3
ToMe
WiliMMe ALAI
Our (fintune))
Ours
ALM
Oun (Inotondd)
Oure
Sensts
50
75
100
PTv3- S3DIS
Sonata S3DIS
80
80
0.3
03
0.5
mltti (9)
70
00
(factuael)
60
Oun (Tasttac))
Our
ALUOX
Onrx
AL.IM
50
50
6x
10'
75
100
150
200
250
500
6=100
75
100
150
200
250
300
4'
GFLOPs
GFLOPS
Figure 4: Off-the-shelf performance comparison between our merging against existing methods on PTv3 Sonata and PTv3 across three datasets ScanNet, ScanNet-200 and S3DIS. The numbers above each data point indicate the merging rate.
Table 1: We compare our method, using a merge rate of 0.8, two settings- fine-tuned (blue rows) and off-the-shelf (gray rows)- against other seg- mentation and point cloud downsampling methods applied to PTv3.
Input
95% Merge 80% Merge 0% Merge
Methods
ScanNet Val ScanNet2000 Val
S3D1S AreaS
MinkUNet
15
72.2
25.0
65.4
ST H7
74.3
72.0
PointNeXX
00
71.5
70.5
OctFormer
75.7
32.6
Swin3D
(
76.4
72.5
PTv1 102
70.6
27.8
70.4
PTv2
93
75.4
30.2
71.6
PTv3
92
77.6
35.2
74.7
Random Drop
70.1
31.1
73.4
FPS
71.2
32.4
70.9
VexelGrid Down.
72.1
Ours
77.0
34.4
72.3
Ours
77.4
35.2
74.3
Table 2: PCA feature visualization of PTv3 on the 3D indoor segmentation task across different merging rates. Even at 95% merging rate, the
PTv3-Somata 91
79.0
30.4
72.2
Random Drop
72.2
25.2
68.5
FPS
73.9
26.1
69.0
VeselGrid Down.
73.9
25,5
68.8
Ours
28.8
latent representations of the point cloud remain largely unchanged.
Ours
78.9
30,9
73.5
Outdoor Segmentation: We additionally evaluate our method on the outdoor dataset NuScenes g Table 3 summarizes the performance and efficiency of different methods on the NuScenes validation
7

---

set in terms of mloU, mAcc, allAcc, peak memory usage, GFLOPS, and latency. We use the default merging rate of 80% and an aggressive merge configuration with K" 32. PTv3 achieves the highest overall accuracy with an mloU of 80.3, mAce of 87.2, and allAce of 94.6. While PTv3 + Ours shows slightly lower mloU (78.0) and mAce (85.5), it maintains a comparable allAcce (94.0), demonstrating minimal performance loss despite substantial computational savings. Notably, our method reduces peak memory usage by over 85% (from 6.20 GB 0.92 GB), lowers GFLOPS by nearly 70%, and cuts latency by about 30%, highlighting significant improvements in speed and resource efficiency.
Methods
mloU
mAcc
allAcc
peakMem (GB)
GFLOPS
Latency (ms)
PTV2 93
80.2
PTv3 92
80.3
87.2
94.6
6.20
101.68
152
PTV3 Ours
78.0
85.5
94.0
0.92
32.45
Table 3: Comparison of semantic segmentation performance and efficiency on the NuScenes valida-
tion set 8
GS0 00D
Objaverse OOD
0.06
0L87
0.86
algo
nigu
ToMe
TaMe
Pruning
0.79
ToFls
S8IM
Tul's
PIToMe
PieMMe
ToMcSD
8.78
To2Ce5D
Bandom Drop
Random Drop
Furthesd Puint
Fur Joed Point S.
Our
Our
0.0
02
0.4
0.8
0.8
0.4 Merge Rate
Merge
Figure 5: 3D Object reconstruction: Off-the-shell performance of MAYC on Objaverse 18 and Google Scanned Object (GSO) 23
PTV3
Random Drup
Vocel Down
FPS
Our
Ground Truih
Figure 6: We visualize the output of various token compression techniques after removing 80% of the tokens, comparing their visual quality degradation (or preservation) on the 3D object reconstruction task.
5.3 3D Object Reconstruction
We also conduct experiments to evaluate the performance of our method on the novel view synthesis task under nut-of-distributiin (OOD) test camera angles. For this task, we adopt SplatFormer [14] the backbone, which also incorporates PTv3 its core to refines flawed 3D Gaussian splats to mitigate artifacts in OOD views.
As shown in Table 6 Figure 5 and 6 our method archives high performance, with only about
0. % drop across metrics, even after reducing up to 90% of the tokens processed by the model, while still outperforming other state-of-the-art methods such as MipNeRF360 33, 3DGS 40), 2DGS

---

38), Nerfbusters 190]. and LaRA II In contrast, alternative token compression techniques such as Random Drop, Voxel Downsampling, and Furthest Point Sampling significantly degrade model performance after reducing 80% number of tokens.
5.4 Language Guided Object Detection
Methods
F1 L25
L50
F1 O25
F1 O50
Time (s)
Mem (GB)
SpatialLM
0.4906
0,3886
0,3356
0.1894
6.009
12.36
Ours (r=0.5)
0.4982
0.3806
0.3490
0.1946
5.269
3.75
+ Ours (r=0.)
0.4809
0.3873
0,3485
0,2006
4.795
2.53
Table 4: Evaluation of our method with off-the-shell setting on the SpatialLM dataset for object
detection at two merging rates (0.5 and 0.8) demonstrates improvements in inference time and memory usage without any degradation in prediction quality.
M a language-guided 3D object de- tection model that uses Sonata as its backbone. processes 3D point clouds to detect spatial layouts and objects based natural language instructions.
We extensively evaluate our method on the language-guided object detection task from Spa- tialLM [6]] which is based on Sonata. The met- rics reported in Table of detection performance and computational ef- ficiency. Specifically, the Fl scores for layouts and objects at 25% and 50% Intersection-over- Union (IoU) thresholds (F1 L25, F1 L500, Fl 025, F1 050; where L refers to scene layouts and O to objects) assess the accuracy of de- tecting spatial layouts and objects with different levels of strictness. Inference time (in seconds) and peak memory consumption (in GB) reflect the computational cost during model inference. Figure shows that even after merging 80% of the tokens, the prediction quality remains unaffected. Our method significantly reduces both inference time and memory usage compared to the baseline SpatialLM model, while maintaining comparable or slightly improved detection performance across the evaluated metrics.
various aspects
Figure 7: With a 0.8 merge rate, SpatialL using our method still produces high-quality predictions.
6 Ablation Study
Energy Threshold. We use threshold to decide which patches P to aggressively merge. As shown Table [0 Appendix when is close -1, no patches are merged, resulting un- changed GFLOPs and 2% drop in mloU. As 7 increases, more patches are merged, reducing GFLOPs until they approach the non-adaptive merging bascline. We select T = 0.2 as of- fers the best trade-off, achieving similar GFLOPs
0.9 while yielding much higher mloU.
Merging Metric In Table 5 we evaluate the effect of using Q, K, or V features as the merging crite- rion. We also compare applying the merging function independently per head versus uniformly across all heads. Results show that using the value feature (V) and merging independently per head yields the best performance.
Table 5: Impact of metric and independent head during token matching.
Metric
K
V
No Independent Heads
76.08
76.37
76.55
With Independent Heads
76.27
76.36
76.98

---

Table OOD-NVS. Comparisons on the GSO-OOD, Realworld-OOD and Objaverse- OOD evalu- ation sets with off-the-shelf evaluation. The metric is evaluated on OOD test views with elevation Dodt 2 70s,
Methods
GSO-OOD
Objaverse-OOD SSIM
RealWorld-OOU
PSNR
SSIM1
IPIPS.
PSNR
T.PIPS.
PSNR 1 SSIM I.PPPS
MipNcRF360
22.90
0.824
0.192
19.6
0.72
0.28
21.99
0.878
0.127
3DGS 40
21.78
0.746
0.25
19.24
0.67
0.29
23.83
0.877
0.109
2DGS 38
23.29
0.816
0.204
19.24
0.67
0.29
23.64
0.891
0.104
Nerfbusters
90
15.95
0.678
0.300
16.9
0.69
0.29
23.93
0.893
0.114
LaRa [
19.0
0.68
0.32
SplatFormer
14
24.71
0.857
0.152
22.43
0.808
0.179
24.33
0,900
0,100
Random Drop
23.77
0.821
0.19
21.80
0.777
0.208
24.02
0,889
0.105
Farthest Point
23.29
0.817
0.194
21.13
0.757
0.223
23.91
0,889
0.107
VoxelGrid Down.
23.74
0.827
0.18
21.47
0.756
0.224
23.88
0.887
0.108
Our
24.56
0.852
0.157
22.34
0.803
0,185
24.06
0.899
0.101
7 Discussion
Our results show that 3D point cloud transformers rely heavily on excessive tokens, and removing attention layers simplify the architecture. Using non trainable strided pooling, such as mean pooling, combined with token shuf- fling for long range information exchange (Figure 8D), achieves competitive performance on ScanNet, with only slight drop compared to attention based models (76.0 77.0 mloU). This underscores the importance of attention for long range interactions while motivating more efficient alternatives.
MLP
Layer Norm
Un-ShuffleO
Token Mixer
Self- attention
Pooling
Shufflel
Additionally, applying 70% token merging rate dur- ing downstream training of Sonata preserves performance while significantly reducing computation and training time (Table Unlike standard fine-tuning, where merging is applied after downstream task training, here merging is integrated from the start. This highlights the potential of token merging for efficient training of large models without sacrificing accuracy.
a) Generic Token Mixing Pipeline
b) Our Shufile-Pooling Pipeline
Figure 8: At each layer, self-attention is substituted with pooling function, combined with a shuffling function to enable information exchange.
A limitation of our method is that the merg- ing rate r is manually specified rather than learned. Automatically optimizing r under a FLOPs constraint would require an end- to-end framework. but this is challenging because sorting and grouping operations non-differentiable, necessitating gradi- ent approximations
Version mIoU mAce allAcc GPU Mem GPU Hours
Sonata-ft 79.0
86.0
92.7
211.25 GB
55.2
Ours 78.9
85.3
92.3
74.95 GB
28.3
Table 7:
Sonata downstream task training performance with and without our token merging method (first row and second row respectively).
Another open question is the lack of a formal frame- work to quantify and reduce redundancy in token representations, which could further improve efficiency and provide stronger theoretical grounding for our approach.
7 94 65
8 Conclusion
In conclusion, our study reveals that state-of-the-art 3D point cloud transformers are significantly over-tokenized, and their performance can be largely retained even after reducing up to 80-95% of the tokens with a proper merging strategy. We also propose a 3D-specific token merging strategy, integrating local geometric structure and attention saliency to estimate voxel importance, thus enabling aggressive token reduction with minimal accuracy degradation. Our findings not only expose inefficiencies in existing works but also introduce a practical path toward more scalable and computationally efficient 3D vision systems, offering a new perspective on transformer design 3D tasks and emphasizing the importance of efficient token utilization over parameter scaling.
10

---

Acknowledgement
This work was supported by Deutsche Forschungsgemeinsccaff (DFG, German Research Foundation) under Germany's 's Excellence Strategy EXC 2075 390740016, the DARPA ANSR program under award FA8750-23- 2-0004, the DARPA CODORD program under award HR001 12590089. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Duy M. H. Nguyen. Tuan Anh Tran, Duy M. H. Nguyen, Michael Barz and Daniel Sonntag are also supported by the No-IDLE project (BMFTR, 16IW23002), the MASTER project (EU, 101093079), and the Endowed Chair of Applied Artificial Intelligence, Oldenburg University. Additionally, Hoai-Chau Tran and Khoa D. Doan are supported by the VinUni-Illinois Smart Health Center (VISHC), VinUniversity.
References
[1] J. Achiam, S. Adler. S. Agarwal, L. Ahmad, I. Akkaya, F. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] I. Armeni, O. Sener, A. R. Zamir, H. Jiang, I. Brilakis, M. Fischer, and S. Savarese. 3d semantic parsing of large-scale indoor spaces. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1534-1543, 2016.
[3] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5470-5479, 2022.
[4] F. M. Bianchi. D. Grattarola, and Alippi. Spectral clustering with graph neural networks for graph pooling. In International conference on machine learning, pages 874-883, PMLR, 2020.
[5] Bolya, C.-Y. Fu, Dai, P. Zhang, C. Feichtenhofer, and J. Hoffman. Token merging: Your vit but faster. International Conference on Learning Representations (ICLR), 2023.
[6] D. Bolya and J. Hoffman. Token merging for fast stable diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4599-4603, 2023.
[7] M. Bonnaerens and J. Dambre. Learned thresholds token merging and pruning for vision transformers. Transactions on Machine Learning Research, 2023.
[8] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11631, 2020.
[9] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to- end object detection with transformers. In European conference computer vision, pages 213-229. Springer, 2020.
[10] A. X. Chang, T. Funkhouser, I. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song. H. Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: urXiv:1512.03012 2015.
[11] A. Chen, H. Xu, S. Esposito, S. Tang, and A. Geiger. Lara: Efficient large-baseline radiance fields. European Conference on Computer Vision, pages 338-355. Springer, 2024.
[12] M. Chen, W. Shao, P. Xu, M. Lin, K. Zhang, F. Chao, R. Ji, Y. Qiao, and Luo. Diffrate: Differentiable compression rate for efficient vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 17164-17174, 2023.
[13] X. Chen, H. Ma, J. Wan, B. Li, and T. Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1907-1915, 2017.

---

[14] Y. Chen, M. Mihajlovic, X. Chen, Y. Wang, S. Prokudin, and S. Tang. Splatformer: Point trans- former for robust 3d gaussian splatting. International Conference Learning Representations (ICIRR, 2025.
[15] Choy, J. Gwak, and Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3075-3084, 2019.
[16] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießnne. Scannet: Richly- annotated 3d reconstructions indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828-5839, 2017.
[17] G. Daras, N. Kitaev, A. Odena, and A. G. Dimakis. Smyrf-efficient attention using asymmetric clustering. Advances in Neural Information Processing Systems, 33:6476-6489, 2020.
[18] M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi. Objaverse: A universe of annotated 3d objects. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13142-13153, 2023.
[19] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pages 4171-4186, 2019.
[20] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition scale. International Conference on Learning Representations (ICLR, 2021.
[21] O. Dovrat, I. Lang, and S. Avidan. Learning to sample. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2760-2766 2019.
[22] L. Downs, A. Francis, N. Koenig, B. Kinman, R. Hickman, K. Reymann, T. B. McHugh, and V. Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA), pages IEEE, 2022.
[23] L. Downs, A. Francis, N. Koenig, B. Kinman, R. Hickman, K. Reymann, T. B. McHugh, and V. Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference Robotics and Automation (ICRA), page 2553-2560. IEEE Press, 2022.
[24] Y. Eldar, M. Lindenbaum, M. Porat, and Y. Y. Zeevi. The farthest point strategy for progressive image sampling. IEEE transactions on image processing. 6(9):1305-145 -315, 1997.
[25] L. Fan, Z. Pang, T. Zhang, Y.-X. Wang, H. Zhao, F. Wang, N. Wang, and Z. Zhang. Embracing single stride 3d object detector with sparse transformer. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8458-8468, 2022.
[26] Fan, Z. Liu, J. Lian, W. X. Zhao, X. Xie, and J.-R. Wen. Lighter and better: low-rank decomposed self-attention networks for next-item recommendation. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pages 1733-1737 2021.
[27] Z. Fan, J. Zhang, W. Cong, P. Wang, R. Li, K. Wen, Zhou, A. Kadambi, Z. Wang, D. Xu, et al. Large spatial model: End-to-end unposed images to semantic 3d. Advances in neural information processing systems, 37:40212-40221 2024.
[28] Z. Fang, X. Li, X. Li, J. M. Buhmann, C. C. Loy, and M. Liu. Explore in-context learning for 3d point cloud understanding. Advances in Neural Information Processing Systems, 19:2232-42339 2023.
12

---

[29] M. Fayyaz, S. A. Koohpayegani, F. R. Jafari, S. Sengupta, H. R. V. Joze, E. Sommerlade, H. Pirsiavash, and Gall. Adaptive token sampling for efficient vision transformers. In European Conference Computer Vision, pages 396-414. Springer, 2022.
[30] X. Feng, H. Du, H. Fan, Y. Duan, and Y. Liu. Seformer: Structure embedding transformer for 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 332-640, 2023.
[31] R. Garcia, S. Chen, and C. Schmid. Towards generalizable vision-language robotic manipula- tion: A benchmark and Ilm-guided 3d policy. arXiv preprint arXiv:2410.01345, 2024.
[32] B. Graham, M. Engelcke, and L. Van Der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 9224-9232, 2018.
[33] C. Guan, X. Wang, Q. Zhang, R. Chen, D. He, and X. Xie. Towards deep and unified understanding of deep neural models in nip. In International conference on machine learning, pages 2454-224442 2463. PMLR, 2019.
[34] M.-H. Guo, J.-X. Cai, Z.-N. Liu, T.JJ. MU, R. R. Martin, and S.-M. Hu. Pet: Point cloud transformer. Computational visual media, 7:187-199, 2021.
[35] Y. Guo, H. Wang. Q. Hu, H. Liu, L. Liu, and M. Bennamoun. Deep learning for 3d point clouds: A survey. IEEE transactions on pattern analysis and machine intelligence, 43(12))4338-44 -364,, 2020.
[36] C. He, R. L., Li, and L. Zhang. Voxel set transformer: A set-to-set approach to 3d object detection from point clouds. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8417-8427, 2022.
[37] Q. Hu, B. Yang. L. Xie, S. Rosa, Y. Guo, Z. Wang. N. Trigoni, and A. Markham, Randla-net: Efficient semantic segmentation of large-scale point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11108-11117, 2020.
[38] Huang, Z. Yu, A. Chen, A. Geiger, and S. Gao. 2d gaussian splatting for geometrically accurate radiance fields. In ACM SIGGRAPH 2024 conference papers, pages 1-11, 2024.
[39] Z. Huang, Z. Zhao, B. Li, and J. Han. Lepformer: Towards effective 3d point cloud analysis via local context propagation in transformers. IEEE Transactions on Circuits and Systems for Video Technology, 499554999944996 2023.
[40] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139-1, 2023.
[41] M. Kim, Gao, Y.-C. Hsu, Y. Shen, and H. Jin. Token fusion: Bridging the gap between token pruning and token merging, In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1383-1392, 2024.
[42] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.- Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015-4026, 2023.
[43] N. Kitaev, L. Kaiser, and A. Levskaya. Reformer: The efficient transformer. International Conference on Learning Representations (ICIR), 2020.
[44] X. Kong, S. Liu, X. Lyu, M. Taher, X. Qi, and A. J. Davison. Eschernet: A generative model for scalable view synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9503-9513 2024.
[45] J. Lahoud, J. Cao, F. S. Khan, H. Cholakkal, R. M. Anwer, S. Khan, and M.-H. Yang. 3d vision with transformers: A survey, arXiv preprint arXiv:2208. 2022.
[46] X. Lai, Y. Chen, F. Lu, J. Liu, and J. Jia. Spherical transformer for based 3d recognition. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pottern Recognition, pages 17545-17555 2023.
13

---

[47] X. Lai, Liu, L. Jiang, L. Wang, H. Zhao, Liu, X. Qi. and J. Jia. Stratified transformer for 3d point cloud segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8500-8500 85098 2022.
[48] A. H. Lang. S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and patiern recognition, pages 12697-12705, 2019.
[49] B. Li, T. Zhang, and Xia. Vehicle detection from 3d lidar using fully convolutional network. Robotics: Science and Systems, 2016.
[50] J. Li, Zhou, Y. Xiong. Chen, and C. Chakrabarti. An adjustable farthest point sampling method for approximately-sorted point cloud data. In 2022 IEEE workshop on signal processing systems (SiPS), pages 1-6. IEEE, 2022.
[51] Y. Li, R. Bu, M. Sun, W. Wu, X. Di, and B. Chen. Pointenn: Convolution on x-transformed points. Advances in neural information processing systems, 31, 2018.
[52] V. Likhosherstov, K. M. Choromanski, Q. Davis, X. Song, and A. Weller. Sub-linear memory: How to make performers slim. Advances in Neural Information Processing Systems, 34:6707-6719, 2021.
[53] S. Lin, P. Lyu. D. Liu, T. Tang, X. Liang, A. Song, and X. Chang. Mlp can be a good transformer learner. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19489- 2024.
[54] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 20112-0022, 2021.
[55] Z. Liu, J. Ning, Y. Cao, Y. Wei, Zhang, S. Lin, and H. Hu. Video swin transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3202-3211, 2022.
[56] Z. Liu, H. Tang, Y. Lin, and S. Han. Point-voxel cnn for efficient 3d deep learning, Advances in neural information processing systems, 32, 2019.
[57] Z. Liu, X. Yang, H. Tang, S. Yang, and S. Han. Flatformer: Flattened window attention for efficient point cloud transformer. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1200-1211, 2023.
[58] F. Lu, G. Chen, Y. Liu, Z. Qu, and A. Knoll. Rskdd-net: Random sample-based keypoint detector and descriptor. Advances in Neural Information Processing Systems, 33:21297-21308. 2020.
[59] W. Lyu, W. Ke, H. Sheng, X. Ma, and H. Zhang. Dynamic downsampling algorithm for 3d point cloud map based on voxel filtering. Applied Sciences, 14(8):3160, 2024.
[60] X. Ma, C. Qin, H. You, H. Ran, and Y. Fu. Rethinking network design and local geometry in point cloud: A simple residual mlp framework. International Conference on Learning Representations (ICLR), 2022.
[61] Y. Mao, J. Zhong, C. Fang. Zheng, R. Tang, H. Zhu, Tan, and Z. Zhou. Spatiallm: Training large language models for structured indoor modeling. In Advances in Neural Information Processing Systems, 2025.
[62] D. Marin, J. R. Chang, A. Ranjan, A. Prabhu, M. Rastegari, and O. Tuzel. Token pooling in vision transformers for image classification. In Proceedings of the IEFE/CVF winter conference on applications of computer vision, pages 12-21, 2023.
[63] D. Maturana and S. Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In 2015 IEFE/RSJ international conference on intelligent robots and systems (IROS), pages 922-928. IEEE, 2015.
14

---

[64] L. Meng, H. B., Chen, S. Lan, Z. Wu, Y.-G. Jiang, and S.-N. Lim. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF conference on compuder vision and patlern recognition, pages 12309-12318, 2022.
[65] M. Niepert, P. Minervini, and L. Franceschi. Implicit mle: backpropagating through dis- crete exponential family distributions. Advances in Neural Information Processing Systems, 34: 14577-4944 2021.
[66] N. Norouzi, S. Orlova, D. De Geus, and G. Dubbelman. Algm: Adaptive local-then then-global token merging for efficient semantic segmentation with plain vision transformers. In Pro- ceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 15773-15782 2024.
[67] R. Qi. H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[68] C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30, 2017.
[69] G. Qian, Y. Li, H. Peng, Mai, H. Hammoud, M. Elhoseiny, and B. Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. Advances in neural information processing systems, 35:23192-23204, 2022.
[70] Z. Que, G. Lu, and D. Xu. Voxelcontext-net: An octree based framework for point cloud compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6042-6051, 2021.
[71] H. Ren, H. Dai, Z. Dai, M. Yang, J. Leskovee, D. Schuurmans, and B. Dai. Combiner: Full attention transformer with sparse computation cost. Advances Neural Information Processing Systems, 34:22470-22483 2021.
[72] D. Rozenberszki, O. Litany, and A. Dai. tanguage-grounderr indoor 3d semantic segmentation in the wild. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
[73] M. E. Sander, J. Puigcerver, J. Djolonga, G. Peyré, and M. Blondel. Fast, differentiable and sparse top-k: a convex analysis perspective. In International Conference on Machine Learning, pages 29919-2999999 299966.. PMLR, 2023.
[74] Z. Shen, M. Zhang, H. Zhao, S. Yi, and H. Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 3531-3539 2021.
[75] S. Song, F. Yu, A. Zeng. A. X. Chang, M. Savva, and Funkhouser. Semantic scene completion from single depth image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1746-1754, 2017.
[76] H. Su, S. Maji, E. Kalogerakis, and E. Learned-Miller Multi-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference on computer vision, pages 945-953, 2015.
[77] P. Sun, H. Kretzsehmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, Caine, al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 145444454, 2020.
[78] P. Sun, M. Tan, W. Wang. C. Li,, F. Xia, Z. Leng, and D. Anguelov. Swformer: transformer for 3d object detection in point clouds. In European Conference on Computer Vision, pages 426 442. Springer, 2022.
Sparse window
[79] H. Tang, Z. Liu, S. Zhao, Y. Lin, J. Lin, H. Wang, and S. Han. Searching efficient 3d architectures with sparse point-voxel convolution. In European conference on computer vision, pages 685-702. Springer, 2020.
15

---

[80] J. Tang, Z. Chen, X. Chen, T. Wang, G. Zeng, and Z. Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. In European Conference on Computer Vision, pages 1-18. Springer, 2024.
[81] C. Tran, D. MH Nguyen, M.-D. Nguyen, Nguyen, N. Le, P. Xie, D. Sonntag, Y. Zou, B. Nguyen, and M. Niepert. Accelerating transformers with spectrum-preservinn token merging. Advances in Neural Information Processing Systems, 37:30772-30$ 30000, 2024.
[82] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, Attention is all you need. Advances in neural information processing systems, 30, 2017.
[83] I. Vizzo, T. Guadagnino, Mersch, L. Wiesmann, J. Behley, and Stachniss. Kiss-icp: In defense of point-to-point icp-simple, accurate, and robust registration if done the right way. IEEE Robotics and Automation Letters, 8(2):1029-1036, 2023.
[84] C. Wang, W. Yang. X. Liu, and T. Zhang. State space model meets transformer: A new paradigm for 3d object detection. International Conference on Learning Representations (ICLR), 2025.
[85] Wang, B. Dedhia, and N. K. Jha. Zero-tprune: Zero-shot token pruning through leveraging of the attention graph in pre-trained transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16070-16079, 2024.
[86] H. Wang, C. Shi, S. Shi, M. Lei, S. Wang, D. He, B. Schiele, and L. Wang. Dsvt: Dynamic sparse voxel transformer with rotated sets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13520-13529, 2023.
[87] P.-S. Wang. Octformer: besree-based transformers for 3d point clouds, ACM Transactions on Graphies (TOG), 42(4):1-11, 2023.
[88] P.-.. Wang, Liu, Y.-X. Guo, C.-Y. Sun, and X. Tong. O-enn: Octree-based convolutional neural networks for 3d shape analysis. ACM Transactions On Graphics (TOG), 36(4):1-11, 2017.
[89] W. Wang, J. Dai, Z. Chen, Z. Huang. Z. Li, X. Zhu, X. Hu, T. Lu, L. Lu, H. Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14408-14419 14419, 2023.
[90] F. Warburg, E. Weber, M. Tancik, A. Holynski, and A. Kanazawa. Nerfbusters: Removing ghostly artifacts from casually captured nerfs. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18120-18130, 2023.
[91] X. Wu, D. DeTone, D. Frost, T. Shen, Xie, N. Yang. J. Engel, R. Newcombe, H. Zhao, and J. Straub. Sonata: Self-supervised learning of reliable point representations. Proceedings of the IEEF/CVF Conference on Computer Vision and Patlern Recognition, 2025.
[92] X. Wu, L. Jiang, P.-S. Wang, Z. Liu, X. Liu, Y. Qiao, W. Ouyang, T. He, and H. Zhao. Point transformer v3: Simpler faster stronger. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4840-4851, 2024.
[93] Wu, Lao, L. Jiang, X. Liu, and H. Zhao. Point transformer v2: Grouped vector attention and partition-based pooling. 35:33330-33342. 2022.
[94] Xie, H. Dai, M. Chen, B. Dai, Zhao, Zha, W. Wei, and T. Pfister. Differentiable top-k with optimal transport. Advances in neural information processing systems, 33:20520-20533 2020.
[95] Q. Xu, X. Sun, C.-Y. Wu, P. Wang, and U. Neumann. Grid-gen for fast and scalable point cloud learning. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 5661-5670, 2020.
16

---

[96] X. Xu, S. Wang, Y. Chen, Y. Zheng, Z. Wei, and J. Liu. Gtp-vit: efficient vision transformers via graph- based token propagation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 86-95. 2024.
[97] Y.QQ. Yang, Y.-X. Guo, J.-Y. Xiong, Y. Liu, H. Pan, P-S. Wang. X. Tong, and B. Guo. Swin3d: A pretrained transformer backbone for 3d indoor scene understanding. Computational Visual Media, 11(1):83-101, 2025.
[98] C. Yeshwanth, Y.-C. Liu, M. Nießner, and A. Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12-22, 2023.
[99] H. Yin, A. Vahdat, M. Alvarez, A. Mallya, Kautz, and Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10809-10818 2022.
[100] Zhang, Guo, P. Gao, R. Fang. Zhao, Wang, Y. Qiao, and H. Li. Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training. Advances in neural information processing systems, 35:27061-27074 2022.
[101] R. Zhang, Z. Guo, W. Zhang, K. Li, X. Miao, B. Cui, Y. Qiao. P. Gao, and H. Li. Pointclip: Point cloud understanding by clip. In Proceedings of the IEEE/CVF conference computer 2022.
vision and pattern recognition, pages
[102] H. Zhao, L. Jiang, J. Jia, P. H. Torr, and V. Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16259-16268, 2021.
[103] Zhou, W. Xiang, C. Li, B. Wang. X. Wei, L. Zhang, M. Keuper, and X. Hua. Sp-vit: Learning 2d spatial priors for vision transformers. The British Machine Vision Conference (BMVC), 2022.
17

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provided experiments to support our claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We included at the end of the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). ). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
All assumptions are already stated.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Ys]]
Justification: we provided experiment descriptions in the main paper and appendix. Further- more, we will release GitHub implementation if the paper is accepted.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reprodueibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification: We will publish the code when the paper accepted or through an anonymous link if some reviewers ask for it.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https:/ /mpps.cc/ppblicc guides htaess//ooSSSSmiiiiooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https wttpsscc/pubic/guuue Sttall/c/ggiidesSodeSSuuuuuuu
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Ys]]
Justification:
We provided details for experiments in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We conduct experiments on diverse datasets and follow the protocol used by previous works for fair comparisons.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This information is included in our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ httpt//p..ce///bbbbee////hhccuue
Answer: [Yes]
Justification: Our paper follows the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our work helps to reduce the carbon footprint when training large models using ViT. There no negative societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have properly cited papers and resources used in our experiment.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, haperswithcodee.com watele/oom//mtset...c /datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We don't have experiments involving crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material fine, but if the main contri- bution of the paper involves human subjects, then much detail possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
23

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: We use LLMs for grammatical checking only.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components. /2025 LLM) for
Please refer our LLM policy what should or should not be described.
24

---

Supplementary Materials for
"How Many Tokens Do 3D Point Cloud Transformer Architectures Really Need?"
Contents
Experiments Setup Details
25
A.I Semantic Segmentation Datasets and Metrics:
25
A.2 Semantic Segmentation Baselines:
26
Replacements for Attention
26
Additional Results
27
C.1
Segmentation Results
27
Reconstruction Results
28
Ablation Study the Number of Bins (K)
28
Local vs Global energy score
29
Complexity Analysis
30
Transfer Entropy Analysis for Token Merging
30
A Experiments Setup Details
A.1 Semantic Segmentation Datasets and Metrics:
S3DIS 2 is a large-scale indoor dataset composed of 3D scans from six areas in office buildings. It includes point-wise semantic annotations across 13 categories, making a common benchmark for semantic segmentation in indoor environments.
ScanNet [16]
is a richly annotated dataset of indoor scenes, consisting of RGB-D videos that are reconstructed into 3D meshes. It provides point-wise semantic labels over 20 object categories and is widely used for evaluating 3D semantic segmentation models.
NuScenes g is an autonomous driving dataset that includes LiDAR point clouds, camera images, and radar data, collected in urban scenes. The 3D semantic segmentation task focuses labeling LiDAR points across 32 object classes.
ScanNet200 198 is an extended version of ScanNet with 200 fine-grained object categories. It introduces a more challenging segmentation task due to its larger label space and long-tail class distribution.
Metries: We evaluate models using several standard metries. mloU (mean Intersection over Union) measures the average overlap between predicted and ground truth labels across all classes. mAce (mean accuracy) computes the average of per-class accuracies, while allAce (overall accuracy) reflects the proportion of correctly classified points over the entire dataset. In addition to accuracy metrics, we report FLOPs (Floating Point Operations) to quantify the computational cost of a model, and PeakMem (Peak Memory Usage), which indicates the maximum GPU memory required during inference. These efficiency metries are critical for understanding model scalability and deployment fcasibility.
25

---

A.2 Semantic Segmentation - -Baselines:
ToMe [5] (Token Merging) is general framework that reduces token count by merging tokens based on feature similarity, originally proposed for vision transformers. PiToMe [8]] extends ToMe to 3D point cloud processing by introducing point-wise importance scores to guide the merging process. Both ToMe and PiToMe are limited to merging up to 50% of the tokens.
ALGM [66] is two-stage token merging approach involving global merging followed by local merging. In our adaptation, we use only the local merging stage, which evenly divides tokens into spatial bins and computes intra-bin similarity. Bins containing highly similar tokens are merged based on a similarity threshold. We evaluate three analytic thresholds for merging: μ. J - o², and μ 20 where j is the mean similarity of tokens within a bin and o² is the variance.
Point Cloud Downsampling Techniques: We evaluate several common downsampling strategies for 3D point clouds. Voxel Downsampling partitions the 3D space into uniform voxels and retains one representative point per voxel. The feature of each representative point is computed as the mean of the features of all original points within the voxel. Furthest Point Sampling (FPS) iteratively selects points such that each newly selected point is as far as possible from previously selected ones, ensuring coverage of the spatial domain. Random Sampling simply selects subset of points uniformly at random from the input set. For all methods, we adjust parameters to ensure that the resulting downsampled point cloud retains approximately 20% of the original points.
B Replacements for Attention
Given the significant reduction in tokens achievable during attention computation, a natural question arises: Is the computationally expensive attention mechanism truly necessary for 3D point cloud processing? ? To investigate this, we explore several alternative token mixing strategies by replacing attention with parameter-free or computationally efficient mechanisms (Fig. (h). The methods are either tested off aff-th fine-tuned (FC) or trained from the scratch (SC). Detailed evaluation is in The methods evaluated include:
Tab. 8
We eliminate the O(N²) attention computation involving keys, queries, and values. Instead, we retain only the linear transformation that produces value features. In this setting, we fine-tune the value projection layer and the subsequent MLP layers, while keeping the rest of the network frozen.
ValueFeat:
MLP: Similar to ValueFeat, self-attention replaced with a multi-layer perceptron (MLP). However, in this configuration, the entire network is trained from scratch.
PoolAttn: To enhance spatial communication, we first apply average pooling with a large kernel size and stride, both set (N) (where N is the number of tokens). Attention is then computed over the pooled features. Finally, unpooling is performed by duplicating the attention outputs back to their corresponding original token positions. is overal attention computation cost is O(log" This can also be considered a token merging methods that rely purely on locality.
AvgPool: We apply average pooling with stride of I and large kernel size. Through empirical evaluation, we found that a kernel size 127 yields the best performance.
ShufflePool: Since AvgPool may lack long-range spatial communication between tokens, we introduce a token shuffling step before pooling. Specifically, we reshape the token sequence as mikknnsreshapeedee (M, N), apply transpose operation cokens.swap swap_axiss(1_ -2), and then flatten it back with tokens. tokens.reshape([ reshape(MxXN] We then apply average pooling as in AvgPool, using stride of and a kernel size of 127, which we found to be optimal.
Progressive-Tomer We design (NN) token merging method by constructing local graph among tokens, where edges are formed between adjacent tokens in a 1D serialized order. In our experiments, we merge the top 80% of edges with the highest similarity by averaging their corresponding tokens. The un-merging process follows the same approach our main method.
Strided-PiTone We integrate PiToMe with our main approach. Following PiToMe [1, we compute importance scores for each token and select the most important tokens within each bin as the destination set for merging.
26

---

Analysis: Our results indicate that without long-range spatial information sharing, the model struggles perform effectively. While the pretrained value features (ValueFeat) retain reasonable performance after fine-tuning (71.3 mloU), the counterpart trained from scratch (MIP) fails to handle the task adequately, achieving only 42.7 mloU.
Introducing spatial information sharing through pooling mechanisms (c.g., PoolAtin, AvgPool) leads significant improvements over ValueFeat (74.8 mloU vs. 71.3 mloU), though the communication remains limited to local neighborhoods. To address this, we employ token shuffling followed by pooling and unshuffling. This parameter-fres method, trained from scratch, nearly matches the performance of the original attention based model (76.2 mloU vs. 77.3 mloU).
This is crucial finding: it suggests that attention layers may not be strictly necessary for 3D point cloud processing. Instead, designing effective spatial information sharing mechanisms could offer a more efficient and competitive alternative.
For off-the-shelf token merging methods, we observe that using a metric-based destination selection, such importance scores from PiToMe, actually degrades performance compared to random sam- pling. We hypothesize that this due to the nature 3D point clouds, where each token corresponds to single point that is sparse and uninformative on its own. Consequently, importance scores computed at high-resolution layers of the U-Net may not be meaningful or reliable.
Additionally, merging tokens based on pooling, without considering similarity, can negatively impact performance. Pooling tends to indiscriminately combine informative and non-informative tokens, leading to a loss of crucial spatial details.
In contrast, find that Progressive Token Merging (Prog-Tome), which relies solely local similarity as the merging criterion, performs comparably to our final method that incorporates both spatial preservation and long-range information sharing (75.5 mloU vs. 77.0 mloU). This highlights the effectiveness of localized, similarity-awaa merging in point cloud processing.
malp
mip
Methods
mAce allAce Lateney (ms)
PTv3 (arigimal)
77.3
85.0
92.3
266
MLP
MLP
ValueFeat (FT)
71.3
81.1
90.6
MLP (SC)
42.7
56.2
77.4
PoolTome (SC)
42.9
57.3
72.7
194
PooTTome (FT)
74.8
83.0
91.2
194
Layer Norm
Layer Norm
AvgPool
68.6
79.1
88.9
AvgPool (FT)
91.2
fattu
Un-Shuffle0
ShufflePool (SC)
92.0
Token Mixer
Self- attention
Pool/Tome
71.0
81.2
90.5
194
Pooling
Prog Prog-Tame
76.7
84.4
91.6
198
Strided-Ptoom
Shuffle0
Ours
84.5
Ours (FT)
92.2
92.2
a) Generic Token Mixing Pipeline
b) Our Shuffle-Pooling
Table 8: Evaluation of attention replace-
Pipeline
ment and token merging methods Scan- Net Val. Methods in the second block are either fully trained from scratch (SC) or fine-tuned (FT). Latency is measured in milliscconds.
Figure 9: Visualization of the token mixing module: a) We evaluate different token mixing techniques replace transformer layers. b) Shuffling the token then
perform pooling layer.
C Additional Results
C.1
Segmentation Results
In Table 10 we present detailed results for two configurations: without branching (i.e., merging is performed uniformly across all regions with fixed merge rate, without using globally informed graph to selectively merge specific 3D patches) and with branching (i.e., merging is guided by
27

---

Table 9: Dataset statistics and evaluation metrics used in the 3D reconstruction task
Dataset
Description
Metric
Description
Objaverse
8
800K+ (and growing) 3D models with descriptive captions, tags, and ani- mations.
SSIM
Structural Similarity Index measures image similarity based on structure, lu- minance, and contrast. Higher is better.
PSNR
Peak Signal-to-Noise Ratio: evaluates reconstruction quality. Higher is better.
RealWorld [14
4 real-world scenes.
Google Scanned
More than 1K 3D- scanned
LPIPS
Learned Perceptual Image Patch Simi- larity: Deep Perceptual Similarity Met- ric. Lower is better.
Objects [3
household
items.
globally informed graph to aggressively combine certain 3D patches). And Figure 10 evaluation of memory consumption during evaluation across different settings.
shows the
ScanNet Peak Memory
ScanNet200 Peak Memory
10
algo
algo
PTv3
PTv3
Our w/o branching
Our w/o branching
Our
Our
RAM
G33
G1
4-
0.0
0.3
0.3
0..0
0.7
0.81
0.9
0.0
0.3
0.3
6..
0..7
0.1
0.9
Merge Rate
Merge Rate
Figure 10: Peak memory consumption evalutation
Table 10: Details results on the segmentation task
ScanNel
ScanNet2000
mloU
mAce
allAce
GFLOPS
mioU
allAce
GFLOPS
PTv3
02
77.68
84.77
91.82
107.5
34.57
45.58
82.79
104.99
branching
91.79
41.37
83.20
w/o branching
77.63
84.62
91.91
66.98
35.09
45.58
83.29
63.89
branching
77.62
83.91
91.57
30.48
34.72
44.37
83.06
27.79
w/o branching
77.69
84.59
91.80 31.80
45.73
34.76
44.92
83.16
42.65
0..6
branching
77.45
83.71
91.48
26.43
34.48
44,07
82.96
24.62
w/o beanching
77.51
84.55
91.79
37.80
34.52
44.55
83.09
34.72
0.7
branching
77.20
83.53
91.39
23.32
34,21
43.74
82.90
22.17
branching
77.31
91.81
83.02
+
0.8
branching
76.98
83.41
91.34
21.10
34.20
43.70
82.98
20.42
branching
+
0.9
branching
76.24
83.05
91.36
19.75
34,38
43.64
83.21
19.39
w/o branching
76.40
87
91.80
27.14
34.54
44.13 13
83.28
21.44
C.2 Reconstruction Results
We provide additional qualitative results for reconstruction task in Figure 13
C.3 Ablation Study on the Number of Bins (K)
The central idea of apatial-presevving token merging to ensure that destination tokens are evenly distributed throughout the input space, which helps maintain both semantic and positional information after merging. our approach, we aggressively merge all source tokens into destination tokens when the merge rate exceeds 50%, making it crucial for these destination tokens to be well-dispersed.
28

---

Without this binning strategy, multiple tokens could collapse into the same destination token, leading significant loss of spatial information. By requiring all source tokens within bin to merge into its corresponding destination token, we promote a more uniform spatial distribution of merged tokens.
'To validate this point, conducted ablation study the ScanNet dataset. Table 11 reports the segmentation result (mloU) for different numbers of bins K and merge rates
r riK
128
64
32
16
0.5
77.04
76.67
76.78
76.80
0.8
76.98
76.33
75.03
73.99
0.9
76.61
75.84
74.38
71.84
Table 11: Ablation study on the number of bins (K) for different merge rates (r). Higher K values
generally preserve spatial information better at aggressive merge rates.
These K parameters, however, are not yet optimal. In our approach, we define K dynamically as:
K= = TT (1-r)
(2)
where the number of tokens each patch. This formulation ensures that the number of bins adapts naturally to the merge rate, maintaining balanced spatial distribution of destination tokens.
Qualitative Comparison: Global vs. Local Energy Score
We provide qualitative comparison between global and local energy scores the ScanNet dataset (without augmentation) to illustrate their impact on performance. Table summarizes the results at different token merge rates.
Table 12: Comparison of global and local energy ScanNet at various merge rates without test-time
augmentation.
Merge Rate Without merging
Global Energy mloU
Local Energy mloU
76.3
76.3
0.3
76.4
76.0
0.5
76.2
75.7
0.8
75.8
74.9
The results show that global energy scores consistently maintain slightly higher performance than local energy scores, especially higher merge rates.
D Local vs Global energy score
Input
PCA-Based Color Mapping
Figure 11:
PCA-Based Color Mapping of all tokens in the last layer of PTv3 model.
To justify the motivation for our globally-informed energy score, we conduct a detailed analysis comparing the behavior of -ocally-informed energy scores used in PiToMe [8]] and explain why it

---

failed for the 3D Point Cloud models. As demonstrated in Figure 11 most points belonging to the same object exhibit similar features, as indicated by their shared color. This suggests that in the initial and final layers, where each patch's receptive field is still local and covers only portion of larger object, individual tokens lack sufficient contextual information. As result, computing the energy score locally within each patch does not accurately reflect a token's alignment with the global feature space formed by all points in the input point cloud.
To mitigate this limitation, we introduce a globally informed energy score. This involves first computing centroids for each patch, followed by calculating each token's energy score as the average of its alignment with all patch centroids. As illustrated in Figure [2] the
energy score provides clearer distinction between foreground and background regions. This enables more effective identification of patches that can be aggressively merged in the initial and final layers. Consequently, tokens representing the foreground are better preserved before entering the middle layers, where the token space is downsampled and each patch has a wider receptive field.
Enenyy Soare B(.)
Lobal Lineryy Scure Visualize
High
Token
Puach Po
Cautoid
Compota Local
Farnyy R(e.)
b) Local Energyy Score
e) Gioball Energy Score Visualize
(4x,7)
(N TTaling)
Comput Global Enengy Stree Bixi]
Comput Caumiid
Input
Global Finergy Score
Figure 12: Visualizing Global (Ours) vs. Local (PiToMe 81
Energy Score for each token
E
Complexity Analysis
We provide our pseudo code for token merging in Algorithm I In our algorithm. the global graph constructed via matrix multiplication between each point and the patch centroids, resulting in a complexity of (NNh). where h is the dimensionality of the input vectors, N is the number of points, and i is the number of patch centroids (with N). The resulting global energy scores are then used to determine which patches should be aggressively merged.
Let denote the number of tokens in each aggressively merged patch, and T' (where n 7) be the number of tokens in each patch. The time complexity of the attention operator can be approximated as O(k(rT))hh + n²h)) (here is the ratio of tokens that remain), capturing the dominant contributors to computational cost. However, actual performance may vary depending on the specific Py Torch version and hardware configuration, optimizations and parallelization may impact the empirical runtime.
F Transfer Entropy Analysis for Token Merging
In this section, we take the first step toward formally demonstrating the redundancy and quantifying the amount of information preserved using the Transfer Entropy framework. Following 33 (3), entropy can be used to measure the information content of network:
30

---

PTv3
Merge Rato #0.5
Merge Rato pa0.9
Ground Truth
Figure 13: Additional Reconstruction Visualizations Generated by PTv3 with different merge rate
H(F) - ( f F.
Since directly measuring the probability distribution of tokens is non-trivial, we approximate it with Gaussian distribution 53
F ~ N(H,0
(4)
The entropy of a feature set is then:
H(F) [logoN(H, (n,
20 (f ui) )
10 log (2 exp
(5)
log(a) + 2
log(2Tr) + 2.
Transfer Entropy Definition: We define Transfer Entropy (TE) as the change in information after applying token merging function:
31

---

Globally Informed Token Merging
Input: Scrialized ID point cloud P e RNxxXC (N patches, K points per patch, C features)
Output: Merged point cloud representation
Step Construct Global Bipartite Graph Compute patch centroids:
1 P3
2k
for each patch P,
24EPs
Construct bipartite graph G (V.E), where:
[xd] (Pz}
. E - directed edges from points to all patch centroids
Step 2: Compute Energy Scores
Define outgoing neighbors (xxx) = (P) (z,, Py) e E
Compute point energy:
F((:;)
Σ
cos(Ei, Pj)
W(xx))
PrEN(a))
Compute patch energy:
E(P3)
Σ F(x)
[P3
.EP
Step 3: Adaptive Merging by Energy foreach patch P3 do
E(,)) > T then
Apply moderate merging [(P,,r)
else end end return Merged point cloud
Apply aggressive merging f(P,. r
Algorithm 1: ALGORITHM FOR GLOBALLY INFORMED TOKEN MERGING
TE H(F) H(Mergod(F)))
(6)
This quantifics the amount of information lost or altered due to merging. The Transfer Entropy Rate is defined as:
TE
Transfer Rate
H.rig
Experimental Setup: We analyze the ScanNet validation set with a default merging rate of 70%. reporting transfer entropy rates for different merging scenarios (the indicate the direction of entropy transfer from source to destination):
1. TE A: Original - Moderate merging (without Global Informed Graph)
2. TE B: Original - Adaptive-aggressive merging (with Global Informed Graph)
3. TE C: Moderate Aggressive merging
32

---

Layer
TE A
B
CEC
0.007
0.045
0.005
0.070
0.030
0.001
14
0.096
0.022
0.051
0.004
0.027
0.031
Table 13: Layer-wise transfer entropy rates.
Layer-wise Transfer Entropy Rate Results
The transfer rates remain consistently small (< 0.1) across layers, indicating minimal information loss from compression via merging functions. This aligns with our segmentation performance results and supports the hypothesis, based on a transfer entropy framework, that token representations in point transformer models contain significant redundancy.
While this does not constitute complete theoretical proof, the transfer entropy framework provides principled analytical assessment that offers clear evidence supporting our findings. We consider this starting point for more rigorous theoretical reasoning.
33


Reviews:
[{"review_id": "qJRdxxIicv", "reviewer": "Reviewer_vPay", "summary": "This research highlights the importance of bringing the idea of \"patchify\" to point cloud transformers, similar to how it's used in image transformers. The motivation is clear: there are just too many tokens to process, which holds back progress in point cloud processing. But applying \"patchify\" to sparse point clouds (without losing too much information) isn't as straightforward as it is with images. This paper tackles that challenge and proposes a method that strikes a solid balance between accuracy and token count.", "questions": "See the discussion in the Weakness section. I initially suggested a `borderline accept` and am open to adjusting my score based on the authors’ rebuttal.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n**[Good Motivation / Direction]** \nOverall, I like this paper’s starting point and think it's doing some of the right things for our community. We've already put a lot of effort into improving the efficiency and scalability of basic point cloud operators. But no matter how far we push those boundaries, the most straightforward way to reduce cost is to process fewer points. That’s why we need the concept of \"patchifying\", similar to what’s been done in image processing. In that sense, the proposed \"merging\" techniques make a lot of sense.\n\n**[Performance]** \nThe accuracy from the merged point cloud reported by the paper is quite acceptable. Compared with the compression ratio, I think it would be helpful to some applications in the robotic learning or autonomous driving area. And also, it is good to extend the downstream tasks from perception to reconstruction.\n\n## Weakness\n**[Efficiency Analysis of Patchifying Itself]**\nI understand there's no doubt that processing a patchified point cloud is more efficient. However, I'm also curious about the efficiency of the patchifying algorithm itself. I think the paper would be stronger if it included some latency benchmarks comparing the different patchifying methods discussed.\n\n**[Absence of ScanNet Semantic Segmentation]**\nI noticed the paper includes semantic results on ScanNet200, so it feels a bit odd that ScanNet itself isn’t included (especially since the main difference is just the label mapping). It would be helpful to at least discuss performance on ScanNet, even if the results aren’t great.\n\n**[Please Polish the TeX (Minor)]**\nI’d suggest the authors double-check the consistency of terminology used throughout the paper. For example, I noticed that Point Transformer V3 is referred to in multiple ways, such as PTv3, PvT-3, and Point Transformer v3. It would be better to stick to a single, consistent naming convention.", "was_revised": true, "final_justification": "The author's rebuttal solved my concern. Therefore, I maintain my recommendation score as \"weakly accept\". Overall, I support the acceptance mostly because I think this is a good direction, worth encouraging the point cloud community to pay more attention to this direction. The manuscript, maybe without too much technical contribution, yet does provide a simple yet effective baseline in this direction.", "rebuttal": "Thank you for your valuable feedback and suggestions. We provide detailed clarifications below.\n\n## ScanNet Results\n\nThe evaluations for the validation sets of both **ScanNet** and **ScanNet200** are included in **Tab. 1** of the manuscript. The results are illustrated in the first and second columns, respectively.\n\n---\n\n## Latency Comparison\n\nWe analyze latency on the validation scenes of **ScanNet** and compare various grouping strategies. For each method, we adjust its parameters so that the number of points retained is around 20% of the original point cloud. The results are summarized below:\n\n| **Method**            | **mIoU (%)** | **Latency (ms)** |\n|-----------------------|--------------|-------------------|\n| PTv3 (base model)           | 77.6        | 266                 |\n| Random Drop           | 70.1         |          154        |\n| FPS                   | 71.2         | 3122                 |\n| Voxel Downsampling    | 72.1         | **160**                 |\n| Progressive           | 75.7         | 198                 |\n| PoolTome              | 71.0         | 194                 |\n| **Ours**              | **77.0**     | 203                 |\n\n---\n\nWe introduce several additional grouping techniques for token reduction (for more information please see **Sec. B** and **Tab. 5** in the supplementary material):\n\n- **Progressive-Tome**  \n  We design an $O(N)$ token merging method by constructing a local graph among tokens, where edges are formed between adjacent tokens in a 1D serialized order. In our experiments, we merge the top 80% of edges with the highest similarity by averaging their corresponding tokens. The unmerging process follows the same approach as our main method.\n\n- **PoolTome**  \n  To enhance spatial communication, we first apply average pooling with a large kernel size and stride, both set to $\\log_2(N)$, where $N$ is the number of tokens. Attention is then computed over the pooled features. Finally, unpooling is performed by duplicating the attention outputs back to their corresponding original token positions. The overall attention computation cost is $O(\\log_2(N))$. This method can be considered a token merging strategy that relies purely on locality.\n\n---\n\nWhile point cloud downsampling techniques such as Random Drop and Voxel Downsampling offer computational efficiency, they are often challenging to apply in off-the-shelf scenarios, as they typically overlook both point-wise feature representations and the latent space of segmentation models. In contrast, our method identifies redundancy using a globally informed graph structure, enabling effective merging and unmerging of points while preserving the features essential for accurate model performance."}, {"review_id": "KovTXArlN1", "reviewer": "Reviewer_7wen", "summary": "This paper adapts and improves token merging techniques from 2D to 3D, demonstrating that 3D point cloud Transformers are heavily over-tokenized. The proposed method is plug-and-play, motivated by the inefficiencies caused by excessive point tokens in both training and inference. However, the merging decisions are based on a fixed heuristic energy score and are only evaluated on indoor benchmarks, raising concerns about the learnability and robustness of the approach in large-scale LiDAR or dynamic scenes.", "questions": "see weakness", "limitations": "see weakness", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The motivation is valuable, as state-of-the-art 3D scene point cloud methods often require a large number of point tokens to achieve strong performance on both local and global tasks.\n\n2. The modifications proposed by the authors to the PTv3 framework are effective, including the design of a globally structure-aware energy definition, adaptive merging rates, and spatially uniform dst.\n\n**Weaknesses**\n1. The use of heuristic energy scores lacks learnability. AdaPT [1], applied in 3D scenes as well, adopts Gumbel-Softmax to enable differentiable token selection and allows dynamic adjustment under computational budgets. However, the current method adopts only two fixed merging ratios, which are neither elegant nor truly dynamic.\n\n2. Several relevant works on token merging or pruning in point clouds have not been discussed, analyzed, or compared, such as YOGO [2], SuperPoint [3], and Token Halting [4]. The current baselines are primarily adapted from 2D methods and directly applied to 3D point clouds, resulting in insufficient baseline comparisons.\n\n3. If further experiments could be conducted in additional domains, such as LiDAR point clouds and object point clouds, it would significantly enhance the impact and credibility of this work.\n\n\n**Lots of Typos**\n1. Line 121      meodels  ->  models\n2. Line 165      Sonana and S3DIS  ->  Sonata\n3. Line 169      GLOPs  ->  GFLOPs\n4. Line 209      differnt dataset  ->  different\n5. Line 240      Furthest Point Sampling  ->  Farthest Point Sampling\n6. Figure 5       nuScenese  ->  nuScenes\n7. Figure 5       Wamo  ->  Waymo\n\n\n\n\n[1] Adaptive Point Transformer\n\n[2] You Only Group Once: Efficient Point-Cloud Processing with Token Representation and Relation Inference Module\n\n[3] Efficient 3D Semantic Segmentation with Superpoint Transformer\n\n[4] Efficient Transformer-based 3D Object Detection with Dynamic Token Halting", "was_revised": true, "final_justification": "The authors’ additional experiments have addressed my concerns; I find the paper’s motivation and significance compelling, and therefore, I raise my rating to 4.", "rebuttal": "## Lacking Learnability\n\nWe agree that while our proposed globally-informed graph provides a way to determine which areas contain more redundant information (and can thus be processed with more aggressive merging), automatically determining the merging rate for each region is indeed a promising next step in our research direction. We view this as a natural extension to make our method more adaptive and fully learnable.\n\n---\n\n## Missing Comparison\n\nWe appreciate the suggestion to compare with other recent token pruning and downsampling strategies. Below, we discuss relevant methods:\n\n- **YOGO**  \n  YOGO performs cross-attention between regions and original points to enable a wider receptive field. However, it relies on furthest point sampling, which is computationally expensive for large scenes with many points and requires many point groups. Additionally, it requires training from scratch. As an attempt to compare, we adopt their grouping strategy in our ablations.\n\n- **TokenHalting**  \n  This method determines which token to prune at each layer using a learnable criterion. At the final detection head, pruned features are restored for full processing. However, this approach is primarily suitable for architectures without explicit spatial downsampling. It is less applicable to our architecture, which benefits from spatial relationships.\n\n- **SuperPoint**  \n  This method proposes a new architecture that gradually groups points into super points. Self-attention is then performed on aggregated features within each group. While efficient, it lacks fine-grained point-level processing and depends on hand-crafted features. It also requires training from scratch.\n\nOur method instead targets redundancy within general-purpose architectures such as UNet and Point Transformer (PTv3, Sonata). These architectures already demonstrate strong performance across both indoor (ScanNet) and outdoor (LiDAR) point cloud benchmarks. Our token merging strategy can be applied off-the-shelf, improving efficiency **without retraining**, which contrasts with the above methods that require full retraining or architectural changes.\n\nWe also tried to adopt YoGo by directly integrating their downsampling methods into the PTv3 model and performing off-the-shelf inference on the ScanNet dataset. we simulated the cross-attention mechanism in two steps: (1) pooling $n$ Key and Value ($K, V$) tokens into $k$ centroids using a kernel size of 64 and a stride of 32 to mimic self-attention across regions, and (2) performing self-attention between the $n$ Query ($Q$) tokens and the $k$ pooled $K, V$ tokens, thus simulating cross-attention between the original points and the regions.\n\n| Method        | mIoU | mAcc | allAcc |\n|---------------|------|------|--------|\n| PTv3          | 77.6 | 87.2 | 94.6   |\n| PTv3 + YoGo   | 66.5 | 72.2 | 86.6   |\n| PTv3 + Ours   | 77.0 | 86.6 | 92.6\n---\n\n## Experiment with LiDAR Point Cloud and Object Point Cloud\n\nFor experiments with LiDAR point clouds, we include results on **nuScenes** [2] in **Tab. 7** of the supplementary material. We report the relevant summary here:\n\n| Method        | mIoU | mAcc | allAcc | peakMem (GB) | GFLOPS | Latency (ms) |\n|---------------|------|------|--------|---------|-------|---------|\n| PTv3          | 80.3 | 87.2 | 94.6   | 6.20|  101.68     |   152      |\n| PTv3 + Ours   | 78.0 | 85.5 | 94.0   | 0.92 |32.45|   106      |\n\n---\n\nWe also evaluate our method with a recent object point cloud segmentation model, **SAMPart3D** [1], on their provided dataset:\n\n| Method              | Human-Shape | Animals | Daily-Used | Buildings&&Outdoor | Transportations | Plants  | Food    | Electronics | Total Mean AP | Peak GPU Mem |\n|---------------------|-------------|---------|------------|--------------------|------------------|---------|---------|-------------|----------------|----------------|\n| **SamPart3D**        | 0.4454      | 0.4224  | 0.3911     | 0.3109             | 0.3683           | 0.4466  | 0.4693  | 0.3576      | 0.3904         | 3.44GB         |\n| **SamPart3D + Ours** | 0.4182      | 0.4194  | 0.3782     | 0.2895             | 0.3329           | 0.4113  | 0.4670  | 0.3955      | 0.3785         |1.45GB         |\n\nWe applied a default merge rate of 0.6 and an aggressive merge rate of 0.9 for the experiments. We did not have enough time to complete our analysis for the GFLOPs in this experiments. However, we reported a significant drop of memory consumption from 3.44GB to 1.45GB while only have 2% drop in total AP score. This suggest that we would likely observe a huge drop rate with GFLOPs. Fine-tunning experiment on this dataset is also promising. \n---\n\nAdditionally, we assess our method on the **SpatialLM** dataset for object detection using the **SpatialLM** 3D language model:\n\n| Configuration   | F1_Layouts_25 | F1_Layouts_50 | F1_Objects_25 | F1_Objects_50 | Avg_Inference_Time | Peak_Memory_GB |\n|-----------------|---------------|---------------|---------------|---------------|--------------------|-----------------|\n| baseline        | 0.4906        | 0.3886        | 0.3356        | 0.1894        | 6.009              | 12.36           |\n| weighted_0.8    | 0.4809        | 0.3873        | 0.3485        | 0.2006        | 4.795              | 2.53            |\n\nThe metrics represent the F1 score of object detection at IoU thresholds of 25 and 50. In the language and 3D-grounded domains, our results show that the tokens in the point cloud transformer remain highly redundant. Our method therefore can significantly boost the computational performance in term of inference speed and peak GPU memory consumption.\n\n---\n\nWe hope these comparisons and extensions clarify the generality and applicability of our method across different domains and architectures.\n\n[1] Mao, Yongsen, et al. \"SpatialLM: Training Large Language Models for Structured Indoor Modeling.\" arXiv preprint arXiv:2506.07491 (2025)."}, {"review_id": "X8Qv4edO8O", "reviewer": "Reviewer_VXJ3", "summary": "This paper tackles the problem of token reduction in 3D point cloud transformers by introducing a token merging strategy aimed at significantly lowering computational cost and memory footprint without sacrificing performance. The proposed method builds upon the Point Transformer V3 architecture and incorporates a domain-specific token merging mechanism at each attention block. Central to this approach is the computation of a globally-informed energy score for each point (token) in the input point cloud. This score aggregates the cosine similarity between a token and all patch centroids, effectively capturing its alignment with the global structure. Based on these energy scores, an adaptive token merging strategy is applied inspired by the ToMe framework: tokens within partitions exhibiting lower average energy, indicating weaker alignment with the global structure, are merged more aggressively, while those with higher energy are preserved, controlled by a tunable partition energy threshold. The method is evaluated on two key tasks, indoor semantic segmentation and 3D reconstruction, demonstrating its effectiveness in maintaining performance while significantly improving efficiency.", "questions": "- The adaptive merging procedure follows a predefined merging ratio $r$, which determines the number of similar edges to retain in the bipartite graph between source and destination tokens. However, for certain partitions, a different ratio  $r^+>>r$ is used. What is the actual value of this higher ratio, and how is it selected?\n- While the discussion on local versus global energy scores in Appendix D is informative, it would be beneficial to include a quantitative comparison to better support the analysis.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- The paper provides a systematic analysis of prior token merging methods and offers a clear comparison with the proposed approach.\n- A thorough ablation study supports the rationale behind key design choices. \n- The proposed method consistently outperforms alternative token merging strategies across all evaluated tasks and matches the accuracy of the original backbone network, while significantly reducing both computational and memory costs.\n\n**Weaknesses:**\n- While the overall analysis is solid, some key aspects remain insufficiently explored. First, prior work such as ToMe suggests that token merging is most effective when applied between the attention and MLP layers within each block, and that an alternating strategy for selecting source and destination tokens in the bipartite graph yields better results. In contrast, as shown in Figure 4, the proposed method applies merging before the attention mechanism, and appears to use a random split of tokens within each patch to form the source and destination sets. This deviation from established findings is not clearly justified.\n- Second, the logic behind the globally-informed energy score raises concerns. According to lines 194–196 and 199–200, tokens with lower energy, i.e., those that are spatially dissimilar to the overall structure, are aggressively merged, while tokens with higher similarity are preserved. However, within each partition, the merging process appears to follow a different logic: similar tokens are merged, while dissimilar ones are retained. This inconsistency suggests that the adaptive merging strategy may be conceptually reversed. Arguably, partitions with tokens that are aligned with the global structure (and thus potentially redundant) should be merged more aggressively, while partitions with spatially distinct tokens should be preserved to maintain diversity.\n- There are structural and editorial inconsistencies throughout the paper. The supplementary material unnecessarily duplicates the entire main paper. Figure 5 presents results for ScanNet200 and S3DIS, yet its caption references additional datasets (nuScenes, Waymo, ScanNet) that are not shown. The second sentence of Table 2’s caption (which is in fact a figure) mistakenly describes Table 1. Figures 1 (main paper) and 10 (supplementary) are duplicates. Additionally, Section 7 of the main paper heavily references analyses from Section B of the supplementary material, which the reader may not have encountered yet, disrupting the flow and self-containment of the discussion.", "was_revised": true, "final_justification": "The authors have addressed my concerns, particularly regarding the token merging placement and the merging strategy for low- and high-energy tokens. I am raising my rating to borderline accept.", "rebuttal": "Thank you for your insightful feedback. We will address both of your concerns, including providing additional justification for the choice of token merging and clarifying any confusion regarding the merging algorithm, and incorporate these changes into our manuscript.## Method Justification: Token Merging Placement\n\nBefore finalizing our approach, we analyzed the bottleneck in the point cloud transformer model and found that while the attention layer is the primary bottleneck, it is also redundant. In this study, we replicate the analysis of different token merging strategies at a 75% merge rate. For this experiment, we adopt a fixed merging rate rather than a weighted scheme. When merging an MLP layer, we use the output features of the self-attention layer before it as the merging criterion. We evaluated three configurations: merging only the MLP layers (MLP), merging only the attention layers (Attention), and merging both the MLP and attention layers (Attention + MLP). We use Scannet validation set for this experiment.\n\n| Placement             | mIoU  | Latency |\n|-----------------------|-------|---------|\n| PTV3 (Base model)     | 77.7  | 260     |\n| MLP                   | 26.3  | 277     |\n| Attention             | 77.4  | 203     |\n| Attention + MLP       | 25.0  | 212     |\n\nThe bottleneck is clearly located in the Attention layer, suggesting that merging at this stage significantly impacts performance. In contrast, the MLP layers, which preserve more detailed information, are critical for maintaining accuracy. Interestingly, increasing computational effort for token merging within the MLP layers led to higher overall computational costs. Based on this analysis, we chose to focus on merging within the Attention layers.\n\n\n---\n\n## Conflicting Reasoning\n\nWe acknowledge a mistake in the original writing (Line 196). We propose the revised clarification for Lines 196–197:\n\n> \"Tokens with lower energy (i.e., **more aligned with global structure**) are considered less informative and can be merged more aggressively, whereas high-energy tokens (i.e., **less aligned with global structure**) are preserved to retain critical information.\"\n\n---\n\n## Writing Improvements\n\nWe appreciate the feedback and will revise the manuscript to enhance clarity. Specific improvements include:\n\n- Reorganizing the order and flow of supplementary material.\n- Correcting figure captions and resolving typographical issues (e.g., repeated \"the\").\n\n---\n\n## Merge Rate Selection\n\nWe empirically determined a **fixed merging rate of 32** using the validation set of **ScanNet** and applied it across all datasets. Although this fixed rate yields strong performance, we acknowledge the potential benefits of **automatically determining adaptive merge rates per patch**—a promising future direction.\n\n---\n\n\n## Qualitative Comparison: Global vs. Local Energy Score\nThank you for your suggestion. We will include a qualitative comparison between global and local energy scores in our revised version. Below are the comparison results on the ScanNet dataset (without augmentation):\n\n| Merge Rate      | **Global Energy Score** | **Local Energy Score** |\n|--------------------|--------------------------|--------------------------|\n|      Without merging      |       76.3                   |            76.3              |\n|      $ r=0.3$        |       76.4                   |            76.0              |\n|      $ r=0.5$        |        76.2                 |              75.7           |\n|      $ r=0.8$        |      75.8                    |               74.9           |"}, {"review_id": "oqLCb4zQzN", "reviewer": "Reviewer_4bMr", "summary": "The paper puts forward an interesting point of view: only 5%-10% of the tokens are needed to represent the original complex and redundant token sequence without significantly affecting the performance. Based on this idea, the paper proposes a specific token merging strategy for 3D transformer, which integrates local geometric structure with attention saliency to evaluate the importance of voxels, thereby achieving significant token reduction while maintaining minimal accuracy loss. Finally, a large number of experiments verify the effectiveness of the proposed token merging strategy in 3D semantic segmentation, reconstruction and detection tasks. It is worth mentioning that the proposed method can even surpass the baseline through a fine-tuning strategy, after removing 90%-95% of the tokens.", "questions": "In addition to the weaknesses mentioned, please solve the following confusions:\n1. Can other token merging strategies, such as ToMe, ALGM, PiToMe, etc., improve their compression rate so that GFLOPS remains at a level comparable to that in this article, and then evaluate their performance? For example, Figure 2.\n2. In Table 1, why did the performance decrease after fine-tuning on the S3DIS dataset? This result should be discussed specifically.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n1. This paper proposes a specific token merging strategy for 3D transformer, which evaluates the importance of voxels by integrating local geometric structure with attention saliency, thereby achieving significant token reduction while maintaining minimal accuracy loss.\n2. This paper conducts a large number of experiments to verify the effectiveness of the proposed token merging strategy in 3D semantic segmentation, reconstruction and detection tasks. It is worth mentioning that the proposed method can even surpass the baseline by fine-tuning the strategy while removing 90%-95% of the tokens.\n\n**Weaknesses**:\n1. The method is too simple. The proposed token merging method does not involve any parameter updates, and the method is too simple.\n2. Lack of more experimental verification. The method proposed in this paper should be plug-and-play. In addition to PTv3, it can also be adapted to other transformer-based models in theory. The author should do more experiments to verify the feasibility and applicability of this method.\n3. The icon title and caption are wrong. Table 2 should actually be Figure 6.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful feedback. We will incorporate both your concerns and the discussion into our manuscript.\n## Other Model\n\nTheoretically, our technique can be applied to other point transformer models such as **OctFormer**[2] or **FlatFormer**[1]. However, **PTv3** and **Sonata** are the first transformer-based models to operate on point clouds at a fine-grained level, processing self-attention across more than 1000 points simultaneously. This scalability is particularly beneficial for large-scale scenes, where millions of points may exist with uneven density and varied semantic importance (e.g., a few points may hold crucial information while many do not). This makes point-level self-attention crucial.\n\nIn contrast, other models group points first (e.g., **PointNet**, **PointNet++**), then apply attention with a limited number of points (up to 128 for **OctFormer** and **FlatFormer**), limiting their capability. This also reduces attention overhead, but at the cost of representation power.\n\nNotably, **Sonata** is the first pretrained foundation model at the scene level, with a large receptive field for self-attention. It has also been used to power 3D-language models such as **SpatialLM**[3].\n\n## Simplicity\n\nA key insight in our work is that point transformer models overuse tokens, leading to excessive computation and memory usage. Our globally-informed graph provides additional context to the local transformer, helping determine which areas contain more informative tokens.\n\nThe simplicity of our method ensures robustness and can be used to for future design decisions in efficient 3D point cloud processing.\n\n## Adaptation of Other Methods\n\nTo extend the evaluation scope for other baselines, such as ALGM, we tested their performance across different threshold settings. For PiToMe, applying merging rates above 50% is theoretically infeasible due to its formulation, which requires maintaining a set of preserved tokens. When the merging ratio exceeds 50%, all tokens become candidates for merging, resulting in the concept of a \"preserved set\" invalid.\nWe explored adapting their energy-based formulation to determine a set of protected tokens, but found that this approach yielded worse results compared to simply allowing all tokens to be considered for merging when the merging ratio surpasses 50%. Here, we present our analysis of adapting PiToMe to our patchified merging technique, evaluated on the Scannet validation set. By fixing a percentage of tokens to remain unmerged (selected using PiToMe's energy score), we merged the remaining tokens, ensuring that the total number of retained tokens (merged and protected) was 20% of the original token count.\n\n| Method                       | mIoU  |\n|------------------------------|-------|\n| Our                           | 77.0  |\n| Our + 1% Protected (PiToMe)   | 76.8  |\n| Our + 5% Protected (PiToMe)   | 76.3  |\n| Our + 10% Protected (PiToMe)  | 75.4  |\n\nBased on the experimental results, our method, by distributing DST sets more evenly across the point cloud to cover a broader range of areas, is more effective than the sparse distribution of DST sets used in PiToMe to maintain a fixed protected set.\n\n\n## S3DIS Score Clarification\n\nWe mistakenly swapped the fine-tuned and off-the-shelf evaluation scores. The corrected results are:\n\n- Fine-tuned: 74.3  \n- Off-the-shelf: 72.3\n\n[1]Liu, Zhijian, et al. \"Flatformer: Flattened window attention for efficient point cloud transformer.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.\n[2]Wang, Peng-Shuai. \"Octformer: Octree-based transformers for 3d point clouds.\" ACM Transactions on Graphics (TOG) 42.4 (2023): 1-11.\n[3]Mao, Yongsen, et al. \"SpatialLM: Training Large Language Models for Structured Indoor Modeling.\" arXiv preprint arXiv:2506.07491 (2025)."}, {"review_id": "HpkSLtsRt5", "reviewer": "Reviewer_cv9K", "summary": "In this paper, the authors observe that existing 3D point cloud transformers rely on excessive redundant tokens, yet only a small critical subset is essential for performance. Merging tokens based on spatial importance reduces usage by 90-95% with minimal accuracy loss, enabling efficient scalability across multiple vision tasks.", "questions": "(1) Could the authors provide a theoretical analysis or formal justification for the observed token sparsity in their Transformer-based framework? Specifically, what underlying principles or properties of point cloud data lead to this sparsity, and how might this connect to existing theoretical work on token efficiency in attention mechanisms?\n\n(2) Could the authors explicitly clarify the key technical distinctions between their Aggressive Merging Method and the prior approach in [1]?\n\n(3) Explicitly contrast their merging process with RandLA-Net [2], highlighting both algorithmic similarities and any substantively novel aspects?\n\n(4) How does the choice of parameter K in the Figure 4 (c) influence the model’s ability to preserve local geometric structures during the SA Merge operation? Could the authors provide an ablation study analyzing the sensitivity of performance to different K values, particularly given that uniform partitioning may not optimally adapt to varying local point densities?\n\n(5) Could the authors discuss potential challenges or limitations in extending the proposed method to other frameworks beyond PTv3?\n\n[1] Bolya D, Fu C Y, Dai X, et al. Token merging: Your vit but faster[J]. arXiv pre\n[2] Hu Q, Yang B, Xie L, et al. Randla-net: Efficient semantic segmentation of large-scale point clouds[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 11108-11117.", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. Timely and impactful:\n- The paper tackles a highly relevant issue—scalability and inefficiency in transformer-based 3D models.\n- It challenges the implicit assumption that “more tokens = better performance,” backed by robust empirical evidence.\n\n2. Methodological novelty:\n- The proposed global-informed energy score for adaptive merging is well motivated and exploits domain-specific spatial structure, improving upon generic token reduction strategies.\n\n3. Strong empirical validation:\n- Experiments span three core 3D tasks with multiple datasets (ScanNet, S3DIS, ShapeNet, Objaverse, GSO, Waymo), showing generality.\n- Ablations on merging strategies, thresholds, and feature types (Q, K, V) are interesting.\n\n4.\tPractical implications:\n- The method operates inference-time without retraining, and can even outperform the baseline with minimal fine-tuning.\n- Clear gains in FLOPs and memory make the work valuable for real-world deployment (e.g., mobile robotics, AR/VR).\n\n5.\tComprehensive related work and discussion:\n- Covers point cloud, transformer compression, and downsampling literature in a balanced way.\n- Acknowledges open problems (e.g., automatic merging rate optimization, extension to meshes or real-time streams).\n\nWeaknesses: \n- While this work is primarily motivated by empirical observations, it lacks a rigorous theoretical analysis regarding the inherent sparsity of tokens in Transformer-based architectures. \n\n- The proposed Aggressive Merging Method demonstrates only marginal technical differentiation from the prior approach [1], thereby constraining the innovative contribution of this work.\n\t\n- The methodological pipeline presented in Figure 4 lacks sufficient detail to clearly elucidate the functional relationships between components (a), (b), and (c). Furthermore, the figure contains labeling inaccuracies, particularly in the misassignment of r⁺ and r in (a), and the incorrect color coding of source (red) and destination (blue) points in (c). Upon closer examination, the merging process appears fundamentally similar to the down-sampling strategy employed in RandLA-Net [2], raising questions about the claimed technical novelty of this aspect of the proposed method.\n\t\n- The manuscript fails to provide a formal definition for term ‘E’ introduced in line 191, which impedes proper understanding of the subsequent mathematical derivation.\n\t\n- The ablation study fails to evaluate the impact of parameter K in Figure 4 (c). The current approach of uniformly dividing each patch into K spatial bins may adversely affect the model’s ability to capture local geometric structures in the SA Merge operation. \n\t\n- While I greatly appreciate the thorough experiments and validations presented in the supplementary materials, the proposed method has only been validated on the PTv3 framework. This limited scope makes it difficult to assess the generalizability of the approach. To strengthen the claims, the authors should demonstrate its effectiveness across additional baselines or diverse benchmarks.\n\n[1] Bolya D, Fu C Y, Dai X, et al. Token merging: Your vit but faster[J]. arXiv pre\n[2] Hu Q, Yang B, Xie L, et al. Randla-net: Efficient semantic segmentation of large-scale point clouds[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 11108-11117.", "was_revised": true, "final_justification": "I maintain my score, which I believe reflects the strenghts and weaknesses of the work.", "rebuttal": "Thank you for the insightful feedback, we would like to answer your questions as follows:\n## Why tokens in point cloud transformers are highly redundant?\n\nThank you for the question, it touches on a key motivation behind our work and points to an important direction for future research. Our intuition regarding the high token redundancy in point cloud transformers stems from the inherent properties of 3D data similar to how single pixels in a 2D image are not very informative on their own. Unlike in Vision Transformers, where semantic information is often captured in structured patches (e.g., 16×16 pixels), 3D point clouds are unordered, spatially noisy, and frequently incomplete. Individual points in 3D space typically carry very limited information, and many of them correspond to repetitive structures within larger objects, such as flat surfaces or repeated geometric patterns, which further amplifies redundancy.\n\nThis perspective is further supported by the design of state-of-the-art methods like Point Transformer v3 and Sonata, which rely on fine-grained attention mechanisms to model local context. In both the first and last layers of these transformer backbones, point tokens tend to exhibit high redundancy, a phenomenon we visualize using PCA projections. At these stages, individual tokens contribute minimal unique information, making self-attention computationally expensive and inefficient. Nevertheless, it's important to retain a subset of these tokens, as some may belong to semantically rich or structurally complex objects that become meaningful only when captured by the broader receptive fields of deeper layers. Identifying which tokens to preserve is non-trivial, and this challenge motivates our proposed globally-informed graph representation, which guides token selection based on global context.\n\nIn future work, we plan to explore this redundancy more formally by applying the transfer entropy framework to analyze and justify the globally-informed graph representation introduced in our method. We believe this will offer both theoretical insights and practical improvements for designing more efficient transformer architectures in the 3D domain.\n\n##  Aggressive Merging Method versus ToMe \n- Findings: ToMe and other prior works on Vision Transformers primarily focus on 2D images, where each token typically covers a relatively large and semantically meaningful region of the image. This makes individual tokens highly informative, which in turn limits the degree of safe token merging, usually only a small fraction of tokens are merged progressively at each layer to avoid significant information loss. In contrast, our study highlights a key difference in the 3D domain: transformer backbones such as PTv2, PTv3, and Sonata frequently encounter substantial token redundancy. This might stems from the structural repetition inherent in 3D scenes. As a result,  we observe that the attention mechanism can become overused on redundant or uninformative tokens, particularly in the early and late layers where token dimensionality is shallow. We believe these findings will open up new directions for designing more efficient transformer architectures tailored to the unique characteristics of 3D data.\n\n- Method: Leverage our finding, we propose a token compression strategy that aggressively reduces redundancy by identifying and merging tokens in low-energy regions of the 3D scene. Our method employs a globally-informed energy scoring mechanism to quantify and detect  redundant regions. By merging up to 99% of the tokens, while preserving the most informative ones, we demonstrate that a well-design token compression method can significantly reduces computational cost with negligible impact on accuracy.\n\n\n## Ablation study for the number of bins ( $K$ )\nThe central idea of spatial-preserving token merging is to ensure that destination tokens are evenly distributed throughout the input space, which helps maintain both semantic and positional information after merging. In our approach, we aggressively merge all source tokens into destination tokens when the merge rate $r$ exceeds 50%, making it crucial for these destination tokens to be well-dispersed. Without this binning strategy, multiple tokens could collapse into the same destination token, leading to significant loss of spatial information. By requiring all source tokens within a bin to merge into its corresponding destination token, we promote a more uniform spatial distribution of merged tokens. To validate this point, we conducted an ablation study on the ScanNet dataset.\n\n\n|               | $k=128$| $k=64$ | $k=32$| $k = 16$ |\n|----------|------- |------|------|----|\n| $r=0.5$    | 77.04  | 76.67 |76.78| 76.80| \n| $r=0.8$    | 76.98  | 76.33| 75.03| 73.99|  \n| $r=0.9$    | 76.61  | 75.84| 74.38| 71.84| \n\nThese $K$ parameters however is not yet optimal, in our paper we define $k$ as: $\\lfloor T * (1 - r)\\rfloor$ , where $T$ is the number of token in each patch.\n\n## Comparision with  RandLA-Net \n\n- A key difference between RandLA-Net and our method lies in their design philosophy and integration. Our method is an off-the-shelf technique aimed at reducing redundant information and computational overhead inherent in large-scale point transformer architectures such as PTv3 and the foundation model Sonata. In contrast, RandLA-Net is an independent architecture that must be trained from scratch.\n- Our framework also includes an analytical tool that identifies redundant regions in the point cloud using a proposed globally-informed graph structure, enabling more effective compression. Before comparing our method directly with RandLA-Net, it is instructive to contrast the underlying architectures of PTv3/Sonata and RandLA-Net.\nWhile both PTv3 and RandLA-Net are designed for large-scale point clouds (on the order of millions of points), they adopt fundamentally different strategies. RandLA-Net employs random sampling and local feature aggregation based on KNN, which can become computationally expensive and less scalable. PTv3, however, demonstrated that the KNN-based aggregation used in RandLA-Net imposes a computational bottleneck and limited receptive field. To address this, PTv3 introduced a serialization scheme that orders and groups points sequentially, replacing the costly KNN approach.\n- Our method builds upon PTv3's serialization strategy by dividing space into smaller scales and selecting a representative set of DST tokens, rather than sampling points randomly as RandLA-Net does. This preserves spatial structure more effectively and avoids the clustering issues inherent to random sampling. Moreover, our method performs aggregation using DST tokens at a more global scale (e.g., 1000 points), as opposed to RandLA-Net's local KNN-based selection, further enhancing efficiency and representation power.\n\n## Beyond PTv3 and Sonata \nPTv3 and Sonata are the first large-scale, scene-level point cloud transformers capable of performing self-attention across thousands of points simultaneously. This design allows them to capture fine-grained and important details during training. However, it also leads to significant redundancy in the data, an issue our method aims to address.\n\nIn contrast, other techniques mitigate computational cost by partitioning the point cloud before processing, either through spatial grouping (e.g., RandLA-Net, Yogo [1], SuperPoint [2]) or via 1D serialization with localized self-attention (e.g., OctFormer [3], which attends to up to 128 points). The grouping-based methods typically exhibit lower redundancy due to their coarse representation and small number of groups, meaning that pruning one or two groups has limited impact on efficiency. Meanwhile, in serialization-based approaches, the self-attention layer is not the main bottleneck, the point cloud itself is. Thus, a different strategy is needed, such as completely or temporarily pruning redundant points.\n\n\nGiven the demonstrated strength of PTv3 and Sonata as state-of-the-art 3D architectures and foundational models for 3D pretraining, we anticipate that future architectures will build upon them. We believe integrating our redundancy optimization method into the training process of such models is a promising direction, particularly as our work so far has focused primarily on improving inference efficiency.\n\n\n[1] You Only Group Once: Efficient Point-Cloud Processing with Token Representation and Relation Inference Module\n\n[2] Efficient 3D Semantic Segmentation with Superpoint Transformer\n\n[3] Octformer: Octree-based transformers for 3d point clouds"}]

Author rebuttal:

Meta-review: This paper proposes token merging techniques for 3D point cloud transformers, showing that existing 3D transformer architectures are heavily over-tokenized. The main strength of the paper is the effectiveness of the proposed modifications to the PTv3 framework -- the method can even surpass the baseline by fine-tuning the strategy while removing 90–95% of the tokens.

Several weaknesses were identified, such as the absence of latency benchmarks, limited discussion of other token-merging/pruning techniques (e.g., Yogo), limited experiments on LiDAR point clouds, missing adaptations of other token-merging strategies, reliance on heuristic energy scores, insufficient theoretical justification, and questions about novelty relative to existing methods (e.g., RandLA-Net).

The authors provided extensive clarifications, including latency experiments, comparisons with other merging techniques, additional results on LiDAR point clouds, adaptations of alternative token-merging strategies. Four out of five reviewers were satisfied with the responses and voted for acceptance. One reviewer, however, remained negative, expressing concerns about novelty relative to RandLA-Net and claiming limited empirical generalization and theoretical justification. The AC finds that these concerns were not sufficiently substantiated by the reviewer. The authors presented experiments on diverse datasets (nuScenes, SAMPart3D, SpatialLM), demonstrating robustness across various settings, from indoor scenes to autonomous driving. The AC agrees with the majority of reviewers that the method is sufficiently distinct from prior work (including RandLA-Net) and provides adequate intuition for its design choices.

Thus, the AC recommends acceptance. The authors are strongly encouraged to incorporate the additional experiments, results, and clarifications from the rebuttal and discussion into the final version of the paper.
Decision: Accept (poster)