Frequency-Aware Token Reduction for Efficient Vision Transformer
Authors: Dong-Jae Lee
Jiwan Hur
Jaehyun Choi
Jaemyung Yu
Junmo Kim
Venue: NeurIPS 2025 poster ()
Keywords: Efficient Transformer
Token pruning
Token merging
Primary area: deep_learning
TL;DR: Token reduction with mitigating rank-collapsing (over-smoothing) through high-frequency token preservation

Abstract: Vision Transformers have demonstrated exceptional performance across various computer vision tasks, yet their quadratic computational complexity concerning token length remains a significant challenge. To address this, token reduction methods have been widely explored. However, existing approaches often overlook the frequency characteristics of self-attention, such as rank collapsing and over-smoothing phenomenon.
In this paper, we propose a frequency-aware token reduction strategy that improves computational efficiency while preserving performance by mitigating rank collapsing. Our method partitions tokens into high-frequency tokens and low-frequency tokens. high-frequency tokens are selectively preserved, while low-frequency tokens are aggregated into a compact direct current token to retain essential low-frequency components. 
Through extensive experiments and analysis, we demonstrate that our approach significantly improves accuracy while reducing computational overhead and mitigating rank collapsing and over smoothing. Furthermore, we analyze the previous methods, shedding light on their implicit frequency characteristics and limitations. The code is available in https://github.com/jhtwosun/frequency-aware-token-pruning.

Freauency-Aware Token Reduction for Efficient Vision Transformer
Dong-Jae Leel
Jiwan Hur
Jaehyun Choi
Jaemyung Yu?
Junmo Kim
KAIST
NAVER AI Lab
(jhtwosun, jiwan. hur, chlvogus> kc.kr, jaemyung- Soaryyuun-uunaavecrrrr com, junmo. kim@kaist.ac.
Abstract
Vision Transformers have demonstrated exceptional performance across various computer vision tasks, yet their quadratic computational complexity concerning token length remains a significant challenge. To address this, token reduction methods have been widely explored. However, existing approaches often over- look the frequency characteristies of self-attention, such rank collapsing and over-smoothing phenomenon. this paper, we propose a frequency-aware to- ken reduction strategy that improves computational efficiency while preserving performance by mitigating rank collapsing. Our method partitions tokens into high-frequency tokens and low-frequende tokens. figh-freuuecc tokens are selec- tively preserved, while low-frequency tokens are aggregated into a compact direct current token retain essential low-frequency components. Through extensive experiments and analysis, we demonstrate that our approach significantly improves accuracy while reducing computational overhead and mitigating rank collapsing and over smoothing. Furthermore, we analyze the previous methods, shedding light their implicit frequency characteristics and limitations. The code is available sewaaretokee-prunn
Introduction
The advent of Vision Transformers (ViTs) [8] has marked significant milestone in computer vision, demonstrating remarkable performance on various benchmarks. Following this success, research has focused on utilizing ViTs in various applications. However, their computational overhead has significantly hindered their deployment in real-world applications, primarily due to the quadratic complexity of self-attention with respect token length. This has motivated extensive work on token- reduction, which aims to lower computational cost by discarding aggregating tokens with minimal impact on performance. Prior token-reduction methods fall into two main categories: token merging and pruning. Merging methods fuse similar neighboring tokens, whereas pruning methods remove less informative tokens and retain only the most important ones. Both strategies have demonstrated significant computational savings with minimal performance loss.
Meanwhile, another line of research has focused on the theoretical properties of ViTs to analyze their characteristics. Recent studies have identified a rank-collapsing phenomenon, in which the self-attention layer in Transformer causes the feature to converge a rank-1 matrix, where all tokens share the same representation [6]. From a frequency-domen
perspective, researchers [26, 18] showed that stacking self-attention blocks is analogous to applying a repeated low-pass filter. Consequently, self-attention primarily preserves the direct current (DC) component, i.e., the gero-frequency component the input, leading to an over-smoothing feature map. This leads to a loss of token-level information, limiting the model's capacity and expressive power. Subsequent work has demonstrated that mitigating rank collapse and integrating both low- and high-frequency components can significantly improve the performance of ViT [34, 19, 32, 18, 26].
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Considering such analysis, token reduction should avoid discarding tokens containing relatively high- frequency information, as this can expedite over-smoothing and limit expressive power. However, previous token reduction methods overlook such connection with high-frequency information and rank collapse. Consequently, they fail to preserve such information effectively, accelerating the rank collapse. Merging-baset approaches naturally suppress high-frequent signals during token aggrega- tion, accelerating rank-collapsing. Pruning-basee methods may retain the preserved tokens capture high-frequency content. However, as we later show (Section 6.3), observe that pruning methods preserve nigh-frequency tokens only in the near-last layers, and their behavior varies substantially across different layer depths.
information
In this paper, we propose a frequency- awwar token reduction, which focuses on mitigating the over-smoothing phenomenon and preserving the expressive power of ViTs while lowering their computational cost. Specifically, we partition the token set into two subsets: one containing primarily high-frequency information and another dominated by low-frequency content. By selectively retaining high-frequency tokens and reducing low-frequency tokens, we minimize the loss of higu-fre signals, thereby alleviating rank collapse and over-smoothing phenomenon. To further mitigate the information loss from discarding low-frequency tokens, we aggregate their DC signals into a single token-acting as compact representation of the low-frequency information.
Through extensive analyses and experiments across various models, we demonstrate that the proposed method effectively alleviates rank collapse and preserves accuracy while reducing computational costs. Furthermore, we analyze previous methods from a frequency perspective, shedding light on their frequency characteristics and limitations.
2 Background
2.1 Vision Transformer
A Vision Transformer (ViT) block consists of two primary components: Multi-Head Self-Attention (MSA) and Feed-Forward Network (FFN). Typically, the ViT transforms input image I R ff W xC into tokens X E R=xd through patch embedding, n = HW/p² represents the number of tokens where p the patch size, and d the feature dimension. Special tokens (e.g.. the CLS token) may optionally be appended to facilitate tasks such as classification.
The key characteristic of the transformer lies in its Self-Attention (SA) layer. SA encodes the token representations from the previous layer X E Rn by aggregating information from other tokens via an attention weight matrix A E R=x", Formally, it is defined as [25]:
SA(X)= AXWv
XW @XWW K) XWV, vdl
softmax
where Wo E Raxd,, WK E Rdddd and Wy Rddxd are the query, key, and value weight matrices, respectively, and dy. de, do denote the dimensions of the query, key, and value vectors, respectively. The term vdz serves as a scaling factor, and softmax(-) is applied row-wise. MSA aggregates outputs from multiple SA heads and projects them into the hidden dimension via linear transformation.
2.2 Over-smoothing of self-attention
Extensive studies show that stacking SA leads to rank collapse or over-smoothing phenomena, which reduces the token diversity and hinders the model from learning rich features [6]. Formally, is shown that the following proposition holds 26].
Proposition 2.1. Let the mean-centered matrix of the feature matrix X be [x] (I- -11-111 which can also be viewed as high-pass filtered version of X. Then, SA reduces the high-rreqqency component with collapsing ratio X by:
|H,|SA(X)]]I S [X|I|
For ViTs, researchers [26, 18] showed that the attention matrix behaves as a low-pass filter, preserving the direct component (DC) signal, i.c., the eero-freyunncy component at the output layer,

---

causing an wvershhnggin problem. As the collapsing phenomenon becomes more severe, A becomes closer low-pass filter. i.e., A = 11 Consequently, SA fails to capture meaningful token interactions, causing the output features to become similar across adjacent layers. A deeper background available in the Appendix A.
To address this issue, recent research has emphasized the importance of counteracting over-smoothing by balancing low-frequency and high-frequency components, which can significantly enhance the performance of ViTs. Various approaches have been proposed, including designing modules to empha- size freuufrreuency information, decomposing A into pww-psss and high- nigh-pass filters, incorporating convolution layers, and improving training strategies [18, 26, 32, 19, 34].
3
Frequency-Aware Token Reduction
3.1 Token reduction in frequency aspect
0.0
-2.58
A Log amp Loggamplnude linde
1p60 Pactening
Log ampöitule:
Figure/ provides empirical visualization of the ralk-colla phenomenon. The ampli- tude of high-free fege-freuenncy components decreases with increasing layer depth. A similar trend is observed when comparing the relative ampli- tude of high-frequency signals across different layers (Figure 1b). The slower decrease of high- frequency signals observed in the early layers can be attributed to their convolution-like charac- teristics. Specifically, early self-attention layers predominantly interact with spatially adjacent tokens. due to the characteristics of natural im- age [18]. As the layer goes deeper, the feature loses high-freeuenge information, leading to limitations in the token diversity.
-1.0-
5.
2.0-
-3.3
-.0-
3.40
0.9m
Froquency
1ibx
Normelized depth
(a) A log amplitude
(b) Relative high free-fryy
Figure 1: Frequency analysis sthe elf-attention layer according to the layer depth. amplitude of frequency. (b) Relative log amplitude of high- tigh-frequency (1.0x).
(a) Relative log
Such rank collapsing and over-smoothing limit the model's capacity and expressive power. Therefore, preserving high-freen figh-frequency signals is crucial for increasing the performance of ViT's [34, 19). However, token reduction inherently leads to the loss of high-frequency signals, exacerbating rank collapse and diminishing token diversity. Existing token reduction methods overlook these frequency perspectives. Specifically, pruning and merging operations decrease high-frequency information unless the selected token is already collapsed, i.e., = Σ3 2g, which leads to accelerating the collapsing. Formally, we have the following propositions.
Proposition 3.1. Let M 1n denotes row-normalized matrix, where n' <, If the (i,j) element of M is binary, i.e. ma,j E {0, 1}. M serve as row-selection matrix, and MX represents the token pruning. Similarly, ma.j E (0,1), then MX represents the token- moken-merging. Then, have:
|M)|SA(MX)]II < |SA((A( S |X]||
Derivation of Equation (4) can be found in Appendix B. Intuitively, merging-baid based methods reduce token diversity thereby accelerating rank collapse. Pruning- based methods are similarly susceptible if the tokens removed predominantly contain high-frequency signals. Consequently, token reduction accelerates the rank collapsing, hindering the model's 's ability to represent diverse features and limiting the model's capacity and expressive power.
Given these limitations, a more effective approach should explicitly consider the frequency aspect when performing token reduction. To address these issues, we propose novel approach that explicitly selects and retains tokens contributing predominantly to high-frequency components while merging those associated with low- www-freqency information. This frequency-awan token reduction strategy aims to mitigate rank collapse while reducing computational costs.
3.2 Selection of Frequency token
For frequency-aware token-reducing, we first separate tokens into two sets: those primarily containing gigh-frequency signals (HF tokens) and those dominated by low-frequency signals (LF tokens). For separating the token sets, a naive method involves calculating the DC signal of the entire feature and

---

Hlgh frequncc Low drequency ukem
0.8
All skoz
4 Log amplitude
CKA Similarit
Accura (%)
20
0.2
lligh- reghffrcenyyy Low Deguenyyyyyyyyy tokm
35
Low Eegonny tokes
Normalized Layer Depth
Normalized Layer Depth
Normalized Layer Depth
(a) A log of high-frequency
(b) Similarity to DC signal
(c) Effect of AWGN
Figure Analysis of HF tokens and LF tokens. (a) Relative log amplitude of high-frequene
(1.0x). HF tokens contain relatively more high-frequency signals than LF tokens. (b) Similarity with DC-signal. The LF tokens dominantly contain DC signal features compared to the HF tokens. (c) Effect of white Gaussian noise (AWGN) on each token set. We report mean accuracy of 10 trials; the gray dashed line represents the initial accuracy. The results show that the HF tokens are more critical in maintaining the model accuracy compared to the LF tokens.
then measuring the similarity of each token to it. Tokens exhibiting high similarity can be classified LF tokens. However, this method introduces additional computational overhead, which contradicts the primary objective of reducing computational cost in token reduction.
Instead, we decompose the attention map A into a low-frequency attention map A T.P and a high- frequency attention map A HP!
A ALP -11T
(5)
11
HP A A LP
(6)
In the frequency view, A I. X represents the zero-frequency component (DC bias) of F(AX). where FF represents discrete fourier transforms. The residual term, A HP X, represents the residual high-frequency terms. In the matrix A HP Raax" the (4, k)-th element quantifies the contribution of the k-th token to the high-frequency component of the q-th output token. Therefore, a centain column of dμP has relatively larger weights than other columns, the token corresponding to this column contributes more to the high-frequency components of the output tokens and can be classified HF token. Conversely, tokens with lower weights are classified as LF tokens.
We select the r tokens with the highest weights in HP as the HF token set, denoted as N WHF and the r tokens with the lowest weights the LF token set, denoted as NLF. Formally, the selection process can be described as:
Åg nh
A HPWN" n'
M= n'=1
NHF argmax
Σ A
SC(1....))) |.|S
KES
NLF argmin SC(L....),,III==
Σ Age
KES
The summation of Equation (7) is applied column-wise (over query dimension) with respect to atten- tion map of each heads A HP(h)) and Aş captures importance of kth token in terms of contribution to high frequency component.
For more precise justification, refer to Appendix C. Empirically, we validate the high- and low- frequency token sets using ViT [8] and ImageNet- validation set [5]. We set r [NT] with = 0.25. Fourier analysis reveals that tokens classified as LF tokens exhibit significantly lower high-frequency components than HF tokens (Figure 2a). While both HF and LF tokens show decline in high-frequency components layer depth increases-consisteen with the rank collapse

---

phenomenon HF tokens consistently retain more high-freen -ree-ffuuency signals than their LF counterparts. This confirms that HF tokens are more effective in preserving high-frequue -egeffreuency features. The first layer has the opposite behavior, since the input to the first layer is an embedding of an image via convolution. Similar to convolution, the attention layer here shows stronger attention to spatially adjacent tokens [18]. Additionally, we measured the similarity of each token to the feature's s DC signal (Figure 2b). The analysis revealed that LF tokens contain significantly more DC signal components than HF tokens, confirming that our method effectively separates tokens based on their frequency characteristics without introducing excessive computational overhead.
To assess the functional importance of these token sets, we evaluated the model's 's accuracy after applying additive White Gaussian Noise (AWGN) to each token set (Figure 2c). The results highlight the critical role of HF tokens in maintaining model accuracy. Since the selection method proposed in Equations (7) to (9) requires only simple averaging, it much more efficient than those methods than rely on Fourier transform or cosine similarity while achieving the similar effect of selecting high frequency components as evidenced by the results in Figure
3.3 Frequency-Aware Token Reduction
Building on our analysis, we propose a token reduction method that retains HF tokens while preserving
the DC components of the remaining tokens through a single DC token. Specifically, we retain the HF token set, FH as defined in Equation (8), However, removing LF tokens (N (NHF)) disrupts the DC signals of the original features (Figure 2b). While these signals are less critical than high-frequency information, they still have a mecainggfu impact on accuracy (Figure 2c). To address this issue, we propose aggregating the DC signals of LF tokens into single DC token. This approach compactly preserves essential DC information, ensuring that the overall feature representation remains balanced. By combining HF token preservation with DC signal aggregation, we can minimize the information loss while maintaining model accuracy and reducing computational costs.
In addition, some LF tokens may still contain high-frequency signals in the early layers, as shown in Figure 1. Thus, treating all LF tokens as purely low- wow-frequency signals may lead to the unintended loss of critical sreeufrreeency information. To address this issue, we leverage spatial locality- characteristic of early layers in vision transformers [18, 20]-by introducing local DC tokens.
To achieve this, we divide the token set into 2 spatially adjacent groups, where the jth local group .i......), n. is a set of token index for the jth group, which has n/w" tokens, i.e. [N' n/w² = H/up- W/ up tokens. The LF tokens in each local group. NY. NLF NNS, are aggregated into local DC token as follows:
denoted as N Each local group, comprises No C (1......
XDC = - {a'no The
1
Σ KENE
Iı,j E {1, [.]...,,w]]]
(10)
INLF
When w = 1, this results in a single global DC token. By employing local DC tokens, we minimize high-frequency signal loss in early layers while still reducing the token count. The reduced token set N, comprising [N HF tokens and 22 DC tokens, is passed to the next layer where |N [nt] with J represents the reducing ratio.
In cases where token reduction is applied across multiple layers (1 - 1,,,, HF token selection process remains unchanged, while the DC tokens in the 1th layer XDC., are updated as follows:
= XDc = =<rbo. -bc.
Σ SEN
IAs1 + Db..-1) E (1...., w w))
(11)
IN +1
This iterative update ensures that the information from LF tokens is retained in the subsequent layers.
We aim to apply the proposed method to pretrained models while mitigating the rank collapse problem. However, self-attention remains prone to collapsing the remaining HF tokens. To address this issue, we modify the attention weight matrix Å follows:
A A (wi + A HP + (wz + Noc
(12)
where W1 22 E ARA are learnable parameters, and A NDC represents the zero-padded attention weights assigned to the DC tokens. The parameter Wi ensures that HF tokens are emphasized to prevent
5

---

collapse, while w/ adjusts their attention weights when DC tokens are present from previous layers. This adjustment accounts for the fact that DC tokens, derived through aggregation, tend to receive lower attention scores due to Jensen's inequality- -ii... the attention weight assigned to the DC token lower than the sum of the attention weight assigned to the individual L.F token. Therefore, we introduce w2 to allow the model to adjust low-frequency information appropriately.
4 Related Works
Token reduction methods aim to improve the efficiency of Transformers. They aim reduce the quadratic complexity of self-attention to token length while minimizing accuracy degradation without modifying the architecture, making them directly applicable to various pretrained models.
A core idea of token reduction can be categorized into two: first, merging- based approaches, such as ToME [1], DTEM [13]. and DiffRate [2] calculate the token similarity and merge them to reduce redundancy. The related approach involves spatial pooling, which also reduces token length but is directly applicable pretrained models since it significantly alters model behavior and learned features. As result, spatial pooling is more commonly employed in designing efficient architectures rather than directly reducing token length in pretrained models [15].
The second category focuses on token pruning, where only the most informative tokens are retained while discarding redundant ones. Dynamic ViT [21] and Ada [16] introduce additional layers to select the informative token with a fixed pruning ratio. More recently, EVIT [14], ATS [10], and Evo-ViT [29] have leveraged the CLS token
selection marker, making their methods efficient with few additional costs. V-V [30] and ATS [10] further improve the previous methods with dynamic pruning. Several approaches have been proposed to mitigate the potential loss information from discarded tokens. ViT, SPVIT, EVIT. and TPS [27] additionally preserve the discarded tokens by utilizing them in the residual connection or fusing them into remaining tokens.
Experiments
In this section, we validate the effectiveness of the proposed method in various models. For discussion and ablation study, we used the widely used configuration as the default configuration, else if otherwise noted. The results of the experiments with the searched Pareto-optimal confi configuration are available in Appendix F. For more experimental results regarding the throughput, additional ablation studies, and application to the downstream tasks, please refer to Appendices G I.
5.1 Experimental Setup
To validate the effectiveness of our proposed method, we conduct experiments on ImageNet-IK [S] and compare our results with state-of-the-ee ahr methods. Specifically. we utilize pretrained models on ImageNet-IK and fine-tune them for 30 epochs. For initial experiments, we apply the common practice of applying reductions at the 4th, 7th, and 10th layers, reducing the token count by 30% at each layer. We evaluate our approach across different model sizes using DeiT' [24]. Additionally, we assess performance across different training strategies, including: DeiT, ViT[22], ViT pre-trained with ImageNet-21K and finetuned with -1K, and ViT trained with self-supervised learning (MAE, DINO) [1]]. For other experimental details, we follow prior works [14, 29, 21J. Specifically, we employ self-distillation, where the original model serves as a teacher model. We set the local window
size = 2, 1, 1 for all experiments except VIT-S, which is used with 2, 1. For quantitative evaluation, we report Top-1 accuracy (%), the number of Multiply-Accumull
operations in giga units (MACs), and the number of parameters in million units. All experiments are conducted using training scripts and pretrained models from timm [28], and MACs are measured using torchprofille ile. Further details the experimental procedures be found in Appendix D.
5.2 Experimental Results
Comparison with SOTA methods Table I, we compare with the recent token-reducing SOTA methods. The proposed method demonstrates better performance compared to existing approaches. In certain cases, it even outperforms the original model, which can be attributed to its ability to mitigate rank collapse effeectively. Although the method introduces additional components, such as

---

Table 1: Comparison of various token reduction methods and the proposed method across models.
For models trained with self supervised models, we fine-tune the baseline model with 30 epochs.
Model
Method
Parums
MACs
Acc. (%)
Model
Method
Params
MACs
Acc. (%)
Baseline
5.6
1.3
72.2
Baseline
22.1
4.6
Dy.
5.9
0.8
71.4
VIT-S
ToMe
22.1
3.0
77.7
DeiT-T
EVO-VIT
5.9
0.8
72.0
EVT
22.1
77.9
Ours
22.1
3.0
79.0
ToMe
5.6
0.8
71.4
EVIT
5.6
0.8
71.9
Baseline
22.1
4.6
81.1
Ours
5.6
0.8
72.3
VIT-S-21K
ToMe
22.1
3.0
80.1
Baseline
22.1
4.6
79.8
EVIT
22.1
Ours
22.1
3.0
81.2
Dy. ViT
22.8
2.9
79.3
EVIT
22.1
3.0
79.5
Baseline
86.6
17.6
83.7
ToMe
22.1
2.9
79.5
VIT---MAE
ToMe
86.6
12.1
82.3
ATS
22.1
2.9
79.7
DiffRate
86.6
12.1
82.9
DeïT-S
METR
22.1
3.0
79.6
Ours
DTEM
22.1
3.0
79.4
Baseline
307
61.6
86.0
Zero-TP
22.1
3.0
79.4
DiffRate
22.1
2.9
79.6
VIT-I-MAEE
ToMe
307
42.3
85,6
Evo-VIT VIT
22.1
3.0
79.4
DiftRate
307
42.3
85.6
Ours
22.1
3.0
79.8
Ours
42.3
85.7
Basclinc
86.6
17.6
81.9
Baseline
86.6
17.6
81.8
VIT-B-DINO
ToMe
86.6
12.1
80.8
Dy. VIT
89.5
11.5
81.4
Ours
86.6
12.1
81.4
DeiT-B
ToMe
86.6
11.5
81.4
EVTT
86.6
11.6
81.4
Baselime
83.3
DiflRate
86.6
11.5
81.5
LV-VIT-S
EVTT
4.7
Ours
86.6
11.6
81.8
Ours
26
4.7
83.3
local DC tokens and learnable parameters for attention weights, these additions are minimal and do not significantly increase the number of parameters computational cost compared to existing methods. the following sections, we analyze the proposed method in detail across various models.
Comparison different pretrained models Rank collapse known to influenced by various training hyperparameters [7, 26]. Therefore, evaluate the proposed method across different pretrained models, including DeiT, ViT, and ViT pretrained on ImageNet-21k and MAE. We visualize the comparison of log amplitude of high-frequency with different models in Figure 3a. DeiT incorporates several techniques, such as drop path and strong data augmentation, which enhance performance and preserve patch diversity more effectively than VIT [3]. Applying the proposed method to these models reveals a more significant performance improvement VIT than in DeiT, VIT is more susceptible to rank collapse, amplifying the method's impact. Similarly, the ViT- 21K model also exhibits better performance gains than DeiT. contrast, for models pretrained via self-supervised methods, we observe varying impacts when applying the proposed method. This can be attributed to reduced rank collapse effects in these methods: MAE explicitly promotes token diversity through reconstruction tasks, and DINO encourages diverse yet consistent representations across augmented views. Nevertheless, our proposed frequency-awar approach minimizes frequency- related information loss compared to existing methods, resulting in relatively smaller performance drops. A detailed analysis of these pretrained models is provided in Appendix E.
Comparison of different model sizes When comparing the performance of the proposed method across different sizes of DeiT, the proposed method improves the performance slightly despite reducing the computational cost. We also observed that smaller models exhibited better performance improvements. The differences between model sizes lie in the number of heads in MSA and channel size. Previous studies have shown that the rank collapsing rate is influenced by the Frobenius norm of the projection matrix in SA and the number of heads, with latter particularly mitigating collapsing rate [26]. Consequently, smaller models, which are more prone to rank collapse, show slightly better performance from the proposed method. We visualize the comparison of log amplitude of high-fregggg frehufrryuenyy with different model sizes in Figure 3b.

---

Th
DaT-8
DxiT-8
à Log smplimade
Log amplinae
CKA Simmarity
Simiarity
3.00
3.00
0.3
0.4
3330
3,30
tokes
Original lokes
0.6
08
Normalzzed depth
Noraalzzdd depth
Normalized Layer Depth
Nomalized Layer Depth
(a) Different pretraining (b) Different model size
(a) Inter-layer
(b) DC signal
Figure 3: Relative log amplitude of high- frequency (1.0r) of models with (a) different training methods (ViT, DeiT, MAE) and (b) dif- ferent model size (tiny, small and base).
Figure 4: (a) Similarity with last layer feature. The gray-dashed line represents the similarity be- tween the models. (b) Similarity with the DC signal feature. In both figures, the gray square represents the reduction layer.
Discussion
6.1 Analysis on the rank collapsing
To assess the effectiveness of method in mitigating rank collapse, analyze feature similarity between the last and intermediate layers. Since rank collapse reduces token diversity, outputs from intermediate layers become increasingly similar to the last layer [34, 26]. As shown in Figure 4a, our method achieves lower similarity compared to DeïT-S, demonstrating reduced rank collapse. Moreover, deeper layers, our approach progressively learns alleviating redundancy seen in DeïT. Additionally, we measured similarity between DC signals from the original features and HF/DC tokens generated by our method (Figure 4b). HF tokens show lower similarity to the overall DC signal, confirming our method effectively selects HF content. Conversely, DC tokens closely resemble the DC signal, effectively preserving low-frequency information. These results confirm that our method mitigates rank collapse while reducing computational cost.
distinct features, effectively
6.2 Ablation study
Effect of high-frequency token To evaluate the impact of high-frequen (HF) and low- frequency (L.F) components on model perfor- mance, we conducted an ablation study where only DC tokens were retained while HF tokens were removed (Figure 5a). The results clearly demonstrate that reducing HF tokens causes more significant performance degradation than reducing DC tokens, underscoring the critical role of high-free geehffreuenccy components in model per- formance. During fine-tuning, we observed that the accuracy of LF token preservation drops to nearly 20%. This suggests that ViTs inherently rely high-frequency signals despite the rank collapsing. In contrast, methods that utilize spa- tial pooling [15, 31J, which rely predominantly
79-
08
HF
LF
layer
layer
layer
(a)
(b)
Figure 5: Results of the ablation study. The gray- dashed line represents the initial accuracy. (a) HF refers to the results of pruming the LF tokens, and LF refers to the results of pruning the HF tokens. (b) Effect the number of the local window. Each number represents the window size. on low-frequency signals, learn distinct features and therefore require training from scratch.
Effect of DC token We examined the importance of preserving DC signals by evaluating model performance without DC tokens (Figure 5b). We experiment with token reduction in a single layer with different local window sizes. The results confirm that DC tokens effectively retain the majority of the original DC signal, and their absence leads to notable performance degradation. Furthermore, we assessed the effect of local DC tokens under varying local window sizes. When reducing tokens in single layer, performance improvements from local DC tokens were more pronounced in earlier layers compared later layers. Considering the increased computational cost as the number of windows increases, set 2 in the earlier layer and in other layers for the default configuration.

---

3.30
-2.00
0. amplitude
Log ampitude
2.41
2.43
Accoray (%g)
2.99
Agrn mantaay
4.15
3.00
2.00
0.6
0.7
105
Ozzi
Edil
3.35
B331
Poeting
08
Pontnn
04
mieU
Strticty
150
Take
Normalized Layer Doych
Normalized Layer Depth
Normalized Layer Depch
Normalized
Depth
(a) 4th layer Reduction
(b) 7th layer Reduction
Comparison with CLS (d) Single-layer pruning
Figure 6: Analysis of the proposed method and the previous methods. (a).()): Relative log amplitude of high-frequency in single-layer token reduction in 4th and 7th layer, respectively. (c): Selection mloU with EViT and similarity of DC signal and CLS token. (d) Accuracy comparison between different methods with the single-layer token reduction in 4th, 7th, and 10th layers. The gray-dashed line represent the initial accuracy of DeiT-S.
6.3 Analysis on previous method
We conducted a detailed analysis of existing token reduction methods and compared them with the proposed approach. Specifically, we compare our method with ToMe [1], which merges similar tokens, and EVIT [14], which identifies tokens with lower CLS attention weights as less important and removes them. For comparison, fine-tuned all models with 30 epochs. To analyze frequency components, we also compare our approach with adding a spatial pooling layer token-reducing layer in DeiT. For analysis, we perform single-layer token reduction at different depths with fixed reduction ratio 50%. For the local window size, we set w for all layers.
When comparing the relative amplitude of high-frequency components (Figures 6a and 6b), the merging-based method exhibits lower high-frequency components than both the token selection- approach and the proposed method. Furthermore, ToMe demonstrates even lower high-frequency amplitude than the model with pooling, which accelerates rank collapse and significantly impacts accuracy (Figure 6d). Applying merging strategies in shallower layers leads to early-layer rank collapse, causing a steeper decline in accuracy as the reduction layer moves to earlier layers.
On the other hand, EVIT exhibits freghffreuency amplitude similar to that of the proposed method. EVIT retains tokens that achieve high attention weights from the CLS token, assuming that these tokens are informative. To further investigate the relationship between the proposed method and EVIT, we measured i) the intersection-over tokens selected by our method, and ii) the similarity of the DC signal with the CL.S token. As shown Figure 6e, the token sets differ in early layers but become increasingly similar at deeper layers. Our analysis reveals that the CLS token gradually incorporates figh-frequeccy signals depth increases. As result, the similarity between the CLS token and the DC signal decreases with depth, leading to greater overlap in token selection. This explains why accuracy differences become less significant when tokens are reduced at deeper layers compared to earlier layers (Figure 6d).
(IoU) between tokens selected by EViT and HF
6.4 Limitation
While our frequency-base based approach successfully reduces computational costs and minimizes perfor- mance degradation, our experiments were limited to vision models. Since rank collapse is inherent all cansformer-basee models, the proposed method could be applied other domains, such as multi-modal models. Future work will focus on exploring adaptations to different modalities.
7 Conclusion
In this paper, we proposed a frequency-awar token reduction method that effectively reduces the computational cost of ViTs while mitigating the rank collapsing and over-smoothing phenomenon. Our approach explicitly selects and retains the HF token set and utilizes the DC token as a compact representation of low-frequency signal, along with learnble attention mechanism. Experimental re- sults demonstrated consistent improvements across various model sizes and pretrained configurations, effectively mitigating the rank-collape while reducing the computational cost.

---

Acknowledgements
This research was partly supported by Artificial intelligence industrial convergence cluster develop- ment project funded by the Ministry of Science and ICT (MSIT, Korea) & Gwangju Metropolitan City, and by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) Next-Generation General Al with Reliability, Ethics, and Adaptability)
Developing the
References
[1] D. Bolya, C.-Y. Fu, X. Dai, Zhang, C. Feichtenhofer, and J. Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.0942 2 2022.
[2] M. Chen, W. Shao, P. Xu, M. Lin, K. Zhang, F. Chao, R. Ji, Y. Qiao, and Luo. Diffrate: Differen- tiable compression rate for efficient vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17164-17174, 2023.
[3] T. Chen, Z. Zhang, Y. Cheng, A. Awadallah, and Z. Wang. The principle of diversity: Training stronger vision transformers calls for reducing all levels of redundancy. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition. pages 12020-12030, 2022.
[4] T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré. Flashattention: Fast and memory-efficien exact attention with io-awareness. Advances neural information processing systems, 35:16344-16359, 2022.
[5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. lece, 2009.
[6] Y. Dong, J.-B. Cordonnier, and A. Loukas. Attention not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pages 2793-2803, PMLR, 2021.
[7] Y. Dong, J.B. Cordonnier, and A. Loukas. Attention not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pages 2793-2803, PMLR, 2021.
[8] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenbomn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1129,, 2020.
[9] Y. Fang, W. Wang, B. Xie, Q. Sun, L. Wu, X. Wang, T. Huang. X. Wang, and Y. Cao. Eva: Exploring the limits of masked visual representation learning scale. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 19358-19369, 2023.
[10] M. Fayyaz, S. A. Koohpayegani, F. R. Jafari, S. Sengupta. H. R. Joze, E. Sommerlade, H. Pirsiavash, and Gall. Adaptive token sampling efficient vision transformers. European Conference on Computer Vision, pages 396-414, Springer, 2022.
[11] K. He, X. Chen, S. Xie, Li, P. Dollár, R. Girshick. Masked autoencoders are scalable vision learners In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 66000-16009 2022.
[12] Z. Kong, P. Dong, X. Ma, X. Meng, M. Sun, W. Niu, X. Shen, G. Yuan, B. Ren, M. Qin, et al. Spvit: Enabling faster vision transformers via soft token pruning. arXiv preprint arXiv:2112. 13890, 2021.
[13] D. H. Lee and S. Hong. Learning to merge tokens via decoupled embedding for efficient vision transformers. arXiv preprint arXiv:2412. 10956,, 2024.
[14] Y. Liang, C. Ge, Z. Tong. Y. Song, J. Wang, and P. Xic. Not all patches are what you need: Expediting vision transformers via token reorganizations. arXiv preprint arXiv:2202.0780 07800, 2022.
[15] S. Mehta and M. Rastegari. Mobilevit: Lightwweigtt general-purpose, and mobile-frienddl vision trans- former. arXiv preprint axX0v:2/10002/77. 2021.
[16] L. Meng, H. Li, B.-C. Chen, S. Lan, Z. Wu, Y.G. Jiang, and S.-N. Lim. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2230-112318 2022.
10

---

[17] L. Noci, S. Anagnostidis. L. Biggio, A. Orvieto, S. P. Singh, and A. Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems, 35:27198-27211, 2022.
[18] N. Park and S. Kim. How do vision transformers work? arXiv preprint arXiv:2202.06700) 2022.
[19] B. Patro and V. Agneeswaran. Scattering vision transformer: Spectral mixing matters. Advances in Neural Information Processing Systems, 36, 2024.
[20] M. Raghu, T. Unterthiner, S. Korbllihh, C. Zhang, and A. Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34:12116-12128, 2021.
[21] Y. Rao, W. Zhao, B. Liu, Lu, Zhou, and C.-J. Hsich. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 11111333-19499 2021.
[22] A. Steiner, A. Kolesnikov, X. Zhai, R. Wightman, J. Uszkoreit, and L. Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106.
2021.
[23] R. Strudel, Garcia, I. Laptev, and C. Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7262-7272. 2021.
[24] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[25] A. Vaswani. Attention all you need. Advances in Neural Information Processing Systems, 2017.
[26] P. Wang, W. Zheng, T. Chen, and Wang. Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory to practice. arXiv preprint arXiv:203.005662, 2022.
[27] S. Wei, T. Ye, Zhang, Y. Tang, and Liang. Joint token pruning and squeezing towards more aggressive compression of vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2092-2101, 2023.
[28] R. Wightman. Pytorch image models. https: htt.a//.../rrigggaaaa/ttttoooooe ieetrn--immmmmmmmmmee 2019.
[29] Y. Xu, Z. Zhang, M. Zhang, Sheng, K. Li, Dong. L. Zhang, C. Xu, and Sun. Evo-vit: Slow-fast token evolution for dynamic vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2964-2972, 2022.
[30] H. Yin, A. Vahdat, J. Alvarez, A. Mallya, J. Kautz, and P. Molchanov. Adavit: Adaptive tokens for efficient vision transformer. arXiv preprint
2021.
[31] W. Yu, M. Luo, P. Zhou, C. Si, Y. Zhou, X. Wang, J. Feng, and S. Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 10819-10829, 2022.
[32] B. Zhang, I. Titov, and R. Sennrich. Improving deep transformer with depth-scaled initialization and merged attention. arXiv preprint arXiv: 1908. 11365, 2019.
[33] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633-641, 2017.
[34] D. Zhou, B. Kang, X. Jin, L. Yang, X. Lian, 7. Jiang, Q. Hou, and J. Feng. Deepvit: Towards deeper vision transformer, arXiv preprint 2rXiv:2103 /888, 2021.
11

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction should clearly state the claims made.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper has limitations and discussed the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
12

---

Justification:
All the theorems, formulas, and proofs in the paper is numbered and cross
referenced.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully disclose all the information needed reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
13

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: This work partially supported by a company, and the code released guidelines for this paper have not yet been discussed. However, the paper fully discloses all the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ce
cublic/guides Seardees/ooeSuuuisseeeee wwt.aiissssiooiiyoyo.oooii for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ stips.cc/publice gwt//uuieeesseee /CodeSubmiiee hww.simmissioooooooioioooooo onliiicyllll iy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specify all the training and test details.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars reported because it would be computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
14

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provide sufficient information on the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : neurrps....pp// oc cc/public/ htlpppulllbthhapeeeeeeeepppaapee
Answer: [Yes].
Justification:
The paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no socictal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
15

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The creators or original owners are properly credited and are the license and terms of use explicitly mentioned and properly respected
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
16

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: the paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
17

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: the core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https http://aaips.cceoooeeeeeeeeeeeee cc//on/eeeeecccc//2/3///// for what should or should not be described.
18

---

A More Background
In this section, we delve deeper into the rank-collapsing phenomenon observed in Transformer archi- tectures and discuss the mechanisms that mitigate this issue. As outlined in Section 2, Transformers can experience rank collapse, where token representations become nearly identical as the network depth increases. This reduction in representation diversity impairs the model's 's ability to capture complex features. Formally, we have following theorem.
Theorem A.1. (Rank Collapse in Pure Attention Networks) Let X E Raxd denote the token embedding matrix at layer f. (with tokens and embedding dimension d) and Hc[X] the high- frequency component of the token feature matrix X. Then, self- attention operation (SAA) reduces this component by:
|H][S((x)]le= & {|Hr([]]rr,
(13)
where the convergence rate < 1 occurs doubly-exponentidd rate in terms of the number of layers & [6, 26].
Dong et al. [6] employed a path-decomposition approach, showing that in pure attention networks, all but one component in the embedding space decay doubly exponentially. The dominant surviving component corresponds to uniform embeddings across tokens, resulting in rank collapse. Com- plementarily, Wang et al. [26] provided frequency-domain analysis, demonstrating that repeated self-attention layers inherently function as low-pass filters, exponentially attenuating high-frequency differences between token embeddings. This further explains the rapid convergence toward uniform embeddings.
In addition, Noci et al. [17] found that rank collapse hinders training by causing the gradients of the queries and keys vanish at initialization, making difficult for the model learn meaningful attention patterns. They suggest that an appropriate depth-dependen scaling of the residual branches can prevent rank collapse and stabilize training. Dong et al. [6] demonstrated that models composed solely of self-attention (SA) layers tend to produce outputs that collapse to a rank-1 I matrix, with all token representations converging to the same vector. This collapse occurs doubly exponentially with depth, severely limiting the model's s expressiveness. Residual connections and feed-forward networks the Transformer architecture that help prevent rank collapse.
(FFNs) are integral components Residual connections preserve the output of previous layers, counteracting the tendency toward rank collapse by maintaining the diversity of token representations across layers. This preservation ensures that the model retains the capacity to learn complex patterns. FFNs contribute by increasing the Lipschitz constant of the transformation applied to token representations, slowing down the convergence rank-1 1 matrix and allowing the model to maintain higher-rank representations greater depths.
For more theoretical analysis, please refer to the following papers [6, 26].
B Proof of Equation (4)
Equation (4) claims that
|| < ||HA|SA(XX]] S [X]I]
(14)
More precisely, previous works on attention collapsing shows that ||H)(A((X]]e < [X]| Equation (4) shows that token pruning and merging may accelerate such collapsing. In below, we provide the detailed proof of Equation (4).
Let X E R be a matrix of token embeddings, and let X' = MX where M E R is a now-normalized matrix which represents either a row-selection matrix (with elements in {0, 1)) row-normalized matrix (with elements in [0, 1) with n' < n. Define H,[X] as the mean-centered (high-frequent component of X. Then, we have:
[H(()((' & [H|[)(()]]IF.
(15)
Proof. Let μ = Ei= X,, and μ = n En' X3. be the mean vectors of X and X', respectively. Then,
Hc[X] X -1p,, Ha[X')= x'- - 1npp..
19

---

and by the definition.
"²'
Σ |x, Mll
|
|X - 1 Σ j=1
Case 1: M is a row-selection matrix.
Then X' is simply a subset of the rows of X. A standard fact from statistics is that the sample variance of any subset points is at most the variance the full set. Concretely, let S S (1,...;7)) be the indices selected by M. Then
# n
X Σ | - pll
iES
and
||x - 1/1 S
I
Σ Xx - 1ll2
Case 2: M is row-normalized.
Here each row of M forms probability distribution over {l..... n}. so each output row X², is convex combination of the original rows:
X
Σ
M₁ Xa.
X
M = 1.
t=1
By the convexity squared Euclidean norm,
|X J1 - -1112
Σ Mxs - 13)l
S >Σ Mx 1111
Summing over j 1...n',
n
Σ
|x - 111 Σ
X - 1/12 S Σ | - -1ll3
where the last inequality again uses that re-centering about the true mean μt minimizes the total squared deviation.
In both cases, the inequality | |a[X]]l | holds. It then follows from Proposition 2.1
that (sA(A(X')III III & [HA(( [F, as claimed.
C Decomposing high- and low-frequency tokens
In this section, we provide a formal justification for decomposing the attention matrix A into low-frequency component ALP and figuefrry component AнP, as defined below:
1
ALP 11
(16)
11
AнP A - ALP.
(17)
Formally, let A E Raxn be an attention matrix derived from sclf-attention (SA), and let be an n-dimensional vector of all ones. Define the matrices:
ALP 1IT Aup = A- ALP-
n
Then, ALP acts as low-pas, filter and Agp acts high-pat pass filter in the token embedding space. We consider token embeddings X E guxd and apply the attention mechanism as:
AX.
(18)
20

---

First, note that ALPX 11 X computes the mean embedding across all tokens and broadcasts this mean to all tokens. Explicitly,
(ALPX)i: 1 E Xj. E mn......).
(19)
This operation corresponds exactly extracting and replicating the Direct-Current (DC) or mean component, thus representing a low-pass filtering operation.
Consequently, the residual term:
Aнr = (A ALP) AX 11
(20)
subtracts the mean component from the attention result, leaving only the differences between tokens (high-frequency variations) and removing the DC component. Thus, by definition, AHP acts high-pass filter.
Formally, the degeppeccompoiition perspective further clarifies this intuition: ALP has single nonzero eigenvalue corresponding to the eigenvector 1, representing the DC component. The orthogonal complement of corresponds to higher-frequency components, preciscly captured by АиР.
Therefore, ALP and Aur indeed represent low-pass and high-pass filtering operations, respectively, as claimed.
D Experiments Details
We conduct the experiments with AdamW with learning rate 0.0001 and weight decay 2 x 10-5, The batch size is set to 128 per GPU. For other hyperparameters and data augmentations, we follow DeiT, except warmup epochs and stochastic depth, which we set O. For EMA, we set the decay factor to 0.9998. Additionally, we utilize the EMA model as distillation. All experiments are conducted in NVIDIA RTX 4090 with mixed precision. We standard cross-entropy and self-distillation for loss term from an unpruned model, following [14, 21, 27J. For DC tokens, we additionally utilize positional embedding terms. For extra distillation from the large teacher model, follow [12]. Specifically, we change the self-distillation term to distillation loss term [24] from the teacher model.
In the experiments, we utilize pretrained VIT and DeiT. The ViT and DeiT share the same architecture with different data augmentations and regularizers. We provide some details of the differences, based on their papers. Note that we change the augmentations when finetuning them following their recipes, except the stochastic depth which we set as following previous works.
Table 2: Hyper-parameters of ViT-S and DeiT-S
Methods
VTT-S
DeiT-S
Optimizer
AdamW
AdamW
learning rate
0.003
0.001
Weight decay
0.03
0.05
Stoch. Depth Repeated Aug
0.1
Rand Augment
2/0
9/0.5
Mixup prob.
0.8
Cutmix prob.
1.0
Erasing prob.
0.25
21

---

E Frequency Analysis of models
As described in Section 5.2, rank collapsing, and over-smoothing phenomenons are affected by model size, training hyperparameters and strategies. Here, we report frequency analysis of different pretraine models and different models. For comparison of training strategies, we conduct the analysis with the base model. For comparsion of model sizes, we conduct the analysis with DeiT. can be seen in the figure below, VIT suffers more from these phenomena than DeïT. On the other hand, MAE shows a lower decrement of high-frequeng components compared to the supervised model (VIT, DeiT).
45
3.50
A. L.Lggaapthis
139
155
115-
199
424
3.4
3.49
-.8
8.5G
Notoellzed
Nounalize
(a) ViT-B
(b) DeiT-B
04
500-
3.00
3.99
2.0
2.3
04
02
C4 Sor
Pragecncy
Nomnainnddgggh
(c) MAE-B
(d) Comparsion
Figure 7: Frequency analysis of different pretrained model. (Left) Frequency analysis along layer depth. (Right) Relative figh-frequeng, component along layer depth.
Log anglinde
Lnganniidde
3.29
-35
3.40
-38
3.99
3.59
-35
3.94
065
3,39
Progercy
Nommalized
Properay
Nomallznd depth
(a) DeiT-T
(b) DeiT-S
Dalt
45.
Date
Dxll
2.00
2.
23
6.4
0F
2.1 Nomodlood
Proparacy
Nomalizo deptt
(c) DeiT-B
(d) Comparsion
Figure 8: Frequency analysis of different model sizes. (Left) Frequency analysis along layer depth. (Right) Relative high-frequency component along layer depth.
22

---

F
Results of the reducing configuration search
The proposed method can adjust the layer and ratio for token reduction. For initial experiments, we apply the common practice of applying reductions the 4th, 7th, and 10th layers, reducing the token count by 30% at each layer. Additionally, we employed Neural Architecture Search (NAS) to identify Pareto-optimal reduction layers and ratios. The NAS optimization
defined
Score (1
MACourrent
ACCeurrent
x
(21)
MACbase
AcCbase
The first term quantifies the relative reduction in computational cost using multiply-accumulan operations (MAC), while the second term evaluates the relative preservation of top-1 accuracy. To narrow the search space, divided the layers into three groups (2nd ~ 4th, 5th 7th and 8th 11th) and searched for one reduction layer in each group with reducing ratio ((0.1, .............. and local window size ((1,2, 4}) to maximize the score.
We report the result of searched configurations for each DeiT model in the below table. We found that larger reduction ratios in deeper layers Here, we report the detailed results of one or two optimized configurations for each DeiT model. The results with "-o" is the results of the pareto-optimal configurations The reducing ratio represents the token-reducing percentage of the remaining lokens. Our analysis reveals that employing larger reduction ratios in deeper layers leads to more Pareto- optimal outcomes. Notably, eliminating greater number of low-frequency (L.) tokens these depths incurs minimal information loss. This is attributed to the increased occurrence of rank collapse in deeper layers, which diminishes the presence of high- fighffreueeccy (HF) tokens. Consequently, the higher incidence of rank collapse at these depths reduces the number HF tokens, making the removal of additional LF tokens less detrimental. Interestingly, by expanding the size of the local window (i.e., increasing [NDC]). we observed that even shallower layers could achieve optimal configurations with higher reduction ratios. This underscores the effectiveness of the proposed local DC token mechanism in preserving essential information while facilitating aggressive token reduction. Furthermore, our approach reduces the computational of DeiTT- level comparable with DeîT-T (ours-o2) while maintaining better accuracy.
Table
Results the searched Pareto-optimal reducing configuration.
Model
Method
Reducing layer
Reducing ratio
W
GFLOPs
Acc. (%)
Baseline
N/A
N/A
72.2
Ours
[4, 7, 10]
[30%, 30%, 30%]
[2, 1, 1]
72.3
DeiT-T
Ours-ol
[4, 6, 91
[50%, 40%, 20%)
[2, 2, 2]
1.0
72.7
Ours-o2
[4, 7.
[50%, 40%, 40%]
2,
0.6
70.2
Baseline
N/A
N/A 130%, 30%, 30%
N/A
4.6
79.8
Ours
7,
12, I,
3.0
79.9
DeïT-S
Ours-ol
[4, 6, 101
[40%, 50%, 90%
[2, 1, 11
2.6
79.6
Ours-o2
[2. 8, 10]
[70%, 60%, 60%]
[4, 2, 1]
1.5
73.5
Baseline
N/A
N/A [30%, 30%, 30%]
N/A
17.6
81.8
DeiT-B
Ours
[4, 7, 10]
[2, I,
11.6
81.8
Ours-ol
7, 11]
[60%, 50%, 50%]
(2,
8.7
80.2
G
Application to Dense Prediction
To assess the generality of our frequency- token reduction beyond image classification, we integrate it into semantic segmentation pipeline. We choose the Segmenter framework [23] with an ADE20K [33]-pretrained ViT-B (EVA) encoder [9]. Our modifications occur during the Transformer encoding stage:
Token Pruning: In each self-attention layer, we apply our frequency- aware pruning operator, removing subset of tokens deemed low-value for high-frequency detail.
Low-Frequency Aggregation: All pruned tokens' information is aggregated into a single token via averaging (the low-frequency component X)PX)).
23

---

Residual Injection for Decoding: To restore full spatial resolution for dense prediction, we inject the final token back into the decoder input as residual. This ensures that low-frequency context flows into the upsampling and classification heads.
Table 4:
Semantic segmentation results on ADE20K using Segmenter. Pruning configurations mirror common settings used in classification experiments.
Model
Throughput (img/s)
mloU (%)
Baseline (no pruning)
21.7
51.3
Config 1: 3% pruning on all layers
68.0
51.1
Config 2: 30% pruning layers 4,7,10
92.1
50.8
These results show that our method can be effectively extended dense prediction. By aggregating redundant tokens into single low-frequency token, we remove unnecessary computation while pre- serving high-frequency details critical for segmentation. Even under 30% pruning selective layers, the mloU degrades by only 0.5 points, while throughput increases by over 4x. This demonstrates the practical utility of frequency-aware token reduction in segmentation and other downstream tasks without significant performance loss.
H
Ablation on Attention Weight Modification
We conducted ablation study the DeiT-S model to examine the effects of attention weight modification. Our results indicate that the accuracy experienced a slight improvement from 79.8% (without reweighting (w)) to 79.9% (with reweighting w1). The primary objective of our proposed method is to mitigate the rank collapsing phenomenon by selectively preserving high-frequency tokens. The attention reweighting through w1 provides additional support in alleviating this issue. Additionally, we observed that incorporating wa effectively reduced the number of epochs required for fine-tuning, with minimal impact on the final accuracy.
I End-to-End Throughput
Due to inconsistent code releases, some baseline methods report only GFLOPs; here we uniformly benchmark all methods under FP32 NVIDIA RTX 4090. The fastest result in each group is bolded, and the recond-fastess underlined.
Table 5: End-to-end throughput and accuracy for various token-reduction methods on ViT-S and ViT-B architectures (FP32 on RTX 4090).
Method
imgs/s
Accuracy (%)
Method
imgs/s
Accuracy (%)
DeiT-S
2803
79.8
DeiT-B
918
81.8
EViT
3742
79.5
EVIT
1338
81,4
ToMe
3555
79.5
ToMe
1223
81.4
DiffRate
3538
79.6
DifiRate
1236
81.5
Ours
3659
79.9
Ours
1310
81.8
(a) ViT-S
(b) VIT-B
Our method consistently matches exceeds both speed and accuracy of competing approaches, underscoring its practical deployability in real-world inference scenarios.
Compatibility with FlashAttention [4]
Our frequency awwree
aware attention modification integrates seamlessly with FlashAttention, requiring only lightweight post-procssiing step without altering the core attention kernel. The proposed method rescales the attention map as follows:
Å
ALP (wi + AhP + (w2 1) ANDC-
(22)
24

---

In practice after FlashAttention calculates the product AV, one needs only to calculate the mean value of each token set NLF and Ne Formally, we can compute the attention as follows:
A = (w1 + 1) A - W1 ALP (w1 W2) ANDO.
(23)
n Vi (w) W2) Σ
ÁV (wi + 1)AV W1 Ev.
V
(24)
iENDc
This design preserves FlashAttention's low-memory, single-pass pass advantages and adds only two vectorized scaling operations.
For selecting the tokens Nur and NLF, FlashAttention did not store the entire attention map, so we can directly apply our methods. However, we only need the argmax in the column-wis value of the attention weight, not the exact value of the attention weight. Therefore, we only have to store the mean attention score Å. Specifically, we store the block-mean attention score S = Σ. Q.KT
Alternatively, we can explicitly compute the Sj = Ex Q out of the FlashAttention, for better generability with the existing frameworks. In either case, we can easily integrate the proposed methods with FlashAttention computational overhead. the table below, we report the throughput of the proposed methods with FlashAttention and explicit computations. Since FlashAttention requires FP16, we changed the configurations of our experiments in the table below.
any other modern attention-optimizze techniques with minimal
Table 6: Throughput (images/s) on an RTX 4090, FP16 with FlashAttention and our frequency-aware post-processing.
Model
Configuration
Throughput
DeiT-S
w/ FlashAttention
9303
Ours
w/ AltenAttention
12759
The results show that the proposed method is compatible with the modern attention-optiimized methods.
K
Visualization of HF/LF tokens
Following your valuable suggestion, we performed an experiment to visualize the selected High- Frequency (HF) and Low-Frequency (LF) tokens on sample images. While cannot attach figures directly to this rebuttal, we will describe our key findings here and commit to adding these visualizations in the main manuscript. Our analysis confirmed that in early to mid-layers, the token selection aligns well with human intuition: HF tokens predominantly correspond to patches with object boundaries and complex textures, while LF tokens are clustered in smooth, uniform background areas. Interestingly, we also observed 'outliers' where some background patches were selected as HF tokens, which found often correlates with high-norm features in the embedding space. In deeper layers, we observed that where tokens corresponding to the salient foreground of an object began to be classified as I.F tokens. Since rank collapse becomes more severe, the unique signals of even the most important tokens in human perception, get 'smoothed out' and converge toward the low-frequency, DC-like representation of the entire feature map.
L Ablation on Different Image Size
To evaluate method's effectiveness across different token lengths, conducted ablation study using various image and patch sizes. We utilized the ViT-2lk-fmetuned an-lk model, which offers pre-trained weights for all tested configurations. The results, presented in Table 7, demonstrate that the benefits of our approach are most prominent with longer token sequences.
For instance, with 16/384 patch/image size (576 tokens), our method improves accuracy from 86.6% 87.0% while reducing computational cost by 35% (from 55.5 to 35.9 GFLOPS). This trend even more pronounced for the 8/224 configuration (784 tokens), where observe 0.7% accuracy gain
25

---

with 36% reduction in GFLOPS (from 78.3 to 50.0). The parenthesized values in the table represent GFLOPS.
Table 7: Performance comparison of ViT-B and VIT-S models with different patch and image sizes. Values are presented as accuracy (9) and a GFLOPS in parentheses.
Patch Size / Image Size
32/224
32/384
16/224
16/384
8/224
Image Token Length
49
144
196
576
784
ViT-B (original)
80.7 (4.4)
83.4 (13.1)
84.5 (17.6)
86.6 (55.5)
85.8 (78.3)
ViT-B (pruned)
77.9 (3.2)
84.0 (8.8)
84.6 (12.1)
87.0 (35.9)
86.5 (50.0)
Difference
-2.8 (-1.2)
0.63 (-4.3)
0.14 (5.5)
0.4 (-19.6)
0.7 (-28.3)
ViT-$ (original)
76.0 (1.1)
80.5 (3.4)
81.1 (4.6)
83.8 (15.5)
ViT-S (pruned)
73.5 (0.8)
80.5 (2.3)
81.4 (3.0)
84.0 (9.9)
Difference
-2.5 (-0.3)
0.0 (-1.1)
0.30 (-1.6)
0.16 (-5.5)
26


Reviews:
[{"review_id": "aAvWRxyPU5", "reviewer": "Reviewer_PN7n", "summary": "The manuscripts describes a novel token reduction method based on prior research on overshooting and rank collapse of self-attention / transformer models, and additional analysis of these phenomena as token reduction tends to exasperate these conditions. In order to avoid this a frequency-aware token reduction method is proposed, where the input tokens are partitioned into two subsets: a low frequency (LF) set of tokens, which are similar to the DC Bias (mean token value), and high frequency (HF) set consisting of tokens which are dissimilar to the DC bias. Furthermore, chunked attention is initially used in order to account for spatial effects, as the LF tokens are found to contain HF signals in early layers.\n\nThe methods is trained using a teacher-student distillation setup and compared across several pretrained backbones (ViT, DeiT, MAE, DINO) on ImageNet-1K. The proposed method is found to consistently matching and outperforming prior methods. Lastly, analysis and ablation studies of the effect and behavior of the LF and HF tokens are conducted, and prior token reduction methods are analyzed in a frequency perspective.", "questions": "1. Why are the results on dense prediction tasks (section G in the supplementary results) not described more in the main paper? The results, as with the other sections, are interesting and currently the reader may not fully recognize this when reading the main manuscript.\n\n2. Does the reduction rate r have to be the same for both LF and HF subsets? Have you conducted any investigation into this, and whether it could be made dynamic such that all relevant tokens could be used?\n\n3. What is the effect of the distillation based training vs standard fine-tuning? Clarifying this would make the results more certain, as currently the effect of the distillation setup and the frequency-aware token reduction is entangled.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed method is in essence very simple, as it simply consists of disentangling the LF and HF parts of the attention maps. This makes the method easy to adopt and understand.\n\n2. The method is founded on theoretical and empirical results related to the rank collapse and oversmoothing of self-attention layers. This again leads to the method to be easily understood, and also serves as a tool for interpreting prior methods as done in Section 6.3.\n\n3. As shown in the experiment section, the proposed method matches or in some cases significantly outperforms prior token reduction methods. This is impressive, considering the large suite of methods compared against.\n\n\nWeaknesses:\n\n1. It is unclear why a student-teacher distillation setup is used. It is not motivated by any part of the manuscript and the addition leads to uncertainty of whether the results are due to the proposed frequency-aware method or the distillation training setup.", "was_revised": true, "final_justification": "Having read the author's responses to my own and the other reviews I have found all of my concerns to be answered. i believe the paper is of high interest to the Token Reduction field and provide interesting theoretical and empirical results. I am therefore raising my recommendation to Accept.", "rebuttal": "We sincerely thank the reviewer for their thorough feedback and insightful comments. We are encouraged that they found our method to be simple, theoretically sound, and experimentally effective. Below, we address each point raised and believe these clarifications will help strengthen the final manuscript.\n\n- **Effects of self-distillation**\n\nWe adopted self-distillation to follow the standard protocol for a fair comparison. The main prior works we compare against, such as EViT ,Evo-ViT , and DynamicViT, also report their performance using this same self-distillation method. This approach is standard practice in the field.\n\nTo  address the concern, we ran an additional experiment without any distillation. The results confirm its impact is minimal: the performance DeiT-S w/ distillation (79.8%) and w/o distillation (79.7%). The similar behavior can be observed in the other methods, for instance, DynamicViT w/ distillation (79.3%) and w/o distillation (79.2%).\n\nThis ablation study demonstrates that the performance gains stem from our proposed frequency-aware method, not the distillation setup. We will add this analysis to the final paper and thank you for helping us strengthen our work.\n\n- **Reduction rates of LF and HF tokens.**\n\nWe use a fixed reduction rate, to precisely control the computational budget (MACs). This is a standard practice in the field that ensures a fair comparison with prior work. Our method then preserves the top-r high-frequency (HF) tokens.\n\nOur paper's primary contribution is the novel frequency-aware selection criterion, which is orthogonal to the scheduling method (fixed vs. dynamic) . While we are aware of dynamic methods like A-VIT and ATS , our focus was on first validating this new criterion. We agree that a dynamic approach is a promising direction. However, dynamic methods introduce implementation challenges, particularly for batched training and inference, as token lengths can vary per sample. We are actively exploring how to efficiently integrate our criterion with a dynamic scheduler, and if the experiments yield meaningful results in time, we will include them in the final manuscript. This investigation builds upon our initial exploration of optimal reduction schedules via NAS (Appendix F) .\n\n- **Dense Prediction tasks**\n\nWe thank the reviewer for this excellent suggestion. The semantic segmentation results were moved to Appendix G primarily due to the main paper's strict page limits. We prioritized a detailed analysis of our core methodology on the primary benchmark of image classification.\n\nHowever, we agree that these results are an important demonstration of our method's generality. In the revised manuscript, we will add a summary of the dense prediction results to the main paper.\n\n- **Paper Formatting**\n\nThank you for your review and for catching these details. We appreciate you helping us improve the clarity of our paper. We will correct all the formatting issues you pointed out in the final version."}, {"review_id": "thjPEP09nG", "reviewer": "Reviewer_pKDi", "summary": "This paper proposed a token reduction method based on frequency analysis. The low-pass filter by a mean filter is applied to attention map. Then, the high-frequency attention map is obtained by subtraction between the original attention map and low-pass filtered map. Then, reduced tokens are selected based on the highest weights in the HF attention map. To maintain the important information from low-frequency tokens, DC signals of LF tokens are aggregated into a single DC token. The experiments are conducted for pre-trained models trained on ImageNet-1K.", "questions": "1. Is it enough to use the mean filtering to obtain the low-pass filtered output of attention maps?\n2. Why did you compare your results with somewhat less recent methods?\n3. Are there any results applied to datasets other than ImageNet?\n4. Is there any reason to limit 30 epochs in the experiments?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The idea of using frequency information when selecting tokens is effective.\n2. The computational overhead is not severe to select high frequency tokens.\n3. Better than previous methods for various models of DeiT, ViT.\n\nWeaknesses\n1. As in Eq. (5), the mean filtering of attention maps is not guaranteed to produce effective low-pass filtered output. Thus, it is not easy to have successful high-pass filtered output. We cannot expect that this simple filtered outputs help to select informative tokens. Similarly, in the process of aggregating DC signals of LF tokens, it is not clear that we can have DC signals by repeatedly aggregating the LF tokens from the mean filtering. \n2. Most of previous methods in Table 1 are published 2 or 3 years ago. Therefore, it is not easy to evaluate that the proposed method outperform the previous state of the art methods. For ViTs, the previous methods are ToMe and EViT, which are published in 2022. There are more recent methods for token reduction, PYRA(ECCV'24), DyT(Neurips'24), OFB (CVPR'24), etc. \n3. It is difficult to determine whether this method can be applied generally because it is only applied to one dataset, ImageNet. It would be better to apply the proposed method to other vision problems, such as VTAB-1K or CIFAR10, Cityscape. \n4. Usually, we need to train at least 100 epochs in previous token reduction methods. 30 epochs may not be enough to compare the performance. There is no clear explanation why 30 epochs are used for the experiments.", "was_revised": true, "final_justification": "I understand why you did not compare it to other 'merging' or 'pruning' techniques. Furthermore, additional experiments alleviated my previous concerns. I will raise the score to 4.", "rebuttal": "We thank the reviewer for the feedback. Below, we aim to address you concerns regarding our methodology and experimental validation to further clarify the contributions of our work.\n\n- **On Experimental Setup, Baselines, and SOTA Comparisons**\n\nWe would like to address the concerns regarding the suggested recent works, datasets, and training epochs.\n\nFirst, we thank you for suggesting the recent papers. However, we would like to clarify that the mentioned works—PYRA and DyT focusing on 'Efficient Task Adaptation,' and OFB on 'Model Compression'—belong to research fields that are orthogonal to our work on 'token reduction for pre-trained models'. These methods aim to improve the efficiency of 'parameter-efficient fine-tuning' or 'compression' via adaptor or network sparsity search, whereas our work concentrates on dynamically reducing the number of input tokens during 'inference'. In fact, the DyT paper itself suggests this complementary relationship by stating, \"Then, we explore the compatibility of our method with token pruning methods,\" indicating that our fields are not direct comparable methods.\n\nAs these research fields differ, so do their standard evaluation protocols. As the reviewer noted, papers in those fields often evaluate on the VTAB-1K dataset with 100 training epochs. However, the standard for our subfield—as followed by recetn SOTA methods (e.g., DTEM('24), Zero-Tp('24)) in Table 1 —is 30-epoch fine-tuning on ImageNet-1K. We adhered to this protocol to ensure a fair and direct comparison. Furthermore, we chose DeiT as our primary comparison model among recent SOTA methods, because it is the standard model used in recent token reduction literature. Therefore, we compared the proposed method with recent SOTA methods, in DeiT. The inclusion of the ViT model was not reported in the most of the previous works, but was a necessary choice to validate our core hypothesis and proposed method; that the degree of rank collapse differs depending on the training recipe, thus requiring verification under different conditions. In addition, the practical challenge of some recent works not having publicly available code also factored into our decision. \n\nIn conclusion, our reason for selecting ToMe / DynamicViT, EViT as primary baselines is that they represent the two foundational pillars of token reduction: 'merging' and 'pruning'. While most subsequent works are incremental improvements on their ideas, our research introduces a fundamental difference by approaching the problem from a new aspect based on 'frequency'. Our goal was to extensively validate the efficacy of this novel methodology across various models, and we respectfully submit that our experimental design was therefore reasonable and well-justified. We hope this explanation helps to address the reviewer's concerns. \n\nNevertheless, to further address the reviewer's concern and further demonstrate the effectiveness of our method, we have conducted new experiments with DyT and AdaptFormer. To ensure fairness, we replicated the experimental environment from Table 5 of the DyT paper (using AdaptFormer on VTAB-1K). Since the DyT paper lacks a precise description of its token pruning setup for that table, we followed the original ToMe experiment settings by pruning 13 tokens in each block. Values measured by us are marked in **bold**; otherwise, they are from Table 5 of the DyT paper. VTAB Accuracy refers to the grouped-mean accuracy, and the parameter count denotes the number of trainable parameters in the backbone.\n\n| Method | VTAB-1K Accuracy ↑ | Params. (M) ↓ | Throughput (img/s) (V100) | Throughput (img/s) (A100) |\n| --- | --- | --- | --- | --- |\n| AdaptFormer | 74.75 | 0.16 | 767.3 | **3296** |\n| DynamicViT+AdaptFormer[12] | 75.48 | 3.1 | 954.82 |  |\n| EViT+AdaptFormer[12] | 74.63 | 0.16 | 1152.38 |  |\n| ToMe + AdaptFormer | 74.3 | 0.16 | 941.7 |  |\n| Ours + AdaptFormer | **74.71** | **0.16** |  | **4752** |\n|  |  |  |  |  |\n| DyT r= 0.5 | 77.14 | 0.16 | 912.39 | **3478** |\n| DyT r= 0.5 + ToMe [5] | 76.6 | 0.16 | 1114.7 |  |\n| DyT r= 0.5 + Ours | **76.77** | **0.16** |  | **4859** |\n\nThe result clearly shows that our frequency-aware approach is a effective solution that surpasses existing token reduction methods, even when integrated with the latest SOTA adaptation frameworks. \nThese experiments on VTAB-1K, together with our results on ImageNet-1K and the ADE20K segmentation task (in supplementary materials), clearly demonstrate the robustness and effectiveness of the proposed method across diverse datasets and tasks.\n\n- **On the Validity of Mean Filtering**\n\nOur use of mean filtering ($A^{LP}$) to isolate low-frequency components is a direct application of a fundamental signal processing principle: a signal's mean is its DC (zero-frequency) component, a quintessential low-pass operation (see Appendix C for a formal proof ). In addition, the mean filter is widely used for a lowpass frequency filter, effectively removing high-frequency content from an image, or token, in our papers [1,2].\n\nWhile our selection method is computationally simple, its effectiveness is empirically validated in Figure 2, which confirms that our selected high-frequency (HF) tokens are indeed richer in high-frequency content and more critical to model accuracy. Furthermore, its effectiveness is validated by the high fidelity of our aggregated DC token to the original DC signal (Figure 4) and the notable performance drop upon its removal (Figure 5). We believe this strong theoretical and empirical evidence demonstrates the soundness of our approach.\n\n[1] Stanford University. (n.d.). CS279 Notes.\n\n[2] Fisher, R., Perkins, S., Walker, A., & Wolfart, E. (n.d.). Hypermedia Image Processing Reference (HIPR2). University of Edinburgh."}, {"review_id": "ekvh83sVg6", "reviewer": "Reviewer_MPrk", "summary": "To address the issue of quadratic computational complexity concerning token length in vision transformer, this paper proposed a frequency-aware token reduction strategy that improves computational efficiency while preserving performance by mitigating rank collapsing. Specifically, the proposed method partitions tokens into high-frequency (HF) tokens and low-frequency tokens (LF), which are selectively preserved and aggregated into a compact DC token to retain essential low-frequency components, respectively.", "questions": "Ablation on different image sizes: The paper mainly focuses on image-based token reduction, which may prove more effective when the sequence length is longer. Could the authors include ablation studies under different image sizes (e.g. 336$\\times$336 or larger)?", "limitations": "See `Weaknesses` and `Questions` part.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Well-motivated: The article presents a detailed discussion and visualization of the rank collapse or over-smoothing phenomena in self-attention and different token diversity of merging-based methods and reduction-based methods.\n2. Clear writing: This paper offers a clear and logical structure that is easy to follow.\n\nWeaknesses:\n1. Limited performance improvement and downstream experiments: In `Tab.1` and `Tab.4` of the appendix, the authors compare their method with other approaches on classification and segmentation, respectively. However, the performance gain in `Tab.1` appears significant only on `ViT-S` and `ViT-S-21K`, while the improvements for other settings are relatively small. Furthermore, beyond the `baseline` comparison on the segmentation task, comparing with more closely related methods on additional vision downstream tasks (such as object detection) would better demonstrate the effectiveness of the proposed approach.\n2. Visualization of HF/LF tokens: An important component of the proposed method is the division of tokens into high-frequency and low-frequency groups. Could the authors provide visualizations based on some images to offer a more intuitive interpretability analysis?\n\n\nI'd be more than willing to revise my score based on the author's further responses.", "was_revised": true, "final_justification": "The authors conducted addition experiments about generalization on the downstream task and ablation on different image sizes, and made a further explanation on HF/LF tokens, which addressed most of my previous concerns.", "rebuttal": "We sincerely thank the reviewer for the thorough and insightful feedback. Your suggestions have been invaluable in helping us strengthen our work. We address each of your points below with new experimental results and analyses.\n\n- **On Performance Improvement and Downstream Task Generalization**\n\nWhile the accuracy improvement is most visible on the ViT-S and ViT-S-21K models, we would like to clarify the goal of our method from a broader perspective, which demonstrates its effectiveness across all experiment settings.\n\nThe primary objective of any token reduction method is to significantly reduce computational cost while minimizing performance degradation. From this viewpoint, the results for models like DeiT represent a major success. For instance, on DeiT-B, our method matches the baseline accuracy of 81.8% exactly, while reducing the computational cost from 17.6 to 11.6 MACs. This result is the ideal outcome, demonstrating that our core objective—maximizing efficiency without sacrificing performance—was successfully achieved. We consider this a significant contribution to the field of efficient models.\n\nThe notable performance increase on ViT-S and ViT-S-21K can be seen as a significant benefit, highlighting a unique strength of our approach. As our paper analyzes, standard ViTs are prone to over-smoothing and rank collapse, where models lose token diversity.\n\nFollowing your valuable suggestion, we conducted additional experiments on object detection. To ensure a fair comparison, we adopted the experimental settings from SViT (Revisiting token pruning for object detection and instance segmentation) [1] and applied the same pruning configuration used in our main classification experiments (removing 30% of tokens at the 4th, 7th, and 10th layers). These experiments were performed on an A100 GPU. In the results table, values in bold represent our own measurements, while others are taken from the SViT paper for reference.\n\nThe results show our method achieves a highly competitive performance-efficiency trade-off. Our method's APbox of 45.0 (-0.8 drop) is significantly better than other pruning-based methods like EViT (-1.3). When compared with SViT, which shows better performance preservation (-0.3 drop), it is important to consider the fundamental architectural differences that lead to this specific trade-off.\n\nSViT recycles information from pruned tokens in all subsequent layers, which helps preserve accuracy but introduces overhead. In contrast, our method is designed to be more lightweight by recycling this information only once at the final layer of the ViT-Adapter block. This key design choice is why our method achieves a notably higher throughput, running over 7% faster than SViT (21.0 vs 19.6 FPS), making it a compelling alternative for latency-sensitive applications.\n\nFurthermore, this trade-off is highly task-dependent. It is worth noting that on ImageNet classification with DeiT-S, our method outperforms SViT on both accuracy and efficiency. Our method achieves 79.8% accuracy (-0.0% drop), while SViT scores 79.4% (-0.4% drop).\n\nWe hypothesize that adopting a more complex, SViT-style recycling mechanism could further improve our detection performance. While we were unable to complete this experiment due to time constraints, we aim to include this important analysis in the main manuscript of our final paper.\n\n| Method | APbox | APmask | FPS (Reported) | FPS (Measured) |\n| --- | --- | --- | --- | --- |\n| DeiT  | 45.8 | 40.9 | 18.45 | **18.02** |\n| SViT | 45.5 (-0.3) | 40.7 (-0.2) | 22.32 | **19.6** |\n| Ours | **44.9 (-0.9)** | **39.9 (-1.0)** | - | **21** |\n| EViT  | 44.5 (-1.3) | 39.8 (-1.1) | 22.76 |  | - |\n| EvoViT  | 44.8 (-1.0) | 39.9 (-1.0) | 22.12 | **19.3** |\n| DyViT + prsv | 44.1 (-1.7) | 39.3 (-1.6) | 22.95 | - |\n\n[1] Liu, Yifei, et al. \"Revisiting token pruning for object detection and instance segmentation.\" *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision*. 2024.\n\n- **Visualization of HF/LF tokens**\n\nFollowing your valuable suggestion, we performed an experiment to visualize the selected High-Frequency (HF) and Low-Frequency (LF) tokens on sample images. While we cannot attach figures directly to this rebuttal, we will describe our key findings here and commit to adding these visualizations in the main manuscript. Our analysis confirmed that in early to mid-layers, the token selection aligns well with human intuition: HF tokens predominantly correspond to patches with object boundaries and complex textures, while LF tokens are clustered in smooth, uniform background areas. Interestingly, we also observed 'outliers' where some background patches were selected as HF tokens, which we found often correlates with high-norm features in the embedding space. In deeper layers, we observed that where tokens corresponding to the salient foreground of an object began to be classified as LF tokens. Since rank collapse becomes more severe, the unique signals of even the most important tokens in human perception, get 'smoothed out' and converge toward the low-frequency, DC-like representation of the entire feature map. \n\n- **Ablation on Different Image Size**\n\nThank you for this excellent suggestion. Following your advice, we have conducted an extensive ablation study on various image and patch sizes to analyze our method's effectiveness on different token lengths. For these experiments, we used the ViT-21k-finetuned-on-1k model, as it provides pre-trained weights across the various configurations. The results, summarized in the table below, strongly support our hypothesis and demonstrate the remarkable scalability of our approach. (GFLOPS are shown in parentheses).\n\n| patch size / image size | 32/224 | 32/384 | 16 / 224 | 16 / 384 | 8/224 |\n| --- | --- | --- | --- | --- | --- |\n| image token length  | 49 | 144 | 196 | 576 | 784 |\n| ViT-B (original) | 80.7 (4.4) | 83.4 (13.1) | 84.5 (17.6) | 86.6 (55.5) | 85.8 (78.3) |\n| ViT-B (pruned) | 77.9 (3.2) | 84.0 (8.8) | 84.6 (12.1) | 87.0 (35.9) | 86.5 (50.0) |\n| Difference | -2.8 (-1.2) | 0.63 (-4.3) | 0.14 (5.5) | 0.4 (-19.6) | 0.7 (-28.3) |\n|  |  |  |  |  |  |\n| ViT-S (original) | 76.0 (1.1) | 80.5 (3.4) | 81.1 (4.6) | 83.8 (15.5) |  |\n| ViT-S (pruned) | 73.5 (0.8) | 80.5 (2.3) | 81.4 (3.0) | 84.0 (9.9) |  |\n| Difference | -2.5 (-0.3) | 0.0 (-1.1) | 0.30 (-1.6) | 0.16 (-5.5) |  |\n\nAs you hypothesized, the benefits of our method are most prominent with longer token sequences. The results clearly show that our approach not only improves efficiency but can also enhance performance, especially when dealing with high token counts.\n\nFor the high-resolution case (16/384, 576 tokens), our method reduces the computational cost by a substantial 35% (from 55.5 to 35.9 GFLOPS) while also improving the accuracy from 86.6% to 87.0%.This trend is even more pronounced with the longest sequence tested (8/224, 784 tokens), where we achieve a +0.7% accuracy gain while cutting GFLOPS by 36% (from 78.3 to 50.0 GFLOPS).\n\nWe thank you for pushing us in this valuable direction. These new experiments clearly demonstrate the effectiveness and scalability of our method. We will add this full ablation study to the final version of our paper."}, {"review_id": "0Ov68Jgv5I", "reviewer": "Reviewer_BU7E", "summary": "This work examines the frequency behavior of self‑attention in Vision Transformers (ViTs), building on prior observations that self‑attention layers act like low‑pass filters—attenuating high‑frequency details and yielding progressively smoother outputs across layers. Noting that maintaining a balance between high‑ and low‑frequency information can boost performance, the authors investigate this balance in the context of token reduction. They quantify the relative importance of high‑frequency tokens versus low‑frequency (DC) tokens and introduce a streamlined method for distinguishing them, using the identity matrix as an efficient proxy for the DC component. By selectively pruning low‑frequency tokens while preserving high‑frequency ones, their frequency‑aware token reduction strategy outperforms conventional methods, both theoretically and empirically, underscoring the value of frequency‑based analysis for more effective token selection in transformer models.", "questions": "- *Token Selection with Windowing*: It remains unclear whether the high‑/low‑frequency distinction is applied globally before window partitioning or independently within each local window. Please specify the exact procedure—ideally with step‑by‑step pseudo‑code—to ensure reproducibility.\n- *Role of DC Subtraction in Token Ranking*: In Section 3.2 (Lines 134–137), the authors use an argmax over each column of the high‑frequency attention map. Since argmax depends only on relative ordering, subtracting the low‑pass (all‑ones) component should not change the ranking. What, then, is the concrete benefit—computational or otherwise—of this subtraction? Please clarify or provide empirical evidence.\n- *Precise Definition of “Relative Magnitude”*: In Section 3.2 (Line 140), the authors cite “relative magnitude” within columns of the high‑frequency attention map. Is this computed via column‑wise sums, means, norms, or another statistic? Please give a formal mathematical definition.\n- *Visualization of Token Reduction*: It is strongly recommend adding qualitative visualizations—e.g., heatmaps showing which patches are deemed high‑ versus low‑frequency, before and after reduction. Such figures greatly enhance interpretability.\n- *Computational Cost Analysis*: It is suggested to include a concrete analysis or empirical benchmark demonstrating the computational savings of your simplified frequency‑based token selection versus the naïve method (i.e., explicitly computing the DC component for every token).", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n- The work introduces a frequency‑aware token reduction strategy for ViTs, achieving state‑of‑the‑art results on multiple benchmarks.\n- It offers a solid theoretical foundation in spectral analysis, linking the rank‑collapse phenomenon in transformers to the loss of high‑frequency components.\n- The proposed method is computationally efficient and seamlessly integrates with existing ViT architectures, making it highly attractive for real‑world deployment.\n\n**Weaknesses**\n- **Clarity**: Some implementation details remain ambiguous—particularly the exact procedure for separating high‑ and low‑frequency tokens and the sequencing of global versus local operations in the token‑reduction pipeline. Including a more explicit algorithmic description or pseudo‑code would greatly enhance readability.\n- **Visualization**: The work omits qualitative visualizations—such as token‑selection heatmaps or spatial maps before and after reduction—that are crucial for interpretability.\n- **Typo**: There are some minor formulaic inaccuracies (e.g., $w^w$ should be $n/w²$), and some phrasing could be clearer regarding the frequency decomposition implementation.", "was_revised": true, "final_justification": "The response addresses most of my concerns. I am maintaining the current rating.", "rebuttal": "We sincerely thank the reviewer for their insightful summary and positive assessment of our work. We are grateful that you recognized our method's solid theoretical foundation, state-of-the-art results, and practical efficiency. Your valuable suggestions regarding clarity, visualization, and computational analysis are well-taken, and we will address each point to further strengthen the manuscript.\n\n- Clarity and Token Selection with Windowing\n\nWe thank the reviewer for this suggestion. To clarify the procedure, our method first performs a global identification of High-Frequency (HF) and Low-Frequency (LF) tokens based on the full attention map, as described in Section 3.2 . Subsequently, the identified global LF token set is partitioned into spatial windows, and tokens within each windowed subset are aggregated into a *local* DC token . HF tokens are always preserved globally.\n\nTo improve clarity and ensure reproducibility, we will add a dedicated algorithm block with pseudo-code in the Appendix of the revised paper.\n\n- **Visualization**\n\nFollowing your valuable suggestion, we performed an experiment to visualize the selected High-Frequency (HF) and Low-Frequency (LF) tokens on sample images. While we cannot attach figures directly to this rebuttal, we will describe our key findings here and commit to adding these visualizations in the main manuscript. Our analysis confirmed that in early to mid-layers, the token selection aligns well with human intuition: HF tokens predominantly correspond to patches with object boundaries and complex textures, while LF tokens are clustered in smooth, uniform background areas. Interestingly, we also observed 'outliers' where some background patches were selected as HF tokens, which we found often correlates with high-norm features in the embedding space. In deeper layers, we observed that where tokens corresponding to the salient foreground of an object began to be classified as LF tokens. Since rank collapse becomes more severe, the unique signals of even the most important tokens in human perception, get 'smoothed out' and converge toward the low-frequency, DC-like representation of the entire feature map. \n\n- **Role of DC Subtraction in Token Ranking**\n\nYou are correct that the argmax ranking is mathematically unchanged by the subtraction. Our reasoning for this formulation is to maintain a clear and consistent theoretical framework throughout our method. This initial decomposition is crucial for consistency with our subsequent attention modification step in Equation (12), where we use a learnable parameter ω1 to amplify high-frequency signals . We will clarify this in the revised manuscript.\n\n- **Precise Definition of “Relative Magnitude”**\n\nThe term “relative magnitude (weight)” in Eq.7 is calculated as the column-wise sum of the high-pass attention map corresponding to a particular token, which is then averaged across all attention heads. We will revise the text in Section 3.2 to make this connection explicit and avoid any confusion.\n\n- **Computational Cost Analysis**\n\nThank you for this valuable suggestion. Following your recommendation, we have performed a more concrete computational cost analysis.\n\nA naïve method that involves calculating the DC signal and then measuring feature similarity between all tokens can have a complexity of up to $O(nd+nd^\n2\n)$. In contrast, our proposed method, which simply post-processes the already computed attention map, requires only an additional $O(n^\n2\n)$  complexity per head.\n\nFor a typical model like DeiT-S with n=197 and d=384, the cost of our $O(n^\n2\n)$  operation is significantly lower than that of the naïve approach. Even with large n, for instance, down stream task such as object detection, cost is still lower since $n << d^2$. This analysis confirms that our frequency-aware selection is not only highly effective, as demonstrated by our experiments, but also computationally very efficient. We will add this detailed cost analysis to the revised manuscript.\n\n- **Typo**\n\nThank you for your careful reading and for pointing out the inaccuracies. We will correct all identified typos and unclear phrasing in the final version of the paper."}]

Author rebuttal:

Meta-review: The paper proposes a strategy to alleviate the high (quadratic) computational cost of vision transformers by performing token reduction.
Importantly, the token reduction is done in a frequency-aware way to avoid the common side effects: rank collapsing and over-smoothing: The tokens are divided into two categories: High-frequency tokens and low-frequency tokens (corresponding to direct current (DC)/bias/zero frequency), which are then handled differently.
Through extensive experiments, the paper shows that the proposed method outperforms existing baselines.

During rebuttal and discussion, the authors provided additional details about the procedure and experiments, which greatly helped convince the reviewers. 
Overall, the reviewers generally liked the paper for its simplicity, theoretical grounding, and extensive experiments that demonstrated practical benfits.
For the camera-ready, the paper should incorporate the clarifications and improvements discussed during review.
Decision: Accept (poster)