A Simple Romance Between Multi-Exit Vision Transformer and Token Reduction
Authors: Dongyang Liu
Meina Kan
Shiguang Shan
Xilin CHEN
Venue: ICLR 2024 poster (2024)
Keywords: Vision Transformer
Token Reduction
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Vision Transformers (ViTs) are now flourishing in the computer vision area. Despite the remarkable success, ViTs suffer from high computational costs, which greatly hinder their practical usage. Token reduction, which identifies and discards unimportant tokens during forward propagation, has then been proposed to make ViTs more efficient. For token reduction methodologies, a scoring metric is essential to distinguish between important and unimportant tokens. The attention score from the $\mathrm{[CLS]}$ token, which takes the responsibility to aggregate useful information and form the final output, has been established by prior works as an advantageous choice. Nevertheless, whereas the task pressure is applied at the end of the whole model, token reduction generally starts from very early blocks. Given the long distance in between, in the early blocks, $\mathrm{[CLS]}$ token lacks the impetus to gather task-relevant information, causing somewhat arbitrary attention allocation. This phenomenon, in turn, degrades the reliability of token scoring and substantially compromises the effectiveness of token reduction. Inspired by advances in the domain of dynamic neural networks, in this paper, we introduce Multi-Exit Token Reduction (METR), a simple romance between multi-exit architecture and token reduction—two areas previously considered orthogonal. By injecting early task pressure via multi-exit loss, the $\mathrm{[CLS]}$ token is spurred to collect task-related information in even early blocks, thus bolstering the credibility of $\mathrm{[CLS]}$ attention as a token-scoring metric. Additionally, we employ self-distillation to further refine the quality of early supervision. Extensive experiments substantiate both the existence and effectiveness of the newfound chemistry. Comparative assessments also indicate that METR outperforms state-of-the-art token reduction methods on standard benchmarks, especially under aggressive reduction ratios.

Published as a conference paper at ICLR 2024
A SIMPLE ROMANCE BETWEEN MULTI-EXIT VISION TRANSFORMER AND TOKEN REDUCTION
Dongyang Liu !.2 Meina Kan'?, Shiguang Shan 12,33 Xilin CHEN
1 Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS
2 University of Chinese Academy of Sciences Peng Cheng Laboratory iudongyang21s, kanmeina, sgshan, xlchen) sict..t... ac.
ABSTRACT
Vision Transformers (ViTs) are now flourishing in the computer vision area. De- spite the remarkable success, VITs suffer from high computational costs, which greatly hinder their practical usage. Token reduction, which identifies and dis- cards unimportant tokens during forward propagation, has then been proposed make ViTs more efficient. For token reduction methodologies, a scoring metric is essential to distinguish between important and unimportant tokens. The attention score from the [CLS] token, which takes the responsibility to aggregate useful information and form the final output, has been established by prior works as an advantageous choice. Nevertheless, whereas the task pressure is applied at the end of the whole model, token reduction generally starts from very early blocks. Given the long distance in between, in the early blocks, [CLS] token lacks the impetus gather task-relevant information, causing somewhat arbitrary attention allocation. This phenomenon, in turn, degrades the reliability of token scoring and substan- tially compromises the effectiveness token reduction. Inspired by advances in the domain of dynamic neural networks, in this paper, we introduce Multi-Exit Token Reduction (METR), a simple romance between multi-exit architecture and token reduction- areas previously considered orthogonal. By injecting early task pressure via multi-exit loss, the [CLS] token is spurred to collect task-related information in even early blocks, thus bolstering the credibility of [CLS) attention as token-scoring metric. Additionally, we employ leff-dssillaation to further re- fine the quality of early supervision. Extensive experiments substantiate both the existence and effectiveness of the newfound chemistry. Comparative assessments also indicate that METR outperforms state-of-the-art token reduction methods on standard benchmarks, especially under aggressive reduction ratios.
INTRODUCTION
The transformer architecture was first introduced in (Vaswani et al., 2017) for natural language processing. Due to advantages like flexible input format and outstanding scaling performance (Zhai al., 2022; Kaplan al., 2020), Transformer has now become universal architecture widely adopted by diverse deep learning areas (Carion et al., 2020; Radford et al., 2021; Alayrac et al., 2022; Radford al., 2023). In computer vision, starting from the monumental work Dosovitskiy et al. (2020), transformers have also achieved state-of tfte--of-the-art performance on variety of tasks.
Despite the remarkable success, vision transformers suffer from high computational cost, which greatly hinders their practical usage, especially under essource-constraing circumstances. In pur- suit of more efficient ViTs, some works revisit the traditional model compression techniques like distillation (Yang et al., 2022; Ren et al., 2023; Liu et al., 2022), pruning (Yang et al., 2021), quan- tization (Lin et al., 2021), etc, and some others turn to efficient architecture design, bringing up innovations like pyramid architecture (Wang et al., 2021) and sliding window attention (Liu et al., 2021). Among these VIT compression efforts, token reduction, which enjoys special affinity with the characteristics of the transformer architecture and the sparse nature of visual information, has emerged as a promising approach. Due to the quadratic complexity of self-attention operation w.nt. token number, reducing the number of tokens can greatly alleviate the computation burden of VITs.

---

Published as a conference paper at ICLR 2024
Block 4
Block 7
Block 10
Block 4
Block 7
Block 10
(a)
(b)
Figure I: Visualization of [CLS] attention. We use DeiT-S and investigate the 4-th, 7-th, and 10-th blocks. Token with top 50% [CLS) attention scores are kept, while others are masked. (a) Official DeiT-S model (b) Further fine-tune the official model for 30 epochs with multi-exit loss.
The methodology design of token reduction can be roughly decomposed into three sub- tobbproblems: ① How to score the importance of each token, so that the least important ones can be pruned? How is token removal implemented exactly (directly discard, merge into reserved ones, etc.)? ① How to allocate the number of reduced tokens in each block?
Among these points, the first one plays a pivotal role, After a period of exploration from the re- search community, the attention score from the [CLS] token has been proved to be the superior choice (Liang et al., 2021; Chen et al., 2023b; Xu et al., 2022; Haurum et al., 2023). Since the [CLS] token bears the responsibility for forming the final result, it has to gather important task-relevant information from the patch tokens. Therefore, how much the [CL.S] token attends to a patch token naturally becomes reasonable proxy for the importance of the patch token.
Nevertheless, whereas the task pressure is applied at the end of the whole network, the token re- duction operation needs commence from very shallow blocks to achieve significant acceleration. Therefore, while in the last few blocks, the task pressure imposes the [CLS) token a strong impetus to pay more attention to the important informative tokens, it is doubtful if, within the blocks far before, the [CLS] token still has the abundant motivation to prioritize the most important tokens. For example, the [CL.) token may otherwise defer the summarization of important information to the last few blocks, leaving the allocation of attention score in early blocks somewhat arbitrary. We empirically validate that this is indeed the fact: as shown in Fig. ((a), tokens with top 50% [CLS] attention scores in block 10 are mostly those containing informative cues such as the head part of the dog, but in block 4, many of these informative tokens are assigned with low attention scores (within the bottom 50%). For normal neural networks, this is okay. However, for token reduction methods, such a phenomenon significantly degrades the reliability of token scoring and substantially compromises the effectiveness of token reduction methods.
2

---

Published as a conference paper at ICLR 2024
Is it possible to make [CLS] attention in early blocks indicative of token importance as in the last block? In this paper, we reveal that the multi-exit architecture, a technique typically adopted in the domain of dynamic neural network (Han et al., 2021b). is good solution to achieve this goal. Both multi-exit architecture and vision transformer token reduction are established topics and have been investigated independently in their respective domains. However, we find that there exists some un- explored special chemistry between these two topics. We thus propose Multi-Exit Token Reduction (METR), an extraordinarily simple yet highly effective and efficient combination. With the multi- exit architecture, pressure from the target task permeates through to the [CLS] token in even earlier transformer blocks, foreing collate useful information from patch tokens immediately. The at- tention score thus makes better reflection of token significance, ensuring accurate token reduction. Additionally, we adopt self-distillation to further refine the quality of early supervision. Note that while our method aims to improve the sub-problem @ of token reduction, we empirically show that METR is compatible with recent progress targeting the other two sub-problems (23).
Overall, our main contributions are summarized as below:
1. We identify the inconsistency between [CLS] attention and token significance in early ViT block, which greatly harms the performance of token reduction methods.
2. We propose METR, simple romance between multi-exit vision transformer and token reduction, to alleviate the aforementioned inconsistency, and further employ self-distillation to improve the quality of early supervision.
3. Extensive experiments substantiate both the existence and effectiveness of the newfound chesm- istry between mutli-exit and token reduction. Comparative assessments also indicate that METR outperforms oftt-of-thhe--- token reduction methods, especially under high reduction ratio.
RELATED WORK
2.1 VISION TRANSFOR MER COMPRESSION
In the realm of computer vision, convolutional neural networks (CNNs) once held sway (Simonyan & Zisserman, 2014; He et al., 2016; Sandler et al., 2018); however, the advent of the Vision Trans- former (ViT) family (Dosovitskiy 2021a) has heralded a new era. The model's unparalleled flexibility, versatility, and performance have driven an increasing number of researchers employ vision transformers for diverse range of computer vision tasks (Carion et al., 2020; Radford et al., 2021; Li et al., 2022).
al., 2020; Touvron et al., 2021; Yuan al., 2021; Han al.,
However, like other deep neural networks, vision transformer suffers from the conundrum between model performance and model complexity. Model compression techniques are then investigated to fight for a better balance. Among the explorations, some works revisit the architecture-agnostic approaches like distillation (Yang et al., 2022; Ren et al., 2023), pruning (Yang et al., 2021), quan- tization (Lin et al., 2021), and neural architecture search (Chen et al., 2021), making them readily applicable to the ViT family. Conversely, the distinctive attributes of the transformer architecture per se provide unique opportunities for model compression, among which token reduction has emerged as particularly promising avenue for research.
2.2 TOKEN REDUCTION
The technique of token reduction relies on two favorable properties of the vision transformer. Firstly, transformers can deal with mutable sequence length and patch-level sparsity leads to real accelera- tion without the need of special hardware/ / alloorithm support (Note that this advantage has also been exploited by works like MAE (He et al., 2022)). Secondly, the information in the image is unevenly and sparsely distributed among patches, making token reduction theoretically feasible.
The core problem of token reduction is how to score the significance of each token so that we can de- cide which to discard and which to retain. Prior works can be roughly divided into the parametric and the non con-aaamettric genres. Parametric scoring strategies introduce specialized policy networks to evaluate the significance of each token. For example, Dynamic ViT (Rao et al., 2021), SPVIT (Kong et al., 202 1), and IA-RED² (Pan et al., 2021) 1) insert additional token importance prediction modules between transformer blocks, AVIT (Yin et al., 2022) implicitly merges the policy network into the
3

---

Published as a conference paper at ICLR 2024
original model, making one feature dimension specially responsible for representing token impor- tance. AdaViT (Meng et al., 2022) generalizes the aforementioned idea, incorporating token-level, head-level, and block-level sparsity into unified framework. Since discarding ken is inherently a discrete problem, re-parameterization tricks (Jang et al., 2016) 6) or reinforcement learning (Sutton & Barto, 2018) are indispensable for optimization. However, the former hampers training-timi time acceleration because tokens can only be masked rather than really discarded, while the latter incurs complicated training curriculum and unstable performance.
retaining a to-
On the other hand, non-parametric methods rely on off-the-shelf indicators to reflect token impor- tance. Since there are no parameters to update within the scoring function, training- -ime acceleration is generally accessible. The attention score from the [CLS] token or its slight variance is the pre- dominant choice of this genre (Liang et al., 2021; Yin et al., 2022). Prior works have validated its superiority (Haurum et al., 2023), and multiple works have based themselves on this metric, making advances from other aspects for creating new SOTA (Chen et al., 2023b; Fayyaz et al., 2022; Xu et al., 2022; Chen et al., 2023a). However, the reliability of [CLS] attention itself and ways for en- hancing it, which is the focus of this paper, has been rarely investigated. In this paper, we point out the inconsistency between [CLS] attention and real token significance, and then propose method- ology to fix it. Meanwhile, the efforts made in this paper are orthogonal to recent advances achieved in other aspects of the token reduction problem and can be seamlessly combined together.
2.3 MULTI-EXIT NETWORK
Adding early-exit heads to neural networks is long-existing topic in deep learning for various purposes. As early as the onset of the current deep learning wave, multi-exit architectures were employed to tackle the vanishing gradient problem in deep neural networks (Wang et al., 2015). At present, one typical purpose to make neural network dynamic (Teerapittayanon et al., 2016; Bolukbasi et al., 2017: Han et al., 2021b), so that for easy samples, whose early results are already accurate, the remaining blocks can be skipped for higher efficiency. With the boom of transformer in the vision community, Bakhtiarnia et al. (2021) also applied the idea of multi-exit dynamic network to vision transformer. Different from existing works, in this paper, we open up a novel utility of the multi-exit structure, revealing its special chemistry with transformer token reduction.
3 METHOD
In this section, we first briefly review the architecture of vision transformer, focusing on components related our method. We then introduce how to inject early task pressure [CLS] tokens in early transformer blocks, making [CLS] attention a more accurate token importance indicator. Finally, we show how use self-distillation to costlessly enhance the final performance.
3.1 VISION TRANSFORMER OVERVIEW
Given an input image, the ViT architecture first divides it into patches and then projects the patches into token embeddings. An extra [CLS] token is then added to the beginning of the token sequence, forming Xo, the input to the stack of transformer blocks:
X Blockz Block ... Block2 ((oock(Xx)),
Xo = z0) z.. ..."" 2."TT
where Block means a transformer block, L is the number of blocks, n is the number of image patches, and superscript means the [CLS] token. X E IR(" 1,44 comprises of the token features Idd for one [CLS] token and n patch tokens, and each æ contains elements. The transformer block typically consists of Norm layer, Multi-Head Self-Attention (MHSA) layer, and Feed-Forward Network (FFN) layer. Considering the i-th block (Block;). ), the computation graph therein is:
X X, MHSA (NOMM()
X = X + FFN(NORM(X x..ttt))).
(2)
In the MHSA layer, every token serves as query to attend to other tokens, gathering information by absorbing value vectors with weights defined by query-key product. Considering the [CLS] token as query, the corresponding [CLS] attention A E R. is computed as follows:
A = Softmax (geK) C
(3)
4

---

Published as a conference paper at ICLR 2024
Early Task Pressure
Ed
[CLS]
Information Flow
Multi-llea Lead Self- Self-AAteenn Attentin
Foed Forward Network
Token Sorting
Su
Patch Tokens
Early Head
Early Result
L
MHSA
FFX
MHSA
Relline
FFN
MHSA
Redhice
FFN
MHSA
Label
Lytmal
An overview METR. In the attention layer, information from patch tokens flows into the [CLS] token (purple arrow), with the [CLS] token's attention score, A defining the contributing ratio. In each transformer block where token reduction is applied. A separates the input tokens into the important set S and the unimportant set SV: token reduction then applied following the principle of retaining S while discarding S To ensure that A accurately reflects the importance of each token for the task at hand, we introduce the self-distillation loss Lad for injecting early task pressure, which spurs the (CLS) token to turn to informative tokens for forming the correct answer.
Figure 2:
where d' is the dimension of attention head, E Rd' is the query vector of the (CT.S) token, and Ke [n denotes the key matrix. With slight abuse of notation, here we make K consist of only patch tokens (i.e. [CLS] excluded).
Finally, at the end of the whole vision transformer, æf, the feature in X corresponding to the [CLS] token, is picked out and fed to the final classification head for task result.
O finot (x)
(4)
L Linnat = CE(O finat, y),
(5)
where yimal is the final classification head, CE is cross-entropy loss, and y is ground-trui truth label.
3.2 MULTI-EXIT TOKEN REDUCTION
For given block, suppose there n input tokens and we reduce the number m, we first propagate through the MHSA layer following Eq. 1, obtaining the intermediate feature Xatta and the [CLS] attention A (we omit the block index for notation convenience), and then use A as the metric to divide the n input tokens to m important tokens and (n m) unimportant tokens.
(6) St and St denote the set of important and unimportant tokens, respectively. p indexes patch token, SP is its corresponding attention score in A and x is the corresponding token feature in Xatin- The top(.,m m. operator selects the largest m elements from (.). The action of token reduction then follows the general principle retain the tokens in S while discarding those in S
S = (=att Xattn la- top(A5, m)). S Xattn S'.
The quality of token reduction is hence dominated by the consistency between [CLS] attention (At) and the real significance of each token. However, the extent of consistency, shown in Fig. 1(a),

---

Published as a conference paper at ICLR 2024
Table 1: Off-the-shelf token reduction performance on ImageNet. While the models have not un- dergone any token-reduction aware training, they are tested directly with token reduction applied. Throughput evaluated on one RTX 4090 with batch size 512.
Keep Ratio
1.0
0.7
0.6
0.5
0.4
0.3
GFLOPs
4.6
3.0
2.6
2.3
2.0
1.8
Throughput (imgs/s)
2360
3582
4158
4712
5256
5864
DeïT-S
79.80
78.52
76.89
74.10
68.67
57.88
DeïT-S (Multi-Exit)
79.78
78.76
77.55
75.12
70.76
61.19
A
-0.02
+0.24
+0.66
+1.02
+2.09
+3.31
in fact unsatisfactory. Specifically, while within the deeper blocks, most the important tokens are given relatively high [CLS] attention score, the consistency degrades significantly when the block goes shallower. We attribute this phenomenon to the lack of task pressure in the early blocks: given the refined patch features in the last several blocks, it may be enough for the [CLS] token to collect task-relevant information majorly in these blocks, leaving [CLS) attention in shallow blocks relatively arbitrary. For token-reduction transformers, this becomes fatal drawback.
In order to calibrate [CL.) attention, we need to make the pressure penetrate through to the early transformer blocks, that the [CLS] token always bears the urgent impetus collect as much task- relevant information and it could. Specifically, we make the transformer network multi-exit, adding an early-exit head H; within every block where token reduction employed. As shown in Fig. 2, the early-exit head takes o aattn the [CLS] token in X Xi-attn: as input, comprising of two-layer bottleneck MLP, outputting an intermediate result O2:
O, =Hi(zi
early task pressure is then applied by the multi-exit loss Lmnt
Cime 1 Σ IEB, CE(O,.y).
[BA]
where B, denotes the set of blocks where token reduction is involved, and |B, is its cardinality. In this way, in every block where token reduction is employed, the [CLS] token is compelled to provide result given existing patch features immediately. The [CLS] attention thus becomes better reflection of token significance, as shown in Fig. 1(b).
3.3 DELF-DISTILLATIO
While the multi-exit loss in Eq. 8 can already effectively improve the reliability of early [CLS) attention, the high-quality final answer, final, offers an opportunity for yet another free lunch. To further facilitate the efficiency and effectiveness of METR, we propose the self-distillatio loss Ced to enhance the quality of supervision for early-exit heads.
Lad
Σ iEB.
KL (O,, detach [O (Ofimat),
where detach[-] denotes the stop gradient operation. Here for simplicity, we use the KL-Divergence based KD loss (Hinton al., 2015) for distillation, but its advanced variants (Zhao et al., 2022) may also be applicable. Finally, the total loss for training fine-tuning a token-reduction transformer with METR becomes:
Liotal
Lifinal + oxLed-
(10)
Note that after training, all of the early-exit heads H removed. Therefore, no extra inference cost is introduced by METR.
4 EXPERIMENT
In this section, we first validate the existence of the claimed special chemistry between multi-es architecture and token reduction in Sec. 4.1. We then delve into the design choices of METR and

---

Published as a conference paper at ICLR 2024
Table 2: Performance ImageNet after 30 epochs of token-reduction fine-tuning.
Keep Ratio
0.7
0.6
0.5
0.4
0.3
GFLOPs
3.0
2.6
2.3
2.0
1.8
DeiT-S
79.32
79.06
78.47
77.68
76.34
DeiT-S (Multi-exit)
79.49
79.22
78.72
77.91
76.72
A
+0.17
+0.16
+0.25
+0.23
+0.38
analyze its performance in different scenarios in Sec, 4.2. Finally, we compare METR with existing token reduction methods to prove its effectiveness in Sec. 4.3. Experiemnts are conducted on the ImageNet (Deng et al., 2009) dataset using DeiT (Touvron et al., 2021) and MAE (He et al., 2022) models. Unless otherwise specified, we base METR on EVIT (Liang et al., 2021), reduce the number of tokens according to the specified keep ratio (e.g. 30% of the tokens are reduced when keep ratio set to 0.7) in the 4-th, 7-th, and 10-th blocks, and fuse the unimportant tokens into additional token. Similarly, for other experiments where we base METR on DiffRate (Chen et al., 2023b), we inherit their allocation of token number and their exact token reduction action. See appendix for detailed experiment settings.
4.1
EXSITENCE OF THE CHESMISTRY
4.1.1
OFF-THE-SHELF TOKEN REDUCTION ON MULTI-EXIT MODEL.
We first fine-tune the officially pre-trained DeïT-S model with Liotat in Eq. 10 for 30 epochs, without incorporating any token reduction operations. Subsequently, we directly add the token reduction operation to the model at inference time. Since the model has not undergone any token-reduction aware training, this forms an off-the-shelf evaluation setting. Based on this setting, we compare the performance of the DeiT-S model before and after muti-exit finetuning in Tab 1.
After 30 epochs of multi- exit tuning, the normal accuracy (i.e. without token reduction) of the model is almost unchanged (79.80%
79.78%). However, the model's robustness token reduction improves discernibly, manifesting an average increase of 1 1.46%, and this margin of improvement amplifies with the application of more aggressive token reduction strategies. This empirical evidence lends credence to our argument that the incorporation of early task pressures augments the alignment between the [CLS] attention score and the token significance.
4.1.2 VISUALIZATION
To elucidate how the model's robustness token reduction has been augmented through 30 epochs of multi-exit fine-tuning, we present visualizations of [CLS) attention maps, revealing insights into the mechanism at work. As depicted Fig ((a), for the original DeiT-S model, the (CLS) atten- tion pattern possesses relatively high correlation with the informativeness of individual image patches in later blocks, such as block 10. However, this correlation substantially diminishes in ear- lier blocks, for example, block 4. Conversely, after 30 epochs of multi-exit tuning (Fig. 1(b)), the congruence between attention patterns and token significance notably strengthened, particularly in the shallower blocks. Consequently, the [CLS] attention in the fine-tuned model serves more reliable metric for token scoring, resulting in less accuracy drop after token reduction.
4.1.3 FURTHER FINE-TUNE MULTI-EXIT MODEL WITH TOKEN REDUCTION
Token-reduction models typically require an additional fine-tuning phase with token reduction ex- plicitly incorporated to fully realize their potential. To ascertain whether the multi-exit model ob- tained in Sec. 1.1.1 has, in a broader context, become more amenable to token reduction, we compare with the original DeiT-S model within this fine-tuning setting. This time, both models undergo an additional 30 epochs of fine-tuning where token reduction is explicitly employed. Early exit losses are excluded from this phase to maintain a fair comparison between the two models. As shown in Tab. 2, while multi-exit loss longer exists in this stage of fine-tuning its influence lasts and leads to better performance. The stable advantage of the multi-exit model supports the special chemistry between multi-exit architecture and token reduction.

---

Published as a conference paper at ICLR 2024
Table 3: Ablation study the design choices of METR. We start from pre-trained DeïT-S model
and fine-tune for 30 or 100 epochs. EE means early exit and SD means self-distillation. Top-1 accuracy (%) on ImageNet validation set is reported.
Keep Ratio
Epochs
Setting
0.7
0.6
0.5
0.4
0.3
EVIT
79.32
79.06
78.47
77.68
76.34
@ EE
79.51
79.28
78.82
78.08
76.79
30
A
0.19
0.22
0.35
0.40
0,45
EE & SD A
79.64
79,45
78.94
78.23
76.90
0.32
0.39
0.47
0.55
0.56
EViT
79.58
79.23
78.85
78.17
77.13
100
EE & SD
80.17
79.86
79.62
78.96
78.02
A
0.59
0.66
0.77
0.79
0.89
4.2 ABLATION STUDY
In Sec. 4.1, split up the multi-exit and the token-reduction fine-tuning process for clearer con- clusion. In practice. however, such a split design is inefficient and unnecessary. For all remaining experiments in this paper, merge the above two fine-tuning stages into single stage. Based on this ground, in this section we ablate over some of the design choices.
4.2.1 ONE-STAGE TOKEN-REDUCTIO FINE-TUNING WITH MULTI-EXIT LOSS
As shown in Tab. 3 (index ① and @), by introducing early task pressure into the token-reduction fine-tuning process, performance considerably enhanced with no additional inference cost and negligible additional training cost (for early heads) imposed. Again, the advantage of @ becomes increasingly large as the pruning becomes more and more aggressive. This intuitive: to prevent significant information loss under aggressive pruning, stringent token scoring metric is necessary, so that the most informative tokens can be retained.
4.2.2 SELF-DISTILLATION
Contrary to lard-labeled-basee Lme- we instead use the self-distillation loss Lad for early task pres- sure. Such substitution incurs additional computation cost, and we empirically find that there's even no need to modify the hyper-paramete a. After the replacement. As shown in Tab. (index @), METR outperforms EVIT by 0.46% average in this setting.
4.2.3 LONGER FINE-TUNING
Since 30 epochs may not be enough fully optimize the multi-exit task, further investigate the performance of METR with longer (100 epochs) fine-tuning, See @ and ( in Tab. 3 for results. As shown in the table, the advantage of METR is magnified with the application of longer fine-tuning recipe, and the average improvement reaches 0.74%.
4.2.4 NUMBER OF MULTI-EXIT HEADS
Considering that methods like DifiRate (Chen et al., 2023b) conduct token reduction in 10 out of trans- former blocks, is it optimal to allocate 10 early heads, one for each them? Interestingly, the answer yes. As shown in Tab 4, on the one hand, an exit head not only influences the block it's in, but also influences the preceding blocks, more early heads help. On the other hand, such influence weaker than directly adding early exit heads in these blocks per se.
Table 4: Effect of carly head number on METR performance. Model: ViT-B with GFLOPs 8.7. We base METR DiffRate.
Early Heads
DeiT
MAE
All
81.11
82,42
Every 2 Every
80.94
82.34
80.81
82.29
8

---

Published as a conference paper at ICLR 2024
Table 5: Comparison with token-reduction methods ImageNet (Deng et al., 2009). We report our reproduced accuracy for EVIT and DiffRate, and the author-reported accuracy for others. means fusing token reduction into model training (i.c. training from scratch for 300 epochs for DeiT, and training from self-supervised checkpoint for 100 epochs for MAE), means starting from supervisely trained checkpoints and further fine-tune for 100 epochs, and for those without special notation we start from supervised checkpoints and fine-tune for 30 epochs.
Model
Method
GFLOPs
Acc.
Model
Method
GFLOPs
Acc.
Baseline
4.6
79.82
Baseline
17.6
81.83
A-ViTi
3,6
78.60
EViT!
11.5
81.30
DynamicVit
2.9
79.32
DynamicViT
11.2
81.30
Evo-Vit
3.0
79.40
VīT-B
ToMe.
11.5
81.41
ToMe
2.9
79.49
(DeiT)
DiffRate
11.5
81.76
EVIT
3.0
79.50
METR+DifiRate
11.5
82.05
METR+EVIT
3.0
79.76
EVITi
8.7
80.00
EVIT
3.0
79.32
DiffRate
8.7
80.51
METR EVIT
3.0
79.64
METR+DiffRate
8.7
81.11
VIT-S
METR + EVITT
3.0
80.09
Baseline
17.6
83.72
(DeiT)
DiffRate
2.9
79.72
ToMe:
11.5
82.94
METR DiffRate
2.9
79.83
DiffRate
11.5
83.25
ViT-B
DynamicViT
2.5
78.50
METR+DifiRate
11.5
83.29
(MAE)
EVIT
2.3
78.50
ToMe
8.7
81.91
METR+EVIT
2.3
79.06
DiffRate
8.7
81.97
EVIT
2.3
78.47
METR+DifRRate
8.7
82.42
METR + EVIT
2.3
78.94
Baseline
61.6
85.95
METR EVITI
2.3
79.62
VIT-L (MAE)
ToMe
31.0
85.05
DiftRate
2.3
79.22
DiffRate
31.0
85.15
METR DiffRate
2.3
79.46
METR+DiHRRte
31.0
85.32
4.3 COMPARE WITH EXISTING METHODS
In this section, compare with state-of-the-an methods to demonstrate the effectiveness of METR. We select EVTT (Liang et al., 2021) 1) and DiftRate (Chen et al., 2023b) as the base method, and build METR on top of them. Note that for these base methods, Additionally, Dynamic ViT (Rao et al., 2021), A-VIT (Yin et al., 2022), Evo-ViT (Xu et al., 2022), and ToMe (Bolya et al., 2022) are compared. For holistic evaluation, consider both the from- scratch training setting and the fine-tuning setting. The results are shown in Tab 5.
report our re-produced results.
Again, we observe stable improvement brought by METR. Specifically, the improvement robust to different base methods, different models, and different schedules. Such enhancement convincingly proves the existence and the effectiveness of the special chemistry between multi-exit architecture and token reduction. Note that since METR investigates transformer token reduction from an aspect different from most existing works on this topic, it can boost state-of-the-art to a new level. For example, even though the recent DiffRate (Chen al., 2023b) has already surpassed other meth- ods by a large margin, METR can still enhance in a stable and considerable manner. On the other hand, there is still a clear tendency that more aggressive token pruning benefits more from METR. For example, considering the ViT-B (DeiT) model, METR improves DiffRate by 0.29% in the GFL.Ps=11.5 setting, and the margin increases to 0.6% with GFLOPs=8.7.
CONCLUSION
this paper, we scrutinize the congruence between [CLS] attention and token significance, a foun- dational premise underlying existing token reduction approaches, and pinpoint its intrinsic unrelia- bility. To ameliorate this issue, we introduce METR- straightforward yet efficacious amalgama- tion of multi-exit architecture and token reduction. Extensive empirical evaluations corroborate that the early task pressure, engendered by the multi-exit loss, is highly effective in calibrating [CLS] attention. Furthermore, this adjustment elevate existing token reduction methods to new level.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work was partially supported by the Natural Science Foundation of China (Nos. U2336213 and 62122074) and Innovation Funding of ICT, CAS under Grant E0000000
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
Arian Bakhtiarnia, Qi Zhang, and Alexandros Iosifidis. Multi-exit vision transformer for dynamic inference. arXiv preprint arXiv:2106. Mxii:2206..1188, 2021.
Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. In International Conference on Machine Learning, pp. 527-536. PMLR,
2017.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffmaan. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
Mengzhao Chen, Mingbao Lin, Ke Li, Yunhang Shen, Yongjian Wu, Fei Chao, and Rongrong Ji. Cf-vit: A general coase-to-ffine method for vision transformer. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 37, pp. 7042-7052 2023a.
Mengzhao Chen, Wenqi Shao, Peng Xu, Mingbao Lin, Kaipeng Zhang, Fei Chao, Rongrong Ji, Yu Qiao, and Ping Luo. Diffraae: Differentiable compression rate for efficient vision transformers. arXiv preprint 230: 17997, 2023b.
Minghao Chen, Houwen Peng. Jianlong Fu, and Haibin Ling, Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 12270-12280 12280, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Mohsen Fayyaz, Soroush Abbasi Koohpayegani, Farnoush Rezaei Jafari, Sunando Sengupta, Hamid Reza Vaezi Joze, Eric Sommerlade. Hamed Pirsiavash, and Jürgen Gall. Adaptive token sampling for efficient vision transformers. In European Conference on Computer Vision. pp. 396-414. Springer, 2022.
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. Advances in Neural Information Processing Systems, 14:15908-15911 2021a.
Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11); 7436-7456, 2021b.
Joakim Bruslund Haurum, Sergio Escalera, Graham W Taylor, and Thomas B Moeslund. Which tokens to use? investigating token reduction in vision transformers. arXiv preprint
arXiv:2308 04657, 2023.
10

---

Published as a conference paper at ICLR 2024
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. Ad- vances in Neural Information Processing Systems (NIPS), 2015.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparamcterization with gumbel-softuax, arXiv preprint arXiv: 00.001144 144, 2016.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0888 2020.
Zhenglun Kong, Peiyan Dong. Xiaolong Ma, Xin Meng, Mengshu Sun, Wei Niu, Xuan Shen, Geng Yuan, Bin Ren, Minghai Qin, al. Spvit: Enabling faster vision transformers via soft token pruning, arXiv preprint arXiv:21 13890, 2021.
Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's-eyer s-eye-viww representation from multi-camera images via spa- tiotemporal transformers. In European conference on computer vision, pp. 1-18. Springer, 2022.
Youwei Liang, GE Chongjian, Zhan Tong, Yibing Song. Jue Wang, and Pengtao Xie. Evit: Ex- pediting vision transformers via token reorganizations. In International Conference on Learning Representations (ICLR), 2021.
Yang Lin, Tianyu Zhang, Peiqin Sun, Zheng Li, and Shuchang Zhou. Fq-vit: Post-training quanti- zation for fully quantized vision transformer. arXiv preprint arXiv:2111.13824, 2021.
Dongyang Liu, Meina Kan, Shiguang Shan, and CHEN Xilin. Function-consistent feature distilla- tion. In International Conference on Learning Representations (ICLR), 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, YunGang Jiang, and Ser-Nam Lim. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 12309-12318, June 2022.
Bowen Pan, Rameswar Panda, Yifan Jiang, Zhangyang Wang, Rogerio Feris, and Aude Oliva. Ia- red 2: Interpretability-awarr redundancy reduction for vision transformers. Advances in Neural Information Processing Systems, 34:24898 24911,, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Ma- chine Leurning, pp. 28492-28518 PMLR, 2023.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems (NeurIPS), 34:13997-13949, 2021.

---

Published as a conference paper at ICLR 2024
Sucheng Ren, Fangyun Wei, Zheng Zhang, and Han Hu. Tinymim: An empirical study of distilling mim pre-trained models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3687-3697, 2023.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 4510 4520, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 2014.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsuung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd international conference on pattern recognition (ICPR), pp. 2464-2469. IEEE, 2016.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMIR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Liwei Wang, Yuen-Yu Lee, Zhuowen Tu, and Svetlana Lazebnik. Training deeper convolutional networks with deep supervision. arXiv preprint arXiv:1505.02444 02496, 2015.
Wenhai Wang. Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. Proceedings of the IEEE/CVF international conference on computer vision, pp. 568-578. 2021.
Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision trans- former. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2964- 2022.
Huanrui Yang, Hongxu Yin, Pavlo Molchanov, Hai Li, and Jan Kautz. Nvit: Vision transformer compression and parameter redistribution. 2021.
Zhendong Yang, Zhe Li, Ailing Zeng, Zexian Li, Chun Yuan, and Yu Li. Vitkd: Practical guidelines for vit feature knowledge distillation. arXiv preprint arXiv:2209 222.03220.. 2022.
Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10809-10818, 2022.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 558-567, 2021.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 12104-12113, 2022.
Borui Zhao, Quan Cui. Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distilla- tion. In JEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pp. 11953- 11962, 2022.
12

---

Published as a conference paper at ICLR 2024
A
EXPERIMENT DETAILS
The experiment hyper-parameters are mostly inherited from the base method on which METR is built. All experiments are conducted on a server with NVIDIA RTX 4090 GPUs.
A.1 METR + EVIT
Under the 30-epoch fine-tuning setting. the official DeiT (Touvron et al., 2021) checkpoints are loaded as model initialization. The model is then trained with effective batch size 2048, actual learn- ing rate (after proportional scaling w.r.t. effective batch size) 8e-5, minimum learning 2e-6, without learning rate warm up. The cosine learning rate scheduler adopted. The optimizer is Adam with weight decay set to Ic-6. For self-distillation, the temperature for distillation loss (Hinton al., 2015) is set 1, and the weight, namely a in Eq. 10, set to 1.0; when Lm instead of fdd is used for ablation studies, the weight for Lo is also 1.0, as we find that Lm and L generally share the same optimal weight.
For the 100-epoch finetuning setting, the learning rate is enlarged to 8c-,, and the other hyper- parameters are left unchanged.
For the from-scratch training setting, the model is trained for 300 epochs. Consistent to the original setting of DeiT, the learning rate and minimum learning rate are set to 2e-3 and le-5, the number of warm-up epochs is set to 5, and weight decay is set to 0.05. The training starts with token keep ratio equal to I, namely no token reduction is applied: subsequently, from epoch 100 to epoch 150, the keep ratio linearly decays to the target value; finally, after epoch 150, the keep ratio is fixed to the target value. Different from the tie-ttning settings, we find that smaller a (weight for L,a), like 0.3, makes the best results, while larger a will hurt the performance.
For all of the above settings, token reduction is applied in the 4-th, 7-th, and 10-th blocks, with inat- tentive token fusion (proposed in the EVIT (Liang al., 2021) paper) turned on. Correspondingly, the early-exit heads are placed in these blocks.
A.2 METR- DIFFRATE
For experiments based on DiffRate (Chen et al., 2023), we adopt the Prune & Merge token reduction operation, and the searched token reduction schedule (i.e. how may tokens should be reduced in each block). Unlike EVIT, DiftRate densely applies token reduction in every block; supported by Tab. 4, we correspondingly add early-exit heads to all these blocks except the first and the last one.
All METR DiffRate experiments follow the 30-epoch finetuming setting, where the officially leased checkpoints for DeiT (Touvron et al., 2021) and MAE (He et al., 2022) (after supervised set effective batch size 1024, actual
finetuning) used for initialization. For all experiments, learning rate (after proportional scaling w.r.t. effective batch size) 8e-5, minimum learning le-6, 1 epoch of learning rate warm up. and weight decay 0.05. As an exception, for the ViT-L (MAE) model, the learning rate is set to 2e-5, The weight for Lad is set to 1 accross all experiments.
B DISCUSSION: OFF-THE-SHELF V.S. TOKEN-REDUCE FINETUNING
AWARE
Comparing Tab. with other experiments in the main text, we find that the advantage of METR numerically more salient in the off-the-shelf setting. We deduce that the reason that, with finetuning, the model can make some compromises, e.g. redundantly copy and store information among tokens, so that it could gain some robustness even when the token scoring metric is not reliable. In contrast, the off-the-shelf setting is way more challenging and poses stricter demands on the accuracy of token scoring. To empirically validate this deduction, we consider the random token selection baseline. Using the official DeiT-S checkpoint (79.80 top-1 accuracy), we first directly evaluate the model the off-the-shell setting, with token reduction applied and keep ratio set to 0.5. When [CLS] attention is used for token scoring and selection, the accuracy drops to 74.10; in contrast, when tokens are random reduced, the accuracy drops dramatically to 71.46. We then consider the 30-epoch finetuning setting, employing f.finat loss function, with token-reduction
13

---

Published as a conference paper at ICLR 2024
operation applied. When [CLS] attention is used for finetuning and evaluation, the accuracy is 78.47; when random token selection is used for finetuning and evaluation, the accuracy 77.12 the gap is much smaller than the previous setting. This contrast provides an explanation over why the advantage of METR is more pronounced under the off-the-shelf setting.
However, indicated by experiment results in main text, though the improvement in the finetuning setting is not as numerically salient as the off-the-shelf setting, it is still fairly consid- erable, especially when taking the average developing pace of the token reduction topic into con- sideration. It indicates that an accurate token scoring metric is still indispensable in fulfilling the potential of token reduction, even with coken-reduction aware finetuning applied. An explanation is that, the aforementioned compromises are unlikely to completely erase the influence of scoring metric quality. Furthermore, the more unreliable the metric is, the stronger the comprises have to be and the more model capacity would be occupied to achieve it, which would inevitably hurt the representation ability and the accuracy of the model.
REFERENCES
Mengzhao Chen, Wenqi Shao, Peng Xu, Mingbao Lin, Kaipeng Zhang, Fei Chao, Rongrong Ji, Yu Qiao, and Ping Luo. Diffrate: Differentiable compression rate for efficient vision transformers. arXiv preprint arXiv:2305. 17997, 2023.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. Ad- vances Neural Information Processing Systems (NIPS), 2015.
Youwei Liang, GE Chongjian, Zhan Tong, Yibing Song, Jue Wang. and Pengtao Xie. Evit: Ex- pediting vision transformers via token reorganizations. In International Conference on Learning Representations (ICLR), 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
14


Reviews:
[{"review_id": "6O7Yt850UY", "reviewer": "Reviewer_9w9u", "summary": "This work has proposed a new token-pruning method, by integrating the multi-exit strategy into ViT. This work diagnoses the inconsistency between [CLS] attention and token importance in early ViT block, which degrades the performance of token reduction methods. To tackle this problem, this work introduces multi-exit architecture that allows the [CLS] token to gather information pertinent to the task in the early blocks. It also adopts self-distillation to improve the quality of early supervision. As a results, it achieves state-of-the-art performance.", "questions": "It would be better to shows the GPU-throughput and compare it with those of SOTA.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### Good Motivation\nThis work has adeptly identified and proposed solutions for a problem in the literature of token pruning method of ViT.\n\n### Novelty and SOTA performance\nTo address the inconsistency between [CLS] attention and token significance at the early blocks, the proposed method that incorporates multi-exit into ViT) is novel and it shows effectiveness clearly by achieving state-of-the-art performance.\n\n### Nice visualization\nThis work shows well-supportive visualization examples.", "weaknesses": "No exists.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u51sD7OTmH", "reviewer": "Reviewer_Zdao", "summary": "This submission introduces METR, a simple and effective technique for informed token reduction applied in Vision Transformer-based image classification. An analysis presented in the manuscript, demonstrates that the commonly used [CLS] token attention scores, acting as an importance metric for token pruning, are far more effective on deeper blocks in contrast to shallower ones. This is attributed to the long gradient distance from the task loss, traditionally applied at the end of the network. \n\nTo remedy this, the manuscript proposes the introduction of intermediate classifiers at training time, forming a multi-exit transformer model, in which all token reduction blocks are exposed to stronger task supervision. Upon deployment, early-exits are removed eliminating any speed overhead, while extensive experiments demonstrate the effectiveness of the proposed method across different models and in comparison with several baselines.", "questions": "1. What motivated the use of self-distillation in place of traditional multi-exit training in the proposed setting? What are the authors' insights about the demonstrated effectiveness of this design choice?\n\n2. Is token-reduction aware fine-tuning indeed limiting the effectiveness of the proposed approach? If yes, this should be commented in the manuscript.\n\n3. In Tab.1,2,3 does “reduce ratio” refer to number of tokens or GFLOPs? Both should be reported to get the full picture. \n\nMinor comments/ Presentation:\n-Notation in Sec.3 is quite loose. Consider defining the dimensionality of each introduced symbol (X,x,A,...).\n-Sec3.2: Symbol a^{c-p} is confusing.\n-Sec4.1.1: without incorporate -> without incorporating,  Subsequently, We (...) -> Subsequently, we (...)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-The manuscript focuses on the very interesting interplay between multi-exit models and token reduction. \n\n-Sec. 3.2, introduces a simple, yet effective solution to the examined problem. The discussion on the use of attention as an importance metric is insightful and many relevant works can benefit from these findings.\n\n-Experiments are extensive in terms of examined models and baselines, and validate the superiority of the proposed approach to the baselines. \n\n-The manuscript is generally well-written and easy to follow.", "weaknesses": "-The use of self-distillation loss for the multi-exit training (Eq.9), in place of traditional multi-exit loss of Eq.8, although effective, is not adequately motivated. Self-distillation is typically used to improve the accuracy of the trained exits, which is not a requirement here as these are discarded at inference time. The manuscript would benefit from a more insightful analysis of what motivated this design choice/ why do the authors believe this works better than the traditional approach.\n\n-Row(2) in Tab.3 seems to be the equivalent of row (4) in Tab.2, where multi-exit and token-reduction fine-tuning are jointly applied (instead of the two-stage ablation in Tab2). If this is the case, it can be deduced that token-aware fine-tuning notably reduces the effectiveness of the proposed approach, leading to significantly smaller gains even when aggressive token reduction takes place. This fact is separate from the commented fading of multi-exit effects after separate fine-tuning and needs to be further investigated/discussed in the manuscript.\n\nNote: An appendix is mentioned in the manuscript (Sec.4), but was not accessible to the reviewer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UxPSqegH7I", "reviewer": "Reviewer_jWbQ", "summary": "The paper introduces METR, a straightforward approach that combines multi-exit architecture and token reduction to decrease the computational burden of vision transformers (ViTs) while maintaining accuracy. The authors discover a discrepancy between the attention score of [CLS] and the actual importance of tokens in early ViT blocks, which negatively affects the performance of token reduction methods relying on this metric. The authors demonstrate that METR can improve existing token reduction techniques and achieve better results than state-of-the-art methods on standard benchmarks, particularly when using high reduction ratios.", "questions": "Please see above", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, METR is a promising method that can help reduce the computational cost of ViTs while maintaining accuracy.\n\n- The paper is clear and well-motivated.\n- The idea is intriguing and demonstrates significant improvement compared to other baselines.\n- The evaluation is well-designed and highlights the core contribution in the design section.", "weaknesses": "- The evaluation demonstrates a notable improvement in accuracy compared to the baseline frameworks. It will be helpful to further demonstrate the reduction in latency with fewer FLOPs compared to other baselines.\n- It would be beneficial if the author could offer more insights in the method section, such as explaining how and why this design can enhance performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SvxynFCMgv", "reviewer": "Reviewer_G39w", "summary": "This paper combines two well-established methods, multi-exit neural networks and vision transformer (ViT) token reduction, to improve the efficiency of ViT. The background is that the prominent ViT token reduction techniques like EViT are based on removing unimportant tokens based on the attention scores that naturally indicate the contribution of the visual tokens to the final ViT prediction. The authors' motivation is that in the previous method, the ViT had no incentive to make sure the attention scores in the shallow layers aligned with the semantic importance of the visual tokens. To motivate the ViT to have such an incentive, the authors propose to exploit the multi-exit training method with ViT exits in the intermediate layers, which requires early class information fusion via the attention scores, thus ensuring the attention scores exhibit semantic importance properties. With the combination of the two techniques, the authors show a noticeable improvement compared to the baseline, especially when a significant number of tokens are removed or with more finetuning epochs.", "questions": "- Is the loss $L_{me}$ in Eq (8) also added to the $L_{total}$? It is not clearly mentioned in the paper.\n- Figure 2 can be improved with better illustration and explanation. Specifically, the arrows from the $A^c$ to the [CLS] are somewhat confusing. And why do the patch tokens have fading colors from the bottom up?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method is clearly motivated. Token reduction based on the existing attention scores in ViT has been shown to be an effective method in reducing computational costs while maintaining most of the classification accuracy. The authors propose to add the pressure of extracting the classification information in the shallow layers via the multi-exit mechanism, which forces the attention scores in the shallow layers to focus on the important tokens as the scores are directly used as weights to gather the information for classification in the multi-exits. \n- The experiments are extensive and show the effectiveness in improving the classification accuracy over the baselines, especially with a longer training schedule. Experiments also demonstrate the proposed method's effectiveness from different perspectives, including different base models (DeiT/MAE), different model sizes, and different base methods (EViT/DiffRate).\n- The visualization seems to support the claim that adding multi-exit modules to the ViT makes the attention scores in the shallow layers aligned better with human perception (higher scores are allocated to more important tokens on the objects).", "weaknesses": "The weaknesses are mostly minor issues, but it is important to address them to make the paper clearer and easier to understand.\n- The phrase \"Reduce Ratio\" is not a good term to indicate the ratio of how many tokens are kept. Please change to another term like \"keep ratio\" to make it clear.\n- Table 1 is not well explained. It took me a while to understand the setting of the experiment. The term \"Off-the-shelf\" is not immediately understandable. It would improve clarity by explicitly explaining the details of the experiments.\n- It would improve the readability of the paper by changing some words/notations to standard ones, e.g., CSE -> CE for cross-entropy.\n- There seems no appendix, but at the end of Section 4 first paragraph, it says \"See appendix for detailed experiment settings.\"\nPlease carefully proofread the whole paper to address these nuanced issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)