EViT: Expediting Vision Transformers via Token Reorganizations
Authors: Youwei Liang
Chongjian GE
Zhan Tong
Yibing Song
Jue Wang
Pengtao Xie
Venue: ICLR 2022 Spotlight (2022)
Keywords: Vision Transformers
multi-head self-attention
efficient inference

Abstract: Vision Transformers (ViTs) take all the image patches as tokens and construct multi-head self-attention (MHSA) among them. Complete leverage of these image tokens brings redundant computations since not all the tokens are attentive in MHSA. Examples include that tokens containing semantically meaningless or distractive image backgrounds do not positively contribute to the ViT predictions. In this work, we propose to reorganize image tokens during the feed-forward process of ViT models, which is integrated into ViT during training. For each forward inference, we identify the attentive image tokens between MHSA and FFN (i.e., feed-forward network) modules, which is guided by the corresponding class token attention. Then, we reorganize image tokens by preserving attentive image tokens and fusing inattentive ones to expedite subsequent MHSA and FFN computations. To this end, our method EViT improves ViTs from two perspectives. First, under the same amount of input image tokens, our method reduces MHSA and FFN computation for efficient inference. For instance, the inference speed of DeiT-S is increased by 50% while its recognition accuracy is decreased by only 0.3% for ImageNet classification. Second, by maintaining the same computational cost, our method empowers ViTs to take more image tokens as input for recognition accuracy improvement, where the image tokens are from higher resolution images. An example is that we improve the recognition accuracy of DeiT-S by 1% for ImageNet classification at the same computational cost of a vanilla DeiT-S. Meanwhile, our method does not introduce more parameters to ViTs. Experiments on the standard benchmarks show the effectiveness of our method. The code is available at https://github.com/youweiliang/evit

Published as a conference paper at ICLR 2022
NoT ALL PATCHES ARE WHAT YOU NEED: EXPEDITING VISION TRANSFORMERS VIA TOKEN REORGANIZATIONS
Youwei Liang'
Chongjian Ge
Zhan Tong?
Yibing Song?
Jue Wang?
Pengtao Xie
UC San Diego The University of Hong Kong
Tencent AI Lab
youwei @uucsi zhettgeefconnect. hk
elliot plxießeng uesd.edu
brrtingsong.cvdgaaan com arphidegmail.com com
ABSTRACT
Vision Transformers (ViTs) take all the image patches as tokens and construct multi-head self-attention (MHSA) among them. Complete leverage of these im- age tokens brings redundant computations since not all the tokens are attentive in MHSA. Examples include that tokens containing semantically meaningless or distractive image backgrounds not positively contribute to the ViT predictions. In this work, we propose to reorganize image tokens during the feed-forward pro- cess of VIT models, which integrated into VIT during training. For each forward inference, we identify the attentive image tokens between MHSA and FFN (i.e., feed-forward network) modules, which is guided by the corresponding class token attention. Then, reorganize image tokens by preserving attentive image tokens and fusing inattentive ones to expedite subsequent MHSA and FFN computations. To this end, method EVTT improves ViT's from two perspectives. First, under the same amount of input image tokens, our method reduces MHSA and FFN computation for efficient inference. For instance, the inference speed of DeiT-S is increased by 50% while its recognition accuracy is decreased by only 0.3% for ImageNet classification. Second, by maintaining the same computational cost, our method empowers ViTs to take more image tokens as input for recognition accuracy improvement, where the image tokens are from higher resolution im- ages. An example is that we improve the recognition accuracy of DeiT-S by 1% for ImageNet classification at the same computational cost of vanilla DeiT-S. Mcanwhile, our method does not introduce more parameters to ViTs. Experi- ments on the standard benchmarks show the effectiveness of our method. The code is available https:: httptggttttt/////////////tttehtt thub. com wang/e/iiiinniiiiiiie
INTRODUCTION
Computer vision research has evolved into Transformers since ViTs (Dosovitskiy al., 2021). Equipped with global self-attention, ViTs have shown impressive capability upon local convolution (i.e., CNNs) on prevalent visual recognition scenarios, including image classification (Dosovitskiy et al., 2021; Touvron et al., 2021a; Jiang et al., 2021; Graham et al., 2021), object detection (Carion al., 2020), and semantic segmentation (Xie et al., 2021; Liu et al., 2021; Wang et al., 2021a;c), with both supervised and unsupervised self-supervised) training (Pan et al., 2021; Ge et al., 2021) configurations. Based on the main spirit of ViTs (i.e., MHSA), there are wide investigations (Liu et al., 2021; Chu al., 2021; Wang al., 202 to explore the network structure of ViT models for continuous recognition performance improvement.
Along with the development of VIT models, the computation burden is becoming issue. As illustrated in (Dosovitskiy et al., 2021), training a ViT from scratch typically requires larger datasets (e.g., ImageNet-21k (Deng et al., 2009) and JFT-300M (Sun et al., 2017)) than those of CNNs (e.g.,
Maaor work done during an internship at Tencent Al Lab. "orresponning author.

---

Published as a conference paper at ICLR 2022
(a) Mask on backgrounds
(h) Mask objects
Figure I: DeïT-S (Touvron et al., 2021a) predictions with incomplete input image tokens. In (a), removing image tokens unrelated the visual content of the corresponding category does not dete- riorate ViT predictions. In (b), removing related image tokens makes VIT predict incorrectly.
CIFAR-10/100 (Krizhevsky et al., 2009) and ImageNet- 1k). Also, using more training iterations is necessity for network convergence (Dosovitskiy al., 2021; Touvron et al., 2021a; Wang al., 2022). Without such large scale training, the VIT models are not fully exploited and perform inferior on visual recognition scenarios. These issues motivate to expedite ViTs for practice usage.
The model acceleration of ViT's is important to reduce computational complexity. However, there are few studies focused on ViT acceleration. This because the significant model difference between CNNs and ViTs prevents CNN model acceleration (e.g.. pruning and distillation) from applying on VITs, Nevertheless, we analyze VIT from another perspective. We observe that not all image tokens in ViTs contribute positively to the final predictions. Fig. shows some examples where part of the input image tokens are randomly dropped out. In Fig. Ib, removing image tokens related to the visual content of the corresponding category makes the ViT predict incorrectly. In comparison, removing unrelated image tokens does not affect ViT predictions, as shown Fig. la. On the other hand, notice that ViTs divide images into non-overlapping tokens and perform self-attention (Vaswani et al., 2017) computation on these tokens. A notable characteristic of self-attention is that it can process varying number of tokens. These observations motivate tokens for ViT model accelerations. Naseer et al. (2021) also showed that ViT are robust to patch drop, which motivates the idea of dropping the less informative patches to accelerate ViT inference.
reorganize image
In this work, we propose token reorganization method to identify and fuse image tokens. Given all the image tokens as input, compute token attentiveness between these tokens and the class token for identification. Then, we preserve the attentive image tokens and fuse the inattentive tokens into one token to allow the gradient back proppproee through the inattentive tokens for better attentive token identification. In this way, we gradually reduce the number of image tokens as the network goes deeper to decrease computation cost. Also, the capacity of the ViT backbone can be flexibly controlled via the identification process where no additional parameters are introduced. We adopt our token reorganization method on representative ViT models (i.c., DeiT (Touvron et al., 202 la) and LV-VIT (Jiang et al., 2021)) for ImageNet classification evaluation. The experimental results show our advantages. For instance, with the same amount input image tokens, our method speeds up the DeïT-S model by 50%, while only sacrificing 0.3% recognition accuracy ImageNet classi- fication. On the other hand, we extend our methods to boost the ViT model recognition performance under the same computational cost. By increasing the input image resolution, our method facilitates Vision Transformers in taking more image tokens to achieve higher classification accuracy. Numer- ically, we improve the ImageNet classification accuracy of the DeiT-S model by 1% under the same computational cost. Moreover, by using an oracle ViT to guide the token reorganization process, our method can increase the accuracy of the original DeiT-S from 79.8% to 80.7% while reducing its computation cost by 36% under the multiply-accumulac accumulate computation (MAC) metric.
RELATED WORK
2.1 VISION TRANSFORMERS
Transformers (Vaswani al., 2017) have drawn much attention to computer vision recently due to its strong capability of modeling long-range relation. A few attempts have been made to add self- attention layers Transformers top of CNNs in image classification (Hu et al., 2019), object de-
2

---

Published as a conference paper at ICLR 2022
tection (Carion et al., 2020), segmentation (Wang al., 2021c), image retrieval (Lu et al., 2019) and even video understanding (Sun et al., 2019; Girdhar et al., 2019). Vision Transformer (ViT) (Doso- vitskiy et al., 2021) first introduced a set of pure Transformer backbones for image classification and its follow-ups modify the ViT architecture for not only better visual recognition (Touvron et al., 2021a; Yuan al., 2021; Zhou al., 2021) but many other high-level vision tasks, such ob- ject detection (Carion et al., 2020; Zhu et al., 2020; Liu et al., 2021), semantic segmentation (Wang al., 202 la; Xie et al., 2021; Chu et al., 2021), and video understanding (Bertasius et al., 2021; Fan et al., 2021), I). Vision Transformers have shown its strong potential as an alternative to the previously dominant CNNs.
2.2 MODEL ACCELERATION
Neural networks are typically overparameterized (Allen-Zhu et al., 2019), which results in signifi- cant redundancy in computation in deep learning models. To deploy deep neural networks on mobile devices, we must reduce the storage and computational overhead of the networks. Many adaptive computation methods are explored (Bengio al., 2015; Wang et al., 2018; Graves, 2016; Hu al., 2020; Wang et al., 2020b; Han et al., 2021b) to alleviate the computation burden. Parameter prun- ing (Srinivas Babu, 2015; Han et al., 2015; Chen et al., 2015b) reduces redundant parameters which are not sensitive to the final performance. Some other methods leverage knowledge distilla- tion (Hinton et al., 2015; Romero et al., 2014; Luo et al., 2016; Chen et al., 2015a) to obtain a small and compact model with distilled knowledge of larger one. These model acceleration strategies are limited to convolutional neural networks.
There are also some attempts to accelerate the computation of the Transformer model, including proposing more efficient attention mechanisms (Wang et al., 2020a; Kitaev et al., 2020; Choroman- ski et al., 2020) and the compressed Transformer structures (Liu et al., 2021; Heo et al., 2021; Wang et al., 2021a). These methods mainly focus on reducing the complexity of the network architecture through artificially designed modules. Another approach to ViT acceleration is reducing the number of tokens involved in the inference of ViT's. Notably, Wang et al. (202 lb) proposed a method to dy- namically determine the number of patches to divide on an image. The VIT will stop inference for an input image if it has sufficient confidence in the prediction of the intermediate outputs. Remark- ably, Ryoo et al. (2021) proposed Tokenl Learnrr to expedite ViTs, where a relatively small amount of tokens are learned by aggregating the entire feature map weighted by dynamic attention map conditioned on the feature map. This can be seen as a sophisticated method for tokenizing the input images. Different from Token our work focuses on the progressive selection of informative tokens during training. Another related work is Dynamic (Rao et al., 2021), which introduces method to reduce token for fully trained ViT, where extra learnable neural network is added to ViT to select a subset of tokens. Our work provides a novel perspective for reducing the compu- tational overhead of inference by proposing token reorganization method progressively reduce and reorganize image tokens. Unlike DynamicViT, our method does not need a fully trained ViT to help the training and brings no additional parameters into ViT.
3 TOKEN REORGANIZATIONS
Our method EVIT is built upon ViT (Dosovitskiy et al., 2021) and its variants for visual recognition. We first review ViT and then present how to incorporate our method into the ViT training procedure. Each component of EVIT, including attentive token identification and inattentive token fusion, will be elaborated. Furthermore, we analyze the effectiveness of our method by visualizing the attentive tokens at different layers and discuss training on higher resolution images with EVIT.
3.1
VIT OVERVIEW
Vision Transformers (ViTs) are first introduced by Dosovitskiy et al. (2021) into visual recognition. They perform tokenization by dividing input image into patches and projecting each patch to token embedding. An extra class token [CLS] is added to the set of image tokens and is respon- sible for aggregating global image information and final classification. All of the tokens are added by a learnable vector (i.e., positional encoding) and fed into the sequentially stacked Transformer encoders consisting of multi-head self-attentic (MHSA) layer and feed-forward network (FFN).
3

---

Published as a conference paper at ICLR 2022
Token keep rate
1.0
0.9
0.8
0.7
0.6
0.5
Top-1 I Acc (%)
79.8
79.7(-0.1) 79.2(-0.6)
78.5(-1.3) 76.8(-3.0) 73.8(-6.0)
Table 1: ImageNet classification accuracy of a straightforward inattentive token removal for a trained DeiT-S (Touvron et al., 2021a). The inattentive tokens are directly removed based on the attention from the class token to other tokens at the 4th 7th and 10th layers.
In MHSA, the tokens linearly mapped and further packed into three matrices, namely Q, K, and V. The attention operation is conducted as follows.
QK
Attention K, V) = Softmax
V Vd
(1)
where d is the length of the query vector. The result of Softmax (QK (QKT/V is square matrix which is called the attention map. The first row of attention map represents the attention from [CLS] all tokens and will be used determine the attentiveness (importance) of each token (detailed in the next subsection). The output tokens of MHSA are sent to FFN. consisting of two fully connected layers with GELU activation layer (Hendrycks & Gimpel, 2016) in between. At the final Transformer encoder layer, the [CLS] token is extracted and utilized for object category prediction. More details of Transformers can be found Vaswani et al. (2017).
3.2 ATTENTIVE TOKEN IDENTIFICATION
Let denote the number of input tokens to ViT encoder. In the last encoder of ViT, the [CLS] to- ken is taken out for classification. The interactions between [CLS] and other tokens are performed via the attention mechanism (Vaswani al., 2017) the ViT encoders:
KT V
Belass Softmax
Jclass
VV a.
Vd
where Jclass K, and V denote the query vector of [CLS), the key matrix, and the value matrix, respectively, in an attention head. In other words, the output of the [CLS] token Telass linear combination of the value vectors V = [V1, 221,22..... U....., un]] with the combination coefficients (denoted by a in Eq. 2) being the attention values from [CLS] with respect to all tokens. Since e; comes from the i-th token, the attention value (i.e., the i-th entry in a. determines how much information of the i-th token is fused into the output of [CLS] ] (i.c., Velass) through the linear combination. is thus natural to assume that the attention value a; indicates the importance of the i-th token.
Caron et al. (2021) also showed that the [CLS] token in ViTs pays more attention (i.e.. having larger attention value) to class- class-specific tokens than to the tokens the non-object regions. To this end, we propose to use the attentiveness of the [CLS] token with respect to other tokens to identify the most important tokens. Based on these arguments, simple method to reduce com- putation in VIT is to remove the tokens with the smallest attention values. However, we find that directly removing those tokens severely deteriorates the classification accuracy, as shown in Table 1. Therefore, we propose to incorporate image token reorganization during the ViT training process.
Moreover,
In multi-head self-attention layer, there are multiple heads performing the computation of Eq. 1 in parallel. Thus, there multiple [CLS] attention vectors (h), h [1....,I] HH) with H being the total number of attention heads (Vaswani et al., 2017). We compute the average attentiveness value of all heads by à = Σm= (h H.. As shown in Figure 2, we identify and preserve the tokens corresponding to the ho largest (top-k) elements in a (k hyperparameter), which we call the attentive tokens, and further fuse the other tokens (which we call the inattentive tokens) into new token. The fusion of tokens detailed in the following paragraph. We define the token keeping rate as Ι k/n.
3.3 INATTENTIVE TOKEN FUSION
Although the tokens on the backgrounds of images are less informative and can be discarded without significantly influencing the performance of the ViT model, they may still be able to contribute to
4

---

Published as a conference paper at ICLR 2022
Taken Reorgenization
Attentive
Multi Multi-Head Head Self- Attention
Taken
Identfication
Feed Forward Network
Add Norm
Add Norm
Image
Inattentive
Taken
Calculated CLS
Anteniveness
Figure 2: Token reorganization within single Transformer encoder. Based on VIT (Dosovitskiy et al., 2021), we reorganize tokens in the original Transformer encoder. Specifically, we calculate the attentiveness of the class token with respect to each image token. Then, we use the attentiveness value criterion to identify the top-k attentive tokens and fuse the inattentive tokens.
the prediction results. On the other hand, some images may have large object parts covering large proportion of the images. Thus, is possible to remove some tokens corresponding to those object parts when we select fixed number of tokens to keep in ViT encoder, which may negatively affect the image recognition performance. To mitigate these problems, we propose to fuse the inattentive tokens at the current stage to supplement attentive ones, as illustrated in Figure 2. The inattentive token fusion benefits our method to preserve more information provided by the inattentive tokens. Specifically, denote the indices set of the inattentive tokens as N. The proposed inattentive token fusion weighted average operation, which can be written as Thused DiEN 0;2; The fused the subsequent layers. The computation
token Brussd is appended the attentive tokens and cost of token fusion is negligible compared to the bulk computation VIT.
3.4 ANALYSIS
Training with higher resolution images. Since our approach is efficient in processing image to- kens, able to feed more tokens into an EVIT while maintaining the computational cost the same level as vanilla VIT. straightforward method to get more tokens is resizing the input im- ages higher resolution and keeping the patch size unchanged. Note that these higher resolution images do not necessarily come from raw images with higher resolution. In our vision recognition experiments ImageNet, we simply resize the standard input images of size 224 224 larger spatial size (e.g., 256 x 256) via bicubic interpolation to obtain the higher resolution images, which further divided into more tokens. Therefore, compared vanilla ViT, EViT uses no additional information from the images to obtain the prediction results in both training and inference. The experimental results Table validate the effectiveness of the proposed method.
Visualization. The proposed EVIT accelerates ViTs by identifying the attentive tokens and discard- ing the redundant calculation on inattentive image tokens. To further investigate the interpretability of EVTT, we visualize the attentive token identification procedure in Figure 3. We present the origi- nal images and the attentive token identification results different layers (e.g., the 4th 7th and 10th layer). It can be seen that as the network deepens, the inattentive tokens are gradually removed or fused, while the most informativeeatttentiee tokens are identified and preserved. In this way, the pro- posed method facilitates the ViTs in focusing class-specifi tokens in images. The visualization results also validate that our EVIT is effective in dealing with images with either simple backgrounds complex backgrounds.
5

---

Published as a conference paper at ICLR 2022
input image
layer 4
layer? 7
layer 10
imput image
layer 4
layer
layer 10
Figure 3: Visualization of inattentive tokens on EViT-DeiT-S with 12 layers. The masked regions represent the inattentive tokens that are fused into a new token. Our method can effectively identify inattentive tokens in images with either simple inattentive tokens (in the first two rows) or complex inattentive tokens (in the last two rows).
Table 2: Comparison on the two variants of EVIT DeiT-S (Touvron al., 2021a). The values of the Top-1 and Top-5 accuracy are averaged over three independent trials. The number behind + is the standard deviation of the three trials. The number in blue is the gap of the corresponding value w.r.t. the baseline DeïT-S.
Keep rate
Top-I Ace (%) Top-5 Acc (%)
Throughpot (images/s)
MACs
Top-I Ace (%)
Top-5 Ace (%)
Throughput (imagesis)
MACs
DeiT-S
79,8
2923
79.8
2923
EVT without inattentive token fusion
EVIT with inattentive token fusion
0.9
79.9:00.1++0]
98.9:0.0
3201 (+10%)
(-13%)
79.8::0.1
(-0.0)
95.0:0.00 (+0.1)
3197 (+9%)
4.0 (-13%)
79.7110.0
94.10.0
(+27%)
(:24%)
79.810.0 0.00
94910.0
3619 (+24%)
(:24%)
0.7
79.4+t0.1 1 (-0.4)
94.740.1 (4.2)
4249 (+45%)
3-35%)
79.550.0
94.8=0.0 (-0.1)
4385 (+50%)
3.0 (-35%)
79.140.2 (-0.) 94,540.11 (-0.)
4967 (+70%)
(-43%)
78.9=0.1
(-09)
94.5+0.0 (-0.4)
4722 (462%)
(-43%)
0.5
78.4::0.
14.1+0.00-03
5325 (+82%)
2.3 (-50%)
78.5+0000
94.220.0 (-0.7)
5408 (+85%)
233 (-50%)
4 EXPERIMENTS
Implementation details. We train all of the models the ImageNet (Deng et al., 2009) training set with approximately 1.2 million images and report the accuracy on the 50k images in the test set. By default, the token identification module is incorporated into the 4th 7th and 10th layer of DeîT-S and DeiT-B (with 12 layers in total) and incorporated into the 5th 9th and 13th layer of LV-VIT-S (with 16 layers in total). The image resolution in training and testing is 224 224 unless otherwise specified. For the training strategies and optimization methods, we simply follow those in the original papers of DeiT (Touvron al.. 2021a) and LV-VIT (Jiang al., 2021). Since our method can be casily incorporated into these models without making substantial modifications to them, the original training strategies work well with our method. Besides, we adopt a warmup strategy for attentive token identification. Specifically, the keep rate of attentive tokens is gradually reduced from I to the target value with a cosine schedule. Unlike Dynamic (Rao et al., 2021), we do not use pretrained ViT to initialize our models in most experiments, except in the experiments with an oracle ViT (see the following paragraphs). We train the models with EVIT from scratch for 300 epochs on 16 NVIDIA 1.00 GPUs and measure the throughput of the models single A100 GPU with a batch size of 128 unless otherwise specified. The multiply-accumulate computations (MACs) metric is measured by torchprof ill (Liu, 2021).

---

Published as a conference paper at ICLR 2022
Table Results of EVIT on LV-ViT-S
Table 4: Results of training EViT-DeiT-S with keep rate of 0.7 for different epochs.
Keep rale
Top-1 Ace (%)
Top-S Ace (%)
Throughput (imuges/s)
MACs (G)
Keep rate
Eppochs
Top-1 (%)
Top-5 Ace (%)
MAC's
LV-VIE-S
83.3
2112
(5.0(-0.3)
96.3
(+40%)
4.7 (-29%)
0.7
0.7
450
300
80.2
79.5
94.8
95.1
3.0
3.0
0.5
82.5 (-0.8)
96.2
3603 (+71%)
3,9 (-41%)
0.7
600
81.0
95.3
3.0
We report the main results of EVIT on Tables 2 and 3. On both DeiT and LV- VVT, the proposed EViT achieves significant speedup while restricting the accuracy drop in relatively small range. For example, DeîT-S trained with EViT with keep rate 0.7 increase the inference throughput by 50% while maintaining the Top-1 accuracy reduction within 0.3% on ImageNet.
Inattentive token fusion. To mitigate the problem of information loss in token removal, we propose inattentive token fusion, which fuses the non-topk tokens according to the attentiveness from the [CLS] token. We experimentally compare the proposed token reorganization method with and without inattentive token fusion. As shown in Table 2, token reorganization with inattentive token fusion typically outperforms the one without inattentive token fusion. Although the improvement small, there indicates the effectiveness of our attentive token identification because the majority of effective tokens are well preserved. Morcover, the accuracy fluctuation in EVIT with inattentive token fusion is smaller than the vanilla inattentive token removal, as shown by the standard deviation in Table 2. Inattentive token fusion also benefits EVTT on high resolution images, as shown in Table 9.
additional computational overhead introduced. From another perspective, it
Epochs of training, Since ViTs do not have inductive bias such as translational invariance processed by CNNs, they typically require more training data and/or training epochs reach comparable generalization performance as CNNs (Dosovitskiy et al., 2021; Touvron et al., 2021a). We find that training longer epochs continues to benefit ViTs in the efficient computation regime. We train the DeiT-S model with 0.7 keep rate for longer epochs of 450 and 600, respectively. As shown in Table 4, their performance steadily improves with training epochs.
Training/Finetuning on higher resolution images. By fusing the inattentive tokens, We are able to feed EVIT with more tokens under the same computational cost. Therefore, we train and/or finetune EVIT resized images with higher resolutions than the standard resolution of 2242, and we report the results on Table 5. We can see that EVIT in most tested cases performs favourably against vanilla DeiT/LV-ViT, while having comparable baselines. Notably, training EViT-LV-ViT-S on images of resolution 224 and further finetuning on higher resolution of 448² for another 100 epochs gives a very competitive Vision Transformer model, achieving an ImageNet top-1 accuracy of 84.7%, which is 0.3% higher than LV-ViT-S @ 384 with basically the same throughput and number of parameters. The experimental results validate our hypothesis that images typically contain tokens that are less informative and contribute little the recognition task. Since ViTs perform global self-attention among all tokens in as early as
higher inference throughput than the
Table 5: Results of training/finetuning high resolution images. EViT-DeiT-S and EVIT-LV-ViT-S have a throughput/MACs comparable to the baselines while achieving better recognition accuracy on ImageNet. The numbers before and after indicate the image size in the 300-epoch training and 100-epoch finetuning, respectively.
(b) LV-VIT-S
(a) DeïT-S
Model
Keep
Image size
Top-1 (%)
Top-5
imgls
MACs (G)
Model
Keep rate
Image
Top-1
Top-5 (%)
imgls
(9)
(G)
(9t)
DeiT.S
224
2923
4.6
IN
1.0
224
83.3
2112
6.6
LV-VIE-S
1.0
224
384
84.4
557
21.9
EVIT
EVIT
0.5
288
256
79.3
94.7
3788
3138
3.9
3.1
EVIT
0.9
2410
83.6
96.5
1956
6.8
EVIT
0.5
304
81.0
95.6
2905
4.4
EVIT
0.8
256
83.6
96.6
1901
6.9
EVIT
256
80.0
EVIT
EVIT
0.7
0.7
256
272
83.5
83.7
96.5
96.6
2102
1829
7.1
EVIT
0.6
288
81.0
95.4
2927
4,5
EVIT
0.5
304
83,4
EVIT
0.7
272
80.3
95.3
2870
4.6
EVIT
0.7
272 1 448
84.7
97.1
548
21.5
7

---

Published as a conference paper at ICLR 2022
Table Results of training EViT-DeiT-S and EViT-DeiT-B using DeïT-S as an oracle. Training for longer epochs continues to benefits the EVIT in efficiency regime.
Model
Orucle
Keep rate
Epochs
Top-1 (%)
Top-5 (%)
Throughput (img/s)
MACs (G)
EViT-DeiT-S
0.7
300
79.5
94.8
4338
3.0
EViT-DeiT-S
0.7
300
80.8
95.4
4,385
3.0
EViT-DeiT-S
0.7
450
81.0
95.5
4.385
3.0
EViT-DeiT-S
V
0.7
600)
81.3
95.5
4388
3.0
DeiT-B
X
1.0
300
81.8
95.6
1295
17.6
EViT-DeiT-B
X
0.7
300
81.3
95.3
2053
11.6
EViT-DeiT-B
0.7
300
82.1
95.6
2053
11.6
85
EVITAD VAVT----V
EVITAD 7-VV
EVITA EWT.D.--VVVV--SS...
EEVIAO. -WW.
Jet BOGDM
SvrT-
EVTT0.7-LY-VVISS
PCPVT-B
ImageNet Top-1 Acc (%)
Coll 306516
Vistrmer
82
PTTL
SVT
Lantil-s
81
Moal-Miri
EVIA0.5 DeIT S000
EficientNet B2
EVIR 5-0eT-5g288
VT.
DWITS
EVITO..-
EVITO 5-D6T-38200
Ef
79
C
Vafome-T
EVIT70.5-De1-5
CoaT-Tiny
78
Utee
EffciennNNe.80
0
1000
2000
3000
4000
5000
6000
Throughput (images/s)
Figure 4: Comparison of different models with various cocuracy-throughygn trade-off. The proposed method EVTT achieves better trade-off than the other methods (marked with circles). The throughput measured NVIDIA A100 GPU using the largest possible batch size for each model. The input image size 224 unless specified after the @.
the first layer, the early interaction and information exchange between tokens makes possible to discard/fuse some of the least informative tokens in intermediate ViT layers since they have been "seen" by other tokens (including the [CLS] token). In comparison, the receptive field of CNN at the shallow layers is relatively small due to the locality property of convolution, which makes it difficult reduce computation at early stages.
Training with an oracle VIT. In EVIT, the criterion of selecting tokens is the attention between the [CLS] token and other tokens. Therefore, would very helpful if we know the importance of each token to the prediction tasks in advance. To this end, we introduce an oracle ViT to guide the [CLS] through the token selection process. A good oracle knows which tokens are important and which are not. For this purpose, use fully trained DeiT-S/B DeiT-S/B with the parameters the oracle ViT. such that the EVIT know which tokens contribute more to the prediction result. We train EVIT equipped with oracle using the same training setup as training a vanilla EVIT. As shown Table 6, training with an oracle significantly improves the recognition accuracy of EVIT.
oracle and initialize EViT-
8

---

Published as a conference paper at ICLR 2022
Table 7: Comparisons on EViT and DynamicViT (Rao et al., 2021). v indicates the model is initialized with pre-trained DeiT-S. For fair comparison, the throughput is measured on the same machine with the same setting using a maximum batch size.
Model
Pre-trained Epochs
Keep
Top-1 (%)
#Params (M) Throughput (img/s) MACs (G)
DynamicViT-Deii - EViT-DeiT-S
30
0.5
77.5
22.8
5579
2.2
30
0.5
78.5 (+1.0)
22.1 (-0.7)
5549
2.3
EViT-DeiT-S
100
0.5
79.1 (+1.6)
22.1 (-0.7)
5549
2.3
DynamicViT-DeiT-S
30
0.7
79,3
22.8
4439
3.0
EViT-DeiT-S
30
0.7
79.5 (+0.2)
22.1 (-0.7)
4478
3.0
EViT-DeiT-S
100
0.7
79.8 (+0.5)
22.1 (-0.7)
4478
3.0
DynamicViT-DeiT-S
X
300
0.5
22.8
5579
2.2
EViT-DeiT-S
X
300
0.5
78.5 (+5.4)
22.1 (-0.7)
5549
2.3
DynmmicWi--DeiT' X 300
0.7
77.6
22.8
4439
3.0
EViT-DeiT-S
300
0.7
(+1.9)
22.1 (-0.7)
4478
3.0
Comparison with DynamicVite We compare the proposed EViT with DynamicViT (Rao et al., 2021). Note that the authors of Dynamic used finetuning strategy. Therefore, we first compare the two methods in finetuning. and then we also train DynamicViT from scratch for 300 epochs to compare it with our baselines. In the case of training from scratch, no pretrained VIT can be used, and thus the distillation loss and KL. loss in Dynamic are discarded. The results in Table show that EVTT outperforms DynamicViT ViT in recognition accuracy under the same computational cost while EVIT uses fewer parameters. Moreover, the accuracy of EViT continues to improve in longer-epoch training. When training from scratch, a significant accuracy reduction is observed in Dynamic which suggests that DynamicViT requires a teacher ViT (i.e., the parameter initializa- tion, the distillation loss, and the prediction (KL) loss from pretrained ViT (Rao al., 2021)) to obtain a good performance.
Comparison with other Vision Transformers. We plot the accuracy and throughput of various ViTs in Figure 4, which shows EVIT is very competitive among the vision models in terms of computation-accuracy trade-off. For EVIT, only plot the EVIT models that are trained from scratch for fair comparisons, except EViT/0.7-LV-viTS @48, which is finetuned for 100 epochs. The comparing methods include DeiT (Touvron al., 2021a), CaiT (Touvron al., 2021b), LV- VIT (Jiang et al., 2021), CoaT (Xu et al., 2021), Swin (Liu et al., 2021). Twins (Chu et al., 2021), Visformer (Chen al., 2021). Cin (d' Ascoli al., 2021). TNT (Han al., 202 la). Efficient- Net (Tan & Le, 2019), and PVT (Wang al., 2021a). These ViTs variants focus modifying the VIT architectures or the interaction methods between image tokens to achieve improvement over the vanilla VIT (Dosovitskiy et al., 2021). Thus, their methods and ours complementary to each other, which makes it possible to incorporate our method into some of the ViT variants.
5 CONCLUSION
In this paper, we present a token reorganization method. By identifying the tokens with the largest attention from the class token, the proposed EVIT reaches better trade-off between accuracy and efficiency than various Vision Transformer models. Moreover, we propose inattentive token fusion, which fuses the information from less informative tokens improves both the recognition accuracy and training stability. Experimentally, we apply the pro- posed token reorganization method two variants of Vision Transformer, namely, DeiT (Touvron et al., 2021 la) and LV- (Jiang et al., 2021). In both variants, EViT achieves a significant speedup in inference while the reduction in recognition accuracy relatively small. Moreover, when training on higher resolution images, EVIT improves the accuracy to various extents while maintaining sim- ilar or smaller computation cost as the original DeiT/LV-Vil VIT models. Besides, when equipped with an oracle ViT which knows which tokens are more important, EVIT can achieve further improve- ment on the trade-off between accuracy and efficiency. The proposed EViT can be easily adapted in VITs and brings no additional parameters, nor does it require sophisticated training strategies. The proposed token reorganization method can serve as an effective acceleration approach for Vision Transformers.
token. Inattentive token fusion

---

Published as a conference paper at ICLR 2022
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang, Learning and generalization in overparameter- ized neural networks, going beyond two layers. In Advances Neural Information Processing Systems, 2019.
Emmanuel Bengio, Pierre-Lue Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint wrXivv1511.00 2015.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In International Conference on Machine Learning, 2021.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv:151 1tti::1511..5555 1.06641, 2015a.
Wenlin Chen, James Wilson, Stephen Tyree. Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In International Conference on Machine Learning, 2015b.
Zhengsu Chen, Lingxi Xie, Jianwei Niu, Xuefeng Liu, Longhui Wei, and Qi Tian. Visformer: The vision-friendly transformer. arXiv preprint arXiv:2104.1280 2021.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. 14794, 2020.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. Arxiv preprint 2104.13840 2021.
Stéphane d' Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv preprint arXiv:2103.10697 2021.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hier- archical image database. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations, 2021.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. arXiv:2104.1122) 2021.
Chongjian Ge. Youwei Liang, Yibing Song, Jianbo Jiao, Jue Wang, and Ping Luo. Revitalizing CNN attention via transformers self-supervised visual representation learning. Advances in Neural Information Processing Systems, 2021.
Rohit Girdhar. Joao Carreira, Carl Doersch, and Andrew Zisserman. Video action transformer net- work. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin. Hervé Jégou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. arXiv preprint arXiv:2 104.0/ 136, 2021.
10

---

Published as a conference paper at ICLR 2022
Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983 2016.
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. transformer. arXiv preprint 21t::203300012,, 2021a.
Transformer in
Song Han, Jeff Pool, John Tran, and William J Dally. Learning both weights and connections for 2015.
efficient neural networks. arXiv preprint arXiv:
Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey, arXiv preprint arXiv:2102.04900 2021b.
Dan Hendryeks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
SEli:1660600415,, 2016.
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. arXiv preprint 1rXiv:2103.16331 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv: urXiv:1503.02533 2015.
Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. Local relation networks for image recognition. In IEEE/CVF International Conference on Computer Vision, 2019.
Ting- Kuei Hu, Tianlong Chen, Haotao Wang, and Zhangyang Wang. Triple wins: Boosting accu- racy, robustness and efficiency together by enabling input-adaptive inference. In International Conference on Learning Representations, 2020.
Zihang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Yujun Shi, Xiaojie Jin, Anran Wang, and Jiashi Feng. All tokens matter: Token labeling for training better vision transformers. arXiv preprint arXiv:2104. 10858, 2021.
Nikita Kitaev, tukasz preprint arXiv:2001.04457, 2020.
Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. 14030, 2021.
Zhijian Liu. torchprofile.
com/ http:://ghhtttt....../aaaaaaaa/r
2021.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task aanostin visiolin- guistic representations for aibon-and-languaaa tasks. arXiv preprint arXiv: 02265, 2019.
Ping Luo, Zhenyao Zhu, Ziwei Liu, Xiaogang Wang, and Xiaoou Tang. Face model compression by distilling knowledge from neurons. In AAAI Conference on Artificial Intelligence, 2016.
Muhammad Muzammal Naseer, Kanchana Ranasinghe, Salman H Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming- Hsuan Yang. Intriguing properties of vision transformers. Advances in Neural Information Processing Systems, 34, 2021.
Tian Pan. Yibing Song. Tianyu Yang. Wenhao Jiang, and Wei Liu. Videomoco: Contrastive video representation learning with temporally adversarial examples. In IEEE/CV Conference on Com- puter Vision and Pattern Recognition, 2021.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamicvit: Efficient vision transformers with dynamic token sparsification. arXiv preprint arXiv: 102.2200000244 2021.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv: 1412.6550, 2014.
11

---

Published as a conference paper at ICLR 2022
Michael S Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. To- kenleamnr:: Adaptive space-time tokenization for videos. In Advances Neural Information Processing Systems, 2021.
Suraj Srinivas and R Venkatesh Babu. Data-free parameter pruning for deep neural networks. arXiv preprint urXiv:1507.061499 2015.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta, Revisiting unreasonable effec- tiveness of data in deep learning era. In IEEE/CVF International Conference on Computer Vision, 2017.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In IEEE/CVF International Conference on Computer Vision, 2019.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference Machine Learning, 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. arXiv preprint arXiv:2 17239, 2021b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, 2017.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.08 04766, 2020a.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In IEEE/CVF International Conference on Computer Vision, 2021a.
Xin Wang. Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph Gonzalez. Skipnet: Learning dy- namic routing in convolutional networks. In European Conference on Computer Vision, 2018.
Yue Wang, Jianghao Shen, Ting-Kuei Hu, Pengfei Xu, Tan Nguyen, Richard Baraniuk, Zhangyang Wang, and Yingyan Lin. Dual dynamic inference: Enabling more efficient, adaptive, and control- lable deep inference. IEEE Journal of Selected Topics in Signal Processing. 2020b.
Yulin Wang, Rui Huang, Shiji Song, Zeyi Huang, and Gao Huang. Not all images are worth 16x16 words: Dynamic vision transformers with adaptive sequence length. arXiv preprint arXiv:2105. 15075, 2021b.
Yuqing Wang. Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2021c
Ziyu Wang, Wenhao Jiang, Yiming Zhu, Li Yuan, Yibing Song, and Wei Liu. Dynamixer: vision mlp architecture with dynamic mixing, arXiv preprint arXiv: 12083, 2022.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. In Advances in Neural Information Processing Systems, 2021.
Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transform- ers. arXiv preprint arXi::2104.0633 2021.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101. 2021.
12

---

Published as a conference paper at ICLR 2022
Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103. 1886, 2021.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr:
Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
13

---

Published as a conference paper at ICLR 2022
A VISUALIZATION
In this part, we present more visualization results in Figure 5 to show the attentive token identifica- tion. The input images randomly selected from the ImageNet dataset. The results validate that our EVIT able to deal with different images from various categories.
input image
layer
layer
layer 10
input image
layer
layer
layer 10
Figure 5: Extended visualization results of inattentive tokens on EViT-DeiT-S with 12 layers.. The regions without masks represent the attentive tokens. The masked regions denote the inattentive tokens that are fused into new token. Our EVIT is effective in dealing with images from different categories.
14

---

Published as a conference paper at ICLR 2022
Algorithm 1: PyTorch-llke pseudocode of EVIT for a VIT encoder.
number attention heads
N: number of input tckens
# C: the dimension of token vector
kt the token keep rate
x: the input tokens with shape IN, C), with the first being the [CLS)
toxen
fc_g, fc_k, fc_v: linear transforms for query, key, and value of self-
attention
8: matrix multiplication:
proj: linear projection in self-attention
norm: layer normalization
ffn: foed-forward network
initialize avg_cls_attn zeros (K-1)
_oout = [] x_residual
(x)
# multi-head se1 f-attention computation for i in range (0, H): # compute se self-attention for each attention head linearly map the tokens to query, key, and value matrices [e_k[i](x) fe_v[i]
q, k, v fc]g(i]
compute the attention map
attn attn softmax (attn, dim-1)
k.transpose ()) sqrt (C/H)
x_head attn 0 x_out. apgend (x_head)
compute the [CLS] attentiveness w.r.t. other tokens, wi without using [CLS] attention to itself cls_attn - [O, 1:] avg clg_cls_att += attn
concatenate the output tokens all heads concat ou_, dim=l)
x (x) # shape: [N, C]
x = x x residual
average the [CLS] attentiveness all heads avg_ole_attn /=
sort the avg_cls_atee attn in descending order sorted idx = sort (avg_cls_attn)
compute the number of attentive tokens, without counting the [CLS] token ceil(k 1))
to topk_idx sorted_cll_attn[:t idx[:K] non_topk_attn, non_topk_idk = ate idx [K:]
separate [CLS] taken and other tokens els_token x[0:1]
withoutt_css = x|1:]
obtain the attentive and inattentive tokens iiteetivv_tokens x_without_elsi topkkkk_ixi inattentive_tokens x_without_cls [non_topk_idk]
compute the weighted combination of inattentive tokens
fuked_token non_topk_attn i inattentive_tokens
concatenate these tokens
x_new concat woncat ([cls_token, attentive_tokens fuked_token], dim=0)
x_residual = x_new
x_n norm (x_new)
X_new ffn (x_new)
x new x new + resiiuual
x_new
15

---

Published as a conference paper at ICLR 2022
Table 8; Results of EViT on DeiT-B
Keep rate
Top-1 Acc (%)
Top-5 Acc (%)
Throughput (images/s)
MACs (G)
DefT-B
81.8
95.6
1295
17.6
0.9
81.8 (-0.0)
95.6 (-0.0)
1441 (+11%)
15.3 (-13%)
0.8
81.7 (-0.1)
95.4 (-0.2)
1637 (426%)
13.2 (-25%)
0.7
81.3 (-0.5)
95.3 (-0.3)
2053 (459%)
11.5 (-35%)
0.6
80.9 (-0.9)
95.1 (-0.5)
2177 (+68%)
10.0 (-43%)
0.5
80.0 (-1.8)
94.5 (-1.1)
2482 (492%)
8.7 (-51%)
Table 9: Comparisons of the two variants (i.e., vanilla inattentive token removal and inattentive token fusion) of EViT-DeiT-S on higher resolution images.
Method
Keep rate
Image size
Top-1 1 Acc (%)
Top-5 Acc (%)
Throughput (images/s)
MACs (G)
DeiT-S
1.0
224
79.8
94.9
2923
4.6
Inattentive token removal
0.5
304
80.9 (+1.1)
95.3 (40.4)
2927
4.4
Inattentive token fusion
0.5
304
(+1.))
95.6 (+0.7)
2905
4.4
B EXTENDED EXPERIMENTS
More results of EVIT on DeïT. We provide the performance of EVIT on DeîT-B (Touvron et al., 2021a) in Table 8. Again, the resulting models of EViT-DeiT-B achieve significant speedup while restricting the accuracy drop in relatively small range. For example, EViT-DeiT-B with keep rate of 0.8 increases inference throughput by 26% while maintaining the Top-1 accuracy reduction within 0. 11% on ImageNet.
More comparisons of vanilla inattentive token removal and inattentive token fusion. We com- pare the performance of the two variants of EVIT, namely, vanilla inattentive token removal and inattentive token fusion, on the training of higher resolution images. In this scenario, EViT with inattentive token fusion also outperforms the EViT with vanilla inattentive token removal, espe- cially in the Top-5 accuracy, as shown in Table 9. Although the improvement is relatively small, it is practical inattentive token fusion brings no additional computational overhead compared to vanilla inattentive token removal.
Using tokens-to-token, attentive scores for inattentive tokens identification. As discussed in Section 3.2, we use the attention from the [CLS] token to other tokens as the attentive scores to identify the class-related tokens that important to visual recognition. This attentive/inattentive token identification approach is motivated by the fact that the final prediction of ViT' is determined by the representation of the [CLS] token in the last layer. However, also find that it is possible to use a tokens-to-token computed the average of the attention from all tokens to all tokens. Specifically, recall the attention map A is computed by:
attention approach to identify the attentive tokens, where the attention scores
QK Vd
A Softmax
(3)
The attentive score is computed by
1 A
(4)
where n is the number tokens and A,,: is the i-th row of attention map A. When there are multiple self-attention heads, the final attentive score is averaged over all heads: à >n (h)) H, where H/ the number of heads. In a nutshell, in the tokens-to-tokens strategy, each token can "vote" to decide which tokens are attentive, while in the token can "vote". As shown in Table 10, EVIT with the tokens-to-tokent attention strategy performs comparably with the [CLS] to-tokens attention strategy but is less efficient. Nevertheless, this experimental observation opens the door for incorporating the proposed EVIT into the ViT variants with no [CLS] tokens (e.g., PVT (Wang al., a)).
strategy, only the [CLS]
16

---

Published as a conference paper at ICLR 2022
Table 10: The performance of EViT-DeiT-S with different attentive token identification strategies.
Method
Inattentive token fusion
Keep rate
Top-1 Acc (%)
Top-5 Ace (%)
Throughput (img/s)
MACs (G)
0.5
78.4
94.1
5325
2.3
[CLS]-to- tokens attentiveness
0.5
78.5
94.2
5408
2.3
0.7
79.4
94.7
4249
3,0
0.7
79.5
94.8
4385
3.0
0.5
78.4
94.2
5241
2.3
tokenssto-tokens
0.5
78.5
94.3
5346
2.3
attentiveness
X
0.7
79.4
94.8
4152
3.0
0.7
79.4
94.7
4290
3.0
Table 11:
Training time for 300-epoch training of DeïT-S and EViT-DeiT-S with different keep rates.
Keep rate
Top-1 Ace (%)
Top-5 Acc (%)
Throughput (images/s)
MACs (G)
Training time
xhors)
DeiT-S
79.8
94.9
2923
4.6
201
0.9
79.8 (-0.0)
95.0 (+0.1)
3197 (+9%)
4,0 (-13%)
188 (-6%)
0.8
79.8 (-0.0)
94.9 (-0.0)
3619 (+24%)
3.5 (-24%)
174 (-13%)
0.7
79.5 (-0.3)
94.8
4385 (+50%)
3.0 (-35%)
160 (-20%)
0.6
78.9 (-0.9)
94.5 (-0.4)
4722 (+62%)
2.6 (-43%)
150 (-25%)
0.5
78.5 (-1.3)
94.2
5408 (+85%)
2.3 (-50%)
145 (-28%)
Training time of EVIT. We provide comparison on the training time of DeiT-S (Touvron al., 2021a) and EViT-DeiT-S Table 11. The results show that EVTT requires considerably smaller GPU>hours to train when compared EViT continues to improve in recognition accuracy with larger training epoch (as shown in Table 4), the performance gap between a vanilla VIT and an EVIT can be narrowed down and even closed when the EVIT is trained for the same amount of time as the VIT.
vanilla DeiT under the same number of epochs. Since
Finetuning of EVIT on high resolution images. We provide results of finetuning EViT on high- resolution images in the last three rows in Table 12. To finetune EViT-DeiT-S from low-resolution images high-resolution images, we need to resolve the discrepancy between the positional embed- ding (Touvron et al., 2021a) of two different resolutions. To this end, we interpolate the positional embedding of lower resolution to the desired higher resolution. This method works quite well in our experiments, shown in the last three rows in Table 12. However, directly training higher resolution images seems better way to train EVIT, as requires less training time (because no finetuning is performed) and typically obtains higher accuracy than finetuning.
Token reorganization locations. It is possible that different combinations of keep rates and to- ken reorganization locations can reach the same computational efficiency. For example, can a) move the reorganization modules into shallower layers and increase the keep rates, or b) move the reorganization modules into deeper layers and decrease the keep rates, to keep the computation cost (approximately) unchanged. To get a deeper understanding of this aspect, we train EViT-DeiT-S models with different token reorganization locations and keep rates, each of which has similar computational cost as the baseline. The results in Table 13 reveal two perspectives. First, moving the reorganization modules into shallower layers deteriorates the accuracy. For example, when the token reorganization module is placed before the third layer (i.c., at the first or second layer), the recognition accuracy drops considerably even though the computational cost is the same. This sug- gests that VIT cannot identify the important tokens at the early stages, which is quite reasonable as the processing of input tokens is just started at shallow layers, where the attention maps are unreli- able for token removal. Second, placing the reorganization modules in different deeper layers has only marginal influence on the accuracy. For instance, when the token reorganization modules are placed behind the third layer, the resulting models have basically the same accuracy. This suggests that EVIT has stable performance as long as the reorganization locations are not in shallow layers. Because of this reason, we refrain from searching for (slightly) better configuration of the reorga-
17

---

Published as a conference paper at ICLR 2022
Table 12: Results of training/linetuning on high resolution images. The numbers before and after indicate the image size in the 300-epoch training and 50-epoch finetuning, respectively.
Model
Keep rate
Image size
Top-1 (%)
Top-5 (%)
Throughput (img/s)
MACs (G)
DeïT-S
1.0
224
79.8
94.9
2923
4.6
EViT-DeiT-S
0.5
256
79.3
94.7
3788
3.1
EViT-DeiT-S
0.5
288
80.1
95.0
3138
3.9
EViT-DeiT-S
0.5
304
81.0
95.6
2905
4.4
EViT-DeiT-S
0.6
256
80.0
95.0
3524
3.5
EViT-DeiT-S
0.6
81.0
95.4
2927
4.5
EViT-DeiT-S
0.7
272
80.3
95.3
2870
4.6
EViT-DeiT-S
304
80.3
95.2
2905
EViT-DeiT-S
0.6
224
288
80.7
95.3
2927
4.5
EViT-DeiT-S 0.7
224
272
80.6
95.2
2870
4.6
Table 13: The performances of EViT-DeiT-S with different of keep rates and reorganization loca-
tions (R.L.). For fair comparison, keep the variants having the same level of computational cost (i.c.. MACs) as our EVIT baseline in the first row by tuming the keep rates.
#R.L.
Reorganization locations
Keep rates
Top-1 Acc (%) Top-5 Acc (%) MACs (G)
[4 7, 10]
0.7
79.50
94.77
3.0
5, 7, 10
[0.64, 0.70, 0.70]
79.57 (+0.07)
94.80
3.0
3, 7,
00.74, 0.70, 0.70
79.47 (-0.03)
94.72
3,0
4, 8,8,10 10
00.70, 0.64, 0.70
79.64 (+0.14)
94.86
3.0
4, 60
(0.70, 0.75, 0.70 0.70
79.54 (+0.04)
94.78
3.0
6,8,
0.6
79.43 (-0.07)
94.69
3,0
6,6,00
0.765
78.72 (-0.78)
94.22
3,0
4.
0.81
78.45 (-1.05)
94,26
3,0
1,5, 9
[0.80, 0.79, 0.79)
76.34 (-3.16)
93.14
3.0
4, 6, 8, 10
0.76
79.38 (-0.12)
94.72
3.0
2, 5, 8, 11]
[0.90, 0.77, 0.62, 0.50)
78.93 (-0.57)
94.52
3.0
7
[4, 5, 6, 9,8,9,10
0.85
79.39 (-0.11)
94.68
3.0
nization locations and keep rates. We adopt simple strategy to decide the reorganization locations our experiments, where the token reorganization locations cut ViT into blocks with the same number of layers. Specifically, for ViT with L layers and t token reorganization layers in total, first obtain the separating length L/(t + 1). Then, the layer indices the reorganization layers 1). which euts the ViT evenly. For the keep rates, simply set them to the same value for each token reorganization module the EVIT.
are 1, 28 1,....
Adopting the DINO attention map for inattentive token identification. In DINO (Caron al., 2021), the authors showed that the [CLS] attention produced by a self-supervised trained ViT is naturally focused on the object parts in images. Therefore, may wonder the good DINO attention can be used in EViT for inattentive token identification. To answer this question, we use the [CLS] attention produced by the last layer of DINO guide the token selection process in EViT. Instead of starting the token reorganization process at the 4th layer of EViT-DeiT-S as we have done in previous experiments, we start the token reorganization the first layer. Specifically, we select the top 50% tokens that correspond to the positions with the strongest DINO attention (i.e., corresponding to the top 50% attention values in the DINO attention). Experimental results in Table 14 show that EVTT works well with the DINO attention. As a comparison, when the reorganization layer is placed the first layer in vanilla EVIT, as shown in Table 13, significant accuracy reduction is observed, while EViT equipped with DINO can maintain its recognition ability even 50% tokens are removed as early as the first layer. However, this is not an efficient approach in practice because obtaining the DINO attention requires forwarding the input through
18

---

Published as a conference paper at ICLR 2022
The performance of EEVIT-DeiT- equipped with the DINO attention. The "MACs" in the table is the computational cost of the EViT-DeiT-S model and "Combined MACs" is the sum of the MACs of both EViT-DeiT-S and the DINO model (i.e., DeīT-S) which is used to obtain the DINO attention.
Table 14:
Method
Keep rate
Drop layer
Top-1 Acc
Top-5 Ace
MACs
Combined MACs
(%)
(%)
(G)
(G)
DINO-EVIT-DeiTSS 0.5
78.7
94.2
2.2
6.8
Table 15: An ablation of token removal strategy in EViT-DeiT-S. EViT-topk is the method of keeping the tokens with the largest [CLS]-to-tokee attention values. EViT-random denotes the method of randomly keeping the tokens, while EViT-mink is the method of keeping the tokens with the smallest [CLS]-to-token attention values.
Method
Keep rate
Top-1 I Ace (%)
Top-5 Ace (%)
MACs (G)
EViT-topk 0.5
78.5
94.2
2.3
EViT-random
0.5
77.3 (-1.2)
93.5 (-0.7)
2.3
EViT-mink
0.5
75.2 (-3.3)
92.4 (-1.8)
2.3
EViT-topk 0.7
79.5
94.8
3.0
EViT-random
0.7
78.4 (-1.)
94.2 (-0.6)
3.0
EViT-mink
0.7
76.4 (-3.1)
93.2 (-1.6)
3,0
the DINO model (i.e., DeiT model) and bringing extra computational burden, as shown by the "Combined MACs" in Table 14.
The correlation between the quality of the attention masks and the performance of EVIT. From Figures 3 and 5, we can see that the masks produced by EVIT are quite intuitive and of good quality, which means that the attention values of the CLS token to other tokens correlate well with the semantic segmentation of images. Therefore, we can deliberately use low quality masks in EViT to see how its performance changes with the quality of the masks. Note that we use the largest (CLs]-to-tokee attention values to obtain the mask for the keeping tokens (we call this method EViT-topk). Thus, the worse possible mask one can obtain is probably using the smallest [CLS] to-tokens attention values (we call this method EViT-mink), and not-so-bad mask can be obtained by randomly generating a mask without considering the [CLS]-to-tole -to-tokens attention (we call this method EViT-random). We perform experiments to see the classification accuracy of EViT under these circumstances. As shown in Table 15, the classification accuracy of these three methods clearly correlates with the quality of the masks. From the experimental results, we conclude that masks of better quality help improve the accuracy. Therefore, we believe it is a potential direction for future research as how to (efficiently) obtain a better mask improve the performance of EVIT
The attention scores of the inattentive tokens w.r.t. the layer depth. In order to find how the attentive scores of the inattentive tokens evolve, we randomly sample several images and plot the change of the [CLS] -to-tokens attention values of n inattentive tokens (n 25) w.r.t. the depth of layers in Fig. 6. Specifically, we use pre-trained DeiT-S and identify the n tokens with the smallest [CLS] -to-tokess attention in the 4th layer. Then we keep track of these selected tokens in the subsequent layers and draw the box plot to see how the statisties (e.g., the mean, standard deviation, and maximum and minimum values) change with the layer depth. The results suggest that the tokens that are identified as inattentive tokens in the shallow layer tend to be less preferred in deeper layers.
The evolution of the inattentive tokens during EVIT training process. To prove the effectiveness of incorporating EVIT into the training process of VIT, we present the evolution of the inattentive tokens during the training of EViT-DeiT-S with a keep rate of 0.7 in Figure 7. It can be observed that the initial training stage, the EVIT model tends misidentify the inattentive tokens (e.g., the bulk of the car in the second column of the first three rows). However, the inattentive tokens gradually converge to the less informative regions of images during the training process. For example, in the last column in Figure 7, the masked regions mainly focus on the background (e.g., the grass). This observation validates the effectiveness of our proposed token identification methods.
19

---

Published as a conference paper at ICLR 2022
3800
03012
6:M
Layer Depen
Layer Ocpth
Image (a)
Image (b)
3800
508K
Layer Depth
Layer Depth
Image (c)
Image (d)
Figure 6: The box plot of the evolution of the attention scores of inattentive tokens with the layer depth. The orange line in the box plot indicates the averaged attention scores of the inattentive tokens at certain layer.
20

---

Published as a conference paper at ICLR 2022
Input image
Epoch
Epoch 9
Epoch 20
Epoch 70
Epoch 180
Epoch 299
layer layer 4
layer 10
Input image
Epoch
Epoch
Epoch 20
Epoch
Epoch 180
Epoch 299
layer layer 4
layer 10
Input image
Epoch
Epoch
Epoch 20
Epoch
Epoch 180
Epoch 299
layer layer
layer 10
Visualization results of the evolution of the removed tokens during the EVIT training process at different layers. The regions without masks represent the attentive tokens. The masked regions denote the inattentive tokens that are fused into a new token. At the initial training stage (e.g., the first epoch), the model is not stable in identifying the attentive tokens. As the training proceeds, EVTT gradually converges, producing meaningful masks.
Figure 7:
21


Reviews:
[{"review_id": "tqunGg2n9NF", "reviewer": "Reviewer_soMj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n- The idea of this  paper is interesting. Different from the previous DynamicViT that reduces tokens with an extra sub-network,\nthis work starts from the working principle of the self-attention module and leverages the relationships between the class token and the normal image tokens. I believe this paper provides a promising way to reduce the computational cost of vision transformers for future research.\n\n- The motivation and presentation of this paper are clear. Pseudo code is provided. It is easy to follow for readers. I am looking forward to the  release of the whole code.\n\n- Experiments are thorough. Results on both DeiT and LV-ViT show that the proposed approach can efficiently compress the computational cost while maintain the baseline results.\n\nCons:\n\n- One of the issue that I concern is the performance of the inattentive token fusion. As shown in Table 2, it seems that the classification performance gap between the two sub-tables is rather small. The question is why the authors introduce the inattentive token fusion?\nIn addition, as described in the late part of the experiment section, training on images with higher resolutions can improve the model performance. So, the question is how the inattentive token fusion would affect the model performance when taking images with larger image sizes as input?\n\n- In Page 7, the authors report that training with a longer schedule benefits the model performance when utilizing the proposed approach. Is there any evidence/observation showing that what leads to such a phenomenon? \n\n- In the experiments, three reorganization layers are used. Have you attempted to use more? \n\n- The proposed approach can be regarded as a type of downsampling operations. So, would the proposed approach perform better than previous classic downsampling operations, such as pooling and convolution. A typical counterpart should be the PVT model.\n\n- In Fig. 4, the authors show the comparison of the proposed method with other vision transformers, but there is no corresponding content describing the differences between the proposed approach and other methods. It would be great to add a new paragraph to give more description.\n\nTypos:   In Page 7, 'wee' should be 'we.' In Page 9, 'reach' should be 'reaches.' Please carefully revised the manuscript and correct the typos.  \n\n", "summary_of_the_paper": "This paper aims to expediting vision transformers by reducing the number of tokens. The main contribution is  the attentive token identification, which is based on calculating the attentiveness of the class token with respect to each image token. Experiments on DeiT and LV-ViT show that the proposed approach is able to successfully reduce the computational cost of the baseline models with nearly no performance drop.", "summary_of_the_review": "In spite of some experiments missing, regarding the significance of the novolty of this paper, I think it deserves a positive score. If the concerns can be well addressed, I would like to lift the rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "rD02LbBigMb", "reviewer": "Reviewer_WtKh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n1. The paper is well written and easy to follow.\n2. The method is simple and does not require a pre-trained model.\n3. The visualization looks good and intuitive.\n4. ViT models are accelerated by a large margin.\n\n## Neutral\n1. Marginal improvements over the baselines with the standard resolution and epochs. The gain is more significant with more input tokens and a longer training schedule. I think this is acceptable as the method reduces the computation cost for each iteration.\n\n## Main Weaknesses\n1. It is arguably difficult to combine the proposed method with multi-scale transformers, e.g. Swin transformer. It might be doable but the real throughput gain could be very limited. The current performance improvement on the vanilla ViT cannot cover this limitation.\n2. The ablation difference of inattentive token fusion is very limited and close to pure training noise. Thus, the discussion of the vanilla model’s training instability and the improved training efficiency lacks enough justification.\n3. It would be interesting to see if one can directly train DynamicViT from scratch. There seems no reason prevents us from doing so. As this is one of the main advantages of the proposed method over DynamicViT, more discussion is expected.\n4. Apart from the class token, one could also use the full tokens-to-tokens attention to compute the attentive score. This is an obvious alternative but is not sufficiently studied.\n\n## Suggestions\n1. It is interesting to see if we can make use of the good DINO mask as an attentive region, although the DINO mask requires much longer training.\n2. The interpretability of the learned mask could be evaluated just like what is done in DINO and then we may be able to see how the mask quality correlates with performance.\n3. To better support the intuition of dropping tokens, one could visualize (in a naively pre-trained ViT) whether a token is never attended by the class token once it is less preferred (i.e. roughly monotonically decreasing attention from the class token). If there are cases where a token is only attended in the last layer, then the intuition of dropping-a-token-forever may not hold.\n", "summary_of_the_paper": "This paper proposes a mechanism to dynamically drop transformer tokens, according to class token attention score. It is able to reduce computation with similar accuracy or improve accuracy with controlled computation.", "summary_of_the_review": "I recommend accepting this paper for its technical novelty, the model acceleration, the performance gain (when it is trained with a long schedule), and the potential of improving more with a better attentive score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oYCOdwdvNQI", "reviewer": "Reviewer_y9ec", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\nThe proposed method is simple while effective. Using the class token to guide image token selection seems reasonable and is well motivated by the observations (Fig. 1 and Table 1). The whole computation does not bring much time cost while reducing inattentive tokens\nduring the ViT feed forward process. This design benefits both offline and online stages. \n\nIn the experiments, the proposed method accelerates recent ViT models (DeiT and LV ViT) by increasing throughput and decreasing MAC. Meanwhile, the recognition accuracy does not drop much. Another perspective is that under a similar computational cost, using more tokens from higher resolution input images improves the prediction accuracies. This will benefit a series of ViTs without bringing additional time costs.\n\nCons:\nThe proposed method illustrates inattentive token removal at the 4th, 7th, and 10th layers. Whatis the rationale behind this choice? How to decide which layer to process when handling other ViT models? A clear illustration upon how to integrate this method into ViT models will be more convincing.\n\nThe token reorganization seems to fuse discarded tokens into one to supplement attentive tokens. As have explained in Sec. 3.3, this might be because the attentive token selection is not stable at the beginning of the training. If that is the case, a visualization upon token variations (initially random token selection, then the discarded token converges to the unrelated content) would be useful to support the claim.\n", "summary_of_the_paper": "In this paper, an EVIT method is proposed for vision transformer speedup. It reduces image tokens based on the token attentiveness, which is measured by the class token. The inattentive tokens are reorganized as one to support attentive tokens. Experiments have shown on the\nbenchmarks for visual recognition.", "summary_of_the_review": "Token processing is promising as it potentially benefits all the ViTs. The proposed method improves ViTs from the efficiency and accuracy perspectives. This will bring a wide range of impacts on the ViTs development. This reviewer recommends acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "6C8DtR5MlD1", "reviewer": "Reviewer_SSye", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "PROS:\n+\tImage token identification via the class token guidance is interesting. The class token adaptively removes image tokens based on the corresponding category influence. The discarded tokens are reorganized/fused into one token to facilitate network training at the initial training stages.\n+\tThe experiments on the efficiency and accuracy evaluations show the proposed method is effective for current Vit models.\nCONS: \n-\tThe proposed method gradually removed tokens from different layers of ViT models. Is there any relationship between these removed tokens from different layers? Since the removed tokens are hardly maintained, will the proposed fusion strategy be able to restore them at the remaining layers?\n-\tGiven a ViT model, is the design of the proposed method arbitrary for all the layers or some fixed layers? Where to drop and fuse tokens within a ViT model deserves further discussion.\n-\tWhat leads to the difference between Table 1 and Table 2(a)? A straightforward inattentive token fusion removal decreases the DeiT more, while without inattentive token fusion decrease the same model less? If the rate of Table 2(a) is set as 0, will the results be the same?\n-\tReporting the actual time cost will benefit a thorough understanding of ViTs acceleration.", "summary_of_the_paper": "Research on Vision Transformer s ( ViTs is heavy Different from the prior investigations that focus on proposing ViTs structures, this work focuses on image tokens and study how to effectively wipe out useless ones to model acceleration This brings a potential advantage that\nwhen maintaining the same time cost, models are able to take more tokens i.e, from a higher resolution image ) for prediction accuracy improvement. These two aspects are evaluated in the experiments.", "summary_of_the_review": "The token reorganization is new. The results are fine. Addressing these raised issues above will make this submission more convincing.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an approach to select visual tokens in images and reorganize them for the object classification, within Transformers. All four reviewers find the paper interesting and novel, and they are also very positive about the experimental results. The authors also addressed minor concerns of the reviewers successfully through the discussion phase, clarifying details and adding experiments.

We recommend accepting the paper.
Decision: Accept (Spotlight)