Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition
Authors: Yulin Wang
Rui Huang
Shiji Song
Zeyi Huang
Gao Huang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Vision Transformer
Efficient Inference
Image Recognition

Abstract: Vision Transformers (ViT) have achieved remarkable success in large-scale image recognition. They split every 2D image into a fixed number of patches, each of which is treated as a token. Generally, representing an image with more tokens would lead to higher prediction accuracy, while it also results in drastically increased computational cost. To achieve a decent trade-off between accuracy and speed, the number of tokens is empirically set to 16x16 or 14x14. In this paper, we argue that every image has its own characteristics, and ideally the token number should be conditioned on each individual input. In fact, we have observed that there exist a considerable number of “easy” images which can be accurately predicted with a mere number of 4x4 tokens, while only a small fraction of “hard” ones need a finer representation. Inspired by this phenomenon, we propose a Dynamic Transformer to automatically configure a proper number of tokens for each input image. This is achieved by cascading multiple Transformers with increasing numbers of tokens, which are sequentially activated in an adaptive fashion at test time, i.e., the inference is terminated once a sufficiently confident prediction is produced. We further design efficient feature reuse and relationship reuse mechanisms across different components of the Dynamic Transformer to reduce redundant computations. Extensive empirical results on ImageNet, CIFAR-10, and CIFAR-100 demonstrate that our method significantly outperforms the competitive baselines in terms of both theoretical computational efficiency and practical inference speed. Code and pre-trained models (based on PyTorch and MindSpore) are available at https://github.com/blackfeather-wang/Dynamic-Vision-Transformer and https://github.com/blackfeather-wang/Dynamic-Vision-Transformer-MindSpore.

Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition
Yulin Wang!" Rui Huang Shiji Song' Zeyi Huang² Gao Huang 1.3f
DDpartment of Automation, BNRist, Tsinghua University, Beijing, China
Huawei Technologies Ltd., China
Beijjng Academy of Artificial Intelligence, Beijing, China (wang-yl19, @mails. Samaals.tssnnhuu edu. en, muangzeyi2@huavei {shijis, gaohuang) gaohuang)(tsinghus @etsingusnnn edu.
Abstract
Vision Transformers (ViT) have achieved remarkable success in large-scale image recognition. They split every 2D image into fixed number of patches, each which is treated as token. Generally, representing an image with more to- kens would lead higher prediction accuracy, while it also results in drastically increased computational cost. To achieve a decent trade-off between accuracy and speed, the number of tokens is empirically set to 16x16 14x14. In this paper, we argue that every image has its own characteristics, and ideally the to- ken number should be conditioned observed that there exist a considerable number of "easy" images which can be accurately predicted with tion of "hard" ones need a finer representation. Inspired by this phenomenon, we propose a Dynamic Transformer to automatically configure a proper num- ber of tokens for each input image. This is achieved by cascading multiple Transformers with increasing numbers of tokens, which are sequentially acti- vated in an adaptive fashion at test time, i.e., the inference is terminated once a sufficiently confident prediction is produced. We further design efficient fea- ture reuse and relationship reuse mechanisms across different components of the Dynamic Transformer to reduce redundant computations. Extensive empirical results on ImageNet, CIFAR-10. 10, and CIFAR-100 demonstrate that our method significantly outperforms the competitive baselines in terms of both theoretical computational efficiency and practical inference speed. Code and pre- trained mod- els (based on PyTorch and MindSpore) are available at hotth: //githbb com/ b
each individual input. In fact, we have
number of 4x4 tokens, while only small frac-
Vision Transf ormer and https: com///aaazzeeatteerrran/aaaaaai http://DDyaaaaiiiioooo..ooom Vision- Trans
Introduction
Transformers, the dominant self-attention-bart models in natural language processing (NLP) [10
40
38 17 recently. In the large scale ImageNet benchmark [9]. while exhibit excellent scalability with the further growing dataset size (c.g., on JFT- 300M (1I).. These models split each image into fixed number of patches and embed them into ID tokens as inputs. Typically, representing the data using more tokens contributes to higher prediction accuracy, but leads intensive computational cost, which grows quadratically with respect to the
have been successfully adapted to image recognition problems [I] 55
particular, vision Transformers achieve state-of-the-an performance
'Equal contribution. CCorresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

token number in self-attention blocks. For a proper trade-off between efficiency and effectiveness, existing works empirically adopt 14x14 or 16x16 tokens [TT 55].
In this paper, we argue that it may not be optimal to treat all samples with the same number of tokens. In fact, there exist considerable variations among different images (c.g.. contents, scales of objects, backgrounds, etc.). Therefore, the number of representative tokens should ideally be configured specifically for each imput. This issue is critical for the computational efficiency of the models. For example, we train T2T-ViT-12
Table 1: Accuracy and computa- tional cost of T2T-ViT-11 with dif- ferent token numbers on ImageNet.
of Tokens
14x14
7x7
4x4
Accuracy
76.7%
70.3%
60.8%
FLOPs
1.78G
0.47G
0.21G
[55] with varying token numbers, and report the corresponding accuracy and FLOPs in Table One can observe that adopting the officially recommended 14x14 tokens only correctly recognizes 155.9% (76.7% v.s. 60.8%) more test samples compared to that of using 4x4 tokens, while increases the computational cost by 8.5x (1.78G v.s. O.21G). In other words, computational resources are wasted on applying the unnecessary 14x14 tokens to many "easy" images for which 4x4 tokens are sufficient.
Motivated by this observation, we propose a novel Dy- namic Vision Transformer (DVT) framework, aiming to automatically configure decent token number conditioned on each image for high computational efficiency. In spe- cific, cascade of Transformers are trained using increas- ing number of tokens. At test time, these models are se- quentially activated starting with less tokens. Once a pre- diction with sufficient confidence has been produced, the inference procedure will be terminated immediately. As a consequence, the computation is unevenly allocated among "easy" and "hard" samples by adjusting the token number, yielding considerable improvement in efficiency. Impor- tantly, we further develop feature-wil -iise and relationship- wise reuse mechanisms to reduce redundant computations. The former allows the downstream models to be trained on the basis of previously extracted deep features, while the later enables leveraging existing upstream self-attention relationships to learn more accurate attention maps. Illus- trative examples of our method are given in Figure [
Easy
97.3% Apple
Terminate Ourput
?
Continue
Reuse Computation
21.4%
?
Egget
Egnet
15.2%
?
43.7% Palace
93.4%
Palace
Palace
H1nd
Figure 1: Examples for DVT.
Notably, DVT is designed as general framework. Most of the state-of-the-an image recognition Transformers, such as ViT [T], DeiT [38], and TiT-ViT (55), can be straightforwardly deployed as its backbones for higher efficiency. Our method is also appealing in its flexibility. The computational criterion. This
cost of DVT is able to be adjusted online by simply adapting the characteristic makes DVT suitable for the cases where the available computational resources fluctuate dynamically or a minimal power consumption is required to achieve a given performance. Both situations ubiquitous in real-world applications (e.g., searching engines and mobile apps).
The performance of DVT evaluated on ImageNet 99 and CIFAR 26 with T2T-VTT (55) and DeiT 38) Experimental results show that DVT significantly improves the efficiency of the backbones. For examples, DVT reduces the computational cost of T2T-ViT by 1.6-3.6x without sacrificing accuracy. The real inference speed on NVIDIA 2080Ti GPU is consistent with our theoretical results.
2 Related Work
vision Transformers (ViT) have recently been developed for image recognition [II. Although ViT by itself not comparable with state-of-the-art convolutional networks (CNN) on the standard ImageNet benchmark, it attains excellent results when pre-trained on the larger JFT-300M dataset. DeiT 38 studies the training strategy of ViT and proposes knowledge distilling-bases approach, surpassing the performance of ResNet [19] Some following works such as T2T- [53, TNT [7]] CaiT 39
Vision Transformers. Inspired by the success of Transformers on NLP tasks [10 40 43
ViT (62), CPVT (6) LocalViT (2 and CrossViT (5) focus on improving the architecture design of VIT. Another line of research proposes to integrate the inductive bias of CNN into Transformers
49
54 16]. There are also attempts to adapt ViT for other vision tasks (e.g., object detection,
semantic segmentation, etc.)
30 44
20
57 60
[4] The most majority of these concurrent

---

works represent each image with a fix number of tokens. To the best of our knowledge, we are the first consider configuring token numbers conditioned on the inputs.
Efficient deep networks. Computational efficiency plays a critical role in real-world scenarios, where
the executed computation translates into power consumption, carbon emission latency. A number
of works have been done on reducing the computational cost of CNNs [22 33 21
53 59 31
35
However, designing efficient vision Transformers is still an under-explored topic. TiT-ViT (55) proposes a light-weighted tokens-to-toke module and obtains a competitive accuracy-paramedt
trade-off compared to MobileNetV2 [33. LiV [16] accelerates the inference of Transformer models by involving convolutional layers. Swin Transformer [30] introduces an efficient shifted window-basee approach in multi- stage vision Transformers. Compared these models with fixed
computational graphs, the proposed DVT framework improves the efficiency by adaptively changing the architecture of the network on per-sample basis.
Dynamic models. Designing dynamic architectures is an effective approach for efficient deep learning (18]. In the context of recognition tasks, MSDNet and its variants [23] 52 [2] develop multi-classifier CNN architecture to perform early exiting for easy samples. Another type of dynamic CNNs skips redundant layers
45 or channels [29] conditioned on the inputs. Besides, 46 has been proposed for efficient image and video recognition. Although these works are related to DVT on the spirit of adaptive computation, they are developed based on CNN, while DVT is tailored for vision Transformers.
the spatial adaptive paradigm 15
Notably, our work may seem similar the multi-exit networks early-exiting. However, DVT differentiates itself from these existing works in several important aspects. For example, a major efficiency gain in both MSDNNet 23 and RANet [52] comes from the adaptive depth, namely processing "easier" samples using fewer layers within single network. In comparison, DVT cascades multiple Transformers to infer several entire networks test time, Besides, DVT does not leverage the multi-scale architecture or dense connection in MSDNet [23]. Compared with RANet [52] DVT introduces the adaptive token number and the relationship reuse mechanism tailored for VIT. In addition, IMTA (27) studies the training techniques of multi-exit models, which are actually complementary to DVT. Another difference is that all these networks adopt pure convolutional models.
23
52
27
47 from the lens of
23
52
27
47
3 Dynamic Vision Transformer
split each 2D image into ID tokens, while model their long range interaction with the self-attention mechanism (40). As aforementioned, to correctly recognize some "hard" images and achieve high accuracy, the number of tokens usually needs to be large, leading to the quadratically grown computational cost. However, "casier" images that make up the bulk of the datasets typically require far fewer tokens and much less costs (as shown in Table [.] Inspired by this observation, we propose a Dynamic Vision Transformers (DVT), aiming to improve the computational efficiency of Transformers via adaptively reducing the number of representative tokens for each input.
Vision Transformers 17
38 55
In specific, we propose deploy multiple Transformers trained with increasing number of tokens, such that one can sequentially activate them for each test image until obtaining a convincing prediction (e.g., with sufficient confidence). The computation is allocated unevenly across different samples for improving the overall efficiency. It is worth noting that, if all the Transformers are learned separately, the computation performed by upstream models will simply be abandoned once downstream Transformer is activated, resulting in considerable inefficiency. To alleviate this problem, we introduce the efficient feature and relationship reuse mechanisms.
3.1 Overview
Inference. We start by describing the inference procedure of DVT, which is shown in Figure For each test sample, we first coarsely represent it using a small number of 1D token embeddings. This can be achieved by either straightforwarddy flattening the split image patches [1] 17 or leveraging techniques like the tokens-to-tokes module (55). We infer a vision Transformer with these few tokens to obtain a quick prediction. This process enjoys high efficiency since the computational cost of Transformers grows quadratically with respect to token number. Then the prediction will be evaluated with certain criterion to determine whether is reliable enough to be retrieved immediately. this paper, early- termmnationn is performed when the model is sufficiently confident (details in Section 3.3
3

---

00000
Forward
Yes
Vision Transformer
Exit?
Oiiput
Relationship Reuse
Image
0..0000 0000000000
Classifiation
Yes
Vision Transformer
Exit?
Output
Relationship Reuse
Fenture
Reuse
No
Position Embelding
Vision Transformer
Exir?
Yes
Octput
Exit?
Termination
Criterion
Figure 2: An overview of Dynamic Vision Transformers (DVT), Under the objective of configuring proper token numbers conditioned on the inputs, we cascade multiple Transformers with increasing number of tokens. At test time, they sequentially activated until convincing prediction (c.g. sufficiently confident) has been obtained or the final model has been inferred. The feature and relationship reuse mechanisms allow reusing computation across different Transformers.
Once the prediction fails meet the termination criterion, the original input image will be split into more tokens for more accurate but computationally more expensive inference. Note that, here the dimension of each token embedding remains unchanged, while the number of tokens increases, enabling more fine-graine as the previous one but different parameters will be activated. By design, this stage trades off computation for higher aceuracy some "difficult" test samples. To improve the efficiency, the new model can reuse the previously learned features and relationships, which will be introduced in Section 3.2 Similarly, after obtaining new prediction, the termination criterion will be applied, and the above procedure will proceed until the sample exits or the final Transformer has been inferred.
representation. An additional Transformer with the same architecture
Training. For training, simply train DVT to produce correct predictions at all exits (i.e., each with the corresponding number of tokens). Formally, the optimization objective
minimize
Em
Σ, LCF(p..))
|Dirain
where (x, y) denote sample the training set Diain and its corresponding label. We adopt the standard cross-entropy loss function (CF(.), while p, denotes the softmax prediction probability output by the 7th exit. We find that such simple training objective works well in practice.
Transformer backbone. DVT is proposed as general and flexible framework. It can be built on top of most existing vision Transformers like ViT (I1), DeiT (38] and T2T-ViT (55) improve their efficiency. The architecture of Transformers simply follows the implementation of these backbones.
3.2 Feature and Relationship Reuse
An important challenge to develop our DVT approach is how to facilitate the reuse of computation. That is, once downstream Transformer with more tokens is inferred, it is obviously inefficient if the computation performed previous models abandoned. The upstream models, although being based smaller number of input tokens, are trained with the same objective, and have extracted valuable information for fulfilling the task. Therefore, we propose two mechanisms to reuse the learned deep features and self-attention relationships. Both of them are able improve the test accuracy significantly by involving minimal extra computational cost.
Background. For the ease of introduction, we first revisit the basic formulation of vision Trans- formers. The Transformer encoders consist of alternatively stacked multi-head self-attention (MSA) and multi-layer perceptron (MLP) blocks 440 [. The layer normalization (LN) [2] and residual

---

00000
FOOD POO
Vision Transformer
Upsample
LN
ML.P
Extract Context
fi+1
Y
Embedding
Flatten
0000000
Self colf-attenio
Multi-head
MLL
IN
Reshape
OC
Transformer Layer
Forward
Image
Classification
Position Embedding
Concateuation
Tokens
Token
Figure 3: Illustration of the feature reuse mechanism. A layer-wise context embedding is learned based on the final representations output by the upstream model, i.c., 21, MLP block of each downstream Transformer layer.
uP and integrated into the
connection [9 are applied before and after each block, respectively. Let 2! R denote the output of the 1th Transformer layer, where N is the number of tokens for each sample, and D is the dimension of each token. Note that N = HW + 1, which corresponds to H x W patches of original image and a single learnable classification token. Formally, we have
zi MSA ((-)) Zt-1,
E ( LE [1....,));
2y MLP(LN + 2/1
(3)
where L is the total number of layers in the Transformer. The classification token in 2,, will be fed into a LN layer followed by a fully-connected layer for the final prediction. For simplicity, here we omit the details on the position embedding, which is unrelated to our main idea. No modification is performed on addition to the configurations of backbones.
Feature reuse. All the Transformers in DVT share the same goal of extracting discriminative representations for accurate recognition. Therefore, is straightforward that downstream models should be learned on the basis of previously obtained deep features, rather than extracting features from scratch. The former more efficient since the computation performed an upstream model contributes to both itself and the successive models. To implement this idea, we propose feature reuse mechanism (see: Figure 3. specific, we leverage the image tokens output by the final layer of the upstream Transformer, i.e., zf, to learn layer-wise embedding E, for the downstream model:
(4) Herein, fr: IR NxP_PR NxD' consists of sequence of operations starting with LN-MLP (R PR which introduces nonlinearity and allows more flexible transformations. Then the image tokens are reshaped to the corresponding locations in the original image, upsampled and flattened to match the token number of the downstream model. Typically, we use small D for an efficient fi-
E (i("))) E BR
Consequently, the embedding E, injected into the downstream model, providing prior knowledge on recognizing the input image. Formally, we replace Eq. (3) by:
(5) where E, is concatenated with the intermediate tokens z/. We simply increase the dimension of LN and the first layer of MI.P from D D+ D Since E, is based the upstream outputs 2 that have less tokens than z/. it actually concludes the context information of the input image for each token in name E, the context embedding. Besides, do not reuse the classification token and pad zero for it in Eq. 55) which we empirically find beneficial for the performance. Intuitively, Eqs. (4) and (5) allow training the downstream model to flexibly exploit the information within = on per-layer basis, under the objective of minimizing the final recognition loss (Eq. (T). This feature reuse formulation can also be interpreted as implicitly enlarging the depth the model.
MLP(LN( (OLLN((Conat(/// E.)) + z/1
LE (1.....,
z/. Therefore,
5

---

Relationship reuse. A promi- nent advantage of vision Trans- formers is that their self-attention blocks enable integrating infor- mation across the entire im- age, which effectively models the long-range dependencies in the data. Typically, the models need to learn group of atten- tion maps at each layer to de- scribe the relationships among to- kens. Apart from the deep fea- tures mentioned above, the down- stream models also have access to the self-atte attentonnoo maps produced in previous models. We argue that these learned relationships are also capable of being reused facilitate the learning of down- stream Transformers.
Multi-head SelF-attentinn
Upstream Attention
Logits AuP
MLP
N
NH
2
Nup
Softmax
N up
Reshape
Flatten
NH
N
Nup
N-Att
N
N
up
Upsample
N
N
Figure 4: Illustration of the relationship reuse mechanism. We leverage the learned self-attention relationships from all upstream layers and attention heads, i.e., AuP, to refine the downstream attention maps. The addition operation of logits adopted. Note that N denotes the head number for multi-head self-attention.
Given the input representation 21, the self-attention is performed follows. First, the query, key and value matrices Qu K; and Vi are computed via linear projections:
Q = w K; = W V = WYWY
(6)
where w w and WY are weight matrices. Then the attention map is calculated by scaled dot-prode dot-product operation with softmax to aggregate the values of all tokens, namely
Attention 2,) Softmax(A.) A, =.Q.T//V
Here d is the hidden dimension of Q or K, and A, E N denotes the logits of the attention map. Note that omit the details on the multi-head attention mechanism for clarity, where A₁ may include multiple attention maps. Such simplification does not affect the description of our method.
For relationship reuse, first concatenate the attention logits produced by all layers of the upstream model (i.e., A E {1, (1,...,T.})
A Concat (AT, ..... AT)) ER NipooN,,pN.
where Nap and NAii denote the number tokens and all attention maps in the upstream model, respectively. Typically, we have N = L, where 2 is the number of heads for the multi-head attention and I is the number layers. Then the downstream Transformer learns attention maps by leveraging both its own tokens and A"P simultaneously. Formally, we replace Eq. ? by
(9) where (-) ( is a transformation network that integrates the information provided by A"P to refine the downstream attention logits A₁ The architecture of rz(+) is presented in Figure 4 which includes MLP for nonlinearity followed by an upsample operation to match the size of attention maps. For multi-head attention, the output dimension of the MLP will be set to the number of heads.
Attention 2x) = Softmax (A; + ry (A A: Q, /Vd,
Notably, Eq. (9) is a simple but flexible formulation. For one thing, each self-attention block in the downstream model has access to all the upstream attention heads both shallow and deep layers, and hence can be trained leverage multi-level relationship information on its own basis. For another, as the newly generated attention maps and the reused relationships are combined in logits, their relative importance can be auto- matically learned by adjusting the magnitude logits. It is also worth noting that the regular upsample operation cannot be directly applied
J'W"
Figure 5: An example for the upsam- ple operation in rz(-).
rz(-). To illustrate this issue, we take upsampling a HW x HW (H (H=l W = attention map to H'W' H'W" (H'=W"=33 for example in Figure 5 Since each of rows and columns corresponds to W image tokens, we reshape the or columns back to H x W, scale them to H' x W', and then flatten them to H' W' vectors.

---

3.3 Adaptive Inference
As aforementioned, the proposed DVT framework progressively increases the number of tokens for each test sample and performs early-termination, such that "easy" and "hard" images can be processed using varying tokens with uneven computational cost, improving the overall efficiency. Specifically, at the 1th exit that produces the softmax prediction p;, the largest entry of p:. i.e., max, Ptj (defined as confidence [23] 52 47), is compared with a threshold 7/.- If max, Pij 2 12. the inference will stop by adopting p, as the output. Otherwise, the image will be represented using more tokens to activate the downstream Transformer. We always adopt zero- threshold for the final Transformer.
The values of (m, n2....) are solved on the validation set. We assume budgeted batch elassification [33 setting, where DVT needs to recognize a set of samples Deal within given computational budget > 0. Let Acc(Dvat, {7/, 722.. .))) and FLOPs(Dval- {m N2, ...)) denote the accuracy and
computational cost on Dan when using the thresholds (m. ........ The optimal thresholds can be
obtained by solving the following optimization problem:
maximize ACe {1 (2:....).)))) Pa.
subject FLOP {m 122.. B.
(10)
Due to the non don-dffferentiabiiity. we solve this problem with the genetic algorithm
48 in this paper.
Experiments
In this section, we empirically validate the proposed DVT on ImageNet [ and CIFAR-10/100
(26). Ablation studies and visualization are presented on ImageNet to give a deeper understanding of our method. Code and pre-trained models based on PyTorch are available at /https:://ggiie con/bblackfeetther-waaggg Saipg//DDmmmiiii-m Transforrrre We also provide the implementa- tion using the MindSpore framework and the models trained on a cluster of Ascend AI processors at Trrassforrrrrrrrrrerrerrr MindSpore
https://gtt hts//gittuuut thub.
Datasets. (1) ImageNet is a 1,000-class dataset from ILSVRC2012 [ containing 1.2 million images
for training and 50,000 images for validation. (2) CIFAR-100100 datasets (26 contain 32x32 colored images in 10/100 classes. Both of them consist of 50,000 images for training and 10,000 images for testing. For all the three datasets, we adopt the same data pre-processing and data augmentation policy as [19 24 223. In addition, we solve the confidence thresholds stated in Section 3.3 on the training set, which we find achieves similar performance adopting cross-valdation.
Backbones. Our experiments are based on several sate-of-thhearr vision Transformers, namely 12T-ViT-12 [55), ViT-ViT-14 (53), and DeiT-small (wo distillation) [38] Unless otherwise specified, we deploy DVT with three exits, corresponding to representing the images as 7x7, 10x10 and 14x 14 tokeng] For fair comparisons, our implementation exploits the official code of the backbones, and adopts exactly the same training hyper-parameters. More training details can be found in Appendix A. The number of FLOPs is calculated using the fvcore toolkit provided by Facebook Al Research, which is also used in Detectron2 [50], PySlowFast [3]. and Classy Vision (I]
Implementation details. For feature reuse, the hidden size and output size of the MLP in fi(-) are set to 128 and 48. In relationship reuse, for implementation efficiency, we share the same hidden state across the MLPs of all rz(-), such that ry (A E {1...., L} can be obtained one time in concatenation by implementing a single large MLP, whose hidden size and output size are 3 L and L. Note that N is the head number of multi-heas attention and L is the layer number.
4.1 Main Results
Results on ImageNet are shown in Figures 6 and 7 where T2T- ViT [55 and DeiT 38 are im-
plemented as backbones respectively. As stated in Section 3.3 we vary the average computational budget, solve the confidence thresholds, and evaluate the corresponding validation accuracy. The performance of DVT is plotted in gray curves, with the best accuracy under each budget plotted in black curves. We also compare our method with several highly competitive basclines, i.c., TNT [17). LocalViT 28). Cross B} PVT 44 ViT [T] and ResNet [9]. It can be observed that DVT consistently reduces the computational cost of the backhones. For example, DVT achieves the 82.3% accuracy with 3.6x less FLOPs compared with the vanilla T2T- When the budget ranges among 0.5-2 GFLOPs, DVT has ~ ~.77-199x -19x less computation than T2T ViT with the same performance. Notably, our method can flexibly attain all the points on each curve by simply adjusting the values of confidence thresholds with single DVT.
3 Although 4x4 tokens also used an example in Section find starting with 7x7 is more efficient.
7

---

13
1.68
63
62-
80
DVT
actaraa
TIT-VI
1.77
TNT
LocalVir
76
1.7%
VioeVVI
75
PVT
BesNet
Vir
1.9%
72
GFLOPs //image
GTLOPs //image
Figure 6: Top-1 accuracy GFLOPs on ImageNet. DVT is implemented top of T2T-VIT-12/14.
Table 2: The practical speed of DVT.
Table 3: Performance of DVT on CIFAR-10/100.
Models
ImageNet (NVIDIA 2080Ti b2=128)
Models
CIFAR-10
CIFAR-100
Top 1 Acc.
Thruughput
Top-1 Acc.
GFLOPs
Top Acc.
GFLOPs
T2T-ViT-7
71.68%
1574 img/s
V2T-VTT-10
97.21
1.53
85.44%
1,53
DVT
78.48% (76.80%)
1574 ing/s
DVT
97.21%
0,50 (13.
85.45%
0.54 (12.8))
T2T-VI--0
75.15%
1286 img/s
12T-VIT-12
97.45%
1.78
86.23%
1.78
DVT
79.74% (14.59%)
1286 ing/s
DVT
97.46%
0.52 (13.4x)
86.26%
0.61 (12.9k)
T2T-VI-11
76.74%
1121 img/s
T2T-VTT-14
98.19%
4,80
89.10%
4,80
DVT
80.43% (13.69%)
1128 img/s
DVI
98.19%
0.77 (16.2x)
89.11%
1.62 (13.0x)
12T-VI-144
81.50%
619 img/s
T2T T2T VIT
98.43%
8.50
89.37%
8,50
DVT
81.50%
877 img/s (f1 (114)
DVT
98.43%
1.44 (15.9x)
89.38%
1.74 (J449%)
T2T-VIT-19
81.93%
382 img/s 666 img/s (1174x)
T2T-V7T-24
98.53%
13.69
89.62%
13.69
DVT
81.93%
98.53%
1.49 (19.2x)
89.63%
1.86 (17.4x)
Practical efficiency of DVT. We test the actual speed of DVT on a NVIDIA 2080Ti GPU under batch inference setting, where mini-batch of data fed into the model at a time. After inferring each Transformer, the samples that meet the early- parlyttermination criterion will exit, with the remaining images fed into the downstream Transformer. The results are presented in Table 2 DVT based on T2T- ViT- using 7x7 and 14x14 tokens, which find more efficient practice. All other implementation details remain unchanged. One can observe that DVT improves the accuracy of small models inference the large T2T- 27T-VVTT-4/19 models by 1.4-1.7x without sacrificing performance.
Here we adopt two-exit
by 3.7-6.8% with the same inference speed, while accelerates the
Results on CIFAR are presented in Table Following the common practice [1 resize the CIFAR images to 224x224, and fine-tune the T2T- VIT and DVT models Figure The official code and training configurations provided by 55 are utilized. We report the computational cost of DVT when it achieves the competitive performance with baselines. Our proposed method shown to consume ~399x less computation compared with T2T-VIT.
55
L
38
we
Table 4: Comparisons between DVT and three tate-offtthear methods for improving the compu- tational efficiency of vision Transformers, i.e., Data distillation (38, DynamieViT (32 and Patch slimming [36]. Both DVT and the baselines are implemented on top of DeiT-small. Since the computational cost of DVT can be adjusted online, match the FLOPs the accuracy of DVT with the baselines, respectively, to see the difference.
DeiT-small Data distillation
DynamicViT
Patch slimming
Top- Acc.
79.80%
81.20%
81.20%
81.67% %
79.30%
79.30%
80.40%
79.40%
79.40%
79.81%
GFLOPwimage 4.61
4.63
2.37
2.90
2.41
Comparisons with SOTA baselines. In Table 4 we compare DVT with several recently proposed approaches for facilitating efficient vision Transformers ImageNet. Both knowledge distillation (KD) (38) and patch pruning based methods [32 36) are considered. One can observe that DVT outperforms the baselines with similar FLOPs, has significantly lower computational cost with the same accuracy. Besides, one can expect that DVT is compatible with these techniques, i.c., DVT can also be trained with KD to further boost the accuracy or leverage patch pruning to reduce the FLOPs.

---

83
3.2%
82-
81
80-
77
accuracy (96)
70
78-
a00 accuracy
77-
70-
DVT
75-
73
DVT (w/l relationsiap reuse)
DeT
72-
DVT (w/o festure reuse)
73-
ResNes
DVT (w/o rews))
VIT
T2T-VI
72
0.5
2.5
CFLOPs/ Pimaage
GFLOPs/inage
Figure 7: Performance of DeiT-based DVT on Figure 8: Performance of the DVT based on T2T- ImageNet. DeiT-small used the backbone.
ViT-12 with and without the reuse mechanisms.
Comparisons with existing carly-exiting networks are pre-
Table 5: DVT v.s. existing multi-exit models with the same adap- tive inference strategy. The accuracy under each budget is reported.
sented in Table [ We adopt
Top-1
Acc.
the same adaptive strategy (see: Section (3.3) for all the base- lines. The Top-1 accuracy on ImageNet under different com- putational budgets is reported.
Networks
0.75G
1,00G
1.25C
1.50G
1.75G
2,00G
MSDNet 23
69.82%
71.24%
72.73%
73.66%
73.99%
74.20%
MTTA-M 27
70.84%
71.92%
73.41%
74.31%
74.64%
74.94%
RANet 52
70.48%
72.24%
73.57%
74.56%
75.02%
75.10%
DVT DVT (IZT-VIT-12)
73.70%
76.22%
77.89%
78.89 %
79.47% %
79.75%
One can observe that DVT significantly outperforms these baselines.
4.2 Ablation Study
Effectiveness of feature and rela- tionship reuse. We conduct exper- iments by ablating one or both of the reuse mechanisms. For a clear comparison, we first deactivate the aurl-terriiation. and report the accuracy and GFLOPs correspond- ing to each exit in Table 6 The three-exit DVT based on 12T-ViT- 12 is considered. One can observe that both the two reuse mechanisms are able to significantly boost the accuracy of DVT at the nd and 3rd exits with at most 6% additional computation, while they compatible with each other to further improve the performance. We also find that involving computation reusing slightly hurts the accuracy at the 1st exit, which may be attributed to the compro- mise made by the first Transformer for downstream models. However, adopted, this difference only results in trivial disadvantage when the computational budget is very small, as shown in Figure g DVT outperforms the bascline significantly most cases.
Table 6: Effects of feature (F) and relationship (R) reuse. The percentages in brackets denote the additional computation com- pared to baselines involved by the reuse mechanisms.
Reuse F R Top-1 Acc. GFLOPs
1 Exit (7x7)
2 Exit (10x10)
3dd Exit (14x14)
Top-1 Acc. GFLOPs
Top-1 Acc. GFLOPs
70.33%
0.47
73.54%
1.37
76.74%
3.15
/
69.42%
0.47
75.31%
(.4
79.21
(.31
69.03%
0.47
75.34%
(2.9%
78. 78.86%
3.34(0.08)
69.04%
0.47
75.65%
1.46
80.00 %
18)
the aarlyteermination is
Design choices for the reuse mechanisms. Here we study the design of the feature and relationship reuse mechanisms. For experimental efficiency, we consider two-exit DVT based T2T- using 7x7 and 10x10 tokens, while enlarge the batch size and the initial learning rate by 4 times. Such training setting slightly degrades the accuracy of DVT, but it is still reliable to reflect the difference between different design variants. We deactivate early-termination, and report the performance of each exit. Notably, as the FLOPS of 1 exit remain unchanged (i.e., 0.47G), do not present it.
We consider four variants of feature reuse in Table 7 (1) reusing features from the corresponding upstream layer instead of the final layer; (2) reusing classification token; (3) only performing feature reuse in the first layer of the downstream model; (4) removing the LN in fi(-). One can see that taking final tokens of the upstream model and reusing them in each downstream layer are both important.
Ablation results for relationship reuse are presented in Table We consider four variants as well: (1) only reusing the attention logits from the corresponding upstream layer; (2) only reusing the attention logits from the final upstream layer; (3) replacing the MLP by a linear layer in r((:); (4) adopting naive upsample operation instead of what is shown in Figure The results indicate that it is beneficial to enable each downstream layer to flexibly reuse all upstream attention logits. Besides, naive upsampling significantly hurts the performance.

---

Table 7: Ablation studies for feature reuse.
Table 8: Ablation studies for relationship reuse.
1" Exit (7x7)
2dd Exit (10x10)
1" Exit (7x7)
2" Exi: (10x
Ablation
Ablation
Top-1 Acc.
Top-1 Acc.
GFLOPs
Top-1 Acc.
Top-1 Acc.
GFLOPs
reuse
70.08%
73.61%
reuse
70.08%
73.61%
Layer wise Layer-wise feature reuse
69,84%
74.31%
1.43
Layer wise relationship reuse
69.63%
73.89%
1.38
Reuse classification token
69.79%
74.70%
1.43
Rease final-layer relationships
69.25%
74.31
Remove f(()).
69.33%
74.73%
1.38
MLP- Lineerr
69.20%
73.84%
1.38
Remove in f((-)
69.635
75.05%
1.42
Naive upsample
69.60%
73.34%
1.41
Ours
69, .4.4%
75.23%
1.43
Ours
69.50%
74.91
1.41
"Easy"
"Hard""
"Easy"
"Hard"
"Fasy"
"Hard"
"Eavy"
"Hard"
30-
DVT (127-111-12
70
CCH
10
GFLOPs (maage
Figure 10: Numbers of images ex- iting at different exits with varying Figure 9: Visualization of the "easy" and "hard" samples in DVT. computational budgets.
Cackatoo
Halloon
Baseball
Beer Bottle
Early-termination criterion. We vary the cri- terion for adaptive inference based on DVT (T2T-ViT-12) and report the accuracy under sev-
Table 9: Comparisons of aarlyttermination criteri-
ons. The accuracy under each budget is reported.
Top Acc.
eral computational budgets in Table Two vari- ants are considered: (1) adopting the entropy of softmax prediction to determine whether to exit [37]; : (2) performing random exiting with the same exit proportion as DVT. The simple but effective confidence-based criterion achieves better performance than both of them.
Ablation
0.75G
1,00G
1,25G
1,50G
Randomly Exit
70.19%
71.66%
72.61%
73.59%
Entropy based
73.41%
75.21%
77.08%
78.40%
Confidence-hased (ours)
73.70%
76.22%
77.89%
78.89%
Vision Transformers with adap- tive depth. We test applying the
Table 10: Effects of applying the adaptive depth mechanism to vision Transformers. The accuracy under each budget is reported.
adaptive depth paradigm (23 [52
VIT in Table 10 Two classi- fiers are uniformly added to the in-
Ablation
Top-1 Acc.
2,00G
2.50G
3.00G
3.50G
4.00G
termediate layers of 1ÝT-V1T-14 and the carly-exiting is performed with the three exits in the same way as DVT. One can observe that the layer-adaptive ViT achieves significantly lower accuracy than DVT using identical computational budgets. This inferior performance may be alleviated by improving the network architecture, as shown in 23
V/T-V with adaptive depth 76.09%
78.88%
79.46%
79.54%
79.55%
DVT (T2T-ViT-14)
79.24%
80.61 %
81.47%
82.10%
82.42%
4.3 Visualization
Figure 9 shows the images that are first correctly classified at the 1st and 3rd exits of the DVT (T2T- ViT-12). The former are recognized as "easy" samples, while the later are considered to be "hard". One can observe that "easy" samples usually depict the recognition objectives in clear and canonical poses and sufficiently large resolution. On the contrary, "hard" samples may contain complex scenes and non-typical poses or only include a small part of the objects, and require a finer representation using more tokens. Figure TO presents the numbers of images that exit at different exits when the computational budget increases. The plot shows that the accuracy of DVT significantly improved with more images exiting later, which is achieved by changing the confidence thresholds online.
5 Conclusion
In this paper, we sought to optimally configure a proper number of tokens for each individual image in vision Transformers, and hence proposed the Dynamic Vision Transformer (DVT) framework. DVT processes each test input by sequentially activating a cascade of Transformers using increasing numbers of tokens, until an appropriate token number reached (measured by the prediction confidence). We further introduce the feature and relationship reuse mechanisms to facilitate efficient computation reuse. Extensive experiments indicate that DVT significantly improves the computational efficiency on top of sate-of-theeart vision Transformers, both theoretically and empirically.
10

---

Acknowledgements
This work is supported in part by the National Science and Technology Major Project of the Ministry of Science and Technology of China under Grants 2018AAA0100701, the National Natural Science Foundation of China under Grants 61906106 and 62022048, and Huawei Technologies Ltd. In particular, we appreciate the valuable discussions with Chenghao Yang.
References
[1] A. Adcock, V. Reis, M. Singh, Z. Yan, L. van der Maaten, K. Zhang. S. Motwani, J. Guerin. N. Goyal. I. Misra, L. Gustafson, C. Changhan, and P. Goyal. Classy vision. https://githul 2019.
[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 16Xiv 2016.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alee Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, NeurlPS, volume 33, pages 1877- 1901. Curran Associates, Inc., 2020.
[4] Shijie Cao, Lingxiao Ma, Wencong Xiao, Chen Zhang, Yunxin Liu, Lintao Zhang, Lanshun Nie, and Zhi Yang. Seernet: Predicting convolutional neural network feature-map sparsity through low-bit quantization. In CVPR, pages 11216-11222 2019.
[5] Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. arXiv preprint arXiv:2103. 14899, 2021.
[6] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. arXiv preprins arXiv:2102.10882 2021.
[7] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPRW, pages 702-703, 2020.
[8] Stéphane d' Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv preprint arXiv:2/03. 10697, 2021.
[9] J. Deng. W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. /CML, pages 248-255, 2009.
[10] Jacob Devlin, Ming-Wal Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[12] Alaaeldin El-Nouby, Natalia Neverova, Ivan Laptev, and Hervé Jégou. Training vision trans- formers for image retrieval. arXiv preprint arXiv:2102.05644 2021.
[13] Haoqi Fan, Yanghao Li. Bo Xiong, Wan- Lo, and Christoph Feichtenhofer. Pyslowfast. bttps:///gh thub. http///ooooooeeeeeeeeeeeeeeoooee
2020.
[14] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. arXiv preprint 12X. 2021.
[15] Michael Figurnov, Maxwell D Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual networks. In CVPR, pages 1039-1048, 2017.

---

[16] Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. arXiv preprint arXiv:2 136, 2021.
[17] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. arXiv preprint arXiv:2 001 12, 2021.
[18] Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey. arXiv preprint D2Xiv:2102.04900 2021.
[19] Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[20] Shuting He, Hao Luo,, Pichao Wang, Fan Wang, Hao Li, and Wei Jiang. Transreid: Transformer- based object re-identification. arXiv preprint arXiv: 044.022,02,04378, 2021.
[21] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chiel Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, al. Searching for mobilenetv3, In ICCV, pages 1314-1324, 2019.
[22] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 1oxXii::704.000 04861,, 2017.
[23] Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q Weinberger. Multi-scale dense networks for resource efficient image classification. In ICLR, 2018.
[24] Gao Huang, Zhuang Liu, Geoff Pleiss, Laurens Van Der Maaten, and Kilian Weinberger. Convolutional networks with dense connectivity. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-1, 2019.
[25] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In ECCV, pages 646-661. Springer, 2016.
[26] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
[27] Hao Li, Hong Zhang, Xiaojuan Qi, Ruigang Yang. and Gao Huang. Improved techniques for training adaptive deep networks. In ICCV, pages 1891-1900, 2019.
(28) Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool. Localvit: Bringing locality to vision transformers. arXiv preprint arXiv:2/04.05707, 2021.
[29] Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In NeurlPS, pages 2181-2191, 2017.
[30] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14000 2021.
[31] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In ECCV, pages 116-131,20 2018.
[32] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dy- namievit: Efficient vision transformers with dynamic token sparsification. arXiv preprint 2xXv:22006.02334, 2021.
[33] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, pages
2018.
[34] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In CVPR, pages 2818-2826, 2016.
[35] Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In JCML, volume 97, pages 1105-61144 2019.
[36] Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming for efficient vision transformers. arXiv preprint arXiv:2106.02852 2021.
[37] Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsume Kung. Branchynet: Fast inference via carly exiting from deep neural networks. In ICPR, pages 2464-2469. IEEE, 2016.
12

---

[38] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012. /2877, 2020.
[39] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. arXiv preprint arXiv:2103. /7239, 2021.
[40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, Bengio, H. Wallach, R. Fergus, Vishwanathan, and Garnett, editors, NeurlPS, volume 30. Curran Associates, Inc., 2017.
[41] Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. In ECCV, pages 3-18, 2018.
[42] Thomas Verelst and Tinne Tuytelaars. Dynamic convolutions: Exploiting spatial sparsity for faster inference. In CVPR, pages 2320-2329, 2020.
[43] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
[44] Wenhai Wang, Enze Xie, Xiang Li, Deng- Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102. 12122, 2021.
[45] Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In ECCV, pages 409-424, 2018.
[46] Yulin Wang, Zhaoxi Chen, Haojun Jiang. Shiji Song. Yizeng Han, and Gao Huang. Adaptive focus for efficient video recognition. arXiv preprint Sttiv:2005.02445 2021.
[47] Yulin Wang, Kangchen Lv, Rui Huang, Shiji Song, Le Yang, and Gao Huang. Glance and focus: a dynamic approach reducing spatial redundancy image classification. In NeurIPS, 2020.
[48] Darrell Whitley. A genetic algorithm tutorial. Statistics and computing, 4(2):65-85, 1994.
[49]
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv: 103.. 15808, 2021.
[50]
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019.
https://gg wtt.b//.ww...cccoooorraarrraaaaa
[51] Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual networks. In CVPR, pages 8817-8826, 2018
[52] Le Yang. Yizeng Han, Xi Chen, Shiji Song, Jifeng Dai, and Gao Huang. Resolution adaptive networks for efficient inference. In CVPR, pages 2369-2378, 2020.
[53] Le Yang. Haojun Jiang, Ruojin Cai, Yulin Wang, Shiji Song. Gao Huang, and Qi Tian. Con- densenet v2: Sparse feature reactivation for deep networks. In CVPR, 2021.
[54] Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. arXiv preprint arXiv:2103. 1816, 2021.
[55] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101. 11986, 2021.
[56] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In 2019.
PICCV, pages
[57] Fangao Zeng, Bin Dong, Tiancai Wang, Cheng Chen, Xiangyu Zhang, and Yichen Wei. Motr: End-to-end multiple-object tracking with transformer. arXiv preprint arXiv:2105.00 002440055..77... 2021. mixup: Beyond
[58] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lapez-Paz.
empirical risk minimization. ICLR, 2018
[59] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, pages 6848-6856, 2018.
13

---

[60] Moju Zhao, Kei Okada, and Masayuki Inaba. Trtr: Visual tracking with transformer. arXiv preprint arXiv:2105.08 N.X.:22005.0381,, 2021.
[61] Zhun Zhong, Liang Zheng, Guoliang Kang. Shaozi Li, and Yi Yang. Random erasing data augmentation. In AAAI, volume 34, pages 13001-13008, 2020.
[62] Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103.11880
2021.
14


Reviews:
[{"review_id": "S3fijjfLujL", "reviewer": "Reviewer_TPUT", "summary": "This paper introduces an efficient inference strategy for vision transformers (ViTs). The motivation is that there are some “easy” images which can be classified using a few tokens. Based on the motivation, the paper proposes a cascading inference framework which do not proceed to next stage if the prediction confidence is enough high (i.e., early exit  strategy). For further efficiency of the cascading scheme, the previous stage’s features are fused with the next stage features. Experimental results show that the proposed method outperforms the baseline methods both in inference speed and accuracy. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- This paper is well-written and the motivation is clear. \n- Experiments are extensive with various datasets and plenty of ablation studies. \n\n\n### Concerns and questions\n- The proposed method provides technical contributions to accelerate vision transformers with an interesting motivation. However, the early-exiting is a  well-known strategy in CNN fields as the authors mentioned in the related work section ([21, 46, 24]). \n  - It would be nice to see in-detail comparisons against the existing multi-stage early-exiting strategies [21, 46, 24] in both empirical and conceptual ways. \n  - Could the previous methods [21, 46, 24] be naively adopted to vision transformer architectures? If not, discussing the reasons would be very interesting. \n- The proposed framework reuses the previous stage’s features to enhance the accuracy. The “reuse” mechanism is quite interesting, so it would be great if the paper provides a discussion about “reusing” the previous features to improve the inference speed of the next stages. \n\n### Post-rebuttal comments\n- The author's response (reply) has resolved most concerns I raised so that I will keep my initial rating (weak acceptance).\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "COsbywybCFA", "reviewer": "Reviewer_kQFQ", "summary": "The paper describes how to dynamically apply a cascade of increasingly expensive ViT models in order to save compute resources at inference time. The architecture reuses features and attention logits between tokens of cheaper models as context for the more expensive ones. The resulting models achieve very strong performance on imagenet against competitive baselines when normalized by inference time and compute (flops). Detailed ablations verify individual design choices.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n* strong inference efficiency gains by coarse to fine processing of an image\n* clearly written\n* detailed ablations\n* dynamic model that is able to adapt based on compute budget\n\n## Weaknesses\n\n* No training efficiency gains (yet), actually the proposed model is a bit more expensive because it involves training the entire cascade of models on every image (is this really necessary?). This fact should be made more explicit (!), as training efficiency is quite important, especially because ViTs have shown larger advantages in high-data regimes [1,2]. The paper typically talks about efficiency gains (in general), but should be more precise at times (e.g., table/figure captions, but also occasionally in text). \n* No comparison/discussion around other methods that speed up inference (e.g., distillation).\n* Only ImageNet.\n\n\n## Conclusion\n\nOverall I like the idea of coarse to fine processing a lot, and combining it with dynamic processing even more. It goes against the main stream application of pyramids that start processing on the most fine grained level and building up ever more complex features. This is a very wasteful process. I would have liked the paper even more if the authors would have made an effort into translating these ideas also into the training process, which I think would be way more interesting than inference speed-ups, as there are other solutions to this.\n\nI dislike the sole focus on ImageNet (which is unfortunately still common practice) because it is totally unclear to me how these speed ups would translate to other datasets or other tasks. It also makes me question how important some of the design choices are in general as ImageNet1k is fairly small by today's standards. I would urge the authors to at least include results for models pretrained on the freely available ImageNet 21k dataset.\n\nAnother important shortcoming of the paper is its lack of comparison to or at least discussion around related work in the space of inference speed-up, e.g. distillation methods.\n\n## Comments & Questions\n\n* capitalize everywhere -> Vision Transformers\n* DeiT-small (w/o distillation) --> ViT-small (It is a ViT model with specs introduced in the DEIT paper, but it is still just ViT).\n* Why concatenating the low res features before the MLP and why not simply concatenating the low res token reps with the higher res patch embeddings  passing them jointly as input to the transformer? This seems somewhat more elegant to me.\n\n[1] ViT. https://arxiv.org/pdf/2010.11929.pdf\n[2] CLIP. https://arxiv.org/pdf/2103.00020.pdf\n\n## UPDATE\n\nI stick with my initial judgement of the paper. I think it deserves acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "7xHXrlhWo6", "reviewer": "Reviewer_SzLL", "summary": "This paper proposes a dynamic transformer network that allocate different computational cost to process different images with different complexity. The key idea is to change the number of tokens to represent an images, and based on the output confidence, the model decides if it goes to the next stage of computation to get more confident results. This paper further proposes two \"reuse\" strategies to reuse the results from the previous stage. Experiments show this improves the performance. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n+ This paper successfully applied the mutl-stage (also called early exit, dynamic models, etc.) strategies to improve the efficiency of Vision Transformers. \n+ One unique contribution of this paper is that it tries to reuse computation from previous stages and use them to improve the performance. Unique to ViT, this paper proposes \"feature reuse\" and \"relationship reuse\". \n\nWeakness:\n- The early-exit idea has been widely adopted in CNN models. The high-level idea of this paper is not that different from early-exit, though it is now applied on transformers. The novelty of this work is therefore limited. \n\n- Since the computation of such multi-stage network is inherently serial, the latency of such networks may be slow even though the theoretical FLOPs are low. \n\n- Different samples can have different latencies, this can be problematic for applications with hard latency constraint. Also, the batch-processing of such method can be much more complicated.\n\nQuestion:\nHow is the actual throughput/latency measured?  I assume in this paper, it reported the \"average\" latency/throughput. Since the speed is measured on GPUs with batch processing, if one sample is more \"difficult\" and requires multiple stages of processing, how is the latency of this batch calculated? Assuming that a sample's latency is t_{b,i}, where b is the batch index and \"I\" is the index within the batch. The time it takes to finish inference of this batch should be max_i t_{b,i}, which means that even there are many simple (fast) samples, the inference time for this batch is determined by the slowest sample's processing time. I hope the author can clarify how their speed is measured, and how to avoid the problem I mentioned above. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No societal impact discussion needed, in my opinion. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers are positive about the paper (also after considering the authors' response and discussion). Here are the main points:

Pros:
1) Well motivated and well presented idea
2) Good inference speed improvements on ImageNet
3) Good ablation studies

Cons:
1) Somewhat related to existing early-exiting strategies
2) Doubts about practical gains in inference/training speed

The authors were successful at addressing the reviewers' concerns. Overall, the idea is clean, is presented well, and works well. Thus I recommend the acceptance of the paper.
Decision: Accept (Poster)