DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
Authors: Yongming Rao
Wenliang Zhao
Benlin Liu
Jiwen Lu
Jie Zhou
Cho-Jui Hsieh
Venue: NeurIPS 2021 Poster (2021)
Keywords: vision transformer
deep model acceleration

Abstract: Attention is sparse in vision transformers. We observe the final prediction in vision transformers is only based on a subset of most informative tokens, which is sufficient for accurate image recognition. Based on this observation, we propose a dynamic token sparsification framework to prune redundant tokens progressively and dynamically based on the input. Specifically, we devise a lightweight prediction module to estimate the importance score of each token given the current features. The module is added to different layers to prune redundant tokens hierarchically. To optimize the prediction module in an end-to-end manner, we propose an attention masking strategy to differentiably prune a token by blocking its interactions with other tokens. Benefiting from the nature of self-attention, the unstructured sparse tokens are still hardware friendly, which makes our framework easy to achieve actual speed-up. By hierarchically pruning 66% of the input tokens, our method greatly reduces 31% $\sim$ 37%  FLOPs and improves the throughput by over 40% while the drop of accuracy is within 0.5% for various vision transformers. Equipped with the dynamic token sparsification framework,  DynamicViT models can achieve very competitive complexity/accuracy trade-offs compared to state-of-the-art CNNs and vision transformers on ImageNet. Code is available at https://github.com/raoyongming/DynamicViT

DynamicVill ViT: Efficient Vision Transformers with Dynamic Token Sparsification
Yongming Raol
Wenliang Zhao'
Benlin Liu
Jiwen Lu
Jie Zhou'
Cho-Jui Hsieh
Tsinghua University
UCLA
3 University of Washington
Abstract
Attention is sparse in vision transformers. We observe the final prediction vision transformers is only based subset of most informative tokens, which is sufficient for accurate image recognition. Based on this observation, we propose a dynamic token sparsification framework prune redundant tokens progressively and dynamically based on the input. Specifically, we devise lightweight prediction module to estimate the importance score of each token given the current features. The module is added to different layers to prune redundant tokens hierarchically. To optimize the prediction module in an end-to-end manner, we propose an attention masking strategy to differentiably prune token by blocking its interactions with other tokens. Benefiting from the nature of self-attention the unstructured sparse tokens are still hardware friendly, which makes our framework easy to achieve actual speed-up. By hierarchically pruning 66% of the input tokens, our method greatly reduces 31 %1% ~ 37% FLOPs and improves the throughput by over 40% while the drop of accuracy within 0.5% for various vision transformers. Equipped with the dynamic token sparsification framework, DynamicViT models can achieve very competitive complexity/accuracy trade-offs compared to state-of-the-ar CNNs and vision transformers on ImageNet. Code is available at https: // thub. com/ raoyongming wew.ooggiinn/ynaniiiiiiiiii
Introduction
These years have witnessed the great progress computer vision brought by the evolution of CNN- type architectures [12, 18]. Some recent works start to replace CNN by using transformer for many vision tasks, like object detection [36, 20] and classification [25]. Just like what has been done the CNN-type architectures in the past few years, it is also desirable to accelerate the transformer-like models to make them more suitable for real-time applications.
One common practice for the acceleration of CNN-type networks to prune the filters that are of less importance. The way input processed by the vision transformer and its variants, i.e. splitting the input image into multiple independent patches, provides us another orthogonal way to introduce the sparsity for the acceleration. That is, we can prune the tokens of less importance in the input instance, given the fact that many tokens contribute very little to the final prediction. This is only possible for the transformer- models where the self-attention module can take the token sequence of variable length as input, and the unstructured pruned input will not affect the self self-attention module, while dropping a certain part of the pixels can really accelerate the convolution operation since the unstructured neighborhood used by convolution would make it difficult to accelerate through parallel computing. Since the hierarchical architecture CNNs with structural downsampling has improved model efficiency in various vision tasks, we hope to explore the unstructured and data deta-deeendent
"ooresponning author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

self
AU/OO
FFN
(a) Structural Downsampling
(b) Dynamic Token Sparsification
Attention Visualization
Figure 1: Illustration of main idea. CNN models usually leverage the structural downsam- pling strategy to build hierarchical architectures as shown (a). unstructured and data-dependent downsampling method in (b) can better exploit the sparsity in the input data. Thanks the nature of the self-attention operation, the unstructured token set is also easy to accelerate through parallel computing. (c) visualizes the impact of each spatial location
the final prediction in the DeiT-S model [25] using the visualization method proposed in [3]. These results demonstrate the final prediction in vision transformers only based on a subset of most informative tokens, which suggests a large proportion of tokens can be removed without hurting the performance.
downsampling strategy for vision transformers further leverage the advantages of self-attention (our experiments also show unstructured sparsification can lead to better performance for vision transformers compared to structural downsampling). The basic idea of our method is illustrated in Figure 1.
In this work, we propose to employ a lightweight prediction module to determine which tokens be pruned in a dynamic way, dubbed as Dynamic ViT. In particular, for each input instance, the prediction module produces customized binary decision mask to decide which tokens uninformative and need to be abandoned. This module is added to multiple layers of the vision transformer, such that the sparsification can be performed in hierarchical way as we gradually increase the amount of pruned tokens after each prediction module. Once a token is pruned after certain layer, it will not be ever used in the feed-forward procedure. The additional computational overhead introduced by this lightweight module is quite small, especially considering the computational overhead saved by eliminating the uninformative tokens.
This prediction module can be optimized jointly in an end-to-end manner together with the vision transformer backbone. To this end, two specialized strategies adopted. The first one is to adopt Gumbel-Softmal [15] to overcome the non-differentiable problem of sampling from a distribution so that is possible perform the end-to-end training. The second one about how apply this learned binary decision mask to prune the unnecessary tokens. Considering the number of zero elements in the binary decision mask is different for each instance, directly eliminating the uninformative tokens for each input instance during training will make parallel computing impossible. Moreover, this would also hinder the back proppapapgn for the prediction module, which needs to calculate the probability distribution of whether to keep the token even if it is finally climinated. Besides, directly wise idea since zero vectors will still affect
setting the abandoned tokens zero vectors is also the calculation of the attention matrix. Therefore, we propose strategy called attention masking where we drop the connection from abandoned tokens all other tokens in the attention matrix based on the binary decision mask. By doing we can overcome the difficulties described above. We also modify the original training objective of the vision transformer by adding term to constrain the proportion of pruned tokens after a certain layer. During the inference phase, we can directly abandon a fixed amount of tokens after certain layers for each input instance as we no longer need to consider whether the operation is differentiable, and this will greatly accelerate the inference.

---

We illustrate the effectiveness of our method on ImageNet using DeiT [25] and LV-ViT [16] as backbone. The experimental results demonstrate the competitive trade-off between speed and accuracy. In particular, by hierarchically pruning 66% of the input tokens, we can greatly reduce 31 % ~ 37% GFLOPs and improve the throughput by over 40% while the drop of accuracy is within 0.5% for all different vision transformers. Our Dynamic ViT demonstrates the possibility of exploiting the sparsity in space for the acceleration of transformer-like model. We expect our attempt to open a new path for future work on the acceleration of transformer-like models.
2 Related Work
Vision transformers. Transformer model is first widely studied in NLP community [26]. It proves the possibility to use self-attention to replace the recurrent neural networks and their variants. Recent progress has demonstrated the variants of transformers can also be a competitive alternative CNNs and achieve promising results on different vision tasks including image classification [8, 25, 20, 35, 23], object detection [2], semantic segmentation [34, 5] and 3D analysis [31, 33]. DETR [2] the first work to apply the transformer model to vision tasks. It formulates the object detection task as set prediction problem and follows the mecderr design in the transformer to generate a sequence of bounding boxes. VIT [8] is the first work to directly apply transformer architecture on non-overlapping image patches for the image classification task, and the whole framework contains no convolution operation. Compared to CNN-type models, ViT can achieve better performance with large-scale pre-training. is really preferred if the architecture can achieve the sate-of-the-ar without any pre-training. DeiT [25] proposes many training techniques so that we can train the convolution- free transformer only on ImageNetIK [7] and achieve better performance than VIT. LV- [16] further improves the performance by introducing new training objective called token labeling. Both ViT and its follow-ups split the input image into multiple independent image patches and transform these image patches into tokens for further process. This makes it feasible to incorporate the sparsity in space dimension for these transformer-lik models.
Model acceleration. Model acceleration techniques are important for the deployment of deep models on edge devices. There are many techniques can be used to accelerate the inference speed of deep model, including quantization [9, 27]. pruning [13, 22], low-rank factorization [30], knowledge distillation [14, 19] and on. There are also many works aims accelerating the inference speed of transformer models. For example, Tiny BERT [17] proposes a distillation method to accelerate the inference of transformer. Star- TransTranffommr [10] reduces quadratic space and time complexity to linear by replacing the fully connected structure with a star-shaped topology. However, all these works focus on NLP tasks, and few works explore the possibility of making use of the characteristic of vision tasks to accelerate vision transformer. Furthermore, the difference between the characteristics of Transformer and CNN also makes it possible to adopt another way for acceleration rather than the methods used for CNN acceleration like filter pruning [13), which removes non-critical or redundant neurons from a deep model. Our method aims at pruning the tokens of less importance instead of the neurons by exploiting the sparsity of informative image patches.
3 Dynamic Vision Transformers
3.1 Overview
The overall framework of our DynamicVIT is illustrated in Figure 2. Our DynamicViT consists of normal vision transformer as the backbone and several prediction modules. The backbone network can be implemented as a wide range of vision transformer (e.g.. ViT [8], DeiT [25], LV- [16]). The prediction modules are responsible for generating the probabilities of dropping/keeping the tokens. The token sparsification performed hierarchically through the whole network certain locations. For example, given a 12-layer transformer, we can conduct token sparsification before the 4th, 7th, and 10th blocks. During training, the prediction modules and the backbone network can be optimized in an end-to-end manner thanks to our newly devised attention masking strategy. During inference, we only need to select the most informative tokens according to a predefined pruning ratio and the scores computed by the prediction modules.
3

---

N₂x
sparsify
Patch Embedding
Self-Atention
Self Actention
FFN
FFN
features
decisions
ds token
prediction module
Figure 2:
The overall framework of the proposed approach. The proposed prediction module is inserted between the transformer blocks selectively prune less informative token conditioned features produced by the previous layer. By doing so, less tokens are processed the followed layers.
3.2 Hierarchical Token Sparsífication with Prediction Modules
An important characteristic of our Dynamic VIT is that the token sparsification is performed hierarchi- cally, Le., we gradually drop the uninformative tokens as the computation proceeds. achieve this, we maintain a binary decision mask D {0, 13.² to indicate whether to drop or keep each token, where N H/ is the number of patch embeddings². We initialize all elements in the decision mask to and update the mask progressively. The prediction modules take the current decision D and the tokens x E R NxC as input. We first project the tokens using an MLP:
local MT.P(x) TR. NxC
where C" can be smaller dimension and C" C/2 in our implementation. Similarly, can compute a global feature by:
global Agg Agg(MLL(xx).DD RC"
(2)
where Agg is the function which aggregate the information all the existing tokens and can be simply implemented as an average pooling:
E D.u, D.
Agg(u, u. D)
u NXCC
The local feature encodes the information of certain token while the global feature contains the context of the whole image, thus both them are informative. Therefore, we combine both the local and global features to obtain local- gooaall embeddings and feed them to another MLP to predict the probabilities drop/keep the tokens:
z,
|z! local
z
global
1 1
(4)
Ceftmax(MLPP(2))
Nx2
(5)
where Taoo denotes the probability of dropping the i-th token and π is the probability of keeping it. We then generate current decision D by sampling from and update D by
D - D.
(6)
where O is the Hadamard product, indicating that once token is dropped, it will never be used.
We omit the class token for simplicity, while in practice we always keep the class token (i.e., the decision
for class token is always "I").

---

3.3 End-to-end Optimization with Attention Masking
Although our target is perform token sparsification, we find non- non-trivial implement in practice during training. First, the sampling from to get binary decision mask D is is non-differettiable, which impedes the end-to-end training. To overcome this, we apply the Gumbel-Softmax tech- nique [15] to sample from the prohabilities π.:
D Gumbel-Soolmaxx(t)
(0.13",
where we use the index "1" because D represents the mask of the kept tokens. The output of Gumbel- Softmax is one-hot tensor, of which the expectation equals т exactly. Meanwhile, Gumbel-Softun differentiable thus makes possible for end-to-end training.
The second obstacle comes when try prune the tokens during training. The decision mask D is usually unstructured and the masks for different samples contain various numbers of I's. Therefore, simply discarding the tokens where D; = O would result in a non-uniform number of tokens for samples within batch, which makes it hard parallelize the computation. Thus, we must keep the number of tokens unchanged, while cut down the interactions between the pruned tokens and other tokens. We also find that merely zero-out the tokens to be dropped using the binary mask D is not feasible, because in the calculation of self-attention matrix [26]
QK
= Softmax
(8)
VC
the zeroed tokens will still influence other tokens through the Softmax operation. To this end, we devise strategy called attention masking which can totally eliminate the effects of the dropped tokens. Specifically, we compute the attention matrix by:
P QK" //C E R N
(9)
I, i=
Gi
1 154J N,
(10)
D i
exp (P
Á.j = E exp(Prs Gi
1 IEÚjSN
(11)
By Equation (10) we construct a graph where Gij = 1 means the j-th token will contribute to the update of the i-th token. Note that explicitly add self-loop each token to improve numerically stability. It is also easy to show the self-loop does not influence the results: if Dj 0, the j-th token will not contribute to any tokens other than itself. Equation (1I) computes the masked attention matrix A, which is equivalent to the attention matrix calculated by considering only the kept tokens but has a constant shape N x N during training.
3.4 Training and Inference
We now describe the training objectives of our DynamicViT. The training of Dynamic includes training the prediction modules such that they can produce favorable decisions and fine-tuning the backbone to make it adapt to token sparsification. Assuming we are dealing with a minibatch of B samples, we adopt the standard cross-entropy loss:
Leks = CrossEntropp (y.roooooyyyyyy)) y),
(12)
where y is the prediction of the Dynamic (after softmax) and y the ground truth.
To minimize the influence on performance caused by our token sparsification, we use the original backbone network as teacher model and hope the behavior of our DynamicViT close to the teacher model as possible. Specifically, we consider this constraint from two aspects. First, we make the finally remaining tokens of the Dynamic ViT close to the ones of the teacher model, which can be viewed as a kind of self-distillation:
Edistill
D
(t: t2
(13)
DB
B
N
Diel
b=1
5

---

Table Main results on ImageNet. We apply our method on three representative vision transform- ers: DeÎT-S, LV-VIT-S and LV-ViT-M DeÎT-S [25] is widely used vision transformer with the simple architecture. LV- and VVT-M [16] are the sate-offthheaa vision transformers. We report the top-1 classification accuracy, theoretical complexity in FLOPs and throughput for different ratio p. The throughput is measured on single NVIDIA RTX 3090 GPU with batch size fixed to 32.
Keeping Ratio at each stage
Base Model
Metrics
1.0
0.9
0.8
0.7
ImageNet Acc. (%)
79.8
79.8 (-0.0)
79.6 (-0.2)
79.3 (-0.5)
DeiT-S [25]
GFLOPs
4.6
(-14%)
3,4 (-27%)
2.9 (-37%)
Throughput (im/s)
1337.7
1524.8 (+14%)
1774.6 (433%)
2062.1 (+54%)
ImageNet Acc. (%)
83.3
83.3 (-0.0)
83.2 (-0.1)
83.0 (-0.3)
LV-VIT-S [16]
GFLOPs
6.6
5.8 (-12%)
5.1 (-22%)
4.6 (-31%)
Throughput (im/s)
993.3
1108.3 (+12%)
1255.6 (+26%)
1417.6 (+43%)
ImageNet Acc. (%)
84.0
83.9 (-0.1)
83.9 (-0.1)
83.8 (-0.2)
LV-VITM [16]
GFLOPs
12.7
11.1 (-13%)
9.6 (-24%)
8.5 (-33%)
Throughput (im/s)
589.5
688.5 (+17%)
791.2 (+34%)
888.2 (+50%)
where t, and t² denotes the i-th token after the last block of the Dynamic and the teacher model, respectively. a is the decision mask for the b-th sample at the s-th sparsification stage. Second, minimize the difference of the predictions between our Dynamic ViT and its teacher via the KL divergence:
LKL KT (ylly'),
(14)
where y' is the prediction of the teacher model.
Finally, we want to constrain the ratio of the kept tokens to a predefined value. Given a set of target ratios for S stages = [p()) p5 we utilize an MSE loss to supervise the prediction module:
B
Lrtio
pla)
Σ
D
(15)
BS
a=1
1=1
The full training objective is combination of the above objectives:
L Lela + AKLCKL + Cdistil + AratioCration
(16)
where we set AKL = 0.5, Adistill = 0.5, Aratio = 2 in all our experiments.
During inference, given the target ratio p. we can directly discard the less informative tokens via the probabilities produced by the prediction modules such that only exact m = tokens are kept at the s-th stage. Formally, for the s-th stage, let
Is argsort(TT..)
(17)
be the indices sorted by the keeping probabilities πT,,1, we can then keep the tokens of which the indices lie in -i... while discarding the others. In this way, our Dynamic prunes less informative tokens dynamically at runtime, thus can reduce the computational costs during inference.
4 Experimental Results
In this section, we will demonstrate the superiority of the proposed Dynamic ViT through extensive
experiments. all of our experiments, we fix the number of sparsification stages S = 3 and apply
the target keeping ratio p as geometric sequence [p, p², p²] where p ranges from (0. 1). During
training Dynamic models, we follow most of the training techniques used in DeiT [25]. We
use the treitreind vision transformer models to initialize the backbone models and jointly train the whole model for 30 epochs. We set the learning rate of the prediction module to hatch use 0.01 x learning rate for the backbone model. We fix the weights of the backbone models in the first 5 epochs. All of our models are trained on a single machine with GPUs. Other training setups and details can be found in the supplementary material.
1024 x 0.001 and

---

80
DyVT-33607
Del (DsT
DefT-5222
DWT:256
Imagehed Top-1 Aoc
Ryl TLV8805
CasTL to Seval
TTT-IT-D
DyVIT-29007
SPVTSmal GAF
Ragharr--
Imagjehar
SubnT
MTTMMaarr
10
CreatiiT
74
(D+TT
2.0
GFLS
GFLOPs
Figure 3: Model complexity (FLOPs) and top-1 ac- curacy trade-offs on ImageNet. We compare Dynam- ieVIT with the state-of of of-the-art art image classification models. Our models achieve better trade-off's com- pared the various vision transformers well carefully designed CNN models.
Figure Comparison our dynamic token sparsi- fication method with model width scaling. We train our DynamicViT based on DeiT models with embed- ding dimension varying from 192 384 and fix ratio p 0.7. We dynamic token sparsification is efficient commonly used model width scaling.
4.1
Main results
One of the most advantages of the DynamicViT that it can be applied to a wide range of vision transformer architectures to reduce the computational complexity with minor loss of performance. In Table summarize the main results on ImageNet [7] where evaluate our DynamicViT used three base models (DeiT-S [25], LV-ViT-S [16] and LV-VIT-M [16]). We report the top-1 accuracy, FLOPs, and the throughput under different keeping ratios p. Note that our token sparsification is performed hierarchically in three stages, there are only [N tokens left after the last stage. The throughput measured on a single NVIDIA RTX 3090 GPU with batch size fixed to 32. We demonstrate that our DynamicViT can reduce the computational costs by 31% ~ 37% and accelerate the inference runtime by 43% 54%, with the neglectable influence of performance (-0.2% -.5%)
4.2
Comparisons with the-state-of-the-arrs
In Table we compare the DynamicViT with the satt--of-the-art models image classification, including convolutional networks and transformer-like architectures. We use the Dynamic ViT with LV-ViT [16] as the base model and use the "/p" to indicate the keeping ratio. We observe that our Dynamic VIT exhibits favorable complexity/accuran trade-offs at all three complexity levels. Notably, we find our SynamieViT-LV-MN are two of the current state-of of-the-arts CNN architectures. This can also be shown clearer in Figure 3, where we plot the FLOPS -ccuurcy curve Dynamic series (where we use DyViT for short), along with other state-of-the-art models. We can also observe that Dynamic ViT can achieve better trade-offs than LV-VIT series, which strongly demonstrates the effectiveness of our method.
beats the EfficientNee-BB [24] and NFNet-FO [1], which
4.3 Analysis
VinamicViT for model scaling. The success of EfficientNet [24] shows that we can obtain model with better complexity/accuracy tradeoff's by scaling the model along different dimensions. While in vision transformers, the most commonly used method to scale the model is to change the number of channels, our Dynamic ViT provides another powerful tool perform token sparsification. We analysis this nice property of Dynamic ViT in Figure 4. First, we train several DeiT [25] models with the embedding dimension varying from 192 (DeiT-Ti) to 384 (DeiT-S). Second, we train our Dynamic VTT based on those models with the keeping ratio p = 0.7. We find that after performing token sparsification, the complexity of the model reduced to be similar to variant with smaller embedding dimension. Specifically, observe that by applying our DynamicVil to DeiT-256,
7

---

Table 2: Comparisons with the state-of-the-arts on ImageNet. We compare our Dynamic ViT models with state-of-the-ar image classifciation models with comparable FLOPs and number of parameters. We use the DynamicViT with LV- [16] as the base model and use the "/p" indicate the keeping ratio. We also include the results of LV-VIT models references.
Model
Params (M)
GFLOPs
Resolution
Top-1 Acc (%)
DeiT-S [25]
22.1
4.6
224
79.8
PVT-Small [28]
24.5
3.8
224
79.8
Coa' Mini [29]
10.0
6.8
224
80.8
CrossViT-S [4]
26.7
5.6
224
81.0
PVT-Medium [28]
44.2
6.7
224
81.2
Swin-T [20]
29.0
4.5
766
81.3
T2T-ViT-14 [32]
22.0
5.2
224
81.5
CPVT-Small-GAP [6]
23.0
817
81.5
CoaT-Lite Small [29]
20.0
4.0
224
81.9
LV-VIT-S [16]
26.2
6.6
224
83.3
Dynamic ViT-LL-/O.
26.9
3.7
224
82.0
Dynamic ViT-LV-S/0.7
26.9
4.6
224
83.0
RegNerY-8G [2]]
39.0
8.0
224
81.7
T2T-ViT-19 [32]
39.2
8.9
224
81.9
Swin-S [20]
50.0
8.7
224
83.0
EfficientNet-B5 [24]
30.0
9.9
456
83.6
NFNet-FO [1]
72.0
12.4
256
83.6
Dynamic DynamicViT-LL--MM ViT-LV-M/0.7
57.1
8.5
224
83.8
ViT-Base/16 [8]
86.6
17.6
224
77.9
DeiT-Base/16 [25]
86.6
17.6
224
81.8
Cross VTT-B [4]
104.7
21.2
224
82.2
T2T-VTT-24 [32]
64.1
14.1
224
82.3
TNT-B [11]
66.0
14.1
224
82,8
RegNerY-16G [21]
84.0
16.0
224
82.9
Swin-B [20]
88.0
15.4
224
83.3
LV-VIT-M [16]
55.8
12.7
224
84.0
DynamieViT-LL-MMN
57.1
9.6
224
83.9
obtain a model that has a comparable computational complexity to DeiT-Ti, but enjoys around 4.3% higher ImageNet top-1 accuracy.
Visualizations. To further investigate the behavior of DynamicVVT, we visualize the sparsification procedure in Figure 5. We show the original input image and the sparsification results after the three stages, where the masks represent the corresponding tokens are discarded. We find that through the hierarchically token sparsification, our Dynamic ViT can gradually drop the uninformative tokens and finally focus on the objects the images. This phenomenon also suggests that the Dynamic leads to better interpretability, i.e., it can locate the important parts in the image which contribute most to the classification step-by-step.
Besides the sample-wise visualization we have shown above, we are also interested in the statistical characteristics of the sparsification decisions, i.e., what kind of general patterns does the Dynamic ViT learn from the dataset? We then use the Din to generate the decisions for all the images in the ImageNet validation set and compute the keep probability of each token in all three stages, as shown in Figure 6. We average pool the probability maps into 7 x such that they can be visualized more easily. Unsurprisingly, we find the tokens in the middle of the image tend be kept, which reasonable because in most images the objects are located in the center. We can also find that the later stage generally has lower probabilities to kept, mainly because that the keeping ratio the stage is p², which decreases exponentially as increases.

---

stage
stage
input
stage
stage
Input
stage
stage
stage
Figure 5: Visualization of the progressively sparsified tokens. We show the original input image and the sparsification results after the three stages, where the masks represent the corresponding tokens are discarded. We see our method can gradually focus on the most representative regions in the image. This phenomenon suggests that the Dynamic has better interpretability.
Table 3: Effects of different losses. We pro- vide the results after removing the distillation loss and the KL loss.
713
Base Model
DefT-S
LVVIT-S
DyIamicVIT
79.3(-0.5)
83.0(-0.3)
w/o distill (Eq.13) 13)
79.3(-0.5)
82.7(-0.6)
w/o KI. (4q..44
79.2(-0.6)
82.9(-0.4)
w/o distill & KL
79.2(-0.6)
82.3(-0.8)
Figure 6: The keep probabilities of the tokens at each stage.
Effects of different losses. We show the effects of different losses in Table 3. We see the improve- ment brought by the distillation loss and the KL loss is not very significant, but it can consistently further boost the performance of various models.
Comparisons of different sparsification strategies. As illustrated in Figure 2, the dynamic token better than other
sparsification is unstructured. To discuss whether the dynamic sparsification strategies, we perform ablation experiments and the results are shown in Table 4. For the structural downsampling, we perform an average pooling with kernel size 2 x 2 after the sixth block of the baseline DeiT-S [25] model, which has similar FLOPs to our Dynamic The static token sparsification means that the sparsification decisions are not conditioned on the input tokens. We also compare method with other token removal methods like randomly removing tokens removing

---

Table 4: Comparisons of different sparsification strategies. We investigate different methods to select redundant tokens based on the DeiT-S model. We report the top-1 accuracy on ImageNet for different methods. We fix the complexity of the accelerated models to 2.9G FLOPs for fair comparisons.
(a) Dynamic sparsification
(b) Different redundant token
(c) Effects of number of sparsifica- tion stages.
static/structural downsampling.
moval methods.
Model
Acc. (%)
Model
Acc. (%)
Model
Acc. (%)
Structural
78.2 (-1.6)
Random
77.5 (-2.3)
Single-stage
77.4 (-2.4)
Static
73.4 (-6,4)
Attention
78.1 (-1.7)
Two-stage
79.2 (-0.6)
Dynamic
79.3 (-0.5)
Prediction
79.3 (-0.5)
Three-stage
79.3 (-0.5)
Table 5; Results on larger models. We apply our method to the model with larger width (i.e., DeiT--) and the model with larger input size (i.e., DeiT-S with 384 384 input).
(a) Results on DeiT-B.
(b) Results the 384 x 384 input.
Model
GFLOPs
Acc, (%)
Model
GFLOPs
Acc. (%)
DeiT-B
17.5
81.8
DeiT-S
15.5
81.6
DynamicViT-BB0.77 V/T-B/0.7
11.2 (-36%)
81.3 (-0.5)
DynamicViT-S/00.7
9.5 (-39%)
81.4 (-0.2)
DynamicViT-s/0.5
7.0 (-55%)
80.3 (-1.3)
tokens based the attention score of the class token. We find through the experiments that although other strategies have similar computational complexities, the proposed dynamic token sparsification method achieves the best accuracy. We also show that the progressive sparsification method is significantly better than one-stage sparsification.
Accelerating larger models. To show the effectiveness of our method on larger models, we apply our method to the model with larger width (i.e., DeiT-B) and models with larger input size (i.e., DeiT-S with 384 x 384 input). The results are presented Table We see our method also works well on the larger DeiT' model. The accuracy drop become less significant when apply our method to the model with larger feature maps. Notably, we can reduce the complexity of the DeiT-S model with 384 x 384 input by over 50% with only 1.3% accuracy drop.
Conclusion
In this work, we open a new path accelerate vision transformer by exploiting the sparsity of informative patches in the input image. For each input instance, our Dynamic model prunes the tokens of less importance from the lightweight prediction module, which fuses the local and global information containing in the tokens. The prediction module is added multiple layers such that the token pruming performed in hierarchical way. Gumbel-Softmal and attention masking techniques are also incorporated for the end-to-end training of the transformer model together with the prediction module. During the inference phase, our approach can greatly improves the efficiency by gradually pruning 66% of the input tokens, while the drop of accuracy is less than 0.5% for different transformer backbone. this paper, we focus on the image classification task. Extending our method to other scenarios like video classification and dense prediction tasks can interesting directions.
dynamic way according to the customized binary decision mask output
Acknowledgment
This work was supported in part by the National Key Research and Development Program of China under Grant 2017YFA0700802, in part by the National Natural Science Foundation of China under Grant 25603, Grant 61822603, Grant U1813218, Grant U1713214, in part by Beijing Academy of Artificial Intelligence (BAAI), in part by National Science Foundation under grant IIS-1901521 IIS- IIS-2008 173, I1S-2048280, and in part by a grant from the Institute for Guo Qiang, Tsinghua University.
10

---

References
[1] Andrew Brock, Soham De, Samuel Smith, and Karen Simonyan. pirh-perfoe large- scale image recognition without normalization. arXiv preprint arXiv:21022061171, 2021. 7,
8
[2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213-229, 2020.
[3] Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. arXiv preprini arXiv:2012.09883, 2020. 2
[4] Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. arXiv preprint arXiv:2 /4899, 2021. 8
[5] Bowen Cheng. Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. NeurIPS, 2021.
[6] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. arXiv preprint arXiv:2102. 10882, 2021.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255, 2009, 3,
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1192) 2020. 3,8
[9] Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional networks using vector quantization. arXiv preprint 1411:114 2014.
[10] Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. Star-transformer. arXiv preprint 190.0:190..09111, 2019.
[11] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. arXiv preprint arXiv:2103.00112 2021.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[13] Yihui Hc, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In ICCV, pages 1389-1397, 3017.
[14] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: 550.0253333333 2015. 3
[15] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax.
In ICLR, 2017. 2,5
[16] Zihang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Xiaojie Jin, Anran Wang, and Jiashi Feng. Token labeling: Training 85.5% top-1 accuracy vision transformer with 56m parameters on imagenet.
arXiv preprint arXiv:2104.10858, 2021. 3, 6, 7,
[17] Xiaoqi Jiao, Yichun Yin, Lifeng Shang. Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint (90i::/900.1033::0 2019. 3
[18] Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. NeurIPS, 15:1097-1105, 2012.
[19] Benlin Liu, Yongming Rao, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Metadistiller: Network self-boosting via meta-learned top-down distillation. In European Conference on Computer Vision, pages 694-709. Springer, 2020.

---

[20] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. 14030, 2021. 1,3,8 8
[21] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. CVPR, pages 10428-10436, 2020. 8
[22] Yongming Rao, Jiwen Lu, Lin, and Jie Zhou. Runtime network routing for efficient image classification. IEEE transactions pattern analysis and machine intelligence, 41(10):2291- 2304, 2018.
[23] Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification. In NeurlPS, 2021. 3
[24] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. ICML, pages 6105-6114, PMLR, 2019. 7,
[25] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012. 2020. 2, 3, 6, 7,8,1 8,
[26] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint 1at.:1700.07622 2017. 5,5
[27] Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quantization with mixed precision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8612-88888 8620,620,220 2019.
(28) Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbone for dense prediction without convolutions. arXiv preprint 2rXiv:2102.12122 2021. 8
[29] Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers. arXiv preprint arXiv:2104.06339) 2021. 8
[30] Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In CVPR, pages 7370-7379, 2017. 3
[31] Xumin Yu, Yongming Rao, Ziyi Wang, Zuyan Liu, Jiwen Lu, and Jie Zhou. Pointr: Diverse point cloud completion with geometry-aware transformers. In ICCV, 2021. 3
[32] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokene-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.11988 2021. 8
[33] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun. Point transformer. In ICCV, 2021.
[34] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng. Tao Xiang, Philip H.S. Torr, and Li Zhang. Rethinking semantic segmentation from a
perspective with transformers. In CVPR, 2021. 3
[35] Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang. Xiaochen Lian, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint 1rXiv:2103.11888 2021.
[36] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint 15X1v2001.00455, 2020.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5. In this paper, we only study the dynamic token sparsification problem on the image classification task. Extending our method to wider range of scenarios like video classification and dense prediction tasks can be interesting future directions.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We study a general framework for deep model acceleration. Our method is not for specific applications, which does not directly involve societal issues.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes]
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] We follow the common practice in previous works, where they didn't 't report the error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See our implementation details in the experiment part.
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "tMS-4ifA9Oa", "reviewer": "Reviewer_s33y", "summary": "This paper works on accelerating the inference of ViT models by dynamically dropping tokens. This is realized by learning a light-weighted importance mask prediction between ViT blocks. The mask prediction module contains a few MLP layers and runs in $O(N)$ time instead of $O(N^2)$ in the self-attention. The authors also identified and resolved a few technical challenges in practice: 1. differentiating into the hard sampling; 2. batching after pruning. Experiments show the proposed method improves throughput by 43% ~ 54% with 0.2 ~ 0.5 accuracy drop.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### strengths\n\n- The overall results of improving throughput by 43% ~ 54% with 0.2 ~ 0.5 accuracy drop is favorable, and is likely to be used in practice in future ViT-based models.\n\n - The technical contribution of using MLPs on each individual token together with a global pooling of tokens to predict importance score makes sense. This is more efficient than the straightforward baseline of using the attention mask as the pruning weight (it will be great if such a baseline comparison is provided in the paper).\n\n - I like the model scaling figure (Fig. 4). It shows the proposed method works under large backbones.\n\n - Code is provided in the supplement.\n\n### Weaknesses\n\n - While the overall throughput improvements are convincing, however I didn't understand where are the speed improvements from. In L143, the authors say in practice they don't actually remove the tokens due to batching. If this is the case, should the actually FLOPs keep unchanged? Is it the same for both training and testing? It seems that testing is also batched (Table. 1 caption). Please make sure to clarify this in the rebuttal.\n\n - The paper does not compare to other efficient ViT baselines. I know there are no neighbors directly close to this work, but some straightforward counterparts I have in mind are the O(n log n) transformers, or just directly pruning using the attention weight (e.g., the cross-attention weight of the class token). I will consider increasing my rating if such experiments are provided in the rebuttal and the results are interesting.\n\n - I don't quite like the distillation loss and KL loss. First, why do we need both distillation loss and KL loss? They seem pretty similar. It would be better if the author can provide ablations of both losses. Note this is NOT a required experiment in the rebuttal.\n\n - The distillation and Eq 9-11 seem to make the training much slower. Please report the training time (e.g., with/ without distillation) for reference. Much slower training time will NOT decrease my rating.\n\n - I don't get the problem of `the zeroed tokens will still influence other tokens through the Softmax operation` in L146. Would weight the zeroed token by negative infinite a simple solution, instead of using EQ 9-11?\n\n - The term `structural downsampling` in Fig. 1 is a bit confusing. From the context, it seems this just refers to pooling. `Structural downsampling` causes confusion to the term `structural pruning` in the network sliming literature, which refers to filtering the weights.\n\n  - Should $D$ in Eq. 2 be $\\hat{\\bf D}$?\n\n### Rating\n\n  Overall this is a good paper with favorable experimental results, novel technical contribution, and good application value. Most of my negative comments are about writing, and I hope the authors can clarify them in the rebuttal.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "bT7EGJ8zFzV", "reviewer": "Reviewer_z3xW", "summary": "This paper proposes a lightweight prediction module to estimate the importance score of each token in the vision transformer. The module is added to different layers to prune redundant tokens hierarchically. To optimize the prediction module in an end-to-end manner, an attention masking strategy is proposed to differentiably prune a token by blocking its interactions with other tokens. Experiments show that the proposed method can significantly reduce the computational cost with a small drop in accuracy.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think that it is an interesting idea to utilize the natural properties of the vision transformer for network pruning. Maybe it is the first paper in this topic. The experimental results are also pretty good. The writing is clear and professional. However, this paper is only evaluated for image classification. I’m wondering if the proposed pruning affects much for downstream vision tasks (e.g., object detection and semantic segmentation)?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "Three hours"}, "extra_text": {}}, {"review_id": "9Jv71pyFR-Y", "reviewer": "Reviewer_m39c", "summary": "A token sparsification method is proposed in this paper for speeding up ViT. Attention masking is added into the ViT framework with learnable dropping out via back-propagations.  Unessential tokens are gradually pruned via attention masking for inference acceleration. The modified ViT, named dynamicViT, maintains the major accuracy while reducing one third FLOPs.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main idea of this work is to develop an attention mask strategy for discarding unimportant tokens. The importance is validated by the attention mask which is learnable via gradient back-propagations. This design functions as a module that is integrated into ViTs. It splits one ViT into two parts. The first part maintains the original operations of ViT while the second part proceeds with reduced tokens.  During training, a knowledge distillation based method is introduced to train dynamicViT based on the original ViT.\n\nThe whole framework is to reduce the tokens while maintaining accuracy. This seems overall fine. There are a few concerns raised regarding the methodology, presentation, and experiments.\n\n1. In Fig. 2, there are $N_1$ blocks before sparsification and there are $N_2$ blocks afterward. It is not clear how $N_1$ and $N_2$ are set according to different transformers.  Moreover, it seems a tradeoff to set $N_1$ and $N_2$ for acceleration and accuracy.\n\n2. The proposed framework is posed as a knowledge distillation problem, which is specified in Sec. 3.4. It raises an interesting thinking that this approach aims at reducing tokens rather than reducing network parameters (e.g., linear projection parameters). It sounds like DeiT and DynamicViT function similarly (i.e., pruning network params v.s. pruning tokens). A discussion upon DeiT would be helpful to improve the motivation. \n\n3. In the experiments, dynamicViT only focuses on LV-VIT architecture. As the proposed method seems general, is it suitable for different transformers? If so, how? (see q1).\n\n4. The attention mask is learned by gradient back-propagation. How the computed loss will affect this mask to discard unimportant tokens. More explanation on this aspect would make the technique more convincing. For instance, how the mask looks like? What if the majority value of the mask is 0? Since the mask is influenced by the gradients, the values of the mask are sometimes not satisfactory. Have the authors considered any regularization upon that?\n\n5. Eq. 9-11 seems to overcomplicate the zero token issue. Reshaping the remaining tokens to make them compact is a way to ensure the following computations.\n\n----------------------------------------------post rebuttal---------------------------------------------\n\nWhile the proposed token sparsification seems not suitable for SWIN-based transformers where the tokens are densely organized for further inference. The authors have addressed most of the concerns. This reviewer turns positive on the current submission.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above. The negative societal impact is not discussed. The checklist is shown in the supplementary file, which does not follow the guideline that it must appear in the submitted PDF, immediately after references.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "99jrL93FUA", "reviewer": "Reviewer_b7CH", "summary": "Based on the observation that the final prediction in ViT is only decided by the most informative tokens, the author proposes DynamicViT which adopts dynamic token sparsification framework to prune redundant tokens progressively and hierarchically dependent on the input. To optimize the pruning in an end-to-end and differentiable way, the author proposes the attention masking strategy. The hierarchically pruning largely reduces the FLOPs and increases the throughout with the price of a very negligible performance drop for ViT, which demonstrates a very good complexity-accuracy trade-off achieved by the proposed DynamicViT.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1) The paper is generally well-written and easy to follow. The code is also provided for reproducibility.\n2) The DynamicViT model is inspired by a very interesting observation and the author conducts extensive experiments to confirm the correctness of the assumption.\n3) The proposed attention masking strategy enables the DynamicViT module to be trained in a differentiable end-to-end manner.\n4) DynamicViT achieves a very promising complexity-accuracy trade-off compared to CNNs and ViT approaches.\n\nWeaknesses:\n1) The hierarchically pruning of DynamicViT works well for the classification tasks. But it seems hard for DynamicViT to extend to object detection & semantic segmentation tasks, which limits the generalization and application scenarios of the proposed approach.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "6LA5BfVomA1", "reviewer": "Reviewer_Gaez", "summary": "The paper introduces a dynamic network to achieve efficient inference for vision transformers. Specifically, the proposed framework adopts a multi-stage architecture. In each stage, a learnable mask with the Gumbel-softmax training strategy is generated to remove some tokens for efficiency. The distillation loss and fine-tuning process are used for higher performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n+ Albeit the dynamic mechanism has been widely explored on Transformers in NLP tasks and CNN networks in Image classification tasks, introducing dynamic token sparsiﬁcation for Vision Transformers is new and interesting.\n+ The visualization and empirical analysis are clear and make sense.\n+ The paper is generally well written.\n\nWeakness:\n- The experimental comparison is not that fair. The proposed framework uses additional distillation losses on features and predictions but the baseline methods do not. According to the observations in DeiT[21], the distillation procedure is essential for classification accuracy. Although some ablation studies are provided in the supplementary material, the performance of the baseline framework with the distillation procedure has not been revealed.\n- Compared with the baseline methods, the DynamicViT adopts an additional fine-tuning procedure with 30 epochs. The fine-tuning procedure with distillation could significantly improve the performance of Vision Transformers, but the paper has not reported it.\n- Can the DynamicViT be trained in an end-to-end manner? How about the performance if it is trained from scratch?\n\nAfter Rebuttal:\nThe response addressed my concerns. I lean to vote for acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Only partially, more discussion is needed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "5kAx5mYldTN", "reviewer": "Reviewer_NNmU", "summary": "The intuition of this paper is that attention is sparse in vision transformers. To this end, this paper proposes a pruning method to estimate the importance score of each token before self-attention layer. An attention masking strategy is proposed to differentiably prune a token by blocking its interactions with other tokens. Experiments on DeiT and LV-ViT demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "No ablation studies on the loss functions\n\nAlthough it's a pruning strategy on the DeiT and LV-ViT, the proposed method is act as a downsampling operation (L120 use C′ = C/2 in our implementation).\nHowever, the baseline is only set to the model doesn't use the pruning method, so baseline comparison with the convolution based downsampling architecture is needed. Comparing with those method in Table 2 (eg PVT) is not enough. The authors need to design fair experiment setting to show the pruning downsampling is better than the convolution downsampling.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "see above", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers agree that the observations made by this submission are interesting and the proposed dynamic token sparsification method is novel and effective. Some concerns on the experiment details are raised by the reviewers but the authors address them well in the response. A clear acceptance. 
Decision: Accept (Poster)