DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
Authors: Yongming Rao
Wenliang Zhao
Benlin Liu
Jiwen Lu
Jie Zhou
Cho-Jui Hsieh
Venue: NeurIPS 2021 Poster (2021)
Keywords: vision transformer
deep model acceleration

Abstract: Attention is sparse in vision transformers. We observe the final prediction in vision transformers is only based on a subset of most informative tokens, which is sufficient for accurate image recognition. Based on this observation, we propose a dynamic token sparsification framework to prune redundant tokens progressively and dynamically based on the input. Specifically, we devise a lightweight prediction module to estimate the importance score of each token given the current features. The module is added to different layers to prune redundant tokens hierarchically. To optimize the prediction module in an end-to-end manner, we propose an attention masking strategy to differentiably prune a token by blocking its interactions with other tokens. Benefiting from the nature of self-attention, the unstructured sparse tokens are still hardware friendly, which makes our framework easy to achieve actual speed-up. By hierarchically pruning 66% of the input tokens, our method greatly reduces 31% $\sim$ 37%  FLOPs and improves the throughput by over 40% while the drop of accuracy is within 0.5% for various vision transformers. Equipped with the dynamic token sparsification framework,  DynamicViT models can achieve very competitive complexity/accuracy trade-offs compared to state-of-the-art CNNs and vision transformers on ImageNet. Code is available at https://github.com/raoyongming/DynamicViT

DynamicVill ViT: Efficient Vision Transformers with Dynamic Token Sparsification
Yongming Raol
Wenliang Zhao'
Benlin Liu
Jiwen Lu
Jie Zhou'
Cho-Jui Hsieh
Tsinghua University
UCLA
3 University of Washington
Abstract
Attention is sparse in vision transformers. We observe the final prediction vision transformers is only based subset of most informative tokens, which is sufficient for accurate image recognition. Based on this observation, we propose a dynamic token sparsification framework prune redundant tokens progressively and dynamically based on the input. Specifically, we devise lightweight prediction module to estimate the importance score of each token given the current features. The module is added to different layers to prune redundant tokens hierarchically. To optimize the prediction module in an end-to-end manner, we propose an attention masking strategy to differentiably prune token by blocking its interactions with other tokens. Benefiting from the nature of self-attention the unstructured sparse tokens are still hardware friendly, which makes our framework easy to achieve actual speed-up. By hierarchically pruning 66% of the input tokens, our method greatly reduces 31 %1% ~ 37% FLOPs and improves the throughput by over 40% while the drop of accuracy within 0.5% for various vision transformers. Equipped with the dynamic token sparsification framework, DynamicViT models can achieve very competitive complexity/accuracy trade-offs compared to state-of-the-ar CNNs and vision transformers on ImageNet. Code is available at https: // thub. com/ raoyongming wew.ooggiinn/ynaniiiiiiiiii
Introduction
These years have witnessed the great progress computer vision brought by the evolution of CNN- type architectures [12, 18]. Some recent works start to replace CNN by using transformer for many vision tasks, like object detection [36, 20] and classification [25]. Just like what has been done the CNN-type architectures in the past few years, it is also desirable to accelerate the transformer-like models to make them more suitable for real-time applications.
One common practice for the acceleration of CNN-type networks to prune the filters that are of less importance. The way input processed by the vision transformer and its variants, i.e. splitting the input image into multiple independent patches, provides us another orthogonal way to introduce the sparsity for the acceleration. That is, we can prune the tokens of less importance in the input instance, given the fact that many tokens contribute very little to the final prediction. This is only possible for the transformer- models where the self-attention module can take the token sequence of variable length as input, and the unstructured pruned input will not affect the self self-attention module, while dropping a certain part of the pixels can really accelerate the convolution operation since the unstructured neighborhood used by convolution would make it difficult to accelerate through parallel computing. Since the hierarchical architecture CNNs with structural downsampling has improved model efficiency in various vision tasks, we hope to explore the unstructured and data deta-deeendent
"ooresponning author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

self
AU/OO
FFN
(a) Structural Downsampling
(b) Dynamic Token Sparsification
Attention Visualization
Figure 1: Illustration of main idea. CNN models usually leverage the structural downsam- pling strategy to build hierarchical architectures as shown (a). unstructured and data-dependent downsampling method in (b) can better exploit the sparsity in the input data. Thanks the nature of the self-attention operation, the unstructured token set is also easy to accelerate through parallel computing. (c) visualizes the impact of each spatial location
the final prediction in the DeiT-S model [25] using the visualization method proposed in [3]. These results demonstrate the final prediction in vision transformers only based on a subset of most informative tokens, which suggests large proportion of tokens can be removed without hurting the performance.
downsampling strategy for vision transformers further leverage the advantages of self-attention (our experiments also show unstructured sparsification can lead to better performance for vision transformers compared to structural downsampling). The basic idea of our method is illustrated in Figure 1.
In this work, we propose to employ a lightweight prediction module to determine which tokens be pruned in a dynamic way, dubbed as Dynamic ViT. In particular, for each input instance, the prediction module produces customized binary decision mask to decide which tokens uninformative and need to be abandoned. This module is added to multiple layers of the vision transformer, such that the sparsification can be performed in hierarchical way as we gradually increase the amount of pruned tokens after each prediction module. Once a token is pruned after certain layer, it will not be ever used in the feed-forward procedure. The additional computational overhead introduced by this lightweight module is quite small, especially considering the computational overhead saved by eliminating the uninformative tokens.
This prediction module can be optimized jointly in an end-to-end manner together with the vision transformer backbone. To this end, two specialized strategies adopted. The first one is to adopt Gumbel-Softmal [15] to overcome the non-differentiable problem of sampling from a distribution so that is possible perform the end-to-end training. The second one about how apply this learned binary decision mask to prune the unnecessary tokens. Considering the number of zero elements in the binary decision mask is different for each instance, directly eliminating the uninformative tokens for each input instance during training will make parallel computing impossible. Moreover, this would also hinder the back proppapapgn for the prediction module, which needs to calculate the probability distribution of whether to keep the token even if it is finally climinated. Besides, directly wise idea since zero vectors will still affect
setting the abandoned tokens zero vectors is also the calculation of the attention matrix. Therefore, we propose strategy called attention masking where we drop the connection from abandoned tokens all other tokens in the attention matrix based on the binary decision mask. By doing we can overcome the difficulties described above. We also modify the original training objective of the vision transformer by adding term to constrain the proportion of pruned tokens after a certain layer. During the inference phase, we can directly abandon a fixed amount of tokens after certain layers for each input instance as we no longer need to consider whether the operation is differentiable, and this will greatly accelerate the inference.

---

We illustrate the effectiveness of our method on ImageNet using DeiT [25] and LV-ViT [16] as backbone. The experimental results demonstrate the competitive trade-off between speed and accuracy. In particular, by hierarchically pruning 66% of the input tokens, we can greatly reduce 31 % ~ 37% GFLOPs and improve the throughput by over 40% while the drop of accuracy is within 0.5% for all different vision transformers. Our Dynamic ViT demonstrates the possibility of exploiting the sparsity in space for the acceleration of transformer-like model. We expect our attempt to open a new path for future work on the acceleration of transformer-like models.
2 Related Work
Vision transformers. Transformer model is first widely studied in NLP community [26]. It proves the possibility to use self-attention to replace the recurrent neural networks and their variants. Recent progress has demonstrated the variants of transformers can also be a competitive alternative CNNs and achieve promising results on different vision tasks including image classification [8, 25, 20, 35, 23], object detection [2], semantic segmentation [34, 5] and 3D analysis [31, 33]. DETR [2] the first work to apply the transformer model to vision tasks. It formulates the object detection task as set prediction problem and follows the mecderr design in the transformer to generate a sequence of bounding boxes. VIT [8] is the first work to directly apply transformer architecture on non-overlapping image patches for the image classification task, and the whole framework contains no convolution operation. Compared to CNN-type models, ViT can achieve better performance with large-scale pre-training. is really preferred if the architecture can achieve the sate-of-the-ar without any pre-training. DeiT [25] proposes many training techniques so that we can train the convolution- free transformer only on ImageNetIK [7] and achieve better performance than VIT. LV- [16] further improves the performance by introducing new training objective called token labeling. Both ViT and its follow-ups split the input image into multiple independent image patches and transform these image patches into tokens for further process. This makes it feasible to incorporate the sparsity in space dimension for these transformer-lik models.
Model acceleration. Model acceleration techniques are important for the deployment of deep models on edge devices. There are many techniques can be used to accelerate the inference speed of deep model, including quantization [9, 27]. pruning [13, 22], low-rank factorization [30], knowledge distillation [14, 19] and on. There are also many works aims accelerating the inference speed of transformer models. For example, Tiny BERT [17] proposes a distillation method to accelerate the inference of transformer. Star- Tran-Tranffommr [10] reduces quadratic space and time complexity to linear by replacing the fully connected structure with a star-shaped topology. However, all these works focus on NLP tasks, and few works explore the possibility of making use of the characteristic of vision tasks to accelerate vision transformer. Furthermore, the difference between the characteristics of Transformer and CNN also makes it possible to adopt another way for acceleration rather than the methods used for CNN acceleration like filter pruning [13), which removes non-critical or redundant neurons from a deep model. Our method aims at pruning the tokens of less importance instead of the neurons by exploiting the sparsity of informative image patches.
3 Dynamic Vision Transformers
3.1 Overview
The overall framework of our DynamicVIT is illustrated in Figure 2. Our DynamicViT consists of normal vision transformer as the backbone and several prediction modules. The backbone network can be implemented as a wide range of vision transformer (e.g.. ViT [8], DeiT [25], LV- [16]). The prediction modules are responsible for generating the probabilities of dropping/keeping the tokens. The token sparsification performed hierarchically through the whole network certain locations. For example, given a 12-layer transformer, we can conduct token sparsification before the 4th, 7th, and 10th blocks. During training, the prediction modules and the backbone network can be optimized in an end-to-end manner thanks to our newly devised attention masking strategy. During inference, we only need to select the most informative tokens according to a predefined pruning ratio and the scores computed by the prediction modules.
3

---

N₂x
sparsify
Patch Embedding
Self-Atention
Self Actention
FFN
FFN
features
decisions
ds token
prediction module
Figure 2:
The overall framework of the proposed approach. The proposed prediction module is inserted between the transformer blocks selectively prune less informative token conditioned features produced by the previous layer. By doing so, less tokens are processed the followed layers.
3.2 Hierarchical Token Sparsífication with Prediction Modules
An important characteristic of our Dynamic VIT is that the token sparsification is performed hierarchi- cally, Le., we gradually drop the uninformative tokens as the computation proceeds. achieve this, we maintain a binary decision mask D {0, 13.² to indicate whether to drop or keep each token, where N H/ is the number of patch embeddings². We initialize all elements in the decision mask to and update the mask progressively. The prediction modules take the current decision D and the tokens x E R NxC as input. We first project the tokens using an MLP:
local MT.P(x) TR. NxC
where C" can be smaller dimension and C" C/2 in our implementation. Similarly, can compute a global feature by:
global Agg Agg(MLL(xx).DD RC"
(2)
where Agg is the function which aggregate the information all the existing tokens and can be simply implemented as an average pooling:
E D.u, D.
Agg(u, u. D)
u NXCC
The local feature encodes the information of certain token while the global feature contains the context of the whole image, thus both them are informative. Therefore, we combine both the local and global features to obtain local- gooaall embeddings and feed them to another MLP to predict the probabilities drop/keep the tokens:
z,
|z! local
z
global
1 1
(4)
Ceftmax(MLPP(2))
Nx2
(5)
where Taoo denotes the probability of dropping the i-th token and π is the probability of keeping it. We then generate current decision D by sampling from and update D by
D - D.
(6)
where O is the Hadamard product, indicating that once token is dropped, it will never be used.
We omit the class token for simplicity, while in practice we always keep the class token (i.e., the decision
for class token is always "I").

---

3.3 End-to-end Optimization with Attention Masking
Although our target is perform token sparsification, we find non- non-trivial implement in practice during training. First, the sampling from to get binary decision mask D is is non-differettiable, which impedes the end-to-end training. To overcome this, we apply the Gumbel-Softmax tech- nique [15] to sample from the prohabilities π.:
D Gumbel-Soolmaxx(t)
(0.13",
where we use the index "1" because D represents the mask of the kept tokens. The output of Gumbel- Softmax is one-hot tensor, of which the expectation equals т exactly. Meanwhile, Gumbel-Softun differentiable thus makes possible for end-to-end training.
The second obstacle comes when try prune the tokens during training. The decision mask D is usually unstructured and the masks for different samples contain various numbers of I's. Therefore, simply discarding the tokens where D; = O would result in a non-uniform number of tokens for samples within batch, which makes it hard parallelize the computation. Thus, we must keep the number of tokens unchanged, while cut down the interactions between the pruned tokens and other tokens. We also find that merely zero-out the tokens to be dropped using the binary mask D is not feasible, because in the calculation of self-attention matrix [26]
QK
= Softmax
(8)
VC
the zeroed tokens will still influence other tokens through the Softmax operation. To this end, we devise strategy called attention masking which can totally eliminate the effects of the dropped tokens. Specifically, we compute the attention matrix by:
P QK" //C E R N
(9)
I, D j. i
i=
Gi
1 154J N,
(10)
exp (P
Á.j = E exp(Prs Gi
1 IEÚjSN
(11)
By Equation (10) we construct a graph where Gij = 1 means the j-th token will contribute to the update of the i-th token. Note that explicitly add self-loop each token to improve numerically stability. It is also easy to show the self-loop does not influence the results: if Dj 0, the j-th token will not contribute to any tokens other than itself. Equation (1I) computes the masked attention matrix A, which is equivalent to the attention matrix calculated by considering only the kept tokens but has a constant shape N x N during training.
3.4 Training and Inference
We now describe the training objectives of our DynamicViT. The training of Dynamic includes training the prediction modules such that they can produce favorable decisions and fine-tuning the backbone to make it adapt to token sparsification. Assuming we are dealing with a minibatch of B samples, we adopt the standard cross-entropy loss:
Leks = CrossEntropp (y.roooooyyyyyy)) y),
(12)
where y is the prediction of the Dynamic (after softmax) and y the ground truth.
To minimize the influence on performance caused by our token sparsification, we use the original backbone network as teacher model and hope the behavior of our DynamicViT close to the teacher model as possible. Specifically, we consider this constraint from two aspects. First, we make the finally remaining tokens of the Dynamic ViT close to the ones of the teacher model, which can be viewed as a kind of self-distillation:
Edistill
D
(t: t2
(13)
DB
B
N
Diel
b=1
5

---

Table Main results on ImageNet. We apply our method on three representative vision transform- ers: DeÎT-S, LV-VIT-S and LV-ViT-M DeÎT-S [25] is widely used vision transformer with the simple architecture. LV- and VVT-M [16] are the sate-offthheaa vision transformers. We report the top-1 classification accuracy, theoretical complexity in FLOPs and throughput for different ratio p. The throughput is measured on single NVIDIA RTX 3090 GPU with batch size fixed to 32.
Keeping Ratio at each stage
Base Model
Metrics
1.0
0.9
0.8
0.7
ImageNet Acc. (%)
79.8
79.8 (-0.0)
79.6 (-0.2)
79.3 (-0.5)
DeiT-S [25]
GFLOPs
4.6
(-14%)
3,4 (-27%)
2.9 (-37%)
Throughput (im/s)
1337.7
1524.8 (+14%)
1774.6 (433%)
2062.1 (+54%)
ImageNet Acc. (%)
83.3
83.3 (-0.0)
83.2 (-0.1)
83.0 (-0.3)
LV-VIT-S [16]
GFLOPs
6.6
5.8 (-12%)
5.1 (-22%)
4.6 (-31%)
Throughput (im/s)
993.3
1108.3 (+12%)
1255.6 (+26%)
1417.6 (+43%)
ImageNet Acc. (%)
84.0
83.9 (-0.1)
83.9 (-0.1)
83.8 (-0.2)
LV-VITM [16]
GFLOPs
12.7
11.1 (-13%)
9.6 (-24%)
8.5 (-33%)
Throughput (im/s)
589.5
688.5 (+17%)
791.2 (+34%)
888.2 (+50%)
where t, and t² denotes the i-th token after the last block of the Dynamic and the teacher model, respectively. a is the decision mask for the b-th sample at the s-th sparsification stage. Second, minimize the difference of the predictions between our Dynamic ViT and its teacher via the KL divergence:
LKL KT (ylly'),
(14)
where y' is the prediction of the teacher model.
Finally, we want to constrain the ratio of the kept tokens to a predefined value. Given a set of target ratios for S stages = [p()) p5 we utilize an MSE loss to supervise the prediction module:
B
Lrtio
pla)
Σ
D
(15)
BS
a=1
1=1
The full training objective is combination of the above objectives:
L Lela + AKLCKL + Cdistil + AratioCration
(16)
where we set AKL = 0.5, Adistill = 0.5, Aratio = 2 in all our experiments.
During inference, given the target ratio p. we can directly discard the less informative tokens via the probabilities produced by the prediction modules such that only exact m = tokens are kept at the s-th stage. Formally, for the s-th stage, let
Is argsort(TT..)
(17)
be the indices sorted by the keeping probabilities πT,,1, we can then keep the tokens of which the indices lie in -i... while discarding the others. In this way, our Dynamic prunes less informative tokens dynamically at runtime, thus can reduce the computational costs during inference.
4 Experimental Results
In this section, we will demonstrate the superiority of the proposed Dynamic ViT through extensive
experiments. all of our experiments, we fix the number of sparsification stages S = 3 and apply
the target keeping ratio p as geometric sequence [p, p², p²] where p ranges from (0. 1). During
training Dynamic models, we follow most of the training techniques used in DeiT [25]. We
use the treitreind vision transformer models to initialize the backbone models and jointly train the whole model for 30 epochs. We set the learning rate of the prediction module to hatch use 0.01 x learning rate for the backbone model. We fix the weights of the backbone models in the first 5 epochs. All of our models are trained on a single machine with GPUs. Other training setups and details can be found in the supplementary material.
1024 x 0.001 and

---

80
DyVT-33607
Del (DsT
DefT-5222
DWT:256
Imagehed Top-1 Aoc
Ryl TLV8805
CasTL to Seval
TTT-IT-D
DyVIT-29007
SPVTSmal GAF
Imagjehar
SubnT
10
CreatiiT
74
(D+TT
2.0
GFLS
GFLOPs
Figure 3: Model complexity (FLOPs) and top-1 ac- curacy trade-offs on ImageNet. We compare Dynam- ieVIT with the state-of of of-the-art art image classification models. Our models achieve better trade-off's com- pared the various vision transformers well carefully designed CNN models.
Figure Comparison our dynamic token sparsi- fication method with model width scaling. We train our DynamicViT based on DeiT models with embed- ding dimension varying from 192 384 and fix ratio p 0.7. We dynamic token sparsification is efficient commonly used model width scaling.
4.1
Main results
One of the most advantages of the DynamicViT that it can be applied to a wide range of vision transformer architectures to reduce the computational complexity with minor loss of performance. In Table summarize the main results on ImageNet [7] where evaluate our DynamicViT used three base models (DeiT-S [25], LV-ViT-S [16] and LV-VIT-M [16]). We report the top-1 accuracy, FLOPs, and the throughput under different keeping ratios p. Note that our token sparsification is performed hierarchically in three stages, there are only [N tokens left after the last stage. The throughput measured on a single NVIDIA RTX 3090 GPU with batch size fixed to 32. We demonstrate that our DynamicViT can reduce the computational costs by 31% ~ 37% and accelerate the inference runtime by 43% 54%, with the neglectable influence of performance (-0.2% -.5%)
4.2
Comparisons with the-state-of-the-arrs
In Table we compare the DynamicViT with the sate--of-the-art models image classification, including convolutional networks and transformer-like architectures. We use the Dynamic ViT with LV-ViT [16] as the base model and use the "/p" to indicate the keeping ratio. We observe that our Dynamic VIT exhibits favorable complexity/accuran trade-offs at all three complexity levels. Notably, we find our SynamieViT-LV-MN are two of the current state-of of-the-arts CNN architectures. This can also be shown clearer in Figure 3, where we plot the FLOPS -ccuurcy curve Dynamic series (where we use DyViT for short), along with other state-of-the-art models. We can also observe that Dynamic ViT can achieve better trade-offs than LV-VIT series, which strongly demonstrates the effectiveness of our method.
beats the EfficientNee-BB [24] and NFNet-FO [1], which
4.3 Analysis
VinamicViT for model scaling. The success of EfficientNet [24] shows that we can obtain model with better complexity/accuracy tradeoff's by scaling the model along different dimensions. While in vision transformers, the most commonly used method to scale the model is to change the number of channels, our Dynamic ViT provides another powerful tool perform token sparsification. We analysis this nice property of Dynamic ViT in Figure 4. First, we train several DeiT [25] models with the embedding dimension varying from 192 (DeiT-Ti) to 384 (DeiT-S). Second, we train our Dynamic VTT based on those models with the keeping ratio p = 0.7. We find that after performing token sparsification, the complexity of the model reduced to be similar to variant with smaller embedding dimension. Specifically, observe that by applying our DynamicVil to DeiT-256,
7

---

Table 2: Comparisons with the state-of-the-arts on ImageNet. We compare our Dynamic ViT models with state-of-the-ar image classifciation models with comparable FLOPs and number of parameters. We use the DynamicViT with LV- [16] as the base model and use the "/p" indicate the keeping ratio. We also include the results of LV-VIT models references.
Model
Params (M)
GFLOPs
Resolution
Top-1 Acc (%)
DeiT-S [25]
22.1
4.6
224
79.8
PVT-Small [28]
24.5
3.8
224
79.8
Coa' Mini [29]
10.0
6.8
224
80.8
CrossViT-S [4]
26.7
5.6
224
81.0
PVT-Medium [28]
44.2
6.7
224
81.2
Swin-T [20]
29.0
4.5
766
81.3
T2T-ViT-14 [32]
22.0
5.2
224
81.5
CPVT-Small-GAP [6]
23.0
817
81.5
CoaT-Lite Small [29]
20.0
4.0
224
81.9
LV-VIT-S [16]
26.2
6.6
224
83.3
Dynamic ViT-LL-/O.
26.9
3.7
224
82.0
VIT-LV-VTT
26.9
4.6
224
83.0
RegNerY-8G [2]]
39.0
8.0
224
81.7
T2T-ViT-19 [32]
39.2
8.9
224
81.9
Swin-S [20]
50.0
8.7
224
83.0
EfficientNet-B5 [24]
30.0
9.9
456
83.6
NFNet-FO [1]
72.0
12.4
256
83.6
Dynamic DynamicViT-LL--MM ViT-LV-M/0.7
57.1
8.5
224
83.8
ViT-Base/16 [8]
86.6
17.6
224
77.9
DeiT-Base/16 [25]
86.6
17.6
224
81.8
Cross VTT-B [4]
104.7
21.2
224
82.2
T2T-VTT-24 [32]
64.1
14.1
224
82.3
TNT-B [11]
66.0
14.1
224
82,8
RegNerY-16G [21]
84.0
16.0
224
82.9
Swin-B [20]
88.0
15.4
224
83.3
LV-VIT-M [16]
55.8
12.7
224
84.0
DynamieViT-LL-MMN
57.1
9.6
224
83.9
obtain a model that has a comparable computational complexity to DeiT-Ti, but enjoys around 4.3% higher ImageNet top-1 accuracy.
Visualizations. To further investigate the behavior of DynamicVVT, we visualize the sparsification procedure in Figure 5. We show the original input image and the sparsification results after the three stages, where the masks represent the corresponding tokens are discarded. We find that through the hierarchically token sparsification, our Dynamic ViT can gradually drop the uninformative tokens and finally focus on the objects the images. This phenomenon also suggests that the Dynamic leads to better interpretability, i.e., it can locate the important parts in the image which contribute most to the classification step-by-step.
Besides the sample-wise visualization we have shown above, we are also interested in the statistical characteristics of the sparsification decisions, i.e., what kind of general patterns does the Dynamic ViT learn from the dataset? We then use the Din to generate the decisions for all the images in the ImageNet validation set and compute the keep probability of each token in all three stages, as shown in Figure 6. We average pool the probability maps into 7 x such that they can be visualized more easily. Unsurprisingly, we find the tokens in the middle of the image tend be kept, which reasonable because in most images the objects are located in the center. We can also find that the later stage generally has lower probabilities to kept, mainly because that the keeping ratio the stage is p², which decreases exponentially as increases.

---

stage
stage
input
stage
stage
Input
stage
stage
stage
Figure 5: Visualization of the progressively sparsified tokens. We show the original input image and the sparsification results after the three stages, where the masks represent the corresponding tokens are discarded. We see our method can gradually focus on the most representative regions in the image. This phenomenon suggests that the Dynamic has better interpretability.
Table 3: Effects of different losses. We pro- vide the results after removing the distillation loss and the KL loss.
713
Base Model
DefT-S
LVVIT-S
DyIamicVIT
79.3(-0.5)
83.0(-0.3)
w/o distill (Eq.13) 13)
79.3(-0.5)
82.7(-0.6)
w/o KI. (4q..44
79.2(-0.6)
82.9(-0.4)
w/o distill & KL
79.2(-0.6)
82.3(-0.8)
Figure 6: The keep probabilities of the tokens at each stage.
Effects of different losses. We show the effects of different losses in Table 3. We see the improve- ment brought by the distillation loss and the KL loss is not very significant, but it can consistently further boost the performance of various models.
Comparisons of different sparsification strategies. As illustrated in Figure 2, the dynamic token better than other
sparsification is unstructured. To discuss whether the dynamic sparsification strategies, we perform ablation experiments and the results are shown in Table 4. For the structural downsampling, we perform an average pooling with kernel size 2 x 2 after the sixth block of the baseline DeiT-S [25] model, which has similar FLOPs to our Dynamic The static token sparsification means that the sparsification decisions are not conditioned on the input tokens. We also compare method with other token removal methods like randomly removing tokens removing

---

Table 4: Comparisons of different sparsification strategies. We investigate different methods to select redundant tokens based on the DeiT-S model. We report the top-1 accuracy on ImageNet for different methods. We fix the complexity of the accelerated models to 2.9G FLOPs for fair comparisons.
(a) Dynamic sparsification
(b) Different redundant token
(c) Effects of number of sparsifica- tion stages.
static/structural downsampling.
moval methods.
Model
Acc. (%)
Model
Acc. (%)
Model
Acc. (%)
Structural
78.2 (-1.6)
Random
77.5 (-2.3)
Single-stage
77.4 (-2.4)
Static
73.4 (-6,4)
Attention
78.1 (-1.7)
Two-stage
79.2 (-0.6)
Dynamic
79.3 (-0.5)
Prediction
79.3 (-0.5)
Three-stage
79.3 (-0.5)
Table 5; Results on larger models. We apply our method to the model with larger width (i.e., DeiT--) and the model with larger input size (i.e., DeiT-S with 384 384 input).
(a) Results on DeiT-B.
(b) Results the 384 x 384 input.
Model
GFLOPs
Acc, (%)
Model
GFLOPs
Acc. (%)
DeiT-B
17.5
81.8
DeiT-S
15.5
81.6
DynamicViT-BB0.77 V/T-B/0.7
11.2 (-36%)
81.3 (-0.5)
DynamicViT-S/00.7
9.5 (-39%)
81.4 (-0.2)
DynamicViT-s/0.5
7.0 (-55%)
80.3 (-1.3)
tokens based the attention score of the class token. We find through the experiments that although other strategies have similar computational complexities, the proposed dynamic token sparsification method achieves the best accuracy. We also show that the progressive sparsification method is significantly better than one-stage sparsification.
Accelerating larger models. To show the effectiveness of our method on larger models, we apply our method to the model with larger width (i.e., DeiT-B) and models with larger input size (i.e., DeiT-S with 384 x 384 input). The results are presented Table We see our method also works well on the larger DeiT' model. The accuracy drop become less significant when apply our method to the model with larger feature maps. Notably, we can reduce the complexity of the DeiT-S model with 384 x 384 input by over 50% with only 1.3% accuracy drop.
Conclusion
In this work, we open a new path accelerate vision transformer by exploiting the sparsity of informative patches in the input image. For each input instance, our Dynamic model prunes the tokens of less importance from the lightweight prediction module, which fuses the local and global information containing in the tokens. The prediction module is added multiple layers such that the token pruming performed in hierarchical way. Gumbel-Softmal and attention masking techniques are also incorporated for the end-to-end training of the transformer model together with the prediction module. During the inference phase, our approach can greatly improves the efficiency by gradually pruning 66% of the input tokens, while the drop of accuracy is less than 0.5% for different transformer backbone. this paper, we focus on the image classification task. Extending our method to other scenarios like video classification and dense prediction tasks can interesting directions.
dynamic way according to the customized binary decision mask output
Acknowledgment
This work was supported in part by the National Key Research and Development Program of China under Grant 2017YFA0700802, in part by the National Natural Science Foundation of China under Grant 25603, Grant 61822603, Grant U1813218, Grant U1713214, in part by Beijing Academy of Artificial Intelligence (BAAI), in part by National Science Foundation under grant IIS-1901521 IIS- IIS-2008 173, I1S-2048280, and in part by a grant from the Institute for Guo Qiang, Tsinghua University.
10

---

References
[1] Andrew Brock, Soham De, Samuel Smith, and Karen Simonyan. pirh-perfoe large- scale image recognition without normalization. arXiv preprint arXiv:21022061171, 2021. 7,
8
[2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213-229, 2020.
[3] Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. arXiv preprini arXiv:2012.09883, 2020. 2
[4] Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. arXiv preprint arXiv:2 /4899, 2021. 8
[5] Bowen Cheng. Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. NeurIPS, 2021.
[6] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. arXiv preprint arXiv:2102. 10882, 2021.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255, 2009, 3,
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1192) 2020. 3,8
[9] Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional networks using vector quantization. arXiv preprint 1411:11412.611,, 2014.
[10] Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. Star-transformer. arXiv preprint 190.0:190..09111, 2019.
[11] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. arXiv preprint arXiv:2103.00112 2021.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[13] Yihui Hc, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In ICCV, pages 1389-1397, 3017.
[14] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: 550.0253333333 2015. 3
[15] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax.
In ICLR, 2017. 2,5
[16] Zihang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Xiaojie Jin, Anran Wang, and Jiashi Feng. Token labeling: Training 85.5% top-1 accuracy vision transformer with 56m parameters on imagenet.
arXiv preprint arXiv:2104.10858, 2021. 3, 6, 7,
[17] Xiaoqi Jiao, Yichun Yin, Lifeng Shang. Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint (90i::/900.1033::0 2019. 3
[18] Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. NeurIPS, 15:1097-1105, 2012.
[19] Benlin Liu, Yongming Rao, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Metadistiller: Network self-boosting via meta-learned top-down distillation. In European Conference on Computer Vision, pages 694-709. Springer, 2020.

---

[20] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. 14030, 2021. 1,3,8 8
[21] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. CVPR, pages 10428-10436, 2020. 8
[22] Yongming Rao, Jiwen Lu, Lin, and Jie Zhou. Runtime network routing for efficient image classification. IEEE transactions pattern analysis and machine intelligence, 41(10):2291- 2304, 2018.
[23] Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification. In NeurlPS, 2021. 3
[24] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. ICML, pages 6105-6114, PMLR, 2019. 7,
[25] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012. 2020. 2, 3, 6, 7,8,1 8,
[26] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint 1at.:1700.07622 2017. 5,5
[27] Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quantization with mixed precision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8612-88888 8620,620,220 2019.
(28) Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbone for dense prediction without convolutions. arXiv preprint 2rXiv:2102.12122 2021. 8
[29] Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers. arXiv preprint arXiv:2104.06339) 2021. 8
[30] Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In CVPR, pages 7370-7379, 2017. 3
[31] Xumin Yu, Yongming Rao, Ziyi Wang, Zuyan Liu, Jiwen Lu, and Jie Zhou. Pointr: Diverse point cloud completion with geometry-aware transformers. In ICCV, 2021. 3
[32] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokene-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.11988 2021. 8
[33] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun. Point transformer. In ICCV, 2021.
[34] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng. Tao Xiang, Philip H.S. Torr, and Li Zhang. Rethinking semantic segmentation from a
perspective with transformers. In CVPR, 2021. 3
[35] Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang. Xiaochen Lian, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint 1rXiv:2103.11888 2021.
[36] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint 15X1v2001.00455, 2020.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5. In this paper, we only study the dynamic token sparsification problem on the image classification task. Extending our method to wider range of scenarios like video classification and dense prediction tasks can be interesting future directions.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We study a general framework for deep model acceleration. Our method is not for specific applications, which does not directly involve societal issues.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] We follow the common practice in previous works, where they didn't 't report the error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See our implementation details in the experiment part.
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "d2I-o7e_ih", "reviewer": "Reviewer_yx6j", "summary": "This paper introduces a new method to accelerate vision transformer (ViT) architectures by dynamically pruning less informative tokens. To enable back-propagation during the training phase, this paper proposed an “attention masking” strategy which eliminates attention values for the pruned tokens. During the inference phase, the pruned tokens are hardly selected and it accelerates the inference speed about 30% against the vanilla baseline, with small accuracy drops (~0.5%).\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- This paper is well-written and the motivation was clearly explained.\n- This paper proposes an effective method to accelerate vision transformers with a simple and clever idea. \n\n### Concerns and questions\n- As shown in Figure 1, the final predictions in vision transformers seem to be related with a subset of most informative tokens. This observation is quite interesting and it would be nice to see further analysis for this phenomenon. For example, at what rate are the informative tokens in ImageNet images? \n- It also would be interesting to provide and discuss some wrongly classified images by the proposed method. Could the missing spatial tokens have an effect on the misclassification?\n- Could the token pruning strategies apply to language modeling transformers? If not, having discussions would be very interesting.\n- Minor comments\n  - In Eq (4), $z_i^{local}$ and $z_i^{global}$ are concatenated? What will be the dimension of $z_i$?, $R^{2c’}$? \n  - Could this method be combined with Swin Transformers? \n\n### Post-rebuttal comments\n- The author's response (reply) has resolved most concerns I raised. \n- After reading the other reviewer's comments, I will keep my initial rating for accepting this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The mentioned like, \"the proposed method is not for specific applications, which does not directly involve societal issues.\"\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "UQSSKzD45jG", "reviewer": "Reviewer_qFNP", "summary": "This paper introduces a dynamic prediction module to reduce the token-level redundancy, the dynamic Vit can achieve comparable performance with fewer Flops on ImageNet1k for Deit and LV-Vit.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths,\n\nthis paper is well-written and easy to follow.\n\nThe performance on ImageNet-1K is impressive.\n\nWeaknesses,\n\n1. lack of ablation study w/wo KD loss.\n\n2. why can the ** dynamic ** token sparsification have throughput gain in Table1(batch size =32) because Different images have different tokens \nremain and may result in parallel problems during the inference phase.\n\n3. why does the learning rate of the prediction module differ from the backbone.\n\n4. The model was initialized from a pre-trained model but compared with the models training from scratch.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "about 6 hours"}, "extra_text": {}}, {"review_id": "H5YVXbuoKv6", "reviewer": "Reviewer_9buE", "summary": "This paper presents an input-dependent dynamic token sparsification framework to prune redundant tokens progressively for efficient vision transformers. Specifically, authors utilize a lightweight prediction module to estimate the importance score of each token given the current features. Experiments show that DynamicViT models achieves competitive complexity/accuracy trade-offs compared to state-of-the-art CNNs and vision transformers on ImageNet.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is well written and easy to follow.\n- The idea of input dependent token sparsification for efficient vision transformer is interesting.\n- Experiments on ImageNet show that the proposed dynamic approach leads to reduction in FLOPs with only 0.5% drop in accuracy.\n\nWeaknesses:\n- The technical novelty of the paper is limited. The idea of using an additional lightweight network for input dependent pruning is very well demonstrated in CNNs. However, it has not been applied to vision transformers. Distillation is also not new for dynamic computation based approaches. So, I believe the main contribution of the paper is on application of such dynamic approach to make vision transformers more efficient. \n\n- How is this dynamic approach for improving efficiency of vision transformer is different from the early exit idea presented in BERT Loses Patience: Fast and Robust Inference with Early Exit? While the proposed approach focuses on data, the idea used in the other paper utilizes changing model depth for improving efficiency. I wonder can such similar ideas be applied to vision transformer for improving efficiency. \n\n- While dynamic pruning of filters or any dynamic computation based approach improves the accuracy while reducing FLOPs in CNNs, the proposed approach shows a drop in accuracy for vision transformers. I wonder what are the possible reasons for drop in accuracy in case of vision transformers?\n\n- Table 1 shows results on DeiT-S including two models based on LV-ViT. Since DeiT-B is more standard, I would recommend authors to include similar results on DeiT-B in addition to DeiT-S.\n\n- What is the effect of number of sparsification stages in the final performance? Ablation on different components of the loss should also be included in the experiments.\n\n- What is the effect of input resolution, e.g., 384x384 in the proposed approach? Does the higher resolution has more redundancy? Authors should perform experiments with different resolutions using DeiT-B on ImageNet. Moreover, can the attention maps generated in Figure 5 be applied to perform weakly supervised image segmentation? \n\n- What about using prediction module only at the beginning of the vision transformer? How is that simple baseline comparable to the proposed method? \n\n- How is the proposed method comparable to a random baseline that drops the same number of tokens as the proposed method? Also, how is the proposed method comparable to a baseline that drops tokens based on the attention map of the pretrained transformer? These experimental comparisons should be included to verify the effectiveness of the dynamic approach over these baselines.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- What are the limitations of the proposed approach? How these dynamic approach is hardware friendly in deployment? \n- While there is no specific negative societal impacts of the work, I recommend authors to include a brief discussion on the negative use case of efficient image classification systems, e.g., in surveillance. ", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "1yAcvnND06-", "reviewer": "Reviewer_mx5s", "summary": "The authors propose to accelerate the inference of vision transformers by sparsifying the tokens dynamically.  The proposed DynamicViT achieves over 40% runtime reduction with negligible accuracy degradation on a variety of vision transformer models.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed DynamicViT is a simple yet effective approach to accelerate the inference of vision transformers. The novelty of this paper is limited because spatial adaptive computing has been extensively studied in the context of convolutional neural networks (e.g., in [1-3]). The paper should discuss these related works more carefully. In my opinion, the contribution of this work lies mainly in the fact that it is one of the first works to apply spatial sparsity to vision transformers and to demonstrate the wall-clock speedup on GPU. The paper is well written and clearly articulates a contribution to the literature. The proposed method is sound as it uses typical treatments such as Gumbel-Softmax for non-differentiable problem and knowledge distillation to improve the accuracy. The experimental results also look promising, as DynamicViT can reduce the runtime by more than 40% with almost no accuracy degradation.\n\nMy main complain is that DynamicViT has only been compared with different transformer architectures. While little work has been done on efficient vision transformer,  both structural pruning (channel/filter pruning) and distillation techniques should be directly applicable to vision transformers. A comparison with some of these static counterparts is useful to help understand the benefits of dynamic sparsification. In addition, DynamicViT is coupled with knowledge distillation. Does the proposed method work without knowledge distillation? For the comparison with structural downsampling in Table 3 , a previous work called Funnel-Transformer [4] points out that “pool-query only” and maintaining a separate non-pooled [cls] token can bring a clear improvement to the\nnaive pooling method. It might be helpful to compare with this improved version of pooling. \n\n---\n[1] Figurnov et al, Spatially adaptive computation time for residual networks.\n\n[2] Verelst et al, Dynamic convolutions: Exploiting spatial sparsity for faster inference.\n\n[3] Wang et al, Glance and focus: a dynamic approach to reducing spatial redundancy in image classification.\n\n[4] Dai et al, Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a dynamic token pruning method for vision transformers. The main concerns raised by the reviewers were limited novelty (with respect to similar techniques that have been applied to CNNs), missing baselines / ablation studies, and unclear runtime speed gains. The author response included new experiments, additional baseline comparisons, and helped clarify several other questions, including why the proposed approach is hardware friendly for throughput gains. As final rating, three reviewers recommend acceptance, while one reviewer recommends rejection. The negative reviewer remained concerned about unfair comparisons and runtime gains, but the AC considers the author response to these issues was convincing. Overall, despite the incremental novelty of the proposed approach, the AC agrees with the majority that the obtained results are compelling and should be interesting to the NeurIPS audience. The authors should add the content of the rebuttal to the camera-ready version, proofread the paper, and improve the related work section (e.g., covering more methods that reduce the computation of neural networks through dynamic computation).
Decision: Accept (Poster)