VTC-LFC: Vision Transformer Compression with Low-Frequency Components
Authors: Zhenyu Wang
Hao Luo
Pichao WANG
Feng Ding
Fan Wang
Hao Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: ViT
compression
low-frequency

Abstract: Although Vision transformers (ViTs) have recently dominated many vision tasks, deploying ViT models on resource-limited devices remains a challenging problem. To address such a challenge, several methods have been proposed to compress ViTs. Most of them borrow experience in convolutional neural networks (CNNs) and mainly focus on the spatial domain. However, the compression only in the spatial domain suffers from a dramatic performance drop without fine-tuning and is not robust to noise, as the noise in the spatial domain can easily confuse the pruning criteria, leading to some parameters/channels being pruned incorrectly. Inspired by recent findings that self-attention is a low-pass filter and low-frequency signals/components are more informative to ViTs, this paper proposes compressing ViTs with low-frequency components. Two metrics named low-frequency sensitivity (LFS) and low-frequency energy (LFE) are proposed for better channel pruning and token pruning. Additionally, a bottom-up cascade pruning scheme is applied to compress different dimensions jointly. Extensive experiments demonstrate that the proposed method could save 40% ～ 60% of the FLOPs in ViTs, thus significantly increasing the throughput on practical devices with less than 1% performance drop on ImageNet-1K.

VTC-LFC: Vision Transformer Compression with
Low-Frequency Components
Zhenyu Wang
Hao Luo
Alibaba Group
Alibaba Group
daner www.allbaba-incc....... com
michuan. thCalibaba
wwwbbaaaaaaaa....eeeeaeeeeeeoeea inc. com
Pichao Wang
Feng Ding
Alibaba Group
Alibaba Group
pichao. wwwgwwawwwawwwaaaaaaaaaaaa com
dingfeng dinggenngaalibbbb--nnncc com
Fan Wang
Hao Li
Alibaba Group
Alibaba Group
fan. wwalibaba-innccom com
lihao. 1h@al www.lbibabannnn.... com
Abstract
Although Vision transformers (ViTs) have recently dominated many vision tasks, deploying ViT models lem. To address such a challenge, several methods have been proposed to com- press ViTs. Most of them borrow experience in convolutional neural networks (CNNs) and mainly focus on the spatial domain. However, the compression only in the spatial domain suffers from dramatic performance drop without fine- tuning and is not robust to noise, as the noise in the spatial domain can easily confuse the pruning criteria, leading to some parameters/channels being pruned incorrectly. Inspired by recent findings that self-attention is a low-pass filter and low-frequency signals/components are more informative to ViTs, this paper pro- poses compressing ViT' with low-frequency components. Two metrics named low-frequency sensitivity (LFS) and low-frequency energy (LFE) are proposed for better channel pruning and token pruning. Additionally, a bottom up cascade pruning scheme is applied to compress different dimensions jointly. Extensive experiments demonstrate that the proposed method could save 40% ~ 60% of the FL.OPs ViT's, thus significantly increasing the throughput on practical devices with less than 1% performance drop on ImageNet-IK. Code will be available at httpp:://githhn wwwc Daner-
snoource-limited devices remains challenging prob-
Seraner-Wann/ae Wang/VTe CTTC-LFC.PG LFC.git
Introduction
have outperformed convolutional neural networks (CNNs) in several vision tasks. However, ViT models still face the challenge of high computational cost when deployed to resource-limited devices. Following previous experiences in compressing CNN models, some pruning methods based on sparse learning [68 61. taylor expansion [60], or automatic searching [9 have been proposed for ViT models to reduce model redundancy via channel pruning. In addition to the redundancy in parameters, recent literature [88] 28 further points out that some noise tokens mainly encoded bisk-irrelevant information (e.g., background), and some tokens become similar deeper layers, showing that great redundancy also exists tokens.
Recently, Vision transformer (ViT) 14 and its variants [50 32 62
"Corresponding author. This work done when Zhenyu Wang was intern Alibaba.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

The mainstream works [28 47 (6) filter out the less informative tokens to reduce the FLOPs without changing the model structure.
Although aforementioned methods have made great progress in ViT compression in spatial domain, we find that they generally suffer from the following two prob- lems: (i) different from CNN pruning which maintains the performance well without finetuning F4 13,, dramatic performance drop is observed when the same method is applied in ViT pruning; (ii) conducting ViT pruning only
Acc1%) after pruning
spatial domain is not robust to noise, and as shown in Figure D after adding noise in the images, the accu- racy of spatial compression dramatically drops. To make ViT compression more effective and robust, we propose to conduct ViT compress with the help of frequency do-
Spatial domain pruning
330
6.0
0.0
530
Ratio of S&P noise ges
Figure 1: Noise resistance of spatial do- main pruning and our pruning. 'S&P' means salt-and-pepper noise, the pruned model is Deil-SSmal, and the perfor- mance evaluated on ImageNet-Il lk.
main. Recent studies 3
55
42
38 52 have indicated
that self-attention (SA) behaves like a low-pass filter, and low-frequency signals/components are more informative to VIT models. Inspired by such low-frreqe www-frequency character- isties of ViT, we propose compression framework named Vision Transformer Compression with Low-Frequency Components (VTC-LFC) which solves the problem from a new angle and emphasizes the contributions of low-freeg fow-frequency components during compression. To our best knowledge, this is the first work that compresses vision transformers the frequency domain. The main contributions of this paper are listed as below:
Channel pruning based on low-frequency sensitivity: Channel pruning a popular structured pruning strategy that aims to remove redundant parameters in fully connected layers of VIT. The mainstream works use some evaluation metrics (e.g. Taylor scores (60), weights norm [6]] or sparse factor (68] to estimate importance scores of parameters. Recent studies B 42 [2 find that ViTs are more reliable to the low- low-frequency components in images, i.e. the low-frequency information is more important for ViTs. Therefore, infer that channels that are less effective in encoding low-frequency components will contribute less to the feature representation for ViT models. Motivated by such a property, propose better channel pruning criterion named low-frequency sensitivity (LFS) based on the Taylor scores [60]. Different from the standard Taylor scores which are computed with the original images, LFS filters high-frequency components from images and uses only low-frequency components to estimate the importance of model parameters. In this way, channels that efficiently encode low-frequency information more likely to be preserved, and the compressed model tends to be more robust to noise. Experimental results show that LFS can alleviate the performance drop after compression without bells and whistles. Token pruning based on low-frequency energy: Token compression/sampling The popular methods dynamically select those tokens with high correlation to other tokens (e.g. the CLS token) as the informative tokens. However, it may be sub-optimal because the selected tokens tend to be similar each other, and the information included in the token itself has been neglected to some extent. As pointed out by [55] 38), the self-attention module in ViTs behaves like low-pass filter, i.e. the tokens with more low-frequency components can pass more information to the next layers. Inspired by this, we propose improving the attention-based token selection with an extra item, token low-frequency energy (LFE), which quantifies the low-frequency information in tokens. By correcting the attention scores with LFE of tokens, the selector can better distinguish informative tokens from both the long-term dependency in spatial domain and low-frequency contributions in frequency domain.
aims to select the informative tokens that store more useful information.
Bottom-up Cascade Pruning Framework: To jointly compress channels and tokens of vision transformers, we propose a bottom-up cascade pruning framework. The model accuracy is further preserved through automatically balancing compression ratios of channel pruning and token pruning block-by- block-by-block.
Related work
NL., the Vision Transformer (VIT) [4] is proposed to encode an image into sequence of tokens and feed them into the pure
Vision Transformer. Inspired by the success of transformers [11

---

transformer architecture. Several studies have shown that ViT performs better than convolution neural networks (CNN) on image classification benchmarks
20
when sufficient training data
66 have also been proposed. For example, DeiT [50] introduces a distillation token structure into ViT, and LV-ViT [24] proposes the token labeling approach for better training of ViT. In addition to image classification, ViT has also achieved great performance in many other computer vision applications, such as semantic segmentation 111 56, [2), image retrieval [22 [7], object detection 12 69] and image
is provided. Many follow-up variants of VIT 4 1
19
54
10
62
53
reconstruction 59]. However, despite of the outstanding performance in a series of tasks, its high computational cost restricts the deployment of ViT, which motivates the study of lightweight ViT models, including pruning [9 68 44 58), block-weights sharing [26, (4), fast distillation [57], and
dynamic prediction architecture [67 45 among which pruning is universal approach for almost all
model structures.
VIT Pruning. As an efficient compression approach, pruning 35 23 29
33 31 has been widely applied on various convolutional neural networks (CNNs) in computer vision. Pruning approaches [63 21 58 48) have also been proposed for ViT to reduce its model size and inference
time. These methods can be roughly grouped into two categories: 1) Channel pruning, which reduces the number of weights, channels, heads or blocks in VIT. SVITE [9] jointly optimizes parameters and explores connectivity for both unstructured pruning (zeroing weights) and structured pruning (removing heads and channels). ViT ViT-Slim [6] applies L1 sparsity on channels and produces compressed VITs with unstructured heads (the shape of heads is different). UVC [6]] drops heads, channels, and blocks in unified framework to achieve a high compression ratio. VTP [68] transfers the sparse-learning scheme in CNN pruning to compress ViT. NViT (60) generates smaller networks from the DeiT-base with the Taylor-based pruning scheme. 2) Token pruning. which focuses on dynamically selecting significant tokens for different inputs. Token pruning would significantly reduce the computational cost while maintaining all parameters. TokenLearner [44] adaptively generates small set of token vectors according to the spatial attention. EViT
28 downsamples tokens every
three blocks and selects tokens with high correlation with the CLS token. DynamicViT 41 estimates the importance of tokens with an MLP (5I) based predictor. IA-RED2 [6] introduces a multi-head interpreter drop uninformative tokens. SP-ViT (25 softly prunes tokens with token selector modules and packages the redundant tokens into one. Different from previous methods, this paper compresses ViTs from novel prospect, frequency domain, to prune both parameters and tokens in a unified framework.
52 of ViTs have indicated that ViTs behave in an opposite way to CNNs in frequency domain. 3 finds out that ViTs perform better than CNNs when only low-frequency components of images are fed into the models, and proposes the HAT method to enhance the capability of ViTs in capturing figh-frequency information. (55) analyzes ViT features from the Fourier spectrum domain and shows that the self- attention module amounts to low-pass filter. [38] also demonstrates that multi-head self self-attentions exhibit opposite behaviors to convolutions. and take advantage of both mechanisms to design a novel AlterNet. To summarize, all these studies point out that the low-frequency components play an important role in information extraction of VIT.
Frequency domain analysis for VIT and CNN. The recent explorations B 55
42
38
3 Methodology
3.1
Preliminary
The necessary notations are defined as below. As shown in Figure 2 a transformer block contains a multi-heaa self-attention (MHSA) module with multiple heads and ferd-forward network (FFN) module with two dully-connection layers. The input images in a mini- batch are denoted as X e RBx3x xxW where B, W and H are the batch size, width and height of images, respectively. The D and X e BBxN'xD
inputs of MHSA and FFN in the 1-th block are denoted as x E RB XDD respectively. ' is the number of tokens, and D is the dimension of a token. In the I-th block, the linear projection matrices W.l.b. W.'hh, and With are used to calculate Qt. (query), KI, (key), and VL, (value) for the h-th attention head. The parameters of the linear projection module in MHSA are denoted as W proj- and two linear projection matrices in FFN are W and W Our goal is to reduce the channel number of linear projection matrices and the token number N!.
3

---

Multi-Head Self Artention (MHSA)
Feed-Forward Network (PFN)
heads
QLa
Meeeeo Worm
Ki
Concat
W
f/
Tppt
ViJa
ViT Block
x12
Channel Pruning
x12
Token Pruning
LFE
Block-l
PHd
FFT
Attention Score
LFS
(Loss
KL)
Original tokens
Less informative tokens
Informative tokens
CLS tokens
Figure 2: Pruming of channels and tokens in one block. 'LFE' is the low-frequency energy extracted from tokens according to Equation [6] 'I.FS" denotes the low-frequency sensitivity used evaluate the importance of channels.
3.2 Channel Pruning based on Low-Frequency Sensitivity
As previously introduced, low-frequency components in images are more valuable for the feature representation in ViT models, iee. channels less effective in encoding low-frequency components will contribute less to the feature representation. Therefore, the key goal to estimate the sensitivity of channel to low-frequency components in images. To achieve this, propose evaluation policy named low-frequency sensitivity (LFS) that estimates the importance scores model parameters by taking more low-frequee cow-frequency components in images into account.
Assume redundant channels have less influence on model outputs, removing redundant channels should hardly change the value of loss when feeding set of training images into the model for loss computation. Thus, the importance of a channel can be quantified by the difference in loss induced by removing this channel. Given number of images X Rx H randomly sampled from the training dataset D, the importance score I, of a weight wy formulated as:
= (C W),Y Y wy = 0) C (X, W),Y))
(1)
where Y RBx1 is the label of data X, (.) denotes the loss function (cross-entropy loss in this paper), M (X, W) is the model output, and W indicates all model weights.
However, the score in Equation can only reflect the importance the original whole images. To separate low-frequency components from images, low-pass filtering is applied on images in the Fourier spectrum domain before feeding them into ViTs. The fre-ureyency components in images X are formulated
(G
where F.)) and F-l (.) denote the fast Fourier transformation (FFT) 40 and the inverse fast Fourier transform (IFFT), respectively, C the Hadamard product, 9 (.) the low-pass filter, ar E (0, 1) determines the cutoff frequency of the low-pass filter which is similar to the radial averaging of the
Fourier spectrum as in [T6 1.5 5 46). Considering that binary filter will cause the Ringing effect when the image is transformed back to the spatial domain, Gaussian filter is chosen for & (-).

---

In addition to the task-specific loss, the pruned model shall also provide robust feature representation as the original model. In other words, the feature representation of the low-frequency images X shall be as close to that of the original images X as possible. Hence, apart from the cross- entropy loss L. for the classification task, Kullback Leibler (KL) divergence loss K-((-) is used measure the error between the CLS tokens corresponding to the low-frequency image and nature image, respectively, Denote the two CLS tokens F (from low-frequency images) and T (from nature images), and simplify the cross- entropy loss L (M (X, W), Y) to C (X). Then, the final importance score 8, of weight wy, named Low-Frequency Sensitivity (LFS), is formulated as:
loss is also taken into account.
8, = -=x(((()) wy = 0) - -()) + (1 ((1-A)((KC(TTTT) W3 =
-K)()) KC(TT))
(3)
where À is the hyper-parameter for the balance of two loss functions.
Calculating the LFS for each parameter with Equation Bi infcasible for models with millions of parameters. Fortunatcly, the score can be approximated with the first-order Taylor expansion [34] 60). Therefore, the approximated version of LFS is represented as below:
(C(X) Ow.
( T)
8y = A (
W/
+ +(1-
Wy
(4)
Owy
where the gradient terms can be easily obtained in the backward procedure the model. The channel importance score can then be approximated by summing over LFS scores of all parameters in the channel, i.e. the LFS of a channel is computed by the sum of 8y:
$J
85
(5)
JEJ
where means the index set of weights in a channel.
3.3 Token Pruning based on Now-Frequency Energy
Token redundancy is another major issue in the ViT compression, and several methods 47 28 25 sample informative tokens via analyzing the relationship or attention scores between tokens. Such solution is sub-optimal because the selected tokens tend to be similar, and the information included in the token itself has been neglected to some extent. To address this problem, the Low-Frequency Energy (LFE) is proposed to make use of the low-frequency preference of ViT for token pruning, Following other works 47 28 25], as shown in Figure 2 the selector located between the multi- head self-attention module and the feed-forward network module. Inspired by (55), we evaluate the low-frequency ratio of the token after transforming tokens X into the frequency domain by
applying FFT each channel of tokens, denoted as X F (x122 b...). We then quantify the
(xx22)
low-frequency information contained in a token by calculating the ratio of its remaining energy to the total energy after low-pass filtering. Given filter G with cutoff factor oe, the LFE is formulated as:
F- (a.) + x4.2
X12
[x ll T1 DDC [2
|c
b,3.:
(6)
X
where DC and [.] denote the direct-current component and the www-frquency component, respectively. Intuitively, token with more low wwwffrquency components will achieve a larger M),
Similar to EVIT 28), the attention scores in the spatial domain also included to evaluate the final importance scores of tokens. For the h-th head in ViT, the attention value is calculated as:
QuaKIT Jatn
A softmax
where A h is the attention score matrix and dr, is the output dimension. The CLS token plays more significant role than other tokens because it is the final output feature which collects information
5

---

from all tokens. Moreover, the head with denser and larger attention values is more important, i.e., with larger proportion. Thus, our proposed modified attention score formulated as:
N N7
1 H
1
Fiis
Σ h=0
On.o 0n.0-
A/ + 0n.1
Σ
A
where 0%,0 = ード= Abj and 0,,1 A0,0 are the head-weights of the CLS attention value A and the other attention value Alj. respectively.
To estimate the importance score of tokens from multiple and diverse aspects, we consider to combine the LFE 7, and attention score Ti. to get the final importance score of token as:
T Tta M1,6
(9)
Note: the CLS token the final output of the ViT model, and is not involved the token pruning.
3.4 Bottom-up Cascade Pruning
Origiaal Meck:
Presing tokems
Presing channels:
Pronel block:
FraEER desii
Brsso dealt
Irqpano dasab
Inqpscy tasah
Brassc dkauls
deadds
Aloel
Alre
Mecht
Work
Wock
Wied
Avck
Dense 1ST model
Compressed 12T model
Figure 3: Pipeline of Bottom-up Cascade Pruning. The pruning starts from the first block to the last one. In each block, the token is pruned first according to the information in the frequency domain (LFE criterion). Once token pruning is done, channels will be compressed based on the LFS criterion.
In our method, the value of LFS and LFE in each block is related to the outputs from previous blocks. Hence, the compression one block will influence the LFS and LFE in its subsequent blocks. is sub-optimal to independently determine the pruning ratios and indices of channels and tokens for all blocks at once and ignore their inter-relationship. Therefore, we design a Bottom-up Caseade Pruning (BCP) process (Figure 3). which promotes pruning from the first block to the last block. A hyper-parameter, named global allowable drop E, is set to control the final performance drop after pruning. During compression of each block, the number of tokens is gradually reduced until the performance drop reaches E = p. E/LL where is a hyper--paae parameter to control the accuracy drop caused by token pruning. Once token pruming is done. channels will be compressed with similar procedure. When the performance drop caused by pruning reaches E/ the compression for a block considered completed. More details are described in Algorithm 1 in Appendix A.1
4 Experiments
In this section, the proposed method is evaluated the benchmark ImageNet (ILSVRC2012) 4 which is a large dataset containing 1.2M training images and 50k validation images of 1000 classes. All the experiments based on the previous study 2 The float operations (FLOPs) of models are evaluated by fvcord") B
deployed with Pytorch [39] on NVIDIA V100 GPUs. The code is modified

---

4.1 Experiments on ImageNet
Implementation details. The proposed method is applied to popular VIT models of three different
sizes, DeiT-Tiny, Deil-Small, and Base. The latest state-of-the-ar (SOTA) methods are compared, including SCOP [49]. CP-ViT [47), PoWER [8] HVT (37) IA-RED [36]. V
EViT (28), and SPVIT 21. In the pruning procedure, the number of training samples used for
evaluating the performance drop in BCP is 5000 (randomly sampling 5 training samples from each category), the number of training samples for calculating LFS is 2000, and the cutoff factors oe and o, are 0.1 and 0.85. For three models, DenT-Tiny, DeiT-Small, and DeiT-Base, the global allowable drop & are 9.5, 14, and 14, and the ratio for the allowable drop is 0.56, 0.35, and 0.3 respectively. The removed channels involve the columns (output channels) of W W.t.h, WLA,, and W ja and
q rows (input channels) of W Wrooj and 2e After pruning, the compressed models are fine-tuned with hard distillation [50] of their corresponding original models. The base learning rate set to 0.0001, and most of the other hyper-parameed Deii--iny/DeiT-- Deaaeiii-imamaiiiTTTaa. -ass models for 300/150/150 epochs. More detailed settings and results of different epochs are listed in Appendix A.3
follow the settings in [9]. We fine-tune the pruned
Results and analysis. The comparison with state-of-the-ane methods shown in Table 1 in which the top-1 accuracy and the reduction ratios of FLOPs are reported. In all three models, the proposed method achieves the highest reduction ratio in FLOPs than previous methods with less than 1% performance drop. Compared to the pure token pruning methods like SCOP 49), CP-VIT [47],
PoWER [8]. HVT (7), and IA-RED² [36], our method achieves not only better performance and more reduction in FLOPs, but also less parameters, which demonstrates the superiority of pruning with frequency domain. Although EViT (28). SPVIT 21 and S ViTE (9] achieves better accuracy on DeiT-Base Basee their reduction in FLOPs (EViT 34.1%, SPVIT 33.1%, 1, S 33.1 % vs Our 57.6%) 6)) is much lower than ours. To evaluate the acceleration on inference speed of our pruning technique, the throughput is assessed on single V100 GPU with batch size 256 in Table 2 model achieves
The DeiT- BeeTT-TiyySmmll/Baaa demonstrates the practicability of the proposed compression approach.
/tt///9770%//.../111 1% speed up after pruning, which
Table I: Comparison with state-of-the-art methods on ImageNet-Ik, 'FLOPs 4' denotes the reduction ratio of FLOPs. We report two versions with different parameter sizes for our method.
DeiT-Tiny
DeiT-Small
DeiT-Base
Method
(5)) FL.PSs & Params Topl/Top5(() FL.PSs 1 Params Topt/ () FLOPss : Params
Baseline
72.2/91.1
5.7M
79.8/95.0
22. IM
81.8/95.6
86. 46
SCOP 49
68.9/ 38.4%
5.7M
77.5/
43.6%
22.
79.7
42.0%
86.4M
PoWER 18
69.4/ 38.4%
5.7M
78.3/
41.3%
22.1M
80.1
39.2%
86.4M
CP-ViT 47
71.2
43.3%
5.7M
79.1/-
42.2%
22.1M
81.1/-
41.6%
86.4M
EVIT 28
78.5/94.2
50.0%
22.1M
81.3/95.3
34.1%
86.4M
HVT (37
69.7
69.7/89.4
46.2%
5.7M
78.0/93.8
47.8%
22.1M
IA 33
-/-
79.1/94.5
31.5%
22. 80.3
80.3/95.0
33.0% 86.4M
SPVITE 9
70.1/- 23.7%
4.2M
79.2/-
31.6%
14.6M
82.2/-
33.1% 56,8M
SPVIT 21
70.7 23.1%
4.9M
78.3/94.3
/44.
28.3% 16.4M
81.6/95.5
33.1%
62.3M
VTC-LFC
71.6/90.7
46.7%
5.1M
79.4
94.6
54.4%
17.7M
$1.3/95.3
57.6%
63.5M
VTC-LFC
71.0/90.4
41.7%
4.2M
79.6/94.8
194.8 47.1% 15.3M
81.6/95.6
95.6
54.4% 56.8M
Table 2: Throughput of baselines and compressed models. 'Speed up' means the improvement in throughput. "base" denotes the baseline model, and 'pruned' is the compressed model.
Top1 Dhase/pruncd
Top5 (ease/pruned)
Throughput (base (asse/)
Model
Speed up
DeiT-Tiny
72.2%
91.1%
2648.7/4496.2
69.7%
DeiT-Small
79.8% 19.8% /7..4% $1.8%/81.32
95.0% /94.6%9
97.0%
DeiT-Base
95.6% /95.3%
314.7 (14.7/651.9
107.1 %
7

---

Table 3: Results of channel pruning and token pruning with different criteria on DeiT-Small. In the column 'Acel (%)", 'FT' means fine-tuning. 'FLOPs 4" denotes the reduction ratio of FLOPs. is noted that the original NVTT compresses large-scale model to the target size (e.g. ViT-Small) and uses extra CNN teacher models, so we implement NVîT" to compress ViT-Small the pruned ViT-Small under the standard pruning setting for fair comparison here. For clear comparison, BCP not applied in any experiments here.
Channel Pruning NVIT
Token Pruning
Ace1 (%)
LFS (ours)
EviT (baseline)
LFE (ours)
FLOPs +
(baseline)
before FT
after FT
X
X
X
X
79.8
0.0%
V
X
X
X
47.5
78.9
32.8%
X
V
X
X
61.2
79.4
32.8%
X
X
V
X
76.8
79.6
43.3%
X
X
X
V
77.6
80.1
43.3%
V
X
V
X
40.5
78.0
55.0%
X
X
V
57.9
78.7
55.0%
AcC3 1NN pruning
Meccingg 1AA witees prouiing
proning
Oriiinll
KV
EVIT
Channel pruning rabe
Ratio of saved tokons
Cutoff factor
(a) Channel pruning
(b) Token pruning
(c) Cutoff factor de
Figure (a) results channel pruning with two different criteria 'NVIT and 'LFS'. (b) results after token pruning with criteria 'EVIT" and 'LFE'. (c) results with different cutoff factors to determine the cutoff frequency of low-pass filters.
4.2 Ablation Study
Effectiveness of LFS and LFE. For fair comparison to analyze the effectiveness of the proposed LFS (for channel pruning) and LFE (for token pruning), we conduct experiments without using BCP framework in Table Two state-of-the-aar methods NViT 60) and EVIT 28 are selected as baselines for channel pruning and token pruning, respectively. NVIT identifies channels based on the Taylor score and EVTT selects tokens according to the attention score. The channels are pruned globally using the manual pruning rate NVIT, and the number of tokens is determined following same ratio as EVIT. The main difference between other methods and ours is whether to leverage the characteristics of VIT on low-frequence components. Table shows the results of only compressing channels, tokens, and both. For channel pruning. can be found that LFS outperforms NVIT* with the same FLOPs reduction. For token pruning, LFE achieves even higher accuracy than the original DeiT-Small. Both comparison results demonstrate the superiority of pruning based on information in the frequency domain over using only information in the space domain. The results of different pruning ratios for channels and tokens are displayed Figure ((a) and (bb) in which the proposed method consistently outperform the other methods under all ratios. The distributions of importance scores for models are displayed in Appendix 10
The influence of the cutoff factor, which determines the ratio of saved low-frequency components to all frequency, is also analyzed. shown in Figure 4(c) a. 0.1 the sweet spot, which further proves the preference of ViT on low-frequency information. For dr, 0.85 is the best choice.

---

Table 4: Automatically searching pruning ratios with 'CP ' is the hyper-parameter that controls the performance drop caused by pruning, and "p" a sypem--aramettr that balances channel pruning and token pruning. All results are achieved by DeiT-Small.
s/p
10/0.33
13/0.35
14/0.15
14/0.35
14/0.55
15/0.35
18/0.35
FLOPs +
48.6%
54.0%
46.0%
54.4%
59.2%
55.2%
58.5%
Params
18.7M
17.8M
17.3M
17.7M
17.9M
17.4M
16.8M
Acel w/o FT 71.6%
69.2%
68.8%
68.3%
69.0%
66.9%
64.6%
Acel w/ FT 79.8%
79.3%
79.9%
79.4%
78.7%
79.1%
78.9%
Factor analysis for BCP. The pruning ratios for channels and tokens are determined by two hyper- parameters and p. As shown in Table when maintaining p 0.35 and increasing from 10 to 18, the model parameter size will be reduced from 18. 7.. to 16.8M, and the performance ranges from 79.8% to 78.9%. Similarly, when maintaining the model size and the FLOPS reduction ratio are increased, while the performance is reduced from 79.9% 78.7%. Compared with the result (78.7% accuracy and 55.0% Flops reduction ratio) without BCP the last row Table 3] we can observe that BCP improves the accuracy or compression ratio of the model, which verifies the effectiveness of the proposed BCP strategy. After comprehensively considering the model parameters size, the FLOPS reduction ratio and model accuracy, we set the
14 and increasing p from 0.15 5 to 0.55, both
14 and p 0.35 in this paper. These parameters be adjusted according to specific requirements in real-world applications.
Influence of each module. To further study how each component affects the proposed method, LFS, LFE, and BCP are respectively removed from our scheme. The modified versions are then applied on the same model. Since BCP will automatically adjust pruning ratios of each block, it is necessary to introduce additional variables for controlled experiments. For a fair comparison, we keep the same pruming ratios (determined by VTC-LFC) of each block in all experiments. As shown in Table the proposed LFS/LFE/BCP improves the performance by 1.3%/0.6 1...0.%%0/99... before fine-tuning and 0.1 0./0.5%//..25 after fine-tuning, respectively. The experimental results show the effectiveness of the proposed modules.
Table 5: Effects of LFS, LFE, and BCP on the proposed compression method. 'Original' is the original model without pruning. 'Ours' the result including LFS, LFE, and BCP. 'NvEv-P' means pruning globally as strategies in NVTT and EVIT rather than the proposed block-by- by-blockk scheme.
Model
Method
Acel (before FT)
Acel (after FT)
FLOPs
Original
79.8% 8%
0.0%
VTC-LFC (Ours)
68.3%
79.4%
54.4%
DeiT-Small
w/o LFS (LFS (LVS")
67.0%
79.3%
54.4%
w/o LFE (LFE-EVVT)) w/o BCP (BCP->NVEv-P)
67.7%
78.9%
54.4%
67.4%
79.2%
54.4%
Influence of pruning sequence. The main reason for pruning from the first block the last one is that the pruning in previous blocks will change the inputs of their subsequent blocks, which may influence the value of our proposed LFS and LFE. If the last block is pruned first, the selected channels and tokens in this block will need to be re-adjusted when former blocks are compressed. Instead, if the former blocks are pruned firstly, the inputs for the subsequent blocks are fixed. Both pipelines are executed and compared on DeiT-Smail and the results shown in (bbleoddemonstee the advantage of the proposed buttom-up (from first to last) pipeline.
Table Pruning with different sequences. 'Bottom-up' means pruning from the first block the last one While'Top-doww denotes pruning from the last block to the first one.
Pipeline
Top1/Top5 w/o FT Top1/Top5 w/ FT FLOPs
Params
Bottom-up (Ours)
68.3/89.1
79.4/94.6
2.1G
17.7M
Tow-down
68.2/89.0
78.9/94.5
2.1G
17.6M

---

Influence of hyper-parameter A. The hyper-pamne parameter À used to balance different terms in LFS during compression analyzed here. As listed in T the models pruned with different values are evaluated after pruning. It can be found that the model achieves the best accuracy when À is 0.1. In addition, the proposed method is not very sensitive the value of > 0).
Table 7: Pruning 20% channels with different X.
Hyper-parameter"
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Top1 accuracy after pruning 65,90
69.76
69.70
69.73
69.73
69.70
69.67
69.67
Compression on other VIT models. In addition to DeiT models, the proposed method is also evaluated on LV-ViT [24] and window-based attention model Swin [32], in which the pruned models are fine-tuned for 150 epochs, respectively. Specifically, due to the token downsampling and the window shifting, token pruning not adapted to Swin yet so that only channel pruning is adopted Swin. To simplify, the token labels in the original LV-ViT are not used during fine- tuning. For LV-, our method obtains 0.3% higher (81.8% vs 81.5%) performance and 0.1G lower FLOPs (3.2G vs 3.3G) than the combination of existing channel pruning (NViT) and token pruning (EViT) approaches. On Swin, the proposed VTC-LFC achieves 0.2/3, 1% higher accuracy than the previous SPVIT /21/STEP 27 with fewer FLOPs (3.3G vs 3,4G/3.5G) and parameters (17.1M vs 25.8M/23.6M4 6M). The results shown in Table & demonstrate that proposed method can still achieve better performance as well as lower FLOPs than previous approaches on other ViT architectures.
Table Results for LV- and Swin Transformer backbones on the ImageNet-Ik.
Model
Method
Top1(% (%)
Top5 (%)
FLOPs (G)
Params (M)
Original
83.2
96.3
6.5
25.8
LV-ViT-S
+EVI
81.5
95.3
3.3
20.2
VTC-LFC (Ours)
81.8
95.6
3.2
20.2
Original
81.1
95.5
4.5
28.3
Swin-Tiny
STEP 27
77.2
93.6
3.5
23.6
SPVIT 21
80.1
95.0
3,4
25.8
VTC-LFC (Ours)
80.3
95.0
3.3
17.1
5 Conclusion and Discussion
This paper reveals the disadvantages of pruning ViT's only the spatial domain and takes advantage of the preference of VITs for low-frequency information conduct compression. Two metrics, low- frequency sensitivity and low-frequency energy are proposed to leverage knowledge in the frequency domain for better channel pruning and token pruning. The comparison with the spatial domain pruning approaches proves that the proposed method can identify the informative channels and tokens more precisely, thus better maintaining the model accuracy. Additionally, with the proposed bottom- up cascade pruning strategy, both channels and tokens are automatically compressed in a unified framework. Extensive experiments of different models on ImageNet demonstrate that more than half of computational costs are saved from ViTs, with significant improvements in inference efficiency, The comparison with the latest compression methods shows the superiority (better performance and more FLOPs reduction) of the proposed approach. As preliminary study about pruning ViT's with the frequency domain, more efficient ways and deeper studies are expected to explored based on
Acknowledgement
This work was supported by Alibaba Group through Alibaba Research Intern Program.
10

---

References
[1] A. Ali, H. Touvron, M. Caron, P. Bojanowski, M. Douze, A. Joulin, Laptev, N. Neverova, G. Synnacve, J. Verbeck, Xcit: Cross-covariance image transformers, Advances neural information processing systems, 34, 2021.
[2] A. Amini, A. S. Periyasamy, and S. Behnke. T6d-direct: Transformers for multi-object 6d pose direct regression. DAGM German Conference on Puttern Recognition, pages 330-544. Springer, 2021.
[3] J. Bai, L. Yuan, S.-T. Xia, S. Yan, Z. Li, and W. Liu. Improving vision transformers by revisiting high-frequency components. arXiv preprint arXiv:220.009993, 2022.
[4] Н. Bao. L. Dong, and F. Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.0822 2021.
[5] K. Chandrasegaran, N.-T. Tran, and N.-M. Cheung. A closer look at fourier spectrum discrepancies for enn-generated images detection. In Proceedings of the IEFF/CVE conference computer vision and pattern recognition, pages 7200-7209, 2021.
[6] A. Chavan, Z. Shen, Z. Liu, Z. Liu, K.-T. Cheng, and E. Xing. Vision transformer slimming: Multi- dimension searching in continuous optimization space. arXiv preprint arXiv:2201.00811, 2022.
[7] C.-F. R. Chen, Q. Fan, and R. Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 357-366, 2021.
[8] H. Chen. Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma, Xu, C. Xu, and W. Gao. Pre-trained image processing transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12299-12 12310, 2021.
[9] T. Chen, Y. Cheng, 7. Gan, L. Yuan, I., Zhang, and Z. Wang. Chasing sparsity in vision transformers: An end-to-end exploration. Advances in Neural Information Processing Systems, 34, 2021.
[10] X. Chen, C.-J. Hsieh, and B. Gong. When vision transformers outperform resnets without pre-training or strong data augmentations. arXiv preprint arXiv:2106. 01548, 2021.
[11] B. Cheng, A. Schwing, and A. Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34, 2021.
[12] L. Ding, D. Lin, S. Lin, J. Zhang, X. Cui, Y. Wang. H. Tang, and L. Bruzzone. Looking outside the window: Wide-context transformer for the semantic segmentation of high-resolution remote sensing images. IEEE Transactions Geoscience and Remote Sensing. 2022.
[13] X. Ding, X. Zhou, Y. Guo, J. Han, J. Liu, et al. Global sparse momentum sgd for pruning very deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
[14] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[15] R. Durall, M. Keuper, and J. Keuper. Watch your up-convolution:
Con based generative deep neural networks are failing to reproduce spectral distributions. In Proceedings of the IELE/CVF conference on computer vision and patlem recognition, pages 7890-7899, 2020.
[16] T. Dzanic, K. Shah. and F. Witherden. Fourier spectrum discrepancies in deep network generated images. Advances in neural information processing systems,
2020.
[17] A. El-Nouby, N. Neverova, I. Laptev, and H. Jégou. Training vision transformers for image retrieval. arXiv preprint 200: 05644, 2021.
[18] S. Goyal, A. R. Choudhury. S. Raje, V. Chakaravarthy, Y. Sabharwal, and A. Verma. Power-bert: Accel- crating bert inference via progressive word-vector climination. In International Conference on Machine Learning, pages 3690-30000 30999, PMLR, 2020.
[19] B. Graham, A. El-Nouby, H. Touvron, P. Stock, A. Joulin, H. Jégou, and M. Douze. Levit: a vision transformer in convnet's clothing for faster inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12259-12269 2021.
[20] J. Guo, K. Han, H. Wu, C. Xu. Y. Tang, C. Xu, and Y. Wang. Cmt: Convolutional neural networks meet vision transformers. arXiv preprint arXiv:2107.06263 2021.

---

[21] H. He. J. Liu, Z. Pan, J. Cai. J. Zhang, D. Tao, and B. Zhuang. Pruning self-attention into convolutional layers in single path. arXiv preprint arXiv:2111. 1802, 2021.
[22] S. He, H. Luo, P. Wang. F. Wang, H. Li, and W. W.aiang. Transreid: Transformer-based object re-identification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15013-15022 2021.
[23] Y. He, P. Liu, Z. Wang. Z. Hu, and Y. Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[24] Z.-H. Jiang, Q. Hou, L. Yuan, D. Zhou, Y. Shi, X. Jin, A. Wang, and J. Feng. All tokens matter: Token labeling for training better vision transformers. Advances in Neural Information Processing Systems, 34, 2021.
[25] Z. Kong, P. Dong, X. Ma, X. Meng, W. Niu, M. Sun, B. Ren, M. Qin, H. Tang, and Y. Wang. Spvit: Enabling faster vision transformers via soft token pruning. arXiv preprint arXiv:21 13890, 2021.
[26] Z. Lan, M. Chen, S. Goodman, K. Gimpel. P. Sharma, and R. Soricut. Albert A lite bert for supervvee learning of language representations. In International Conference on Learning Representations, 2019.
[27] J. Li, R. Cotterell, and M. Sachan. Differentiable subset pruning of transformer heads. Transactions of the Association for Computational Linguistics, 9: 14444944559 2021.
[28] Y. Liang, G. Chongjian, Z. Tong, Song. Wang, and Xie. Evit: Expediting vision transformers via token reorganizations. In International Conference on Learning Representations, 2021.
[29] M. Lin, R. Ji, Y. Wang, Y. Zhang, B. Zhang, Y. Tian, and I... Shao. Hrank: Filter pruning using high-rank
feature map. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 1529-1538, 2020.
[30] S. Lin, R. Ji, Yan, B. Zhang, L. Cao. Q. Ye, Huang, and D. Doermann. Towards optimal structured cnn pruning via generative adversarial learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2790-2799, 2019.
[31] N. Liu, X. Ma. Z. Xu, Wang. Tang, and Ye. Autocompress: An automatic dnn structured pruning framework for ultra-high compression rates. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4876 4883, 2020.
[32] Z. Liu, Y. Lin, Cao, H. Hu, Y. Wei, Z. Zhang. S. Lin, and Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[33] Z. Liu, H. MU, X. Zhang, Z. Guo, X. Yang, K.-T. Cheng, and J. Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE International Conference on Computer Vision, pages 3296-3305, 2019.
[34] P. Molchanov, A. Mallya, S. Tyree, I. Frosio, and J. Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11244-11272, 2019.
[35] P. Molchanov, A. Mallya, S. Tyree, I. Frosio, and J. Kautz. Importance estimation for neural network pruning. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[36] B. Pan, R. Panda, Y. Jiang, Z. Wang, R. Feris, and A. Oliva. la-red: Interprotability-wwaa redundancy reduction for vision transformers. Advances in Neural Informatio Processing Systems, 34, 2021.
[37] Z. Pan, B. Zhuang, Liu, He, and Cai. Scalable vision transformers with hierarchical pooling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 377-386, 2021.
[38] N. Park and S. Kim. How do vision transformers work? In International Conference on Learning Representations, 2021.
[39] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang. Z. Detito, Z. Lin, A. Desmaison, L. Antiga, and
A. Lerer. Automatic differentiation pytorch. 2017.
[40] C. Rader and N. Brenner. A new principle for fast fourier transformation. IEEE Transactions on Acoustics, Speech, and Signal Processing, 24(3):264-266 1976.
[41] Y. Rao, W. Zhao, B. Liu, J. Lu. J. Zhou, and C.-J. Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34, 2021.
12

---

[42] Y. Rao, W. Zhao, Z. Zhu, J. Lu, and J. Zhou. Global filter networks for image classification. Advances in Neural Information Processing Systems, 34, 2021.
[43] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma. Z. Huang. A. Karpathy, A. Khosla, Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-25) 2015.
[44] M. S. Ryoo, A. Piergiovanni, A. Amab, M. Dehghani, and A. Angelova. Tokenlcarner; What can learned tokens do for images and videos? arXiv preprint arXiv:2106. 002.00061119... 11297, 2021.
[45] R. Schwartz, G. Stanovsky, S. Swayamdipta, J. Dodge, and N. A. Smith. The right tool for the job: Matching model and instance complexities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6640-6651, 2020.
[46] K. Schwarz, Y. Liao, and A. Geiger. On the frequency bias of generative modcls. Advances in Neural Information Processing Systems, 11188826-181388 2021.
[47] 7. Song, Y. Xu, 7. He, I., Jiang. N. Jing, and X. Liang. Cp-vit: Cascade vision transformer pruning via progressive sparsity prediction. arXiv preprint arXiv:2203.04577 2022.
[48] Y. Tang, K. Han. Y. Wang, Xu, Guo, C. Xu, and D. Tao. Patch slimming for efficient vision transformers. arXiv preprint arXiv:2106.0288 2021.
[49] Y. Tang. Y. Wang. Y. Xu, D. Tao, C. Xu, C. Xu, and C. Xu. Scop: Scientific control for reliable neural network pruning. Advances in Neural Information Processing Systems, 33:10936-10947 2020.
[50] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[51] A. Vaswami, N. Shazeer, N. Parmar, J. Uszkoreit, I.. Jones, A. N. Gomez, F.. Kaiser, and I. Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[52] H. Wang, X. Wu, Z. Huang, and E. P. Xing. figh-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8684 8694, 2020.
[53] P. Wang, X. Wang, H. Luo, J. Zhou, Z. Zhou, F. Wang. H. Li, and R. Jin. Scaled relu matters for training 2021.
vision transformers. arXiv preprint 1xxxx:210990388
[54] P. Wang, X. Wang, F. Wang, M. Lin, S. Chang, W. Xie, H. Li, and R. Jin. Kvt: k-nn attention for boosting vision transformers. arXiv preprint arXiv:2 005/55 2021.
[55] P. Wang, W. Zheng, T. Chen, and Z. Wang. Anti-oversmoothing in deep vision transformers via the fourier domain analy sis: From theory to practice. In International Conference on Learning Representutions. 2021.
[56] Y. Wang, Xu, X. Wang, C. Shen, Cheng, H. Shen, and H. Xia. End-to-end video instance segmen- tation with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattenn Recognition, pages 8741-8750, 2021.
[57] K. Wu, J. Zhang, H. Peng, M. Liu, B. Xiao, J. Fu, and L. Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. arXiv preprint arXiv:2207. 10666, 2022.
[58] Y. Xu, Z. Zhang, M. Zhang, K. Sheng, K. Li, W. Dong, L. Zhang, C. Xu, and X. Sun. Evo-vit: Slow-fast token evolution for dynamic vision transformer. arXiv preprint arXiv:2108.01 01390, 2021.
[59] F. Yang, H. Yang, J. Fu, H. Lu, and B. Guo. Learning texture transformer network for image super- resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5791-5800 2020.
[60] H. Yang, H. Yin, P. Molchanov, H. Li, and J. Kautz. Nvit: Vision transformer compression and parameter redistribution. arXiv preprint orXiv:2/ 10,04869, 2021.
[61] S. Yu. T. Chen, J. Shen, H. Yuan, J. Tan, S. Yang, J. Liu, and Z. Wang. Unified visual transformer compression. In International Conference on Learning Representations, 2021.
[62] L. Yuan, Q. Hou, Z. Jiang, J. Feng. and S. Yan. Volo: Vision outlooker for visual recognition. arXiv preprint 110i 13112 2021.
13

---

[63] C. Yun, Y.-W. Chang, S. Bhojanapalli, A. S. Rawat, S. Reddi, and S. Kumar. o(n) connections expressive enough: Universal approximability of sparse transformers. Advances Neural Information Processing Systems, 33:13783-13794, 2020.
[64] J. Zhang, H. Peng. K. Wu, M. Liu, Xiao, J. Fu, and L. Yuan. Minivit: Compressing vision transformers with weight multiplexing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12145-12154, 2022.
[65] C. Zhao, pruning. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Zhang, Q. Zhao, Zhang, and Q. Tian. Variational convolutional neural network
[66] Zhou, Wang. Wang, Q. Liu, H. Li. and Jin. Elsa: Enhanced local self-attention vision transformer. arXiv preprint arXiv:211..127860 2021.
[67] Zhou, Ge, J. McAuley, K. Xu, and Wei. Bert loses patience: Fast robust inference with early exit. Advances in Neural Information Processing Systems, 33:18330-18344 2020.
[68] Zhu, Han, Tang, and Y. Wang. Visual transformer pruning. arXiv e-prints, pages arXiv-2104, 2021.
[69] X. Zhu. Su, Lu, B. L..X. Wang, and Dai. Deformable Deformable transformers for end-to-end object detection. In International Conference on Learning Representations, 2020.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes] Please see Section
(c) Did you discuss any potential negative societal impacts of your work? [No] This work
scientific in nature, and we do not believe has immediate negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] Our work does not contain theoretical results.
(b) Did you include complete proofs of all theoretical results? [N/A] Our work does not contain theoretical results.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material as URL)? [Yes] We have released the code.
(b) Did you specify all the training details (e.g.. data splits, hyper- parameters, how they were chosen)? [Yes] Please see Section 4 and appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] We did not report the error bars since training vision transformer on ImageNet is stable. The error bars can be neglected, and the results can be reproduced with the ppper-paramtters introduced in the paper.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] We describe the details of computation resources in Section 4
4. If you using existing assets (e.g., code, data, models) curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We used publicly available data, i.e., ImageNet, in our experiments. We cited the corresponding papers published by the creators in Section 4
(b) Did you mention the license of the assets? [N/A] The license of ImageNet is included in the paper that we have cited.
(c) Did you include any new assets either in the supplemental material or as a URL? [No] The ImageNet we used are publicly available.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We did not collect/curate new data.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] All ImageNet datasets are already publicly available and broadly adopted. We do not think there are any issues of personally identifiable information or offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "oW0I_g5KFW", "reviewer": "Reviewer_ZeeK", "summary": "In this paper, the authors proposed the pruning method with low-frequency for ViTs. The metrics named low-frequency sensitivity for channel pruning, and low-frequency energy for token pruning are proposed, where the low-frequency image and feature are introduced. Besides, a bottom-up cascade pruning is proposed to prune the channels and tokens gradually. The experiments show a good reduction ratio of FLOPs and throughput with less performance drop, and the effectiveness of each proposed components.", "questions": "1) In the section 4.1 Line 228, the pruned models DeiT-Tiny/DeiT-Small/DeiT-Base are finetuned for 300/150/75 epochs, respectively.\n     - Why the number of epochs for the three models are different?\n     -  How many epochs to finetune the pruned models for other pruning methods, like SPViT, S2ViTE? Is it comparable to that in this paper?\n     -  It usually takes 300 epochs to train DeiT-Tiny on ImageNet-1k. Why the pruned DeiT-Tiny needs finetuning for 300 epochs? \n\n2) In the section 3.4 Bottom-up Cascade Pruning, the model is pruned from the first block to the last one. Is there any comparison to the method from the last block to the first one?\n3) In line 164, how to set the hyper-parameter λ in the experiment? Is there any ablation study on it?", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors introduced low-frequeny image and feature to prune the ViT models. It is the first work to compress the model in the frequency domain.\n2. The improvement on the throughput is significant with less performance drop.\n3. The formulas of the proposed pruning method are clear and correct.\n4. The paper is well written.\n\nWeaknesses:\n1. The performance metric is not sufficient. The pruned model can be evaluated on ImageNet-Real/V2 and downstream classification datasets, like CIFAR-10/100.\n2. In Table 1, there is no comparison on the number of the finetuning epochs.\n3. In Table 5, the accuracy of the model without LFS and BCP after finetuning is close by 0.2 to VTC-LFC. The difference is too small to show the effectiveness of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "H-AGPlkcRr", "reviewer": "Reviewer_2SYW", "summary": "The major contributions of this paper are 2-fold:\n1) By leveraging on the understanding regarding the low-frequency information preference of ViTs, this work proposes improved channel pruning and token pruning schemes to compress ViTs. This is achieved using 3 components below:\n- **Channel pruning** : A quantitative metric called LFS (Low Frequency Sensitivity) is proposed to estimate the importance of model parameters by largely considering low-frequency components in images. This is used for channel pruning.\n- **Token pruning** : A quantitative metric called LFE (Low Frequency Energy) is proposed to estimate the percentage of low-frequency information in tokens. \n- **A bottom-up cascaded pruning framework** is proposed to jointly compress channels and tokens of ViTs.\n\n2) The proposed method obtains large improvements in ViT compression (i.e.: reduce FLOPs by 40%-60% at the expense of only 1% top1 accuracy) in ImageNet based classification setups.\n", "questions": "Please see Weaknesses section above for a list of questions. Further please consider answering the following questions:\n\n1) Can the authors include Top-5 accuracies for Tables 1, 2?\n", "limitations": "Briefly mentioned in Checklist 1(C).", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1) This paper is written well. It is easy to follow.\n2) The improvements in terms of FLOPs and parameter reduction using VTC-LFC is impressive. VTC-LFC has potential to be useful in edge-intelligence applications (where ViTs are superior to CNN based architectures).\n\n**Weaknesses:**\n1) **The overall compression procedure is unclear**. I.e.: There is also a knowledge distillation component (Line 159). Can the authors show the overall algorithm (a pseudocode will be useful)? \n2) **Only final top1 accuracy of VTC-LFC is insufficient to understand the significance of the proposed method / More analysis is needed**: This paper could significantly benefit from more analysis / statistics regarding channel pruning and token pruning. Given that LFS and LFE are scalar values, can the authors show the distribution of these values in Baseline / VTC-LFC ViTs? (A simple histogram could be useful). \n3) **The cutoff point between low-frequency and high frequency for the experiments is not clear**: How is Figure 4 ( c ) obtained? Is it obtained by radial averaging of 2D Fourier spectrum similar to these deepfake works ( [1, 2, 3, 4] )?\n4) The caption of Figure 3 is too plain. Please consider explaining the details as Figure 3 is very important in this paper.\n\nOverall I enjoyed reading this paper. In my opinion, the strengths of this paper outweigh the weaknesses by a small margin. I’m happy to change my opinion based on the rebuttal. \n\n=================\n\n[1] Dzanic, T., Shah, K., & Witherden, F. (2020). Fourier spectrum discrepancies in deep network generated images. Advances in neural information processing systems, 33, 3022-3032.\n\n[2] Durall, R., Keuper, M., & Keuper, J. (2020). Watch your up-convolution: Cnn based generative deep neural networks are failing to reproduce spectral distributions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 7890-7899).\n\n[3] Chandrasegaran et al., 2021: \"A closer look at fourier spectrum discrepancies for cnn-generated images detection.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[4] Schwarz et al., 2021: \"On the Frequency Bias of Generative Models.\" Advances in Neural Information Processing Systems 34 (2021).\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8w19rqu2dXx", "reviewer": "Reviewer_KA3x", "summary": "This paper introduces a model compression approach based on low-frequency components for vision transformers. Channel pruning based on low-frequency sensitivity, token pruning based on low-frequency energy, along with bottom-up cascade pruning scheme effectively reduce the model complexity while maintaining high model accuracy. ", "questions": "1. How effective would this approach be when applied to other ViT models such as LV-ViT? \n2. Would it be useful for window-based attention and hierarchical model structure such as Swin?", "limitations": "adequate ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths: \n1. it proposes a novel approach for model compression with low-frequency component, by leveraging the intrinsic property of self-attention modules as low-pass filters. \n2. it introduces the bottom-up cascade pruning framework for model compression.  \n3. it outperforms many SOTA vision transformer compression methods with higher accuracy at reduced model complexities \n\nweaknesses: \nit's unclear how effective this approach would be when applied to other ViT models such as LV-ViT. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept