Data-independent Module-aware Pruning for Hierarchical Vision Transformers
Authors: Yang He
Joey Tianyi Zhou
Venue: ICLR 2024 poster (2024)
Keywords: Filter Pruning; Model Compression; Vision Transformer
Primary area: general machine learning (i.e., none of the above)

Abstract: Hierarchical vision transformers (ViTs) have two advantages over conventional ViTs. First, hierarchical ViTs achieve linear computational complexity with respect to image size by local self-attention. Second, hierarchical ViTs create hierarchical feature maps by merging image patches in deeper layers for dense prediction. However, existing pruning methods ignore the unique properties of hierarchical ViTs and use the magnitude value as the weight importance. This approach leads to two main drawbacks. First, the "local" attention weights are compared at a "global" level, which may cause some "locally" important weights to be pruned due to their relatively small magnitude "globally". The second issue with magnitude pruning is that it fails to consider the distinct weight distributions of the network, which are essential for extracting coarse to fine-grained features at various hierarchical levels. 

To solve the aforementioned issues, we have developed a Data-independent Module-Aware Pruning method (DIMAP) to compress hierarchical ViTs. To ensure that "local" attention weights at different hierarchical levels are compared fairly in terms of their contribution, we treat them as a **module** and examine their contribution by analyzing their information distortion. Furthermore, we introduce a novel weight metric that is solely based on weights and does not require input images, thereby eliminating the **dependence** on the patch merging process. Our method validates its usefulness and strengths on Swin Transformers of different sizes on ImageNet-1k classification. Notably, the top-5 accuracy drop is only 0.07% when we remove 52.5% FLOPs and 52.7% parameters of Swin-B. When we reduce 33.2% FLOPs and 33.2% parameters of Swin-S, we can even achieve a 0.8% higher relative top-5 accuracy than the original model. Code is available at: [https://github.com/he-y/Data-independent-Module-Aware-Pruning](https://github.com/he-y/Data-independent-Module-Aware-Pruning).

Published as a conference paper at ICLR 2024
DATA-INDEPENDENT MODULE-AWARE PRUNING FOR HIERARCHICAL VISION TRANSFORMERS
Yang Joey Tianyi Zhou"
CFAR, Agency for Science, Technology and Research, Singapore IHPC, Ageney for Science, Technology and Research, Singapore {He Yang, Joey Zhou) 2hou) @cfar. wtt.cff.rasstr...m edu.seedusse
ABSTRACT
Hierarchical vision transformers (VITs) have two advantages over conventional VITs. First, hierarchical ViTs achieve linear computational complexity with re- spect to image size by local self-attention. Second, hierarchical ViTs create hi- erarchical feature maps by merging image patches in deeper layers for dense prediction. However, existing pruning methods ignore the unique properties of hierarchical ViTs and use the magnitude value as the weight importance. This approach leads to two main drawbacks. First, the "local" attention weights are compared "global" level, which may cause some "locally" important weights to be pruned due to their relatively small magnitude "globally". The second issue with magnitude pruning is that fails to consider the distinct weight distributions of the network, which are essential for extracting coarse to fine- ine-grained features at various hicrarchical levels.
To solve the aforementioned issues, we have developed a Data-independent Module-Aware Pruning method (DIMAP) compress hierarchical ViTs. To en- sure that "local" attention weights at different hierarchical levels are compared fairly in terms of their contribution, we treat them as module and examine their contribution by analyzing their information distortion. Furthermore, we introduce a novel weight metric that is solely based on weights and does not require input images, thereby eliminating the dependence the patch merging process. Our method validates its usefulness and strengths on Swin Transformers sizes on ImageNet-1K classification. Notably, the top-5 accuracy drop is only 0.07% when we remove 52.5% FLOPs and 52.7% parameters of Swin-B. When we reduce 33.2% FLOPs and 33.2% parameters of Swin-S, 0.8% higher relative top-5 accuracy than the original model. Code is available at: https//gghhhba.oommmme///tmtttte Moddalepeeeneee---o----eeeeee Awwell---Awae--uunnnnn
different
achieve
INTRODUCTION
Vision transformers Dosovitskiy et al. (2020); Touvron et al. (2020); Yuan et al. (2021) have achieved state-of-the-art (SOTA) performance in the area of computer vision, including image clas- sification, detection, and segmentation. However, the utilization of self-attention and the removal of the convolutions cause vision transformers Dosovitskly al.; (2020): Liu al. (2021) heavy com- putational burdens and significant parameter counts. Therefore, it is necessary to trim the model to reduce the computational cost and required storage.
It is challenging to apply conventional CNN weight pruning methods directly to Vision Transformers since they have different structures and weight properties Raghu et al. (2021); Park & Kim 2022).
The most important structure of modern CNNs such as VGGNet Simonyan & Zisserman (2015) and ResNet He et al. (2016) are convolutional layers. shown in Figure I the convolutional layers of ResNet-50 contribute to 92.0% parameters and 99.9% foating-poin operations (FL.Os) of the whole network. In contrast, convolutional layers merely contribute 0. 1% to parameters and 0. 1% to FLOPs in the Swin Transformer. The special structure of VITs should thus be considered during the pruning process.
"Corespondiigg Author

---

Published as a conference paper at ICLR 2024
Although there have been attempts prune con- ventional ViTs, the unique characteristics of hier- archical ViTs are not fully explored. Local atten- tion is a fundamental feature of hierarchical ViTs, requiring attention computation within a window rather than over the entire image. While this re- duces computation on a large scale, it also poses a question: attention weights within one window cannot be compared with those of another window, even if both windows belong to the same image.
135.4M 15.90
2:30
25:0M
$7.5N
Faro. FLOPs
Comy
ATT
23.8%
VGG-16
Inceptionv3
ResNel-50
Swin-B
Figure 1: Parameters and FLOPs for different components of CNNs (VGG-16, Inceptionv3, ResNet-50) and Swin Transformer. "Conv"= Convolutional layers, "FC" = Fully-connected layers, "ATT" Attention layers. The numbers on top of the histograms are the total parame- ters and FLOPs of the network. The percentage numbers listed inside histograms are the con- tribution from the corresponding layers. Note that norm layers and down wwwn-sampling layers are not included for better visualization.
We use an example to explain the problem of the previous magnitude pruning methods. Assume the attention value of window A ranges from to 0.3, and that of window B ranges from 0 to 0.7. Previ- ous magnitude pruning methods Han et al. (2015b) will make this pruning decision: before removing the weights larger than 0.3 in window B, all the weights in window A should be removed. How- ever, this decision may not always be correct as pruning all weights in a particular window, such as window A. could lead to the loss of an important object in the image. Therefore, when determining the least important attention weight in an image, we should not take magnitude into account, but rather rank them based on their contribution.
The second characteristic of hierarchical ViTs is that image patches are merged at higher hierarchical scales. In contrast, conventional V
al. (2020); Touvron al. (2020) have the same number of image patches at different levels. The patch merging characteristic of hierarchical ViTs designed to extract features ranging from coarse to fine-grained for dense prediction. To achieve this property. the weight distribution of different layers should adapt to the image patches at those layers. To minimize the impact of varying image patch sizes, the weight importance metric should be data-independent. Using the magnitude value weight metric is the casiest way to ensure data independence. However, it is not valid when comparing attention values across windows or weight values across layers.
In the paper, we propose a Pata-independent Module- Aware Pruning method (DIMAP) to address the above two problems together. First, include different layers within module and evaluate weight importance at the level of module rather than window or layer. By analyzing the Frobenius distortion Park al. (2020) incurred by single weight, safely compare the contribution of all weights. This allows us to evaluate the importance of weights within module, regardless of their location within the windows or layers. Further, we analyze the information distortion at the module level and develop a novel data iata-independent weight importance metric. The proposed inteeinneeendent module-aware weight importance metric in DIMAP has several desirable properties. Firstly, it provides a "global" ranking of the importance of weights at the module level without altering the "local" ranking within a layer, making it generalization method for layer-wise magnitude pruning. Secondly, it does not use any extra parameters and is purely based on weight values, making it simple and efficient. Finally, it can be applied without the need for complex pruning sensitivity analysis, allowing for efficient one-shot pruning.
obtain their weight importance and
Contributions are summarized as follows: (I) We analyze the Frobenius distortion incurred by Bongle-weight and module-level pruning. (2) We propose data-independent weight importance metric to address the issues of previous pruming methods. (3) Experiments validate that our results comparable the ofate-of-the-ard results on image classification benchmarks.
2 RELATED WORKS
CNNs Pruning. 1) Weight Pruning. Weight pruning LeCun et al. (1900); Han al. (201 a)
aims to prune the fine-grainn mne-grained weights of the network. For example, Han et al. (2015b) proposes to remove the small weights whose absolute values are below the threshold. Guo et al. allaal/221011 (2016) proposes to discard weights dynamically during the training process. Lebedev & Lempitsky (2016) utilizes
the group-sparsity regularization the loss function to shrink some entire groups of weights toward
2

---

Published as a conference paper at ICLR 2024
zeros. However, these weight pruning methods are designed for CNNs, so it is difficult to directly
apply these methods to vision transformers with different properties Raghu et al. (2021): Park &
Kim (2022). 2) Filter Pruning. Filter pruning Li et al. (2017; He et al. (2017 2017);10 Luo et al. (2017);
He et al. (2018a) Sa 2019); Suau et al. (2018); He & Xiao (023) removes the filters entirely to obtain
models with structured sparsity, the pruned convolutional models can achieve better acceleration.
Li ct al. (0017) uses Ll-norm to evaluate the importance of filters of the network. He et al. (2018b) leverages reinforcement learning to find the redundancy for each layer automatically. However, there are not so many "filters" in vision transformers, so we are not able to utilize these algorithms.
Variants of Vision Transformer. Vision transformer firstly proposed in ViT Dosovitskiy et al. (2220), which utilizes attention not on pixels but instead on small patches of the images. After this, different variants of vision transformers (Srinivas et al. b; Yuan et al.
(202T):
Wang et al.
2021 ;
Chen et al.] (2021a); Chu et al. (202Tb): Han et al. (2021 Xu et al.
2022 Jiang et al.
2021 Xia et al. (2022); Mangalam et al. (2022; Gong et al. (2022): Li et al.
(2022b): Mehta &
Rastegari 2022); ; Song et al. (2022); Shao et al. 20222) Tang et al. (2022a by: Yang et al. (2022a)
are emerging. T et al. (2020) introduces a distillation token to make the student network
learns from the teacher network through attention. Boe Srinivas et al. (022 proposes to replace the spatial convolutions with global self-attention in the final three bottleneck blocks of a ResNet. T2T-VIT Yuan et al.] (202T) introduces layer-wise Tokens-To-Toker transformation to train the vision transformer from scratch. PVT Wang et al. (202T) utilizes a pyramid structure for dense prediction tasks. Cross Chen et al. (202Ia) learns multi-scale feature representations with ViT. TNT Han et al. (202]) introduces Transformer in Transformer for excavating features of objects in different scales and locations. CoaT Xu al. (20TT) empowers image transformers with enriched multi-scale and contextual modeling capabilities. LV- et al. (2021) proposes a new training loss for VIT by taking advantage of all the patch tokens. Swin Transformer Liu et al. (202T) attracts attention for its hierarchical structure and efficient shift window attention.
Vision Transformer Compression. Some parallel methods Heo et al. (2021) ) Graham et al. (20 l); Pan et al.) (202T):
| Yue et al. | (202T); (hhu et al. (202T): [Cuu et al. (202Ta) Yang et al. (202Tb) Liang
ct al. (2022); Yin et al. (2022); )); Zhang et al. (2022a); Yang et al. (2022b); Li et al. (2022a); Chen et al.
(202bb); Zhang al. 2022b); Yu et al. (2022); Yu & Xiang (2023) Yang al. 2023); ; Chuanyang
et al. (2022) are proposed for vision transformer compression and acceleration. 1) This direction
focuses the redundancy of networks, and the structures of the original network mostly kept.
An important direction is to reduce the input image tokens Lee et al. (2023). For example, Dynam- icViT Rao al. (202T) prunes redundant tokens progressively. EViT Liang al. (2022) reorganizes the token to reduce the computational cost of multi-head self-attention. SVITE Chen et al. (2021c)
proposes sparse ViT with a trained token selector. 2) Another direction is handle the network itself. VTP Zhu et al. (2021) prunes the unimportant features of the ViT with sparsity regulariza- tion. AutoFormer Chen et al. (2021b) uses an architecture search framework for vision transformer search. As- Chen et al. (022c) automatically scales up ViTs. UVC et al. (202T) assembles three effective techniques, including pruning, layer skipping, and knowledge distillation, for effi- cient VIT. NViT Yang et al. (2021a) uses structural pruning with latency-aware regularization on all parameters of the vision transformer. Most previous compression literature Chen et al. (202Tc); Zhu 2222 Chen et al. 2022c 2021b) focus pruning conventional ViTs such as DeiT Touvron ct al. (2020). which use global self-attention. In contrast, hicrarchical ViTs such as Swin Trans- former utilize local self-attention to save computational costs. Therefore, it is more difficult to compress hierarchical ViTs than conventional ViTs.
et al.
3 MODULE-AWARE PRUNING
3.1 PRELIMINARIES
Assuming a Swin Transformer has L layers, and (! B is the weight for the la layer, where is the kernel size, (:(()) and (/(0) is the number of input and output channels, respec- tively. The first layer of the Swin Transformer is the patch-embedding layer. It is a convolutional layer, so K > 1 and the dimension of this patch-embedding layer is 4. For the latter attention related layers and fully- colleccondected layers, the dimension of these layers is 2, so
view the value of K as K = 1. LayerNorm (LN) layers are I-dimension vectors, and these layers only contribute a small portion of FLOPPs and storage.
3

---

Published as a conference paper at ICLR 2024
First Block
Second Block
Different Modules
40
QRV
OKYM
AUX-M
Swin Transformer
Blocks
Pelih Merging
2
Transformer
MLP-FC2
Blocks
MLP.FC1
Patch Merging
Rabio Weights (9)
20
LN
Swin Transformer
Blocks
Patch Merging
ATT-PRJ
FCE
FC2
Swin Transformer
Blocks
ATT-OKY
20
Embodding
Images
FC2
FC2
Different modules inside
(b) Swin Transformer network with four stages: ① 2 + ?
swin transformer block
20
Figure 3: Categorize network layers to different modules regarding (a) Swin Transformer block; (b) Swin Transformer network. The left fig- ure contains detailed layers of Swin Trans- former block. There are four stages in Swin Transformer. Different colors represent dif- ferent modules, including the attention module (QKV-M and PRJ-M), the multilayer perceptron--eelated module (MLP-M), and the auxiliary module (AUX-M).
-01
0.1
Values of Weights
Figure 2: Weight distributions for different func- tional layers for the first and second Swin Trans- former block. The meaning of QKC, PRI, FC1 and FC2 can be found in Figure 3 The x-axis in- dicates the values of the weights, and the y-axis denotes the ratios of weights.
Weight Distributions. The weight distributions for different functional layers shown in Figure The first observation is a notable difference in the weight distribution of the QKV layer between the first and second blocks: the QKV layer of the first block contains significantly higher number of zeros than that of the second block. The dissimilarity in the distributions supports our statement that direct comparison of attention with their magnitude is appropriate. Moreover, even with the same block, different functional layers have various distributions. Applying a magnitude pruning threshold of 0.03 would lead to the removal of approximately 30% of the weights from the QKV and PRJ layers, but up to 80% of the weights from the FC1 and FC2 layers. It is clear that magnitude pruning results in unbalanced pruning outcomes for these layers. To ensure fair comparison of weights across different layers, we employ the information distortion analysis as the weight metric for pruning, as shown 3.2
Module Definition The definition of modules is shown in Figure More details can be found in Appendix B
3.2 DISTORTION ANALYSIS AND WEIGHT METRIC
Single Layer Analysis. We first utilize single fully-connected layer to consider the pruning problem from the perspective of minimizing &, distortion after pruning Neyshabur et al. (2013): : Lee al. (2020p. Supposing input
gn and weight tensor W E _ the pruning operation can viewed as W := M <) W, where M is a binary mask whose entries are either 1. the sparsity constraint ||o I We aim to find pruning mask M to minimize the difference between the output from pruned and an unpruned layer:
min sup |M
Wa (M O W)
|
Attention layers include similar matrix multiplications for Q, K and
4

---

Published as a conference paper at ICLR 2024
where || v is supported the unit ball. Then the &g distortion can be bounded as:
[ x (M @ a & |W - -M @ W |
(2)
Therefore, solving Equation is equivalent to solving:
min
IIW M WIl
(3)
|M
where || denotes the spectral norm || = - @
With Cauchy-Schwarz inequality, the TElla
operation can be relaxed to Frobenius distortion minimization:
min I|W M W
(4)
|| S
Module Level Analysis. Suppose the output of a network W(I.)) = (W()), W(D) given the input is:
c: (e-))))) w (... (w())))...
(5)
As shown in the previous section, a module consists of multiple layers. Suppose P: (2,,2..... .......,,l.. 1,3 the set that includes all the layers in the module. Pruning a module aims to minimize the following:
| (r: (a.L)) f (a:WaL))
min
sup 11
where (I:L) is the network after pruning a module. Since directly solving Equation 6 is difficult, we approximate the distortion in Equation 6 by pruning single connection. Assume ( == M o ( indicates the leh. weight layer after pruning and I E (4,,42....,44. J. In this situation, we have an upper bound
f ( (x; (((l)))))))
f (2: W w((h)))
sup |
w - w(()
II JENL.1]
Weight Metric, With the above analysis of distortion minimization, the last step is to convert the above weight selection process to a data-independent weight importance metric. To rank weights inside module, we rewrite the right side of the Equation 7
as:
W
W JE(LL]]
II
F
In this formulation, the right side of Equation has influence on pruning since W is not included the product term 1/ ||W6) p. Then the distortion pruning can be evaluated with the left side (the algehraic fraction) of Equation g We look into the numerator of the fraction:
[ - W
Σ
(1
M1j) W
(9)
iE JE(L.....
where and j the index of W and M. To minimize Equation for the top largest W (1 Miy) should be O.
We sort the weights of (1)) in descending order and get (Wp, wp....., Wpmxn J. where Wp, is the largest weight. In the step of removing WT p, the current W()) consists of (W pry W Wp2y.., Wp,)
Please proof in Supplementary.
5

---

Published as a conference paper at ICLR 2024
since weights smaller than W already pruned. After pruning W pus the current W consists of {Wp: p.r Wp....,, Wp-.. }. Then we can rewrite the left side of Equation to get the weight importance to minimize the distortion:
(Wp)) )²
[wp) = Ligp, (W.)²
(10)
The proposed weight importance has several advantages. 1) This is data-independent and does not require extra parameters other than the weight values. 2) We can conduct one-shot pruning to save computational costs.
4 EXPERIMENTS
4.1 EXPERIMENTAL. SETTING
Dataset and Architecture. We follow previous works Chen et al. (2021c); Yu ct al. (2021); Chen al. (02Tb) by validating our method on the ImageNet- I Russakovsky al. (2015) benchmark dataset. ImageNet-1K KK contains 1.28 million training images and 50k validation images of 1, 000 classes. In our experiments, the input resolution is 224 x 224. The architectures include Swin-T, Swin-S and Swin-B. Note that the numbers of Swin Transformer are cited from the authors' official Github r These numbers are slightly different from those reported in their paper Liu et al. (022T: 0.19% higher for Swin-S, 0.14% lower for Swin-T.
Network Training. We utilize the same training schedules as Liu et al. (2021) when training. Specifically, employ AdamW optimizer for 300 epochs using cosine decay learning rate scheduler and 20 epochs of linear warm-up. The initial learning rate is 0.001, the weight decay is 0.05, and the batch size is 1024. Our data augmentation strategies also the same those of Liu et al. (2021) and include color jitter, AutoAugment Cubuk et al. (2018), random erasing Zhong et al. 2020), mixup Zhang et al. (017)) and CutMix Yun al. (2019).
Pruning Setting. We utilize different pruning ratios to analyze the accuracies for models with different sizes. For the fine-tuning process, we use AdamW optimizer for 30 epochs using cosine decay learning rate scheduler Loshchilov & Hutter (2017). The base learning rate is 2c-5, and the minimum learning rate is 2e-7. The number of warm-up epochs is 15, and the final learning rate of the linear warm-up process is 2e-8. The weight decay is le-8. During pruning, we keep the AUX-M and prune the other three modules.
We follow UVC Yu et al. (2021) in utilizing FLOPs for acceleration measurements and use the number of non-zero parameters to measure the required storage Han et al. (2015b). We compare with state-of-the-arr image classifiers variants including Dosovitsk et al. (2020), DeiT Touvron ctt al. (2020), BoTNet Srinivas et al. (202I). T2T- ViT Yuan et al. (2021), PVT Wang et al. (2021). Cross Chen et al. (202 CPVT Chu et al. Ib) TNT et al. (2021). TransMix Chen et al.] (2022a), MobileViTve Mehta & Rastegari (2023), STVIT Chang et al.] (202)), Slide-Swin- T al. (2023) Maaten-SSin--TPe
al. (2023) Long et al. De Long al. (2023) and
vision transformer acceleration methods including VTP Zhu et al. (2021), UVC Yu et al. (2021), SVITE Chen al. (2021c), AutoFormer Chen al. (2021b), As- ViT Chen
(202cc), and ViT Chen et al. (2022d). A-ViT Yin et (2022). ViT-Slim Chavan et al. (2022). 2) WDPruning! Yu al. (2022). NVTT-S Yang al. (2023), X-Pruner Yu & Xiang (2023). For all of these comparisons, results with the nearest parameters and FLOPs with our methods are listed for a better comparison.
4.2 PRUNING SWIN TRANSFORMER
The results of pruning Swin Transformers are shown in Table For every network, we set three tar- get FLOPs reduction ratios, naming them DIMAPI, DIMAP2, DIMAP3. For pruning Swin-B, B, when we remove 14.3% parameters and 14.4% FLOPs, we can improve top-1 accuracy by 0.04%. This result indicates that pruning Swin Transformer has similar regularization effect pruning CNNs.
httg:://ghhhha.ccmmmmicsosssssii Transforrmer

---

Published as a conference paper at ICLR 2024
Top-1 acc.(%)
Top-1 acc._
Top-5
Top-5 acc
Para
FL.PPs
Method
acc.(%)
Para.
((%)
(%)
(%)
Swin B et
2021
83.48
96.46
87.8M
15.4G
Swin-B-DIMAPI
83.52
-0.04
96.43
0.03
75.2M
13.2G
B-MMAP2
83.43
0.08
96.42
58.4M
Swin-R-DIMAP3
83.28
96.39
41.7M
Swin Liu et.l.
2021
83.19
96.23
49.6M
8.7G
Swin-S-DIMAPI
83,08
0.11
96.25
-0.02
42.5M
7.5G
Swin-S-DIMAP2
82.99
0.20
96.26
-0.03
33.1M
33.2
5.8G
33.2
Swin S.DIMAP3
82.63
0.56
23.7M
Swin-T Liu 2021
81.16
95.48
28.3M
4.5G
Swin-T-DIMAPI
81.17
-0,01
95.47
0.01
24.4M
13.7
3,80
13.9
Swin-T-DIMAP2
81.11
0.05
95.42
0.06
19.2M
32.0
3.0G
32.4
Swio-T-DIMAP3
80.35
0.81
95.22
0.26
14.0M
50.3
2.2G
50.8
Table 1: Comparison of the pruned Swin Transformer ImageNet- IK. The "acc. |" is the accuracy drop between pruned model and the baseline model (smaller values are better). 'he negative value of "acc. 4" means the performance of the pruned model is better than the original model. "Para. 4" and "FLOPs 4" are the ratio of removed parameters and FLOPs, respectively.
When pruning about 33.2% of the FLOPs Swin-S, the top-5 accuracy improves by 0.03%. Al- though Swin-T is the smallest Swin Transformer model, we can still achieve superior performance. When we remove 13.7% FLOPs or 13.9% FLOPs, the top-1 accuracy drop improves by 0.01% %. we further increase the pruning ratio to 50.8% FLOPs, the top-5 accuracy drop is 0.26%. These results demonstrate that DIMAP works well on variants of the Swin Transformer.
Comparing the "large pruned" to the "small unprumed" model is interesting. We find our pruned models achieve better accuracies with fewer parameters. For example, when we compare Swin-B- DIMAP3 ("large pruned") and Swin-S ("small unpruned"), 0.09% higher accuracy with 7.9M fewer parameters. Similarly, we look further at Swin-S- DIMAP3 ("large pruned") and Swin-T ("small unpruned"), we find this scenario happens again: our Swin-S-DIMAP3 achieves 0.53% higher accuracy with 4.6M fewer parameters. These results indicate that our DIMAP method effectively shrinks large models into small models and finds mod- els with better accuracy-comppttttion tradc-offs than the originally designed models.
Swin-B-DIMAPE model achieves
4.3 COMPARISON WITH
METHODS
Comparison with Vision Transformer Variants (Table 2a Our Swin-T-DIMAP2, Swin-S- DIMAPI and Swin-B-DIMAP3 perform better than PVT-Small, PVT-Medium, PVT-Large Wang et al. 2021), respectively. Our Swin-S-DIMAP3 achieves 0.33% higher top- 1 accuracy with 0.5G fewer FLOPs required than Slide- Sinde-Swiin--r Pan et al. (023). ) Our Twin-S-DIMAN has a similar im- provement over Flatten-Swin-T Han et al. (2023). The Flatten Han al. (2023) and Slide Pan al. 2023) methods demonstrate a better trade-off on Swin-S and Swin-B architectures as they focus on improving the attention process. Our proposed method focuses on weight and can be combined with their approaches to achieve higher performance. We only report the results of BAT Long et al. 2023 DeiT' since the Swin transformer is not included in the original BAT paper. These results validate the effectiveness of our DIMAP method.
Comparison with Vision Transformer Compression Methods (Table 2b). Our Swin-B-DIMAPE
utilizes 6.3M fewer parameters and 2.7G fewer FLOPs than VTP Zhu et al. (2021) with 2.58% better accuracy. We also achieve 0.7G fewer FLOPss than UVC et al. (2021 with 2.68% higher accuracy. Our Swin-B-DIMAP3 achieves 1. 1.72% better accuracy than S SVITE Chen et al. (2021c) with 10.3M fewer parameters and 3.5G fewer FLOPs. Our Swin-S-DIMAP3,, Swin-B-DIMAP3, Swin-B-DIMAP also performs better than AS-VIT-Small Chen et al. (2022c), As-VIT-Base Chen et al. (2022c). As- VIT-Large Chen et al. (002cc), respectively. Compared to X-Pruner Yu Xiang (023), , S-DAA22 has 0.99% better accuracy and 0.2G fewer FLOPs. These results demonstrate that our DIMAP achieves a better trade-off between accuracy and computation than existing compression methods.

---

Published as a conference paper at ICLR 2024
method
Image Para. FL.Pss ImugeNet
top-1
method
image size
#paramn F1.0Ps
ImageNot
top-1 acc.
VIT-B/16 0088/16Dooe Dosovitskiv et
2020
384
86M
55.4G
77.9
VTT-L/16
Dosovilsky et
2020
384
307M
190.7G
76.5
VTP Zhu et al
2021
224
48.0M
10.00
80.7
VIP
Zhu et al.
2021
224
67.3M
13,80
81.3
DefT-S Touvron al
2020
224
22M
4.6G
79.8
Defl-B
Loryron et a
A001
224
86M
17.5G
81.8
UVC
Yu etal
2021
224
8.0G
80.6
DeiT
al.
2021
384
86M
55.4G
83.1
SVITE
Chen
202
224
13.3M
2.90
80.26
BoT-S11-0 Srinivas
2021
224
20.8M
4.27G
VTV
C hen el
224
14.6M
3.1G
79.22
SVTTE
hen
224
VIT.4.14 19
224
82.4
S 2PVITE
VITE
Chen et
2021c
224
56.8M
11.7G
82:22
T2T-V/T-4-44 Yuan et ml
0121
224
64,IM 15.0G
82.6
VT
Meng al.
2022
224
3.9G
81.1
PmTl Wans al. 2021
224 24.5M 3.8G
79.8
PVT-Mediim Wang et 2021
2242 44.2M 6.7G
81.2
ViT-Slim Chavan
224
17.7M
PVT-Large Wang et
2021
224 61.4M
9.8G
81.7
AutoFormer Chen et al
202
224
54M
11G
82.4
CrossViT--81 Chen Chen
2021a
224
44.3M 9.5G
A-ViI Yin et
2022
224"
22M
3.60
80.7
102
224
22M
79.9
CPVT-R Chu
2021h
224 86M
81.9
WDPruning Yu al.
2022
224
7.6G
82.41
WDPruning Yu 2022
224a
6.3G
81.80
As-ViT-Small Chen
Cheen
2022g
224
29.0M
5.3G
81.2
As-Vil-Base Teen et
224
52.6M 8.90
82.5
NVIT-S Yong cl
2023
224
21M
4.2G
82.19
Vir-Large
224"
88.1M
X-Pruner
Ya & Xiang
2023
224
3.2G
80.7
TNT Han et
2021
224
65.6M 14.1G
82.9
X Pruner
Yu & Xiang
2023
224
6.0G
82.0
TransMix Chan ot al. 2022a
224
86.6M
17.6G
81.8
W-In-P- Swin-T-DIMAP!
224
14.0M
2.20
80.35
224
19.2M
3.0G
81.11
MobileViTv2 Mehta & Rastegari
2023
224
28.3M
4.5G
81.3
Swin DIMAPPI
224
STVIT-Swin- Chane al.
STVT Swin
STVTT-Swin-B Chiing
Chany P
10.48G
3.14G
3,95G
83.2
82.8
81.0
Swin S. S-DIMAP2
Swin S S-DIAP3
224
224
35.1M
23.7M
5.8G
4.1G
82.63
82,99
Swin-S-DIMAL
224
42.5M
7.5G
83.08
Slide-Swin-T T Pan
2023
30M
4.6G
Bein-B-DIMAP3
224
41.7M
7.3G
83.28
Flatten Swin
Tan et al
2023
224
29M
4.5G
82.1
Swin-B-DIMAP2
224
58.4M
10.2G
83.43
BAT-DeiT-S] Long Loogget
2023
3.0G
79.6
Swin-B-DIMAPI
224
75.2M
13.2G
83.52
Vision transformer variants.
(b) Vision transformer compression methods.
Table 2: Compare results on ImageNet- 1K classification.
Method
Swin-B 19%
Swin-S 19%
Swin-T 19%
Swin-B 28%
Swin-S 28%
Swin-T 2812
Swin-B 38%
Swin-S 38%
Swin-T 38%
Swin-B 50%
Swin-S 50%
Swin-T
Ratio
50%
Uniform
83.25
82.91
82.27
81.91
77.73
15.26
13.93
6.02
Ours
83. 33.36 82, au (0..7 83.02, (4.79) 82.58 82.58 80. 24..2444.....111555 81.55400 56,,6.mmmmm
78.5
78.54
78. 18.13 76.
76.78
1.36
Table 3: Comparison of uniform magnitude-basee pruning with our DIMAP. The ratio here means the proportion of removed parameters after pruning. The numbers in brackets top- accuracy gain over uniform pruning.
4.4 ADDITIONAL ANALYSIS
Module Pruning Rate Analysis. As shown in Figure analyze pruning results based proposed module-level weight importance. Specifically, we first calculate the module-level weight importance for each module. Then we conduct magnitude pruning to remove the least important weights from the module. Note that a module merely has one threshold for pruning. The pruning thresholds for QKV-M, PRJ-M and MLP-M are 3.8E-7, 4.99E-7 and 1.5E-6, respectively. After pruning, the ratios of the kept parameters for different layers are shown as the y-axis in Figure We have several observations. 1) The lower layers keep a larger ratio of parameters than the higher layers. This phenomenon indicates the higher layers have greater redundancy. 2) In the first and the second Swin Transformer block, we find that the attention-related layers (layer 1,2,5,6) have larger ratios than MLP-relate layers (layer 3,4.7.8). This result shows that the attention operation is extremely important for low-level features. 3) We can take a look at Figure and Figure together. Layer 4 and layer 8 in Figure correspond two FC2 layers in Figure 2 In Figure 5 layer 4 has a smaller weight-kept ratio than layer This pruning result reasonably corresponds to distributions in Figure 2 there more weights close zero in layer 4 than in layer 8.
8

---

Published as a conference paper at ICLR 2024
Muliiple Modules (%)
Single Module (S)
Model
Difference (%)
Swin-B-DIMALE
83,28
83,17
0.11
Swin SIDIMAP3
82.63
82.31
0.32
Swin-T-DIMAP3
80.35
Figure 4: Comparison of the theoretical and realistic ac- celeration. Only the time consumption of the forward pro- cedure is considered.
0.8
pro)
mlp
0.4
13 37
03
Figure 6: Visualization four pruned layers in Swin-B. B. The pruning ratio is 50%. The x-axis is the input dimen- sion, and the y-axis indicates the out- put dimension. The red areas indicate kept weights, and the grey areas rep- resent pruned weights.
Figure 5: Results of pruning 45% of the parameters from the Swin-S with our module-level weight importance. The x-axis denotes the layer index, and the y-axis indicates the ratio of the remaining parameters. Different colors repre- sent different modules.
Comparison with Uniform Pruning, In Table 3 we conduct an ablation study by comparing our method to magnitude-bras uniform pruning. Apparently, we are better than uniform pruning at all settings. Explicitly, at small pruning ratio like 19%. our accuracy gain is just about 0.1%. When increase the pruning ratio 50%. our DIMAP method achieves a 60+% accuracy gain over uniform pruning. These results validate that the necessity of DIMAP in guaranteeing performance, especially when the pruning ratio is large.
Visualization. We visualize the four DIMAP-prunce layers in the first Swin Transformer block in Figure 6 First, for the figure ATT-QKV, y-indexes for "Query, Key, and Value" are [0, (0,1 [128,255], [256,383], respectively. We find the "Query" and "Key" matrix of ATT-QKV is denser than the "Value" matrix. This implics that "Query" and "Key" may suffer more from pruming than "Value". Second, although ATT-PRJ, MLP-FCI and MLP-FC2 serve fully-comnected layers, the pattern of ATT-PRJ is quite different from the other two layers. A possible reason that the input of ATT-PRJ also includes the relative position bias of attention, and ATT-PRJ needs project this position information. Third, we can even find several rows of weights are removed entirely. For example, in MLP-FCI, may further guide us on how to design the size of MLP. Fourth, we find that the pattern of MLP- FC1 is similar to MLP-FC2. This result is another proof that we should categorize MLP-FCI and MLP-FC2 into the same module.
weights
kept when the y-index equals 228 and 231. This phenomenon
Model Module. is interesting to view the entire model single module. this setting, we keep the auxiliary layers unpruned and view all other layers as module. This means all the weights are compared based on our novel weight metric. The results shown in Tab, 4demonttrate our fine- grained module definition in Sec. 3.1 can achieve better performance than "Model as Module".
CONCLUSION AND FUTURE WORK
We introduce DIMAP to compress hierarchical ViT's while considering two key properties of these models. We present two main contributions: an analysis of information distortion at the module level, and a data-independdnt weight importance metric based on the distortion analysis. A drawback of weight pruning is that does not bring realistic acceleration on GPUs, and we will explore this direction in the future. Furthermore, are interested in the performances downstream tasks such as object detection and segmentation. We are also interested in the performance our method on other ViT variants.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGEMENT
This work was supported in part by A*STAR Career Development Fund (CDF) under C233312004, in part by the National Research Foundation, Singapore, and the Maritime and Port Authority of Singapore/ Singapore Maritime Institute under the Maritime Transformation Programme (Maritime Al Research Programme- Grant number SMI-2022-MTP-06).
REFERENCES
Shuning Chang, Pichao Wang, Ming Lin, Fan Wang. David Junhao Zhang, Rong Jin, and Mike Zheng Shou. Making vision transformers efficient from a token sparsification view. In Proc. IEEE Conf. Comput. Vis. Pattern Recog-, pp. 1195-6205, 2023.
Arnay Chavan, Zhiqiang Shen, Zhuang Liu, Zechun Liu, Kwang-Ting Cheng, and Eric P Xing. Vi- sion transformer slimming: Multi-dimension searching in continuous optimization space. Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4931- 4941, 2022.
Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. In Proceedings of the IEEE/CVF International Con- ference Computer Vision, pp. 357-366, 2021a.
Jie-Neng Chen, Shuyang Sun, Ju He, Philip HS Torr, Alan Yuille, and Song Bai. Transmix: Attend to mix for vision transformers. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12135-12144, 2022a.
Minghao Chen, Houwen Peng, Jiamlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 12270-12280, 2021b.
Tianlong Chen, Yu Cheng, Zhe Gan, Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. Advances in Neural Information Processing Systems, 34, 2021c.
Tianlong Chen, Zhenyu Zhang, Yu Cheng, Ahmed Awadallah, and Zhangyang Wang. The principle of diversity: Training stronger vision transformers calls for reducing all levels of redundancy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 12030- 2022b.
Wuyang Chen, Wei Huang, Xianzhi Du, Xiaodan Song, Zhangyang Wang, and Denny Zhou. Auto- scaling vision transformers without training. In ICLR, 2022c.
Xianing Chen, Qiong Cao, Yujie Zhong, Jing Zhang, Shenghua Gao, and Dacheng Tao. Dearkd: Data-efficient early knowledge distillation for vision transformers. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12052-12062. 2022d.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural Information Processing Systems, 34, la.
Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. Arxiv preprint 2102.10882 2021b. URL hwtphhtts////attte http://wwwvvv.r/g/ee../
org wwwpg/pp.ff.fmm 10.822..088 108110882.00 pdf
Zheng Chuanyang, Zheyang Li, Kai Zhang, Zhi Yang, Wenming Tan, Jun Xiao, Ye Ren, and Shiliang Pu. Savit: Structure-aware vision transformer pruning via collaborative optimization. Adv. Neural Inform. Process. Syst., 2022.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv: 1805.0000000000 2018
10

---

Published as a conference paper at ICLR 2024
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. / 1929, 2020.
Chengyue Gong, Dilin Wang, Meng Li, Xinlei Chen, Zhicheng Yan, Yuandong Tian, Vikas Chandra, et al. Nasvit: Neural architecture search for efficient vision transformers with gradient conflict aware supernet training. In International Conference on Learning Representations, 2022.
Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer convnet's clothing for faster inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12259-12269, 2021.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient DNNs. In Proc. Adv. Neural Inf. Process. Syst., 2016.
Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention. In Proc. Int. Conf. Comput. Vis., pp. 5961-5971, 2023.
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. In Adv. Neural Inform. Process. Syst., 2021.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In Proc. Int. Conf. Learn. Represent., 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Proc. Adv. Neural Inf. Process. Syst. 2015b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Yang He and Lingao Xiao. Structured pruning for deep convolutional neural networks: A survey. IEEE Trans. Pattern Anal. Mach. Intell., pp. 1-20, 2023, doi: 10. 10.109//PAMM.2023333444
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In Proc. Int. Joint Conf. Artif. Intell., 201 8a.
Yang He, Ping Liu, Ziwei Wang, and Yi Yang. Pruning filter via geometric median for deep convolu- tional neural networks acceleration. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2019.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proc. IEEE Inf. Conf. Comput. Vis. (ICCV), 2017.
Yihui He, Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Amc: Automl for model compression and acceleration mobile devices. In Proceedings of the European Conference Computer Vision (Eur. Conf. Comput. Vis.), pp. 784-800, 2018b.
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11936-11944 2021.
Zi-Hang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Yujun Shi, Xiaojic Jin, Anran Wang, and Jiashi Feng. All tokens matter: Token labeling for training better vision transformers. Advances in Neural Information Processing Systems, 34, 2021.
Vadim Lebedev and Victor Lempitsky. Fast ConvNets using group-wise brain damage. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2016.
Yann LeCun, John Denker, and Sara A Solla. Optimal brain damage. In Proc. Ady. Neural Inf. Process. Syst., pp. -98-605, 1990.
11

---

Published as a conference paper at ICLR 2024
Heejun Lee, Minki Kang, Youngwan Lee, and Sung Ju Hwang. Sparse token transformer with attention back tracking. In Int. Conf. Learn. Represent. 2023.
Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer-adaptive sparsity for the magnitude-basee pruning. arXiv preprint arXiv:2010.07611, 2020.
Changlin Li, Bohan Zhuang, Guangrun Wang, Xiaodan Liang, Xiaojun Chang, and Yi Yang. Au- tomated progressive learning for efficient training of vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11486-11446, 2022a.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient ConvNets. In Proc. Int. Conf. Learn. Represent., 2017.
Kunchang Li, Yali Wang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uni- former: Unified transformer for efficient spatiotemporal representation learning. In International Conference Learning Representations, 2022b.
Youwei Liang, Chongjian Ge, Zhan Tong. Yibing Song, Jue Wang, and Pengtao Xie. Not all patches
are what you need: Expediting vision transformers via token reorganizations. In International Conference on Learning Representations, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. 14030, 2021.
Sifan Long, Zhen Zhao, Jimin Pi, Shengsheng Wang, and Jingdong Wang. Beyond attentive tokens: Incorporating token importance and diversity for efficient vision transformers. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pp. 00334-10344 2023.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In Proc. Int. Conf. Learn. Represent., 2017.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. ThiNet: A filter level pruning method for deep neural network compression. In Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2017.
Karttikeya Mangalam, Haoqi Fan, Yanghao Li, Chao- Wu, Bo Xiong, Christoph Feichtenhofer, and Jitendra Malik. Reversible vision transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10830-10840 2022.
Sachin Mehta and Mohammad Rastegari. Mobilevit: light, general- and mobile- friendly vision transformer. In International Conference Learning Representations, 2022.
Sachin Mchta and Mohammad Rastegari. Separable self-attention for mobile vision transform- ers. Trans. Mach. Learn. Res., 2023. ISSN 2835-8856. URL http::://peeeeeeeew
forum?1d=tB14ype [Ki.
Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser- Nam Lim. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12309-12318, 2022.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory, pp. 1376-1401. PMLR, 2015.
Xuran Pan, Tianzhu Ye, Zhuofan Xia, Shiji Song, and Gao Huang. Slide-transformer: Hierarchical vision transformer with local self-attention. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pp- 2082-2091, 2023.
Zizheng Pan, Bohan Zhuang, Jing Liu, Haoyu He, and Jianfei Cai. Scalable vision transformers with hierarchical pooling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 377-386, 2021.
Namuk Park and Songkuk Kim. How do vision transformers work? In ICLR, 2022.
12

---

Published as a conference paper at ICLR 2024
Sejun Park, Jaeho Lee, Sangwoo Mo, and Jinwoo Shin. Lookahead: far-sighted alternative of magnitude-basee pruning. In Proc. Int. Conf. Learn. Represent., 2020.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy.
Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34, 2021.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNet large scale visual recognition challenge. Int. J. Comput. Vis., 2015.
Wenqi Shao, Yixiao Ge, Zhaoyang Zhang, Xuyuan Xu, Xiaogang Wang, Ying Shan, and Ping Luo. Dynamic token normalization improves vision transformer. In International Conference on Learning Representations, 2022.
Han Shu, Jiahao Wang. Hanting Chen, Lin Li, Yujiu Yang, and Yunhe Wang. Adder attention for vision transformer. Advances in Neural Information Processing Systems, 34, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. Proc. Int. Conf. Learn. Represent., 2015.
Hwanjun Song, Deqing Sun, Sanghyuk Chun, Varun Jampani, Dongyoon Han, Byeongho Heo, Wonjae Kim, and Ming-Hsuan Yang. Vidt: An efficient and effective fully transformer-based object detector. In International Conference on Learning Representations, 2022.
Aravind Srinivas, Tsung- Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition, 2021.
J Michael Steele. The Saucky-Schware master class: an introduction to the art of mathematical inequalities. Cambridge University Press, 2004.
Xavier Suau, Luca Zappella, Vinay Palakkode, and Nicholas Apostoloff. Principal filter analysis for guided network compression. arXiv preprint arXiv:1807.10585, 2018.
Shitao Tang, Jiahui Zhang, Siyu Zhu, and Ping Tan. Quadtree attention for vision transformers. In International Conference on Learning Representations, 2022a.
Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming for efficient vision transformers. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 12165-12174 2022b.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint ar Xriv:20012.1287, 2020.
Wenhai Wang. Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2 12122, 2021.
Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li. and Gao Huang. Vision transformer with de- formable attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4794-4803, 2022.
Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transform- ers. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 9981 9990, 2021.
Chenglin Yang, Yilin Wang, Jianming Zhang. He Zhang, Zijun Wei, Zhe Lin, and Alan Yuille. Lite vision transformer with enhanced self-attention. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. -1988- 2022a.
13

---

Published as a conference paper at ICLR 2024
Huanrui Yang, Hongxu Yin, Pavlo Molchanov, Hai Li, and Jan Kautz. Nvit: Vision transformer compression and parameter redistribution. arXiv preprint arXiv:2110.04869, 2021a.
Huanrui Yang, Hongxu Yin, Maying Shen, Pavlo Molchanov, Hai Li, and Jan Kautz. Global vision transformer pruning with hessian-aware saliency. In IEEE Conf. Comput. Vis. Pattern Recog, pp. 18547-18557, 2023.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal attention for long- longrrange interactions in vision transformers. Advances Neural Information Processing Systems, 34, 2021b.
Shusheng Yang. Xinggang Wang. Yu Li, Yuxin Fang, Jiemin Fang, Wenyu Liu, Xun Zhao, and Ying Shan. Temporally efficient vision transformer for video instance segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2885-2895 2022b.
Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov, A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10809-10818, 2022.
Fang Yu, Kun Huang, Meng Wang, Yuan Cheng, Wei Chu, and Cui. Width depth pruning for vision transformers. In AAAI, pp. 3143-3151, 2022.
Lu Yu and Wei Xiang. X-pruner: explainable pruning for vision transformers. In IEEE Conf. Comput. Vis. Patlern Recog., pp. 24355-24363, 2023.
Shixing Yu, Tianlong Chen, Jiayi Shen, Huan Yuan, Jianchao Tan, Sen Yang, Ji Liu, and Zhangyang Wang. Unified visual transformer compression. In International Conference on Learning Repre- sentations, 2021.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.11986, 2021.
Xiaoyu Yue, Shuyang Sun, Zhanghui Kuang, Meng Wei, Philip HS Torr, Wayne Zhang, and Dahua Lin. Vision transformer with progressive sampling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 387-396, 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceed- ings of the IEEE/CVF international conference on computer vision, pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 17Xiv:7110.09112. 2017.
Jinnian Zhang, Houwen Peng. Kan Wu, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Minivit: Compressing vision transformers with weight multiplexing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12145-12154, 2022a.
Yuxin Zhang, Mingbao Lin, Zhihang Lin, Yiting Luo, Ke Li, Fei Chao, Yongjian Wu, and Rongrong Ji. Learning best combination for efficient n: M sparsity. In Adv. Neural Inform. Process. Syst.,
pp. 941-953, 2022b.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmen- tation. In Proceedings of the AAAI conference on artificial intelligence, 2020.
Mingjian Zhu, Kai Han, Yehui Tang, and Yunhe Wang. Visual transformer pruning. arXiv e-prints, pp. arXiv-211 -104, 2021.
14

---

Published as a conference paper at ICLR 2024
A BOUND PROOF
In this section, we provide proof for the upper bound of pruning process:
| (a:WiLL))
(2:1-11-1) w((t
|
sup |
x [ - W())
(G)
(11)
F
JE[1,]]
Inspired by Neyshabur al. 2015 its layers:
Lee al. (2020), write the network / (a; W(1:)) with
(1; ((;)) = Wio (Wi.-10 (Wr.-2 (Wix))))))))))
(12)
Then we can peel the highest layer WL:
r x; w(ni)) - (o:; w(it-1) W() m
(13)
= =Il((
(z; (0 (((L-1)) -o
a (5 (x; W(1:-11) Wi, w((+ 1|2
With Cauchy- Saucch-Schwarz inequality Steele 2004), we have the upper bound of Equation 13 as:
[ r 11o (5 (z: w((l-1) - a (f x: W(L:-1), W00 )(
(14)
Suppose is the ReLU activation, lg norm for the upper bound of Equation 14]a::
the I-Lipschitzness of ReLU activation with respect to
|
||s (z: (() f (z: (z: ((L--1) W( -
(15)
Then we continue peeling the second highest layer WL-1 to lower layers and stop the peeling at the pruned layer ( We have this upper bound for Equation 13
| cx w(ll)))) - (2; ((1-1)) W(), w(((
(12) )
(16)
S 1
| (a; w(aw)) f (e:W(attt,,TWt
We consider the effect pruning for the term on the right side and use Cauchy-Schwer inequality again:
|s (1:W)) - (z; (z: ((tt-) w 111 = =1 (w -
-1(w a (5 (z; (((tt-)))
(17)
$ - ( (1=2
For the activation term, based on o0)) = O, have:
| (f (z: w(( - (1:
< -0
-00)
(
- 0// - -1|s (z: w(atti) =2)((=:Wauu)))l
(18)
If continue the peeling process shown in equation 13 we can achieve equation
II
B MODULE DEFINITION
Attention-related Module (QKV-M and PRJ-M). Self-attention is an important operation in Swin Transformer. In every transformer block, there are two layers related to self-attention, that is, ATT- QKV and ATT-PRJ in Figure? ATT-QKV means the "Query, Key, and Value" matrix for self- attention, and ATT-PRJ indicates the projection layer for self-attention. Assuming z' is the feature map of Ltp, layer, two attention-related layers are:
15

---

Published as a conference paper at ICLR 2024
1 ATT-QKV (z)), z = ATT-PRJ (2)
(19)
where z' is the input feature map of ATT-QKV, and z z is the input feature map of ATT-PRJ. After these two layers, there is a residual connection between the output of the ATT-PRI layer and the feature map before the LN layer:
2 22 2
(20)
Assume there are J transformer blocks in the whole network. For the transformer block shown in Figure 3[a), assume the range of the layer index [t, 1+ 5]. For the jen. block after this block, the layer index range is 11 + Pj; 1 P5 5], where Pj means the number of layers between these two blocks. Then we can define QKV-M and PRJ-M as:
QKV-M { W W . for jE [L,]]. PRJ-M : [ W :...] for jE [1.].
(21)
Multilayer Perceptron-related Module (MLP-M). Another important part of the Swin Trans- former is the multilayer perceptron. There are two MLP layers in every Swin Transformer block. We name these two layers MLP-FCI and MLP-FC2:
++ MLP-FCI (z/+)), z'+5 = MMP--FC
(22)
where + is the input feature map of MLP-FCI, and 2/4 is the input feature map of MLP-FC2. Similarly, after two MLP layers, there is residual connection between the output of the MLP-FC2 layer and the feature map before the LN layer:
2 15 +/2
(23)
Considering J blocks in the whole network, and j E [1, J], we can define the MLP-M as:
MLP-M: (Wi+1 w ..... W w..t.tts......
(24)
Auxiliary Module (AUX-M), The auxiliary module consists of the auxiliary layers inside and out- side the Swin Transformer block. Inside the Swin Transformer block, shown in Figure there are two LN layers. Some other auxiliary layers outside the Swin Transformer blocks, including a patch embedding layer and several patch merging layers, are also included in AUX-M. We do not prune these layers due to their important contribution to the network and their relatively small parameter count.
16


Reviews:
[{"review_id": "oMiEXX9wXo", "reviewer": "Reviewer_qfKu", "summary": "Hierarchical Vision Transformers (ViTs) offer linear computational complexity concerning image size and create hierarchical feature maps. To solve the aforementioned issues, the authors  have developed a Data-independent Module-Aware Pruning method (DIMAP) to compress hierarchical ViTs. To ensure that \"local\" attention weights at different hierarchical levels are compared fairly in terms of their contribution, they treat them as a module and examine their contribution by analyzing their information distortion. Furthermore, they introduce a novel weight metric that is solely based on weights and does not require input images, thereby eliminating the dependence on the patch merging process. The experiments are fine to me.", "questions": "To me the extensive applications will be beneficial to be new contributions of field.\nCould you discuss more the new applications of your method.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The authors introduce a new pruning approach called Data-independent Module-Aware Pruning (DIMAP). DIMAP compares \"local\" attention weights fairly by treating them as a module and examining their contribution based on information distortion. A new weight metric that doesn't need input images is also introduced, making it data-independent. The method has been validated on Swin Transformers, demonstrating its effectiveness and efficiency.", "weaknesses": "To me the extensive applications will be beneficial to be new contributions of field.\nCould you discuss more the new applications of your method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KfRRk0zXRp", "reviewer": "Reviewer_uDum", "summary": "Traditional pruning methods ignore the unique properties of hierarchical ViTs by utilizing magnitude as the weight importance, which leads to issues in weight pruning at both local and global scales. DIMAP addresses these challenges by treating attention weights at different hierarchical levels as a \"module\" and examining their significance by analyzing their information distortion. The proposed method demonstrates superior performance on Swin Transformers when compared with other approaches, especially in terms of accuracy and efficiency in the pruning process.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The idea is somewhat novel. The unique properties of hierarchical ViTs leads to the proposing of a module-aware approach for pruning hierarchical ViTs.\n2. The paper is well-structured, with clear explanations of the challenges with existing pruning methods, the introduction of their novel method, and mathematical derivations supporting their approach.\n3. The experiments are good. The top-5 accuracy drop is only 0.07% when we removing 52.5% FLOPs and 52.7% parameters of Swin-B.", "weaknesses": "1. Why it's called \"data independent\" since the weights are updated with training data?\n2. A comparative study with existing pruning methods, including their weaknesses when applied to hierarchical ViTs, could offer more insights into the significant advantages of DIMAP.\n3. More information on the implementation specifics, hyperparameters used, and other nuances would be helpful for reproducibility and understanding the intricacies of DIMAP.\n4. It's not clear that uniform pruning means pruning parameters uniformly or pruning FLOPs uniformly?\n5. While the method has been validated on the Swin Transformer architecture on ImageNet-1k, it would be beneficial to see its performance on other Vision Transformer architectures or datasets to establish its versatility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E4Ds1J7erB", "reviewer": "Reviewer_VGj7", "summary": "The paper introduces the Data-independent Module-Aware Pruning (DIMAP) for hierarchical vision transformers (ViTs). Instead of relying on magnitude values like traditional pruning techniques, DIMAP evaluates \"local\" attention weights in hierarchical ViTs by their impact on information distortion. By examining weight distribution at different hierarchical levels, DIMAP offers an alternative to magnitude-based methods. Tests on Swin Transformers highlight its efficiency, with a small decrease in top-5 accuracy and notable reductions in FLOPs and parameters.", "questions": "The computational cost and threshold explained in weakness section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper presents DIMAP, a unique pruning method for hierarchical ViTs. Instead of the prevalent focus on weight magnitudes, DIMAP evaluates attention weights based on their effect on information distortion, setting it apart from standard methods.\n\n2. The approach is grounded in solid theory, with mathematical models reflecting a comprehensive grasp of the issue. The choice to use distortion analysis as a measure of weight significance stands out.\n\n3. The paper's organization is logical, detailing the obstacles and answers related to pruning hierarchical ViTs. Its equations, coupled with their explanations, offer transparency and ease of understanding for readers.\n\n4. Given the rising importance of hierarchical ViTs in computer vision, the paper's relevance is significant. Efficient pruning is vital for these transformers in limited-resource scenarios. DIMAP, focusing on maintaining crucial attributes while cutting down computations, meets a current community demand.", "weaknesses": "Complexity: While the module-aware method proves efficient, it could complicate the pruning process, particularly for extensive models or vast datasets. How much computational burden does this additional step bring?\n\nThe threshold determination is at the module level. Could this be determined at the model level instead? What differentiates these two levels of thresholding?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)