SAViT: Structure-Aware Vision Transformer Pruning via Collaborative Optimization
Authors: Zheng Chuanyang
Zheyang Li
Kai Zhang
Zhi Yang
Wenming Tan
Jun Xiao
Ye Ren
Shiliang Pu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Vision Transformer
Pruning
Compression

Abstract: Vision Transformers (ViTs) yield impressive performance across various vision tasks. However, heavy computation and memory footprint make them inaccessible for edge devices. Previous works apply importance criteria determined independently by each individual component to prune ViTs. Considering that heterogeneous components in ViTs play distinct roles, these approaches lead to suboptimal performance. In this paper, we introduce joint importance, which integrates essential structural-aware interactions between components for the first time, to perform collaborative pruning. Based on the theoretical analysis, we construct a Taylor-based approximation to evaluate the joint importance. This guides pruning toward a more balanced reduction across all components. To further reduce the algorithm complexity, we incorporate the interactions into the optimization function under some mild assumptions. Moreover, the proposed method can be seamlessly applied to various tasks including object detection. Extensive experiments demonstrate the effectiveness of our method. Notably, the proposed approach outperforms the existing state-of-the-art approaches on ImageNet, increasing accuracy by 0.7% over the DeiT-Base baseline while saving 50% FLOPs. On COCO, we are the first to show that 70% FLOPs of FasterRCNN with ViT backbone can be removed with only 0.3% mAP drop. The code is available at https://github.com/hikvision-research/SAViT.

SAViT: Structure-Aware Vision Transformer Pruning via Collaborative Optimization
Chuanyang Zheng', Zheyang Li Kai Zhang's Zhi Yang', Wenming Tan', Jun Xiao², Ye Ren', Shillang Pu²
Hikvision Research Institute, Hangzhou, China Zhejiaan University, Hangzhou, China fzhengchuanyang, lizheyang, zhangkai23, yyhhggahhi1hhhiiiiiiiio tanwermiing@@-kkviisioom com, junxdcs zju. edu.cn Crenye,pushille hww.hhikiision.ccoommmmm
Abstract
Vision Transformers (ViTs) yield impressive performance across various vision tasks. However, heavy computation and memory footprint make them inacces- sible for edge devices. Previous works apply importance criteria determined independently by each individual component to prune ViTs. Considering that heterogeneous components in ViTs play distinct roles, these approaches lead suboptimal performance. In this paper, we introduce joint importance, which integrates essential structural-aware interactions between components for the first time, to perform collaborative pruning. Based on the theoretical analysis, we construct Taylor-based approximation evaluate the joint importance. This guides pruning toward more balanced reduction across all components. To further reduce the algorithm complexity, we incorporate the interactions into the opti- mization function under some mild assumptions. Moreover, the proposed method can be seamlessly applied to various tasks including object detection. Extensive experiments demonstrate the effectiveness of our method. Notably, the proposed approach outperforms the existing -fate-of-the-ar approaches on ImageNet, in- creasing accuracy by 0.7% over the DeiT-Base baseline while saving 50% FLOPs. On COCO, we are the first to show that 70% FLOPs of Faster R-CNN with VIT backbone can be removed with only 0.3% mAP drop. The code is available at btaaps://itthan www.wikvwsiiiioooooooooooooooooo -ee.eiiccrreeeaacSS//SSSaa.
Introduction
Convolutional neural networks (CNNs) have dominated nearly every aspect of computer vision for long time [1]. Recently, the emerging ViT's [2, 4, 4, 6, 7] have shown competitive performance on image classification, object detection, and other vision tasks. Despite the success, ViTs contain complicated submodules and require more intensive computational cost [8], limiting their deployment on resource-restricted devices. This naturally calls for compression and acceleration in ViTs those done in CNNs.
One common direction for speeding up deep neural networks is to remove a group of less important connections that have a negligible impact on the network's ' performance. As pruning is proven very powerful accelerate CNNs [9, 10, 11. 12], the exploration of pruning ViTs has just emerged. Different from CNNs whose parameters mainly rely on the homogencous component, i.c., convo- lutional filter, transformers contain heterogeneous components such multi-head self-attention (MSA), hidden neurons, and embedding neurons, as shown Figure These components play different functional roles in capturing contextual information and each one is associated with distinct
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

drp xd
dx
x
d
Linear
Scaled Dot Product Attention
x hd
d ds!
Head
Layer LayerNorm Norm
Linear
Concat
Lipey Norn
Head
Linear
PPP
bpv
x
Head
Linear
ht
Prunable Heads
ddy
Prunable Hidden Dimensions
Prunable Embedding Dimensions
Figure I: The illustration of prunable components in ViT block. da denotes the feature dimension in each head. ViTs consist of multiple repeated blocks.
structure. This makes pruning ViTs more challenging. Many previous approaches focus on trimming
either heads [13] or patches [ [14, 15), while neglecting the time-consuming embedding component that accounts for the majority of redundancy shown in our analysis. Most recent works [16, 17] aim at pruning multiple components in ViTs and gain better acceleration than pruning a single component. However, this line of methods ignores complicated interactions between components, leading to suboptimal compressed models. As pointed out by some studies [18] that worked on the interpretability of DNNs, interactions make meaningful contributions the inference. Thus, interactions should be taken into account to minimize the impact of pruned parameters in ViTs.
In this work, explore efficient VITs through pruning all components comprehensively, which offers better flexibility to search for the optimal subnetwork in parameter design space and potentially reaches maximum acceleration for ViTs. Since all components collaborate with each other to achieve superior modeling capability instead of playing individually, we propose to quantitatively analyze the joint importance of pruned components, which contains the individual importance of each component as well as the interactions between components. Although preceding approaches including OBD [19] and OBS [20] have studied the interactions in pruning CNNs, the contribution of interactions is limited in CNNs [10], which consist of homogeneous components. Nevertheless, this is not the case for ViT. In this paper, we investigate the interactions between components and demystify that interactions are crucial in heterogeneous ViTs. Specifically, we propose to exploit the Hessian matrix to capture the interactions between components. To reduce the high computational cost of the Hessian matrix and facilitate the proposed algorithm for more applications, we encode the interactions represented by the Hessian matrix into pruning ratios of different components efficiently approximate the optimization target. Finally, the Evolutionary Algorithm [21] is leveraged to solve the optimization problem. The proposed collaborative pruning algorithm is fast and consumes less than epochs on the whole training dataset. In addition, the pruning framework can be scamlessly extended to accelerate more complicated networks such as detection networks. For example, by regarding the neck and head as new components, can prune all components in the detection network together. We summarize our main contributions as follows:
To the best of our knowledge, this work is the first to explicitly incorporate the interactions between different components into pruning ViTs.
Based on the theoretical analysis, we transform the joint importance into an approximated optimization target to prune efficiently.
We conduct a systematical analysis of ViTs and propose a comprehensive framework for pruning all components in ViTs by solving optimization problem. The framework can also generalize well to more complicated networks such as detection networks, giving it the potential for widespread applications.
The experimental results on various models and tasks demonstrate that the proposed approach brings great acceleration while preserving excellent performance. On ImageNet, our method can achieve a surprising 0.7% accuracy increase over DeiT-Base while reducing 50% FLOOP.. On COCO, 70% FL.PPs of the Faster R-CNN with Swin-T backbone can removed with only 0.3% mAP drop.

---

2 Related Work
2.1 Vision Transformers
Transformer [22] is originally designed for natural language processing (NLP) tasks. It relies on self-attention to capture long-term dependencies and has long been dominant preference in NLP. Recently, the pioneering work of ViT [2] has demonstrated that directly applying a pure transformer to sequence of image patches brings exciting results on various image classification benchmarks with large-scale pre-training. The powerful modeling capability from ViT spawns many novel vision transformer models in computer vision. DeiT [23] makes of bunch of training techniques and particularly proposes a distillation procedure to release the ViT from heavy dependency on large datasets, outperforming the original ViT as well. Later, PVT [24] and Swin [3] introduce the pyramid structure into transformers to generate multi-scale feature maps, making it a unified backbone for multiple downstream vision tasks. Subsequently, many vision transformers [5. 25, 3, 6, 24, 26] are presented to improve the performance and achieve astonishing results on multiple benchmarks. Until now, ViTs have been widely applied across various vision tasks, including image classification [3. 7], object detection [3, 4J, segmentation [27], point cloud [28], and so on. However, the computational cost of ViTs is still intensive and scales up quickly as numbers of MSA heads, embedding width [29].
2.2 CNN Pruning
Extensive efforts have been conducted to prune CNNs for better efficiency via weight magnitude [30], filter norm [31], scaling factor [32]. learned performance proxy [12], etc. More relevantly, some works utilize Taylor expansion to approximate the loss of pruning. OBD [19] and OBS [20] aim to prune weights with the least error approximated by second-order Taylor derivatives, which is intractable for millions of parameters in modern deep models. [33] introduces an importance metric based first-order Taylor expansion, avoiding the expensive computational cost of the Hessian matrix. L-OBS [9] proposes a layer-wise pruning method using a criterion based on second-orde -order derivatives of a layer-wise error function. CCP [34] analyzes the layer-wist overall impact of pruned channels based on the second-order Taylor expansion when pruning each layer. [10] measures the importance by squared difference of prediction errors and approximates it using Taylor expansion. In experiments, they show that the Hessian matrix makes negligible improvements, which indicates that the interactions are limited in pruning CNN.
2.3 Transformer Pruning
Compared to pruning CNNs. pruning transformers still in the early stage. Some previous works put efforts into pruning different substructures, including attention head [13], basic block [35]. The recent advances in ViTs motivate many researchers to design specific methods for ViTs. Instead of training full ViTs, [8] integrates sparse training into the transformer to train the smaller subnetwork from scratch. Another line of works [14, 36, 37, 38) extract data-related redundancy and remove unnecessary image patches. Unfortunately, pruning patches breaks down the spatial structure of ViTs. Recently, NVP [16] first shows that pruning all components reaches better acceleration. UP [17] considers KL-Divergence change for each parameter a proxy dataset, using the same compression ratio in all blocks. ViT-Slim [39] designs differentiable soft masks on each component and imposes l1 sparsity constraint to force the mask to be sparse. Different from the hand-crafted pruning ratios in prior works, we first quantitatively analyze the interactions between components and adaptively learn the proper pruning ratio for each component by solving the optimization problem, offering the best flexibility to find the optimal subnetwork in a more collaborative manner.
3 Method
In this section, based on the relationship between computational cost and prunable VIT components, we present the scheme of pruning all components in ViTs comprehensively.
3.1
Components Analysis
To reveal how each component affects the computational cost of VITs and analyze the trade-off between accuracy and FLOPs reduction for the network, we conduct experiments pruning the
3

---

Table Computation analysis. The last two columns give examples of FLOPs in the practical DeiT and Swin model. n is the number of patches, d,, is the feature dimension of each head.
Part
Computation
DeîT-B
Swin-B
MSA Projects
3ndhdn, + thhdad
5.57G
4.91G
MSA attention
2nPhdp, 2ndd
0.72G
0.31G
FFN
11.2G
9.87G
Total
2nhdj, (2d + n1) +2ddgs
17.6G
15.4G
commonly used DeiT. Figure 2a clearly shows that pruning single component results in the unsatis- factory performance pruned models. Meanwhile, each component makes distinct contributions to the network. Furthermore, we break down the computation of VIT into three parts, while VIT variants follow a similar configuration. As shown Table 1, MSA and Feed- Feed-Foowwar Network (FFN) account for 36% and 64% computation respectively. Combining with the architecture characteristics illustrated by Figure 1, structural pruning can be applied to three components to accelerate ViTs: 1) attention head, which only affects MSA computation; 2) hidden dimension, which only affects FFN; 3) embedding dimension, which is shared across both MSA and FFN. Despite embedding dimension being responsible for the largest amount of computation, it is more tricky evaluate the importance of embedding dimension than other components. The reason behind this lies that embedding dimensions of different linear layers entangled due to skip connection. Consequently, collaboratively pruning all components needs to handle the difficulty in evaluating the importance of distinct components.
3.2
Collaborative Pruning
Based on the above observations, our collaborative pruning scheme adjusts all components together to achieve better trade-off between accuracy and latency. Given ViT model, apply structural pruning on all components by removing unnecessary parameter groups that cause a minimum performance drop the network. The performance drop can be reflected by the perturbation in the loss function L. Inspired by OBD [19], we use the Taylor expansions to construct an approximated expression of the loss function and analytically estimate the joint influence of pruning. Formally, define collaborative pruning as kind of perturbation Aw on the whole weight vector w, under certain computational cost constraint Chudget (e.g. FLOPs). The optimal subnetwork can be found through the following optimization problem:
min AL C(w + Aw) - L(wr), Am
s.t. C(w + Aw) S Coudget
where C(w Am) represents the computational cost of the pruned model and Aw b @ u w. The binary mask indicates whether each weight should be pruned and @ is Hadamard product, b, 0 means the i-th weight should be pruned. AL denotes the difference of loss function before and after pruning, which can be further formulated as follows:
AC Aw" g + Aw= HAW + 0(lAw|))),
where g and H are the gradient vector and the Hessian matrix w.r.t w.
According quadratic approximation [19], the third term in Equation 2 can be neglected. Generally, the first term Aw g measures the impact of each parameter group playing individually. As we employ structural pruning, the first term can be further rewritten the sum over perturbation of all pruned parameter groups:
Aw g E1.
where 7, - Eier Wigs is the individual importance for the pruned parameter group. S is the set of all prunable parameter groups. w; and gi is the weight and gradient for the i-th parameter respectively in the parameter group 8. And the Awo g can be easily computed since gi is already available from backward propagation.

---

Head
1921 Acc
Hicken Neurons
70
76
Baseline
Embeddingg
Hoad only
Neurons
74
Fmbeddiin only
Comprehensive
Cross-Compomertt
0.0
0.1
0.5
3.6
0.7
0.0
Interactions
KUPS reduction ratio
(a)
(b)
Figure 2: (a) Accuracy of models applied with different pruning strategies on ImageNet. The baseline DeiT-Base (FLOPs reduction ratio 0). X-only means model is pruned along a single X component. "Comprehensive" means employing pruning across all components. (b) An example illustration of the full Hessian matrix. Green blocks indicate the interactions between components. Other blocks mean the interactions within the corresponding component.
The second term in Equation 2 provides rich knowledge about the interactions between components, which is essential for pruning ViTs. However, the brute-force method suffers from huge memory space and computational cost required by the large-scale full Hessian matrix in Equation 2. This is infeasible on many devices and restricts the application of our method.
To tackle this problem, we propose more efficient approximation. Due to the different roles that different components play in ViT, we split all interactions encoded by the full Hessian matrix into intra- component interactions (blue, orange, yellow parts in Figure 2b) and inter-component interactions (green parts in Figure 2b). Correspondingly, Aw during pruning can be partitioned into weight pertur- bations within component of head, hidden neurons and embedding neurons (1) Au (2), (1) Substituting the partitioned weight perturbations into the second term in Equation 2, we have
3
1 - HAW
Σ (k) 2 ArIml
1
H O
where H(h) is the Hessian block matrix that describes the interactions between component and component For example, (II)), illustrated by the blue block in Figure 2b, represents the interactions within heads, while the green block H (12) indicates the interactions between heads and hidden neurons.
Denoting all the learnable weights belongs to component & by (k), we re-formulate each Hessian block in Equation 4 as:
(
Aw (K) H (kl)
Aw
Σ
(kl) (k) (kl) (1) m
w, bij w
i=1
(T(
(O)
M
u
H w()
(5)
(kl) where (1k) and (1)) are the total number of parameters in component and respectively. m the mask of the interaction between the i-th parameter in component k and the j-th parame- (kl) means the interaction is added into weight perturbation due to
ter in component mij
(kl)
pruning of the corresponding parameter and m
means the interaction should be neglected. (R) H ((() denotes the mean of Hessian block correspond to the pruned network.
Without loss of generality, every pruning process can be regarded as an independent weight sampling from the network. Thus the Hessian block can be transformed into the following expression:
Aw (k) H (k)) ( = (()) (()) (k)) W H
(6)

---

where (RI) denotes the mean of the mask matrix, w(kt) = w((k) (k)) is the mean of the full Hessian block matrix term, this paper, computed by Monte Carlo method [40] to further reduce computational cost.
During pruning, (kk) is generated by:
M = (1 - (1 b(0) = b(k)) [ b(OjT,
(7)
where (k4) denotes the binary mask vector for the k-th component and 6(4)) is identical the pruning ratio. We define the pruning ratios of attention head, hidden, embedding by p()) p(2) and p(3) respectively:
(k))
1
-
((k) lo k 112=,,2,3. 1,2,3,
((k)
where || llo is the fo-norm, ((k is the number of all parameters in component k. So (R) is equal to plb) ,!)), and we can simplify the interactions as:
x( M (k) w((k H (b0) (()) NO p+0 p00) (R0)
(10)
Therefore, the loss perturbation in Equation 2 can be approximated as:
3
AC ~ Aw" 9 +
N( (k)) (
(11)
k11=
As described above, the Aw can be partitioned into weight perturbations along each component: Au (1) A2) (3) Summarizing all above, the final objective of collaborative pruning that minimizes the joint importance can be rewritten as follows:
1 2 Σ
min pll) pl p()) p(3)
Aw g
s.t. C(w + Aw) & Chudget;
(w) = (r) k == 1,2,3.
(12)
Here Prune represents an operation to prune weights within each component individually. Specif- ically, for the k-th component with pruning ratio (()), the Prune will update the mask of the least important (R) p(k) weights as zero. Correspondingly, weight perturbation (k) becomes b(k) @ (k) - (k), To evaluate the importance of each weight, we apply the Fisher information metric [11]:
SC. Ow;?
1 N
acn W4
2
(13)
Ows
where N is the number of samples. Finally, the whole optimization problem can be solved by Evolutionary Algorithm (EA) [21]. We provide complete view of the collaborative pruning in Algorithm 1. When dealing with the more complicated detection network, we regard the neck and detection head as new components. Then collaborative pruning leverages the interactions involving all components to trim down the network.
Relations to CNN pruning methods using the Hessian matrix As described Section 2.2, L-OBS [9] and CCP [34] leverage the second-order derivatives for pruning homogeneous CNNs. However, they both fail to deal with networks containing multiple heterogeneous components such as ViTs since the structural characteristic of each component is quite different and the importance of different components is incomparable. Simply applying them on ViT's will cause non-optimal pruning ratios for different components. Furthermore, these layer-wise pruning methods discard important relationships between layers, for example, the relationship between MSA and FFN. The layer-wise wise design also needs to set pre-defined ratios for each layer, which not optimal. Going beyond both, we establish the benefits of Hessian as a tool to capture the essential interactions across different components and different layers in pruning ViTs. Through solving the optimization problem, our work adaptively learns optimal pruning ratios for each component.

---

Algorithm Collaborative Pruning with EA
Input:
Pre-trained model To. FLOPs constraint Obudget, dataset D, search iterations E, population size Q, components number M, fitness value f
Output: Optimal pruned model
1: Initialize samples 9; (p,
(M)
Pj
2: Calculate importance / for all structural groups according to Equation 13 and (tt), i,j
1, ...,..,,M .M according to Equation 6 on D
3: for iteration e 1 E do
4: for sample = 1 to Q do
5: (Q:)) < Cuddget then
6: fi 0, continue
7: end if
8: for pruning ratio 0 in current sample q do
9:
Prune the corresponding component j according to importance I
10: end for
Compute the f according to Equation 11 12: end for
13: Keep the Top-k fittest samples
14: Generate new samples by Crossover and M utation
15:
end for
16:
return Optimal pruned model
4
Experiment
4.1
Pruning DeiT
In this section, first analyze the performance of the proposed pruning method on the DeiT family of different model sizes, i.e., DeiT-Base/ SeTT-Base/Small/iny Then we show the benefits of pruning equipped with knowledge distillation.
The pruning process is performed on the pre-trained DeiT released from official implementation on ImageNet-Ik [41]. As our method is a one-shot method, the whole pruning process is performed fast on a single GPU, After pruning, we fine-tune the pruned network using the same setting as DeiT [23] without warm-up.
Implementation Details
Results We present summarized results on Table 2. On DeiT-Base, SAVIT can steadily outperform previous state-of-the-art works under various complexity settings. Impressively, when reducing the model size by 50% FL.OPs, SAVIT achieves 0.32% improvement with 17% fewer FLOPss than ViTTE and even surpasses the baseline by 0.7%. When we further increase the compression ratio to 70%, our approach still obtains competitive performance with only a O. 18% accuracy drop, outperforming the recently proposed UVC by 1.1%. The strong performance indicates that SAVIT benefits a lot from the interactions. On the other hand, the pruned models by our method show better trade-off between accuracy and efficiency than hand-crafted models well models based on Neural Architecture Search (NAS). For example, our compressed model obtains higher accuracy than T2T VIT-24 with fewer FLOPs. A similar observation holds for DeiT-Small, where our method outperforms SSP and ViTE by large margin. Furthermore, the proposed method also achieves the best t accuracy when pruning smaller DeiT--Tiny. The consistently remarkable performance of pruning DeiT family demonstrates the proposed collaborative pruning algorithm works very well with different model complexity, suggesting strong potential for widespread applications.
4.1.1 Knowledge Distillation
For better performance, we also investigate fine-tuning with knowledge distillation. Using Deit- Base-Distilles [23] as baseline, we prune into different model size, i.e., SAVIT-B/S/T, shown in Table 3. Then we fine-tune the pruned models with a simple knowledge distillation strategy. The fine- tuning configuration is detailed in Appendix B. Table 3 lists the results of our method and previous

---

Table 2: Results of pruning DeiT Base/Small/Tiny on ImageNet-Ik dataset. We compare the parameters, FLOPs, and Top-1 accuracy of the pruned model with various models. X-600e means training the baseline model for longer schedule, i.e., 600 epochs following the original recipe in [23]." means hand-crafted models with comparable FL.OPs. means NAS-based models. Others are pruned models.
Model
Param.(M)
(4%)
FLOPP((G)
(4%)
Top-1 Acc. (%)
A
DeîT-B
86.6
17.6
81.84
DeiT-B-600e
86.6
17.6
82.01
+0.17
T2T-ViT-24" [42]
64.1
26.0
13.8
21.6
82.30
+0.46
PVT-L" [24]
61.4
29.1
9.8
44.3
81.70
-0.14
AutoFormer-Bt [43]
54.0
37.6
11.0
37.5
82.40
+0.56
SSP-B [8]
56.8
34.4
11.8
33.1
80.80
-1.04
ViTEEB [8]
56.8
34.4
11.8
33.1
82.22
+0.38
Evo-ViT [36]
86.6
0.0
11.7
33.5
81.30
-0.54
EViT-DeiT-B [38]
86.6
0.0
11.6
34.1
81.30
-054
Dynamic VIT [14]
86.6
0.0
11.5
34.7
81.30
-054
ViT-Slim [39]
52.6
39.3
10.6
39.8
82.40
+0.56
SAVIT (ours)
51.9
40.1
10.6
39.8
82.75
+0,91
vTP-B [44]
47.3
45.4
10.0
43.2
80.70
-1.14
PS- DS-ViT-B [37]
86.6
0.0
9.8
44.3
81.50
-034
SAViT (ours)
42.6
50.8
8.8
50.0
82.54
+0.70
UVC [45]
8.0
54.5
80.57
-1.27
SAVIT (ours)
25.4
70.7
5.3
69.9
81.66
-0.18
DeiT-S
22.1
4.6
79.85
DeiT-S-600e
22.1
4.6
80.02
+0.17
AdaViT-S [15]
22.1
0.0
3.6
21.7
78.60
-1.25
Dynamic VTT [14]
22.1
0.0
3,4
26.1
79.60
-0.55
EViT-DeiT-S [38]
22.1
0.0
3.0
34.8
79.50
-0.35
SSP-S [8]
14.6
33.9
3.1
31.6
77.74
-2.11
S2ViTE-S [8]
14.6
33.9
3.1
31.6
79.22
-0.63
SAVIT (ours)
14.7
33.5
3.1
31.7
80.11
+0.26
DeiT-T
5.7
1.3
72.20
SSP-T [8]
4.2
26.3
0.9
23.7
68.59
-3.61
S'VITE-T [8]
4.2
26.3
0.9
23.7
70.12
-2.08
SAVIT (ours)
4.2
25.2
0.9
24.4
70.72
-1.48
compression methods. As expected, assembling knowledge distillation boosts the performance of SAVIT. Notably, SAVIT achieves 77.0% accuracy with only 1.3G FLOPs, substantially reducing DeiT-Base-Distie -iistillleee computational cost by 92.6% FLOPs and surpassing the distillation-only DeiT- Tiny-Distilled well NVP-T. This means heavily pruned model from larger model achieves better accuracy and indicates the necessity of pruning, consistent with the previous observations [47]. Besides, SAViT substantially improves accuracy other compression methods.
4.2 Pruning Swin Transformer
To verify the generalization our method, we also conduct experiments on more challenging ViT, i.e., hierarchical Swin Transformer. It adopts the local windows attention mechanism to reduce the computational cost for downstream tasks and becomes a general general-purpose purpose backbone [3, 26J. We compare approach with the baseline pruning method -ormm, which is extended from CNN methods.
compression [31] and belongs to importance-bart
Implementation Details Similar to DeiT, we perform pruning the official pre-trained Swin? on ImageNet-1k. Finally, we fine-tune the pruned network for 300 epochs under the same strategies as Swin [3].
hhun::/iiittbbccom//ncrrotttsiis Transfommer

---

Table 3: Results of combining pruning and knowledge distillation for compressing DeiT-Base- Distilled.
indicates the pre-trained baseline from DeiT family [23]. UP-DeiT and NVP adopt pruning before knowledge distillation. The unpruned DeiT-B-Distille used as the teacher for distilling SAVIT-B/S/T.
Model
Param.
FLOPs
Top-1 Acc.
86.6M
17.6G
83,36
NVP-B [16]
34.0M
6.8G
83.29
SAViT-B (ours)
33.6M
6.7G
83.31
DeiT-S-Distilles
22.1M
4.6G
81.20
Manifold [46]
22.1M
4.6G
81.48
UP-DeiT-S [17]
22.1M
81.56
NVP-S [16]
21.0M
4.2G
82.19
SAVIT-S (ours)
20.6M
4.2G
82.38
DeiT-T-Distilled
5.6M
1.3G
74.50
Manifold [46]
5.6M
1.3G
75.06
UP-DeiT-T [17]
5.7M
75.79
NVP-T [16]
6.9M
1.3G
76.21
SAViT-T (ours)
6.6M
1.3G
76.95
Results Table compares the results of our approach with the baseline pruning methods. Pruning Swin more challenging since the elaborately designed hier- archical architecture has better pa- rameter efficiency and less redun- dancy. Our approach can reduce the computational costs by 50% with slight influence, consistently surpass- ing the baseline pruning method E norm. The impressive performance of pruning Swin Transformer again demonstrates the superiority of the proposed pruning algorithm.
Table 4: Results of pruning Swin-Base on ImageNet-Ik
dataset.
Model
Method
Param.
FLOPs
Top-1 Acc.
Baseline
87.8M
15.4G
83.51
Swin-B
f []-nor[[3]]
36.0M
7.8G
80.95
ours
33.0M
7.7G
82.62
4.3 Pruning Detection Network
Since the detection network consists of more components such as neck, it becomes harder to prune such complicated architecture. Benefiting from the collaborative optimization of our method, the proposed approach can be easily transferred accelerate the detection network by adding the neck and detection head as new components, We employ pruning on the popular object detection framework Faster R-CNN [48] with Swin-Tiny backbone on COCO 2017 dataset [49] and report mean Average Precision (mAP) for comparison. To the best of our knowledge, we are the first to apply pruning to detection networks with ViT as the backbone. Besides, we extended -norm pruning our baseline method for comparison.
As noticed in Table 5, when target- ing considerable compression ra- tio (i.e., 58% parameters and 70% FLOPs reduction). our approach only has slight mAP drop. outperform- ing the l norm by a large margin, e.g. 3.1% % mAP. This indicates that the proposed method can capture the interactions between multiple com- ponents in more complicated network and excavate the redundancy well.
Table 5: Pruning Faster R-CNN detection network on COCO.
Model
Method
Param.
FLOPs.
mAP
Baseline
45.2M
221.7G
45.5
Faster R-CNN
no-oom[[3]]
21.7M
67.8G
42.1
(Swin-')
ours
19.2M
67.8G
45.2

---

4.4 Ablation Study
Latency Measurement We strictly measure the lateney of compressed ViTs using CUDA bench- mark mode, please see Appendix for details. Table shows that the proposed method can bring 0.7% accuracy gains when compressing the DeiT-Base achieve 2.0x FLOPs reduction and 1.55 inference speedup. More significantly, a larger 2.05x inference speedup of DeiT-Base can be ob- tained with merely 081 loss of accuracy. We further provide operator-level speedup analysis in Appendix C. These observations suggest that our method can prune ViT's for better latency in practical applications.
Table 6: Run time speedup of compressed DeïT on Nvidia V100.
Model
Param. (x)
FLOPs (x)
Speedup (x)
Top-1 Acc.(%)
86.6M (1.00)
17.6G (1.00)
1.00
81.84
DeiT-Base
42.6M (2.03)
8.8G (2.00)
1.55
82.54
25.4M (3.41)
5.3G (3.32)
2.05
81.66
The Impact of Interactions An appealing feature of our method is the consideration of interactions between components, which helps identify redundancy accurately. To demonstrate its effectiveness, prune DeiT-Bat Base and Faster R-CNN according the optimization function with and without cross-components interactions under the same acceleration target, i.e., 70% FLOPs reduction. For DeiT-Base, fine-tune the pruned models for 80 epochs following the identical setting in Sec- tion 4.1. For Faster R-CNN, the pruned model is fine-tuned using the same setting as Section 4.3. Table 7 clearly shows that optimizing individual importance and interactions together gives the best result. Moreover, we observe that interactions help adjust all components toward a more balanced architecture. These observations suggest that interactions indeed play an important role identifying structural redundancy and can boost the performance of the pruned model.
Table 7: Ablation study of pruning with and without cross-components interactions. P1 P5 are the pruning ratios for the head, hidden dimension, embedding, Feature Pyramid Network, and detection head respectively.
Pruning Ratio
Model
Interactions
Top-1 Acc//AP
P1
P2
P3
P1
P5
X
0.13
0.12
0.69
79.68
DeiT
/
0.42
0.40
0.52
80.78
X
0.55
0.68
0.61
0.72
0.61
44.5
Faster R-CNN
/
0.55
0.53
0.71
0.66
0.52
45.2
Conclusion
In this paper, we present versatile ViT accelerating framework that collaboratively prunes all components. Based on the theoretical analysis, we construct a Taylor-based optimization function take full advantage of the interactions between heterogeneous components. As the Hessian matrix requires huge computation cost, we derive an approximation to transform the Hessian matrix into pruning ratios and achieve fast pruning. Then the optimization problem solved toward the optimal trade-off between accuracy and computational cost. We also show the proposed framework can be applied to prune more complicated architecture, c.g., detection network. Extensive experiments demonstrate that the proposed framework significantly reduces computational costs without compromising performance on various models as well as tasks.
Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China (No. U19B2043).
10

---

References
[1] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[2] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprins arXiv:2010. 1929, 2020.
[3] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021.
[4] Xiaoyi Dong. Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. arXiv preprint arXiv:2 100.0062, 2021.
[5] Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Improved multiscale vision transformers for classification and detection. arXiv preprint arXiv:2112.01526 2021.
[6] Sitong Wu, Tianyi Wu, Haoru Tan, and Guodong Guo. Pale transformer: A general vision transformer backbone with pale-shaped attention. arXiv preprint arXiv:2112.14000, 2021.
[7] Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan. Davit: Dual attention vision transformers. arXiv preprini arXiv:2204.03645 2022.
[8] Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. In NeurIPS, pages 19974-19988, 2021.
[9] Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In NeurIPS, 2017.
[10] Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In CVPR, pages 1 11264-1 11272, 2019.
[11] Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang. Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. In ICML., pages 7021-7032, 2021.
[12] Shangqian Gao, Feihu Huang, Weidong Cai, and Heng Huang. Network pruning via perfor- mance maximization. In CVPR, pages 9270-9280, 2021.
[13] Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In NeurIPS, 2019.
[14] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. In NeurIPS, pages 13937- 13949, 2021.
[15] Hongxu Yin, Arash Vahdat, Jose Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. Adavit: Adaptive tokens for efficient vision transformer. arXiv preprint 12.06521 2021.
[16] Huanrui Yang, Hongxu Yin, Pavlo Molchanov, Hai Li, and Jan Kautz. Nvit: Vision transformer compression and parameter redistribution. arXiv preprint urXiv:2100.4466) 2021.
[17] Hao Yu and Jianxin Wu. A unified pruning framework for vision transformers. arXiv preprint arXiv:211..11127 2021.
[18] Hao Zhang, Yichen Xie, Longjie Zheng, Die Zhang, and Quanshi Zhang. Interpreting multivari- ate shapley interactions in dnns. arXiv preprint arXiv:2010.05043 2020.

---

[19] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In NeurlPS, 1989.
[20] Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. In NeurIPS, 1992.
[21] Hans-Georg Beyer and Hans-Paul Schwefel. Evolution strategies-a comprehensive introduction. Natural computing, (11:3-52, 2002.
[22]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[23] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICMI., pages 10347-10357, 2021.
[24] Wenhai Wang, Enze Xie, Xiang Li. Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, pages 568-578, 2021.
[25] Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. In NewrIPS, pages 3965-3977, 2021.
[26] Qiang Chen, Qiman Wu, Jian Wang, Qinghao Hu, Tao Hu, Errui Ding, Jian Cheng, and Jingdong Wang. Mixformer: Mixing features across windows and dimensions. arXiv preprint arXiv:2204002557, 2022.
[27] Namyup Kim, Dongwon Kim, Cuiling Lan, Wenjun Zeng, and Suha Kwak. Restr: Convolution- free referring image segmentation using transformers. arXiv preprint 1676:22003.16668, 2022.
[28] Xin Lai, Jianhui Liu, Li Jiang, Liwei Wang. Hengshuang Zhao, Shu Liu, Xiaojuan Qi, and Jiaya Jia. Stratified transformer for 3d point cloud segmentation. arXiv preprint arXiv:2203 14508, 2022.
[29] Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. Adavit: Adaptive vision transformers for efficient image recognition. arXiv preprint arXiv:2111.15668 2021.
[30] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NeurlPS, 2015.
[31] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv: 087100 2016.
[32] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In ICCV, pages 2736- 2744, 2017.
[33] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz, Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440, 2016.
[34] Hanyu Peng, Jiaxiang Wu, Shifeng Chen, and Junzhou Huang. Collaborative channel pruning for deep networks. In ICML, pages 5113-5122, 2019.
[35] Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. arXiv preprint 1rXiv:1909.11555 2019.
[36] Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng. Ke Li. Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision trans- former. arXiv preprint arXiv:2 2/0i: 108.0139... 2021.
[37] Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming for efficient vision transformers. arXiv preprint arXiv:2106.0288 2021.
12

---

[38] Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. arXiv preprini arXiv:2202.07880, 2022.
[39] Arnay Chavan, Zhiqiang Shen, Zhuang Liu, Zechun Liu, Kwang-Ting Cheng, and Eric Xing. Vision transformer slimming: Multi-dimension searching in continuous optimization space. arXiv preprint arXiv:2201.0055 2022.
[40] Dirk P Kroese, Tim Brereton, et al. Why the monte carlo method is important today. Wiley Interdisciplinaryy Reviews: Computational Statistics, 6(6):386-392, 2014.
[41] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255, 2009.
[42] Li Yuan, Yunpeng Chen, Tao Wang. Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. ICCV, pages
2021.
[43] Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching trans- formers for visual recognition. In ICCV, pages 12270-12280, 2021.
[44] Mingjian Zhu, Kai Han, Yehui Tang, and Yunhe Wang. Visual transformer pruning. arXiv e-prints, pages arXiv-2104, 2021.
[45] Shixing Yu, Tianlong Chen, Jiayi Shen, Huan Yuan, Jianchao Tan, Sen Yang, Ji Liu, and Zhangyang Wang. Unified visual transformer compression. arXiv preprint arXiv:2203.08244 082443, 2022.
[46] Ding Jia, Kai Han, Yunhe Wang, Yehui Tang, Jianyuan Guo, Chao Zhang, and Dacheng Tao. Effi- cient vision transformers via fine-grained manifold distillation. arXiv preprint arXiv:2107.01378, 2021.
[47] Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E Gonzalez. Train large, then compress: Rethinking model size for efficient training and inference of transformers. arXiv preprint arXiv:200.. 2020.
[48] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
[49] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, pages 740-755, 2014.
[50] Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self-attention and convolutional layers. arXiv preprint arXiv: 1X11:1911.03584 2019.
[51] Jinnian Zhang, Houwen Peng, Kan Wu, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Minivit: Compressing vision transformers with weight multiplexing. arXiv preprint arXiv:2204.07154, 2022.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "tYKA4xEWww", "reviewer": "Reviewer_HogS", "summary": "This paper proposes SAViT,  a structure-aware pruning method for transformer-based architecture, which jointly prunes parameters in different components by considering interactions between these components. Experiments on different ViT architectures and vision tasks demonstrate the effectiveness of SAViT.", "questions": "See \"Weaknesses\"", "limitations": "The authors haven't discussed any limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The idea of collaboratively pruning all components of a model is interesting.\n- The performance gain is impressive when compared to sufficient state-of-the-art methods.\n- This paper is well-written and easy to follow.\n\nWeaknesses\n- It's not clear how much computation cost and data is needed for the proposed method during pruning, when compared to state-of-the-art network pruning methods.\n- The main novelty of this work is the interaction of different components during pruning, so the ablation study on this design is important. In my understanding, Line 297-305 and Table 7 aim to give such ablation studies, while it's not clear whether the setting of \"without second-order interations\" (Line 300) in this ablation study means dropping all Hessian-based terms in Eq. (4) or only drop the cross-components terms (green blocks in Figure 2(b))? I think the latter one can better reflect the main contribution of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UOwyAnChqsY", "reviewer": "Reviewer_2B6b", "summary": "This paper presents a new neural network pruning method for vision transformers.  The proposed technique can effectively accelerate most vision transformers such as ViT and Swin Transformer by collaboratively pruning components such as multi-head self-attention, hidden neurons, and embedding neurons. Extensive experiments show that the proposed technique is efficient yet competitive in accuracy as compared with the state-of-the-art. ", "questions": "No questions.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The manuscript has the following pros as follows: 1) The paper has a clear motivation and the proposed technique is based on the theoretical analysis. 2) The paper is clearly written, and easy to follow in general; 3) A large number of experiments have been performed to validate the performance in various aspects.\n \nI have some small concerns about this paper as listed. First, in Sec 4.4, the author said that the proposed method can bring accuracy gains when compressing the DeiT-Base. I think this conclusion is a bit inappropriate. As described in Sec. 4.1 and 4.2, the pruned models will further fine-tune 300 epochs, which means that the pruned models have a longer training schedule than baseline models. The accuracy gains may come from the longer training schedule.\n \nIn addition, the pseudo-code for the whole pruning algorithm in the Appendix is the core contribution of this work, which should be moved to the main paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BtIUuKhbtf", "reviewer": "Reviewer_Mowb", "summary": "This work proposes to prune ViT from all components comprehensively, which considers the interactions between different components in pruning ViTs. Different from the homogeneous components of CNN, the components of ViT are always heterogeneous, thus this work constructs a Taylor-based optimization function to take full advantage of the interactions between heterogeneous components. To avoid the huge computation cost of the Hessian matrix, this work derives an approximation to transform the Hessian matrix into pruning ratios. Finally, it solves the optimization target towards the optimal trade-off between accuracy and computational cost. This work is validated on DeiT and Swin on ImageNet and also on detection experiments.", "questions": "Overall, I think this work has a reasonable motivation and good idea, but the definition of ∆w confused me.", "limitations": "Please refer to Weaknesses. ", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This work is well-written and easy to follow.\n2.  The motivation of this work is clear.\n\n\nWeaknesses：\n\n1. Please verify the definition of ∆w, as ∆w = b ⊙ w − w, while the optimization target (eq.1) is: min ∆L = L(w + ∆w) − L(w), thus ∆L = L(w + ∆w) − L(w) = L(b ⊙ w) - L(w). And  C(w + ∆w) = C(b ⊙ w). I think this target is not correct, which maybe cuased by the definition of ∆w.\n2. Based on the definition of ∆w, I think we cannot get eq2 from eq1.\n3. The pruning results on Swin are directly trained on ImageNet-1k, while Swin can achieve much better performance after pretraining on ImageNet-22k. So the author should also provide experimental results on it. No need to train Swin on ImageNet-22k, but directly load the pretrained weights on ImageNet-22k then do pruning along with finetuning on ImageNet-1k.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3qu4VOu8qCC", "reviewer": "Reviewer_ipJy", "summary": "This paper presents a model pruning method for vision Transformers by jointly considering multiple possible pruning dimensions. To address the problem of joint optimization, a new collaborative pruning is designed. Experiments on multiple backbones (i.e., DeiT and Swin) and multiple tasks (i.e., ImageNet classification and COCO detection) show the effectiveness of the method.", "questions": "The paper presents a thorough study of the emerging area of efficient vision Transformers. The method is tested on multiple backbones and datasets. However, the joint optimization framework is similar to previous methods for CNNs, which makes the novelty of the pruning algorithm relatively low. Besides, I still have some concerns about the insufficient experiments, actual speedup on GPU, and reproducibility.  This paper can be stronger if the issues mentioned in the weaknesses subsection can be addressed.", "limitations": "Limitations of the proposed method are not discussed. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The idea of jointly considering multiple pruning dimensions is natural and well-motivated. \n\n- The method is tested and works well on multiple tasks and backbones.\n\nWeaknesses:\n\n- The idea of joint optimization of the vision Transformer architecture is not very new. Many previous methods have explored the joint optimization problem for network acceleration. Recent work like AutoFormer also considers multiple dimensions for vision Transformers.\n\n- Table 3 presents an important experiment to compare with previous state-of-the-art pruning methods. Since ViT/DeiT-B/S are considered as the standard models in many previous papers, it is better to provide the results on multiple model sizes (e.g., ViT-B/S/T) to clearly show the effectiveness of the proposed method.\n\n- According to Table 6, pruning multiple dimensions may not lead to ideal actual speedup on GPUs.\n\n- The method introduces a new pruning algorithm, which may not be easy to implement. Since the code is not available, I am a bit worried about the reproducibility of the method.\n\n\n-----------------\nPost rebuttal:\n\nI would like to thank the authors for the detailed feedback and additional results. The response addressed my concerns about the insufficient experiments and actual speedup on GPUs. I would like to upgrade my rating to 5.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept