oViT: An Accurate Second-Order Pruning Framework for Vision Transformers
Authors: Denis Kuznedelev
Eldar Kurtic
Elias Frantar
Dan Alistarh
Venue: Submitted to ICLR 2023 (2023)
Keywords: neural network pruning
vision transformer
sparsity
model compression

Abstract: Models from the Vision Transformer (ViT) family have recently provided breakthrough results across image classification tasks such as ImageNet. Yet, they still face barriers to deployment, notably the fact that their accuracy can be severely impacted by compression techniques such as pruning. In this paper, we take a step towards addressing this issue by introducing \textit{Optimal ViT Surgeon (oViT)}, a new state-of-the-art method for the weight sparsification of Vision Transformers (ViT) models. At the technical level, oViT introduces a new weight pruning algorithm which leverages second-order information, specifically adapted to be both highly-accurate and efficient in the context of ViTs. We complement this accurate one-shot pruner with an in-depth investigation of gradual pruning, augmentation, and recovery schedules for ViTs, which we show to be critical for successful ViT compression. We validate our method via extensive experiments on classical ViT and DeiT models, as well as on newer variants, such as XCiT, EfficientFormer and Swin. Moreover, our results are even relevant to recently-proposed highly-accurate ResNets. Our results show for the first time that ViT-family models can in fact be pruned to high sparsity levels (e.g. $\geq 75\%$) with low impact on accuracy ($\leq 1\%$ relative drop), and that our approach outperforms prior methods by significant margins at high sparsities. In addition, we show that our method is compatible with structured pruning methods and quantization, and that it can lead to significant speedups on a sparsity-aware inference engine. 


Under review as a conference paper at ICLR 2023
OVIT: AN ACCURATE SECOND-ORDER PRUNING FRAMEWORK FOR VISION TRANSFORMERS
Anonymous authors Paper under double-blind review
ABSTRACT
Models from the Vision Transformer (ViT) family have recently provided break- through results across image classification tasks such as ImageNet. Yet, they still face barriers to deployment, notably the fact that their accuracy can be severely impacted by compression techniques such as pruning. In this paper, we take a step towards addressing this issue by introducing Optimal ViT Surgeon (oVIT), a new state-of-the-att weight sparsification method, which is particularly well-suited Vision Transformers (ViT) models. At the technical level, oVT introduces a new weight pruning algorithm which leverages second-order information, and in par- ticular can handle weight correlations accurately and efficiently. We complement this accurate one-shot pruner with in-depth investigation of gradual pruning, augmentation, and recovery schedules for ViTs, which we show to be critical for successful compression. We validate method via extensive experiments classical ViT and DeiT models, hybrid architectures, such as XCTT, EfficientFormmrr and Swin, as well general models, such as highly-accurate ResNet and Efficient- Net variants. Our results show for the first time that ViT-family models can in fact be pruned to high sparsity levels (e.g. 2 75%) with low impact on accuracy (S 1% relative drop). In addition, we show that our method is compatible with structured pruning methods and quantization, and that it can lead to significant speedups on a sparsity- aware inference engine.
INTRODUCTION
Attention- based Transformers (Vaswani et al., 2017) have revolutionized natural language processing (NLP), and have become popular recently also in computer vision (Dosovitskiy et al., 2020; Touvron Carion al., 2020). The Vision Transformer (ViT) (Dosovitskiy al., 2020; Touvron et al., 2021) and its extensions (Ali et al., 2021; Liu et al., 2021; Wang et al., 2021) which are the focus of our study, have been remarkably successful, despite encoding fewer inductive biases. However, the high accuracy of ViT' comes at the cost of large computational and parameter budgets. In particular, VIT models are well-known to be more parameter-heavy (Dosovitskiy et al., 2020; Touvron et al., 2021), relative to their convolutional counterparts. Consequently, a rapidly-expanding line of work has been focusing reducing these costs for ViT models via model compression, thus cnabling their deployment in resource-constrained settings.
al., 2021;
Several recent references adapted compression approaches to ViT models, investigating either structured pruning, removing patches or tokens, or unstructured pruning, removing weights The consensus in the literature is that ViT models are generally less compressible than convolutional networks (CNNs) at the same accuracy. If the classic ResNet50 model (He et al., 2016) can be compressed to 80-90% sparsity with negligible loss of accuracy, e.g. (Frantar et al., 2021; Peste et al., 2021), the best currently- known results for similarly-accurate ViT models can only reach at most 50% sparsity while maintaining dense accuracy (Chen et al., 2021). It is therefore natural to ask whether this "lack of compressibility" is an inherent limitation of ViTs, or whether better results can be obtained via improved compression methods designed for these architectures.
Contributions. In this paper, propose new pruning method called Optimal VIT Surgeon (oVIT), which improves the state-of-the-an art accuracy-vs-spar -pssarsitty trade-off for ViT family models, and shows that they can be pruned to similar levels CNNs. Our work is based on an in-depth investigation of ViT performance under pruning, and provides contributions across three main directions:
A new second-order sparse projection. To address the fact that ViTs tend to lose significant accuracy upon each pruning step, we introduce novel approximate second-order pruner called o'T,, inspired by the classical second-order OBS framework (Hassibi et al., 1993). The key new feature of our pruner is that, for the first time, it can handle weight correlations

---

Under review as a conference paper at ICLR 2023
during pruning both accurately and efficiently. by a new theoretical result which reduces weight selection to a sparse regression problem. This approach leads to state-of-the-ar results one-shot pruning for both ViTs and conventional models (e.g. ResNets).
Post-pruning recovery framework. To address the issue that ViTs are notoriously hard to train and fine-tune (Touvron et al., 2021; Steiner et al., 2021), we provide set of efficient sparse fine-tuning recipes, enabling accuracy recovery at reasonable computational budgets.
End-to-end framework for sparsity sweeps. Our accurate oVTT pruner enables us avoid the standard and computationally--eet target independently, e.g. (Gale et al., 2019; Singh & Alistarh, 2020). Instead, we propose simple pruning framework that produces sparse accurate models for a sequence of sparsity targets single run, accommodating various deployments under a fixed compute budget.
procedure of gradual pruning for every sparsity
Our experiments focus on the standard ImageNet-IK benchmark (Russakovsky et al., 2015). We show that, under low fine-tuning budgets, the o'TT approach matches improves upon the state- of-the-art SVITE (Chen et al., 2021) unstructured method at low-to-medium (40-50%) sparsities, and significantly outperforms it at higher sparsities (2 60%) required to obtain inference speedups. Specifically, our results show that, at low targets (e.g. 40-50%) sparsity acts as a regularizer, sometimes improving the validation accuracy relative to the dense baseline, by margins between 0.5% and 1.8% Top-1 accuracy. At the same time, we show for the first time that VIT models can attain high sparsity levels without significant accuracy impact: specifically, we can achieve 75-80% sparsity with relatively minor (<1%) accuracy loss. Figure I summarizes our results.
Conceptually, we show that ViT models do not re- quire over-parametrization to achieve high accuracy, and that, post-pruning. they can be competitive with residual networks in terms of accuracy peeuurccyeee-eeee parameter. Practically, we show that the resulting sparse ViTs can be executed with speedups on a sparsity-aware in- ference engine (Kurtz et al., 2020). In addition, oViT complementary to orthogonal techniques such as token sparsification (Rao et al., 2021) and/or quanti- zation (Gholami et al., 2021), 1), and that it applies to newer ViT variants such as EfficientFormer (Li et al., 2022), XCIT (Ali et al., 2021), and Swin (Liu et al., 2021). Moreover, viT can provide strong results CNNs (Wightman et al., 2021; Tan & Le, 2021) and detection mod- els (Carion et al., 2020). In addition, it outperforms all known pruners on the challenging recent SMC- Bench compression benchmark (Anonymous, 2023).
Val accuraicy
Buftih SVITE
DeIT-S, GM
Bull:: uNT1
DalT-S, SVITE
DafT-B
Nonzero Params (M)
Figure 1: Validation accuracy versus non- zero parameters for DeiT-Tiny, -Small and -Base models, as well as a highly- accurate ResNet50D model, pruned to (50%, 60%, 75%, 80%, 90%} sparsities us- ing iterative Global Magnitude (GM), SViTE, and oVTT methods.
even for pruming highly-accurail
Overall, our correlation-aware pruner outperforms existing approaches on ViTs, CNNs (ResNet and EfficientNet), and language models (RoBERTa). Specifically for ViT', our recovery recipes and sparsity sweep framework enable o ViT to achieve state-of-the-art compression, roughly doubling sparsity at similar accuracy (Chen et al., 2021), with only half the compression budget.
Related work. Vision Transformers (ViTs) (Dosovitskiy et al., 2020) have set new accuracy bench- marks, but are known to require careful tuning in terms of both augmentation and training hyper- parameters. Identifying efficient recipes is an active research topic in itself (Touvron et al., 2021; Steiner et al., 2021). We propose new and general recipes for fine-tuning ViTs, which should be usefuul to the community. Several prior works have investigated ViT compression, but focusing structured pruning. such as removing tokens (Zhu et al., 2021; Kim et al., 2021; Xu et al., 2021; Pan et al., 2021: Song et al., 2022; Rao et al., 2021; Hou & Kung. 2022). We show experimentally that these approaches are orthogonal to unstructured pruning. which can be applied in conjunction to further compress these models.
Unstructured pruning. on which we focus here, considers the problem of removing individual network weights, which can be leveraged for computational savings (Kurtz et al., 2020; Hoefler et al., 2021). The only existing prior work unsfructured ViT pruning is SVITE (Chen al., 2021), which applied the general Rigl. pruning method (Evei et al., 2020) to the special case of ViT models. We also present results relative to well-tuned magnitude pruning, the best existing second-order pruners (Singh &
2

---

Under review as a conference paper at ICLR 2023
Frantar et al., 2021; Kurtic et al., 2022) and AC/DC pruning (Peste et al., 2021). oViT improves upon existing methods across almost all benchmarks, by large margins at high sparsity.
Alistarh, 2020;
Research on accurate pruning using second-order information was initiated by LeCun et al. (1989); Hassibi et al. (1993), and has recently garnered significant attention, e.g. (Dong et al., 2017; Wang et al., 2019; Singh & Alistarh, 2020; Yu et al., 2022). This approach can lead to good results for both gradual pruning (Frantar et al., 2021; Kurtic et al., 2022) and one-shot (post-training) compression (Frantar & Alistarh, 2022). The closest work to ours is (Frantar et al., 2021; Kurtic et al., 2022), who propose different approximations.
We introduce a new approximation of second-orde -odder information for pruning, which can efficiently take weight correlations into account. We show this to be especially- accurrae for ViT models. Another source of novelty is our in-depth investigation of sparse fine-tuning approaches for efficient accuracy recovery of ViT models. Together, these two techniques lead to tfte-of-thhe-rr results.
2 BACKGROUND AND PROBLEM SETUP
The pruning problem assumes a fixed model architecture with weights E Rd (d is the total number of parameters), and aims to find configuration of weights with as many zeros as possible while preserving the performance of the original dense model. Gradual pruning, e-g. (Hoefler et al., 2021), usually starts from an accurate dense model, and progressively removes weights by setting them to zero, followed by fine-tuning phases.
Weight Saliency. The pruning step usually relies on proxies for weight importance, defined according to certain criteria. For instance, weight magnitude is arguably the most popular criterion, e.g. (Han et al., 2015; Zhu & Gupta, 2017; Gale et al., 2019). Specifically, given model of each weight is its absolute value (magnitude) P = |umل for j E {1, 2.... d): weights with the smallest scores are prumed away. Gradual magnitude pruning is usually a strong baseline across most models and settings. Many other criteria exist, such as gradient magnitude (Evci et al., 2020) or "rates of change" in the weights (Sanh et al., 2020).
R the saliency
The Optimal Brain Surgeon (OBS). LeCun et al. (1989) and Hassibi et al. (1993) proposed a framework for obtaining weight saliency scores by leveraging (approximate) second-order information about the loss. Specifically, they start from the Taylor approximation of the loss L in the vicinity of the dense model parameters w*. Assuming that w is close to the optimum (hence V VL(w") = 0), one seeks a binary mask M (with elements E {0, 1}) and new values for the remaining weights wM,, such that the resulting increase in loss is minimal. A standard approach approximate the loss increase is to expand the loss function up to the second order in model weights:
1 L(wM)- L(w") 2 (w ww" " (HHH(w"((e) (w")(W w")
where (ww") ) is the Hessian of the model at w*, and w-M represents weights after the pruning step. In this setup, LeCun et al. (1989) and Hassibi al. (1993) showed that the "optimal" weight to remove, incurring the least loss, and the update to the remaining weights, can be determined via a flosed-form solution to the above inverse problem. Specifically, the saliency score P; for 7th weight and the optimal weight update dw for the remaining weights after climination of the zth weight are as follows:
P. 2 H
8w"
(w")e,,
(2)
where e; is the 7hh basis vector. Theoretically, the procedure would have to be executed one- -ne-weight-at- a-time, recomputing the Hessian after each step. In practice, this procedure suffers from a number of practical constraints. The first is that direct dessian-inverse computation is computationalllyinneeee -inieeasileeeeee for modern DNNs, due to its quadratic-in indiiiiniiiiiioonn storage and computational costs. This has led to significant recent work on efficient second-order approximations for pruning and quantization (Dong et al., 2017; Wang et al., 2019; Singh & Alistarh, 2020; Yu et al., 2022).
WoodFisher and the Optimal BERT Surgeon. The empirical Fisher approximation (Amari, 1998) a classic way of side-stepping some of the above constraints, and can be formally-stated as follows:
N
He(w")= ~ F(w*) = ALaxa +
VL, VL (WW)
(3)
N
3

---

Under review as a conference paper at ICLR 2023
where (ww) E IR is gradient computed on sample of data, À > is dampening constant needed for stability, and is the total number of gradients used for approximation. Note that the resulting matrix is positive definite.
The memory required to store the empirical Fisher matrix still quadratic in d, the number of parameters. Singh & Alistarr (2020) investigated diagonal block-wise approximation with predefined block size B. which reduces storage cost from (dd)) to ( and showed that this approach can lead to strong results when pruning CNNs. Kurtic et al. (2022) proposed formula for block pruning, together with a set of non-trivial optimizations to efficiently compute the block inverse, which allowed them to scale the approach for the first time to large language models.
A second obvious limitation of the OBS framework is that applying the procedure and recomputing the Hessian one weight at a time is prohibitively expensive, so one usually prunes multiple weights at once. Assuming are searching for the set of weights Q whose removal would lead to minimal loss increase after pruning, we get the following constrained optimization problem:
1 .ww T F (w")ōw
min
Eqow Eqw" O,
(4)
dw 2
where Eq E RIQ is matrix of basis vectors for each weight in Q. The corresponding saliency score for the group of weights Q and the update ówQ of remaining weights is (Kurtic et al., 2022):
pQ -
2 w T (F-1(w")qq
wQ dw2 = - Fw-((""EE,
(w")(w())))))
wQ-
(5)
However, an exhaustive search over all subsets of size [Q] from d elements requires (1²)) evaluations,
which makes it prohibitively expensive for [Q] 1. To alleviate this issue in the case of unstructured pruning Singh & Alistarh (2020) ignores correlations between weights and in the case of block pruning Kurtic et al. (2022) ignores correlations between blocks and solves only for correlations between weights within the same block. Despite these approximations, both approaches yield state- of-the-art results in their respective setups. As we will demonstrate later, our oVi method improves upon these approximations by reformulating this problem and proposing a correlation-aue aware solution that is fast and memory-efficient even for models as large as DeiT-Base (86M parameters).
THE OVIT PRUNING FRAMEWORK
3.1 WHY Is PRUNING VISION TRANSFORMERS HARD?
The literature suggests that ViT models are difficult to compress: the best known unstructured pruning results for ViT models only achieve 50% sparsity before significant accuracy loss (Chen et al., 2021). It is natural to ask why. Our investigation suggests the following factors at play.
Factor 1: Accurate One-shot Pruning is Hard. ViT models tend to lose a significant amount of
accuracy at each pruning step: Figure 3 shows the accuracy drops of Tiny, Small and Base model variants under the Global Magnitude (GM), WoodFisher/oBERT (WF), and Optimal VIT Surgeon (oViT) pruners, at various sparsities. In this one-shot setup, the higher accuracies of oVTT come from the significantly improved sparse projection step. Despite these improvements, accuracy drops are still not negligible, so we need a solid post-pruning recovery phase.
Factor 2: Accuracy Recovery is Difficult. Once accuracy has been dropped following pruning step, it is hard for ViT models to recover accuracy. This is illustrated in Figure 4, showing recovery under gradual pruning for various fine-tuning strategies. Results demonstrate the importance of the learning rate schedule and augmentation recipes.
Our work introduces new techniques to address both these factors: we introduce a new state-of-the-art one-shot pruner for ViTs, complemented with generic recipes for post-pruning fine-tuning.
3.2 INGREDIENT I: AN EFFICIENT CORRELATION-AWE AWARE SECOND-ORDER PRUNING
Our aim is to solve the pruning problem stated in the previous section: given a weight pruning target k. find the optimal set of weights Q to be pruned, such that [Q] k and the loss increase is minimized. Exactly solving for the optimal Q is an NP-hard problem (Blumensath & Davies, 2008), so we will investigate an iterative greedy method for selecting these weights, similar to the ideal version of the OBS framework discussed above. Importantly, our method properly considers weight correlations, while being fast and space-efficient. In turn, this leads to significant improvements over other pruners, especially in the context of vision transformers.
4

---

Under review as a conference paper at ICLR 2023
Formally, correlation- aware greedy weight selection approach would perform pruning steps iter- atively, as follows. Given a set of already-selected weights Q, initially 0, we always add to Q the weight q which has minimal joint saliency PQu[g). repeating until the size of Q equals the pruning target k. The fact that we add weights to the set one-by-one allows us to take into account correlations between pruned weights. However, naive implementation of this scheme, which simply recomputes saliency at each step, would be prohibitively expensive, since requires (kd) evaluations of pQ= each of which involves an O(B)) matrix inversion, where B is the Fisher block size.
The centerpiece of our approach is a reformulation of the OBS melth- pruning problem in Equation 5 which will allow us take correlations into account, while being paatically-efficien Specifically, we now show that, when using the empirical Fisher approximation, the problem of finding the optimal set of weights Q to be removed, while taking correlations into account, is equivalent to the problem of finding the set of sparse weights which best preserve the original correlation between the dense weights w" and the gradients (w,(w") on an fixed set of samples i E S. Formally, this result, whose proof we provide in Appendix L., is stated as follows.
Theorem 1.
Let S be a set with m samples, and let VL1 (WC((w)) (..,,..., Wm (w") be a set of their gradients, with corresponding empirical Fisher matrix F Fw"(""). Assume a sparsification target of k weights from w". Then, a sparse minimizer for the constrained squared error problem
1 Σ ) w')
min
VVL(W")
s.t. w/ has at least k zeros,
(6)
2m i=1
is also a solution to the problem of minimizing the Fisher-based group-OBS metric
arg min _wàT (="((")))))
w
Q
An Efficient Sparse Regression Solver. The formulation in Equation (6) reduces pruning to sparse regression problem, where the "input" is given by gradients over calibration samples. A related problem arises in the context of one-shot (post-training) pruning (Hubara et al., 2021; Frantar & Alistarh, 2022), where authors solve a related sparse lz-fitting problem, but sparse weights are determined relative to the layer inputs rather than the layer gradients. Specifically, the OBC solver (Frantar& & Alistarh, 2022) observes that, since the loss is quadratic, the row-wise Hessians are independent, and only depend on the layer input. Therefore, the method processes rows independently, and zeroes out weights from each row greedily, one-by-one, in increasing order of squared error. It then updates the remaining weights to reduce the l error. This essentially implements the OBS selection and update in Equation 2 exactly, assuming layer-wise lg loss. We build on this strategy to implement our greedy weight-subset selection, with some significant modifications.
A direct implementation of this approach to remove (dd) weights would have ((d) runtime, where is the layer dimension, as the O(d2)-time selection + update process is repeated ((d) times. This can be reduced to O(d- max(d, B)) by using the block diagonal structure of the Fisher matrix with block size B, when performing updates after each weight elimination. However, as B is typically much smaller than d, this would still be extremely slow. We apply a different approach: we treat each block of the Fisher matrix as an independent row for the purposes of the &g solver, and then we merge the all the blocks into a single global approximation of the Fisher matrix. This allows us to perform global weight saliency calculation, and weight updates following Equations and 7. The resulting algorithm, given in full in Appendix 1, requires O(d B²) runtime and Odd- B) space. When working with small block sizes, our method is very fast and has practically no overhead over existing Fisher-base OBS approaches, while yielding significantly improved one-shot pruning results. Detailed pseudo-code and an efficient implementation are provided as supplementary material.
3.3 INGREDIENT 2: FINE-TUNING AND PRUNING PROCEDURE
ViTs are notoriously difficult to train: they need longer training relative to CNN architectures, and the choice of the training procedure (learning rate schedule, regularization, augmentation) can have a major impact on convergence and accuracy (Touvron et al., 2021; Steiner al., 2021). We found the same to apply to post-pruning accuracy recovery, which is key in gradual pruning; below, we describe the main ingredients to obtaining highly- accurate me-ttnnng schedules as part of our method.
Learning Rate Schedule. First, to achieve good performance during gradual pruning, the learning rate (LR) schedule is crucial. Specifically, we propose to a cyclic linear schedule:
5

---

Under review as a conference paper at ICLR 2023
t%T
(tt) Mmax
(7max
7mmin)
(8)
T
where %e means taking the remainder after integer division by x. We chose linear decay for simplicity; we obtained similar results for other functions (e.g., cubic decay). By contrast, as we illustrate in Section 4.2, the cyclic nature of the schedule is key for accurate pruning. Specifically, this choice is justified theoretically by tying back to the original assumptions of the OBS framework: for Equation I to hold, the pruned model should be well-optimized (i.e. have small gradients) at the point when pruning is performed. Moreover, right after the pruning step, having larger value of the learning rate is useful since gives the model chance to recover from the sub-optimal point induced via pruning. We note that this learning rate schedule is different from prior work on pruning, which typically uses single decay cycle (Kusupati et al., 2020; Singh & Alistarh, 2020; Peste et al., 20 or dynamic learning rate rewinding, e.g. (Frankle et al., 2019; Renda et al., 2020).
Regularization and Augmentation. Another important ingredient for achieving high accuracy is the choice of the regularization/augmenaaa pipeline. Specifically, we observe that smaller models such as DeiT-Timy benefit from lower levels of data augmentation during fine-tuning as in Steiner et al. (2021), whereas larger mod- els such as DeiT-Base behave best with more complex augmentation and regularization, such as Touvron et al. (2021). Intuitively, fine-tuning sparsified small models with high augmentation likely may exceed model capacity, rendering the optimization process unstable. We provide de- tailed parameter values and ablations for this training component in the AppendixB.
0.0023
2.0024
2,0083
Leaming
2,0022
0,0030
20
1.0
250
200
250
300
Epoch
Figure 2: Blue: Cyclic linear learning rate sched- ule used in the work. Violet: Dependence of the global model sparsity on the epoch. Every change in sparsity corresponds to a pruning step.
An Efficient Pipeline for Sparsity Sweeps. We
propose simple iterative pruning framework, which takes a set of target sparsity configurations and produces models which match these configurations in single run. Specifically, we start from a standard gradual pruning setup, which prunes in sequence of steps of increasing sparsity, followed by sparse fine-tuning. We then set the intermediate values in such a way that all intermediate target sparsity levels are achieved. For example, if one wishes obtain checkpoints with sparsity levels 40%, 50,, 75%, 90%, one can set the lowest sparsity level on the gradual pruning schedule to 40%, the highest sparsity level to 90% ,and 50%, 75% intermediate points. Between any two such pruning steps, we apply the cyclic retraining schedule above.
We emphasize the fact that having accurate pruner is key to support this compressed pruning approach: virtually all previous high-accuracy pruning methods, e.g. (Kusupati et al., 2020; Singh & Alistarh, 2020) redo the entire training run for each sparsity target in turn. In our experimental section, we also examine the impact of additional fine-tunn tine-tuning applied to each checkpoint, and show that it induces small-but-consistern improvements.
4 EXPERIMENTAL SETUP AND RESULTS
Setup and Goals. We consider the ImageNet (Russakovsky et al., 2015) image classification benchmark, and aim examine how sparsity impacts accuracy for different model variants. We consider three scenarios: one-shot, single-step pruning of pretrained model, where performance is clearly tied to the quality of the second-order approximation, one-shot + fine-tuning, in which we follow one-shot pruning by a short period of fine-tuning, and, finally, iterative gradual pruming, where one applies pruning periodically, with some retraining interval, gradually increasing sparsity.
4.1 ONE-SHOT PRUNING, WITHOUT AND WITH FINE-TUNING
We start by examining the quality of existing one-shot pruners relative ViT. We compare against carefully-tuned variants of Magnitude Pruning (Magn), Gradient times Weight (GrW) Sanh et al. (2020), and the SOTA second-order methods WoodFisher (Singh & Alistarh, 2020; Kurtic al., 2022) and M-FAC (Frantar et al., 2021). Our tuning process, optimal hyper-parameter choices, and ablations are detailed in Appendices G and H. To demonstrate the importance of weight correlations,

---

Under review as a conference paper at ICLR 2023
also compare with an implementation of LeCun et al. (1989) via WoodFisher, denoted by WF-1 on the Figure 3, which performs diagonal approximation.
For Magnitude and oViT we present results for both uniform layer-wise sparsity and global sparsity. This choice does not make a difference for Magnitude, whereas global sparsity better for ViT, as it can adjust saliency globally. We only investigate global sparsity for the other methods, in Figure 3.
DeiT-Tiny
ResNet50-D
0.8
0.7
0.8
Val acceursyy
Val Bocwracy
0.5
0.4
(U)
Maga (G)
64
GAV (G)
Magn
WF-1 (G)
0.
Mago (G)
M-PAC (G)
92
WF (G)
WF (G)
oVIT (T))
4.3
oVT (G)
oVTT
4.3
0.
9.7 08
04
64
0.7 0.8
09
Sparsity
Sparsity
Figure 3: One-shot pruning for DeiT-Tiny (left) and ResNer50-D (right).
The full comparison between all these SOTA methods is presented for DeiT-Tiny and highly- accurate ResNet50-D, in Figure 3. First, all first and second-order methods outperform Magnitude. All 2nd order methods are better than the 1st order saliency described above. WoodFisher with small block size is better than both the diagonal approximation and large-block M-FAC. This suggests that is beneficial to take weight correlations into account, but attempting to incorporate dependencies between large groups of weights may lead to noisy estimates that detrimental performance. We also present comparison between different pruners on several variants ViT models Appendix C.
We note that VTT outperforms all other methods, both DeiT-Tiny and ResNet50, setting new SOTA results. Remarkably, the margin is so large that ViT with uniform sparsity still outperforms global sparsity WoodFisher. The computational cost of WF is approximately the same for oVT:: ViT pruning step on DeiT-Small takes 23 minutes, compared 20 minutes for WF. (The majority of the cost for both methods comes from the collection of gradients, not Hessian estimation.)
Oneshot + Fine-tuning, Next, we consider the one-shot + fine-tune setup where one first prunes the model to certain sparsity level, and then per- forms a short period of fine-tuning. Specifically, below prune to 50% sparsity, and fine-tune for 20 epochs. In addition to ViT/DeiT, we also consider similar models based on variants of self- attention (Liu et al., 2021; ; Ali et al., 2021), 1), and compare against simple GM baseline. We use linearly-decaying learning rate schedule between Tmax 10 to M/ma 10-5 and the training DeiT training recipe (Touvron al., 2021). The results are given in Table 1, and show that oViT can almost fully-recover accuracy for all models; the gaps from GM and WF (see DeiT-Small 75 and 90%) are still very significant. Given this, in the following experiments, we will mainly focus on oViT as the second-order pruner.
Table 1: One-shot + fine tuning on ImageNet.
Model
Method
Sparsity (%)
Topl- Aecuracy (%)
Dense
79.8
GM
79.0
oVIT
50
79.5
GM
74.3
DeiT-Small
WF
75
75.8
oVTT
76.9
GM
45.6
WF
59.3
oVT
65.1
Dense
GM
81.5
oWT
this setup
DaiT-Base
GM
80.1
WP
80.2
oWI
81.0
GM
68.1
WF
90
69..2
oVi
76.3
Dense
82.0
4.2 POST-PRUNING RECOVERY
XCT--Seall
oViT
GM
50
81.9
81.7
For ViTs, the choice of augmentation parameters and learning rate schedule is critical. For exam- ple, reducing the level of augmentation during fine-tuning for smaller models, e.g. DeiT--Tiyy significant improves performance, whereas larger models, e.g. the 4x larger DeiT-Small, requires
Donse
81.3
Swin-Tiny
oii
GM
50
80.5
80.9
7

---

Under review as a conference paper at ICLR 2023
strong augmentations for best results even during fine-tuning, See Figure 4 for an illustration; provide full details and results Appendix B.
Moreover, the choice of eyclic learning rate (LR) schedule is critical as well. To illustrate this, we compare convergence obtained when using a cosine annealing schedule, which very popular for pruning CNNs (Kusupati et al., 2020; Singh & Alistarh, 2020; Peste et al., 2021), from Maaa = 5-10-a Tymán 10-3 while performing pruning updates times more frequently (one update per 10 epochs) than in our standard setup from the following section 4.3. The results are provided Figure 4, where cosine annealing (no cycles) in red. All experiments use the oiT pruner, and highlight the importance of the learning rate and augmentation schedules for recovery.
Table 2: Securacy-vs-spariie for gradual prun-
DeiT-Tiny
ing on ImageNet. (t) denotes accuracy after additional fine-tuning for 100 epochs.
Val accuraa
Spansity (%)
Topl
Model
Method
Reduction
Accuracy (%)
(56
light1
Dense
72.2
GM
43.9
73.5
oNTT
Epoch
SVITE-Tiay
43.9
69.6
DuiT-Small
GM
52.6
73.1 (73.2
light
oWT
52.6
73.3 (73.6
Deil-Tiny
GM
65.8
71.4 (71.9 T)
oNT
65.8
72.3 (72.6
SVITE-Tiny
65.8
63.9
GM
69.7
70.5 (70.9 1)
oNTT
GM
79.0
66.2 (66.6
oWI
90
79.0
67.4 (68.0
Epoch
SVITE-Tiay
79.0
49.7
Figure 4: Ablations of the training setting on DeiT-Tiny (up) and DeiT-Small (down). Green curves correspond to the lightl (Steiner al., 2021) augmentation recipe, blue curves to the deit (Touvron et al., 2021) recipe. The curve follows training with single (acyclic) cosine annealing schedule, as in (Kusupati al., 2020; Singh & Alistarh, 2020).
Dense
79.8
GM
46.7
79.3 (79.8
(79.9
Small
46.3
79.7
GM
56.1
79.0 (79.5
oNT
56.2
79.3 (79.8 T)
SVITE-
55.4
79.4
DelT-Small
GM
78.0 (78.7
oNTT
75
70.2
78.5 (79.0 ?
SVITB- Smalll
DeiT-Base
GM
74.2
(77.9
oVTT
80
74.9
78.0 (78.6
DeiT-Small
DelT-Tiny
GM
84.0
74.1 (74.7
Val cy
7B
eNTT
84.1
75.2 (75.8 T)
SWITB Small
84.1
70.1
oVTT
50
SViTE-Base
48.0
81.5
DaiT-Base
xNTT
58.2
81.5
SVITE-Bass
57.5
81.3
68
75
72.8
81.1 (81.2
Throughput (img/s)
90
87.4
79.7 (80.1 1)
Figure 5: Accuracy throughput for dense and sparse ViT models when executing on the DeepSparse inference engine. corresponds to dense models.
4.3 GRADUAL PRUNING RESULTS
Finally, we execute gradual pruning with the sparsity schedule, augmentation choices, and cyclic linear learning-rate scheduler discussed above. The whole gradual pruning procedure lasts for 300 epochs, in Touvron al. (2021). We aim to obtain accurate sparse checkpoints for 50%, 60%, 75%, 80%, and 90% sparsity. For this, we prune to 40% in the initial step, and increment sparsity every 20 epochs, until reaching 90%, with fine-tuning in between. (See Figure 4.) We select the
8

---

Under review as a conference paper at ICLR 2023
accuracy of intermediate models which match the target sparsities; examine the impact of fine- tuning, we trained each of the resulting sparse checkpoints for an additional 100 epochs, marked with (t). We compare with global magnitude (GM) following the same schedule as oVTT, as well as the state-of-the-art SVTE (Chen et al., 2021) paper, which trains the sparse model from scratch using a variant of Rigl. (Evci et al., 2020), but for a total of 600 epochs. The results are in Table 2.
For DeiT-Tiny and 50% sparsity, we achieve significant improvements upon SViTe, and even manage to improve test accuracy relative to the dense model. We believe this is due to more proper choice of augmentation for fine-tuning this smaller model. At 75-80%, we recover the dense model accuracy, showing for the first time that VIT models can be pruned to such sparsities without loss, We observe a significant accuracy drop only at 90%. GM pruning also benefits from the choices made in our schedule, outperforming SVITE at 50% sparsity; yet, there are significant gaps in favor of oVTT at higher sparsities, expected.
On the 4x larger DeiT-Small model, SVITE performs remarkably well at 50% sparsity (79.7%).
almost matching the dense model, but viT outperforms it very slightly after fine-tuning (79.9%). In terms of total training budget, SVITE uses 600 epochs to produce each model (and so, the 50%-sparse one as well), whereas we use a total of 40 epochs for pruning to 50% + initial fine-tuning, and 100 additional epochs for sparse model fine-tuning. Even if we take into account the original 300 epochs dense DeiT checkpoint (Touvron et al., 2021), our approach is significantly more efficient (440 vs. 600 epochs), and savings compound across sparse models. At 75% sparsity, oVIT drops ~ 1% of accuracy relative to dense post- post-finetuning. with a significant gap of 1% Top-1 relative to GM, and 2% Top-I I relative SViTE, The trend continues for higher sparsities, where we note remarkable gap of 5.7% Top-1 I vs SVITE at 90% sparsity. We obtain similar numbers for DeiT-Base; generally, we achieve > 99% recovery at 2 75% sparsity.
for training the publicly-availably
Additional Experiments. Due to space constraints, several experiments are deferred to the Appendix. Specifically, show that o'TT also provides high accuracy when pruning other highly-accurate models (EfficientFormer, ResNet50D, and EfficicntNetV2) in Appendix D, while having similar runtime to the fastest implementation of WoodFisher (Kurtic et al., 2022) (Appendix E). In Appendix F we show that ViT can also be applied in conjunction with other types of compression, in particular token pruning (Rao et al., 2021), quantization-aware training, and semi-structured (2:4) sparsity. Specifically, we show that oViT can induce 2:4 sparsity with minor accuracy drop across DeiT models. In Appendix J, we show oVTT also outperforms AC/DC pruning (Peste et al., 2021), while in Appendix M we show that ViT outperforms all other methods on SMC-Bench (Anonymous, 2023).
Sparse Speedups. Finally, in Figure 5, we examine the speedups obtained by ViT from unstructured sparsity, when executed on sparsity-aware CPU inference engine (Kurtz et al., 2020). Specifically, we executed the models from Table 2 using 4 cores of an Intel(R) Xeon(R) Gold 6238R CPU, at batch size 64. We find it interesting that sparse ViTs build almost-contiguous Pareto frontier from 82% to 68% Top-1 accuracy (Y axis), with a 25x span in throughput (from 11 imgs/second to 260 imgs/second, X axis). Notably, the DeiT-TTiy model obtains a speedup of 2.4x without any accuracy loss (thanks to the accuracy increase), while Base and Small ViT's show 1.5x speedups with minimal loss of accuracy. Thus, sparsity can be promising approach to speeding up ViTs. We note that competing methods (e.g. SVITE) would provide similar speedups at the same sparsity levels, but lower accuracy, as shown in Table 2.
5 DISCUSSION, LIMITATIONS AND FUTURE WORK
We examined the trade-off between parametrization and accuracy in the context of unstructured pruned VIT models, and presented a new pruner called oVIT, based on a new approximation of second-order information, which sets new state-of-the-a sparsity-acuuracy accuracy trade-off. Specifically, we have shown for the first time that ViT variants can support significant weight pruning (2 75%) at relatively minor accuracy loss (5 1%), similar to CNNs. Therefore, our results seem to suggest that, despite their weaker encoded inductive biases, ViT models do not require over over- parametriizanio post-training, and in fact can be competitive with CNNs in terms of accuracy per-parameter. Our approach extends to other model families, and is complementary to other compression approaches.
One limitation is that we still require rather high computational budgets for best results. This is currently an inherent limitation of ViTs; to address it, we provide efficient oneshot fine-tune recipes, leading to good results. Further, we have only focused on Top-1 as a leading accuracy metric, without covering transfer accuracy or potential bias (Hooker al., 2020). Future work should also be able to extend our pruner to structured compression of ViT models, or employ our o ViT pruner inside different, more computationally-intessiss pruning algorithms such as RigL. (Evci et al., 2020).

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alaacldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross-covariance image transformers. Advances in neural information processing systems, 34, 2021.
Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10(2):251-276. 1998.
Anonymous. Sparsity may cry: Let us fail (current) sparse neural networks together! In Submitted to The Eleventh International Conference on Learning Representations, 2023. URL https: / /pppneevveww... net / ffoum? i iediidd-6F0 J6
under review.
Thomas Blumensath and Mike E Davies. Iterative thresholding for sparse approximations. Journal of Fourier Analysis and Applications, 14(5-6):629 654, 2008
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. Advances in Neural Information Processing Systems, 34:19974-19988, 2021.
Xin Dong, Shangyu Chen, and Sinno Jialin Pan. Learning to prune deep neural networks via layer- wise optimal brain surgeon. In Conference on Neural Information Processing Systems (NeurIPS), 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition scale, 2020. URL https: arxiv. org/ abs / 2010.00000 11929.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. PMLR, 2020.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv: 1903.0161, 2019.
Elias Frantar and Dan Alistarh. Optimal Brain Compression: A framework for accurate post-training quantization and pruming. arXiv preprint arXiv:2208. 115800 2022.
Elias Frantar, Eldar Kurtic, and Dan Alistarh. M-fac: Efficient matrix-free approximations of
second-order information. Advances in Neural Information Processing Systems, 34, 2021.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.099 09574, 2019.
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103. 13630, 2021.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv: 10.000499 2015.
Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, pp. 293-299, IEEE, 1993.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1-124, 2021.
10

---

Under review as a conference paper at ICLR 2023
Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. Characterising bias in compressed models. arXiv preprint (XXi:20110003088, 2020.
Zejiang Hou and Sun- Yuan Kung. Multi dineeiiiimniinnl model compression of vision transformer. In 2022 IEEE International Conference on Multimedia and Expo (ICME), pp. 01-06. IEEE, 2022.
Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry. Accelerated sparse neural training: A provable and efficient method to find n: m transposable masks. Advances in Neural Information Processing Systems, 11-21099-21111 2021.
Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. Learned token pruning for transformers, 2021. URL https://aaaatt www.:/iw..iioooooooooooooooooooe org/o/////ao.ooo. /abs/ 2107. 00910.
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal bert surgeon: Scalable and accurate second-orde pruning for large language models. arXiv preprint arXiv:2203.072.9, 2022.
Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Bill Nell, Nir Shavit, and Dan Alistarh. Inducing and exploiting acti- vation sparsity for fast inference on deep neural networks. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Proceedings of Machine Learning Research, pp. 5533-5543, Virtual, 13-18 Jul 2020. PMLR. URL http:://pt pptp:///ccceeinggss
Machine Learning. volume 119 of
www.. press / v11 9 /urr z20a html.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity, In International Conference on Machine Learning, pp. 5544-5555. PMLR, 2020.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Yanyu Li, Geng Yuan, Yang Wen, Eric Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren. Efficientformer: Vision transformers at mobilenet speed. arXiv preprint arXiv:2206.01 191, 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
2021.
Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks, 2021. URL https: arxiv. org abs 2104. 08378.
Bowen Pan, Rameswar Panda, Yifan Jiang, Zhangyang Wang, Rogerio Feris, and Aude Oliva. Ia- red 2: Interpretability-awaar redundancy reduction for vision transformers. Advances in Neural Information Processing Systems, 34:24898-22 -29981, 2021.
Alexandra Peste, Eugenia Iofinova, Adrian Vladu, and Dan Alistarh. Ac/de: Alternating com- pressedddecomppesssed training of deep neural networks. Advances in Neural Information Process- ing Systems, 34:8557-8570, 2021.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamievit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34: 34:13937-13949, 2021.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. arXiv preprint 20f::20000.0388., 2020.

---

Under review as a conference paper at ICLR 2023
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-252, 2015. doi: 1007// 1263-015-0816-y.
Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning, Advances in Neural Information Processing Systems, 33:20378- 20389, 2020.
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. Advances in Neural Information Processing Systems, 15:18088-18009 2020.
Zhuoran Song, Yihong Xu, Zhezhi He, Li Jiang, Naifeng Jing, and Xiaoyao Liang. Cp-vit: Cascade vision transformer pruning via progressive sparsity prediction, 2022. URL atttps://arxe org/abs 2203. 04570.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers, 2021. URL https: /toe:///axxii /ttp/aaxivvv.///////ee//aaaevlea org/ abs./10// 10270.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensequ: question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00933 2018.
Mingxing Tan and Quoe Le. Efficientnetv2: Smaller models and faster training. In International Conference on Machine Learning, pp. 10096-10106. PMLR, 2021.
Hugo Touyron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Chaoqi Wang, Roger Grosse, Sanja Fidler, and Guodong Zhang, Eigendamage: Structured pruning in the kronecker-factored eigenbasis. In International Conference on Machine Learning, pp. 6566-6575. PMLR, 2019.
Wenhai Wang, Enze Xie, Xiang Li, Deng- sing Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer. A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 568-578, 2021.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:2110.00476, 2021.
Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision transformer, 2021. URL https: : / arxiv. wwg. abs 201..218...000 01390.
Shixing Yu, Zhewei Yao, Amir Gholami, Zhen Dong, Schoon Kim, Michael W Mahoney, and Kurt Keutzer. Hessian-awarr pruning and optimal neural implant. Proceedings of the IEFF/CV Winter Conference on Applications of Computer Vision pp. 3880-3891, 2022.
Michael Zhu and Suyog Gupta. To prune, not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv: 2017.
Mingjian Zhu, Yehui Tang, and Kai Han. Vision transformer pruning, 2021. URL https:/ // arxiv. www.oorgg/abba/eooeoo /2104 08500.
12

---

Under review as a conference paper at ICLR 2023
A OVIT ALGORITHM DESCRIPTION
The section below illustrates the VIT pruning algorithm step-by-step. Prunable model weights IR are partitioned into blocks of fixed size B. Below P: inside block belongs to and Pe is the score across the whole model. The steps of the algorithm are listed below:
(B) denotes the saliency scores for weight zth
Algorithm oViT pruning algorithm
1: P; saliency scores for weights
2: Accumulate Fisher inverse blocks
3: for each block do
4: CIT 0
5: for element in block do
6: Select the weight w, with smallest score (B))) (using the equation 2 for pc)
7: Prune
8:
Update remaining weights the block via equation
err+ P: (B)
9:
Pe f err
11: Save current state of the block for later merging
Update Fisher inverse block
13:
end for
14: end for
15: Sort the scores ij in ascending order
16: Mark the weights with smallest scores P: pruned
17: for each block do
18:
Load the saved state of the block with the weights marked pruned and all remaining alive.
19: end for
B TRAINING DETAILS
Augmentaaton/reguarizaiioo recipe
Table 3: Summary of the augmentation and regularization procedures used in the work.
Procedure
DeiT
lightl
Weight decay
0.05
0.03
Label smoothing
0.1
0.1
Dropout
X
X
Stoch.Depth
0.1
0.0
Gradient Clip.
X
1.0
H.flip
RRC
Rand Augment
9/0.5
2/0.5
Mixup alpha
0.8
0.0
Cutmix alpha
1.0
0.0
Erasing prob.
0.25
0.0
Erasing count
Test crop ratio
0.9
0.9
For the gradual pruning experiments (with 300 epochs) we have used cyclic learning schedule, with high learning rate directly after the pruning step with gradual decrease up to the pruning step.
13

---

Under review as a conference paper at ICLR 2023
Table 4: Hyperparameters the schedules used gradual pruning.
Model
Prune freq
LR sched [/decay- Tmaxx 7minn
Augm
Batch size
Epochs
DeiT-Tiny
20
(cyclic [cyclicllinear, 5 10 1 10-b)
lightl
1024
300
DeiT-Small
20
[cyclic [cyclic_linear, 5 10 10 10
deit
1024
300
For both DeiT-Tiny and applied cosine annealing schedule with 7max = 5 7wiin 11 10 and all other parameters the same as in the Table 4.
model during the additional fine-tuning for 100 epochs 've
C ONE SHOT PRUNING OF DIFFERENT VIT VERSIONS.
In this section we present comparison of Global Magnitude (GM), WoodFisher (WF) and oVTT pruners in one-shot pruning setting for DeiT models (Touvron al., 2021) of diffent size (i.e DeiT-Tiny, Deil-Small, DeiT-Bare BBaase) to study the scaling behaviour of ViT sparsification.
Decl-Small
GH
GM
GM
WT
Sponsity
Spansty
Sparsit
Figure 6: One-shot pruning of different DeiT versions.
Notice, that the gap between magnitude pruning and second order methods is very pronounced for all models, whereas the difference performance between oVT and WF decreases with increase of the size of model. Nevertheless, oVTT performs still noticeably better than WF, especially in high sparsity regime.
D EXPERIMENTS WITH OTHER MODELS
Despite the fact that in this work we aim at the pruning of VIT models the proposed approach can be applied to any architecture for image classification, in particular, convolutional neural network (CNN) or a ViT-CNN Hybrid. We have applied o oVTT gradual pruning to the recently proposed EfficientFormer (Li al., 2022) using the same setting and hyperparameters CNN architectures ResNet50-D and EfficientNerV2 2-Tnnyy (Tan & Le, 2021) considered this work were trained with the use of augmentation and regularization procedure described in the recent PyTorch blog post. Differently from most of the prior art we have used the ResNet50-D trained with the modern recipe from timm repository.
for DeiT-Small. Two
For ResNet50-D we prune all convolutional weights except of the first convolution and we keep the classification layer dense. In EfficientNetv2-Timm we do not prune depthwise convolutions since each channel is processed separately and the only block size that makes sense in this case would be most of the size of the convolutional kernel spatial dimensions product. We have set the block size to be 256 for ResNet50-D and 16 for EfficientNetV2-TE ViIT the same as for DeiT experiments. Such a small block size was chosen for CfficientNee(2-TTiinn due the fact that the largest common divisor of the prunable weights.
while keeping all the other hyperparameters of
First of all, we conducted comparison between one-shot pruning methods for ResNet50-D. We compare between Uniform and Global magnitude pruning. WoodFisher with block size of 256, M-FAC with block size of 2048 and ViT with uniform and global sparsity. One can observe that oViT' outperforms all previous methods even when comparing uniform sparsity with global sparsity.
"resnet500d checkpoint with 80.5 % accuracy for dense model "effiicietteetv2 ____t checkpoint with 82.3 % accuracy for dense model
14

---

Under review as a conference paper at ICLR 2023
Contrary to the case of DeiT where there is much difference in performance between uniform and global magnitude pruning for ResNer50-D global sparsity turns out to much better. This results quite expectable since CNN are not uniform and deeper layers are mode wide than those close to the input.
ResNet50-D
0.7
0.5
Val accoracy
0.5
0.4
Magn (U)
0.3
Magn (G)
M-FAC (G)
WF (G)
oVIT (U)
oVIT
0.3
0.4
0.5
0.5
0.7
0.8
Sparsity
Figure 7: Comparison between different pruning methods on pretrained ResNet50-D model on the ImageNet dataset in one-shot pruning setup.
Next we carried out one-shot finetuning experiments with ResNet50-D keeping setup the same as for DeiT models in Table 1. We have selected 75 % and 90 % as the difference between methods becomes pronounced only at sufficiently high sparsity. Notably, the performance of Global magnitude and WF is roughly the same, the initial difference after one-shot pruning between WF and Global Magnitude vanishes during the finetuning procedure, whereas there is still gap in performance between o ViT and other methods.
Table 5: One-shot + fine-tuning on ImageNet.
Model
Method
Sparsity (%)
Topl -Accurcyy (%)
Dense
80.5
GM
79.0
WF
75
79.0
ResNet-50D
oViT
79.2
GM
74.7
WF
90
74.8
V
75.2
Finally we conducted gradual pruning runs following the same sparsification schedule as for DeiT- models. The EfficientFormer and EfficientNet models despite being already very optimized and parameter efficient can be still compressed with small drop accuracy.
E TIMINGS
Any algorithms involving second order loss information are believed to require tremendous amounts of compute. Time required for calculation of pruning scores and the OBS update comprises collection of grads, Fisher inverse rank-1 updates and additional pruning iteration for ViT. We have measured the time of single pruning step for DeiT-Small and present the results in Table 7. All measurements were done on a single RTX A6000 GPU with 48GB of memory. One can observe that the amount of time needed to perform a pruning update is not very large, especially when compared to the duration of typical training procedure of modern computer vision models ImageNet that usually takes several days on a multi-gpu node. Note that the additional step for oVT adds small fraction total computational time relative other steps of the OBS method.
15

---

Under review as a conference paper at ICLR 2023
Table 6: Performance of gradual pruning on ImageNet. Numbers in the parentheses followed by the
upwards directed arrow denote additional fine-tuning for 100 epochs.
Model
Method
Sparsity (%)
Aopl-Accuracy (%)
Dense
0
78.9
50
78.0
EffFormer-I.1
60
77.4
oVIT
75
76.4
90
72.4 (72.8 1
Dense
0
80.5
50
79.8
ResNet-50D
60
79.7
oVTT
75
79.2 (79.6 1)
90
77.1 (77.5 f)
Dense
0
82,4
50
81.0
60
80.6
75
79.6 (80.0 f)
90
75.0
Table 7: Minutes per pruning step for DeiT-Small.
Model
Method
Time (minutes)
Fast WoodFisher (Kurtic et al., 2022)
DeiT-Small
oViT
23
F COMPOSITE COMPRESSION
In addition to weight pruning decrease storage and inference cost with the help of other compression approaches: quantization (casting weights and activations to lower precision) and token pruning specific for the transformer architecture.
F.1 QUANTIZATION-AWWARE TRAINING
Weight quantization done in the following way one takes sparse checkpoint and then runs quantization aware training (QAT). We ran QAT training for 50 epochs with linearly decaying learning rate schedule from 7mma 10 7mmin 10 5, Models are quantized 8-bit precision. In all experiments performed accuracy of quantized model almost reproduces the accuracy of the sparse model stored in full precision.
Table ImageNet- IK top-1 accuracy for sparse models after QAT training.
Model
Sparsity (%)
Accuracy (%)
DeiT-Tiny
75
72.2
DeiT-Small
75
77.7
DeiT-Base
75
81
F.2 TOKEN PRUNING
There are different approaches for token pruning proposed the literature. In this work follow the one from (Rao et al., 2021). Specifically, in DynamicViT one selects the ratio of tokens being pruned each step with the lowest importance score, predicted by the model itself. Following the main setup from the paper we prune tokens after 3rd 6th gth block, and the token pruning ratio after each block is p 0.2 (i.e 20% least improtant tokens are pruned).
16

---

Under review as a conference paper at ICLR 2023
Table 9: ImageNet- top-1 accuracy for sparse models with token pruning.
Model
Method
Sparsity (%)
TopI-Accuracy (%)
50
72.0
DynamicViT-Tiny
VVIT
60
71.6
75
70.2
50
DynamicVit ViT-Small
V'T
60
79.4
75
78.7
F.3 SEMI-STRUCTUREE SPARSITY.
While CPUs can utilize sparsity patterns of arbitrary form to speed-up the computations at the present time modern GPU accelerators can handle only restricted form of unstructured sparsity, namely the N . sparsity pattern that enforces exactly N non-zero values for each block of M weights. Namely, since the introduction of Ampere architecture NVIDIA GPUs have special kernels that can work with 4 sparse matrices (Mishra et al., 2021). One can integrate the M sparsity in the oViT framework without significant changes. The only difference with the original o ViT approach that while running the o oVTT iterations one doesn't 't prune a given weight in case a group of weights to which this weights belongs to there M N zero weights. Since the sparsity pattern is significantly constrained compared to generic unstructured sparsity pattern drop in performance after doing pruning step and consequent fine-tuning is more challenging than it would be for unconstrained sparsity. experiments below we prune models with linearly decaying learning rate schedule.
sparsity and fine-tune them for 50 epochs
Table 10: Semi-structured 2 : pruning of ViT models.
Model
Accuracy (%)
DeiT-Tiny
72.7
DeiT-Small
79.0
DeiT- 80.5
G OVIT/WF HYPERPARAMETER,
Following the oBERT's directions (Kurtic et al., 2022) on identifying the optimal set of hyperpa- rameters via one-shot pruning experiments, we conduct grid search over the three most important hyperparameters:
Number of grads collected for Fisher inverse
Dampening constant À
Block size
The more grads are collected, the more accurate the empirical Fisher inverse estimate, however, more compute is required at the same time. We chose N 4096 point from which further increase of Fisher samples doesn't 't improve performance performance at different sparsities vs number of grads is presented on Figure 8.
Dependence of the one-shot pruning
The next parameter be studied is the dampening constant À in. This constant regularizes the empirical Fisher matrix and allows to avoid instabilities in computation of the inverse. However, this constant decreases the correlation between different weights and in the limit - oo OBS reduces to magnitude pruning. The optimal dampening constant for oVi (Aapt = 10 10-$)
(0-8) is smaller than the one for WoodFisher (Aopt = 10 10-6), 6), i. ViT remains numerically and computationally stable with smaller amount of regularization compared WF (we observed that for X < 10-1 WF performance starts to deteriorate rapidly).
And the last but not the least important parameter is the block size in (Singh & Alistarh, 2020). The larger the block size is, the more correlations between different weights are taken into account. However, mentioned in 2 the computational and storage cost scales with the block size. Moreover,
17

---

Under review as a conference paper at ICLR 2023
DelT-Small
DelT-Small
16
16
64
64
256
1024
1024
Val aceuracy
accuracy
04
0.3
61
01
0.0
09
04
0,6 Sparsity
Sparsity
Figure Left: One-shot pruning performance of WoodFisher. Right: One-shot pruning performance of VVT.
DelT-Small
DelT-Small
le-5
1e-7
1e-8
1e-8
1e-7
1e:9
accuracy
Val accuracy
0.5
0.4
00
0.4
Sparsity
Sparsity
Figure Left: One-shot pruning performance of WoodFisher. Right: One-shot pruning performance of VfT.
for a fixed number of gradients in the Fisher estimate matrix with larger block sizes is likely to be worse conditioned. Therefore, one would like to work with smaller block sizes but not to keep the approximation as accurate as possible. We' 'e selected block size according to the accuracy-effciency trade-off.
DeiT-Small
DeiT-Small
0.8
0.6
48
0.7
0.7
192
192
0.6
Val accuracy
acouracy
0.3
0.1
Sparsity
Sparsity
Figure 10: Left: One-shot pruning performance of FoodFisher. Right: One-shot pruning perfor- mance of VIT.
In addition, we've ve studied the benefit from application of multiple recomputations in the one-shot pruning setting for WoodFisher and ViT. Since the assumption of static Fisher matrix F ((w*) doesn't hold in general, we expect that multiple recomputations are likely to result in higher one-shot accuracy in accordance with the result from (Frantar et al., 2021). This is indeed the case. The gain from recomputations more pronounced for WoodFisher, since ViT already performs implicit Fisher inverse updates in its operation. Yet, the efect is not vanishing even for the case of VIT.
18

---

Under review as a conference paper at ICLR 2023
DeiT-Small
Val accuracy
WF (Nr-1) 1)
(Nr=2)
WF (Nr=5)
WF (Nr=8)
VVI (Nr=1)
oVIT (Nr=5)
Sparsity
Figure 11: One-shot performance for WF and oVTT with different number of recomputations N,-
H DETAILS AND HYPERPARAMETER CHOICES FOR OTHER PRUNING METHODS.
In this section we provide some additional details about methods compared on Figures 3 and 7.
The popular Movement Pruning (Sanh al., 2020) computes the weight saliency scores during the training procedure, hence it is not a one-shot pruning method by the definition. We have observed that use of the naive elementwise product of gradient and weights i.e p; = wN Ww ((w)) leads to poor performance, significantly below even the Magnitude Pruning baseline. However, the following first order pruning criterion:
N
P
(R) I
V
((R) (w)ll
allows to get reasonable saliency scores that produce more accurate sparse models than Magnitude Pruning. However, its performance is still inferior to any of the second order pruning methods. This method is denoted as GrW (Gradient times weight) on Figures and
M-FAC Pruner proposed in (Frantar et al., 2021) is pruner leveraging second order information that doesn't 't require an explicit construction Fisher Inverse matrix. Therefore, unlike WoodFisher and oViT that require ((Bd) memory computation and storage cost of this method is constant with respect the block size and one can take into account correlations between larger groups of weights for free. Following the original paper we chose block size of 2k as the best performing one. However, from Figures 3 and 7 that smaller block sizes turn explanation of this phenomenon is that the Fisher Inverse estimate becomes too noisy and unstable for large blocks.
perform better. possible
I EXECUTION LATENCY.
In addition to the plot throughput vs accuracy shown in the main part we present in this section execution latency per sample latency when running models on the DeepSparse engine. The results are presented on Figure 12.
J COMPARISON WITH AC/DC TRAINING
In addition to the sparse training from scratch with periodic updates of the sparsity weights with some salincy criterion for weight elimination and regrowth (Evci et al., 2020) one can consider alternating compressedddeecompessed training (AC/DC), proposed in (Peste et al., 2021). Namely between dense stages with standard unconstrained training of the model, and sparse stages when the model is pruned to the target sparsity level and trained with the frozen sparsity mask until the
switches
19

---

Under review as a conference paper at ICLR 2023
DefT-Base
DelT-Small
DefT-Tiny
78
Val accuracy
76
74
68
Latency (ms)
Figure 12: Accuracy vs lateney on ImageNet-1k.
beginning of the next dense stage, when the sparsity mask is removed. This procedure produces both accurate dense and sparse models.
Following the original paper use magnitude pruning as saliency criterion for weight pruning. The augmentation and regularization pipeline follows the settings from Touvron et al. (2021). All models with AC/DC
trained for 600 epochs in total with first pruning step epoch 150 followed by 7 sparse stages 25 epochs long each, and dense stages of the same length. The last dense stage lasts 50 epochs and the last sparse is 75 epochs long. Learning rate is gradually decayed from Mmax 5 10
to 7m = 10 6 with cosine annealing. Initial warm-up up phase with linearly increasing learning rate is 20 epochs. We compare AC/DC with o VIT models finetuned for additional 100 epochs.
Table 11: AC/DC vs oVTT (finetuned for additional 100 epochs) ImageNet-Ik
Model
Method
Sparsity (%)
Topl-Accuray (%)
oViT AC/DC
79.9
60
80.4
DeiT-Small
VVT
79.0
75
AC/DC
79.0
oVTT
75.8
90
AC/DC
72.0
One can observe that at low sparsity AC/DC achieves higher accuracy for the same sparsity target (even outperforming the dense baseline by 0.6%), whereas for 75% performance of both methods is equal, and ViT outperforms AC/DC at higher sparsity. However, one should note, that oViT computational budget (including the training of original model) of 440 epochs for 60% sparsity, 520% for 75% and 700% for 90% 600 epochs used in AC/DC.
K ONE-SHOT PRUNING OF DETR
The approach presented in the paper is not limited the image classification task, but be applied to other computer vision tasks, such as object detection. We chose the DeTR model (Carion et al., 2020) with ResNet50 backbone and ran one-shot pruning procedure with global magnitude, WoodFisher and o ViT pruner. Specifically, we pruned convolutional layers in the CNN backbone expect the first one and all linear projections in transformer encoder and decoder blocks while keeping the detection heads dense. The results are presented Table 12. Following the standard protocol we used bbox mAP for evaluation. One can observe, that difference between the second order methods and magnitude pruning very pronounced even for relatively small sparsity of 50%, and ViT outperforms WF pruner.
20

---

Under review as a conference paper at ICLR 2023
Table 12: One-shot pruning of DeTRR.
Model
Method
Sparsity (%)
bhox mAP
Dense
0.42
GM
0.16
DeTR
WF
50
0.36
oViT
0.38
L PROOF OF THEOREM 1
Theorem 1. Let S be a set of samples, and let Ve, (w"), )...., Lmn (w") be a set of gradients with i S, with corresponding empirical Fisher matrix F Fw'(("). Assume sparsification target of weights from w Then, a sparse minimizer for the the constrained squared error problem
minlw
2m "vex("") "'' [Ww )=)
w' has least k zeros,
(10)
is also solution the problem of minimizing the Fisher-based group- OBS metric
argmin Q.1) 2 wQ (F-1(w")1Q
WQ.
(11)
Proof. We start by examining the unconstrained squared error function in Equation (10), which denote by G. Clearly, the function G is d-dimensional quadratic in the variable w', and has minimum at w Next, let us examine G's second-order Taylor approximation around w", given by
(w'
Ve, (w"
w").
(12)
where used the fact that w is minimum of the squared error, and thus the function has gradient at it. However, by the definition the empirical Fisher, this is exactly equal to
(w' w F((") -w).
(13)
The Taylor approximation is exact, as the original function a quadratic. and so the two functions equivalent. Hence, have obtained the fact that, under the empirical Fisher approximation, k-sparse solution minimizing Equation 10 will also be a k-sparse solution minimizing Equation However, the question of finding a k-sparse solution minimizing Equation 1 is precisely the starting point of the standard OBS derivations (see e.g. (Singh & Alistarh, 2020) (Kurtic et al., 2022)), which eventually lead to the formula in Equation (11). This concludes the proof.
M PERFORMANCE ON THE "SPARSITY MAY CRY (SMC)" BENCHMARK
Concurrently to our submission, novel benchmark targeting sparse language models has been proposed. The "Sparsity May Cry" Benchmark (SMC-Bench) (Anonymous, 2023) introduces a set of challenging tasks, spanning from commonsense and arithmetic reasoning to protein termostability prediction and multilingual translation, on which it claims that all existing pruning approaches fail to produce accurate and sparse models, sometimes even at trivial sparsities as low as 5%. As such, poses an open question on whether the existing datasets and metrics are reliable for evaluation and comparison between different pruning techniques.
We conduct experiments with our proposed o VVT pruner on the hardest benchmark presented in the AMC-Bench work, according to its authors, which all pruning approaches appear to fail. Namely, the task evaluates commonsense reasoning capabilities of the massive RoBERTa-Large (Liu et al., 2019) model, with 354M parameters, on CommonsenseQA, question-answent challenge targeting commonsense knowledge (Talmor al., 2018).
In Figure 13 we present our results on various sparsity targets. More specifically, for 20%, 50%, 60% and 70% targets simply do one-shot pruning with oVT followed by 5-epochs of fine-tuning. For higher sparsity targets, 80% and 90%, we run gradual pruning with oVT over the span of 20-epochs. As can be seen from the Figure, our proposed ViT pruning approach does not fail even the hardest
21

---

Under review as a conference paper at ICLR 2023
SMC-Bench task and significantly outperforms all other existing techniques. These results also demonstrate that our method, even though developed in the context vision transformers, produces state-of-the-art results when applied to pruning of large language models in the NLP domain.
Since the exact data of SMC-Bench is not yet released, we sketch our oVTT results Figure and provide exact numbers in Table 13.
RoBERTa CommonsenseOA
80
Dense model
SNIP w/o E (Before)
Sparsity (%)
Accuracy (%)
60
oVIT
LTH (Attei)
20
78.38
Accuracy
(After)
50
76.16
OMP we (Atter)
60
73.05
40
GMP (During)
wio (During)
70
70.35
OMP (Before)
80
56.10
OMP w/o (Before)
90
50.37
20
CVIT
0.488
0.59
0.672
0:738
0.791
0.8325
0.86
0.893
Sparsity
Figure 13: Evaluation of various pruning techniques on the
Table 13: Sparsity-accuracy results obtained with oViT pruning ap- proach, illustrated in Figure 13.
challenging SMC-Bench, pruning of the RoBERTa-Large model on the CommonsenseQA task.
22


Reviews:
[{"review_id": "ycvUqVLx6jK", "reviewer": "Reviewer_TCn5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new weight sparsification for vision Transformer. It considers second-order information to recognize redundant weights. By investigating various strategies, the proposed method can acheive high sparsity levels with low impact on accuracy.", "summary_of_the_review": "See Strength And Weaknesses", "strength_and_weaknesses": "\nStrength:\n\n-The authors conduct extensive experiments to validate the effectiveness of this method. The experiments are conducted with multiple backbones including ViT, XCiT, Swin-Transformer, e.t.c.\n\n-This paper is well-written and easy to follow.\n\nConcerns:\n\n-Using second-order information to prune neural network is a common strategy. It has been widely used in spare convolutional neural networks. The work brings these methods to vision transformer. Could the second-order method for CNN be used in vision transformer? The authors are required to clarify the contribution and novelty about the method transferring.\n\n- It is interesting that the unstructured pruning can achieve practical acceleration. However, only the speed of proposed method is shown in Table 5. How about the competeting method? Compared with other methods, could the proposed still show superiority with similar practical speeds?  Besides, it is better to illustrate how to achieve practical acceleration on the a CPU platform more detailedly.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cXRqrLbdA3N", "reviewer": "Reviewer_rRsE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Paper proposes a method to prune neural networks. Most experiment are shown for Transformers, results on CNN are presented in the Appendix. The method is based on second order statistics, Hessian. The method is inspired by Optimal Brain Surgeon (1993) and is the modification of it. Results are presented on Imagenet image classification. \n\n", "summary_of_the_review": "Paper proposes a general framework for pruning neural networks. It is based on iterative saliency pruning. Novelty of the method is in the computation of the saliency metric that is based on Hessian inverse and a modification of OBS (1993). In my understanding, the paper proposes a way to simplify computation of the Hessian inverse, therefore novelty is questionable. Paper lacks comparisons with simpler techniques based on first order Taylor expansion methods that are simpler (gradient times weight), as well as full versions of OBS and OBD. Finally, the method is not specific to Transformers and more comparisons are required to the SOTA in CNN pruning. ", "strength_and_weaknesses": "Strengths:\n\n- Transformers are popular and pruning them is important. The proposed method is generic and it is not clear how it is specific to transformers. \n- Working with saliency metric other than the magnitude is a promising direction. Specially because weight magnitude does not scale for global pruning and requires sensitivity analysis. \n- Taylor decomposition based techniques are demonstrated to be better for CNNs, it is good to see this works for transformers as well.\n- Proposed method computed saliency and the required weight update. \n\nWeakness/questions:\n\n- What is exact pruning setup? Is it global unstructured sparsity or uniform unstructured sparsity? For the the former one it is clear that GM will not perform well as weights are not scaled with respect to the layer number. \n- With all approximation, what is the final equation to compute saliency? authors mention eq 2 in Algorithm 1, but this is the start point not oViT. Sharing code will help to understand the algorithm. \n- In the paper authors state and try to answer why pruning transformers is hard. However, pruning of ViT should not be harder as transformers are overparametrized. [1] shows that Diet pruned for structured sparsity of 50% maintains accuracy. For N:M sparsity, for example 2:4, pruning with magnitude gives no accuracy drop. \n- Finetuning is an important aspect of the pruning method. Authors tune the recipe, what happens if the standard recipe is used.\n- Method is based on the Taylor expansion of the the loss function. The method is inspired by OBS and therefore should be compared directly even if compute time is significant. Having comparison with optimal brain damage will be helpful as well. \n- How would the method perform if we use diagonal approximation of the Hessian: $\\rho = w_i^2*\\Delta L(w_i)^2$ ? This is another popular saliency metric for pruning. \n- Transferability of the concluded insights: the experiments are conducted on classification tasks only. While the observations on different tasks maybe different, thus I am not sure whether the extracted insights can be transferred to other tasks or it is only a not general conclusion for classification tasks.\n- What is special in the proposed method for transformers? The method seem to be general but title specifically days \"for vision transformers\"\n- Minor: LeCun proposed optimal brain damage (OBD) while Hassibi proposed optimal brain surgeon. \n\n\n[1] Yang, Huanrui, et al. \"Nvit: Vision transformer compression and parameter redistribution.\" arXiv preprint arXiv:2110.04869 (2021).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IRPdz3ndsf", "reviewer": "Reviewer_rP9A", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on unstructured sparsity of vision transformer. The paper proposes a second-order pruner called oViT, and a set of general sparse fine-tuning recipes. The proposed oViT allows pruning in a single run. The experiments on ImageNet-1K show the effectiveness of the proposed method.", "summary_of_the_review": "The paper is well written but the novelty should be further clarified, compared to the existing OBS based pruning methods. The experiments on finetuning recipe should be more comprehensive.", "strength_and_weaknesses": "Strength:\n- The paper is well organized and easy to follow.\n- The observation on vision transformer pruning is interesting and provides much insight.\n- The proposed correlation-aware second-order pruning is reasonable.\n\nWeaknesses or questions:\n- Optimal Brain Surgeon [1] is a widely-used second-order technology and has been used in neural network pruning [2,3,4]. This paper is an extension of OBS for vision transformer. The novelty is limited. What's new in the proposed method?\n- Could the proposed method be applied on CNN pruning?\n- The proposed finetuing recipe is a set of tricks, which is more like engineering contribution.\n- The proposed finetuing recipe includes learning rate schedule, regularization, augmentation and an efficient pipeline for sparsity sweeps, The ablation study of these components are not fully conducted.\n\n[1] LeCun Y, Denker J, Solla S. Optimal brain damage[J]. Advances in neural information processing systems, 1989, 2.\n\n[2] Hassibi B, Stork D. Second order derivatives for network pruning: Optimal brain surgeon[J]. Advances in neural information processing systems, 1992, 5.\n\n[3] Hassibi B, Stork D G, Wolff G J. Optimal brain surgeon and general network pruning[C]//IEEE international conference on neural networks. IEEE, 1993: 293-299.\n\n[4] Dong X, Chen S, Pan S. Learning to prune deep neural networks via layer-wise optimal brain surgeon[J]. Advances in Neural Information Processing Systems, 2017, 30.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject