A Fast Post-Training Pruning Framework for Transformers
Authors: Woosuk Kwon
Sehoon Kim
Michael W. Mahoney
Joseph Hassoun
Kurt Keutzer
Amir Gholami
Venue: NeurIPS 2022 Accept (2022)
Keywords: Pruning
Compression
Transformers

Abstract: Pruning is an effective way to reduce the huge inference cost of Transformer models. However, prior work on pruning Transformers requires retraining the models. This can add high training cost and high complexity to model deployment, making it difficult to use in many practical situations. To address this, we propose a fast post-training pruning framework for Transformers that does not require any retraining. Given a resource constraint and a sample dataset, our framework automatically prunes the Transformer model using structured sparsity methods. To retain high accuracy without retraining, we introduce three novel techniques: (i) a lightweight mask search algorithm that finds which heads and filters to prune based on the Fisher information; (ii) mask rearrangement that complements the search algorithm; and (iii) mask tuning that reconstructs the output activations for each layer. We apply our method to BERT-base and DistilBERT, and we evaluate its effectiveness on GLUE and SQuAD benchmarks. Our framework achieves up to 2.0x reduction in FLOPs and 1.56x speedup in inference latency, while maintaining < 1% loss in accuracy. Importantly, our framework prunes Transformers in less than 3 minutes on a single GPU, which is over two orders of magnitude faster than existing pruning approaches that retrain the models.

A Fast Post- Post-Training Pruning Framework for Transformers
Woosuk Kwon
Sehoon Kim" UC Berkeley sehoonkim@eerkeeey.. edu
Michael W. Mahoney UC Berkeley, ICSI, & LBNL
UC Berkeley koosuk.kwon@berkee.. edu
maboneynw@berkelley edu
Joseph Hassoun Samsung Semiconductor, Inc. j. hassoun@samsung.. com
Kurt Keutzer
Amir Gholami UC Berkeley
UC Berkeley
keutzer@berkeleyy.dde
amirgh@berkeley..om
Abstract
Pruning is an effective way to reduce the huge inference cost of Transformer models. However, prior work pruning Transformers requires retraining the models. This can add high training cost and high complexity to model deployment, making it difficult to in many practical situations. To address this, propose fast post- training pruning framework for Transformers that does not require any retraining. Given resource constraint and sample dataset, framework automatically prunes the Transformer model using structured sparsity methods. To retain high accuracy without retraining, introduce three novel techniques: (i) lightweight mask search algorithm that finds which heads and filters to prune based on the Fisher information; (ii) mask rearrangement that complements the search algorithm: and (iii) mask tuning that reconstructs the output activations for each layer. We apply our method SERTBASE and DistilBERT, and we evaluate its effectiveness on GLUE and SQuAD benchmarks. Our framework achieves up to 2.0x reduction in FLOPs and 1.56x speedup inference latency, while maintaining 1% loss accuracy. Importantly, our framework prunes Transformers in less than 3 minutes single GPU, which over two orders of magnitude faster than existing pruning approaches that retrain the models
Introduction
In recent years, Transformer 76 has become de facto standard model architecture in Natural Language Processing
and it becoming common in many domains including Computer 7 and Speech Recognition (2 7 26 However, efficient deployment of Transformer architectures has been challenging due their large model size and high inference latency, As promising way to tackle this challenge, structured pruning of Transformers has been widely studied.
Vision 14
While prior work pruning Transformers substantially reduces the inference time, it is often difficult to use in practice for several reasons. First, previous approaches require retraining the pruned model and/or jointly learning the pruning configurations during training. This increases the training time by adding significant computational overhead. Second, previous methods add many moving parts to the model deployment process. That is, the pruning pipelines are often complex and require additional hyperparameter tuning. Such techniques demand significant engincering efforts for implementation and debugging, which impedes their adoption in production pipelines. Third,
up to 10x
38 88
"qqual contribution.
Our code is publicly available at http://///ggghh..oooooooooooooee
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

User Intervention
Fine-tuned Transformer
FLOPs/latency Constraints
Sample Dataset
Fine-tuned
Dataset
Transformer
(a) Existing Pruning Frameworks
(b) Our Pruning Framework
X
X
Retraining
-30 hrs
No Retraining
"3 mins
Figure (a) Prior pruning frameworks require additional training on the entire training set and involve user intervention for hyperparameter tuning. This complicates the pruning process and requires a large amount of time (c.g., ~30 hours). (b) Our pruning framework does not require retraining. It outputs pruned Transformer models satisfying the FLOPs/latency constraints within considerably less time (c.g., ~3 minutes), without user intervention.
these previous methods do not directly adapt to the users' constraints. They either rely on vague regularization hyperparameters control the model sparsity use fixed model architectures selected independently of the user settings. This can result in sub-optimally pruned models that are not tailored the users' constraints and hardware.
To address the above limitations, we propose a fast post-training pruning framework for Transformers that does not require any retraining of the models. As illustrated in Figure [ framework takes as input a Transformer model, a sample dataset, and a FLOPs/latency constraint. It then outputs a pruned Transformer model that be deployed immediately. By avoiding expensive retraining, the end-to-end pruning pipeline can be extremely fast and simplified, which typically takes a few minutes without any user interventions that complicate the whole process.
Indeed, post-training compression has been widely studied for quantization, and it gained considerable attention in both academia and industry 3 27 197). Although quantization- aware training methods achieve higher compression rates general, post-training quantization (PTQ) has often been more preferred in practice due its retraining- free advantages. Importantly, PTQ allows quantization to happen seamlessly at the model deployment time using the tools such as TensorRT [56], TFLite [I9], and OpenVINO that enables pruning of Transformers without engineering efforts.
29 Similar to the PTQ methods, framework provides an out-of-the- tot-o---he-boxx tool
Our contributions can be summarized as follow:
We propose a novel post-training pruning framework for Transformers that does not require model retraining. To retain accuracy without retraining, framework consists of three stages: (i) the mask search process guided by the Fisher information matrix to select which heads/filters to prune (Section (ii) the mask rearrangement process that reselects the heads/filters prune by capturing intra-layer interactions (Section 4.2 ; and (iii) the mask tuning process that adjusts the mask variables to ensure that the output signal recovered for each layer (Section
We extensively test our framework by applying it to BERT BASE RASE and DistiBBERT GLUE and 5.2). Within 1% accuracy drop, our framework reduces 30-50% of the original FLOPs (Figure 5). resulting in up to 1. 1.56x speedup on an NVIDIA V100 GPU (Table
SQuAD tasks (Section
We show that method achieves comparable even better FLOPs-accuracy trade-off than prior structured pruning methods without retraining (Section 5.3 Figure 6). Our end-to-end pruning pipeline finishes in only 39 and 135 seconds average for GLUE and SQuAD (Section 5.4 Table 5 which is over 100x faster than the retraining-baat based methods.
Related Work
Efficient Transformers. In order to improve the inference speed and reduce the memory footprint of Transformers, multiple different approaches have been proposed. These can be broadly categorized as follows: (i) efficient architecture design
PS
72
8
87]: (ii) hardware-softwar co-
design 21
22 73 80 (iii) knowledge distillation
3
82 (iv) quantization 33 66 95 [6]:
(v) neural architecture search
77
68
79
90
and (vi) pruning. this paper, we focus on
pruning and briefly discuss the related works.

---

Transformers Pruning. Pruning has been a promising way to remove unimportant weights in neural networks. Pruning can be largely categorized into unstructured and structured pruning. For unstructured pruning, magnitude-basee and the lottery ticket hypothesis [9 TC.
[8] first-order (64] and second-order [36] pruning methods,
16 58 have been explored for Transformers. While these
methods can substantially compress the model size, commodity hardware such as GPUs can hardly take advantage of the unstructured sparse patterns for model inference speedup.
For this reason, a number of structured pruning methods have been introduced to remove coarse- grained sets of parameters in Transformers. For example, to prune structured sets of parameters in
weight matrices, low-rank factorization 3 block-wise sparsity [4 and tile-wise sparsity [20
were studied. Furthermore, as more coarse-gres grarseegrineddd methods, attention head pruning [5T
77 and
layer dropping [15
62
have been popularly used. Taking a step further, recent approaches 8 32 38
92 have explored jointly pruning Transformers with different pruning granularity and
principles, maximizing the model efficiency in every dimension. Orthogonally, another thread of
work 15
25
40
98
has shown that Transformers can be dynamically pruned at inference time.
Unfortunately, while the structured pruning methods can achieve high compression rates and speedups, they are often difficult use in practice. One reason for this is the high computational cost of additional training during or after pruning, which can be up to 10x 38% 88] compared to that of the original model training. Another reason is the high complexity of the pruning pipelines [25 38 47l 1921. where each pruning stage often requires rewriting the training code and introduces additional hyperparameters to tune.
Post-training Model Compression. Post-training compression methods have been widely studied in quantization. These methods, categorized as post-training quantization (PTQ), perform quantization without any retraining, thereby avoiding the additional training cost and user intervention. Multi- ple PTQ techniques have been proposed to effectively mitigate the accuracy degradation without
retraining [3 27
54 97).
Although not much for quantization, post-training schemes have also been explored for unstruc- tured
40 [3
and structured pruning of CNNs. For structured pruning. [3
770 94 proposed
find it difficult to extend those techniques to pruning Transformers because they require the model to have a repeating structure of a linear layer and element- wise nonlinearity, which is not the case for the multi-heat attention layers of Transformers. Even for the feed-forward network layers of Transformers, [34] (70 can hardly be used because they rely on a certain characteristic of ReLU while many Transformers GELU 24 instead of ReLU,
ways to group and merge similar neurons in CNN. However,
Motivated by the fact that the existing post-training CNN pruning techniques cannot be applied to Transformers, in this paper we propose a novel post-training pruning method with a focus on Transformers. However, would like to note that the underlying principles in approach are general enough to be extended to pruning other types of model architectures such as CNNs.
3 Overview
3.1 Background
Transformer Architecture. In this paper, we focus on the pruning of encoder- based Transformer 76 models, especially the BERT [22 architecture family. BERT is a stack of homogeneous Transformer encoder blocks, each of which consists of a multi-head attention (MHA) layer followed by a point- wise Freed-Fowwart Network (FFN) layer. Specifically, MHA layer consists of H independently parameterized attention heads:
"
MHA(x) Σ (x), Xчнд = LayerNorm (x + MHA(x)),
where Att is dot product attention head, and is the input sequence. The output of the MHA layer then fed into the FFN layer, which consists of N filters:
N
FFN(x)=
w (W!! + b(D)) + + Xout = LLaerNNrmmXx + FFN((MMa))
where W() (222 BOO) and ((2) are the FFN parameters, and is the activation function, typically GELU 24). Note that (H, N) (12, 3072) for BERT base, and (16, 4096) for BERTLARGE We also
denote L as the number of Transformer layers (c.g., 12 for BERT
3

---

PPN
FFN
m
MILA
m
MHA
m
m
MHa
MILA
O
1:2
O
1.1
Initial Mask
(b) Mask Search
(c) Mask Rearrangement
Mask Tuning
Figure 2: Overview of our pruning framework. (a) The mask variables are initialized as 1. Then they undergo the three-stage pipeline of (b) mask search (Section 4.1 (c) rearrangement (Section 4.2). and (d) tuning (Section
Granularity of Pruning. Our framework considers the structured pruning of both heads in MHA and filters in FFN layers. We do not prune the embedding and the final classifier, computation of those layers takes a negligible portion of the total inference latency. Since our pruning framework always produces a smaller dense architecture, the model can be readily accelerated without the need of specialized hardware logic, which often required for unstructured sparsity to gain latency speedup.
Notations. We pose the pruning problem as finding a sparse mask for the heads and filters. To formalize this, we introduce mask variables associated with the outputs of heads and filters:
H
MHA(x; (x; m
MHA)
MRA m
(x);
FFN ml FFN)
W
(W X + ((2) + 6
where MIIRA E R and m E RN are the mask variables for MHA and FFN in the /-th layer, respectively, and mi and m are their i-th elements. Furthermore, denotes the Hadamard product. Originally, the mask variables are all initialized to 1, which does not change the model outputs. After pruning, the mask variables become zero or any nonzero values, affecting the model accuracy and sparsity. Especially, setting mi, and MILA MILA head and filter, respectively.
is equivalent to pruning the i-th
Overall, there are LH head mask variables and LN filter mask variables, summing up to L(H N) number of total mask variables in a Transformer model. To simplify notations, we additionally define MMA E RLH mIN E BLL, and m E R(H+N) as flattened vectors of the head, filter, and total mask variables, respectively, across all layers. In what follows, discuss how find the optimal sparse masks under a given cost constraint and how to adjust their values to recover accuracy.
3.2 Framework Overview
Figure bb) and Figure illustrate the overview of our framework.
Inputs. Our framework has inputs: a Transformer model; a sample dataset; and a resource constraint. The input Transformer model should contain weights fine-tuned for downstream task. The sample dataset is a small partition of the training dataset (typically 1-2K examples) for the downstream task. The resource constraint can be given either as the number of floating point operations (FLOPs) or as an actual latency on target hardware. In the later case, we further assume that a latency lookup table for the target hardware is provided.
Compression Pipeline. As illustrated in Figure our framework consists of 3 stages: Fisher-based mask search; Fisher-based mask rearrangement; and mask tuning. During the Fisher-based mask search stage (Section I. we search for a binary mask applied to the heads and filters based on the Fisher information of the mask variables. Intuitively, the mask variables with relatively higher Fisher information are considered more important, and they should be less likely be pruned 41 45 52
As finding the optimal mask that minimizes the Fisher information loss is intractable due to the large size of the full Fisher matrix, we propose a lightweight search algorithm that finds the optimal mask under reasonable approximations. Second, in the Fisher-based mask rearrangement stage (Section 4.2).. the framework adjusts the searched mask patterns to better take into account the intra-layer interactions between the mask variables. Lastly, in the mask tuning stage (Section 4.3) the framework tunes the nonzero mask variables to recover the accuracy drop by reconstructing the layer-wise output signal.

---

4 Methodology
The pruning problem can be seen as finding an optimal mask under sparsity constraint. However, without retraining, the problem becomes intractable. To address this, we decompose Transformer pruning into three sub-problems, each of which can be efficiently solved; the first two stages of our pipeline address the problems of finding an optimal binary mask, and the last stage further optimizes into real-valued mask.
Note that the number of the mask variables is much less than the number of the parameters in a Transformer (e.g., 37K 110M in case of BERT This allows the framework number of examples without overfitting to the sample dataset, and thus to be extremely faster than the retraining-based pruning methods which typically the model "as is" and only decides the mask variables, we henceforth regard the model parameters as constants and consider the mask variables as the only parameters for our pruning problem.
only small
the entire dataset. As the framework keeps
Problem Formulation. We formulate Transformer pruning as a constrained optimization problem on the mask
arg min (m)) s.t. Cost(m) S C
where C denotes the loss function, Cost is the FLOPs/latency of the architecture pruned by the mask, and C' is the given FLOPs/latency constraint. Unfortunately, such a problem is generally intractable as Cost is usually a function (o-norm of the mask m, which is non-differentiable. Thus, in what follows, we introduce several assumptions and approximations to simplify the problem.
We start by approximating the loss function using the second-ordes Taylor expansion around the initial mask 1: [(m) [(1) gT(1 (1-1m)) (1) + ] (1- TH m)
1
f(1 2 T) m):
where g = El has converged local minima, where the gradient term close to 0 491 As L(1) a constant, can rewrite the optimization objective as follows:
5m [1)] and H = E 7m [()) Eq. 3 is deduced from an assumption that the model
82
arg min [(m) arg aigmin(1- m) TH m).
Eq. shows that the optimal mask is determined by the Hessian of the loss with respect to the mask variables. Since forming the exact Hessian matrix explicitly is infeasible, we approximate the Hessian with the (empirical) Fisher information matrix I of the mask variables:
2
=
Σ
C(x, vi 1))
C(x y 1))²,
(5)
D
Fm
Fm
(a,y)ED
where D is the sample dataset and (xr, y) is a tuple of an input example and its label.
4.1 Fisher-based Mask Search
Diagonal Approximation of the Fisher Information Matrix. It is intractable to solve the opti- mization objective in Eq. Jusing the full Fisher information matrix I. Thus, we first make a simple assumption that I is diagonal. This further simplifies Eq. ass follows:
arg min ((m) arg min (1 mm;)
(6)
Since we restrict the possible mask values to either or 1, the following can be derived from Eq. 6
arg min ((m) ~ arg min m
Σ In where Z(m) = (i m. 0}.
m
(EZ(m)
We can interpret each diagonal element of I as the importance score of the head/filter associated with the mask variable, and Eq. as a process of minimizing the total importance scores of the pruned heads and filters. Such an importance score has also been introduced in [52] 74 to guide pruning.
5

---

Algorithm Mask Search with FLOPs Constraint
Input: FLOPs constraint C, diagonal Fisher information matrix I
1: for 0 to LH do
# remaining heads
2: kj LH - n
D # heads to prune
3: HI indicies of k1 least important heads
4:
f = L(C' -nFbaadd /fiter]
> # remaining filters
5:
h2 LN f
# filters to prune
6:
Fl indicies of l2 least important filters
7: S[n] ΣiEHLHI Iu
8: R|n (HI, FI)
9: end for
argmin, S[n]
optimal # remaining heads > indicies of heads/filters to prune
11: HI" "I", =
R(n=]
Initialize MIlA and m as 1
13: m [MI
prune the selected heads > prune the selected filters
FFFN(FF [FI"] =0
Output: m²
= (m MHA
m FPN
Solving FooPseconstrained Problem. We need to solve Eq. 7 given a cost constraint. For a given target FLOPs cost, denoted by C, we can formulate the binary mask search problem as follows:
arg min m
Σ
Iis
s.t. Fhead MIA
MIA
Ilo + Fnterl m
1o S C,
(m)(m))
where Fhead E R and Friter E R are the FLOPs for computing a head and a filter, respectively. Note that the number of FL.OPs of head/filter is constant across all layers. While such optimization problem can be generally solved by knapsack algorithm [ [5 the following observations allow faster polynomial-timg solution: (1) having more heads and filters unpruned always optimizes Eq. since the diagonal elements of I are non-negative; and (2) if a certain number of heads needs to be pruned, they should be the ones with the lowest importance scores because each head accounts for the same amount of FLOPs. The same statement also holds for pruning filters. The two observations lead to our mask search algorithm described in Algorithm
Algorithm I partitions the solution space by the total number of remaining heads in the pruned architecture (n in line For each n, the number of remaining filters should be the largest possible number that satisfies the cost constraint by observation (1), which can be described as f line Then by observation (2), the heads/filters with the lowest important scores are selected be pruned. Therefore, S'n] is a solution of Eq. under additional constraint of fixing the number of remaining heads be n. When the loop terminates, the output is the mask that minimizes S'n] across all possible n (line I0 and [1p. In Section A. we prove that the output mask m of Algorithm D is optimal. That is, any other mask satisfying the given FLOPs constraint will result higher loss:
Σ
I.
Σ I&.
iEZ(m")
iEZ(m)
Solving Latency-constrained Problem. If the cost constraint is given in terms of latency target hardware, we have a new optimization problem with different cost constraint than Eq.
arg min
ΣI s.t.
LAAT(M) MHA + Σ LAT(M) PEN CC
(10)
imZ(m)
1=1
where the function LAT indicates the latency of MHA/FFN layer after pruning. We assume that latency lookup table on the target hardware provided so that evaluating LAT takes negligible time.
Unfortunately. the latency constraint makes the problem more challenging as directly applying Algorithm is no longer possible. This because LAT not linear the number of remaining heads or filters after pruning [59]. as shown in Figure ( (Left). We can interpret this as follows: (1) with sufficient number of heads/filters in a layer, the hardware resources such as parallel cores can be fully utilized, resulting in latency roughly proportional to the number of heads/filters: and (2) otherwise,
the hardware is underutilized and constant overhead dominates the latency 33 50
Thus, pruning
more heads/filters below certain threshold does not translate into actual speedup.

---

Barch
MHHAL
Latency
N FEN1
Latency 1
Slope
11
LHH+M
H
MHHAZ
2004
2048
3072
Filters
Heads/Filters
Figure 3: (Left) Real latency of a single FFN layer with differ- ent numbers of remaining filters. (Right) Schematic plot for the approximated latency as a piece-wise linear function.
Figure 4: Illustration of the block diagonal Fisher matrix.
Based on the above analysis, we approximate LAT as a piece-wise linear function as in Figure (Right) such that (m)(m;) is 0 if |mm/loo = O, c if < ||millo I T. and a(IImyllo- - T) + ||millo > 7, where e R is the constant overhead, 7 E N is the threshold number of heads/filters that the latency starts to become linear, and E the slope of the linear part. This can easily obtained by fitting the actual latency data in the lookup table with the minimum mean squared error.
The piece-wise linear approximation of LAT allows to extend Algorithm to the setting with latency constraints. The core idea is to separately consider the constant part and the linear part of LAT; after handling the constant part, we can apply the Algorithm the linear part. The detailed modification to Algorithm is described in Section A.2
4.2 Fisher-based Mask Rearrangement
Block Diagonal Approximation of the Fisher Information Matrix. Although it simplifies the problem, the diagonal assumption in Section 4.1 alone might not find the best solution, as it does not take into account the interactions between different mask variables. For example, if there two attention heads playing a similar role in a layer, pruning only one of them might not affect the model accuracy. However, when both of them degraded. Such interactions are captured by the non-diagonal elements of the Fisher information matrix, which were ignored in the previous stage. Thus, we can better consider the interactions in our pruning problem by using a block diagonal approximation to the Fisher matrix, where a block corresponds to a MHA layer or a FFN layer as illustrated in Figure
pruned, the model accuracy can be significantly
However, the block diagonal approximation results in an intractable optimization problem over the binary mask. To alleviate this, use the results from the previous stage warm start the optimization problem. First, we constrain the number of heads/filters to prune for each layer to be the same as the binary mask we obtained in the first stage. In other words, given the mask m² obtained in Section 4.1 we constrain |ymloo to be equal to |mm|loo for each layer l. Second, we use the mask m* as the starting point of the greedy search solve the optimization problem.
Given the two assumptions that (i) there is no interaction between the mask variables in different layers (i.e., the block diagonal approximation), and (ii) the number of heads/filters to prune are pre-determined for each layer (i.e., warm-start), Eq. 4 breaks down to set of layer-wise optimization
problems, follows based on the derivation in Section A.3
m arg min (1- m:) ) TI mz),
(11)
mi
where Z, is the (-th diagonal block I. We approximately solve this problem with greedy algorithm. After initializing the mask my as m; (i.c., warm-start), we pick for every round a pruned head (or filter) with the highest Fisher information and exchange it with unpruned head (or filter) in the current mask that can further optimize Eq. 11 After every pruned head/filter goes through one round, obtain approximate solution to Eq.
Because this process does not change the number heads/filters each layer, the obtained mask mi results in the same FL.OPs/latency as that of the mask m. searched in Section 4.1 In effect, this process rearranges the binary mask variables of each layer to find a better arrangement of pruning locations and capture the latrr-layer interactions.

---

BERT
BERT
DistIBERT
DistiIBERT
MNLI
QQP
QNLI
SST-2
Accuracy
Accuracy
Accuracy
Accuracy
Relative FLOPs
Relative FLOPs
Relative FLOPS
Relalive FLOPS
STS-B
MRPC
SQUAD 1.1
SQUAD 2.0
Accurecy
12.5
10.0
05.0
Rolative FLOPs
FLOPs
Relative FLOPs
Rolative FLOPs
Figure 5: Accuracy of our pruning method applied to BERTBASE and DistilBERT with different
FLOPs constraints. The dashed horizontal lines indicate 1% accuracy drop from the baseline models. Note that these results can be achieved in only 39 and 135 seconds for GLUE and SQuAD benchmarks, respectively, on a single GPU system, as described in Table F
(in Section A.CIionAA.I0...
Table 1: Latency speedup of BERT BASE on single NVIDIA V100 GPU with different batch sizes. The latency is measured using Porch.. We constrain the accuracy degradation to be at most 1 1% from the baseline accuracy, and we report the largest speedup among those that satisfy the constraint.
Batch size
MNLI
QQP
QNLI
SST-2
STS-B
MRPC
SQuAD1.1
SQuAD2.0
Geo. mean
32
1,27x
1,42x
1.42x
1.23x
1.34x
1,36x
1,33x
1.37
1.34x
256
1.34x
1.54x
1.53x
1.56x
1.54x
1,55x
1.34x
1.40x
1.47x
4.3
Mask Tuning
In the previous two stages, the possible mask values are restricted to either 0 or in order to simplify the search process. In this stage, we further relax this restriction. The nonzero variables in the mask mn from Section 4.2 are tuned to any real values such that the pruned model recovers its accuracy.
Layer-wise Reconstruction via Linear Least Squares. We tune the mask variables toward min- imizing the layer-wise reconstruction error, similarly to (3). From the first to the last layer, we reconstruct the output activation of the original model with the remaining heads/filters in the pruned model. This can be formally written as follows:
arg min Xx + layer(x; (x' layer(x'; 1)
(12)
where layer either MHA or FFN, and x and are the inputs to the layer of the pruned model and the original model, respectively. Here we compare the activations after the residual connection. Note that this stage does not incur any change in model FLOPs/latency, as we only tune the nonzero mask variables. We show in Section A.4 that Eq. 12 can be reduced to linear least squares problem of arg minm, ||Am, - bllž, where the matrix A denotes head/filter-wiz output activations of the model pruned by the binary mask and the vector b is the difference between the output activations of the two models. Concretely, when there are T tokens the sample dataset and D the hidden size of the model, the size of the matrix A is TD x H for head masks and TD x N for filter masks.
Due to the large size of the matrix A, naively solving the least squares problem can lead numer- ically unstable results. To address this, our framework uses the LSMR solver in CuPy (55) with a regularization hyperparameter (i.e., damp). Concretely, we re-parameterize the least squares problem as arg min, ||, + A 1 --bbl bl12 where m; = 1 and solve it with the damp value fixed to 1. Then, to prevent the case in which the tuned mask rather hurts the accuracy, we restrict the acceptable range of the tuned mask variables to [-10, 10]. When the solver finds a layer mask that exceeds this range, we discard the mask for that layer and stop mask tuning. In our experiments, we find that the aforementioned heuristics make the mask tuning process highly stable across different models, tasks, and seeds. Furthermore, while the use of the heuristics involves the two hyperparameters (i.e., damp and the acceptable range), we empirically find that these need not be tuned for different tasks and models. In all of our experiments, we fixed the two hyperparameter values mentioned here.

---

Ours
SLIP
EBERT
Sajjad et al.
A 8MP
FLOP
DynaBERT
*
CoFi
Q0P
QNU
S5T-2
MRPC
-0.2
A,ccurrcy
Accuracy
Accuracy
-16
-3.5
-35
-00
-246
033
3.5
0.83
3.5
2.5
6-6
6.70
6/8 0R 6.1
335
055 043
67
FLOPS
FLOPs
FLOPs
Relative
Figure 6: Amount of accuracy degradation from the baseline when pruning BERT BASE using our method and the prior structured pruning methods with different relative FLOPs. Note that our method does not require retraining. whereas all the other methods involve significant retraining overheads as described in Table
5
Evaluation
5.1
Experimental Setup
Our framework is implemented top of PyTorch [57 and the HuggingFace Transformers 86 library. We evaluate the effectiveness of our approach using BERTBASE [I2] and DistilBERT 63 on GLUE 78 and SQuAD [60] 61 benchmarks. We 2K examples from the training sets for pruning, and we evaluate the resulting models on the development sets. All of the results are averaged over the runs with 10 different seeds. More details the experimental setup can be found in Section A.5
5.2 Performance Evaluation
FLOPs. Figure shows the accuracy BERTBASE BASE and DistilBERT with different FLOPs constraints on GLUE and SQuAD datasets. As can be seen in the plots, with only 1% of accuracy drop, BERT achieves 60-70% of the original FLOPs for all tasks. DistilBERT also shows a similar pattern and achieves up to 50% FLOPs reduction (in STS-B and MRPC) compressed architecture. More results using larger sample datasets are provided Section A.6
though it is already
Latency. We further measure the latency on real hardware by pruning BERT BASE with latency constraints and deploying the resulting models on an NVIDIA V100 GPU. Table lists the latency speedup with maximum accuracy drop of 1% for GLUE and SQuAD datasets. With batch size of 256, we achieve speedup of 1.47 x on average and up to 1.56x.
5.3 Comparison with the Prior Methods
FLOPs and Accuracy Comparison. Here, we compare our method with the prior structured pruning methods for Transformers including Flop 83 SLIP
4F Sajjad al. [2] DynaBERT (5)
by the FLOPs-accuracy trade- off of BERTBASE on GLUE tasks. We use the results without knowledge distillation and data augmentation reported in each paper. Since the baseline accuracy differs slightly from paper to paper, we compare the amount of the accuracy drop from the baseline instead the absolute accuracy. The results are plotted as Figure 6 We include the comparison details and full table in Section A.7
EBERT 4Р Block Movement Pruming (BMP) 38 and CoFi NN
Interestingly, our method exhibits comparable better results than the prior methods without any model retraining and with substantially lower pruning costs. This empirically demonstrates that retraining and complex pruning pipeline Transformers. For high sparsity, we find that our framework with retraining works comparably to or better than the prior methods the same pruning cost (See Section A.8
necessary for moderate level of pruning of
Retraining Cost. We select DynaBERT 255, EBERT 49 BMP 38 and CoFi 88] that achieve comparably good accuracy in Figure and systematically analyze their end-to-end retraining costs on MNLI dataset. As shown in Table 2 these methods require 5-33 hours of retraining, On the other hand, method finishes in less than minute, which is 2-3 orders of magnitude faster. We also highlight that this training latency analysis only accounts for a single hyperparameter, and the entire cost should be multiplied by the size of the hyperparameter space. While the prior methods rely on a considerable number of hyperparameters, ours introduce only two hyperparameters (in Section which fix for all experiments. See Section A.9 for more details.

---

Table Pruning cost comparison between the prior structured pruning methods and ours. We compare the number of training epochs and the end-to-end (E2E) time required for pruning.
Weight magnitude
Gradient- based
Ours
QQP (BERT-bol bbsee)
90
Accuracy
80
# Epochs
E2E time (hr)
DynaBERT
2
12
70
EBERT 40
BMP 38
20
17
0.50
0.58
0.66
0.75
0.83
CoFi
40
33
Relative FLOPs
Figure 7: Retraining- accuracy without (dotted) and with (solid) mask tuning.
Ours
0.01
Table 3: Ablation of our mask search, rearrangement, and tuning methods, described in Section We use BERTBASE BASE as the baseline model, and we prune it with a 60% FLOPPs constraint.
MNLI QQP QNLI SST-2 STS-B
MRPC SQuAD,.1 SQuAD2.0 Avg. Diff
Baseline
84.53
91.00 91.41
93.57
88.90
86.27
88.48
76.82
Mask Scarch
81.21
89.99
88.38
92.13
87.10
83.14
82.66
71.12
Mask Rearrangement
81.81
90.08
88.77
92.09
87.68
83.23
84.47
72.38
0.60
Mask Tuning
82.51
90.35
90.06
92.49
88.00
85.27
86.72
75.26
1.27
5.4 Discussion
Ablation Studies. Table 3 lists an ablation of the mask rearrangement (Section 4.2) and tuning (Section 4.3) stages for pruned BERT BASE with 60% of FLOPs. We find that both stages help recover the baseline accuracy, and that mask tuning is in particular critical, recovering up to 2.88% accuracy.
To further investigate the importance of the mask search and rearrangement, we compare the retraining-free performance of the binary masks obtained by our method and other pruning cri- teria: weight magnitude and the gradient- layers using the two criteria with different width multipliers. Figure shows that the two methods significantly degrade the accuracy under the low sparsity regimes. Even with mask tuning, the accuracy not fully recovered. The results demonstrate that our mask search and re-arrangement are necessary to get optimal binary masks, and that mask tuning is only effective when the binary mask preserves high accuracy. More ablation studies can be found in Section [A.IT] and Section A.
method used in DynaBERT. We uniformly pruned the
Time Breakdown. We break down our pruning pipeline into 4 parts gradient computation, mask search, rearrangement, and tuning-and we measure the latency for each stage Table 5 (Sec- tion 10). For GLUE and SQuAD tasks, our framework finishes in 39 and 135 seconds on average.
Conclusion
In this work, we have proposed a novel post-training pruning framework for Transformers that does not require model retraining. The core of our framework is the three-stage decomposition of the pruning process. It uses a fast Fisher-bases mask search algorithm to decide which heads/filters to prune, rearranges the pruned heads/filters, and tunes the mask variables to recover the output signal for each layer. We empirically evaluate our framework using BERT BaSE and DERT where our pruning method achieves up to 50% FL.PPs reduction within only 1% accuracy degradation on GLUE and SQuAD datasets. This results in up to 1.56x latency speedup on an NVIDIA V100 GPU. Importantly, our end-to-end pruning pipeline only needs 39 and 135 seconds for GLUE and SQuAD, which is 2-3 orders of magnitude faster than the prior methods.
Aedmowledgments and Disclosure of Funding
The authors would like thank Suhong Moon who helped with brainstorming. We also acknowledge gracious support from Google Cloud, Google TRC team, and specifically Jonathan Caton, Prof. David Patterson, and Jing Li. Prof. Keutzer's lab is sponsored by Samsung. Intel corporation, Intel VLAB team, Intel One-API center of excellence, as well as funding through BDD and BAIR. Woosuk Kwon and Sehoon Kim acknowledge the support from Korea Foundation for Advanced Studies. Amir Gholami was supported through funding from Samsung SAIT. Michael W. Mahoney would also like to acknowledge the UC Berkeley CLTCC ARO, NSF, and ONR. Our conclusions do not necessarily reflect the position the policy of our sponsors, and no official endorsement should be inferred.
10

---

References
[1] Yonathan Aflalo, Asaf Noy. Ming Lin, Itamar Friedman, and Lihi Zelnik. Knapsack pruning with inner distillation. arXiv preprint arXiv:2002.0842 08258, 2020.
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self- supervieed learning of speech representations. arXiv preprint arXiv:2006.1147, 2020.
[3] Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry. Post-training 4-bit quantization of convolution networks for rapid-deployment. arXiv preprint wwXiv:1810.05723, 2018.
[4] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot leamners. arXiv preprint 2005v 14165, 2020.
[5] Daniel Cer, Mona Diab, Eneko Agirre, Inigo I: Semantic textual similarit-mulliinguaal and cross-lingual focused evaluation. arXiv preprint arXiv: 170.00/70 2017.
and Lucia Specia. Semeval-2017 task
[6] Daoyuan Chen, Yaliang Li, Minghui Qiu, Zhen Wang, Bofang Li, Bolin Ding, Hongbo Deng, Jun Huang, Wei Lin, and Jingren Zhou. Adabert: Task-adaptive bert compression with diffcrentiable neural architecture search. arXiv preprint arXiv:2001. 04246, 2020.
[7] Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, al. Wavim: Large-scale self-supervised pre-training for full stack speech processing. arXiv preprint a 10. 13900, 2021.
[8] Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end- cod exploration. Advances in Neural Information Processing Systems, 34:19974- 199888 2021.
[9] Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang. Zhangyang Wang. and Michael Carbin. The lottery ticket hypothesis for pre-trained BERT networks. arXiv preprint arXiv:2007. 1222, 2020.
[10] Xiaohan Chen, Yu Cheng. Shuohang Wang, Zhe Gan, Zhangyang Wang, and Jingjing Liu. Earlybert: Efficient bert training via early-birt lottery tickets. arXiv preprint Stti:2201100006: 2020.
[11] Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177-190. Springer, 2005.
[12] Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv: 1810.04805, 2018.
[13] William B Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005.
[14] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11 19999, 2020.
[15] Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv:190..15555 2019.
[16] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprini 10X.v:1803.03635, 2018
[17] Elias Frantar and Dan Alistarh. Spdy: Accurate pruming with speedup guarantees. arXiv preprint arXiv:2201.13096, 2022.
[18] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv: 192.00944 2019.
[19] Google.
Tensorflow Lite: http::/www..cnnsfffooo.ortttttt. 2017
[20] Cong Guo, Bo Yang Hsueh, Jingwen Leng. Yuxian Qiu, Yue Guan, Zehuan Wang. Xiaoying Jia, Xipeng Li. Minyi Guo, and Yuhao Zhu. Accelerating sparse dan models without hardware- via tile- wise sparsity. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-15. IEEE, 2020.
11

---

[21] Tae Jun Ham, Sung Jun Jung, Seonghak Kim, Young H Oh, Yeonhong Park, Yoonho Song, Jung-Hun Park. Sanghee Lee, Kyoung Park. Jae W Lee, al. A 3: Accelerating attention mechanisms in neural networks with approximation. In 2020 IEEE International Sympusium on High Performance Computer Architecture (HPCA), pages 2288341. IEEE, 2020.
[22] Tae Jun Ham, Yejin Lee, Seong Hoon Seo, Soosung Kim, Hyunji Choi, Sung Jun Jung, and Jae Lee. Elsa: Hardware-softweed co-design for efficient, lightweight self-attention mechanism in neural networks. In 202/ ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), pages 692-705. TEEE, 2021.
[23] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pages 1389 2017.
[24] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv: arXiv:1066.000 1606.0841, 2016.
[25] Lu Hou, Zhiqi Huang. Lifeng Shang, Xin Jiang. Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. arXiv preprint (xXxx2004404037. 2020.
[26] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self supervised speech representation learning by masked prediction of hidden units. arXiv preprint arXiv:2106.0m 07447, 2021.
[27] Itay Hubara. Yury Nahshan, Yair Hanani. Ron Banner, and Daniel Soudry. Improving post training neural quantization: Layer-wise calibration and integer programming. arXiv preprint arXiv:2006. 10518, 2020.
[28] Forrest N landola, Albert E Shaw, Ravi Krishna, and Kurt W Keutzer. Squeezebert: What computer vision teach nlp about efficient neural networks? arXiv preprint wXii:2006.113166 2020.
[29] Intel. Open
htmll, 2021.
[30] Shankar Iyer, Nikhil Dandekar, and Kornl Csernai. First quora dataset release: Question (2017. URL. https://daia. quora. com/First-Quonn Quu-u--Datsee--Reeeeeee Reltase--Relesse--
-eleeeea---e--essinnn-eee Aduestion-Pairs. 2017.
[31] Xiaoqi Jiao, Yichun Yin, Lifeng Shang. Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint 1rXiv:190..10333 2019.
[32] Ashish Khetan and Zohar Karnin, schubert: Optimizing elements of bert. arXiv preprint 200:: 2020.
[33] Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. I-hert: Integer-only bert quantization. arXiv preprint arXtv:2101.0132// 2021.
[34] Woojeong Kim, Suhyun Kim, Mincheol Park, and Geonscok Jeon. Neuron merging: Compensating for pruned neurons. arXiv preprint arXiv:2010.13100 2020.
[35] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2019.
[36] Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. arXiv preprint (tXXx:2223300255. 2022.
[37] Woosuk Kwon, Gyeong-In Yu, Eunji Jeong, and Byung-Gon Chun. Nimble: Lightweight and parallel gpu task scheduling for deep learning. arXiv preprint arXiv:2012.02732, 2020.
[38] François Lagunas, Ella Charlaix, Victor Sanh, and Alexander Rush. Block pruning for faster transformers. arXiv preprint arXiv:2109.0445 04838, 2021.
[39] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut, Al- bert: A lite bert for superviue learning of language representations. arXiv preprint arXiv:1909, 19442, 2019.
[40] Ivan Lazarevich, Alexander Kozlov, and Nikita Malinin. Post-training deep neural network pruning via layer-wise calibration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 798-805, 2021.
[41] Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pages 598-605, 1990.
12

---

[42] Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth International Conference on the Principles Knowledge Representation and Reasoning. Citeseer, 2012.
[43] Binghing Li, Zhenglun Kong, Tianyun Zhang. Ji Li, Zhengang Li, Hang Liu, and Caiwen Ding. Efficient transformer-base large scale language representations using fardeale-friendly block structured pruning. arXiv preprint arXiv:2009.0000 2020.
[44] Zi Lin, Jeremiah Zhe Liu, Zi Yang, Nan Hua, and Dan Roth. Pruning redundant mappings in transformer models via spectral-normalized identity prior. arXiv preprint arXi::20/001791, 2020.
[45] Liyang Liu, Shilong Zhang. Zhanghui Kuang, Aojun Zhou, Jing-Hap Hao Xue, Xinjiang Wang. Yimin Chen, Wenming Yang. Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. In International Conference on Machine Learning, pages 7021-7032. PMLR, 2021.
[46] Yinhan Liu, Myle Ott, Naman Goyal. Jingfei Du, Mandar Joshi, Dangi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint Xrv: / /907. /692, 2019.
[47] Yuanxin Liu, Zheng Lin, and Fengcheng Yuan. Rosita: Refined bert compression with integrated techniques. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 8715 8722, 2021.
[48] Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint 1rXiv:2103.14330. 2021.
[49] Zejian Liu, Fanrong Li, Gang Li, and Jian Cheng. Ebert: Efficient bert inference with dynamic structured pruning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 481 4-- 4823, 2021.
[50] Lingxiao Ma, Zhiqiang Xie, Zhi Yang, Jilong Xue, Youshan Miao, Wei Cui, Wenxiang Hu, Fan Yang, Lintao Zhang, and Lidong Zhou. Rammer: Enabling holistic deep learning compiler optimizations with rtasks. In 14th (USENIX) Symposium on Operating Systems Design and Implementation ( ((OSDI) 20). pages 881 881-897, 2020.
[51] Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? arXiv preprint arXiv: 10650, 2019.
[52] Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11264-11272, 2019.
[53] Ben Mussay, Margarita Osadchy, Vladimir Braverman, Samson Zhou, and Dan Feldman. Data-independent neural pruning via coresets. arXiv preprint arXiv: 040188 2019.
[54] Markus Nagel. Rana Ali Amjad, Mart Van Baalen, Christos Louizos. and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pages 7197-7206. PMLR, 2020.
[55] ROYUD Nishino and Shohei Hido Crissman Loomis. Cupy: calculations. 31 confernce neural information processing systems, 151, 2017.
A numpy compatible library for nvidia gpu
[56] NVIDIA. TensorRT: https//deeeeooeerrvddddd.oooooes httpppppeeeeeiiiiacoooo//eooottt 2018
[57] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, sigh--perfommnce deep learning library. Advances in neural information processing systems, 32:8026-8037, 2019.
[58] Sai Prasanna, Anna Rogers, and Anna Rumshisky. When BERT plays the lottery, all tickets are winning. arXiv preprint arXiv:2005.008 2020.
[59] Valentin Radu, Kuba Kaszyk, Yuan Wen, Jack Turner, José Cano, Elliot J Crowley, Bjöm Franke, Amos Storkey, and Michael O' Boyle. Performance aware convolutional neural network channel pruning for embedded gpus. In 20/9 IEEE International Symposium on Workload Characterization (IIWC), pages 24-34. IEEE, 2019.
[60] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable questions for squad. arXiv preprint arXiv:1806 18088880082.... 2018.
13

---

[61] Pranav Rajpurkar. Jian Zhang, Konstantin Lopyrev. and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.
[62] Hassan Sajaad Fahim Dalvi, Nadir Durrani, and Preslav Nakov. On the effect of dropping layers of pre-trained transformer models. arXiv preprint arXiv:2004.0388 2020.
[63] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 108, 2019.
[64] Victor Sanh, Thomas Wolf, and Alexander M Rush. Movement pruning: Adaptive sparsity by fine-tuning. arXiv preprint arXiv:2005.0766 2020.
[65] Maying Shen, Hongxu Yin, Pavlo Molchanov, Lei Mao, Jianna Liu, and Jose M Alvarez. Halp: Hardware- aware latency pruning. arXiv preprint arXiv:21 2021.
[66] Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of hert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8815-8821, 2020.
[67] David So, Quoe Le. and Chen Liang. The evolved transformer. In International Conference on Machine Learning, pages 5877-5886. PMLR, 2019.
[68] David R So, Wojciech Manke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V Lc. Primer: Searching for efficient transformers for language modeling, arXiv preprint 1iti:::2009006666 2021.
[69] Richard Socher, Alex Perelygin, Jean W Jason Chuang, Christopher D Manning. Andrew Y Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedingy of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, 2013.
[70] Suraj Srinivas and R Venkatesh Babu. Data-free parameter pruning for deep neural networks. arXiv preprint 1rXiv:1507.06144 2015.
[71] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint arXiv: arXiv:/908 199800993555 2019.
[72] Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: a compact task aanossti bert for resource-limited devices. arXiv preprint arXiv:2004.02984, 2020.
[73] Thierry Tambe. Coleman Hooper. Lillian Pentecost, Tianyu Jia, En- En Yang, Marco Donato, Victor Sanh, Paul Whatmough, Alexander M Rush, David Brooks, et al. Edgehert: Sentence-level energy optimiza- tions for latency-aware multi-task nip inference. In MICRO-54; 54th Anmual IEEE/ACM International Symposium on Microarchitecture, pages
2021.
[74] Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferenc Huszár. Faster gaze prediction with dense networks and fisher pruning. arXiv preprini arXiv:1801.0577 2018
[75] Hugo Touvron, Matthieu Cord, Mathhij Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10351 PMLR, 2021.
[76] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
[77] Elena Voita, David Talbot, Fedor Moiscev, Rico Sennrich, and Ivan Titov, Analyzing multi-head self- attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv: 190.:0995509 2019.
[78] Alex Wang, Amanprect Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman, GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 1804.0746,, 2018.
[79] Hanrui Wang. Zhanghao Wu, Zhijian Liu, Han Cai. Ligeng Zhu, Chuang Gan, and Song Han. Hat: Hardware-aware transformers for efficient natural language processing. arXiv preprint arXiv:2055. 14187, 2020.
[80] Hanrui Wang, Zhekai Zhang, and Song Han. Spatten: Efficient sparse attention architecture with cas- cade token and head pruning. In 2021 IEEE International Symposium on Perforaneee Computer Architecture (HPCA), pages 97-110. IEEE, 2021.
14

---

[81] Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint 047762 2020.
[82] Wenhui Wang, Furu Wei, Li Dong, Hangho Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. arXiv preprint 200:::::002.1095555 2020.
[83] Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. arXiv preprint 1110:499 2019.
[84] Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641 2019.
[85] Adina Williams, Nikita Nangia, and Samuel R Bowman, A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint (xXiv1100.054226 2017.
[86] Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer. et al. Transformess: Sfate-of-the-art natural language processing. Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pages 38 45, 2020.
[87] Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. Lite transformer with long-short range attention. arXiv preprint arXiv:2004. 1186, 2020.
[88] Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 15 1513-1528, 2022.
[89] Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. Deebert: Dynamic early exiting for accelerating bert inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2246-2251, 2020.
[90] Jin Xu, Xu Tan, Renqian Luo, Kaitao Song, Jian Li, Tao Qin, and Tin Liu. Nas-bert: Task-agnostic and adaptive-size bert compression with neural architccture search. arXiv preprint arXiv:2105. 10..21005..4444 14444, 2021.
[91] Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. Xlnet: Generalized autorcgressive pretraining for language understanding. Advances neural information processing systems, 32, 2019.
[92] Zhewei Yao, Linjian Ma. Sheng Shen, Kurt Keutzer, and Michael W Mahoney. MIpruning: A multilevel structured pruning framework for transformer- models. arXiv preprint arXiv:2105. 14636, 2021.
[93] Yichun Yin, Cheng Chen, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Autotinybert: Automatic syper-parameter optimization for efficient pre-trained language models. arXiv preprint urXiv:2107. 13686. 2021.
[94] Edouard Yvinec, Arnaud Dapogny, Matthieu Cord, and Kevin Bailly. Red: Looking for redundancies for data-frrestructuuee compression of deep neural networks. Advances in Neural Information Processing Systems, 34:20863 2222736666283,,,..- 2021.
[95] Ali Hadi Zadeh, Isak Edo, Omar Mohamed Awad, and Andreas Moshovos. Gobo: Quantizing attention- based nlp models for low latency and energy efficient inference. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). pages 811- IEEE, 2020.
[96] Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized Sbit bert. arXiv preprini arXiv:1910.06155. 2019.
[97] Ritchie Zhao, Yuwci Hu, Jordan Dotzel, Christopher De Sa, and Zhiru Zhang. Improving neural network quantization without retraining using outlier channel splitting. Proceedings of Machine Learning Research, 2019.
[98] Wangchunshu Zhou, Canwen Xu, Tao Ge. Julian McAuley, Ke Xu, and Furu Wei. Bert loses patience: Fast and robust inference with early exit. Advances in Neural Information Processing Systems, 33: 18330-18844, 2020.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] 'he notations
are defined in Section 3.1 and our problem is formally described in Section
(b) Did you include complete proofs of all theoretical results? [Yes] Section A. contains
the proof of the optimiality of Algorithm !I Section A.3 and Section describe the derivation processes omitted in the paper.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material
URL)? [Yes]] The GitHub
link to our code can be found in the first page.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] Our method has only two hyperparameters which were fixed in all of our experiments (See Section 4.3) p. For details on the datasets, see Section 5.]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No] Because our method is training- free, the results are very stable across seeds. For simplicity,
omitted the error bar.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] For all experiments, we used an AWS p3.2xlarge instance which has NVIDIA V100 GPU. See Section 5.T for details.
4. If you using existing assets (e.g., code, data, models) curating/releasing
assets... (a) If your work uses existing assets, did you cite the creators? [Yes] We cited the HuggingFace transformers library, from which we got the pre-trained Transformer weights.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "sSV1Y2960CD", "reviewer": "Reviewer_CMhk", "summary": "This paper proposes three  techniques to obtain a high accuracy transformer without  retraining. A search algorithm to find which heads and filters are need to prune based on the Fisher information. An algorithm which rearrange the mask that complements the search algorithm. And tuning the mask which reconstructs the output activations for each layer. The experiments show the author can get better results than compare methods.", "questions": "1、How to obtain latency exactly? Is the latency become different when obtain them from different hardware and inference engines\n\n", "limitations": "Please see my weaknesses", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThis paper is well-written, well-motivated, and clear presentation.\n\nThe proposed algorithm improves transformer throughout efficiency with competitive accuracy and small latency, outperforming prior pruning and distillation approaches.\n\nFormulating the hardware-aware structural pruning as a knapsack problem is interesting.\n\nWeaknesses:\n\nThis paper should compare their algorithms with some SOTA pruning methods on transformer such as COFI[1]\n\n[1]Xia M, Zhong Z, Chen D. Structured pruning learns compact and accurate models[J]. arXiv preprint arXiv:2204.00408, 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "rO9dYgiVg5P", "reviewer": "Reviewer_Lp5s", "summary": "This paper proposes a three-stage post-training pruning framework for Transformers.  It first uses Fisher information to search the binary mask, i.e, layer-wise pruning rate. Then, the framework modifies the binary mask in a layer-wise manner. Lastly, it tunes the non-zero valued mask for minimizing the layer-wise reconstruction error.\n\nSuch a framework could retain the performance of the model without retraining, thus it can finish the pruning in less than 3 minutes on a single GPU and can obtain actual speedup in inference because of structured pruning.\n\nExtensive experiments show that the proposed post-training pruning framework has a comparable performance with prior methods.\n", "questions": "1. Figure 3 (Left) only shows that  the actual speed is linear to the number of filters, which does not support the claim in line206-207. I suggest authors re-plot Figure 3 (Left) with fewer filter number.\n\n2. Technical merits are limited.  Each component used in the three-stage pruning has been used in prior works. For example,   taking Fisher Information Matrix as an importance score and  tuning the mask variables toward minimizing the layer-wise reconstruction error.\n\n3. I can not understand why the mask by Eq. (4) and Eq. (11) is different because the importance score is the same.   Line 237 claims that $||m_l||_0$ is  equal to $||m_l^*||$. Line 242 claims that $\\mathcal{I}_l$ is the  $l$-th diagonal block of $\\mathcal{I}$. In such situation, $m_l^*$ selected by $\\mathcal{I}$  and $m_l$ selected by $\\mathcal{I}_l$ should be the same. This can be proved by contradiction.\n\n4. The ablation experiments are not clear enough. There are two ablative modules in Figure 6, including importance score and mask search algorithm. Such an ablation study with one more ablative module can not support the claim of Line 319.  The ablation experiment in Figure 7 should be divided into two experiments. 1) The importance score. Keeping the three-stage pruning pipeline and just modifying the importance score, which can reflect the efficacy of the importance score. 2) Mask search and rearrangement. Keeping the importance score and using uniformly prune, can reflect the efficacy of mask search and re-arrangement.\n\n\ntypos:\n\n- Line 613: Eq. ?? -> Eq. 9\n- Algorithm 2.2 pf->of\n\n==================post rebuttal============\nMy concerns are addressed. I raise up my score. Thanks.", "limitations": "No societal impact discussion needed, in my opinion.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. To my knowledge, this is the first work to use post-training pruning in Transformer. I recognize the contribution of applying technologies to new areas.\n\n2. Proposing simple mask search solutions based on FLOPs and Latency,  which avoids user intervention.\n\n3. Experiments show the efficacy  of the proposed framework, which could retain high accuracy without retraining.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "iQB9-BELUb", "reviewer": "Reviewer_92Qy", "summary": "The paper proposed a post-training pruning framework for Transformers, without retraining. It prunes of both heads in MHA and filters in FFN layers in a structured  way. The process is done by applying a lightweight Fisher based mask search along with a Fisher mask rearrangement and mask tuning. The results are comparable or even better FLOPs-accuracy trade-off than prior methods.", "questions": "1. The reason to use second-order Taylor. It seems that first-order Taylor can be faster and doesn't lose much accuracy? Some other works use first-order [1][2]\n\n\n[1]Accelerating Sparse DNN Models without hardware-Support via Tile-Wise Sparsity\n[2]Chasing sparsity in vision transformers: an end-to-end exploration", "limitations": "Please see the above comments. I will consider changing my ratings based on the author's rebuttal.\n\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper is well written, and the authors proposed the method in quite a detail. The post-training pruning framework does not require retraining, which is very good. Adding latency-constrained in consideration is also good.\nThe experiments are quite sufficient and are able to support their claims and conclusions. The results show the effectiveness of the proposed methods. The paper also compared existing structured pruning works on GLUE\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7e5_6VgnVW", "reviewer": "Reviewer_hRDe", "summary": "In this paper, the authors proposed a fast post-training pruning framework for transformer-based language models which does not require retraining. The algorithm takes a model, sample dataset, and compression constraint to generate the compressed model. It introduces three techniques to retain high accuracy: 1. mask search; 2. mask rearrangement; 3. mask tuning. Experiments show that the proposed method achieves 2x FLOPs reduction and 1.6x speed up within 1% accuracy drop.", "questions": "1. The authors may discuss if the proposed method is only restricted to transformers. Can it be applied to other models like CNNs? Is it possible that the post-training method only works well here due to the large redundancy of BERT models on a specific downstream dataset?\n", "limitations": "There lacks a discussion of potential negative societal impacts. It might be acceptable due to the nature of the work; but a discussion is still encouraged. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Firstly, the paper is well written and easy to follow. \n2. The proposed method solves a complex optimization problem by introducing several approximations and using a multi-step approach. The process is introduced clearly. The effectiveness of each step is demonstrated by ablation studies. \n3. The experimental results are solid. The proposed method achieves similar performance compared to existing work without a large training cost, which could be valuable to real-life applications. \n4. I like the discussion of latency-aware compression, where the authors used a piece-wise linear function to approximate the latency LUT, which is integrated into the optimization objective. It is a smart design to fit both settings under the same optimization framework. \n\nWeakness:\n1. Firstly, it seems that the proposed method is not limited to pruning transformers; it can be also applied to other models like CNNs by just using the channel mask. Is there any consideration why the work is limited to transformers? Will the algorithm perform well on other models like CNNs? \n2. There are some other works on post-training channel pruning (e.g., [a]). It would be better if the authors can show the proposed method can also outperform general data-free pruning methods when applied to transformers. But I also agree the current experimental results are already solid.\n\n[a] Lazarevich et al., Post-training deep neural network pruning via layer-wise calibration, ICCVW. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept