Accurate Retraining-free Pruning for Pretrained Encoder-based Language Models
Authors: Seungcheol Park
Hojun Choi
U Kang
Venue: ICLR 2024 poster (2024)
Keywords: Retraining-free
Pruning
Compression
Transformers
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose Kprune, an accurate retraining-free structured pruning algorithm for Transformers. Kprune shows up to 58%p higher F1 score than existing retraining-free pruning algorithms on the SQuAD benchmark.

Abstract: Given a pretrained encoder-based language model, how can we accurately compress it without retraining? Retraining-free structured pruning algorithms are crucial in pretrained language model compression due to their significantly reduced pruning cost and capability to prune large language models. However, existing retraining-free algorithms encounter severe accuracy degradation, as they fail to handle pruning errors, especially at high compression rates. In this paper, we propose KPrune (Knowledge-preserving pruning), an accurate retraining-free structured pruning algorithm for pretrained encoder-based language models.
KPrune focuses on preserving the useful knowledge of the pretrained model to minimize pruning errors through a carefully designed iterative pruning process composed of knowledge measurement, knowledge-preserving mask search, and knowledge-preserving weight-tuning. As a result, KPrune shows significant accuracy improvements up to 58.02%p higher F1 score compared to existing retraining-free pruning algorithms under a high compression rate of 80% on the SQuAD benchmark without any retraining process.

Published as a conference paper at ICLR 2024
ACCURATE RETRAINING-FREE PRUNING FOR PRE- TRAINED ENCODER-BASEL LANGUAGE MODELS
Seungcheol Park', Hojum Choif & U Kang"!
SSouul National University, Seoul, South Korea
2 Kim Jaechul Graduate School of AI, KAIST, Seoul, South Korea (ant 6si, ukang)@ ann..accenc......m.. ac.kr hchoi@kaist.ac..k
ABSTRACT
Given a pretrained encoder-basee language model, how can we accurately compress it without retraining? Retraining-free structured pruning algorithms are crucial in pretrained language model compression due to their significantly reduced pruning cost and capability to prune large language models. However, existing retraining- free algorithms encounter severe accuracy degradation, as they fail to handle pruning errors, especially at high compression rates, In this paper, propose K-prune (Knowledge- www.wlldd--peeeringggggg pruning), an accurate retraining- structured pruning algorithm for pretrained encoder-based language models. K K-prune focuses on preserving the useful knowledge of the pretrained model to minimize pruning errors through a carefully designed iterative pruning process composed of knowl- edge measurement, knowledge- pereeeerrgggggggingggg mask search, and knowwlcdge-reeservingg weight-tuning. As result, K-prune shows significant accuracy improvements up 58.02%p higher FI score compared to existing retraining- free pruning algorithms under high compression rate of 80% on the SQuAD benchmark without any retraining process.
INTRODUCTION
How can accurately compress pretrained encoder-based language models without retraining? Transformer- boasd PLMs dominate (Devlin et al. 2019; Clark et al. 2020 Liu et al. 2019 Brown al.) 2020 Zhang et al., 2022) the field of Natural Language Processing (NLP) based on their remarkable performance. The superiority of PLMs comes with a massive increase in their size, and the unaffordably scaled models necessitate compression algorithms that effectively reduce the size of PLMs without compromising accuracy.
prominent for compressing pretrained language models (PLMs) since they require dramatically lower computa- tional costs and a smaller amount of data than existing retraining based algorithms (Hou et al. 2020)
Retraining free structured pruning algorithms (Kwon al. 2022b Nova al. 2023)
Liu et al. 202 Lin et al. 2020: Wang et al.) 2020bb [Sajaad et al. 2023 Xia et al. [022]] Lagunas ct al. 202 ). Retraining-fe free algorithms achieve remarkable efficiency by replacing an expensive retraining process with shot mask search process followed by a lightweight mask-tuning process. However, when comes to the high compression rate, retraining-fre freee algorithms exhibit severe accuracy degradation. The accuracy degradation comes from failure of handling pruning errors which represent the distortion of the model's prediction by the accumulated deformations of the outputs of the pruned intermediate layers.
In this paper, we propose K-prune (Knowleegge-reservingg pruning), an accurate retraining- free structured pruning algorithm for encoder-based PLMs. We conceptualize pruning error as the loss of useful knowledge to explicitly measure the amount of pruning error. We observe that the main reason of severe accuracy degradation in previous retraining free pruning algorithms is an unrecoverable knowledge loss from multiple layers. Therefore, we carefully design an iterative pruning process that distributes the knowledge loss across multiple iterations overcome the accuracy degradation
Work done while at Seoul National University
Corresponning author

---

Published as a conference paper at ICLR 2024
K-Prune (ours)
Kwon SST-2
KCM
3% of the baseline
MRPC
QQP
STS-B
SQuAD 1.1
SQuAD 2.0
Spesemnn Com Pl
80
16.42%p
Accunayy 30
19p0
Arqumna (S
35.67%pp
& & a at &
8.53%ap
58.02%Pp
13
56
10
60
10
60
10
70
80
80
60
To
50
60
Reduoed FLOPs (%)
Reduced FLOPs
Reduced FLOPs (N)
Reduced FLOPs (hi)
Reduoed FLOPs (%)
Reduzed FLOPs (5)
(+)
MRPC
QQP
SST-2
STS-B
SQUAD 1.1
SQuAD 2.0
90
Spearmun Con
& & &
15
50.20%
24.42%p
15.50%
20.37%up
43.0
39.09%p
3
70
60 T0
70
Reduoed FLOPs
Reducod FLLOPs
Roduced (LOPs(N)
Redwved (1)))s
Redhced FLOPs (%)
Reduoed F1.0Ps (%)
(b) DistiBERT
Figure 1: Accuracy vs. reduced FLOPs of retraining free pruning algorithms using BERT and DistilBERT where the dotted line indicates the accuracy degradation of 3%p. K-prune (blue star) largely outperforms competitors in all settings.
problem. Our iterative pruning process consists of three steps which aim to preserve the model's useful knowledge: (1) knowledge measurement, (2) knowledge- preeeeegeeeriingggg mask search, and (3) knowleddg--ressevvnng
knowledge- preserving weight-tuning. Our iterative pruning different from previous atraining-based iterative pruning approaches (Frankle Carbin 2019 Han et al., 2015) since K-prune systemically controls the degree of pruning in each iteration. K-prune efficiently prunes the pretrained language models by an efficient weight-tuning technique which runs within a second requiring only a small sample dataset. As result, K-prune successfully overcomes the accuracy degradation problem and shows up to 58.02% II higher F1 score compared to the other retraining-free pruning algorithms as depicted in Figure We summarize our main contributions follows:
Algorithm. We propose K-prune, an accurate retraining-free free pruning algorithm for PLMs. K- K-prune consists of three novel ideas to preserve the useful knowledge of the pretrained mod- els: knowledge measurement, wnowledge-preserving mask search, and www.legeepreserving weight-tuning.
Accuracy. We perform extensive experiments on GLUE and SQuAD benchmarks to demon- strate the performance of K-prune. K-prune shows up to 58.02 88.02%p higher Fl score than the best results existing retraining-fres algorithms under a high compression rate 80%.
Efficiency. We demonstrate that K-prune shows the best accuracy-cost trade-off among the state-of-the-art pruning algorithms. K-prune shows comparable or higher accuracy than retraining- based algorithms on GLUE benchmarks with up to 422x lower pruning cost.
Our souree code is available at http://:://ggithuu.oooooo///eeee www.oo//ssumsssseommmmm
PRELIMINARIES
2.1 RICODER-BASEE PRETRAINED LANGUAGE MODEL (PLM) COMPRESSION
We define an encoder- based PLM compression problem as follows. We have an accurate PLM T finetuned for the target task, which predicts the label y for each instance and sample dataset
((x;, )1)) We assume that PLM T is too large and exceeds our FLOPS budget TFLOPs- Our goal is to compress the PLM T to tiny model S to satisfy our FLOPs budget TPLOPs while maintaining
its accuracy.
percent-point
2

---

Published as a conference paper at ICLR 2024
2.2 TRANSFORMER ARCHITECTURE
Transformer Encoder. In this paper. we focus compressing the encoder-based Transformers, such BERT (Devlin et al. 2019) and DistilBERT (Sanh al. 2019). The encoder-based Trans- formers consist of two types of sublayers: multi-head attention (MHA) and feedforward network (FFN) sublayers. For given input X E R of tokens each of which of dimension d. outputs of
sublayers are as follows: ((X + M(X)) for MHA sublayers or ((X + F(X)) for FFN sublayers where N refers to layer normalization (Ba et al. 2016). The output M(X) of multi-head attention with H attention heads is the sum of the outputs h2 () E Rdx+ of attention heads as in Equation (1) where B E Rdx is a bias. The output (x) of the ith attention head is decomposed into the output projection Wont Rd and the intermediate feature f(() Rd which are the outputs of a dot-product self-attention with dimension dy-
11
M(X) - Σ E):(()) + Bout where hx(X)- W fi(X)
The output F(X) of a feedforward network with N intermediate neurons is in Equation 2 where n,(X) E Rdxx is the partial output of the ith neuron and Cout Rdx* is bias. The output n, (X) of the ith neuron is computed by two linear transformations and is decomposed into the output projection voue gdxl and intermediate feature (X) R
F(X)=
Σ n. (x( + Coout where nx(X= = offut (x)
Pruning Criteria. In this paper, we aim to identify and prune unnecessary attention heads and neurons following previous works (Michel et al. 2019; Kwon et al. 20226). We introduce mask variables Ç - [S1, Sa) R and & [&1, ..., En]" E R to indicate the pruning status of attention heads and neurons, respectively; Çi = means the ith attention head is pruned. The masked outputs of M(X;c)
() and F(X;E) are described in Equation 3
H
M(X:c)= = Σ (m)((() + Bout and F(X:E)=
Σ (5) + C.out
All mask variables are initialized to 1, which preserves the original inference result. Once the mask variables are determined after mask search, pruning of attention heads and neurons whose mask variables are zero does not affect the inference results.
2.3 THE LOss OF KNOWLEDGE AFTER PRUNING
In existing works (Hinton et al.
2015 Romero et al. 2015 Mirzadeh et al. 2020 Son et al., 2021
Kim et al. 202 Jang al. 2023). large models employed to enhance the accuracy of smaller models by transferring their knowledge, and pretrained models are widely adopted for this purpose in the context of model compression (Sun et al.) 2019 Sanh al. 2019 Jiao al., 2020 Wang et al. 2020al Kim et al.] 2022 2023) Cho & Kang 2022). It is demonstrated that the knowledge of the pretrained models can be extracted from their soft label prediction and intermediate representations, and imitating them improves the generalization performance of the compressed model. For a given input æ, the amount K Kpred (z; m) of the lost predictive knowledge of the compressed model S out of the pretrained model T is defined in Equation (4) (Hinton et al.) 2015) Sun et al.) (019) Jiao et al. 2020) where m E [ is the pruning mask of the compressed model S with L layers. DKL KL-divergence, and 2 1 (m)) and ãs (x; m)) are logits of the pretrained and the compressed models, respectively. s., is a softmax function with the temperature of y. 1 1 |mll is vector of ones indicating unpruned status.
Kpred (r; m, 7) = "x (ss m))))
For the /th sublayer, the amount Kyep,1 of lost representational knowledge regarding intermediate representations is defined in Equation (5) (Romero et al. 2015 Sun et al. 2020; Tang et al.) 2019) where subscript of S and T represents the compressed model S and the pretrained model 7, respec- tively. X, is the input of the Ith sublayer and is added due the residual connection. Sub; is the
3

---

Published as a conference paper at ICLR 2024
Decrease the FLOPs budget (by the number of FLOPs of n2.3) and move on to the next sublayer
Knowledge Measurement
(b) knowledge-preservibn Mask Search (KPMS)
Knowledge-peeserving
Weight-tuning (KPWT)
R43
FFN
(1) Compule importance acores
(1) Prune the selected units only in target sublayer
predictive
fra,2
h1.1
know lodge
222
MHA
FFN
representational knowlodge
Imsorttncce score
Iarget
122,2
122,3
(2) Tune the weights of the survived
FFNs2 8,2
(2) Sort and select units
High
to recover knowledge
Baa
22.2 1i.1
has
7A4.2
FFN
MHA
selected
pruning units Achead
unit status
pred. knowlege
rep. knowledge
Importances score
n:neuron
normal
umed
2. promed
low
high
low
high
low
high
Figure 2: Illustration of K-prune when the second sublayer our target (best viewed in color). See Section 3.1 for details.
sublayer function of the Ith sublayer which either M(X) or F(X), and my a vector of mask variables in the Ith sublayer S.
( Xs. m.) = XT., Suby,"(T,,t mmi)) Xs. - Subs, (Xs, (5)
It is crucial to reduce the amounts Kpred and K Krep.t of the lost knowledge retain the accuracy of the pretrained model during compression.
3 PROPOSED METHOD
3.1 OVERVIEW
In this section, we propose K-prune, accurate retraining-free pruning algorithm which preserves the knowledge of PLMs through sublayer-wise iterative pruning process. Before describing our main ideas, we summarize several challenges that must be tackled.
C1. Importance critcrion. What aspects should we consider to find salient attention heads and neurons for preserving the knowledge of the PLM?
C2. Identifying uninformative components. How many attention heads and neurons should prune in each iteration, and how can select attention heads and neurons that minimize knowledge loss?
C3. Minimizing the loss of knowledge. Pruning induces the loss of knowledge the PLM. leading to severe accuracy degradation. How can we efficiently recover the lost knowledge of PLM after pruning?
We address these challenges with the following main ideas.
11. Knowledge measurement (Section 3.2 We gauge the amount of inherent knowledge regarding both label prediction and intermediate representations to estimate the saliency of masked units.
snowweegg-preeervinng preserving mask search (Section 3.3). In every iteration, we identify the I2. Knowledge meaningless masked units in the target sublayer considering their global importance which reflects both predictive and representational knowledge.
I3. Knowledge- ingless masked units only in the target sublayer and reconstruct the knowledge of the PLM through efficient weight-tuning.
www.wleeeeereeeriiiie preserving weight-tuning (Section 3.4 We remove the identified mean-
K- K-prune iteratively performs sublayer- wise pruning, from the bottom to the top sublayer, with the following three steps: (a) knowledge measurement, (b) knowledge-preserving preserving mask search, and (c) knowledge prngge-preervingg weight- tuning. We illustrate a pruning process for a two-layered Transformer Encoder with four sublayers when the second sublayer is target in Figure 2 In the first step,
4

---

Published as a conference paper at ICLR 2024
(a) measure the amount of inherent predictive and representational knowledge in each masked unit (attention head and neuron) in the target sublayer and sublayers above the target sublayer (e.g.. from the second to the fourth sublayers in Figure 2) The red and blue colors indicate the amounts of predictive and representational knowledge, respectively; darker colors denote richer knowledge. We measure the amount of knowledge in the above sublayers to consider the global importance of masked units in the target sublayer in step (b). We do not measure the knowledge of the sublayers (e.g., MHAs,1) 1) below the target sublayer since they have already been pruned. Then, (b) we compute the importance scores for each masked unit considering both predictive and representational knowledgr and sort them. We select the masked units with the least importance scores to be pruned considering the FLOPs budget. The number of selected masked units in the target sublayer is determined according to the global importance of the target sublayer since we evaluate and compare the importance scores of masked units in all of the unpruned sublayers. After that, (c) we prune the selected components (e.g., R2,1 and n2.2)) from the target sublayer and tune the weights of the remaining components (c.g.. small sample dataset recover the PLM's knowledge. We decrease
n223)). in the target sublayer the FLOPs constraint by the number of FLOPs of the remaining components and then move on to the next sublayer. K K-prune accurately compresses the model since it iteratively prunes small amount of masked units in each sublayer considering their global importance after reconstructing the ledge of the previous sublayers. The running time of K-prune is significantly low since performs only an efficient weight- tuning on small sample dataset. We elaborate on the details of each step in the following sections.
3.2 KNOWLEEDGE MEASUREMENT
We use the amount of both predictive and representational knowledge in each attention head and neu- ron as a metric to estimate their saliency in identifying uninformative attention heads and neurons. We measure the amount of knowledge contained within each attention head and neuron by evaluating the loss of knowledge after pruning it. For ith pruning mask my, in the /th sublayer, we reformulate the functions in Equations (4) and (5). which state the amount of knowledge loss, as single-variable func- tions by assuming that all mask variables are independent, i.e. ( m, = K (a;; mi,i, 7) and Krep. (X T. Xs.;; m.) x Krep.
Xs,i; mi,)), respectively. Then, the predictive and representational knowledge within an attention head neuron which corresponds to the mask my,i is estimated as (2; my, = 0, and Krep(XT.1 S mi, = 0), respectively.
We approximate the average of the amount K Kprea( (2;; my, 7) of predictive knowledge of my,i on the sample dataset as in Equation (6) by applying Taylor expansion and Fisher Information (LeCun) et al. 1989 Kwon et al. 2022b).
@K (repta(1;; m1, = 1,7) 2
Σ RDD
prealla; mi,1 = 0, ~
(6)
D
2y2
Ōm,,i
EDD
We estimate the amount K ith component in the /th sublayer, which corresponds the target mask m4,s- by the MSE loss between the outputs of the /th sublayers of the pretrained model T and the compressed model S as in Equation ( We introduce a mask vector my E Rlmal to indicate the pruning of the ith masked unit, and all elements of mn, are one except for the ith element which is zero. By assuming that the two inputs X and Xs. are the same, K (Xx., Xs.5 ma, = 0) becomes the norm of the output of the components as in Equation (8) since the masked outputs of sublayers are computed as the sum of unpruned attention heads or neurons as in Equation B..
Xs. my, 0) of representational knowledge within the
Krepa (XT,1, Xs.ú mi, m, =)) X SubT (XT,,, 1mm) - (Xs,,, (x),, (7)
[X,,((ss for MHA sublayers
(8)
(X,,)(Xs.) for FFN sublayers
3.3 BHOWLEEGGE-RESSEVVINN MASK SEARCH (KPMS)
We propose snowledge-preservvng mask search (KPMS), an accurate mask search algorithm which finds an accurate non-uniform pruning mask for each sublayer. In KPMS, we estimate the importance of each masked unit using the amount of knowledge in the masked unit to minimize the knowledge loss after pruning. We estimate the importance score not only in the target sublayer but also in the sublayers above the target sublayer control the number of masked units prune in the target
5

---

Published as a conference paper at ICLR 2024
Algorithm 1 www.lledgee-rserving. Mask Search (KPMS)
Input: Sample dataset D1, pretrained model T, compressed model S. FLOPs F', and Fn of a head and a neuron, FLOPs budget TFLOPs temperature y for Equation (4) and balance coefficients À and μ for Equation (9) Output: the sets Phead and P neuron of attention heads and neurons to be pruned
1: Khoad Kheade Koored
head? neuron" neuron + measure- -nw.uurre-.oolleee(..... (, &. Equations 66.
2: Zhead, Zneuron scoring (K head' Kheat Kered K neuron' & H, A, Fh, Fn)
head'
neuron
Equation 9
3: Z <
order( Zheed:Znearor Zeewonn)
4: p < 0, f --heaal Fn + | neuurn!! Fn
5: while f /> -T..PPs do
6:
1/ + -Z[p]
candidate threshold b remained heads and neurons
7:
< Shead 2 v)), nn t [M|SSeeuron()) 2 pt-p- I. nA Fi + Fn
8:
FLOPs of the compressed model
9: end while
10: V² --
11: Phead < Zhead [h] < <v*), neuron {nl Z meuron [n] v*}
selected to be pruned
sublayer, considering their global importance. KPMS is described in Algorithm [ We begin KPMS by measuring the amount of knowledge in attention heads and neurons estimate their importance score (line 1). We evaluate the amount of both predictive and representational knowledge in attention heads and neurons on the sample dataset following Equations (6) and (8) Khead K head E RLLH and Krrred Reuran: Keeeoon E I.N in Algorithm | represent the vectors of the estimated amount of knowledge in all attention heads and neurons in the model, respectively, where L is the number of layers, i.e. there are I MHA sublayers and I FFN sublayers. In detail, we set the amount of knowledge as for the attention heads and neurons in the sublayers below the target sublayer. which was pruned in previous steps, in order to ignore them during the mask search. Then, we estimate the importance score of each attention head and neuron as the weighted sum of the amount of the predictive and representational knowledge with balance coefficient À as in Equations (9) (line 2). divide the scores of attention heads and neurons by their number of FLOPs (F,, for attention heads and Fn, for neurons) in order to consider the amount of importance score per FLOP. We multiply the scores Zhead of attention heads by another balance coefficient μ to reflect the different sensitivity between attention heads and neurons.
Zhead=
K
pred head
+ XKTeR head) /Fh and Znearon
pred (K
AKrepp neuron) /Fx
(9)
neuron
We concatenate two vectors Zneuron and Zieead and then sort the concatenated vector in increasing order to find the threshold for pruning (line 3). We sequentially obtain threshold candidates v from the sorted score vector Z until the FLOPs f of the compressed model pruned by the threshold is smaller than our FLOPs budget TEloss (lines 4-9). Consequently, get the optimal threshold v*, and find the sets P Pread and IP, Pneeurnn containing the indices of heads and neurons whose importance score is lower than v²" respectively (lines 10-11).
3.4 KNOWL.EDGF. WEIGHT- (KPWT)
We propose pereeeevignr weight-tuning (KPWT), an efficient weight-tuning process that reconstructs the distorted knowledge the PLM after pruning. In every sublayer-wise iteration of K-prune, we prune only masked units in the target sublayer to formulate the problem of knowledge reconstruction as a problem which requires an extremely short time to solve. When the /th sublayer target, prune masked units in the (th sublayer if they are included in PPeea or IP Pneuro of KPMS. Then, we formulate the knowledge reconstructing problem as the problem of minimizing the loss K Kegp.1
(irp.((7: (XT,i, Xs.1, m:) of representational knowledge of the Ith sublayer in Equation (5). Equation (10) is the reformulated problem of Equation (5) for MHA sublayers where Cī, represents
the found mask of the ith attention head in the Ith sublayer in Algorithm I i.e. the value of mask Gī, is 0 if the index (IH i) of its corresponding attention head is in IP IPhead or 1 otherwise. We modify the problem as the linear least square problem over the set of output projections { (Wort) ) to exploit the efficiency of the linear solver. We collect the sublayer outputs (X,,1 + M7.((X...) 1)) of the pretrained model, which does not change during a pruning process, at the first iteration of

---

Published as a conference paper at ICLR 2024
K-prune and reuse them for every iteration. We collect the set (f..(X5.))< Rim of features when measure the knowledge in KPMS (line in Algorithm (T) Analogously, we formulate the problem for FFN sublayers in Equation (11) where Eī, represents the found mask of the ith neuron of the /th sublayer. The subscript ( in symbol represents that the symbol is related to the /th sublayer. We
tune weights (Wirr or v1,5 to achieve high accuracy even at high compression rates.
H
arg min (Wost)
X MT,(XT, 1x) Xs.1
Σ
Si Wex (ts((Xs.))
B
(10)
arg min (ef Fimz
XT.+Fri(X IN) Xs, o M1. (Xs.)
out
Opue
(11)
We use linear solve in PyTorch (Paszke et al. 2019) to solve Equations 10 and 11.. Note
that the time for solving the problems in Equations (IO) and (11) is shorter than a second a
typical desktop computer, which is several magnitudes smaller than those of conventional retraining processes in existing works (Xia et al.] 2022 Hou et al. 2020; Lagunas et al. [02 Liu et al. 2021
and does not require any hyperparameter tuning. After pruning and knowledge reconstruction, we decrease our FLOPs constraint by the FLOPs of the remaining attention heads or neurons in the Ith sublayer. Then, we move on the (I + 1)th sublayer with the adjusted FLOPs constraint. This enables K-prune to satisfy the FLOPs constraint through a single run without any interventions from users.
EXPERIMENTS
We perform experiments to answer the following questions about K-prune:
Q1. A eccrrcy (Section 4.2 How accurate are the models compressed with K-prune compared to the models compressed with existing retraining-fres pruning
a algorithms?
Q2. Inference speed (Section 4.3). How fast the models compressed with K-prune com-
pared to the models compressed with existing retraining-fres pruning algorithms?
Q3. Efficiency (Section 4.4). How efficient is K-prune compared the existing pruning algo-
rithms including retraining- based ones in terms of both accuracy and pruning cost?
Q4. Ablation study (Section 4.5 Do ideas of K-prune, i.e. knowledge-based importance criteria, KPMS, and KPWT, improve the accuracy of the compressed models?
4.1
EXPERIMENTAL. SETUP
Sctup. We Py Torch (Pazkke et al. 2019), and the weights of the pretrained models in Transform- ers (Wolf et al. 2020). We evaluate the performance of compressing the pretrained BERT (Devlin al. 2019) and DRRT (Sanh al., 2019) models on GLUE (Wang et al., 2019). SQuAD vI.1 Rajpurkar er al. 2016), and v2 (Rajpurkar et al. 2018) under diverse compression rates. We use FLOPs as a compression measure which is computed on the average sequence length of each dataset. We report the compression rate as ratio of the removed number of FLOPs after pruning. We use NVIDIA 1080 for all experiments
slyperparametes. We use 100K tokens from the training dataset as a sample dataset, and exploit the pretrained tokenizers in Transformers (Wolf et al. (2200) for counting. The size of the sample dataset is small compared to the GLUE and SQuAD datasets, e.g. around 0.64% of MNLI (Williams) et al. 2018) dataset. We fix random seeds from 0 to and report the average performance of the runs. We use two combinations of hyperparameters (7, A, n1) E {(2, 0, 64), (2, 0.00025, 64)) for all experiments of K-prune.
We compare the performance of K-prune with existing retraining-free pruning algorithms for PLMs: Kwon et al. 2022b) and KCM (Nova et al. (023). We compare the pruning efficiency with state-of-the art retraining- based pruning algorithms for PLMs, DynaBERT (Hou et al. 2020> and EBERT (Liu et al. (02T) which show the best tradeoff in terms of accuracy vs. pruning cost, outperforming FLOP (Wang et al. 2020b), (Sajjad al. (2023). Colii (Xia al. (022), and BMP (Lagunas et al. 2021) as reported in Kwon et al. (2022b). We use entire datasets for training retraining- brased algorithms.
Competitors.
listg..sstq
7

---

Published as a conference paper at ICLR 2024
Table I: Comparison of inference speed of the models compressed by K-prune and competitors. We report the best result of the compressed models whose accuracy degradation is lower than 3%p. K- K-prune shows the highest acceleration, giving up 2.93 faster speed than the uncompressed model.
Method
MRPC
STS-B
SQuAD1.1
SQuAD2.0
Avg."
KCM (Nova al. 2023
1.08x
1.23x
1.20x
1.08 x
1.15x
Kwon el al. 20022b)
1.59x
2.10x
2.09
1.75x
1.87
K- K-prune (ours)
2.66x
2.43x
2.60x
2.93
2.65
* Geometric mean
* K-Prima (our))
Kwen atal
Table 2: Evaluation of K-prune and its variants un- der a compression rate of 80%. Each of the pro- posed ideas successfully improves the accuracy of the compressed models, and K-prune shows the best results. We get the largest accuracy improvement from KPWT.
+ EBERT
DymBERT-=
D DyuBERT-4
BEST
MNLI
422
BEST
QNLI 213x
Accuracy (%)
G3 %
90
+
Accuracy
20
80
8
60
10
10"
10
10"
10
10"
10'
10
Pruning time (h)
Pruming time
Method
MRPC
SQuAD*
K-prune
84.80
74.16
Figure 3: Accuracy of compressed models
K K-prune- Kpred, Krep
84.07
72.55
vs. time cost for pruning under compression rate of 75%. K-prune (blue star) shows the best trade-off among both retraining- frre and retraining- based pruning algorithms.
K K-prune- KPMS
81.71
67.10
K K-prune- KPWT
68.38
16.50
SQuAD 1.1
4.2 ACCURACY OF THE COMPRESSED MODELS (Q1)
Issows a comparison of the accuracy vs. reduced FLOPs of the compressed models generated by K-prune and competitors diverse tasks and models. The black dotted line indicates the 3%pp accuracy degradation from the baseline models. In all settings, K-prune outperforms all competitors in large gaps by up 58%p. The accuracy gap between K-prune and the competitors grows larger as the compression ratio gets higher since their one-shot pruning process fails to cope with the pruning errors; especially, KCM shows drastic accuracy degradation as the ratio of reduced FLOPs increases since it cannot prune attention heads. Our results demonstrate that K K-prune effectively addresses the significant accuracy degradation problem by preserving the knowledge PLM via thoughtfully designed iterative pruning process incorporating novel ideas: KPMS and KPWT.
4.3 ACCELERATION ON COMMODITY HARDWARE (Q2)
We compare the inference speed of the compressed models whose accuracy drop is lower than 3%p compared to the baseline model. We use randomly generated input sequences whose length equal to the average length of input sequences in each task. We We summarize the highest acceleration ratio of K-prune and competitors compared to the baseline model in Ta K-prune consistently shows the highest acceleration compared to existing methods on all tasks. K-prune achieves up to 2.93 faster inference speed compared to the baseline model on commodity hardware, while other methods achieve at most 2010x faster inference speed.
batch size of 32 for all experiments.
4.4 COMPARISON WITH RETRAINING-BASEE PRUNING ALGORITIMMS (Q3)
In Sections 4.2 and 4.3 we demonstrate that K-prune outperforms existing retraining-fre free algorithms with large margins in terms of both accuracy and inference speed. In this section, we compare K K-prune with both retraining-free and retraining- K-prune. We compare the cost of each pruning algorithm by measuring the time for pruning in hours, and the accuracy of the compressed models for 75% compression rate on MNLI and QNLI datasets in Figure 3 DynaBERT-d and DynaBERT-w are two variants of DRT with and without applying depth multipliers, respectively. Note that K X-prune shows comparable or better accuracy in all settings compared to Kwon al., EBERT, DynaBERT-w, and DynaBERT-d while showing up 422 x lower pruning cost. Thus, K K-prune shows the best trade-off regarding the accuracy and pruning time among both the retraining-based based and retraining-free pruning algorithms.
pruning algorithms to show the efficiency of
8

---

Published as a conference paper at ICLR 2024
4.5 ABLATION STUDY (Q4)
We perform an ablation study to show that each technique of K-prune, such as knowledge-based importance criteria, KPMS, and KPWT, improves the accuracy of the compressed model. We summarize the results in Table under the compression rate of 80% on MRPC and SQuAD Each row of Table depicts the change of performance when an individual idea is omitted from K-prune. -predd Keep shows the results from using the magnitude of the derivative of cross entropy instead of the knowledge-based importance criterion, -KPMS denotes cases where pruning performed uniformly across sub-layers without considering global importance, and -PPWT represents that iterative pruning and weight-tuning are not employed. Our results show that all ideas contribute to the performance enhancement, and KPWT shows the most significant impact.
RELATED WORKS
5.1
TRANSFORMER COMPRESSION
Transformer compression algorithms are designed to reduce the size and inference time of Transformer.
These algorithms are categorized based on the aspects they focus on: quantization (Kim et al. 2021b
Piao et al., [022] K Kwon et al.] (022a), low-rank approximation (Wang et al., [022]] Cordonnier et al.
2020 parameter sharing (Lan et al. 2020; Jeon et al., 2023), structured pruning (Hou et al. 2020
Liu et al.
2021 Kwon et al. 2022b; Nova al.) 2023), and unstructured pruning (Sanh et al. 2020 Yao et al.
2021). In this paper, we focus on structured pruning which guarantees instant acceleration
on commodity hardware. Note that other types of algorithms are complementary structured pruning in achieving a higher compression rate, as they address different kinds of inefficiencies (Lazarevich
al. 2021
Frantar & Alistarr 2023).
5.2 STRUCTURED PRUNING FOR TRANSFORMERS
Structured pruning algorithms for Transformers are divided into two groups: retraining- -based and retraining-fie ones. Earlier approaches for structured pruning (Hou et al. 2020: Liu et al. 102 Lagunas et al. 2021 Xia et al. 2022) are retraining- algorithms which generate highly sparse and accurate models based on their sophisticated training using entire datasets. However, these algorithms demand extensive retraining costs and intensive hyperparameter tuning, limiting their usage; for large language models (Brown et al. (2020] Zhang et al. 2022). retraining-
algorithms intractable. For example, DynaBERT (Hou et al.] (020)) requires three individual retraining processes for pruning BERT. Retraining- free algorithms (Kwon et al. 2022b Nova et al., (023) are proposed to reduce the expensive pruning cost by removing retraining processes. However, they face significant accuracy drop since they fail to cope with pruning errors. Our proposed K-prune resolves the accuracy degradation problem, achieving both speed and accuracy.
CONCLUSION
We propose K-prune, an accurate retraining-free structured pruning algorithm for encoder-based PLMs. We address the problem of severe accuracy degradation in prior retraining-free pruning algorithms by carefully designing an iterative pruming algorithm to preserve the knowledge of PLMs. K-prune achieves remarkable accuracy improvement up to 58.02%p better performance than existing retraining-fres pruning algorithms. Future works include extending our method for decoder-hased models.
wcknowledggeet This work was supported by Youlchon Foundation. This work was also supported by Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MISTT) Compression Method for Various Applications and Enviromments), [No.202 2020-000223000 Artificial Intelligence Graduate School Program (Seoul National University)), and [NO.2021 202-0-00-000,,, Artificial Intelligence Innovation Hub (Artificial Intelligence Institute, Seoul National University)). The Institute of Engineering Research at Seoul National University provided research facilities for this work. The ICT at Scoul National University provides research facilities for this study. U Kang is the corresponding author.
Flexible and Efficient Model

---

Published as a conference paper at ICLR 2024
REFERENCES
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, 06450
abs/1607.064500 2016.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell,
Balcan, and Hsuan-Tiee Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Ikhyun Cho and U Kang. Pea-kd: Purameter-efficient and accurate knowledge distillation bert. Plos one, 172))e0263592, 2022.
Kevin Clark, Minh- Luong, Quoe V. Le, and Christopher D. Manning. ELECTRA: pre-training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. Multi-head attention: Collaborate instead of concatenate. CoRR, abs/2006. 16362, 2020. URL
2006. 1 16362
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 4171-4186. Association for Computational Linguistics, 2019, doi: 10. 10.18653/v1/n1911433
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 10323-10337. PMLR, 2023.
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural network. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 1135-1143, 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1003 2015.
Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic BERT with adaptive width and depth. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Jun-Gi Gi Jang, Chun Quan, Hyun Dong Lee, and U Kang. Falcon: lightweight and accurate convolution based on depthwise separable convolution. Knowl. Inf. Syst. (205))2222-22249 2023. doi: 1007/ S101 15-022-01888---11
httt.httss://ee org 1
022 01 11188-8
10

---

Published as a conference paper at ICLR 2024
Hyojin Jeon, Seungcheol Park, Jin-Gee Kim, and U. Kang. Pet: Parameter- knowledge distillation on transformer. PLOS ONE, 18(7):1-21, 07 2023.
doi: 10.137 1jouma/jouall.om 002800000000.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling BERT for natural language understanding. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL. pp. 4163-4174. Association for Computational Linguistics, 2020.
Junghun Kim, Jinhong Jung, and U. Kang. Compressing deep graph convolution network with multi- staged knowledge distillation. PLOS ONE, 16, 08 2021a. doi: 10.1371/journa 1ttp:rmaa.......26o11011 0256187.
Minsoo Kim, Sihwa Lee, Sukjin Hong. Du-Seong Chang. and Jungwook Choi. Understanding and improving knowledge distillation for quantization aware training of large transformer encoders, In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.). Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 6713-6725. Association for Computational Linguistics, 2022.
Minsoo Kim, Sihwa Lee, Janghwan Lee, Sukjin Hong. Du-Seong Chang, Wonyong Sung, and Jungwook Choi. Token-scaled logit distillation for ternary weight generative language models. Advances in Neural Information Processing Systems, 36, 2023.
Schoon Kim, Amir Gholami, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. I-BERT: integer-only BERT quantization. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 5506-5518 PMLR, 2021b.
Woojeong Kim, Suhyun Kim, Mincheol Park, and Geunseok Jeon. Neuron merging: Compensating for pruned neurons. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Se Jung Kwon, Jeonghoon Kim, Jeongin Bae, Kang Min Yoo, Jin-Hwa Kim, Baeseong Park, Byeong- wook Kim, Jung-War Woo Ha, Nako Sung, and Dongsoo Lee. Alphatuning: Quantization-awe parameter-efficient adaptation of large-scale pre-trained language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguis- tics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 3288-3305. Association for Computational Linguistics, 2022a.
Woosuk Kwon, Schoon Kim, Michael W. Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. A fast post-training pruning framework for transformers. NeurIPS, 2022b.
François Lagunas, Blla Charlaix, Victor Sanh, and Alexander M. Rush. Block pruning for faster transformers. In Maric-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 10619- 10629. Association for Computational Linguistics, 2021. doi: 10. 2t.18653/v1/2221 2ttp:////222...mlppmmmia. .ww.lpmmann822
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representationss ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL Bttttttts///peeeeeeeee
www.///////u/mmm -ieeiidd-llle 7AEtvS
Ivan Lazarevich, Alexander Kozlov, and Nikita Malinin. Post-training deep neural network pruning via layer-wise calibration. In IEEE/CVF International Conference on Computer Vision Workshops, ICCVW 2021, Montreal, BC, Canada, October 11-17, 2021, pp. 798 805. IEEE, 2021. doi: 10. D0.1009JCCVW542222222000000
URL https: /tt. wtt.a//ggg...11///cccsc......
2021. 00094]
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In David S. Touretzky (ed.), Advances in Neural Information Processing Systems 2, INIPS Conference, Denver, Colorado, USA, November 27-30, 1989), pp. -98-605. Morgan Kaufmann, 1989.
11

---

Published as a conference paper at ICLR 2024
Zi Lin, Jeremiah Liu, Zi Yang, Nan Hua, and Dan Roth. Pruning redundant mappings in transformer models via opectral-normalize identity prior. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 719-730, Online, November 2020. Association for Computational Linguisties.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeutlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.1 1692, 2019.
Zejian Liu, Fanrong Li, Gang Li, and Jian Cheng. EBERT: efficient BERT inference with dynamic structured pruning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics; ACI/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/UCCLP 2021 of Findings of ACL, pp. 4814-4823. Association for Computational Linguisties, 2021. doi: 10.18653/v1/2021 aaddings-acl.425,
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, 2023.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In ICLR 2017, 2017.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 14014-14024, 2019.
Seyed-Iman Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In The Thirty- Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium Educational Advances in Artificial Intelligece, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 5191-5198. AAAI Press, 2020.
Azade Nova, Hanhun Dai, and Dale Schuurmans. Gradient-free structured pruning with unlabeled data. CoRR, abs/2303.04185, 2023. doi: 10.4885500anxxi.2333...o
Seungcheol Park, Jachyeon Choi, Sojin Lee, and U Kang. A comprehensive survey of compression algorithms for language models. CoRR,
2024.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performaffree deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8024-8035, 2019. URL htttss: Abstract bdbca288fees (tt.://feeee722222aaaaa21112.21. -ts. html
newrips cc/ppaprr/219//haae
Tairen Piao, Ikhyun Cho, and U Kang. Sensimix: Sensitivity-wwarc 8-bit index & 1-bit value mixed precision quantization for bert compression. Plos one, 17(4)ee2266221, 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21 (1(1): -5555 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics.
12

---

Published as a conference paper at ICLR 2024
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Papers), pp. 784-789, Melbourne, Australia, July 2018. Association for Computational Linguistics.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and
Yoshua Bengio. Fitnets: Hints for thin deep nets. In Yoshua Bengio and Yann LeCun (eds.), 3rd
International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. BtLphtt::///e //axivv.///x org/ abs / 1412. 6550
Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. On the effect of dropping layers of pre-trained transformer models. Comput, Speech Lang. 77(C), jan 2023. ISSN 0885-2308.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1900001008, 2019.
Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement pruning: Adaptive sparsity by fine-tuning. In Hugo Larochelle, Marc' Aurelio Ranzato. Raia Hadsell, Hsuan-Tien Lin (e Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Balcan, and
Wonchul Son, Jaemin Na, Junyong Choi, and Wonjun Hwang. Densely guided knowledge distillation using multiple teacher assistants. In 2021 IEEF/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 9375 9384. IEEE, 2021. doi: 10. 10.1000CCCV49992.00000000 2021 00926
URL https: /doi org 10. 1 110 199 ICCV48922
Suraj Srinivas and R. Venkatesh Babu. Data-free parameter pruning for deep neural networks. In Xianghua Xie, Mark W. Jones, and Gary K. L. Tam (eds.), Proceedings of the British Machine Vision Conference 2015, BMVC 2015, Swansea, UK, September 7-10, 2015, pp. 31.1-31.12. BMVA Press, 2015.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for BERT model compression. In Kentaro Inui, Jing Jiang. Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 4322-4331. Association for Computational Linguistics, 2019.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: compact task-agnostic BERT for resource- lesttrec-limited devices. In Dan Jurafsky, Joyce Chai, Natalic Schluter, and Joel R. Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 2158-2170. Association for Computational Linguistics, 2020.
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task-specific knowledge from BERT into simple neural networks. CoRR, abs/1903 12136, 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL. httpps://openreeeeeeee ew net/ forumm d=
rJ rJ4km2R5t7
Benyou Wang, Yuxin Ren, Lifeng Shang, Xin Jiang, and Qun Liu. Exploring extreme parameter compression for pre-trained language models. The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for fask-agnostic compression of pre-trained transformers. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Essun-TTien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020a.
13

---

Published as a conference paper at ICLR 2024
Ziheng Wang. Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, Association for Computational Linguistics, 2020b.
2020, pp. 6151-6162
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, NAACI-HI.T 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume / (Long Papers), pp- 1112-1122. Association for Computational Linguistics, 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-ar natural language processing. In Qun Liu and David Schlangen (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2020- Demos, Online, November 16-20, 2020, pp. 38 45, Association for Computational Linguistics, 2020.
Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 1513 1528. Association for Computational Linguisties, 2022. doi: 12663//1120022.allloooo 107.
Zhewei Yao, Xiaoxia Wu, Linjian Ma. Sheng Shen, Kurt Keutzer, Michael W Mahoney, and Yuxiong He. Leap: Learnable pruning for transformer-bast models. arXiv preprint X1v:22005..1636. 2021.
Edouard YVINEC, Arnaud Dapogny, Matthieu Cord, and Kevin Bailly, Red Looking for redundancies for data-free structured compression of deep neural networks. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 20863-20888 Curran Asso- ciates, Inc., 2021. URL URLhhttpss//prroceddnsss neurips wttplps...ccppppp ccppaper__.p.ee.peeeeeeeeee... fi leel/epprrr 2021 22207///111//eeeeeeee00000aaaaa2 Papp
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov. Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang. and Luke Zettlemoyer. OPT: open pre-trained transformer language models. CoRR, abs/2205. .0000000006..... 2022.
14

---

Published as a conference paper at ICLR 2024
A SYMBOLS AND DEFINITIONS
We summarize the definitions the symbols in Table For simplicity, we omit the notation representing the /th sub-layer if omiting does not introduce any confusion.
Table 3: Symbols and descriptions.
Symbol
Description
T.S Sub(.)
pre-trained and compressed models
sub-layer function
M(-) F(-)
sub-layer functions for MHA and FFN sub-layers an attention head and neuron an intermediate layer intermediate features of an attention hcad and neuron output projections for attention head and neuron biases for output projections in MHA and FFN sub-layers masks for an attention head and a neuron a length d vector filled with ones
h(), ml
fO,, 9(-)
WoUC your
Bad C
S
la
map
mask vector filled with ones except the ith element which is zero
H
the number of attention heads in an MHA sub-layer
N
the number of neurons in FFN sub-layer
d
the dimension token embeddingg a sequence length the dimension of projected embeddings in attention heads
8
d,
TD)
a sample dataset
(xr y) X
a tuple of data point and its label in D input of sub-layer
K preeds Keep Zieend Zoeuron
predictive and representational knowledge importance scores of attention heads and neurons the temperature of softmax functions a coefficient for balancing Kpred and Krep a coefficient for balancing Shead and Snearon
X
TELOPs FLOPs())
a FLOPs constraint
function for measuring FLOPs of the model the number of FLOPs for computing the output of attention head the number FLOPs for computing the output of neuron
Fh
Fr
15

---

Published as a conference paper at ICLR 2024
B DERIVATIONS
We provide additional derivations for completeness.
B. DERIVATION OF EQUATION 8 IN SECTION 3.2
For MHA sublayers, we derive K (Xp T. X s.. my, = 0) ~ ( as follows under the
same assumption. We assume that XT, Xs, since we reconstruct the output of the previous sublayer.
Krep. (XT, Xs,6m 0) = | SubT, Im))
Xs.i Subs,,(Xss m
H
Σ h (Xs,i) + Br
Σ
h (Xs.)) Bi 14 h1,
= ||,,,
Analogously, we derive Krep, (X,,s X s =0) (Xs,)( for FFN sublayers as follows.
Krep. Xs,im,, = 0) = X Suby, (X,,, 1m;) - Xs. - -Suu, mp,))
N
Σ
(Xs()) + Cput
Σ j=1
m.,((sss) + +Crut n1, (x..(ss.)
j=1
(Xs:)ll
B.2 REFORMULATED PROBLEM OF KNOWELDGE RECONSTRUCTION FOR LINEAR SOLVERS (SECTION 4.4)
We reformulate Equations 10 and 11 in main text form of linear least square problem
use linear solvers, such as tina. Istsqq as in Equation (22
W = arg min PW W
Q
(12)
We derive P. W, and Q for MHA sub-layers as in Equation (13) where || columnwise concatena- tion. P is transpose of concatenated feature matrix of the remained attention heads after pruning and W is concatenation of transposed weight matrices of the output projections in the remained attention heads after pruning.
P (INE())))))) s((Xs)) T
W =1lee(+() (Wout)7 T
(13)
(X- + My Xx,, 11) Xs Bout
We derive P. W, and Q for FFN sub-layers of Equation sub-layers.
14 in the same logic as the MHA
P (1lettie) di (Xx))) T
W = =11e((1 (Doat)" T
(14)
Q (X, FT FT 1N) - Xs - C
16

---

Published as a conference paper at ICLR 2024
C DETAILED EXPERIMENTAL SETTINGS
C.I DATA DESCRIPTION
We summarize the characteristics of GLUE and SQuAD benchmarks in Table
Table 4: Summarization of benchmark datasets.
Name
Samples
Tokens
Task
Metric
MRPC
3.7k
195k
paraphrase
accuracy
QQP
364k
11,123k
paraphrase
accuracy
SST-2
67k
897k
sentiment
accuracy
STS-B
7k
160k
sentence similarity
Spearman cor..
MNLI
393k
15,629k
NLI'
accuracy
QNLI
105k
5,176k
QA NLI
ONI
accuracy
SQuAD
88k
15,116k
QA
Fl score
SQuAD2.0
132k
22,455k
QA
Fl score
natural language inference
** question answering
C.2 FINE-TUNING OF PLMs
We fine-tune BERT (Devlin al. 2019) following standard training recipe. We fine-tuned checkpoints of DistilBERT in the github" We summarize the performance of fine-tuned BERT and
DistilBERT in Table
Table 5: Accuracy of the fine-tuned BERT and DistilBERT.
MRPC
QQP
SST-2
STS-B
MNLI
QNLI
SQuAD11
SQuAD2.u
BERT
87.01
91.54
93.12
89.08 86.12
84.90
91.87 88.55
88.51 85.73
76.54
DistilBERT
84.80
89.99
91.39
82.10
68.84
C.3 TRAINING DETAILS OF K-PRUNE
Code. We attach our implementation of K-prune in the supplementary material. We attach scripts and detailed instructions for reproducing our experimental results.
Hyperparameter. In addition to the hyperparameter settings {(2, 1, 64), (2, 0.00025, 64)) used in the main text, provide additional results with wider range of hyperparameter settings. We perform experiments on SQuAD,, under compression rates of 40%, 60%, and 80%.
Sensitivity analysis regarding y.. Figure shows the change of the F1 score of the model with regard to the change of the temperature y for softmax functions. We use E {0.5, 1.0, 1.5, 4.0) where a higher 7 represents smoother prediction after softmax. The F1 score of the compressed model is weakly sensitive to the change of ς. We get an accurate compressed model with y 2 which is used for comparison with existing works the main text, and we get additional accuracy improvement when we use 1.5.
Scnsitivity analysis regarding A. Figure shows the change the Fl score of the model with regard to the change of the balance coefficient À for representational knowledge. We use E {0.25, 0.025, ..., 0.0000025) where higher imposes higher importance on representational knowledge than predictive knowledge. We additionally depict the results of two cases that use only predictive or representational knowledge with the leftmost and rightmost stars in each figure. Overall, predictive knowledge plays important role and shows higher fll scores than representational knowledge. However, when it comes to the high compression rate, i.c. 80%, we find that using representational knowledge improves the performance of the compressed model compared to the case
pruning
17

---

Published as a conference paper at ICLR 2024
used for comparison in the main text
others
SQuAD 1.1 (40%)
SQuAD 1.1 (60%)
SQuAD 1.1 (80%)
90
90
80
1r 2r
(1)
85
85
1
12
75
70
0
2 V
3
3
4
2
3
4
Figure 4: Change of fl scores with regard to the change of the temperature Y on SQuAD j.j under compression rates of 40%, 60%, and 80%. The fl scores of the compressed model exhibit weak sensitivity to the alteration in y.
used for comparison in the main text
X others
SQuAD 1.1 (40%)
SQuAD 1.1 (60%)
SQuAD 1.1 (80%)
90
90
80
1
75
x
(%)
85
Red
Kigg only
85
1
70
only
Kyred only
65
Ryret only
Kigp
60
K only
only
55
2.50-6 2.5c-4 2.5c-2
2.5e-6
2.5c-4
2.50-2
1
2.50-6
2.5c-4
2.5c-2
À
Figure 5: Change of fl scores with regard the change of the balance coefficient À on SQuAD1.1 under compression rates of 40% 60%, and 80%. The leftmost and rightmost stars represent the cases that use only predictive or representational knowledge, respectively. Representational knowledge not effective by itself in general, however, improves the accuracy of the compressed model when combined with predictive knowledge.
in which we use only predictive knowledge. We get an accurate model with À E f0, 0.00025) which is used for comparison with existing works in the main text. We get additional accuracy improvement when we use X 0.0025 at the compression rate of 80%.
Sensitivity analysis regarding μs. Figure 6 shows the change of the F1 score of the model with regard to the change of the balance coefficient μ for scores of attention heads. We use μ E (1,2, 4, 8, ..., 2048} where higher imposes higher importance heads than neurons, and encourages the pruning of neurons. As a result, we find that μ E [32, 128) consistently shows accurate results all compression rates, and too-low or too-high value of μ shows severe accuracy degradation. We conjecture that this accuracy degradation comes from the imbalance of pruning of attention heads and neurons. We recommend using μ 64 which consistently shows accurate results.
the scores of the attention
C.4 TRAINING DETAILS OF COMPETITORS
We summarize the training details of competitors.
C.4.1 KOON ET AL.
2002B
Code. We use the code implemented by authors in githut
Hyperparameters We use damp = 1 for LSMR solver in CuPy and acceptable range of tuned varialbes as [010, 10] following the original paper (Kwon et al. 2022b
"https//github.com/ http:://ggghubccoo/oowwuusunkn WwwswWWWwwwkko wwtpw/////wwtraiiinn-eeee-eunnne huapyys.cipy.spase..iinn.
18

---

Published as a conference paper at ICLR 2024
used for comparison in the main text
others
SQuAD 1.1 (40%)
SQuAD 1.1 (60%)
SQuAD 1.1 (80%)
90
90
90
75
60
12
60
60
45
45
45
17
15
IS
15
4 16
64 256 1024
4
16
64 256
1024
16 64
256 1024
Figure 6: Change fl scores with regard to the change of the balance coefficient μ on SQuAD 1.I under compression rates of 40%, 60%, and 80%. The results with μr E [32, 128] are accurate in all settings, and too-low or too-high value of shows severe performance degradation.
C.4.2 KCM NOVA ET AL, 2023
Code. We reimplement the KCM since there is public implementation of authors.
Hyperparamcters We use width 0 = the Gaussian kernel and convergence rate a = 0.01 as
in the original paper (Nova al. 2023 We use Min-Max normalization for normalizing D2 scores.
C.4.3 DYNABERT (HOU AL. 2020
Code. We use the code implemented by authors in githut
Hyperparameters We use the same hyperparameters summarized in Table of the paper (Hou et al. 2020). We use (mw, ma) (0.25, 1.0) for DynaBERT-w and (mz, ma) = (0.5, 0.5) for DynaBERT-d where mw and my are width and depth multipliers, respectively. We do not use data augmentation for fairness since other algorithms do not use data augmentation. We report the accuracy after finalfinetuning.
C.4.4 EBERT LIU ET AL.
2021
Code. We use the code implemented by authors in githut
Hyperparameters We use the same set of hyperparameters introduced in Section 4.1 of the paper (Liu et al. 2021).
D RETRAINING-FRER MODEL COMPRESSION IN CNN
There are retraining-fiee free structured pruning algorithms YVINEC et al. 2021 Kim et al. 2020
Srinivas & Babu 2015) for CNNs which reduce the size of pre-trained models by finding similar neurons based on their weight distribution, and integrating the similar neurons. However, we do not compare them with K-prune since they are not directly applicable to the PLM compression problem. The main reason is the architectural difference between CNN and Transformer. The structured pruning algorithms for CNN do not consider pruning attention heads, and thus they can prune only FFN sub-layers like KCM (Nova et al. 2023) which shows severe accuracy degradation in Figure of our main text.
E EXPERIMENTS ON LARGE LANGUAGE MODELS
We provide an experimental result on decoder- based large language models (LLMs) considering the growing interest in reducing the cost of LLMs via compression (Park et al. 2024). We prune OPT-1.3B and OPT-2.7B models (Zhang et al. 2022 using 128 sentences in C4 dataset Raffel et al.
19

---

Published as a conference paper at ICLR 2024
(2020) and measure the perplexity Wiki-text2 (Merity et al. 2017) dataset for evaluation. We summarize the experimental results in Table 6 Lower perplexities mean better results.
Table 6: Perpelxities on Wiki-text2 dataset (Merity et al. 2017) of OPT (Zhang et al. 2022) models
pruned by K-prune. The term "Difference" represents the ratio of the amount of increased perplexity after pruning compared to the perplexity of the unpruned model, i.e. (Difference) = ((perplexity after pruning) (perplexity before (perpxit) before pruning) 100.
OPT-1.3B
Pruning rate
0%
5%
10%
15%
20%
Perplexity
14.67
14.41
13.96
14.67
15.74
Difference
-1.77%
-4.84%
0.00%
7.29%
OPT-2.7B
Pruning rate Perplexity 12.46
0%
5%
10%
15%
20%
12.23
11.94
12.01
12.51
Difference
-1.85%
-4117%
-3.61%
0.40%
As result, K-prune successfully prunes billion-scale LLMs maintaining its performance. Surpris- ingly K- K-prune shows negligible performance degradation of 0.4% for OPT-2.7B under 20% pruning rate. Note that structured pruning of secoder-based language models much more difficult than that of encoder-basee based models. For example, LLM-pruner (Ma et al.) (203)) shows severe performance degradation of over 30% for LLaMA-7B models on Wiki-text2 dataset. Combined with the obser- vation that larger models achieves higher pruning rates with minimal performance degradation for language models larger than 2.7B. Therefore, applying K-prune decoder-based LLMs is promising future work.
easier prune (Frantar & Alistarh, 2023), we expect that K- K-prune
20


Reviews:
[{"review_id": "CmxHL3xikf", "reviewer": "Reviewer_akzo", "summary": "This paper introduces a new algorithm called Kprune that can significantly improve the accuracy of pretrained language models while compressing them without the need for retraining. The authors explain that while pruning is a common technique for compressing deep neural networks, it is often difficult to apply to pretrained language models due to their complex architecture and the difficulty of preserving their useful knowledge during the pruning process. Kprune addresses these challenges by using an iterative pruning process that selectively removes neurons from the model based on their importance to the overall performance of the model. The authors evaluate Kprune on several benchmark datasets and compare its performance to existing retraining-free pruning algorithms.", "questions": "Can the proposed technique applied to LLMs?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper introduces a new algorithm, Kprune, that can significantly improve the accuracy of pretrained language models while compressing them without the need for retraining. \n- The authors provide a detailed explanation of the iterative pruning process used in Kprune and how it helps preserve the useful knowledge of the pretrained model. \n- The authors evaluate Kprune on several benchmark datasets and compare its performance to existing retraining-free pruning algorithms, providing evidence of its effectiveness. \n- The paper could have practical applications in the field of natural language processing, where the ability to compress pretrained language models without sacrificing accuracy is highly desirable.", "weaknesses": "- The paper focuses specifically on encoder-based language models, so it may not be applicable to other types of language models. \n- The authors do not provide a detailed analysis of the computational resources required to implement Kprune, which could be a potential limitation for some applications. \n- The paper does not apply to large language models which are commonly used now, experiments on larger models are expected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l77BRwvaxB", "reviewer": "Reviewer_9HrX", "summary": "The paper presents KPrune, a new retraining-free structured pruning method for compressing task-specific models while retaining their knowledge. Unlike prior techniques, KPrune considers both the overall loss impact and the effect on representation preservation when deciding which units to prune. It does this by analyzing the expected output of each head/neuron. As a result, KPrune requires minimal pruning time compared to retraining methods and outperforms previous retraining-free approaches. This work differs from Kwon et al. (2022) in two main ways: 1) it incorporates layer-wise representation loss when measuring unit importance, and 2) it recovers weights by tuning the output matrix with linear solvers. These innovations enhance pruning quality with minimal overhead versus retraining. In summary, KPrune efficiently compresses models while maintaining performance by carefully accounting for loss impact and representation preservation when pruning.", "questions": "I don't have further questions for the paper.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and easy to follow!\n- KPrune is the first retraining-free pruning method to incorporate layer-wise representation preservation loss and KL loss on outputs, techniques commonly used in retraining-based pruning.\n- KPrune shows considerable performance improvements over previous retraining-free methods in high sparsity regimes (>80%). This demonstrates its ability to maintain model quality even with extreme compression rates.", "weaknesses": "- While not a direct comparison, it would be interesting to compare KPrune's performance to stronger training-based methods like CoFiPruning. KPrune's  major advantage is requiring much less time for pruning, though training-based approaches may achieve better end results. \n- Since BERT models are relatively small in scale nowadays, reducing training compute is less impactful compared to large language models. It would be interesting to evaluate if KPrune can effectively scale up to larger and more powerful models where saving compute will be more significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cvJqneQCQO", "reviewer": "Reviewer_sVyn", "summary": "The paper introduces Kprune (Knowledge-preserving pruning), a novel retraining-free structured pruning algorithm designed for pretrained encoder-based language models. The key challenge addressed by this work is the accurate compression of such models without requiring retraining. Existing retraining-free algorithms suffer from significant accuracy degradation, particularly at high compression rates, due to their inability to handle pruning errors effectively.\n\nKprune employs an iterative pruning process that focuses on preserving the valuable knowledge contained within the pretrained model. This process includes three main steps: knowledge measurement, knowledge-preserving mask search, and knowledge-preserving weight-tuning. By implementing these steps, Kprune achieves remarkable results, with accuracy improvements of up to 58.02% higher F1 score compared to existing retraining-free pruning algorithms. These improvements are observed under a high compression rate of 80% when tested on the SQuAD benchmark, all without the need for any retraining.", "questions": "Please refer to Weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Kprune presents a significant advancement in the field of pretrained language model compression by demonstrating the feasibility of high compression rates without compromising model accuracy. The approach's success lies in its ability to preserve the essential knowledge within the model, ultimately leading to impressive gains in performance compared to existing retraining-free pruning methods.\n2. The equations are clearly described.", "weaknesses": "1. The writing of the introduction section seems unreasonable. I think some challenges and other content should be written in the introduction instead of the method.\n2. In the main experiment table (Table 1), there are not enough baselines for comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ceTvCiDT89", "reviewer": "Reviewer_nct7", "summary": "This paper presents a method to prune an encoder-only pre-trained transformer-based language model without retraining it. The method is based on the notion of a global importance score for each attention head or feed-forward neuron, considering both the prediction loss (KL divergence between pruned and unpruned models) and the representational loss (L2 distance between weights at each layer before and after pruning). It uses a layerwise pruning of attention heads and feed-forward neurons, starting with the bottom sublayer and working up. In each layer, three steps are performed: 1) Global importance computation: Compute the global importance score for each attention head or neuron. 2) Mask search: Search for a mask for each attention head or neuron, considering both prediction and representational knowledge. 3) Linear projection layer tuning: Tune the linear projection layers for the heads or neurons that are not pruned using a linear solver on a small amount of data. The proposed approach achieves an improvement between 8.5-58.0%p in F1 across tasks/models relative to a training-free approach in Kwon et al. (2022). Compared to the unpruned baseline, the proposed approach achieves an inference speedup of 2.65x, while the method of Kwon et al. gives at most a 1.87x speedup. It achieves comparable accuracy to retraining approaches (such as DynaBERT) with significantly less computation.", "questions": "* Sec 1 : What does the \"%p\" notation mean?\n* Sec 2.2: \"Once the mask variables are established after mask search, pruning of attention heads and neurons whose mask variables are zero does not affect the inference results.\" What does 'established' mean in this context? \n* Equation 7: The LHS K_{rep,l}(X_{\\tau,l},X_{\\rep,l};0)  should be a function of i. \n* Sec 3.3: \"We estimate the importance score not only in the target sublayer but also in the sublayers above the target sublayer\" Why is it necessary to estimate the importance score for sublayers above the target sublayer considering that only the heads/neurons in the target sublayer are pruned when a given layer is considered i.e. From the description/Figure 2, it seems like Algo 1 is run for each sub-layer.\n* The algorithm performs sublayer-wise pruning from the bottom to the top sub-layer. If we refer to this as a single-pass, is it ever necessary in practice to do a second pass to achieve the desired number of FLOPs?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "*  Presents a new method for pruning a pre-trained encoder-only transformer language model without retraining.\n\n*  The proposed approach achieves F1 score improvements of 8.5% to 58% over existing training-free pruning approaches, while achieving similar performance to retraining approaches at a much lower computational cost.\n\n* Reports an ablation study that shows the relative importance of each component of the approach, revealing that layerwise pruning and weight tuning are critical.", "weaknesses": "* Approach is restricted to encoder only models but many of the current LLM approaches are decoder-based.\n* Some aspects of the paper are unclear. See below in questions.\n\nUpdate after author rebuttal:\n* The authors have clarified most of my questions in their rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)