ZipLM: Inference-Aware Structured Pruning of Language Models
Authors: Eldar Kurtic
Elias Frantar
Dan Alistarh
Venue: NeurIPS 2023 poster (2023)
Keywords: LLMs
pruning
compression
inference
TL;DR: ZipLM is a novel efficient framework for producing entire families of state-of-the-art structurally compressed models with runtime guarantees.

Abstract: The breakthrough performance of large language models (LLMs) comes with major computational footprints and high deployment costs. In this paper, we progress towards resolving this problem by proposing a novel structured compression approach for LLMs, called ZipLM. ZipLM achieves state-of-the-art accuracy-vs-speedup, while matching a set of desired target runtime speedups in any given inference environment. Specifically, given a model, a dataset, an inference environment, as well as a set of speedup targets, ZipLM iteratively identifies and removes components with the worst loss-runtime trade-off. Unlike prior methods that specialize in either the *post-training/one-shot* or the *gradual compression* setting, and only for specific families of models such as BERT (*encoder*) or GPT (*decoder*), ZipLM produces state-of-the-art compressed models across all these settings. Furthermore, ZipLM achieves superior results for a fraction of the computational cost relative to prior distillation and pruning techniques, making it a cost-effective approach for generating an entire family of smaller, faster, and highly accurate models, guaranteed to meet the desired inference specifications. In particular, ZipLM outperforms all prior BERT-base distillation and pruning techniques, such as CoFi, MiniLM, and TinyBERT. Moreover, it matches the performance of the heavily optimized MobileBERT model, obtained via extensive architecture search, by simply pruning the baseline BERT-large model. When compressing GPT2, ZipLM outperforms DistilGPT2 while being 60\% smaller and 30\% faster. Our code is available at: https://github.com/IST-DASLab/ZipLM.

ZipLM: Inference-Aware Structured Pruning of Language Models
Eldar Kurtic IST Austria eldar. wurtioiist..ccaat
Elias Frantar
Dan Alistarh
IST Austria
IST Austria & Neural Magic dan. alistarh@ist, ac
elias. frantarGist.ac.
Abstract
The breakthrough performance of large language models (LLMMs) comes with major computational footprints and high deployment costs. this paper, we progress to- wards resolving this problem by proposing novel structured compression approach for LLMs, called ZipLM. ZipLM achieves state-of-the-art securacy-s--peeddee while matching a set of desired target runtime speedups in any given inference environment. Specifically, given model, dataset, an inference environment, as well set of speedup targets, ZipLM iteratively identifies and removes compo- nents with the worst loss- trade-off. Unlike prior methods that specialize or the gradual compression setting, and only for specific families of models such as BERT (encoder) or GPT (decoder), M produces state-of-the compressed models across all these settings. Furthermore, ZipLM achieves superior results for a fraction of the computational cost relative to prior distillation and pruning techniques, making it a cost-effective approach for generating an entire family of smaller, faster, and highly accurate models, guaran- teed to meet the desired inference specifications. In particular, ZipLM outperforms all prior BERT base distillation and pruning techniques, such CoFi, MiniLM, and Tiny BERT. Moreover, it matches the performance of the heavily optimized MobileBERT model, obtained via extensive architecture search, by simply prun- ing the baseline BERT model. When compressing GPT2, ZipL. outperforms DistilGPT2 while being 60% smaller and 30% faster. Our code is available https://gghubb.coom////AAALL///
in either the wost-rraiiigg/oe-ae
1 Introduction
The high accuracy of modern language models from the Transformer family 553 comes at the price of massive computational cost, which hinders their practical adoption in resource-constrained settings. This has motivated the development of model compression techniques, which can be categorized into pruning [17). quantization (9) and distillation [II]. In this paper, we focus on structural compression, whose goal is to reduce model size by removing entire sub-components. such as rows or columns from the model's weight matrices. The key advantage of structured pruning, relative unstructured pruning of individual weights, is that the model can be reshaped to new dimensions, and the resulting computational savings can be leveraged on any hardware, without specialized computational support. At the same time, structured pruning introduces significant challenges. First, models are usually highly-sensitive structured compression, and most methods require gradual compression, including retraining cycles designed to allow the model to recover accuracy. In addition, structural compression significantly complicates the use knowledge distillation [16], which usually done via manual or dynamic layer mapping 121l [9]] On the practical side, another challenge is that most existing techniques do not provide rantime speedup guarantees: the model pruned to fixed sparsity or FLOPS target, and then must be evaluated in the target inference environment. If the pruned model fails to meet the target inference specifications, the whole process must be repeated from scratch.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Overview. In this paper, we resolve these issues and provide a novel structured pruning approach called ZipLM, which achieves state-of-the-ar performance, both in the post post-training/one setting, where retraining is not desirable, as well as in the popular gradual compression setting, where retrain- ing possible. We accomplish this via an inference-aware algorithm, which successfully balances the loss-runtime trade-off at each pruning step. By taking runtime into account, we avoid removing components that do not bring significant speedup gains. Additionally, our algorithm provides speedup guarantees for compressed models, a highly-desirable property in practical applications.
We summarize our contributions as follows:
We introduce a novel structured pruning approach, which unifies the saliency criteria investi- gated by prior work-weight magnitude, activation impact, and removal of lineall-rrdunndat structures, while considering local (layer-wise) and global correlations. We augment it to be inference-awaree ensuring desired latency or throughput in any given configuration.
We complement the algorithm with novel layer-wise token-level distillation, which con- sistently boosts accuracy on small datasets and does not require manual layer matching, circumventing a limitation of prior structured pruning techniques.
ZipLM is the first structured pruning approach that achieves state-of-the-art results for both, post-training/one-stoo compression and gradual praning settings, while being applicable to both, BERT (encoder) and GPT (decoder) language models, without any modifications.
ZipLM is practical and efficient. For a set of desired speedups (e.g. 2x, 5x, 10x) in the target leguence-length-8844 device=V100), in single run and under the same set of hyper-parameters, it produces the entire family of com- pressed models, one for each speedup target. Consequently, it leads to state-of-the- art results in GPU-based inference environments. Moreover, it is compatible with unstructured pruning and quantization, leading state-of-the-ar results even for CPU-based environments.
inference environment (e.g. batch-size=128.
2 Related Work
Distillation-basee compression methods focus on training a smaller student model to mimic the representations of larger teacher model. The "distance" between the representations of student and teacher is often architecture-specffcc MiniLM (55) uses a deep self-attention mechanism to replicate the attention mechanism of the teacher, and TinyBERT [21] employs a bespoke distillation mechanism, for a manually picked subset of layers. Both methods offer a very strong baseline, generally outperforming other approaches, except for MobileBERT. MobileBERT [51] involves first training a custom large BERT teacher model from scratch, and then deviates from the standard architecture 22 by introducing heavily-optimized components with reduced latency, whose combi- nations are decided in neural architecture search (NAS)-like fashion. It achieves strong results in at the cost of significant computational costs in the search process.
terms of ceuuuacy-per-prrnnee DistilBERT and DistilGPT2 [44) involve training a fixed student obtained by removing every other layer from the teacher, while BERT-PKD (50) employs incremental knowledge extraction through the distillation of intermediate layers. Well-Read-Studde architecture through principled downscaling of internal dimensions. DynaBERT [8]] on the other hand, distills knowledge to student model that is both depth- and width-adaptive.
[52) reduces the size of the standard BERT
Structural pruning methods usually start from large pre-trained model, and iteratively reduce the dimensions of weight matrices. Block Movement Pruning (27) identifies and removes redundant rectangular blocks of weights while following the movement pruning intuition 45) that weights moving towards zero during fine-tuning should be removed. FL.OP (56) and Low-Rank (40 use matrix decomposition techniques progressively remove rank- I components from factorized weight matrices during training. -hesses [60] employs similar approach, but replaces entire submodules with smaller counterparts. Methods like LayerDrop 3 and Poor Man's 's BERT 43) address structured compression through various layer-dropping techniques. LayerDrop uses structured layer-dropout regularization to train a model resilient to sub-network selection during inference, while Poor Man's 's BERT explores wide range of layer-dropping strategies. The recent CoFi method (39) employs masks of different granularities to jointly prune coarse and fine-grained submodules during fine-tuning, combined with an optional customized distillation technique. CoFi is the state-of-the-art structural pruning method; relative to distillation methods, CoFi outperforms MiniL. and TinyBERT, but not MoRi,cBERT, in terms of

---

Other compression methods such as the ones that exploit dynamic forms of sparsity which appear at runtime [35], or the ones that utilize lower bit-width representation of weights and/or activations [6] 32 are complementary to our approach. We demonstrate this in Section where we apply quantization
to obtain even higher compression ratios for edge deployment environments like commodity CPUs.
3 Method
Removing large structures like entire matrix columns or attention heads from language model quickly leads to severe aceuracy degradation, from which it is often difficult to recover even with extensive finetuning. This is why current tate-of-theeart approaches like Block Movement Pruning [27] or CoFi [59] opt for integrating pruning directly into training (via sampling or differentiable approximations), rather than performing it in the standard gradual pruning fashion of discrete steps with finetuning in between. However, as we will show, by designing a new highly accurate pruning algorithm which is able to account for both local correlations of structures within single layers as well as global correlations across layers, we can actually apply the gradual pruning paradigm, with all its advantages, to improve significantly over the current of-te-of-the-art
3.1 The ZipLM Structured Pruning Algorithm (Local Correlations)
Most existing structured pruning criteria 33 are based on one or two of the following assumptions about saliency: structures with lower (average) weight magnitude are easier to prune [I5] 29) structures with small input activations can be removed little loss 34 and structures that are close
to linear combination of other structures are the most redundant [4 49). We will now show how all these aspects can be jointly considered in a principled manner via our new ZipLM technique.
Problem formulation. Our approach starts from the idea of applying structured compression layer- wise, a way that allows the layer to preserve most of its output characteristics. This setup is popular in the post-training quantization and unstructured pruning literature [38] 19 g) and can be implemented as follows. We are given a small amount of calibration data, which we run through the network, to obtain "reference" inputs and outputs for each layer. Then, for each layer, given the calibration inputs X and the original layer weights W, aim to find compressed weights W respecting the compression constraint C, which best approximate the original output, measured via the squared error metric. If we assume that the input and weight matrices have an appropriate rectangular form, the problem can be formalized as:
argminw || - W subject to
W E.
This objective can be decomposed across the rows of W, leading to a set of sparse linear regression problems, one per row. These row-wise problems are independent, which forms the basis of related work [8]: yet, since do structured pruning, they become dependent, same weight indices across all rows, i.c. prune entire columns. Thus, finding the optimal weights
would like to prune the
W E C is equivalent to finding: 1) the optimal structure S of the desired shape to be removed, which we assume to be applied across all rows, with corresponding pruning mask Ms. where pruned indices have value in the mask, and others are O: and 2) the corresponding update 0g to all of the remaining weights, optimally compensating for the error caused by the removal of weights in S.
Saliency scores and weight update. Let H X be the Hessian matrix for the fz-minimization problem in Equation I which is independent of the weights. Define W W.,s to be the subset of weights under the mask Mg in row i, and by (H-*) .Ms the submatrix of the inverse Hessian corresponding to the entries under the mask Mg. Then, we can obtain the optimal mask and weight update as follows:
dron
argming
WeMs
((H-1)
Ms. Ma.
)
W.T.ss
(2)
8g W Ms
(HH
M2. Mg
H
(3)
Mg.:
We obtain this by extending the Optimal Brain Surgeon 13
24 formulas for solving Equation I to cover all drow weight matrix rows simultaneously. Importantly, the subselection of the inverse
3

---

Hessian ((H-) Ms MsMMs) .Ms)' is shared between all rows. Further, since we generally consider only non-overlapping sets S of the same size, we pay just Oddool Ms|²) total cost for all extra inversions. Since the number of structures in the mask [Ms] is usually small, c.g. attention heads usually consist of 64 columns, the overall cost of these inversions is low.
Simply selecting the structures to prune according to the criterion in Equation 2 unifies the weight magnitude and activation influence criteria (via the Hessian), but still ignores any correlations between structures. We address this by pruning structures one-at-a-timet while always applying update ōs and fully recomputing H -1 relative to the remaining structures. For example, if there exist two redundant structures S1 and 52, we will first drop Si and update S2 compensate for this removal, at which point S2 is no longer easy to prune. Without this one-at-a-time removal, both structures would have been incorrectly removed they each individually seem easy to prune according to Equation 2 Executing this strategy naively will require a full (dCaa) recomputation of the inverse Hessian relative the remaining structures at each step, which would be very slow, However, this can be avoided by removing the rows and columns corresponding to Ms directly in the inverse with one step of Gaussian elimination
[8]. applied block -wiee to cover larger structures, as follows:
H
((H-))
H Mg.:
.M
Mg.
which takes only O([Ms} d time. We provide complete pseudocode in Algorithm 1
We utilize the fact that the values cor- responding to pruned weights in W and in the inverse Hessian H do not affect any subsequent calculations and can therefore be ignored even they are not exactly zero. However, in the end we have to prune them ex- plicitly again by multiplying with the overall mask to ensure that they are exactly zero In practical imple- mentation, ((H Ms MM,) should only be computed once and reused when computing the corresponding sum across all rows.
Algorithm The ZipLM pruning algorithm. Given inverse (2Xx + AI) -. we remove exactly it
Hessian H structures from the corresponding weight matrix W.
R + set of all possible structures for times do
S + argming Edeo W. W.Mss ((H (MHT'MMs (M) MMg
ds < - ...Mg ((H "MM )= (H M)
W + W + ōs
H + H H. ((H Mg. - H Mg
Mg
R +-++ R - (S}
end for
W W @ MR
Pruned structures. Focusing on Transformers, we consider three types of structural removal: dropping attention heads, shrinking the expanded intermediate dimension of the fully-connected network (FC) layers, and removing entire residual parts, i.e. attention or FC-modules. We implement this by dropping dhead consecutive columns in the out-matrix of the attention block and individual columns in the second linear layer of the feed-forward network. Once these colunn-structures are zeroed out, corresponding rows in previous layers can be safely removed without any output change. Crucially, by pruning e.g. columns in the FC2 layer rather than equivalent rows in FCI, we can utilize the input correlations via
using the ZipLM pruner.
Novelty relative to existing Optimal Brain Surgeon (OBS) approaches. The original frame- work 133, well modern efficient versions [47] 7 g. have been explicitly developed for unstructured pruning, iee removing individual weights. is nontrivial to extend them to structured pruning, this involves considering additional correlations, both within well across multiple blocks (such blocks are usually employed for computational tractability). For example, the state-of- the-art layer-wise approach of [8] performs unstructured pruning by handling weight matrix rows separately, and then greedily merging results. In contrast, we perform structured pruning jointly across multiple rows, which not only necessary for correctness but additionally enables us to design an algorithm with a computational complexity that is lower by a full factor of the hidden dimension size. Additionally, structured pruning requires explicitly matching matrix shapes for consecutive layers and a dedicated strategy for utilizing weight updates even when entire blocks/rows are pruned.
3.2 Inference-Aware Structured Pruning (Global Correlations)
We now describe how to augment the algorithm to be inference-aware, in the sense that it accepts inference specifications, such as batch-size, sequence-length, and speedup on the target hardware, as additional inputs to optimize for.

---

Motivation. The main benefit of inference-aware structured pruning is the fact that pruning decisions are not guided purely by saliency scores, but instead by spspeedup trade-offs associated with the removal of each component in the model. Prior methods, e.g. [24 27 59 focus solely on pruning until a specific sparsity threshold is reached, without taking into account the real-world speedups corresponding to the compression threshold, which can vary significantly between settings. For example, a 95% sparse BERT produced by CoFi [59] has 12x speedup on a V100 GPU, but only 5x on an A100 GPU. With existing methods, if real-world timings fail to meet the inference requirements, the entire process has to be repeated with different sparsity values until the target speedup is achieved, which is both tome-consuming and error- proore An additional advantage of which we showcase in our GPT experiments in Section is that it enables optimizing for different real-world metrics, such as latency or throughput.
Runtime awareness. We integrate runtime constraints via a lateney table for our target inference environment, where we record the time to run an attention block, including all overheads, with 0,..., Nheuss 1 heads pruned and similarly for the fully-connected block with the intermediate dimension shrunk by factor of 0.9', for i = O,... 42; in relative steps of 10% up until se 99% spar- sity, following [5]. This allows rapid runtime estimation for different per-layer sparsity configurations. We provide an example of our latency table in Appendix E
per-layee
Finding the optimal sparsity configuration. Ultimately, our goal is to find a per-layer-sparsie configuration that satisfies certain speedup-constrain while maximizing accuracy. A popular paradigm of doing this 15 30] is to produce a large number of pruned models with different sparsity distributions across layers and then select the one, satisfying target constraint, with the highest accuracy. To make this computationally feasible, ZipLM treats each layer independently. which makes it possible to precompute a database of several pruned versions with different sparsities for each layer. The entire database can be produced in a single run, utilizing the algorithm's 's one-at-a-time nature. While our algorithm is compatible with various search methods for finding layer-wise profiles 15 12
crucial that pruning is cheap, yet accurate.
we adapt the recent SPDY approach (5).
Structured SPDY search. The SPDY approach is designed for unstructured pruning and assigns a quadratic prior to per-layer sensitivity of different sparsity levels. This is not valid in our structured pruming scenario, since for instance it would suggest that dropping a full layer is only slightly more difficult than pruning it to 99% sparsity. Thus, using standard SPDY would lead the algorithm to explore large number of sub-optimal configurations, significantly wasting computational resources. To alleviate this problem, for structured sparsity s, we introduce a better prior ps as the relative
layer-wise squared error incurred by pruning, defined as p. = || - WX [X|]|]|/s |2, which simply has value of for fully dropped layer. Furthermore, the original SPDY approach uses shrinking neighborhood search, which has high variance in both runtime and solution quality for structured compression. Therefore, we perform fixed number of 1000 steps, randomly mutating in expectation 10% of the layer-wise sensitivity coefficients. Finally, we note that any candidate evaluated by this procedure actually achieves the target speedup, leading to significantly decreased search time. We validate our approach in Appendix [F] where we demonstrate that our speedup estimations are indeed very accurate in practice. Specifically, real-world on-device measurements deviate at most by 5.28% from their expected values.
3.3 Layer-wise Token Distillation
For structured pruning. it common to apply layer-wise distillation objectives to transfer intermediate representations. However, structured pruning creates compatibility issues relative to the fixed teacher architecture, leading most methods to develop customized distillation strategies. A popular approach, introduced in (21) and improved by [59], solves the problem via static [21] dynamic (59) mapping of a subset of teacher layers to a subset of student layers. Their main limitation is manual layer selection, where making the "optimal" choice would require evaluating all possible combinations, which can be very expensive. Another limitation is shape-maacheeee matching between intermediate layers, which is solved by introducing learnable linear transformation matrix attached student outputs.
Our approach. We address these challenges differently, by leveraging the fact that ZipL.M preserves the hidden dimension size, and propose to use distillation of intermediate token representations across the entire model. The resulting minimization objective consists of three components:
C(0 (0))) = Craak (00 ()) + Liogig (0 001x))
Xg (0", 0'(zz),
(5)
5

---

where H and 6" represent student and teacher models respectively, z are the inputs, Lrask is the loss associated with the task (e.g. cross-entropy for text-classfficttion) Liogit is the KL-divergence between output logits as described in [16]. and Ltoken is our token-level distillation loss. Hidden tensors passed between consecutive transformer layers are of constant shape H TR. Bx where B stands for the batch-size, seq for the sequence length, and H for the hidden size defined by the R
model architecture. This tensor can be interpreted as collection of x seq vectors each carrying intermediate model representations of input tokens We define the loss Ltoken as an Euclidean distance between vectors h corresponding to each non-padded token the input sequence, averaged over all unpruned layers. Formally, for layer k, it is defined as
Bxseq Σ
kc token
1 A(h
Σe3re 16EP
where stands for the set of padding tokens. This formulation encourages the student model to generate vector representations for each token that are similar to those produced by the teacher model. In Appendix [B] we present ablation studies and comparisons for ZipLM and CoFi, with and without their respective distillation objectives.
4 Experiments
Setup. Given pre-trained model, dataset, and a of desired speedups in a target inference environment, we iteratively fine-tune and prune the model in structured way such that in the end we obtain set accurate compressed models, one for each speedup target. We consider pruning of the standard BERT and BERT large architectures, evaluating on dev-sets of established benchmarks: SQuADv1.1 42 and a subset of GLUE (54) tasks: SST-2 [48]. QNLI [54], MNLI [57], checkpoints from prior work. For precise comparison to prior work [59], our inference environment is single NVIDIA V100 16GB GPU, batch size of 128, and sequence lengths of 384 and 128 for SQuAD and GLUE tasks, respectively. In addition to encoder- based BERT models, we also consider pruning of the decoder- GPT2 model on the WeeneeeeextCorpus [10], for which consider two inference environments: pruning for throughput (batch-size=16, prompts with varying lengths). For illustration, our pipeline is depicted in Figure In Appendix H and [ we report exact values for all results, as well as hyper-parameters for reproducibility.
and QQP [46]. selected to match publicly-availabee
and pruning for latency (batch-size=l, a set of
model: BERT-large
task: SQuADe1.1
batch-size: 128
seq-length: 384
Krai- Prabec
Pecket
device: NVIDIA V100 GPU
speedupe: (2x,
50x)
Inference etvirunmeen specifications
On-Device Multi-leud Attention timing
binter
Hinter N
Leantte
b
proe
ZipLM4 pruning and fine-tuning
On-Device Feed-Fooward Network timing
Figure I: Illustraiion of the ZipLM pipeline: 1) inference specifications, 2) runtime benchmarking of candidates for pruning. 3) gradual structured pruning until all speedup targets are met.
Baselines. In the gradual pruming setting, we explore the performance of ZipLM pruning of BERT- and GPT2-family models, across a wide range of inference speedup targets, ranging from to 15x, in unit increments. This allows us compare the effectiveness of our approach against diverse set of structured pruning and distillation- bbase techniques, including state-of-the-art CoFi pruning, competitive Block Movement Pruning, and distillation approaches including TinyBERT, DistilBERT,

---

DistilGPT2, MobileBERT, MiniLM, and DynaBERT. Additionally, we include comparisons with other relevant methods. For faimesss we follow [59] and report TinyBERT and DynaBERT results without data augmentations. In the post-truining//oesshoo setting, which does not allow retraining, we demonstrate that ZipL.M outperforms the prior state-of-the-an approach of [26]. We evaluate inference speedups of all models in the same environment, unless the models are not publicly available, in which case we report speedups from their respective papers. We refer to ZipLM compressed BERT models as ZipBERT, and to ZipLM compressed GPT2 models as ZipGPT2.
4.1 Gradual Structured Pruning
BERT base results. In Figure 2 we compare structured compression methods the SQuADvl. 1 task. ZipLM outperforms both CoFi and TinyBERT, prior state-of-the-ar techniques, by 3 points in the FI score at the same speedup factor, while at the same Fl score it able to improve inference speedups by at least 60%. In Figure 3 we extend this comparison to a subset of GLUE tasks and provide an exhaustive overview of various structured compression techniques. Results on the other four remaining GLUE tasks are provided in Appendix Figure As can be observed, distillation- based methods usually provide either one or few ceatuurall--ooppessseee models, due to the massive costs associated with training from scratch for each new model. Relative to the most competitive approaches, such as TinyBERT, CoFi, and MiniLM, ZipLM provides consistent improvements in terms of both, accuracy and speedup. while providing guarantees for each compressed model in terms of the expected speedup in the target inference environment. Interestingly, on tasks like QQP and SST-2, ZipLM is able to compress the BERT base model up to 6x and 10x speedups, respectively, while maintaining the accuracy of the dense model. In Appendix [ we provide additional comparisons against CoFi on test-set results from the official GLUE evaluation server.
ZitliKi (om))
Block Novement Frrzing
Dasiitaat FLOP SaBERts
ZiyBERT sgy (MMS))
Block Movertert Prusing
SQuADv1.1
SQuADe1.
F1 soore (%)
Fiscore
66
90
2
5
6
9
10
11
12
13
14
7
10
11
12
18
14
15
Speedop
Spacdap Sacius
Figure 2: Structured compression of BERT base (left) and BERT large (right) on the SQuADv1. task.
Dashed horizontal lines represent full and 99% accuracy recovery of the uncompressed model.
Cafi
DysABERT
BERT
Man's
QNLI
MINLI
SST:2
QQP
92.5
V9.0
Accuncy
Accunac
91.5
91.
90.0
905
908
98.2
MA3
12131411
1811
14
Spuzdup Bactor
Spocilap factorr
Specdap Sectorr
Specshap Sectee
Figure 3: Structured compression of BERT hase on QNLI, MNLI, SST-2, and QQP tasks. Dashed horizontal lines represent full and 99% accuracy recovery of the uncompressed model.
BERT large results. To verify that our approach does not pertain only to the BERT model, we apply ZipLM structured pruning to the 3x larger BERT model on the SQuADvl task. In this setup, we compare against the only two approaches that attempted to structurally compress this larger model, Block Movement Pruning and distillation-bai based MobileBERT. As can be seen in Figure 2 ZipLM is able to compress BERT large up to 4x faster inference while maintaining the F1 score of the uncompressed model. At the same Fl score as the fastest Block Movement Pruning model (3x), MipLM doubles the inference speedup (6x). A result worth emphasizing is that ZipLM is even able to match the performance of the highly optimized MobileBERT model by simply compressing the baseline BERT architecture, without the many additional optimizations and custom components
7

---

used by MobileBERT. Specifically, some of the module- and operator-level optimizations used by MobileBERT include: bottleneck structures and carefully-balancee self-attention and perw-forward modules, embedding layer factorization, bespoke closed-soure teacher model, replacement of LayerNorm layers with lower-latency NoNorm layers, and replacement of GELU activation functions with ReLU activations.
% recovery. The MLPerf Benchmark [36] largets recovery of >99% of the baseline accuracy. At this industry-defined threshold, ZipLM models set new state-of-the-art performance across all of the considered datasets with the following BERT base inference speedups: 5x on the SQuADvl task, 6x on QNLI and MNLI, and, surprisingly. 13x and 15x on SST-2 and QQP, respectively. When compressing BERT large on the SQuADv1 task. ZipLM produces a 6x faster model 99% recovery.
GPT2 results. To validate that our approach does not only apply to encoder-based models, we apply ZipLM structured pruning the decoder-based GPT2 model. In addition this, further demonstrate the inference-awareneen applications, we consider two different regimes: pruning for throughput and pruning for latency. An example application for the former regime is server-side deployment where the model processes many queries the same time, while an application for the latter regime is text-generation scenario where the model is used in an online fashion to auto-complete user's text.
property of our approach and its importance for real-world
For fair comparison. we follow the DistilGPT2 setup 44 and prune the 124M parameters GPT2 variant on the Open WebTextCorpus dataset, followed by zero-shor evaluations, without any fine- tuning, on the test-split of the WikiText (37 dataset. Because of the enormous vocabulary size, the maximum achievable speedup in the throughput regime for this model is roughly 3.5x. Thus, we ZipLM pruning to 1.5x, 2x, 2.5x, and 3x speedup targets. For the latency regime, report the median time to process sequences of various lengths when generating text with Top-K sampling [4]. In Table [ we present zero-shot evaluations of the uncompressed GPT2 model which serves baseline relative to the competing DistilGPT2 approach, and four variants of our ZipLM pruned GPT2. In the pruning for throughput scenario, at similar speedup and decoder size (1.6x-vs-1.55 and 12.5M-vs-47.3MM ZipGPT2 achieves significantly lower perplexities relative to DPst. Further, slightly better (lower) perplexities. ZipGPT2 reduces the decoder size from 42.5M only 26.5M parameters (60% reduction) and improves speedup from 1.6x to 2.1x (30% faster). In the pruming for latency scenario, a similar speedup of 1.9x-v8-2.0K Ox, ZipGPT2 reduces the decoder size by 3M params while providing almost 2 points improvement in the zero-shot perplexity.
Table 1:
Zero-Shot perplexity (PPL) of compressed GPT2 in two regimes: pruning for throughput and pruning for latency. *GPT2 was trained by OpenAl [41] on much larger closed- source dataset and for significantly longer. The only direct com- parison is between DistilGPT2 and ZipGPT2.
Table 2: One-shot (post- training) structured pruning of BERT on three down- stream datasets and two speedup targets.
Pruning for throughput
Pruning for latency
Kwon et al. ZipBERT
Model
Decoder size
Wiki
Decoder size
Wiki
Speedup
26
hase
Speedup
Text-103
Speedup
Text-103 103
SQuAD,
P
PPL $
1.5x
86.2 76.5
87.1
GPT2"
1.0x
85.0M
28.5
1.0x
85.0M
28.5
2,0x
84.1
QQP. acc.
DistilGPT2
1.6x
42.5M
43.0
1.9x
42.5M
43.0
1.5x
89.5
89.7
1.5x
47.3M
35.4
1.6x
48.7M
37.8
2.0x
83,9
84.8
ZipGPT2
2.1x
26.5M
41.5
2.0x
39.2M
41.2
MNLI, acc.
(ours)
2.7x
14,0M
50.4
2,2x
26.6M
49.0
1.5x
82.8
83,0
3.3x
5.7M
72.1
2.5x
20.7M
55.0
2.0x
78.1
78.2
4.2 On the Importance of Inference-Awareness
Depth vs. width pruning. A particularly interesting illustration of the importance of inference- awareness in the pruning algorithm is given by our GPT2 models running directly in the PyTorch--Huggnngaace framework, which can used in two different modes: batch-prediction (thouughput-coostraineee)) and text-generation (latency-constrained) cally large, and shrinking weight matrices is an effective way to achieve speedups. However, for the latter, the inputs much smaller, and the size of weight matrices is no longer the primary bottleneck.
For the former, inputs are typi-
8

---

In this scenario, the only way to achieve substantial speedups is to completely drop some modules, which prior methods cannot account for as they solely optimize for overall model sparsity. However, with ZipI.M, LM, runtime measurements from the target inference environment guide pruning decisions, allowing learn the best way to compress the model for an optimal speedup-accuracy trade-off. Our GPT2 compression results in Table @ clearly illustrate and support these statements. Even though pruned for the same speedup larget, the final architectures of ZipGPT2 models are drastically different. scenario, the model's 's depth was preserved but the matrix dimensions
For the htroughput-coostrriiee were significantly reduced (roughly by a factor of 10) making the corresponding multiplications with large input tensors much faster. In contrast. for the aatency-constraiie (shapes of weight matrices) was mostly preserved but the depth was shrunk almost by a factor of 4, making the forward pass with small inputs faster by reducing the effective number of modules.
scenario, the model's width
Inference device capabilities. Incorporating capabilities of the inference device is another important aspect for effective structured pruning which prior methods do not account for as they solely optimize for higher sparsities. As noted in Section 3.2 this reflects in larges discrepancies between speedups obtained on different devices, e.g. a compressed model with 12x speedup on an 100 GPU. This arises because the A A100 GPU is significantly more powerful and thus faster on the dense model; at the same time, it is highly underutilized for small matrices, which significantly limits the speedups for very high sparsity. To illustrate this, we have measured the speedup from reducing the MLP size for both GPU types (see Table 3). As can be seen, pruning =90% sparsity (3072 + 302) gives 7 speedup on a V100 but only ==3x speedup on an A100. Such differences automatically captured by ZipLM, where pruning for sparsity is replaced by pruning for speedup.
V100 is only 5x faster
Pruning for speedup vs. pruning for sparsity. Figure 4 we compare results with ZipLM pruning when the target for pruning is sparsity (like prior approaches) and when the target for pruning is speedup (the ZipLM approach). Pruning for speedup brings significant improvements, up to 10 points, especially at higher speedups where inference-awarence is very important the algorithm does not remove components that do not bring any further speed and therefore helps preserving accuracy.
SQuADeL.1
Table 3:
Speedups from shrinking the intermediate size of MLPs in the FFN section of a Transformer layer, on different GPUs.
score (%0)
Speedup
MLP size
V100
A100
75
Praning for spamity
Pruming for spoclup
3072
1.0x
1.0x
1814
1.6x
1.1x
7
10
11
13
14
1322
2.0x
1.4x
Speedup factor
302
6.9x
3.1x
130
11.8x
4,4x
Figure 4: Ablation study for the impact of the pruning target: pruning for sparsity (like prior approaches) versus pruning for speedup (the ZipLM approach).
76
13.1x
4,4x
33
14.8x
4.4x
4.3 Post-training/Ome-shoo Structured Pruning
We now study the performance of ZipLM when applied purely in one-shot, without any retraining. In this setting, compare against the state-of-the-ar method of Kwon et al. [26] which combines several heuristics: Fisher-based mask search, mask rearrangement, and mask tuning. Instead of heuristics, our pruning framework utilizes direct end-to-end loss information to find the optimal sparsity configuration. During the warm-start phase, (26) utilizes a diagonal Fisher matrix to estimate the significance of heads and filters, which discards correlations caused by off-diagonal elements. Although the approach attempts to address this limitation by approximating correlations within single layer, it will not capture global dependencies. Furthermore, the weights are adapted for layer- wise reconstruction at the very end of the compression step, whereas our method does it continuously during the pruning (please see Section for the significance doing this). For a fair comparison, we apply the authors' own implementation in latency-constrained mode on the exact same model weights Table presents results on several datasets and speedups, showing that ZipL.M is even more accurate than the approach designed and optimized specifically for the post-training/one pruning.

---

Sensitivity to calibration data. Additionally, we have found that ZipLM is very robust to the amount of calibration data. In T we present sensitivity analysis with respect the number of calibration samples. We one-shot prune BERT hase on the 1QuADv1. task for two speedup targets: 1.5x and 2.0x. In this setup, we compare results against Kwon et al. [26], which uses 2048 samples by default. As can be seen from the table, ZipL.M outperforms prior state-of-the
starting at only 32 samples. As we increase the number of samples, the results improve, up to 2 points in Fl score.
5 Discussion and Extensions
CPU as an LLM-inference environment. In Section 4 we have focused on various GPU-based inference environments as enabled us to conduct fair comparisons against prior structural compres- sion techniques. However, CPUs present another compelling inference environment focused on edge deployment of LLMs. Therefore, we target the recently proposed compound compression pipeline of 224, which involves three steps: structured pruning, unstructured pruning, and quantization. We replace their structured pruning approach based on layer dropping with ZipLM. As a result, at full accuracy recovery, are able to improve speedup from 3x to 13x, and at the largest compression ratio from 30x to 50x. Due to space constraints, we provide full results in Appendix A
Computational efficiency. Relative to iivilllationbbasee methods, structured pruning is an order of magnitude more efficient in terms of GPU hours due to the massive costs associated with pretraining from scratch for each compressed model [59] [1 21]. For efficiency comparisons to CoFi, we consider the task of producing a full family of compressed BERT base models with speedup targets ranging from 2x to 15x. In this setup, ZipLM requires only 115 epochs in total, whereas CoFFi would require 560 epochs. Therefore, ZipL.M 4.87 87 times more efficient than CoFi. In terms end-to-end runtime, ZipLM produces the entire family of compressed BERT base models GPU in ~35 hours on larger datasets (e.g. MNLI) and only 10 hours on smaller ones (c.g. SST2). Finally, it is worth emphasizing that have not taken into account the cost of hyper-parameter tuning in the above comparisons, but that this is very favorable to ZipLM: it uses a single set of hyper-parameters produce entire family of compressed models while other methods require hyper- hyper-parameter tuning for each model independently.
single RTX A6000
SQuADvL11
Table 4: Sensitivity the number of calibration samples.
BERT distillation
90
ZipBER N (ours)
F1 score (%)
base
ZipBERT Tuzge (ours)
Fl score at
85
medium
Num samples
Method
1.5x
2.0x
snall
80
4
82.3
48.4
32
86.8
82.6
75
mnir
ZipLM
128
86.8
83.6
10
20
30
40
50
60
70
80
512
86.8
84.1
Speedup factor relative to BERT
2048
87.1
84.1
4096
87.6
84.7
Figure 5: Scaling laws of structured pruning vs. distilla- tion on the standard BERT architecture.
Kwon et al.
2048
86.2 76.5
Scaling laws for structured pruning. To further understand the eecuracy-speedie trade-offs, we run ZipLM larger speedup ratios, up 55x for BERT large and 75x for BERT base To the best of our knowledge, this is the first result in literature demonstrating that such extreme compression ratios achievable with structured pruning without model collapse. In Figure 5 compare these results against distillation- based downscaling of the BERT architecture (52). The results clearly demonstrate that each of the pruned models, based either on BERT large or BERT base significantly outperforms comparable pre-trained variants. An emergent behavior that can be observed is that structurally pruned models tend to follow linear scaling law, meaning that the accuracy decreases linearly with the increase of the speedup ratio, at a slope given by the original model. Fitting linearly via least squares produces the following expressions for the accuracy-speedup relationship: Fliarge Ri 92.1 - 0.3 x speedup and F1 R 90.3 0.6 x speeduppase- Thus, the rate of decrease in accuracy for BERT is twice as large as that of BERT large. which can be attributed to the presence of more redundant representations in the larger model, making it more resilient to pruning. In Appendix G we provide additional analysis of the structure of pruned models.
10

---

References
[1] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv: (t0::9988.0999 09791, 2019.
[2] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[3] Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth demand with structured dropout. In International Conference on Learning Representations, 2019.
[4] Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arXiv preprint arXiv: 04833, 2018.
[5] Elias Frantar and Dan Alistarh. SPDY: Accurate pruning with speedup guarantees. arXiv preprint arXiv: 13096, 2022.
[6] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprini arXiv:2210. 2022.
[7] Elias Frantar, Eldar Kurtic, and Dan Alistarh. M-fac: Efficient matrix-free approximations of second-order information. Advances in Neural Information Processing Systems, 34, 2021.
[8] Elias Frantar, Sidak Pal Singh, and Dan Alistarh. Optimal Brain Compression: A framework for accurate post-training quantization and pruning. arXiv preprint arXiv:2208. //800, 2022. Accepted to NeurlPS 2022, to appear.
[9] Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103.13630, 2021.
[10] Aaron Gokaslan and Vanya Cohen. Openwebtext corpus, 2019.
[1]] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey, International Journal Computer Vision, 129(6):179--18)) 2021.
[12] Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16, pages 164--60. Springer, 2020.
[13] Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
[14] Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4340-4349, 2019.
[15] Yihui He, Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In Proceedings of the European conference on computer vision (ECCV), pages 804, 2018
[16] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, al. Distilling the knowledge in a neural network. arXiv preprint arXiv: 02551. 2(7), 2015.
[17] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and A Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1-124, 2021.
[18] Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. Advances Neural Information Processing Systems, 33:9782-9793,, 2020. 2020.
11

---

[19] Itay Hubara, Brian Chmiel, Moshe Island. Ron Banner, Joseph Naor, and Daniel Soudry. Accelerated sparse neural training: A provable and efficient method find m transposable masks. Advances in Neural Information Processing Systems,
2021.
[20] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko.. Quantization and training of neural networks for efficient integer-arithmetic-oe
inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2713,, 2018
[21] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding, In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4 4163-4174, 2020.
[22] Aran Komatsuzaki. One epoch is all you need. arXiv preprint arXiv:/906. 1ee.11996606699 2019.
[23] Eldar Kurtic and Dan Alistarh. Gmp*; Well-tuned global magnitude pruning outperform most bert-pruning methods. arXiv preprint arXiv:2210.06384, 2022.
[24] Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 4163-4188 2022.
[25] Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Bill Nell, Nir Shavit, and Dan Alistarh. Inducing and exploiting activation sparsity for fast inference on deep neural networks. In Hal Daumé IIII and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5533-5543, Virtual, 13-18 Jul 2020. PMLR.
[26] Woosuk Kwon, Sehoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. A fast post-training pruning framework for transformers. arXiv preprint 1rXiv:2204009655 2022.
[27] François Lagunas, Ella Charlaix, Victor Sanh, and Alexander Rush. Block pruning for faster transformers. In Proceedings of the 202/ Conference Empirical Methods in Natural Language Processing, pages 10619-10629, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[28] Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue. Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175-184. Association for Computational Linguistics, November 2021.
[29] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 1608::6600.07/00 2016.
[30] Yawei Li, Kamil Adamczewski, Wen Li, Shuhang Gu. Radu Timofte, and Luc Van Gool. Revisiting random channel pruning for neural network compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 191-201, 2022.
[31] Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruning for efficient neural networks, arXiv preprint arXiv:1911.07412. 2019.
[32] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for Imm compression and acceleration. arXiv preprint a7X.v:2306.00978, 2023.
12

---

[33] Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang. Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. In International Conference Machine Learning, pages 7021-7032. PMIR, 2021.
[34] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference computer vision, pages 2736-2744 2017.
[35] Zichang Liu, Jue Wang, Tri Dao. Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivas- tava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient Ilms at inference time. In International Conference on Machine Learning, pages 22137-22176. PMLR, 2023.
[36] Peter Mattson, Vijay Janapa Reddi, Christine Cheng, Cody Coleman, Greg Diamos, David Kanter, Paulius Micikevicius, David Patterson, Guenther Schmuelling, Hanlin Tang, et al. MIperf: An industry standard benchmark suite for machine learning performance. IEEE Micro, 40(2):8-16, 2020.
[37] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
[38] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pages 7197-7206. PMLR, 2020.
[39]
NeuralMagic. Deep sparse: A fast cpu inference engine, 2021.
[40] Matan Ben Noach and Yoav Goldberg. Compressing pre-trained language models by matrix decomposition. In Proceedings of the Ist Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 884 889, 2020.
[41] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[42] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In EMNLP, 2016.
[43] Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. Poor man's bert: Smaller and faster transformer models. arXiv preprint arXiv:2004.03844 2020.
[44] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01100 108, 2019.
[45] Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: fine-tuning. Advances in Neural Information Processing Systems, 33:20378-20389 2020.
Adaptive sparsity by
[46] S. Shankar. Identifying quora question pairs having the same intent. 2017.
[47] Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. Advances in Neural Information Processing Systems, 33, 2020.
[48] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, 2013.
[49] Yang Sui, Miao Yin, Yi Xie, Huy Phan, Saman Aliari Zonouz, and Bo Yuan. Chip: Channel independence-ben pruning for compact neural networks. Advances Neural Information Processing Systems, 34:24604-24616 2021.
13

---

[50] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language P Processing (EMNLP-IJCNIPP), pages 4323-4332, 2019.
[51] Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mo- bilebert: compact task agnossti bert for susource-limited devices. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2158-2170 2020.
[52] Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: The impact of student initialization on knowledge distillation. ArXiv, abs/1908.055 0988. 2019.
[53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[54] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. ArXiv, abs/1804.07465 2018.
[55] Wenhui Wang. Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33:5776-5788, 2020.
[56] Ziheng Wang. Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6151-6162, 2020.
[57] Adina Williams, Nikita Nangia, and Samuel Bowman. broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics, 2018.
[58] Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Piemie Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics.
[59] Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 1513-15288 1528, 2022.
[60] Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. Ber--of-theseus: Com- pressing bert by progressive module replacing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7859-7869, 2020.
14

---

A Compound Compression for Edge Deployment
Deploying large language models in edge environments requires running inference on low-power devices such as CPUs. Therefore, we follow the compound compression approach from [24] which bundles together structured, unstructured pruning, and quantization for efficient inference on CPUs. We start with ZipLM structurally pruned models, and apply unstructured pruning method [24] to 80% sparsity. After structured and unstructured pruning, we apply prrntiiatioonaaa--traaiie
the state-of-the-a ofrtte-rttttttt oBERT
(QAT) 2 to quantize FP32 weights into INT8 representations. We benchmark these compound compressed models by running inference in the DeepSparse [39] engine, single-core of Intel Cascade Lake CPU. In this setting, we compare compound compression pipeline (24) which applics layer dropping as a form of structured pruning. As can be seen from Figure when substitute layer dropping with principled structured pruning via Li,LM, the resulting compound compressed models achieve very competitive aatency-s--ccureee performance in the edge-inference regime. At full accuracy recovery, ZipLM improves the speedup from 3x to 13x, while at the largest compression ratio ZipLM improves the speedup from 30x to 50x.
results against the
batch-size-1, scq-len=384, #cores=1
15 ms
CPU inference speedup
50
ZipLM+oBERT+OAT
40
20 ms
30
30 ms
20
60 ms
10
310 ms
100
99
98
97
96
95
94
93
92
Fl recall (%)
Figure 6: Improvements in CPU-inference speedups for compound compressed BERT models on the SQuADvl.1 task when ZipLM used for structured pruning. End-to-end latency indicated by the dashed line.
B Ablation Studies
In Table we present ablation results for ZipLM and CoFi, with and without their respective layer- wise distillation techniques. ZipLM outperforms CoFi in all tasks when both methods use distillation, and in three out of four when distillation is not used. For example, ZipLM outperforms CoFi with a significant point increase in Fl score on the SQuAD task both setups. Furthermore, when comparing ZipLM results with and without layer-wise distillation, it can be observed that benefits pronounced for low data tasks, where accuracy improvements reach up to points.
C Additional GLUE results
Due to space constraints, Figure we present results only on four GLUE tasks. Therefore, for completeness, in Figure 7 we present results on the remaining four GLUE tasks, namely: CoLA, MRPC, STS-B, and RTE, Results show the same trends as the other four GLUE tasks, with large improvements for ZipLM, especially at higher compression rates.
15

---

Table 5: Comparison of ZipLM and CoFi dev-set results, with and without layer-wise distillation.
SST-2
QNLI
MNLI m-ace.
SQuAD
acc.
acc.
F1
CoFi ZipBERT
90.4
86.1
80.6
82.6
91.7
88.6
81.7
85.7
CoFi w/o Linyer
91.1
85.1
79.7
82.5
ZipBERT w/o Looken
89.2
86.5
81.2
85.7
ZEy (mn))
DIAIBERT The BERT
Pooe Mtrts BERT Malli
Well Road Stadiets Lon Haank BERT
LayeDOop BErt-FDD
of
Cafl
MRPC
STS-8
RTE
Mathoon cee
33
15
Speedup
Speedap
Speedop
Speedap factor
Figure 7: Structured compression of BERT base on CoLA, MRPC, STS-B, and RTE tasks. Dashed horizontal lines represent full accuracy recovery of the uncompressed model.
D Additional Validation
Evaluating and comparing compressed models the development (dev-set) is standard practice, as enables comparisons with off-the-shelf results from the literature. However, an implicit assumption behind such comparisons that all methods tune their hyper-parameters only on a subset of the dev-set before evaluating and reporting results on all samples, which is not always the case. Moreover, specifically- -uuned peeee-ppraeeeers can lead to large performance differences, especially when compressing LLMs [23). To ensure that there is no such "overfitting" on the dev-set, in Table we compare ZipLM against the prior state-of-the-en the-art CoFi approach on unseen test-set, obtained by submitting predictions to the official GLUE evaluation server. The results show consistent improvements over CoFi, on both dev- and test-
E Latency table used for ZipLM pruning
As described in Section 3.2 we record the time to run an attention block, including all overheads, with O, ..... Nheads 1 heads pruned (pruning everything with runtime 0 is also considered) and similarly for the fully-connected block with the intermediate dimension shrunk by factor of 0.9*, for i = o..... 42: in relative steps 10% up until & 99% sparsity, following (5). In Table 7 we present an example of such latency table used in ZipLM pruning approach.
F Speedup Evaluations
As shown in Figure D ZipLM is based on measuring runtimes higher-level modules, such as attention heads and fully connected matrices, rather than low-level operators. This makes our
Table 6: Dev- and test-set comparison of ZipBERT bass and CoFi models with comparable speedups.
dev-set ZipBERT
test-set
CoFi
CoFi
ZipBERT hase
QNLI, acc.
86.1
88.6
85.8
88.4
SST-2, acc.
90.4
91.7
88.2
91.8
MNLI, m-ace-
80.6
81.7
80.7
81.9
MNI., mm-acc.
80.7
82.0
79.9
80.6
SQuAD, F1
82.6
85.7
N/A
N/A
16

---

Table 7: An example of the latency table used by the ZipLM pruning approach.
Intermediate size
Latency (ms)
Number of heads
Latency (ms)
3072
11.9
12
7.9
1814
7.4
10
6.7
1322
5.8
5.8
302
1.6
4.4
130
1.0
3.2
76
0.9
2
1.9
0.7
0
Table 8: Comparison of target (desired) inference speedups with achieved (on-device measured) speedups obtained with our ZipLM pruning approach.
BERT on SQuADv1.1
BERT large on SQuADv1.1
Target speedup
Achieved speedup
Deviation
Target speedup
Achieved speedup
Deviation
I.98
-1.00%
2.01
+0.50%
4
4.05
+1.25%
4
4.05
+1.25%
6
6.16
+2.67%
6
6.09
+1.50%
8
8.25
+3.12%
8
8.27
+3.37%
10
10.36
+3.60%
10
10.33
+3.30%
12
12.31
+2.58%
12
12.46
+3.83%
14
14.33
+2.35%
14
14.74
+5.28%
approach independent of underlying optimizations in different inference engines and frameworks, which usually perform further optimizations such as operator-fusion. Our runtime lookup table contains information about the runtime of Transformer layer with different numbers of attention heads, and various dimensions of the fully connected matrices. This implies that we measure runtimes of layer with 12 heads, 11 heads, 10 heads, and so on, as well as the runtimes of fully connected matrices with hidden sizes ranging from 3072 to O. We utilize this information to guide pruning decisions.
To fully validate the ability of MipL. to compress the model while satisfying desired speedup constraints via the described approach, provide the timing results in Table comparing the desired (target) speedup and the achieved (measured) speedup for different models.
As can be seen from the Table 8 the deviation between the desired (target) and the achieved (measured) speedup is at most 5.28%. This confirms that our approach indeed provides reliable runtime information guide the pruning decisions.
G Structure of Pruned Models
Through comprehensive examination of ZipLM pruned BERT models across all datasets considered in Section we aim to identify trends the pruning of key components the Transformer layer, namely attention heads and intermediate size, needed to achieve specific speedup target. As illustrated in Figure g we observe that the intermediate size is pruned at a higher rate relative to attention heads, which aligns with the fact that the intermediate size dictates the dimensions of the two large linear layers in the feed-forward part of the Transformer block. For instance, to attain a 2x speedup, roughly 60% of the intermediate size and 40% of the attention heads need to be removed. Additionally. in Figure 9 we visualize the entire encoder size needed to reach a specific speedup target. Interestingly,
find that 15x faster models retain average only 2% of intermediate size and 6% attention heads which amounts to only 2.9M parameters overall, while at the same time recovering more than 95% of the uncompressed model's accuracy (see Figure
we visualize number of remaining heads and intermediate size across all Transformer layers and various speedup targets subset of GLUE datasets.
Additionally, in Figures TO 12
13
17

---

100
Percentage of pruned (%)
80
60
40
BER T
Attention heads
BERT
Attention heads
large
20
BERT
Intermediate size
base
BERT
Intermediate size
large
0
1
2
3
5
6
8
9
10
11
12 13 14 15
Speedup factor
Figure 8: Percentage of pruned attention heads and intermediate size to reach a specific speedup target with M.
85.1
80
Encoder-size (M)
60
40
38.1
23.7
20
17.0
13.1
10.3
8.5
7.2
6.1
5.2
4.5
4.0
3.5
3.2
2.9
2
7
8
9
10
11
12
13
14
15
Speedup factor
Figure 9: Encoder size vs. speedup factor of ZipLM pruned BERT base models, averaged over all considered datasets in Section
15x
Fx
12x
800
1b035
6
600
2
Trandremeer layan indian
Transforrict Dayan indlen
Figure 10: Remaining number of attention heads and intermediate size across layers of the ZipLM compressed BERT base model at various speedups and MNLI dataset.
H Experiments Additional Results
In Table we report accuracy and model size of ZipL. pruned models visualized in Section in Figures
and
18

---

6c 12X 15x
3c
12x 15k
800
Ahods
600
1
10
0
Trunsfoetner layer Index
T2 Transforract layret indix
Figure 11: Remaining number of attention heads and intermediate size across layers the ZipLM compressed BERThase hase model at various speedups and QNLI dataset.
12X
15k
9x
12x
15x
1250
thods
1000
730
000
14
250
10
Transforroer layer Endex
Transformrr layer Lodex
Figure 12: Remaining number of attention heads and intermediate size across layers of the ZipLM compressed BERT model at various speedups and QQP dataset.
12x
15x
5x
6x
12x
18x
1250
rhods
1000
750
000
23
250
Transformeer layer index
Transforer layer index
Figure 13: Remaining number of attention heads and intermediate size across layers of the ZipLM compressed BERT model at various speedups and SST-2 dataset.
I Hyper-parameles for Reproducibility
To facilitate reproducibility, we conduct experiments in the open-source Transformers library (58) and use publicly available datasets (28). We plan to open-source our entire framework which supports one-shot and gradual structured pruning via SparseML (25]. making very easy experiment with other models and datasets. In addition to our code, we plan to open-source all of our compressed models via the popular HuggingFace Hub. Table 10 we report syper-parametert used to produce our ZipLM pruned models in Section Because of the excessive memory overhead, we don't 't make use of any kind of knowledge distillation when pruning the GPT2 model. Following insights from DistilGTP2, we hypothesize that this can further improve our results. We follow [22] and disable dropout regularization while pre-training ZipGPT2 models at OeenWebTextCorput dataset.
Broader Impact and Limitations
Our results contribute the line of work on efficient language models. Thus, it should help reduce the energy and monetary cost of inference over such models, and allow them to be used without access to powerful hardware. While this is a mainly positive outcome, also reduces the cost of employing these models for detrimental purposes, such as spam generation. Thus, this significant cost reduction for inference should also seen as further motivation for methods to ensure safe usage of these models, such watermarking alignment.
19

---

Table 9: Accuracy and model size for ZipLM pruned models in Section
BERT Jase
BERT Targe
QNLI Encoder size (M)
MNLI
SST2
QQP Encoder size (M)
SQuADv1
SQuADv1
Encoder size (M)
Encoder size (M)
Encoder size (M)
Encoder
Speedup
Acc.
Acc.
Acc.
Acc.
F1
size (M)
2x
91.4
38.0
84.8
38.5
93.4
38.7
91.3
37.8
89.1
37.3
91.6
141.1
3x
91.1
23.8
84.8
23.5
93.4
24.1
91.3
23.8
88.6
23.4
91.4
88.3
90.9
16.9
84.0
93.0
16.8
88.0
16.8
63.1
5x
90.8
12.5
84.0
13.5
93.0
13.5
91.1
13.0
87.5
13.0
90.8
48.5
6x
90.4
9.5
83.5
10.5
93.0
11.0
91.1
10.2
86.7
10.4
90.2
39.1
7x
89.8
8.0
83.2
8.8
93.0
9.0
90.9
8.1
86.1
8.7
89.9
32.7
89.2
6.4
83.1
7.5
93.0
7.6
90.9
6.8
85.7
7.5
89.7
27.5
89.1
82.8
93.0
6.7
5.8
85.3
23.8
10x
88.6
4.9
82.7
5.4
93.0
5.7
90.8
4.9
84.2
5.3
89.1
20.9
11x
88.6
4.0
82.5
4.7
92.7
4.9
90.7
4.3
83.8
4.7
88.8
18.4
12x
87.8
3,6
81.7
4.1
91.7
4.2
90.6
4.1
83.2
4.0
88.4
16.4
13x
87.6
81.3
3.5
3.8
3.7
82.5
87.9
14.9
14x
87.4
2.8
81.2
3.3
91.7
3.6
90.3
3.3
81.7
3.2
87.7
13.7
15x
87.2
2.6
80.8
2.9
90.7
3.2
90.3
2.9
81.4
2.9
87.6
12.5
Table 10:
used for gradual ZipLM runs in Section
BERT
BERT
GPT2
16 SQuADvl
batch-size
128
32 GLUE
384 SQuADv1 128 GLUE
max mex-sch--en
1024
finetune before pruning
epochs
50k steps
finetune in between pruning steps
8 epochs
10 epochs
2 opochs
LR schedule in between pruning steps
linear decay
linear decay
initial LR
8e-5
Se-5
le-3
#calibration samples
2048
512
speedup-targets
[2, 3, 4, 5, .... 15)x
(1.5, 2, 2.5, 3)x
knowledge distillation
1.0
1.0 SQuADvl GLUE
knowledge distillation Xa
0
SQuADvI 0.5 GLUE
knowledge distillation Aa
0
weight-decay
0.03
20

---

As any academic study, our work is not without its limitations, All of our benchmarks are focused on English-language
English language datasets and therefore our results do not provide insights into compression effects for low-data languages. Unfortunately, this limitation is inherent to all of the existing works on compression due to the lack of standardized benchmarks. Given that our structured pruning approach relies on small sample of calibration data to perform pruning decisions, we hypothesize that our approach should able to provide satisfying results the low-data setup as well. At the moment we do not have data support these claims, but we see it as opportunity for future work.
21


Reviews:
[{"review_id": "Wo7i1rEH0i", "reviewer": "Reviewer_27Y8", "summary": "This paper proposes a structured compression method to optimize inference efficiency for language models. The proposed method uses the accuracy-efficiency trade-off under specific inference objectives as the importance measure of the model component( attention head, MLP neurons, entire FC/attention blocks.) Extensive evaluation results are provided to demonstrate the effectiveness of the proposed method. ", "questions": "Table 1 shows a significant drop in PPL compared to the original GPT2, even at the smallest speed-up. To better understand the trade-off, can the authors comment on at what speedup we can expect no loss in performance? ", "limitations": "The authors didn't discuss limiation or negative social impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "I appreciate the thoroughness of the evaluation, in which the authors consider the encoder/decoder and decoder-only model,  retraining compression and zero-shot compression, throughput and latency objective, etc. Evaluation provides a better understanding of the proposed method. ", "weaknesses": "This paper focuses on improving the language model's inference efficiency. Other than distillation and pruning, quantization is also a popular direction[1], which is not discussed/compared. Quantization generally reduces the model size. At the same time, a reduction in model size would require less number of GPUs for large model inference, leading to latency speed up.  Further, there is a line of work on dynamic sparsity[2] that improves inference latency, which is also not discussed/compared.\n\n[1]Frantar, Elias, et al. \"Gptq: Accurate post-training quantization for generative pre-trained transformers.\" arXiv preprint arXiv:2210.17323 (2022).\n[2]Liu, Zichang, et al. \"Deja vu: Contextual sparsity for efficient LLMs at inference time.\" (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1: Discussion of alternative approaches (quantization, Deja Vu)**\n\nPlease note that quantization is complementary to both distillation and pruning; in practice, it is applied in conjunction with these two. Thus, rather than comparing pruned and quantized models, one can _combine_ these compression techniques to obtain compound improvements.\n\nThis was our approach in this submission, where _we did apply quantization_ to ZipLM models. Specifically, in the “CPU as an LLM-inference environment” paragraph in section “5 Discussion and Extensions”, we describe how we combine ZipLM pruning with quantization. We used quantization-aware training (QAT) which, instead of quantizing only weights (like GPTQ), quantizes both weights and activations, thus enabling direct deployment of LLMs on edge devices as well as computational speedups for arbitrary batch sizes and number of input tokens (GPTQ only leads to speedups for generative inference at very low batchsize).\n\nDeja Vu leverages dynamic forms of sparsity, which appear at runtime, such as activation sparsity. Thus, Deja Vu is also complementary to ZipLM, which induces and leverages static structured sparsity. Further, while promising, Deja Vu comes with a few key limitations:\n\n(1) As of yet, DejaVu has only been shown to work on a few very specific models (OPT-30B/66B/175B and BLOOM-175B), which are extremely large and not very efficient to start with.\n\n(2) DejaVu requires complex custom CUDA kernels to produce runtime speedups, which are optimized for one-token-at-a-time inference, while speedups quickly diminish for larger batch-sizes or non-generative applications.\n\nThe structured pruning approach we adopt is much more general; we have shown it to be applicable to both batch-prediction (throughput-constrained) and text-generation (latency-constrained) use cases. In addition to text-generation (GPT) use cases, our work also shows viability in various (non-generative) downstream tasks such as: text-classification, question-answering, spam detection, etc, for which Deja Vu currently _does not yield speedups_.\n\nIn summary, both quantization and Deja Vu (contextual sparsity) are complementary to ZipLM. We have already shown compatibility with quantization in our submission (Section 5), and plan to investigate compatibility with contextual sparsity in future work: in particular, we plan to investigate whether ZipLM structured sparsity can induce significant additional contextual sparsity. \nWe will add a clarifying discussion on both approaches in the next version. \n\n\nOn a technical note: the Deja Vu paper was not publicly available at the time of the NeurIPS submission deadline, and therefore we would not have been able to cite or discuss it.\n\n**Question 2: PPL difference relative to the original GPT2**\n\nGPT2 has been trained by OpenAI on a much larger (> 10x) closed-source dataset, for significantly longer than both the DistilGPT2 and ZipGPT2. In contrast to that, DistilGPT2 and our ZipGPT2 models are trained on the same open-source dataset, for a much shorter period of time. Therefore the only possible direct comparison in this setup is between DistilGPT2 and ZipGPT2, which is clearly in favor of our approach: relative to DistilGPT2, we provide higher accuracy at the same speedup, and higher speedup for the same accuracy. \n\nIn fact, to support these claims we pretrain GPT2 model from scratch on the open-source dataset used by DistilGPT2 and ZipLM. After evaluating the resulting model in the same setup as the other two models, we are observing the perplexity of 38.5 (the closed-source trained GPT2 has 28.5). This means that when trained on the same data and under similar computational budget, DistilGPT2 and ZipLM models do exhibit competitive performance to the uncompressed GPT2 model.\n\n**Question 3: Limitations and social impact**\n\nPlease note that, following the NeurIPS23 submission guidelines, we have provided a discussion of limitations and impact in the 'Appendix H: Broader Impact and Limitations'. \n"}, {"review_id": "R0ts09Kild", "reviewer": "Reviewer_mFy7", "summary": "This paper proposes ZipLM, a structured pruning and reconstructing + layer-wise distillation + inference-aware pruning algorithm. The authors first extend the pruning formula of OBS to structured pruning and utilize estimation of inference for each structure and structured SPDY search to achieve more accurate inference-aware pruning. Then, the authors distill the model in a token-wise manner. Experimental results demonstrate that ZipLM achieves state-of-the-art (SOTA) performance in both one-shot and pruning/knowledge distillation settings.", "questions": "1. Could you provide a more comprehensive ablation experiment, specifically to identify which module within the entire framework contributes significantly to performance improvement?\n2. Since Kwon [1] also employed a pruning metric and the layer-wise reconstruction via LLS for structured pruning, have you directly compared your pruning metric and the performance after reconstruction of OBS with their method to determine which one performs better, apart from the effect of distillation?\n3. Line 192-193: For example, a 95% sparse BERT produced by CoFi has 12x speedup on a V100 GPU, but only 5x on an A100 GPU. Since this is an interesting observation that FLOPs-based calculation != inference speed, can you explain what is the potential reason of this?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. ZipLM demonstrates exceptional results among existing compression algorithms. It not only outperforms retraining-constrained approaches but also surpasses the performance of some pruning + knowledge distillation algorithms.\n2. The authors propose a systematic framework for structured pruning, encompassing aspects such as pruning and reconstruction, inference-aware structure search, and knowledge distillation for performance recovery. This comprehensive framework exhibits a well-designed structure that, in my opinion, contributes to the entire community.", "weaknesses": "1. The methodology is somewhat incremental. The core contributions of the authors revolve around extending previous methods (OBS, SPDY) to structured pruning since those methods couldn't be directly applied. While the authors have made these extensions, the core essence of the method remains largely based on the previous framework.\n2. The experimental section of this paper has some shortcomings. For instance, although the authors conducted a simple ablation experiment, they did not analyze which specific parts of the framework played a crucial role in improving performance. Additionally, in the majority of compression papers, the GLUE benchmark's eight datasets are evaluated as a whole since they represent the fundamental measure for assessing the performance of compressed models. However, the authors only evaluated four of these datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1: Ablations**\n\nIn our submission, “Appendix B: Ablation Studies” presents results of an ablation study on the impact of distillation during the fine-tuning stage. As can be seen, our distillation technique can improve results up to _2 points_ in accuracy. Additionally, in Figure 2 of the response PDF, we focus on ablations specifically for the pruning metric. We compare results with ZipLM pruning when the target for pruning is sparsity (like prior approaches did) and when the target for pruning is speedup (the ZipLM approach). As can be seen from the Figure, the choice of pruning for speedup brings significant improvements, up to _10 points_, especially at higher speedups where inference-awareness is very important. For additional ablations on inference-awareness, please see our answer below for the difference between V100 and A100 speedups.\n\nIn summary, these ablation studies decouple improvements from the three major components of our framework: token distillation, pruning metric, and inference-awareness. Please let us know if you would like to see any additional ablation studies, and we will be more than happy to provide them during the discussion period. \n\n**Question 2: Evaluation on the remaining tasks in the GLUE suite**\n\nWe ran ZipLM on the remaining four tasks from the GLUE benchmark suite (CoLA, MRPC, STS-B, and RTE) and presented results in Figure 1 of the response PDF. As can be seen, ZipLM provides state-of-the-art results across these datasets as well, especially at higher speedup targets relative to prior structured pruning and distillation based approaches.\n\n**Question 3: Comparison of pruning metrics with Kwon et al. without distillation**\n\nWe would like to highlight that we already show in “Table 2: One-shot structured pruning of BERT-base” that ZipLM outperforms the prior state-of-the-art one-shot approach of Kwon et al., in a setup without fine-tuning and thus also without distillation. In this setup only pruning metric and reconstruction are being compared.\n\n**Question 4: Difference in speedups on V100 vs A100**\n\nThis discrepancy for CoFi models arises because the A100 GPU is significantly more powerful and thus faster on the dense model; at the same time, it is highly underutilized for small matrices, which significantly limits the speedups for very high sparsity. To illustrate this, we have measured the speedup from reducing the MLP size for both GPU types (see Table below). As can be seen, pruning to ~90% sparsity (3072 -> 302) gives ~7x speedup on a V100 but only ~3x speedup on an A100. \n\n*Table 1: Speedup improvements from shrinking the intermediate size of MLPs in the FFN section of a Transformer layer.*\n| MLP size | V100 | A100 |\n| :--------: | :----: | :----: |\n| 3072     | 1.0    | 1.0    |\n| 1814     | 1.6  | 1.1  |\n| 1322     | 2.0   | 1.4  |\n| 302      | 6.9  | 3.1  |\n| 130      | 11.8 | 4.4  |\n| 76       | 13.1 | 4.4  |\n| 33       | 14.8 | 4.4  |\n\nSuch discrepancies are captured by inference-awareness of ZipLM approach, where pruning for sparsity is replaced by pruning for speedup which can utilize this information to guide pruning decisions.\n\n**Question 5: Novelty**\n\nRegarding novelty, we would like to note a few additional points, relating to the answers to your questions above:\n\n- Conceptually, ZipLM introduces two innovations relative to prior instances of the Optimal Brain Surgeon (OBS) framework: it focuses on structured pruning (so it can produce speedups on any hardware), and it is inference-aware (so it directly relates accuracy loss with real-world speedup gains). This is enabled by new technical derivations for the layer-wise structured compression problem, and by algorithmic insights which serve to speed up the combinatorial search (see the discussion in lines 172-182). \n- In addition, we also propose a very effective form of distillation for structured pruning. \nPractically, this leads to a very powerful framework, as we can produce the entire Pareto frontier corresponding to accuracy-vs-compression a lot more efficiently than prior methods: as shown, our method is on average _10 times more computationally efficient_ for this task than CoFi, the prior SOTA method.  \n- As illustrated by the V100/A100 runtime discrepancy, a key differentiating feature of ZipLM is taking inference characteristics directly into account during compression. As such, ZipLM is the first method that yields state-of-the-art results for all 8 GLUE tasks, question answering (SQuAD), and text generation (WikiText), across BERT-base, BERT-large and GPT2 models, for both GPU and CPU deployment targets. \n\n"}, {"review_id": "sRbsBmRwko", "reviewer": "Reviewer_h7Df", "summary": "The paper proposes ZipLM, a structured pruning method that can achieve desired target runtime speedups.​ The idea of ZipLM is to solve a layer reconstruction problem with a structural constraint which minimizes the output changes on a set of calibration examples if the layer is reconstructed as it. The proposed ZipLM is shown to be effective on both decoder-only models and encoder-only models, outperforming prior distillation approaches and structured pruning approaches.", "questions": "* Do you think the proposed approach can apply to larger language models such as LLAMA-7B? If not, what is the barrier?\n* How sensitive the proposed approach is to the calibration examples?\n", "limitations": "I didn’t see a clearly potential negative societal impact of this paper.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "* The paper focuses on structured pruning problems, where the pruned model can easily get real speedups. Moreover, the proposed approach takes as input a target speedup as well as the hardware environment and optimizes the model on this specific setup to ensure the model to achieve desirable speedup. I admire this realistic setting and believe that this can be practically impactful.\n* The paper has conducted extensive experiments studying the effectiveness of the proposed ZipLM approach, including on both encoder and decoder models, both one-shot pruning setting and gradual compression setting, and with different levels of sparsity. \n* The experimental results show the proposed approach is more effective compared to the existing baseline methods.\n", "weaknesses": "* I believe the paper (especially the methodology part) can be presented better and a lot more details should be added (even in the appendix). For example, how do you solve the reconstruction problem and obtain the optimal mask and weight update (equation (2) & (3))? How do you consider the constraint C in equation (1) in the solution equation (2) & (3)? What exactly does a latency table look like and how do you get the table given a hardware environment? How many calibration inputs did you use in your experiments and how sensitive the final results will be to the number/quality of the calibration inputs?\n* The paper is motivated by structured compression of large language models (LLMs). However, all experiments are conducted on models with hundreds of millions of parameters. Given the current state-of-the-art language models (e.g., LLAMA-7B/13B/65B) contain a couple orders of magnitude more parameters compared to BERT, it is not clear how the proposed method works in larger models.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1: I believe the paper (especially the methodology part) can be presented better and a lot more details should be added (even in the appendix). For example, how do you solve the reconstruction problem and obtain the optimal mask and weight update (equation (2) & (3))? How do you consider the constraint C in equation (1) in the solution equation (2) & (3)?**\n\nThank you for the comments on the presentation, which we will follow in the next revision. \nThe main idea for deriving (2) and (3) is that (1) consists of the sum over row-wise linear regression problems. For linear regression, the exact error incurred by removing a set of weights as well as the optimal adjustment to the remaining weights can be determined in closed form, e.g., via solving a Lagrangian. Then, to couple structures across rows, we utilize the fact that there is no interaction between rows in the error of (1). Consequently, we can sum up the per-row formulas in appropriate ways to introduce our required coupling.\n\n(2) and (3) provide optimal formulas for removing a *single* structure; we then efficiently iterate those formulas until we have reached the desired sparsity target dictated by $C$ (we note that this iteration is not necessarily globally optimal but appears to be a good approximation in practice).\n\nIn the next revision, we will improve the presentation of this part and include more details in the Appendix.\n\n**Question 2: What exactly does a latency table look like and how do you get the table given a hardware environment?**\n\nFor a given model, batch-size, sequence-length and a device, we measure time of the forward pass through the main components of the Transformer layer, running in isolation: the attention and the FFN module. For the attention module we measure timings for varying number of heads, and for FFN module for varying intermediate size.\nThe supplementary material of our submission contains an example of such a latency table at: code/bertbase_squad_V100.txt. Since the table is formatted such that it can be directly consumed by ZipLM algorithm, we provide a human readable example here:\n\n| Intermediate size | Latency (ms) |     $\\|$      | Num of heads | Latency (ms) |\n| :-----------------: | :------------: |--- | :------------: | :------------: |\n| 3072              | 11.9         | $\\|$ | 12           | 7.9          |\n| 1814              | 7.4          | $\\|$ | 10           | 6.7          |\n| 1322              | 5.8          | $\\|$ | 8            | 5.8          |\n| 302               | 1.6          | $\\|$ | 6            | 4.4          |\n| 130               | 1.0            | $\\|$ | 4            | 3.2          |\n| 76                | 0.9          | $\\|$ | 2            | 1.9          |\n| 33                | 0.7          | $\\|$ | 0            | 0            |\n\n**Question 3: Do you think the proposed approach can apply to larger language models such as LLAMA-7B? If not, what is the barrier?**\n\nIn principle, there is no major obstacle to appling ZipLM to massive models such as LLAMA-7B, and in fact we believe this is a very interesting question for future work. Our results show that the method translates to good speedups both for generative inference (ZipGPT2) and for large-batch scenarios (ZipBERT). The main challenge to porting ZipLM to billion-scale models would be computational: since structured compression methods require significant fine-tuning for best results, one would need to have the computational budget (as well as sufficient GPU memory to train with optimal settings) to reproduce a fraction (e.g. 5%) of the original training in order to allow for good recovery.  \n\n**Question 4: How sensitive the proposed approach is to the calibration examples?**\n\nWe have found the method to be very robust to the amount of calibration data. \nTo illustrate this, in the table below, we present a sensitivity analysis with respect to the number of calibration samples. We one-shot prune the fine-tuned BERT-base model on the SQuADv1 task for two speedup targets: 1.5x and 2.0x. In this setup, we compare results with the current state-of-the-art one-shot pruning approach of Kwon et al, which uses 2048 samples by default. As can be seen from the table, ZipLM outperforms prior state-of-the-art starting at only 32 samples. As we increase the number of samples, the results improve, up to 2 points in F1 score.\n\n|       Speedup = 1.5x    |   |  $\\|$  |  Speedup = 2.0x | |\n| :----------------: | :-------- | :--------------: | :--------------: | :-------------: |\n| Num samples      | F1 score |     $\\|$           | Num samples | F1 score |\n| ZipLM, 4         | 82.33    |        $\\|$        | ZipLM, 4 | 48.41 |\n| ZipLM, 32        | 86.76    |        $\\|$        | ZipLM, 32 | 82.63 |\n| ZipLM, 128       | 86.79    |      $\\|$          | ZipLM, 128 | 83.56 |\n| ZipLM, 512       | 86.79    |       $\\|$         | ZipLM, 512 | 84.06 |\n| ZipLM, 2048      | 87.08    |       $\\|$         | ZipLM, 2048 | 84.14 |\n| ZipLM, 4096      | 87.62    |      $\\|$          | ZipLM, 4096 | 84.68 |\n| Kwon et al, 2048 | 86.20     |     $\\|$           | Kwon et al, 2048 | 76.50 |\n\n"}, {"review_id": "hoUwIqLzaS", "reviewer": "Reviewer_rTQd", "summary": "This paper looks into structure pruning (e.g. neurons, or attention heads) of Transformer based models.\nAuthors propose to formulate the pruning objective by requiring the output of each pruned layer to be as close as possible to that of an unpruned layer. Then they adopt optimal brain surgeon algorithm to come up with the weights to keep (mask) and the update to such weights. The proposed algorithm prunes each group (e.g. a neuron in a layer) one by one, ensuring that correlated groups are properly accounted for (which is hard to do when choosing a number of neurons to prune in one go). \nAdditional extensions to the method include inclusion of inference aware criteria into the algorithm for choosing the groups to keep (using pre-computed knowledge of latency table)\nSuggested approach can be used for one shot pruning and gradual pruning. Experimental results indicate good performance on attention head pruning, fully connected layers pruning and removing full attention modules.", "questions": "1) How does pruning two consecutive layers happens? First pruning first layer (Algorothm 1) and then pruning the next layer?\nIf yes you are essentially assuming that layers are independent (so assuming block diagonal structure of the hessian). \n2) In your formulation (1) you seem to suggest that X is the input to the layer (whose weights W you are considering to prune). In the original Optimal Brain Surgeon, X is actually based on the gradients (and the global objective is considered, instead of local l2 objective of matching each layer's output). I think this all needs to be made clearer \n2) for experiments, it would be nice to report flocs or any other time measurement for your method (not the inference) and competing methods. \n3) Potentially you can apply your idea of pruning only one structure at a time (line 155 in Algorithm 1)  to any of the existing methods, so ablation is needed (is your improvement comes from the fact that u do essentially more gradual pruning?)\n4) The method you propose does not seem to be LLM specific - why not to compare it with methods on structured pruning for non LLM models\n\n\n- I am confused about contribution on line 56 (produces a family of compressed models). How is it any different from say using gradual pruning with various levels of sparsity (and at each step updating the copy for that sparsity level). You are still saving/updating multiple copies of the models right?\n\n\nMinor:\n- Algorithm 1: Mask Mr is not defined and is not updated in the body of the algorithm. I think you want to compile it based on R at the end of k iterations (the same also applies to Ms - please define how this mask looks like and its shape)", "limitations": "n/a", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Really good experimental results\n- Well written and easy enough to follow (modulo my comments below)", "weaknesses": "- Not very clear novelty\n- seems computationally expensive\n-  Ablations are missing (is the improvement coming from the fact that you use formulation (1) for each layer? is it from line 155 of the algorithm? Is it the inference awareness inclusion?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1: Computational Cost**\n\nPlease note that in the section “5 Discussion and Extensions”, paragraph “Computational efficiency”, we report two additional metrics: end to end runtime and efficiency with respect to the number of epochs, comparing also against the strongest competitor, CoFi:\n\n- In terms of the total number of epochs, ZipLM is more efficient than CoFi by a factor of _4.87x_ on larger tasks and _14.5x_ on smaller tasks.\n- In terms of the end to end runtime, ZipLM is very efficient as well: for large GLUE tasks, it produces all 14 compressed models in only 35 hours on a single GPU, while for the small tasks it does so in only 10 hours.\n\n**Question 2: Ablations**\n\nIn our submission, “Appendix B: Ablation Studies” presents results of an ablation study on the gains from our distillation technique and demonstrates improvements up to _2 points_ in accuracy. Additionally, in Figure 2 of the response PDF, we focus on ablations specifically for the pruning metric. We compare results when pruning for sparsity (like prior approaches did) and when pruning for speedup (the ZipLM approach). The results suggest that the choice of pruning for speedup brings significant improvements, up to _10 points_, especially at higher speedups where inference-awareness is very important.\n\n**Question 3: How does pruning two consecutive layers happen?** \n\nIn step 1 of our framework (Section 3.1), we indeed consider individual layers in isolation; but account for all correlations between different channels within each to perform accurate local pruning. However, in step 2 (Section 3.2) we also handle global interactions between layers by constructing (in an inference-aware manner) and evaluating many global pruning candidates based on the layer-wise solutions produced in step 1. Thus, overall, ZipLM captures both local as well as global correlations, which is a key feature of our algorithm.\n\n**Question 4: Formulation (1) vs. original OBS** \n\nYes, this is correct. In the local pruning step (Section 3.1), we apply similar techniques to the OBS, but in the context of the layer-wise pruning problem defined in equation (1). For this particular formulation, the Hessian, which OBS approximates via Fisher gradients, can be calculated directly as XX^T (and our OBS is thus exact in this case). We will improve the clarity of this aspect in the next revision.\n\n**Question 5: Potentially you can apply your idea of pruning only one structure at a time to any of the existing methods.**\n\nThe idea of pruning only one structure at a time cannot be applied to most other structured pruning approaches because they do not apply weight updates to remaining weights during the pruning step (e.g., they simply remove structures with lowest saliency); hence there would be no benefit of such a strategy. In contrast, in ZipLM, after removing the first structure, we can very efficiently update the remaining weights to compensate for this removal and re-evaluate their scores before the next selection. We would further like to emphasize that this one-at-a-time removal is only enabled by the high efficiency of our algorithm; applying this idea to other approaches which compute updates in less efficient ways would be far too slow (e.g., the MLP-layer of BERT-base has > 3k structures to remove one-at-a-time).\n\n**Question 6: The method you propose does not seem to be LLM specific - why not to compare it with methods on structured pruning for non LLM models**\n\nIt is true that aspects of our method are general, and could be extended to other model types (CNNs or ViTs). We have chosen to focus on LLMs in our practical comparison since they are a clear focus for efficiency research these days, and, consequently, we have several high-quality baselines to compete against. \n\n**Question 7: I am confused about contribution on line 56 (produces a family of compressed models).**\n\nThis contribution warrants special attention because all other structured pruning methods necessitate _a complete gradual pruning process for each sparsity target independently_, since intermediate checkpoints are not accurate enough. In contrast, ZipLM achieves the generation of all models within a single run, producing remarkably accurate intermediate models featuring varying levels of sparsity.\n\nTo elaborate, if one were to obtain models with compression ratios of 2x, 3x, ..., up to 15x using the CoFi method, it would entail distinct runs for each ratio: one for 2x, another for 3x, and so forth, up to 15x.\n\n**Question 8: Please define how the masks look like and their shapes**\n\nWe use $M_A$ to denote a pruning mask where all entries corresponding to the set of indices in $A$ are 1 and other elements 0; e.g., $M_R$ is a mask of all remaining indices after pruning. We will make this more clear in the next revision.\n\n**Question 9: Not very clear novelty**\n\nRegarding novelty, we would like to note a few additional points, relating to the answers to your questions above:\n\n- From the practical perspective, ZipLM is the first _gradual structured pruning approach_ which enables it to produce the entire Pareto frontier of models a lot more efficiently than the previous state-of-the-art method CoFi. As we detailed in Question 1 and Question 7, ZipLM is _4.87x_ more efficient on larger datasets and _14.5x_ more efficient on smaller datasets.\n- This advancement over prior work is enabled by a number of conceptual innovations: a new highly accurate pruner (as shown to be state-of-the-art in one-shot scenario), implemented by a highly efficient algorithm (as demonstrated by the ability to do one removal at a time with re-evaluation of the pruning scores), and a new distillation technique. All of this is complemented by inference-awareness (as demonstrated in the ablation study in Figure 2 of the response PDF).\n"}, {"review_id": "WJl7OYTcGc", "reviewer": "Reviewer_5zkW", "summary": "This paper proposed a novel structured compression approach for the inference of LLM, which reduces the model size by removing entire sub-components, like rows or columns from the model’s weight matrices. Based on their algorithm, they manage to drop out some attention heads and shrink the size of FFN layers. Besides, the proposed method considers the runtime speedup while deciding the sparsity of the pruning, which leads to a better trade-off between the loss and the performance. ", "questions": "Please refer to the Weekness", "limitations": "The potential social impact is not discussed in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work comes up with a new structured pruning method for the transformer models and achieved reasonable speedup in the experiments with multiple models.\n- The theoretical analysis is solid. ", "weaknesses": "- The performance of the proposed method seems strongly dependent on the data distribution of inference, which may make it not that practical in real-world inference settings where the incoming inputs can vary a lot.\n- In the latency-constrained scenario, the authors said \"However, for the latter, the inputs are much smaller, and the size of weight matrices is no longer the primary bottleneck.\", however, in my opinion, when the batch size is very small (like 1), loading weights from GPU HBM is exactly the bottleneck. Therefore, I guess sparsifying model parameters should lead to reasonable speedups?\n- In the throughput-constrained (aka. large batch size), it will be interesting to see how ZipLM could support an even larger batch size since we can take advantage of the memory saved by pruning to hold more inputs. In this way, we may further increase the throughput?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1: The performance of the proposed method seems strongly dependent on the data distribution of inference, which may make it not that practical in real-world inference settings where the incoming inputs can vary a lot.**\n\nWe would like to note that encoder-only models, such as BERT, are designed for per-task finetuning. Similarly, ZipLM enables per-task pruning, to very large speedup ratios. This per-task setting is completely standard in pruning & distillation literature around encoder-only models. Additionally, we also apply ZipLM to decoder-only models, which are typically used for diverse tasks without additional finetuning. Consequently, we perform pruning on a very general dataset (OpenWebText) and evaluate the model in zero-shot fashion on the unseen WikiText dataset.\n\nIn terms of sensitivity to different amounts of calibration data, we have performed additional experiments and found that ZipLM is very robust: even with only 32 samples, we outperform the prior state-of-the-art Kwon et al. using 2048 samples, in the one-shot setting. We present results in the table below:\n\n|       Speedup = 1.5x    |   |  $\\|$  |  Speedup = 2.0x | |\n| :----------------: | :-------- | :--------------: | :--------------: | :-------------: |\n| Num samples      | F1 score |     $\\|$           | Num samples | F1 score |\n| ZipLM, 4         | 82.33    |        $\\|$        | ZipLM, 4 | 48.41 |\n| ZipLM, 32        | 86.76    |        $\\|$        | ZipLM, 32 | 82.63 |\n| ZipLM, 128       | 86.79    |      $\\|$          | ZipLM, 128 | 83.56 |\n| ZipLM, 512       | 86.79    |       $\\|$         | ZipLM, 512 | 84.06 |\n| ZipLM, 2048      | 87.08    |       $\\|$         | ZipLM, 2048 | 84.14 |\n| ZipLM, 4096      | 87.62    |      $\\|$          | ZipLM, 4096 | 84.68 |\n| Kwon et al, 2048 | 86.20     |     $\\|$           | Kwon et al, 2048 | 76.50 |\n\nFinally, to ensure that there is no overfitting of hyper-parameters to the validation set, we have also performed test set evaluations on the official GLUE servers in Appendix C: Additional Validation.\n\n**Question 2: In the latency-constrained scenario, the authors said \"However, for the latter, the inputs are much smaller, and the size of weight matrices is no longer the primary bottleneck.\", however, in my opinion, when the batch size is very small (like 1), loading weights from GPU HBM is exactly the bottleneck. Therefore, I guess sparsifying model parameters should lead to reasonable speedups?**\n\nYes, structured pruning does, in general, bring speedups for memory-bound applications like small batch-size generative inference as well. What we are referring to in the paper is that for our particular model and inference environment, generative runtime is significantly impacted by overheads (e.g. kernel launches, layer-norms, under-utilization, etc.) as low batch-size matmuls are so fast (on larger models or on low-power devices, this should be much less significant). This means size reduction does not necessarily lead to proportional speedups. ZipLM, however, understands this automatically and accounts for it during pruning. This is what we wanted to illustrate in this part. Nevertheless, we agree that this can be made more clear and will make improvements in the next revision.\n\n**Question 3: In the throughput-constrained (aka. large batch size), it will be interesting to see how ZipLM could support an even larger batch size since we can take advantage of the memory saved by pruning to hold more inputs. In this way, we may further increase the throughput?**\n\nYes, this is an excellent suggestion, and we therefore ran some benchmarking tests on an 11GB RTX 2080Ti. For each model, dense (1.0x) and ZipLM compressed (4.9x, 9.8x, 14.5x) models, we increase batch-size to the maximum value and evaluate the throughput (number of samples processed per second). The table below presents throughput improvements from increased batch-size for the corresponding speedup targets.\n\n| Speedup | Throughput gain |\n| :-------: | :---------------: |\n| 1.0x    | 1.0x            |\n| 4.9x    | 6.5x            |\n| 9.8x    | 12.0x           |\n| 14.5x   | 17.6x           |\n\n**Question 4: The potential social impact is not discussed in the paper.**\n\nPlease note that, following the NeurIPS23 submission guidelines, we have provided a discussion of limitations and impact in the 'Appendix H: Broader Impact and Limitations'. \n"}]

Author rebuttal:
We wish to sincerely thank the reviewers and the AC for their work, and for the valuable feedback. We have provided individual responses for each reviewer question. We outline answers to some common topics below: 

- Regarding the discussion of limitations and potential negative social impact in our work (Reviewers **27Y8** and **5zkW**), we would like to clarify that these aspects were taken into account in our original submission. Following the NeurIPS 2023 Call for Papers, we have addressed them in a dedicated section called '*Appendix H: Broader Impact and Limitations*,' where we delve specifically into the limitations and potential negative social impact of our research.
- Similarly, the ablation studies on various components of our method were already present in ‘*Appendix B: Ablation Studies*’ which isolates the accuracy impact of e.g. distillation on the final accuracy of the model. 

In addition, we have performed the following additional experiments, which further address the reviewer concerns:

- To address the question of Reviewer **mFy7**, we have run ZipLM on the remaining 4 GLUE tasks. The results are illustrated in Figure 1 of the PDF response, and show the same trends as the original tasks. Specifically, ZipLM provides significant gains over prior methods, especially at higher compression rates. Results also showcase the fact that our method is significantly more computationally-efficient than prior work: we were able to obtain the full trade-off for each task in a single execution, whereas prior work would need a separate execution for each target, requiring on average _10x more computation_.

- To address the questions regarding differences in speedup on different GPUs and using different metrics (Reviewers **mFy7** and **5zkW**) we have provided a detailed analysis of how these differences arise, and how pruning for size compares with pruning directly for speedup using our method (see also Figure 2 in PDF response). The results show that pruning directly for speed is critical to obtaining good practical performance at the same accuracy level. The full answers are provided in the individual responses.

- We also provide an ablation with respect to the number of samples, showing that our method is extremely stable with respect to sample complexity. Specifically, we can outperform the prior SOTA method of Kwon et al. (which uses 2048 samples by default) starting at just 32 samples. 

We thank the reviewers again for their feedback, and look forward to the discussion. 


Meta-review: This paper proposed a structural pruning method for language models, with the consideration of runtime at the inference time.
The reviewers generally like the comprehensive studies offered in this paper, and the good empirical results obtained. However the concerns are around the novelty and technical contribution w.r.t. the prior works. Nevertheless, this paper has made good contribution to the field of network pruning, and I encourage the authors to refine the paper further by incorporating the reviewers' comments into the revisions.
Decision: Accept (poster)