BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation
Authors: Peng Xu
Wenqi Shao
Mengzhao Chen
Shitao Tang
Kaipeng Zhang
Peng Gao
Fengwei An
Yu Qiao
Ping Luo
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Model Compression
LLM Pruning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: a differentiable pruning technique for large language models, strong perfromance on pruning various LLMs

Abstract: Large language models (LLMs) have demonstrated outstanding performance in various tasks, such as text summarization, text question-answering, and etc. While their performance is impressive, the computational footprint due to their vast number of parameters can be prohibitive. Existing solutions such as SparseGPT and Wanda attempt to alleviate this issue through weight pruning. However, their layer-wise approach results in significant perturbation to the model's output and requires meticulous hyperparameter tuning, such as the pruning rate, which can adversely affect overall model performance. To address this, this paper introduces a novel LLM pruning technique dubbed blockwise parameter-efficient sparsity allocation (BESA) by applying a blockwise reconstruction loss. In contrast to the typical layer-wise pruning techniques, BESA is characterized by two distinctive attributes: i) it targets the overall pruning error with respect to individual transformer blocks, and ii) it allocates layer-specific sparsity in a differentiable manner, both of which ensure reduced performance degradation after pruning. Our experiments show that BESA achieves state-of-the-art performance, efficiently pruning LLMs like LLaMA1, and LLaMA2 with 7B to 70B parameters on a single A100 GPU in just five hours. Code is available at [here](https://github.com/LinkAnonymous/BESA).

Published as a conference paper at ICLR 2024
BESA: PRUNING LARGE LANGUAGE MODELS WITH BLOCKWISE GARAMETER-EFFICIEN SPARSITY AL-
LOCATION
Peng Xu Wenqi Shao Mengzhao Chen² Shitao Tang
Kaipeng Zhang² Peng Gao? Fengwei An
Yu Qiao² Ping Luo" 2122
The University of Hong Kong OpenGVLab,
Shanghai AI Laboratory
3 Southern University of Science and Technology
ABSTRACT
Large language models (LLMs) have demonstrated outstanding performance various tasks, such as text summarization, text their performance impressive, the computational footprint due to their vast num- ber of parameters can be prohibitive. Existing solutions such as SparseGPT and Wanda attempt to alleviate this issue through weight pruning. However, their layer-wiss approach results in significant perturbation the model's output and requires meticulous hyperparameter tuning, such as the pruning rate, which can adversely affect overall model performance. To address this, this paper introduces a novel LLM pruning technique dubbed blockwise parameter-efficient sparsity al- location (BESA) by applying a blockwise reconstruction loss. In contrast the typical layer-wise pruning techniques, BESA is characterized by two distinctive attributes: i) it targets the overall pruning error with respect to individual trans- former blocks, and ii) it allocates layer-specific sparsity in a differentiable manner, both of which ensure reduced performance degradation after pruning. Our exper- iments show that BESA achieves state-of-the-an aht performance, efficiently pruning LLMs like LLaMAI, and LLaMA2 with 7B to 70B parameters on single A100 GPU in just five hours. Code is available here.
and etc. While
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable performance in a wide range of NLP tasks, including language modeling, code generation, machine translation, sentiment analysis, and question answering (Zhang et al., 2022a; Touvron et al., 2023a;b; Xu et al., 2023; Team, 2023; Zeng et al., 2022). However, LLMs have vast number of parameters, resulting in high memory consumption and slow inference speed (Dettmers et al., 2022). For example, it requires 335GB GPU memory (i.e. five A100 GPU with 80G memory) to load parameters in FP16 of Falcon-180P (Penedo et al., 2023), which corresponds to the inference speed of merely 4 tokens per second. Thus, there has been considerable interest in compressing LLMs to make LLMs more efficient and practical for deployment in various applications.
One of the approaches to compress a network is weight pruning. Although it has a long history in model compression (Hassibi et al., 1993; Hassibi & Stork, 1992), few pieces of work can be used to prune LLMs due to the requirement of extensive retraining. Recent studies, such as SparseGPT (Frantar & Alistarh, 2023) and Wanda (Sun et al., 2023) aim to tackle this challenge by reconstruct- ing the layer-wise output of LLMs, as illustrated in 1(). Specifically, SparseGPT proposes to prune unimportant with importance metric derived from the hessian matrix. and then recon- struct layer-wise output. Moreover, Wanda removes intricate computation in SparseGPT by only leveraging the product of weight and activation magnitudes.
While these approaches can eliminate considerable unnecessary weights, they typically operate within each weight by minimizing each layer's pruning error, which has two drawbacks. First, layer- wise pruning eтror minimization does not effectively mitigate the impact of pruning on the model's
"Corresponding authors: Ping Luo, pluo@cs..k hku.h;;; Wenqi Shao, shaowenqi @ pjlah.org.cn

---

Published as a conference paper at ICLR 2024
Input
Mageitate
Input
Rewroedff who vardltte
Attention
Sparce Attention
Werdd
Weight
Sparan Weight
Sparse
Layer Reconstruction
1
85 59
02
04 87
Block Reconstructio
(a) Block-wise pruning error
[b] Perplesitty vs. Sparsity
[c] Prior work
(d) Our BESA
Figure (a) shows that layer-wise pruning methods such as Wanda (Sun et al., 2023) produce a more significant error than our block-wis wiiee pruning technique BESA. (b) presents the curves of perplexity v.s. sparsity for different layers on WikiText2 (Merity, 2016). We see that layers do not contribute equally to the final performance.
(c) shows that prior works prune all linear projections in the transformer block by layer reconstruction. (d) expresses that our proposed BESA compresses LLMs under a block-wis wise reconstruction pipeline.
output because the pruning error would accumulate layer by layer as demonstrated in Fig. (a). Sec- ondly, layer-wise pruning also requires handcrafting the sparsity for all layers, as the individual contributions of layers to the final model performance exhibit significant variation, as illustrated in 1 Applying a uniform pruning rate to all layers, as seen in prior methods, poses the risk of removing important weights, given the unequal contributions of layers to the final performance.
To address these challenges, we propose the Blockwise Parameter-Effiiien Sparsity Allocation (BESA) technique for compressing L.MMs, which optimizes pruning rates across different layers as shown in Fig. (d). Toward this goal, we first formulate the sparsity allocation problem to mini- mize block-wise reconstruction error with a learnable binary mask for each weight. BESA enjoys two advantages for LLM compression. Firstly, the sparsity that was previously considered differentiable hyperparameter can be now equivalently represented by differentiable binary masks. Hence, layer-wise pruning sparsity can be optimized using simple gradient descent algorithm. Secondly, unlike traditional approaches (Kang & Han, 2020) that learn sparsity for the entire model, BESA optimizes pruning rates sequentially within each transformer block. This enables efficient and differentiable pruning of LLMS ranging from 7B to 180B parameters on a single A100 GPU.
However, directly learning binary masks challenging because it involves a huge solution space. To mitigate this issue, BESA encodes the fact that a more important weight would have a lower pruning probability in a parameter-efficien block in LLaMA 1-7B). The binary mask can be generated with element-wist pruning probabilities whose gradients are easily obtained through straight-through estimation (Bengio et al., 2013). Such procedure remarkably reduces the solution space and alleviates the learning difficulty. We further develop a comprehensive LLM compression framework where weight pruning and quantization are jointly optimized in differentiable manner. Extensive experiments show that BESA achieves state- of-the-art performance in pruning various LLMs such as LLaMAI (Touvron et al., 2023a), and LLaMA2 (Touvron et al., 2023b).
way (e.g. 2.10% extra parameters of transformer
Overall, this work has three contributions. (1) We propose a model pruning framework named BESA for compressing LLMs which searches for optimal pruning rates for each layer in differ- entiable manner. To the best of our knowledge, BESA is the first differentiable pruning algorithm for L.Ms.. (2) Our BESA is parameter-efffccient and easy to optimize, exhibiting high efficiency and effectiveness in pruning various LLMs such as LLaMA1, and LLaMA2. For example, BESA can prune 50% parameters of LLaMA2-70B (Penedo et al., 2023) within five hours on a single A A100-80GB GPU with 0.16 perplexity improvement on WikiText2 (Merity, 2016) compared with SparseGPT (Frantar Alistarh, 2023). (3) Extensive experiments on language modeling tasks such as Text PTB (Marcus et al., 1994), and C4 (Raffel et al., 2020) and various downstream tasks show that BESA establishes new state-of-the-art performance compared with prior arts. Finally, we demonstrate the practical speedup of the pruned model in a hardware simulator.
2 RELATED WORK
Compression of Large Language Models. Numerous technologies aim to mitigate the memory and computation demands of Large Language Models (LLMs). These techniques can be broadly
2

---

Published as a conference paper at ICLR 2024
categorized into primary types: quantization (Frantar et al., 2022: Lin et al., 2023; Shao et al., 2023) and pruning (Sun et al., 2023; Frantar & Alistarh, 2023; Ma et al., 2023). Quantization con- verts full full-precision values to low-bit representationss while pruning selectively eliminates insignifi- cant weights. These two compression strategies are distinct but can be synergistically combined to enhance the compression ratio (Frantar et al., 2022; Kim et al., 2023). In this paper, focus on impelling the performance of LLM pruning.
Pruning of Large Language Models. Pruning methods for neural networks can be broadly classi- fied into structured pruning (Ma et al.. 2023; Huang et al., 2020) and unstructured pruning (Frantar al.. 2022; Sun al., 2023; Zhang al., 2023; 2022b). Conventional techniques such as those in (Huang et al., 2020; Zhang et al., 2023) are ill-suited for LLMs due to their reliance on extensive retraining, challenge amplified within the context of L.LMs. In contrast, L.IM-specifie pruning methods emphasize data and time efficiency. Regarding structured pruning, L.MMpruner (Ma et al., 2023) delves into the structured pruning of L.LMs and employs LoRA to recuperate the performance of pruned models. In the realm of unstructured pruning. SparseGPT (Frantar & Alistarh, 2023) in- troduces an efficient technique for estimating the Hessian matrix, thereby adapting the traditional OBS approach (Hassibi et al., 1993) to large-scale models. Furthermore, Wanda (Sun et al., 2023) adopts straightforward strategy, eliminating weights based on the product of weight and activation values. Those methods prune LLMSs using layer-wise strategy and employ consistent pruning rate throughout all layers, leading to rapid error accumulation. Contrarily, our approach emphasizes block-wise pruning, coupled with a differential allocation of layer-specific sparsity, which effec- tively minimizes performance degradation.
Sparsity Allocation in Network Pruning. Several previous methods (Chen al., 2023; Kusupati et al., 2020; Evci et al., 2020)) have been proposed to achieve adaptive layer-wise sparsity. For instance, STR (Kusupati et al., 2020) ) and LATS (Chen et al., 2023)) introduce learning- approaches to determine pruning thresholds for each layer. leading to a non non-uniform sparsity distri- bution. However, directly adapting these techniques to L.LMS presents challenges, primarily due to the extensive retraining needed on vast datasets. Our approach is tailored to efficiently address this issue.
3 METHOD
This section introduces our Blockwise Efffiieer-Efficient Sparsity Allocation (BESA) framework for compressing LLMs. shown in Fig.2, our proposed BESA sequentially prunes the param- eters of one transformer block before moving on to the next under the supervision of bloek-wise reconstruction error minimization. Such a pipeline reduces the GPU memory overhead remarkably. In addition, we develop parameter-effiieent sparsity learning algorithm to optimize sparsity for each layer in a block. We introduce the proposed BESA framework the following. The overall algorithm is presented in Algorithm
3.1 BLOCK-WISS PRUNING
BESA solves the optimization problem via block-wise pruning, making it possible to prune LLM with the parameter size of 7B 180B learning in the block-wise setting, our objective becomes minimizing the reconstruction error be- tween the blockwise outputs of pruned and dense models as shown in Fig.2(a) and Fig.2(a).
single A100 GPU. To facilitate differentiable sparsity
For each transformer block, we drop the superscript 'I' for simplicity of notation. In this way, block-wise pruning can be expressed as
1 T E k(M) - à)² (1)
argmin cllckk
|F(W, X) F(W M 1E +X
Crecon
Laporne
where W and M are the set of all linear weights in self-attention and feed-forward modules and their corresponding learnable binary masks. T X, and F denote the total parameter size of the transformer block, input token, and the mapping function, respectively. k(-) returns the number of zero entries and M is the binary mask for each linear weight whose zero entry indicates that the corresponding weight is pruned, || || is the Frobenuous norm, and À is a hyperparameter.
In (q). block-wise pruning is built with a reconstruction loss reemn which minimizes the prun- ing error, and sparsity penalty loss Teperse which encourages the pruned model satisfy the
3

---

Published as a conference paper at ICLR 2024
SA Output
FFN Input
Weight +
Mark
o_.proj
jgate_proj
upop Pruning with Differentiable Sparsity
Intermediste
Attention
FRN Intermediate
candidate sparsity
down_proj
layer/row weights
19, (4.proj
proj
proj
Ru
learnable
importance
Output
Input
Forward
Backward
Dense FFN Output
Block Reconstruction
sorted weights
pruning probability
pruning
Layer-specfficc sparsity allocation
Parameter-effliiott sparsity learning
Figure 2: The pipeline of our BESA. (a) shows that BESA prunes weights in the self-attention and feed-forward networks by block reconstruction, which enables efficient and differentiable search for layer-specific pruning rates. (b) describes that weight pruning is achieved by differentiable binary masks which obtained in a pfffmeter-efficient way by taking weights' importance into modeling. Note that only a small number ratios {3u)2=1 are learnable during pruning while the original weights in the LLM are frozen.
sparsity constraint. The sparsity penalty is instantiated with lg loss, which we find works well to attain the target sparsity á for each transformer block. The block-wise pruning in Eqn.(I) sequen- tially prunes the weights of one transformer block before moving on the next. In this way, is sufficient to guarantee the global sparsity of the whole LLM. Moreover, since each linear weight maintains binary mask whose 0-1 values can optimized through gradient descent algorithm, our BESA can obtain the optimal sparsity for each linear weight.
Although BESA reduces the memory footprint overhead by block-wise pruning, it still requires learning binary masks M for all linear weights, which involves huge solution space. Instead of directly learning binary masks with massive parameters, we develop a parameter-efficient algorithm to learn layer sparsity with marginally additional learnable parameters in Sec.3.2.
3.2 CARAMETER-EPPICIENT SPARSITY LEARNING
Our BESA employs a parameter-efficient sparsity learning technique to enable weight pruning with optimal pruning rate for LLMs. Motivated by the fact that pruning unimportant weights minimizes performance degradation, we propose to remove the top-K least important weights for each layer. Note that K can be different for layers which also implies that each layer has its own optimal sparsity a" (i.e. a = K K/N where N denotes the parameter size of linear weights), considering that layers in transformer block do not contribute equally to the final performance as shown in Fig.
To optimally select the top-K least important weights for each layer, our main idea is to first sort weights with weight importance metric and then assign important (unimportant) weights with mask 1 (mask 0) in differentiable manner, as shown in Fig.
Weight Sorting. Various metrics have been proposed to measure the importance of weights of LLM. For example, SparseGPT (Frantar & Alistarh, 2023) estimates the importance of weight by the incurring error when the underlying weight is masked. Moreover, for each individual weight, Wanda (Sun et al., 2023) evaluates its importance by the product of weight magnitude and the corresponding input feature norm, which simplifies SparseGPT by avoiding calculating the Hessian inverse. Here, we directly adopt Wanda the weight importance metric sort weights.
Given layer weight W E RCi. and layer input x E SxCx where Cin, Cout and 5 are weight input dimension, weight output dimension, and input sequence length, respectively, sort the weights of each row by
6.j
WWal || W,
Sort Sort(W.,,
where Wi, is i-th row and j-th column entry W and x. of the j-th column vector of x. The weight importance 8i, takes both weight and activation magnitude into consideration. It works well
4

---

Published as a conference paper at ICLR 2024
in our BESA find the least i weights. With oa,j, we can obtain the sorted weight sequence W1 in ascending order by the Sort(.) function. We also experiment with other metrics of weight importance in Appendix Sec.A. Note that we only need to sort weights of cach row in each layer once by (qn.(2) as shown in Algorithm considering that the weight's importance is invariant to the pruning procedure.
Mask Generation. We now turn to generate differentiable masks in parameter-efftcient way. Towards this goal, we parameterize the sparsity with the learnable combination of a set of candidate pruning rates {pa]2=1 where pa P4+1 and D denotes the number of candidate pruning rates. In this way, we formulate the layer sparsity as
Σ d=1 BaPd:
E A-1 are learnable coefficients lying in simplex and Ba is the probability of layer sparsity being Pa- Note that the top-(Cout Pd) least important will be pruned if the layer sparsity is pa- Given candidate pruning rates (pa)a=1. we can derive the element-wise weight pruning probability as
where B [.,....,
P(W,)) = D 2
3d if Cout Phe 3 < Cout Pk+1
(4)
where P(W.) (.) indicates the probability that weight Wa3 is pruned. We set the boundary condition as Po = 0 and 3D = O which ensures that the most important weights are always retained. From (qn. we have P(W.)) 2 P(W.,#1) Hence, our modeling of element wise weight pruning probability explicitly encodes the fact that the less important weights have higher pruning proba- bility, which would reduce the optimization difficulty in finding unimportant weights. Given the pruning probability of each weight, the weight mask M in Eqn.(1) can be generated by
M = 0. if P(M,)) 2a, esle,
(5)
where Mr,j indicates that the weight Wes is preserved, and vice versa.
Differentiability of Sparsity a. Our modeling of binary mask M make the loss function Llock differentiable with respect to sparsity a. Note that the gradient of mask My,; with respect to pruning probability P(W.,) be estimated using Seahht--Trouuh-sttiiin
(STE). Hence, the gradient
w.r.t. a can be calculated by
Cloock
D &Colock aCblock
Cous bbblock aP(M.s)
= Pd
(6)
da
0p
033
0M.,
033
With (q). the sparsity (pruning rate) can be optimized through simple gradient descent algo- rithm for different layers.
The learnable parameters of the sparsity learning in Eqn. 5) come from the combination coefficients ( By default, we learn sparsity for each row, which results in additional D Oin parameters for each layer. However, learning sparsity on a row basis would cause unstructured pruning mask, which slows down the learning process because unstructured masks cannot be implemented in a parallel way. To address this, we have designed a customized CUDA operator to accelerate the row-wise probability pruning mask generation in this setting. We also implement a lightweight version with fewer learnable parameters by sharing the same set of combination coefficients {.2) d- which adds only D parameters for each layer. In experiments, we set D = 100. Take LLaMA-65B as an example, our BESA introduces 2. 2.10% and 0.0003% extra parameters each block for row-wise and layer- wise settings, respectively.
Parameter Efficiency.
3.3 JOINT OPTIMIZATION WITH QUANTIZATION
Pruning can save memory by only storing unpruned weights and binary masks while quantization reduces memory by saving weights in the low-bit format. Thanks to the inclusion of block-wel wise reconstruction, our BESA pruning algorithm can be jointly optimized with the weight-only quanti- zation technique. Following OmniQuant (Shao et al., 2023), we consider the Min-Max quantization scheme with learnable clipping strengths. To jointly optimize quantization and pruning parameters,
5

---

Published as a conference paper at ICLR 2024
Algorithm Overall algorithm of BESA.
Input: calibration dataset X, pre-trained LLM model (W") JT-1 and target sparsity â. Output: pruned model.
1: Initialize X, =
b init inputs of pruned model. block-wise pruning
2: for / in 21, } do:
3: calculate the full-precision output F(X, fps W
4: sort weights for all W E W by Eqn.(2),
only sort weights once for each block generate element-wise pruming mask M, with learnable ratios { by Eqn. - 5),
5: while optimal sparsity of not converge do:
6:
7:
calculate pruned output F(Xp: W @ M²))
8:
calculate block loss Cblock by Eqn.(1).
9: update learnable ratios {3a) d=1 by back-propagation,
10: end while
11: forward propagation X, = =F(X,, W M M!),
12: store the weight mask M
13:
end for
14: return pruned model (W' @ M
we first quantize the model weights and then prune the quantized weights with our BESA. This can be implemented by substituting W with its quantized version Q(W) which is expressed
W
+ 2), with h "1 max () -20 min W)
min (W)
Q(W) = clamp(L h
2 -
for all W W. In (qn. Q(W) and W denote the quantized and full precision weights, respec- tively. is the normalization factor for weights and z is the prro-point value. The clamp operation constrains the value within the range of --bit integer, specifically [0, 2 - 1] where N is the target bit number. 1-] indicates round operation. max W) and min (W) are the maximum and minimum in W, respectively. 70 [o, and 7/ [0, 1] are learnable clipping strengths for the lower and the upper bound of weights, respectively. When performing pruning and quantization simultancously, optimize the combination coefficients {Ba]a= for generating pruning masks and quantization clipping thresholds {70, 71}.
EXPERIMENTALS
In this section, we present a comprehensive series of experiments designed to evaluate the effec- tiveness of our proposed methods. We begin by providing a detailed overview of our experiment settings, encompassing the configuration of our experiments, the specific Large Language Model (LLM) model under evaluation, the benchmark dataset utilized, and the baseline method employed for comparison. Subsequently, we assess both the perplexity and the zero-shot capability of the pruned LLM models. Finally, we concurrently perform pruning and quantization, and we include series of ablation studies, which can be found in Appendix Sec.A. Additionally, we explore the real-world acceleration performance of our proposed method using a customized accelerator known ViTCoD (You et al., 2023).
4.1 EXPERIMENT SETTINGS
Setup. All pruning experiments were executed on a single NVIDIA A100 GPU equipped with 80GB of memory. Our proposed method, along with the baseline methods, was implemented using the PyTorc framework. The calibration set used consisted of 128 sequences, each comprising 2048 tokens, sampled from the first shard of the C4 training dataset, mirroring the approach adopted in the baseline methods. LLM models and datasets were sourced from the Huggingface Transformers library (Wolf et al., 2020). Zero-shot experiments were conducted with the assistance of the Lan- guage Model Evaluation Harness (LM-Eval) library (Gao et al., 2021). In this configuration, proposed method achieved full sparsity the LLaMA-65D model within a remarkable time frame of 4.5 hours.

---

Published as a conference paper at ICLR 2024
Datasets
Methods
1-7B
1-13B
1-30B
1-65B
2-7B
2-13B
2-70B
Dense
5.68
5.09
4.10
3.53
5.47
4.88
3.31
Wikitex12
SparseGPT
7.22
6.21
5.33
4.60
6,99
6.02
4.25
Wanda
7.26
6.15
5.25
4.60
6.92
5.97
4.22
BESA
6.86
5.92
5.00
4.33
6.60
5.75
4.09
Dense
7.34
6.70
6.13
5.81
7.26
6.73
5.71
C4
SparseGPT
9.31
8.12
7.33
6.66
9.23
8.22
6.45
Wanda
9.34
8.14
7.29
6.71
9.24
8.30
6.50
BESA
8.96
7.90
7.09
6.49
8.88
7.96
6.38
Dense 41.25
28.10
23.51
25,07
32.91
48.82
20.76
PTB
SparscGPT 79.25
37.24
26.33
27.93
108.71
70.87
22.67
Wanda 80.30
36.42
26.63
25.75
48.15
69.65
23.20
BESA 66.96
36.07
25,41
24.76
44.09
58.58
22.87
Table 1: Perplexity results for LLaMA models with unstructured 50% sparsity. In the table, 1- 7/13/30/65B denotes LLaMA-7/13/30065BB and 2-7/13/70B represents LLaMA2-7713/705 models. The best performing result is indicated in bold, while the second best result is shown as underlined.
Models. Our primary focus for evaluation centers the LLaMA (Touvron et al., 2023a) family of models, renowned as one of the most prominent series of Large Language Models (LLMs).. Specif- ically, assessed our methods across various model sizes, including HLLMA-7BB13B/30BB and LLaMA2-7B/13B/70E Notably, our methodology exhibits consistent improvements and is not bound by the size of the LLaMA model.
Benchmarks. Our initial assessment centers on evaluating the perplexity of pruned LLM models, a widely recognized metric renowned for its reliability and resilience in appraising LLM performance. In alignment with prior studies (Frantar & Alistarh, 2023; Sun et al., 2023), we primarily measure model perplexity using the WikiText2 (Merity, 2016), C4 (Raffel al., 2020), and PTB (Marcus et al., 1994) datasets. In addition to assessing perplexity, we undertake an exhaustive examina- tion of the zero-shot capabilities of pruned models across six standard common-sene sense benchmark datasets. These benchmarks encompass PIQA (Bisk et al., 2020), BoolQ (Clark et al., 2019), Hel- laSwag (Zellers et al., 2019), WinoGrande (Sakaguchi al., 2021), as well the ARC Easy and Challenge (Clark et al., 2018) tasks.
Baselines. We evaluate the following established methods as baselines: (i) SparseGPT, which di- vides the task of pruning LLM models into a sparse regression problem for a set of transformer blocks, subsequently solving these problems with an approximate sparse regression solver. It worth noting that GPT updates the values of unpruned weights. (ii) Wanda, method that leverages the product weight magnitude and L2 normalization input activations to determine the importance of LLM model weights, followed by pruning weights with lower importance.
4.2 PERPLEXITY EXPERIMENTS
In this experimental evaluation, we conducted a comprehensive assessment of the entire LLaMA model family. We pruned all linear layers, excluding embeddings and the model head, achieving 50% unstructured sparsity level. The perplexity scores for the pruned models the Wikitext2, C4, and PTB datasets are presented in Table The results displayed in Table demonstrate a consistent improvement in BESA when compared to existing methods. To further explore the impact of differ- ent sparsity levels, we conducted experiments with varying sparsities, and the results, measured in terms of Wikitext2 perplexity, are visualized Fig.3.
4.3 ZERO-SHOT EXPERIMENTS
In addition utilizing perplexity reliable and robust metric for assessing LLM performance, have expanded our evaluation to encompass a range of downstream prompt-based zero-shot tasks. We provide concise summary of detailed performance metrics in Table 2. When considering

---

Published as a conference paper at ICLR 2024
Models
Methods
PIQA
BoolQ
WellaSwag
Winogrande
ARC-e
ARC-c
Average
Dense
78.67
75.08
56.94
70.01
75.25
41.89
66.31
1-7B
SparseGPT
76.39
72.97
51.41
69.38
71.30
37.29
63.12
Wanda
75.41
71.04
51.95
66.14
69.36
36.95
61.81
BESA
76.66
72.17
54.31
67.64
70.79
37.20
63.13
Dense
79.16
77.89
59.93
72.69
77.36
46.42
68.91
1-13B
SparseGPT
78.35
76.85
54.88
71.35
72.47
41.98
65.98
Wanda
77.42
76.27
55.77
72.30
73.32
43.86
66.49
BESA
77.97
76.91
57.61
72.06
73.86
46.16
67.43
Densc
81.07
82.72
63.34
75.93
80.43
52.90
72.73
1-30B
SparseGPT 79.65
82,87
59.21
73.64
78.91
48.89
70.53
Wanda 79.33
81.87
60.96
73.88
79.38
50.09
70.92
BESA
79.82
83.12
62.39
75.06
79.67
49.57
71.61
Dense 81.23
84,83
64.55
77.43
81.31
52.90
73.71
1-65B
SparseGPT
80.52
85.08
62.21
77.82
79.88
50.26
72.63
Wanda BESA
80.63
85.47
62.77
77.43
80.26
50.34
72.82
80.74
85.54
64.35
77.27
81.10
53.38
73.73
Dense 78.07
77.71
57.14
68.90
76.35
43.60
66.96
2-7B
SparseGPT
76.17
76.02
52.81
68.67
71.63
36.95
63.71
Wanda BESA
76.55
75.29
52.65
67.17
72.18
38.99
63.81
76.66
74.83
54.60
68.59
73.86
40.96
64.92
Dense
79.05
80.55
60.06
72.14
79.42
48.46
69.95
2-13B
SparseGPT
77.69
81.41
55,93
71.59
74.66
42.06
67.22
Wanda
78. 62
81.04
56.97
71.51
76.26
43.26
67.94
BESA
79.11
81.68
59.19
70.80
76.64
43.26
68.45
Dense
82.21
83.79
64.77
77.90
82.70
54.44
74.30
2-70B
SparseGPT 81.56
85.05
62.23
78.30
81.65
53.33
73.69
Wanda
81.01
83.24
62.66
77.19
81.14
52.05
72.88
BESA
81.72
85.38
63.81
77.27
81.52
53.41
73.85
Table 2: LLaMA accuracies for zero-shot tasks with unstructured 50% sparsity. In the table, 1- 7/13/30/65B denotes 1LaMA-7/13/300651 and 2-7/13/70B represents ILLaMA-7/11/700 models. The best performing result is indicated bold, while the second best result shown underlined.
the average accuracy across the six tasks we examined, BESA consistently demonstrates superior performance compared to existing methods. Notably, the disparity average accuracy between pruned model and the original dense model diminishes as the model size increases. While it is important to acknowledge that the evaluation results for these prompt-based zero-shot tasks exhibit more variability compared to perplexity, BESA even achieves higher average accuracy than the original dense model LLMMA-65B.
4.4 JOINT COMPRESSION
We explore the synergy of combining both pruning and quantization techniques. Introducing sparsity into quantized models enhances their potential for achieving significant gains in terms speed and memory efficiency, thereby facilitating the deployment of L.MMs edge devices. As detailed in Sec.3.3, we have harnessed the cutting-edge OmniQuant method (Shao et al., 2023) to implement 4-bit weight-only quantization in conjunction with our pruning algorithm, employing block-wise approach. The performance of the jointly compressed models in LLaMA-7/1 3/30B and LLaMA2- 7/13B is presented Table For the sake of comparison, we have also applied post-pruning the quantized model using the Wanda method. As demonstrated in Table 3, under the joint compression framework, BESA consistently outperforms Wanda across various models and datasets.
8

---

Published as a conference paper at ICLR 2024
Wikitex12
C4
PTB
Models
Dense
Joint
Joint Wanda
Dense
Joint
Joint Wanda
Dense
Joint
Joint-Wanda Wanda
1-7B
5.68
7.00
7.44
7.34
9.16
9.64
41.25
73.14
92.79
1-13B
5.09
6.01
6.27
6.70
8.02
8.30
28.10
35,43
36,30
1-308
4.10
5.08
5.34
6.13
7.20
7.44
23.51
25.63
27.11
2-7B
5,47
6.77
7.12
7.26
9.10
9.50
32.91
49.91
53.26
2-13B
4.88
5.85
6.10
6.73
8.07
8.44
48.82
61.17
71.10
Table 3: Perplexity (ppl) Evaluation of LLaMA Family Models with Joint Compression (lower ppl indicates superior performance). In this table, Dense refers to the original dense model, Joint corresponds to the outcomes achieved through concurrent BESA-based pruning and quantization, and Joint-Wande signifies the results obtained by pruning the quantized model with Wanda.
Layer name
q.proj
k.proj
v.proj
o_proj
gate gate_proj
up_proj
down proj
Dense Runtime
4096
4096
4096
4096
10128
10128
10128
Average Runtime (SparseGPT)
2952.22
2932.0
3041.31
2950.13
7941.88
7865.81
7441 1.4
Average Runtime (Wanda)
2887.84
2871.34
3000.91
2461.59
7701.41
7670.84
7388.97
Average Runtime (BESA)
2232.31
2230.50
2720.59
2698.53
5207.53
5125.0
6850.03
BESA Sparsity
53.87%
54.54%
48.96%
47.15%
50.20%
50.36%
46.52%
BESA Speedup
1,83x
1.84x
1.51
1.52x
1.94
1.98
1,48
Table 4: Runtime (cycles) and speedup across various layer shapes in LLaMA-7B. The term "cy- cles" denotes the number of instruction cycles necessary for the ViTCoD accelerator perform the associated computational workloads.
4.5 SPEEDUP IN SIMULATION
Prior unstructured pruning techniques (Frantar & Alistarh, 2023; Sun et al., 2023) exploit a fine- grained structured sparsity scheme (e.g. n : sparsity), achieve acceleration real computing devices. The m technique can be effectively implemented on NVIDIA Ampere GPUs using the cuSPARSELI library to achieve practical speed improvements. Our BESA seeks the optimal pruning rate for each layer. which yet poses challenges in achieving the structured n m sparsity pattern. To comprehensively investigate the speedup potential of pruned Large Language Models (LLMs), we have utilized specialized neural network accelerators other than NVIDIA GPUs.
Specifically. we employ the simulator of ViTCoD accelerator (You al., 2023), assess the real- istic speedup capabilities of our proposed method. The ViTCoD accelerator incorporates denser and sparser engine, designed concurrently process computation workloads with varying levels of sparsity. This simultaneous processing enhances the utilization of Processing Elements (PEs) within the accelerator. In this work, we extend the capabilities ViTCoD to handle the computation of all sparse matrix multiplications within pruned transformer block. We provide more details about the configurations of ViTCoD in Appendix Sec.B.
Given that sparsity significantly influences the runtime of computation, and considering that BESA prunes the model with imbalanced layer sparsity within each transformer block, we calculate the average simulated runtime across all transformer blocks within L.aMA-7B. Detailed speedup values for each pruned layer within the transformer block, along with their corresponding average sparsity, are provided in Table 4, accompanied with the simulated runtime of the model pruned by SparseGPT and Wanda for comparison.
CONCLUSION
In this work, we propose blockwise parameter-efficien sparsity allocation (BESA), which is com- prehensive framework to jointly prune and quantize large language models (LLM). We find that layer- wise pruning error minimization adopted by previous arts does not effectively mitigate the impact of pruning the model's output because the pruning error would accumulate layer by layer. By contrast, our PESA operates under blockwise pruning framework. By minimizing block- wise error and optimizing sparsity rates across layers, BESA is able prune various LLMs such as LLaMA1, and LLaMA2. Our experiments show that BESA achieves state-of-ther of-the---the-n artt performance, with moderate performance drop compared with the unpruned one.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This paper is partially supported by the National Key R&D Program of China No. No.2022200011000 and the General Research Fund of Hong Kong No. 1720022..
REFERENCES
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint 3r8ivv1308.3432 2013.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Yanqi Chen, Zhengyu Ma, Wei Fang, Xiawu Zheng, Zhaofei Yu, and Yonghong Tian. A unified framework for soft threshold pruning. arXiv preprint arXiv:2302.13011 2023.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018.
arXiv preprint
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 O: 8-bit matrix multiplication for transformers at scale. arXiv preprint (xXiv:2208.07339 2022.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943-2952. PMLR, 2020.
Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. arXiv preprint arXiv:2301.0077 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210. 17323, 2022.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. framework for few-shot lan- guage model evaluation, September 2021. URL https: (tocl.orr//10..21/eeeoo. 5371628.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In JEEE international conference on neural networks, pp. 293-299. IEEE, 1993.
Zhongzhan Huang, Wenqi Shao, Xinjiang Wang, Liang Lin, and Ping Luo. Convolution- weight-distrriution assumption: Rethinking the criteria of channel pruning. arXiv preprint a00iv:2004. I 11627, 2020.
Minsoo Kang and Bohyung Han. Operation-aware soft channel pruning using differentiable masks. In International Conference on Machine Learning, pp. 5122-51211111 5131. PMLR, 2020.
Schoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-spars. quantization. arXiv preprint arXiv:2306. 02..22066.769, 07629, 2023.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference Machine Learning, pp. 5544-5555 PMLR, 2020.
10

---

Published as a conference paper at ICLR 2024
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq:
Activation-awart weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.00097 00978, 2023.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. arXiv preprint arXiv:2305.11621) 2023.
Mitch Mareus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. The penn treebank: Annotating predicate argument structure. In Human Language Technology: Proceedings of a Workshop held Plainsboro, New Jersey,
March 8-11, 1994, 1994.
Stephen Merity. The wikitext long term dependency language modeling dataset. Salexforce Meta- mind, 9, 2016.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon IIm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.0/ 16, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555) 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande; An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Ommidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137 2023.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. / 1695, 2023.
InterLMM Team. Internlm: A multilingual language model with progressively enhanced capabilities, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2rX1v:2302.13971,,000 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 2.00:22300.00288,, 2023b.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierrie Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. As- sociation for Computational Linguistics. URL https: /wwww/www../////////// aclwebe org/ a antholgyy/ 2020 wwwee-dlllddeeseee 6.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprini 2rXiv:2304.12244 2023.
Haoran You, Zhanyi Sun, Huihong Shi, Zhongzhi Yu, Yang Zhao Yongan Zhang, Chaojian Li, Baopu Li, and Yingyan Lin. Vitcod: Vision transformer acceleration via dedicated algorithm and accelerator co-design. In The 29th IEEE International Symposium on High-Performan Computer Architecture (HPCA-29), 2023.
11

---

Published as a conference paper at ICLR 2024
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint arXiv:195..07830, 2019.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414, 2022.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01066; 2022a.
Yuxin Zhang, Minghao Lin, Zhihang Lin, Yiting Luo, Ke Li, Fei Chao, Yongjian Wu, and Ron- grong Ji. Learning best combination for efficient n: sparsity. Advances in Neural Information Processing Systems, 35:941-9553 2022b.
Yuxin Zhang, Mingbao Lin, Yunshan Zhong, Fei Chao, and Rongrong Ji. Lottery jackpots exist in pre-trained models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
A ABLATION STUDIES
In this section, we conduct ablation studies to comprchensively investigate the performance scala- bility of our method. We delve into how different pruning configurations influence the performance of our pruned model. To expedite experimentation and obtain results more efficiently, we focus on the LLaMA-7B model with unstructured sparsity level of 50% in the following trials.
Calibration Size. Our initial investigation centers on assessing how the performance of our prun- ing methods varies with different sizes of calibration data. The results, measured by Wikitex12 perplexity, are presented graphically in Fig.4. Notably, BESA demonstrates the ability to achieve satisfactory results even with limited number of calibration samples. With fewer than 64 calibra- tion samples, increasing the calibration dataset size leads to significant improvement; however, this improvement tapers off rapidly after reaching 64 calibration samples. For example, increasing the number of calibration samples from 128 to 256 only results in marginal decrease of 0.02 in Wikitext2 perplexity.
10.0
SparseGPT
SparsePTT
10
Wanda
Wanda
BESA
BESA
9.0
Perplexity
Perplexity
8.5
8.0
7.5
7.0
0.1
0.2
0.3
0.4
0.5
0.6
16
64
128
256
Sparsity
# Calibration Samples
Figure 3: Model sparsity ablation
Figure 4: Calibration size ablation
Epochs and Sparsity Steps. Next, we investigate the influence of training epochs and sparsity steps on our pruning methodology. Detailed performance results are presented in Table 5. Given that the calibration data is drawn from the C4 dataset, we observe that the C4 perplexity of the pruned model decreases with an increasing number of training epochs. However, it is noteworthy that this trend not consistently mirrored the Wikitext2 and PTB datasets, suggesting that a larger number of training epochs may not necessarily yield superior pruned model. Consequently, adopt epoch of training as our default setting, as consistently produces pruned models with satisfactory perplexity across various datasets.
Then, we explore the significance of sparsity steps, which determine the sparsity candidates used in method. For example, sparsity step of 0.01 implies sparsity candidates ranging from 1.0 to 0.0 with a step size 0.01. In Table 5, we compare the performance of three different sparsity steps: 0.1, 0.01, and 0.001. Notably, the C4 perplexity of the pruned model ceases improve beyond a sparsity step of 0.01, prompting us to select as our default sparsity step. Despite the seemingly better Wikitext2 perplexity associated with 0.1 sparsity step, opt for 0.01 for two primary reasons: i) Training with a 0.1 sparsity step requires more manual tuning to achieve model convergence the target sparsity level. ii) Given that the calibration is drawn from the C4 dataset and training with 0.1 sparsity step results in higher C4 perplexity, it performs less favorably than other options block-wise reconstruction.
Table 5: Ablation across epochs (left), sparsity steps (middle), and importance metrics (right).
Epochs
Spursity Step
0.001
Metri
Weight
Wanda
SparseGiPT
Wikites22
6.86
6.85
6.84
6.86
Wikitext2
6.84
6.86
6.86
Wikiiext2
7.43
6.86
8.73
C4
8.96
8.95
8.95
8.94
C4
8.98
8.96
8.96
C4
9.81
8.96
11.32
PTB
66.96
67.37
66.83
67.09
PTB
69.29
66.96
83.60
66.96
140.60
13

---

Published as a conference paper at ICLR 2024
Learning Granularity. Another pivotal facet of our method concerns the selection of learning granularity. As depicted in Fig. the contribution of weights to the final performance exhibits sig- nificant variations across different layers. Previous methodologies (Frantar & Alistarh, 2023; Sun et al., 2023) have consistently applied a uniform pruning rate to all layers, thus introducing the peril of eliminating critical weights and detrimentally impacting overall performance. Consequently, conducted an ablation study to meticulously assess the influence of learning granularity, as suc- einctly summarized in Table 6. We rigorously explored three distinct choices for learning granular- ity: Attn-MLP, block, and two blocks. Within the Attn-MLP setting, we permitted the sparsity of the Attention module and MLP module to converge to the target sparsity without imposing constraints on the individual layers within these modules. Analogously, the choices of block and two blocks followed similar principles.
Drawing upon the insights derived from Table 6, it becomes evident that larger learning granularity holds greater potential for preserving essential weights within the model, thereby leading to demon- strable performance enhancements, Furthermore,
delved into in-depth investigation of the reconstruction errors associated with each block, corresponding to different learning granularities. These results are meticulously presented in Fig.5. Considering the combined improvements ob- served in perplexity, the reduction in reconstruction error, and the memory requirements associated with various learning granularities, we judiciously select the block as default learning granular- ity.
Layer (Wenda)
Table 6: Learning granularity ablation.
Att--MLP
Recesstreeba crear
2.5
2.0
Granularity
Wikitext2
C4
PTB
1.5
Layer (Wanda)
7.26
9.30
80.30
Aun-MMP
6.97
9.14
70.70
0.5
Block (BESA)
6.86
8.96
66.96
Two Blocks
6.80
8.85
62.55
10
15
20
25
Block Index
Figure 5: Reconstruction error for learning granularities.
Importance Metric. Finally, we ascertain the most suitable importance metric for our proposed method. Given that our method learns pruning probabilities based on sorted weights, we explore the impact of various importance metrics, with the results presented in Table 5. Notably, our method ex- hibits sensitivity to the choice of importance metrics. Consequently, we opt for Wanda's importance metric, which consistently delivers superior performance.
B THE COMPUTATION MECHANISM OF VITCOD ACCELERATOR
As previously mentioned in Sec.4.5, our evaluation incorporates the ViTCoD accelerator (You et al., 2023) to assess the practical speedup capabilities of our proposed method. When considering the pruned weight matrix as a sparse matrix and the input activations as a dense matrix, the matrix multiplication performed in our pruned model can be characterized as a form of sparse-dense ma- trix multiplication (SpMM), as illustrated in Fig.6. ViTCoD addresses the challenges of SpMM computation, as depicted in 7 This approach employs an -utput-stationary dataflow to reduce on-chip buffer requirements and minimize the need for frequent input matrix loading. Specifically, it spatially tiles both the sparse and dense matrices along the dimension shown in Fig.6 and ac- cumulates partial sums along the dimension illustrated in the same figure. During computation, viTCoD initially transfers the tiled blocks of input matrices to memory buffers and subsequently assigns computation tasks to either the Denser or Sparser Engine based on the sparsity of the tiled matrix columns. The partial sums computed in the Denser Engine are then transferred to the Sparser Engine and accumulated within the Sparser Engine's accumulator. This tiling and computation map- ping strategy efficiently reuses input matrices and requires only small on on-chip buffer for storing calculated outputs. Furthermore, distributing computation workloads to the Denser and Sparser Engines enhances the utilization of Processing Elements (PEs).
14

---

Published as a conference paper at ICLR 2024
Temporal
Temporal
Spatial
->
Spatial
Spatial
Figure 6: Illustration of Sparse-Dense Matrix Multiplication (SpMM). The leftmost matrix repre- sents the sparse matrix, with zero values denoted by white blocks. The red dashed box highlights ViTCoD's tiling strategy for SpMM computation, while the spatial/temporal arrows illustrate ViT- CoD's computation mapping strategy.
Denser Engine
Sparser
Accumblaan
Engine
Input Buffer
Weight Buffer
Output Buffer
DRAM
Figure 7: ViTCoD's mechanism for addressing SpMM computation.
15


Reviews:
[{"review_id": "GXhzGGYe9J", "reviewer": "Reviewer_VXVS", "summary": "This paper studies the weight pruning problem of large language models. In order to solve the problems of significant output disturbance and the need for careful hyperparameters tuning in existing layer-wise methods, a novel LLM pruning technique named block-wise parameter-efficient sparsity allocation (BESA) is proposed, with two distinctive characters: minimizing pruning error for individual blocks and ensuring the layer-specific sparsity differentiable. Finally, this paper verified the performance and efficiency of the method through detailed experiments on strong baselines.", "questions": "1)Considering the abstract mentions existing methods require meticulous hyperparameter tuning, it would be better to study the vital hyperparameter’s sensitivity of the proposed method and experiments are needed, which will make this paper more convincing. \n2) Due to the uniqueness of the proposed method BESA, which seeks the optimal pruning rate for each layer, compared to existing methods, will implementing specialized neural network accelerators (ViTCoD in the experiments) consume additional time or make model faster than others?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This paper introduces a novel approach, BESA, for compressing Large Language Models through block-wise pruning with a differentiable sparsity allocation, which maintains the performance of LLMs well and improves computational efficiency compared to existing methods. Besides, this paper is well written, clearly explaining the method of block-wise tuning and parameter-efficient sparsity learning through detailed mathematical and textual expression. Finally, credible experimental design and solid experimental results and increase the credibility of the paper.", "weaknesses": "There may be a few things that need to be modified or clarified clearly. In page 3 BLOCK-WISE PRUNING equation (1), it is better to add the meaning of “W” together with “M, F, X, …”; Since the article mentioned that existing methods require meticulous hyperparameter tuning, adding the sensitivity of some vital hyperparameters of the proposed model and clarify the advantage in the appendix will make this paper more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8WrzSh3HGI", "reviewer": "Reviewer_qTC9", "summary": "The paper presents a novel pruning technique called Blockwise Parameter-Efficient Sparsity Allocation (BESA) for compressing large language models (LLMs). BESA aims to address the computational footprint and memory consumption issues associated with LLMs by optimizing pruning rates across different layers in a differentiable manner. The proposed method achieves state-of-the-art performance in pruning various LLMs, such as LLaMA1 and LLaMA2, and efficiently prunes them on a single A100 GPU.", "questions": "NA", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- BESA is the first differentiable pruning algorithm for LLMs, which allows for efficient optimization of pruning rates across layers.\n- The method is parameter-efficient and easy to optimize, exhibiting high efficiency and effectiveness in pruning various LLMs.\n- BESA achieves state-of-the-art performance in pruning various LLMs, such as LLaMA1 and LLaMA2, with reduced performance degradation after pruning.\n- The proposed method can be jointly optimized with weight-only quantization techniques, further enhancing the compression ratio and efficiency of LLMs.", "weaknesses": "- The paper does not provide a detailed analysis of the trade-offs between different pruning rates and their impact on model performance, which could be useful for understanding the optimal pruning strategy.\n- The paper does not provide a comprehensive comparison of BESA with other pruning techniques, such as structured pruning, which could help in understanding the relative strengths and weaknesses of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bOgTCPRYKl", "reviewer": "Reviewer_JpTs", "summary": "The authors propose a blockwise model pruning framework for compressing LLMs, which searches for optimal pruning rates for each layer in a differentiable manner. The authors have done experiments on different models and datasets.", "questions": "The model seems not significantly better than SparseGPT with a smaller number of parameters. Could you have significant test?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors demonstrate the practical speedup of the pruned model in a hardware simulator.\n2. The method is parameter-efficient and easy to optimize.\n3. The authors have done extensive experiments on language modeling and few-shot learning benchmark datasets. The authors also explore models with different numbers of parameters from 7b to 70b.", "weaknesses": "1. It would be better to involve the computational cost of attention weight in Table 4.\n2. It would be better to have an ablation study of block-wise pruning. Maybe directly pruning all the models instead of layer by layer.\n3. There has been a lot of research on pruning in CV and NLP. The baselines are far from complete, such as \"The Lottery Ticket Hypothesis\" and its following works. While considering not much work on LLM, I would not penalize too much on it.\n\nOverall, I think the experiments are solid. The novelty is ok, but it would be better to explore more classical pruning methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1C74UDZ2SM", "reviewer": "Reviewer_MvCd", "summary": "The authors proposed a novel method for blockwise pruning of LLMs. It was evaluated with generation tasks, as well as for zero-shot downstream tasks, and outperformed SparseGPT and Wanda baselines.", "questions": "Please refer to the weaknesses Section.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed method outperformed recent baselines.\n- The motivation for this research is clear, and the proposed method is helpful for practitioners.", "weaknesses": "Authors claimed BESA has different advantages compared to other baselines. E.g., the fact that BESA differentiably optimizes masks, unlike SparseGPT. However, it is not clear whether other methods could or could not use such specific techniques and what makes BESA better than them. The current ablation study does not answer these questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)