LLM-Pruner: On the Structural Pruning of Large Language Models
Authors: Xinyin Ma
Gongfan Fang
Xinchao Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: model compression
structural pruning
large language model

Abstract: Large language models (LLMs) have shown remarkable capabilities in language understanding and generation. However, such impressive capability typically comes with a substantial model size, which presents significant challenges in both the deployment, inference, and training stages. With LLM being a general-purpose task solver, we explore its compression in a task-agnostic manner, which aims to preserve the multi-task solving and language generation ability of the original LLM. One challenge to achieving this is the enormous size of the training corpus of LLM, which makes both data transfer and model post-training over-burdensome. Thus, we tackle the compression of LLMs within the bound of two constraints: being task-agnostic and minimizing the reliance on the original training dataset. Our method, named LLM-pruner, adopts structural pruning that selectively removes non-critical coupled structures based on gradient information, maximally preserving the majority of the LLM's functionality. To this end, the performance of pruned models can be efficiently recovered through tuning techniques, LoRA, in merely 3 hours, requiring only 50K data. We validate the LLM-Pruner on three LLMs, including LLaMA, Vicuna, and ChatGLM, and demonstrate that the compressed models still exhibit satisfactory capabilities in zero-shot classification and generation. The code will be made public.

LLM-Pruner: On the Structural Pruning of Large Language Models
Xinyin Ma Gongfan Fang Xinchao Wang"
National University of Singapore
maxinyinou. nus edu, gongfanOu.. mus edu, xinchaodnus cecmusseddie .dd.
Abstract
Large language models (LLMs) have shown remarkable capabilities in language un- derstanding and generation. However, such impressive capability typically comes with substantial model size. which presents significant challenges in both the deployment, inference, and training stages. With LLM being a general-purpos task solver, explore its compression in task-agnostic manner, which aims preserve the multi-task solving and language generation ability of the original L.M.. One challenge achieving this is the enormous size of the training corpus of LLM,, which makes both data transfer and model post-training wwer-burdensome. Thus, we tackle the compression of LLMs within the bound of two constraints: being task- agnostic and minimizing the reliance on the original training dataset. Our method, named LLM-Pruner, adopts structural pruning that selectively removes non-critical coupled structures based on gradient information, maximally preserving the ma- jority of the LLM's functionality. To this end, the performance pruned models can be efficiently recovered through tuning techniques. LoRA, in merely hours, requiring only 50K data. We validate the ILaMA, Vicuna, and ChatGLM,, and demonstrate that the compressed models still exhibit satisfactory capabilities in zero-shot classification and generation. The code is available at: https: thub........ com www.coo.////.ccoooo hwt.://oooeeeeeeeee PLM-Prrre
on three LLMs, including
Introduction
64 4 74 have demonstrated remarkable proficiency in language understanding and generation. With the increase in model size, they are better equipped to handle complex tasks 3 5 However, notwithstanding their impressive performance, LLMs pose challenges in deployment and inference. Their extensive scale engenders substantial computational demands, and the multitude of parameters involved induce long latencies and other related issues. Several techniques
Recently, Large Language Models (LLMs) 39 51
58 60) and even exhibit emergent abilities [57].
proposed to solve these problems, like model pruning [56] 61 72 21, knowledge distillation [6 47 quaantizzaii
1 within the context of pre-trained language model (PLM).
While previous methods have effectively maintained model performance amidst parameter reduction, they primarily target compression within specialized domains for designated tasks in the context of task-specific compression. For instance, PLM fine-tuned on a particular dataset, such as one of the classification tasks in the GLUE benehmark (53), after which these models are distilled into a smaller classification model [46] [8]. Although this paradigm could potentially be employed for LLM compression, it compromises the LLM's capacity versatile task solver, rendering it suited to single task exclusively.
" Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Compression
Evaluation QA
3
(i) Task Specific Compression
Dataset
Task Dataset
Fine-tune
LLM
-20GB Corpus
Task Dataset
(ii) TinyBRRT
3.5 days GPUs)
lask-specific model
-50B Corpus
(iii) L.LM-Pruner
3 hour (1 GPU)
C LLB by LLM-Pruner
LLaMA-7B
The Leaning Tower of Pisa is known for its unusual tilt, which result a number factors. When tower was built in twelfth century, the soil beneath it was extremely soft, allowing the buttresses to settle unevenly. This resulted in tilt towards one side.
The Leaning Tower of Pisa is known for being tilted and unstable. However, its story is much more fascinating. Although construction hegan 1173, the tower was never meant tilted. simply became that way because was built on unstable ground.
Figure I: Illustration of LLM-Pruner. (i) Task-specific compression: the model is fine-tuned then compressed on specific task. (ii) TinyBERT: First distill the model on unlabeled corpus and then fine-tune it on the specific task. (iii) LLM-Pruner: Task-agnostic compression within 3 hours.
Thus, we strive to compress the LLM new setting: to reduce the LLM's size while preserving its diverse capabilities general-purpose task solvers, depicted in Figure T This introduces the task-agnostic compression of LLMs, which presents two key challenges:
The size the training corpus of the LLM enormous. Previous compression methods heavily depend on the training corpus. The LLM has escalated the corpus scale to trillion tokens or more [T]. The extensive storage needs and protracted transmission times make the dataset difficult acquire. Furthermore, if the dataset is proprictary. acquisition of the training corpus verges impossibility, a situation encountered in 774 39].
The unacceptably long duration for the post-training of the pruned LLM. Existing methods
require substantial amount of time for post-traing pest-training the smaller model 555 28 For instance, the general distillation BERT takes around 14 GPU days (20] Even post-training a task-specific
compressed model of BERT demands around 33 hours [6] [2] As the size of both the model and
corpus for L.MS increases rapidly, this step will invariably consume an even more extensive time.
To tackle the aforementioned challenges associated with the task task-agnostic compression of LLMs, we introduce novel approach called L.LM-Pruner. Since our goal is to compress LLMS with reduced data dependency and expedited post-training. how to prune model with the minimal disruption the origin is crucial. To accomplish this, propose a dependency detection algorithm that identifies all the dependent structures within the model. Once the coupled structure is identified, we employ an efficient importance estimation strategy to select the optimal group for pruning under the task-agnostic setting, where the first-order information and an approximated hessian information is taken into account. Finally, rapid recovery stage is executed to post-train the pruned model with limited data.
Contribution. In this paper, we propose novel framework, LLM-Pruner, for the task-agnostic compression of the large language model. To the best of our knowledge, LLM-Pruner is the first framework designed for structured pruning of LLMMs. We conclude the advantages of the LLM- Pruner as (i) Task-agnostic compression, where the compressed language model retains its ability to serve multi-task solver. (ii) Reduced demand for the original training corpus, where only 50k publicly available samples are needed for compression, significantly reducing the budget for acquiring the training data (iii) Quick compression, where the compression process ends up in three hours. (iv) An automatic structural pruning framework, where all the dependent structures are grouped without the need for any manual design. To evaluate the effectiveness of extensive experiments on three large language models: LLaMA-7B, Vicuna-7B. 7B, and ChatGLM-6B, The compressed models are evaluated using nine datasets to assess both the generation quality and the zero-shot classification performance of the pruned models. The experimental results demonstrate that even with the removal of 20% of the parameters, the pruned model maintains 94.97% of the performance of the original model.
we conduct

---

2
Related Work
Compression Language Model. Language models |1 have gained much attention and
increase the need to reduce the size of parameters and reduce the latency [23 48 To compress the
language model, previous works can be divided into several categories: network pruning [21 63 32
15
knowledge distillation [46 47 40]. quantization [68 71 and other techniques, like early exit
62 or dynamic token reduction [69 We focus on the pruning of the language models, especially
structural pruning (26). Structural pruning removes the entire filter from the neural network, which is
more hardware friendly. There are several ways to remove the structure, such as 11 -deeenddent pruning
[6 72], first-order importance estimation [88, bessian-basee estimation [2] [54] or the optimal brain surgeon (24] 21] As for the pruning unit in structural pruning, some works adopt the entire layer
10
as the minimal unit, and others take the multi- head attention [52] the feed-forward layers
18
36 as the basic structure to prune. CoFi 61 studies the pruning unit in different granularity.
Efficient and Low Resource Compression. With the growing size of neural network models, there is an increasing demand for efficient and low-resource compression [67 66 30 29 65). As for the efficient compression, [22] accelerate the post-training by defining the reconstruction error as a linear propose the layer-wise optimal brain surgeon. As for the constraint of availability of the training corpus, data- free pruning [45] 70 come up with several strategies to prune the model by measuring neurons' similarity. Besides, [34
least squares problem. [33 [2]
proposes methods that distill the model without reliance on the training corpus of the model. However, those methods are too consccon,,,ing, involving synthesizing samples by backpropagating the pre-trained models.
33 42
3 Methods
In this section, we provide a detailed explanation of LLM-Pruner. Following the conventional model compression pipeline [22), LLM- Mrrrrerrr consists of three steps: (1) Discovery Stage (Section 3.1). This step focuses on identifying groups of interdependent structures within LLMs. (2) Estimation Stage (Section 3.2). Once the coupled structures are grouped, the second step entails estimating the contribution of each group to the overall performance of the model and deciding which group to be pruned. (3) Recover Stage (Section 3.3 This step involves fast post-training that alleviates potential performance degradation caused by the removal of structures.
3.1 Discover All Coupled Structure in LLMs
In light of the limited availability of data for post-training. it becomes imperative to prioritize the removal of structures with minimal damage when compressing the model. This underscores the dependency-baie structural pruning. which ensures coupled structures are pruned in unison. We provide experiment in Section 4.3 to show the importance of dependency-basee structural pruning when compressing the large language model.
Structure Dependency in LLMs. Similar [1], the pruning begins by building the dependency for L.Ms.. Assume N, and N, are two neurons in the model, In(N,) and Out(N,) represents all the neurons that point towards point from N₁. The dependency between structures can be defined as:
N₃ Out(N.) A Deg (N₂) 1 N, is dependent on N;
(1)
where Deg (N₂) represents the in-degree of neuron N,. Noting that this dependency is directional, we can therefore correspondingly obtain another dependency:
N E In(N,) A Degt = 1 + N₂ is dependent on Nj
where Deg" + (N,) represents the out-degree of neuron N;. The principle of dependency here is, if current neuron (e.g., N₁) depends solely on another neuron (e.g.. N2), and the neuron N, is subjected to pruning, follows that the neuron N, must also undergo pruning. We provide a detailed case in Appendix
Trigger the Dependency Graph. By having the definition of dependency, the coupled structures in the LLM can be analyzed automatically. Considering any neuron within the LLM as the initial
3

---

Group Type A: MLP
Group Type B: Multi-head Attention
Group Type C: Channel-wise Grouping
Projection
Normi
Norm
Head:
MLP
Gate Projection
Up Projection
Head
Lx
Norm:
HEME
Hoidn
Norm
MIE:
QKV
Multi-head
Attention
Embest
dos::
Query
Key
Value
Embedding
Figure 2: Illustration of the coupled structures in LLaMA. We simplify the neurons in each layer to make the dependent group clear. The trigger neuron, marked as a circle with a bell, cause weights with dependency pruned (dashed lines), which may propagate (red dashed lines) to coupled neurons (dashed circles). A group can be triggered by variety of trigger neurons. Taking Group Type B as an example, the trigger for this group involves (i) the attention head, (ii) the output neuron in Query, Key or Value, and (iii) the input neuron in the final output projection.
trigger, it possesses the capability activate neurons that depend on it. Subsequently, these newly triggered neurons can serve as the subsequent triggers to identify the dependency and activate their respective dependent neurons. This iterative process continues until no new neurons are detected. Those neurons then form group for further pruning. Taking LLaMA as an example, by searching over all the neurons as the initial trigger, we can locate all the coupled structures, as shown in Figur
Given the diversity in the structure of different LLMs, manual analysis and removal of coupled structures in each LLM could be extremely consumnns.,.g. However, by employing LLM-Prunee all coupled structures can be automatically identified and extracted.
3.2 Grouped Importance Estimation of Coupled Structure
Till now, all coupled structures within the model are grouped. Weights within the same group should be pruned simultaneously, as partial pruning not only increases parameter size but also introduces misaligned intermediate representations. Therefore, estimate the importance of the group whole, as opposed to evaluating the importance modules. Given the limited access to the training dataset, we explore the use of public datasets manually created samples as alternative resources. Although the domains these datasets may not perfectly align with the training set, they still provide valuable information for assessing the importance.
Vector-wise Importance. Suppose that given a dataset D = (wi, ye J where N is the number of samples. In our experiments, we set N equal to 10 and we use some public datasets the source of D. A group (as previously defined as a set of coupled structures) can be defined as G = [W; W.M where M is the number of coupled structures in group and W; is the weight for each structure. While pruning, our goal to remove the group that has the least impact on the model's prediction, which can be indicated by the deviation in the loss. Specially, to estimate the importance of W., the change in loss can be formulated as [24]:
7L (D) OW; W. 2 W. HW, + O (IWW)))
Tw. |AC(D) Lw: (D)
(3)
where H is the hessian matrix. Here, L represents the next-token prediction loss. The first term is typically neglected prior work [24] 54 [2], as the model has already converged on the training dataset, where OL /aW; 0. However, since D here is not extracted from the original training data, which means that OL TCT/OW, /aW, & 0. This presents desirable property for determining the importance of W. by the gradient term under LLMs,, since computation of the second term, the Hessian matrix, on the LLM is impractical with O (N2) complexity.

---

Element-wise Importance. The above can be considered as estimate for the weight W,. We can derive another measure of importance finer granularity, where each parameter within W; is assessed for its significance:
(L(D) ow? W
w = |C(D) = 1Cw (D) Lw:==(D)
- Hu. +0 (IIW)1") (4)
W
Here, represents the k-th parameter W. The diagonal of the hessian HA can be approximated by the Fisher information matrix, and the importance can be defined as:
Tw 1c (D) L (D)
AC(D) OW? 2
W
1
Σ
OC(D,)
2 OW! W + (1WW11)) (5)
Group Importance. By utilizing either ls or Iw., we estimate the importance at the granu- larity of either parameter or vector of weight. Remembering that our goal is to estimate the importance of G, aggregate the importance scores in four ways: (i) Summation: 1G = Di List Iw, or I Ei= Σx /w., (ii) Production: I II Iw; or I II-" Σ. TwY. (iii) Max: lg max:=1 /w; Is max Σx :w: (iv) Last-Only: Since deleting the last executing structure in dependency group is equivalent to erasing all the computed results within that group, we assign the importance of the last executing structure as the importance the group: lg = /w; or lg = En Aw., where / is the last structure. After assessing the importance of each group, we rank the importance of each group and prune the groups with lower importance based predefined pruning ratio.
3.3 Fast Recovery with Low-rank Approximation
In order to expedite the model recovery process and improve its efficiency under limited data, is crucial to minimize the number of parameters that need optimization during the recovery phase. To facilitate this, we employ the low-rank approximation, LoRA[19), to post-train the pruned model. Each learnable weight matrix in the model, denoted W, encompassing both pruned and unpruned linear projection in the LLM, can be represented as W. The update value AW for can be decomposed as AW = PQ E Rd xat where P E IRd xd and Q E Rdxa The forward computation can now be expressed as:
f(x) = (W + AW)X+++- (WX (PQ)X
where b is the bias in the dense layer. Only training P and Q reduces the overall training complexity, reducing the need for large-scale training data. Besides, the extra parameters /' and Q can be reparameterized into AW, which would not cause extra parameters in the final compressed model.
4 Experiments
4.1
Experimental Settings
Foundation Large Language Model. To showcase the effectiveness and versatility of LLM-Pruner, we test it over three open-source large language models with two kinds of structure: LLaMA-7B [1]. Vicuna-7B and ChatGLM-6B 774].
Evaluation and Datasets. To assess the performance of the model in the task-agnostic setting, we follow LLaMA's evaluation to perform zero-shot task classification on common sense reasoning datasets: BoolQ (6), PIQA (2), HellaSwag (73). WinoGrande (43) ARC-easy 77. ARC-challenge 7 and OpenbookQA (88 Follow [14, the model ranks the choices in the multiple choice tasks or
generates the answer in the open-ended generation [ Additionally, we complement our evaluation
with zero-shot perplexity (PPL) analysis on WikiTexx2 [37] and PTB [35].
hhaps://huggingffcccco///m hhtp://wwwww....ooo///a/a/a/eeee
5

---

Table 1: Zero-shot performance of the compressed LLaMA-7B. The average is calculated among seven classification datasets. 'Underline' indicates the best pruning- cruning-only performance, while "bold" represents the overall best performance with the same pruning ratio, considering both pruning and post-training. The 'Channel' strategy only prunes the dependent group of Type C, while all other methods employ the 'Block' strategy to prune dependent groups in both Type A and Type B. Since
[51] did not provide its prompt, the evaluation of the result with is performed under different
prompts, which is lower than the official results.
Pruning Ratio
Mothod
WikiTexii)
PTB.
BooiQ
PIQA
HelloSwag
WinoGrande
ARC-c
ARC-2
OBQA
Average
LLaMA-7BB5 7B0
76.5
79.8
76.1
70.1
72.8
47.5
57.2
68.59
1aMM-7B8
22.14
73.18
78.38
72.99
67.01
67.45
41.38
42,40
63.25
L2
582.11
1022.17
59.66
58.00
37.04
52.41
33.12
28.58
29.80
42.65
Random
27.51
43.19
61.83
71.33
56.26
54,46
57.07
32.85
35,00
52,69
Ratio = 20%
74.63
153.75
62.75
62.73
41.40
27.90
30.40
45.38
w/o
Vodiar
22.28
41.78
61.44
71.71
57.27
54.22
55.77
33.96
38,
53.25
Hlemens
36.66
59.39
75.57
65.34
39.80
Elemont
19.09
34,21
$7.06
75.68
66.80
59.83
60.94
36.52
40.00
$6.69
Channel
22.02
38.67
59.08
73.39
64,02
60.54
57.95
35,58
38.40
55.57
Ratio= = 20% w/
Voctor
18.84
33.05
65.75
74.70
64.52
59.35
60.65
36.26
39, 39:40
57.23
Hlement
17.37
30.39
69.54
76.44
68.11
65.11
63.41
37.88
40.00
60.07
Element
17.58
30.11
64.62
77.20
68.80
63.14
64.31
36.77
39.80
59.23
Table 2: Zero-shot performance of the compressed LLaMA- 13B. Here we adopt Element* as the importance estimation for 'Channel' and 'Block',
Pruning Ratio
Meshod
WikiText2,
PTB.
BoolQ
PIQA
HellaSwag
WinaGrande
ARC-e
ARC-c
OBQA
Average
LLaMA-13B'
11.58
20.24
78.89
76.21
70.09
42.00
64.97
1.2
61.15
91.43
61.50
67.57
$2.90
57.54
50.13
31.14
36,80
$1.08
Rutio
Random
19.24
31.84
63.35
73.18
63.55 63.54
60.85
64.44
36.26
38.00
57.09
wio tune
Channel Block
106.48
33.20
50.29
16.01
29.28
67.68
77.15
73.41
65.11
58.35
38.40
42.40
61.79
L.2
20.97
38.05
73.24
76.77
71.86
64,64
67.59
39.93
40.80
62.12
Ratio m 20%
Random
16.84
31.98
64. 19
76.06
66.88
38. 31
40.80
59.78
w/ tune
Channel
17.58
29.76
69.20
76.55
68.89
66.38
62.08
38. 99
39.60
60.21
Block
15.18
28.08
70.31
77.91
75.16
67.88
71.09
42.41
43.40
64.02
Implementation Details. In the model pruning process, we use 10 randomly selected samples from Bookcorpus (75), each truncated to a sequence length of 128, as the calibration samples for establishing dependency and calculating the gradient for both LLaMA and Vicuna. For ChatGLM, select 10 random samples from DailyDialog [27]. During the recovery phase, utilize the cleaned version of Alpaca [49], which comprises approximately 50k samples. Remarkably, tuning these samples requires merely 3 hours a single GPU with only 2 epochs. More hyper-pale seper-parameters
of pruning and training can be found in Appendix B
Statistics of the Compressed Model. Table 3 presents the statistic of the 7B models that are used in our experiments: the pa- rameter count, MACs, memory re- quirements and latency for running each model. 'he statistical evalu- ation is conducted using the infer- ence mode, where the model is fed sentence consisting of 64 tokens. The latency is tested under the test set of WikiText2 on single A5000. Here, the 'Block' strategy implics that the pruned unit in the model consists of Group Type A and Group Type B as illustrated in Figure 2 whereas 'Channel' indicates that the unit to be pruned is Group Type C. We delve into an analysis of these two choices in Section 4.2 Channel Strategy vs. Block Strategy). The pruning ratio stated here denotes the approximate ratio of parameters to be pruned since the number of parameters within each pruned structure does not perfectly match the total number of pruned parameters.
Table 3: Statistics of the base model and the compressed model.
Model
Stratogy
Ratio
#Params
#MACss
Memory
Latency
6.74B
424.02G
12884,5MiB
69.32s
Channel
20%
5,39B
339.36G
10363.6MiB 6MiB
61.50s
LLaMA-7B
Block
20%
5.42B
339.60G
10375.5MiB
58.55s
Vicuna-7B
Channel
50%
3.37B
212.58G
6556.3MiB
40.11s
Block
50%
3.35B
206.59G
6533.9MiB
37.54s
4.2 Zero-shot Performance
Table 12 and 5 shows the zero-shot performance of the pruned model. Based on the evaluation conducted LLaMA, employing 20% parameter reduction without post-training, the pruned

---

Table Zero-shot performance of the compressed Vicuna-7h
Pruned Model
Method
WikiTex2 $
PTR.
BoolQ
PIQA
HellaSwag
WinoGmade
ARC-e
ARC-c
OBQA
Average
Ratio 0%
Vicuna-7B
16.11
61.37
76.57
77.75
70.64
67.40
65.11
41.21
40.80
62.78
12
3539.98
5882.21
55.90
56.15
32.37
51.85
30.01
28.41
28.20
40.41
random
34.63
112.44
61.47
70.89
54.67
56.27
55.60
31.74
34.60
52.18
Ratio 20%
Channel
71.75
198.88
51.77
63.93
42.58
55.17
29.27
33.40
45.72
weo tume
Vector
27.03
92.51
62.17
71.44
55.80
53.43
55.77
33.28
37.80
52.81
Element
24.70
62.87
25.41
64.00
58.41
37.12
39.00
56.83
Element
25.74
92.88
61.70
75.30
63.75
56.20
63.22
36.60
37.00
56.25
Vector
74.66
74.59
60.30
36.60
56.64
Ratio 20% tune
Element
18.97
76.78
60.40
75.63
65.45
63,22
63.05
37.71
39.00
57.78
Flement
19.09
78.25
63.33
76.17
65.13
60.22
62.84
37.12
39.20
57.71
model manages to retain 89.8% the performance exhibited by the unpruned model. Furthermore, through the efficient post-training, the classification accuracy further improves 60.07%, achieving 94.97% of the accuracy attained by the original model. This demonstration proves the feasibility of using LLM-Pruner to effectively compress the model, even without relying training data, and within a remarkably short period of time. Surprisingly, we discover that on most datasets, the pruned model with 5.4B LLaMA even outperformed chatGL.M-6B. This highlights the superiority of the L.LM-rruner: a smaller model with a customized size is required, LLM--Punner more cost-effective compared to retraining another model with satisfying performance. However, with 50% parameters pruned, a large accuracy degradation is observed (see Appendix C.4). Compressing LLMs under high compression rates still remains large challenge.
The compression results of Vicuna-78 7B align with those of LLaMA, as pruning 20% of parameters on Vicuna-7B maintains performance at 92.03% of the original model. We test smaller pruning rate of 10% on chatGLM-77. where the pruned model only experiences a marginal performance decrease of 0.89%. which can be recovered through post- tratntraining. Despite the pruned model outperforming the uncompressed model, we don't I assert it is better than the original model. This is largely because chatGLM-6B, bilingual model, has limited English pre-training exposure. Post-training, however, introduces to more English corpus, albeit limited, improving its English comprehension.
Ablation: Impact of Importance Estimation. We conduct tests all proposed importance estimation techniques mentioned in Section [32]] The results can be found Table and 4 Here, Element" represents the importance evaluation utilizing the n-th order term in Eq Vector represents the result corresponding to Eq Based on the results obtained from LLaMA-7B and Vicuna-7B, pruning algorithms achieved the best average performance mostly by leveraging the second-order derivatives for each parameter. Nonetheless, given that first-order derivatives are considerably more efficient than second-order derivatives, though yielding slightly inferior results, still vote for the first-order term as a competitive method. Besides, the results on chatGLM-75 differed significantly from these findings. The importance estimation on each parameter fails, performing even worse than 12, while the importance estimation on the weight matrix reaches the best performance.
Channel Strategy vs. Block Strategy. From the results presented in Table it is evident that pruning 'Channel' significantly deteriorates performance compared to pruning 'Block'. This discrepancy arises because the layers within the stacked transformer do not evenly distribute their
Ablation:
Wiiteet2
PTB
Wilkteetz Perplexity
Perple hoppycyyyy
importance, As shown in Figure 3 the first and last layers have a profound impact on the model's 's performance, and pruning them results in more sub-
stantial performance degradation compared to other layers. However, due to the uniform treatment the 'Channel' group across all layers, it becomes inevitable to prune the first and last layers, leading to significant decline in performance.
Figure 3: Layer sensitivity for Pruning: Re-
moving Groups in only one layer.
7

---

Table Zero-shot Performance of the compressed ChatGLM-6B
Pruned Model
Method
PIQA
HellaSwag
WinoGrande
ARC-c
ARC-c
OBQA
Average
Ratio = 0%
ChatGL M-66
67.95
46.37
52.33
48.36
29.95
37.40
47.05
L2
61.97
37.22
49.72
42.05
28.24
35.40
42.43
Ratio = 10%
Random
65.29
43.18
51.30
47.52
29.52
34.60
45,
w/o tune
Vector
66.32
43.51
53.04
47.56
30.72
35.80
46, 16
Element
54.35
28.07
50.59
27.82
24.66
33.20
36.45
w/ lune
Vector
67.74
46.35
53,99
51.01
29.95
35,00
47.34
Withest Oun
Wiccher? D2
Walnott Gun.
Wichwet
PTB Our
PTB
PTB Oun
P13
Wikitext2
4013
24.0
58.5
28.5
30.0
2240
78.1
2003
2005
28.2
1120
250
500
750
1000
3250
2500
1750
Pruning Ralio 1561
Pruning Ratlo 1%6
Step
Figure 4: The pruning results on LLaMA-7B (left) and Vicuna- Figure 5: Perplexity on zero-shot 7B (right) with different pruning rates.
datasets across varyhing steps.
4.3 More Analysis
Impact of Different Pruning Rates. We investigate the impact of pruning the LLM at various pruning ratios in Figure We compare our pruning results with the L2 strategy because L2 is also data-free pruning algorithm. is observed in the experiment of LLaMA that when the pruning ratio reaches approximately 20%, the magnitude-dependee algorithm experiences rapid collapse, leading to the loss of information. Conversely, by employing LLM-Pruner, the pruning ratio to around 60% while achieving an equivalent perplexity level. Furthermore, in the case of Vicuna-7B, removing 10% parameters results in a performance decline equivalent to that of LLM-Pruner with 60%. The utilization of HLM--runee enables significant increase in the number of model parameters that can be pruned, thereby substantially reducing computational overhead.
able increase
Tuning the External Dataset. To tune the pruned model. utilize the external dataset Alpaca (49] The evaluation curves the pruned model on two zero-shot datasets during the post- training process are depicted in Figure 5 The results demonstrate rapid decrease in the perplexity of the pruned model within 300 steps, followed by gradual increase. We provide a more comprehensive evaluation in Appendix C.3 It is important note that if the model is trained for an excessive number of steps, it runs the risk of overfitting the external dataset, potentially compromising its performance in other general-purpose tasks.
Impact of Dependency- Structured Pruning. To study the importance of dependency-based structural pruning. we conduct an experiment to disrupt dependencies within groups, where each weight matrix W; is pruned solely based the importance score estimated on itself. Table 6 presents the results demonstrating the impact of dependencies in structural pruning. In the absence of dependencies, the model nearly fails in the zero-shot generation and classification tasks. Even with tuning, the model fails to recover, showing a substantial difference compared to the results in rependency-basee pruning.
Impact of Different Aggregation Strategies. We conduct tests on the aggregation algorithms proposed in Section 3.2 Our experimental results unveil notable discrepancies model performance across different aggregation strategies, with particular emphasis on the 'Last-only' strategy. Among the evaluated approaches, the 'Max' strategy attains the most favorable outcomes in terms of perplex- ity, signifying enhanced coherence and fluency in sentence generation. However, it is important to note that the 'Max' strategy exhibits the poorest zero-shot classification results compared to all four strategies. Conversely, the 'Last-only' strategy showcases superior classification performance but
8

---

Table 6: Effect of the dependency-baad based struc- tural pruning. Average represents the average performance on classification datasets.
Table 7: Impact of different aggregation strate- gies on group importance estimation. Experi- ments are performed on LLaMA-7B.
Method
WikeTex12.
PTB.
Averagel
Method
WikiText2
PTB.
ARC-ct
PIQAT
OBQAT
wio Tuning
w/ dependency
dependency
68378.42
19.09
79942.47
34.21
56.09
Summation 66.13
Max
62.59
144.38
164.25
40.70
39.60
63.49
63.71
34,60
34,80
w/ Tuning
wlo dependency
dependency
13907.46
17.58
13548.08
30.11
38
Production
Last-only
130.00
77.63
192.88
170.88
41.92
37.84
64.75
62.08
35.20
35.00
suffers from the poorest generation quality. In our experiments, we make a trade-off by selecting the "Summation" strategy since shows both good generalization quality and classification performance.
Comparison with DistiBBERT We show the comparison re- sults of DistilBERT and LLM- on LLaMA-7B in Table 8 LLM-Pruner outperforms DistilBERT by 4.24% on average with even smaller size. The reason lies in that LLM-Pruner minimizes model disruption during pruning, whereas Distil- BERT merely selects one layer out of two. As result, the model pruned by LLM-Pruner demands less data to recover its performance compared with DistilBERT, consequently achiev- ing superior performance.
Table DistilBert vs. The average here means the average score on the above seven datasets.
Pruning Ratio
#Param
Average
DistilBert LLM--runer
3,50B
44.64
3,35B
48.88
Scratch Training vs. Pruning- We compare LLMM-Pruner with StableI with similar parameter size. To ensure fairness, both models are fine-tuned on the Alpaca dataset. The experimental results of these two models are shown in the Table L.L.--runnr crafts lightweight L.MSs with low resources, and even can sometimes achieve better performance than LLMs from scratch training. However, we also acknowledge that the LLaMA-3B obtained by LLM-Prunc will not always outperform other 3B models from scratch training, due to the huge gap in the size of training corpus.
Table 9: Scratch Training (StableLM-3B) vs. Pruning (LLaMA-3B by LLM-Pruner)
Pruning Ratio
#Param
Latency
BoolQ
PIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Average
StubleLM-BB
3.6B
31.69s
48,78
69.48
44.52
54.62
50.93
25.17
27.40
45.84
LLaMA-3B
3.6B
37.96s
61.41
70.08
$1.01
55.01
46,80
30.38
37.40
50.30
More Data for Recovery Despite our primary experiments being conducted using 50k samples, we remain convinced that the inclusion of additional data could substantially enhance the recovery process, albeit at a considerably higher computational cost. Consequently, we conduct an experiment aimed at model recovery with more data, employing dataset comprising 2.59 million samples (59]. The results are detailed Table I0 From the results, it is evident that the performance of the com- pressed model closely approximates that of the base model, exhibiting only marginal performance decrease of 0.89%.
Table 10: Model Recovery: 50k samples vs. 2.59M samples
Model
#Samples
BoolQ
PIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Averuge
LLaMA-7B
73.18
78.35
72.99
67.01
67.45
41.38
42.40
63.25
LLLMA--54B
50k
64.62
77.20
68.80
63.14
64.31
36.77
39.80
59.23
LLaMA-5.4B
2.59M 59
76.57
77.37
66.60
65.82
70.62
40.70
38.80
62.36
Case Study. We provide some examples of sentences generated by the model compressed using LLM-Pruner in Table We made efforts ensure a minimal overlap between these generated sentences and the information contained in the tuning corpus, which demonstrates that the information originates from the original model rather than the tuning corpus. We provide additional examples in the Appendix, including the generated sentences of the model without post-training. From the

---

Table 11: Some examples of generations obtained with the original model and the pruned model.
Model
Original Model
Pruned Model
LLaMa
universe entirety space. time, matter, and energy lhat exists, includes all maller and energy. whether visible or not, and in fact the whole universe is invisible. There are many theuries regarding its origin and ultimate fate....
universe is the entirety space, time, matter, and energy that exists. The laws that make up physics elec- trons arhiring atoms, magnets attracting other objects, planets orbiling Sun have been consistent over
Vicuma
10 steps to build an ios app: I. Research and develop the idea. Decide the platform you want use (iOS, Android, both). 3. Create plan and budget. Hire leam of developers. 5. Design the user interface (IIT). 6. Develop the user interface 7. the upp the store. 9. Optimize the upp for performance and user experience (UX). 10, Market and promote the upp.
10 steps to build an los app: 1. Plan your app: decide what should what
cost look like, what build it 2. Hire developer: A good iOS developer will help you design the user interface, create the app, and make sure bug- free. Create user interface: where the magic happens. Your app's is's user interface will be designed and mocked up at this point..
iterate the app. 8, Subenit
ChaGGLM
Write poetry about love and peace. In the midst of life's hustic and bustle. I find your love soothing peace. It is lceling cannot conveyed, connection be distanced. With your smile, your eyes, your embrace. leel a sense of belonging that only you can give. is a love lasts forever. love that cannot defeat...
Write poetry about love and peace. Love and peace, two such wonderful things) A pure and endless desire in my heurt) both of them, must seek
cannol
long. long time, know.. Love, know, is fooling of being' A perfect partner. in every sensel And peace, need it, so much, day' long. long way. my heart will go..
evident that the sentences generated by the compressed model are comparable to those produced by the original model. They exhibit fluency, relevance, and informativeness regarding the given topic. Nevertheless, during our experiments, we observed that the pruned model's performance deviates from that of the original model, particularly when generating lengthy sentences. Occasionally, may generate sentences that are meaningless or contain repetitive tokens.
cases in Table 11
Conclusion
In this paper, we propose LLM-Pruner, a structured pruning approach for large language models. LLM-Pruner aims to compress sizable language models in a task-agnostic manner while minimizing the dependency on the original training corpus and preserving the linguistic capabilities of LLMs. LLM-Pruner accomplishes this by iteratively examining each neuron within the model as a trigger for identifying dependency groups, thereby constructing the L.M'' dependency graph. Subsequently, LL-M-rruner assesses the importance of these groups using both parameter-wist and weight-wise estimation. Finally, we utilize LoRA for fast recovery and adjustment of the pruned model. We evalu- ate the efficacy of --rurerrerr on three distinct LLLMA, Vicuna, and CharGLM-utilizin various zero-shot datasets. Our experimental results indicate that LLM-Pruner successfully prunes the model, reducing computational burden while retaining its zero-shot capabilities. Nevertheless, considerable performance degradation occurs when employing high pruning rates, such as the removal of 50% of LLaMA's parameters, resulting in substantial decline in model performance. Additionally. we observe instances in which the model generates incoherent sentences. Addressing the challenges associated with compressing L.MSs at higher pruning rates remains challenging task.
Acknowledgment
This project is supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (Award Number: MOE-T2EP200 12EP20122 22-0006), and the National Research Foundation, Singapore, under its Medium Sized Center for Advanced Robotics Technology Innovation.
10

---

References
[1] Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang. Jing Jin, Xin Jiang. Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit of bert quantization. arXiv preprint urXi::2012.15701, 2020.
[2] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
[3] Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, ct al. Language models are fow-shot leamerrs. Advances in neural information processing systems, 33: 33:1877-19011 2020.
[4] Wei-Lin Ln Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph F., Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality, March 2023. URL htttp:///mmss org/ blog/
2023-03- 03-300000000000 vicuna/
[5] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint (rXiv:2204.02311 2022.
[6] Christopher Clark, Kenton Lee. Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Compulational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2936,, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 1118//61/111-11//1111111 URL https://aclanther
org//1g-/ 1300
[7] Peter Clark, Isaac Cowhey, Oren Etzioni. Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803. 80.45330.. 2018.
[8] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm, int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208. 07339, 2022.
[9] Jacob Devlin. Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[10] Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv: 1909. 1556, 2019.
[11] Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning, 2023.
[12] Elias Frantar and Dan Alistarh. Massive language models can be accurately pruned in one-shot. arXiv preprint wxXi:2301.00770 2023.
[13] Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate prsiingaining quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17325 2022.
[14] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff. et A framework few-shot language model evaluation. Version vO. O. Sept, 2021.
[15] Fu-Ming Guo, Sijia Liu, Liu, Finlay S. Mungall, Xue Lin, and Yanzhi Wang. Reweighted proximal pruning for large-scale language representation. CoRR, abs/1909. 12486, 2019. http://////rxxi org/abs /bs/ 1909. 12486
[16] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gar- nett, editors, Advances Neural Information Processing Systems, volume 28. Curran Asso- ciates, Inc., 2015. URL httpp:: h/proceed/o.......... http:/rrrreeiiiinne.eiip.ppppppp ctturipp.cc/pppee.. httppperrtiieeepppp////////// aa0eeeeeddsa22ccf444644444400000 Paper .pdf
[17] Jordan Hoffmmann Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute -ottima large language models. arXiv preprint arXiv:2203.15556. 2022.

---

[18] Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang. Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Systems, 33:9782-9793, 2020.
[19] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation large language models, 2021.
[20] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163-4174, 2020.
[21] Eldar Kurtie, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal bert surgeon: Scalable and accurate recond-order pruning for large language models, arXiv preprint 22.3v:2200.072559 2022.
[22] Woosuk Kwon, Sehoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. A fast post-training pruning framework for transformers. arXiv preprint arXiv:2204.00 220::22044009666 2022.
[23]
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel. Piyush Sharma, and Radu Soricut. Al- bert: A lite bert for sule-super learning language representations. arXiv preprint arXiv: 11942. 2019.
[24] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
[25] Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Chazvininejad. Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Demoising sequenceto-sequeni pre-training for natural language generation, translation, and comprehension. arXiv preprint 191..:9910.146111 2019.
[26] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 160.v::/08.087/// 2016.
[27] Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. Dailydialog: manually labelled multi-turn dialogue dataset. Proceedings of The 8th International Joint Conference on Natural Language Processing (IJCNLP 2017), 2017.
[28] Chen Liang. Haoming Jiang. Zheng Li, Xianfeng Tang. Bin Yin, and Tuo Zhao, Homodistil: Homotopic task-agnostic distillation of pre-trained transformers. arXiv preprint arXiv:3302.00632 2023.
[29] Songhua Liu, Kai Wang, Xingyi Yang. Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. In Advances in Neural Information Processing Systems, 2022.
[30] Songhua Liu, Jingwen Ye, Runpeng Yu, and Xinchao Wang. Slimmable dataset condensation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[31] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, arXiv preprint arx::: 907. 1692, 2019.
[32] Zejian Liu, Fanrong Li, Gang Li. and Jian Cheng. Ebert: Efficient bert inference with dynamic structured pruning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4814 4823, 2021.
[33] Xinyin Ma. Yongliang Shen, Gongfan Fang, Chen Chen, Chenghao Jia, and Weiming Lu. Adversarial self-supervised data-free distillation for text classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6182-6192, 2020.
[34] Xinyin Ma. Xinchao Wang, Gongfan Fang, Yongliang Shen, and Weiming Lu. Prompting to distill: Boosting data-free knowledge distillation via reinforced prompt. arXiv preprint MtXiv:2205.07523. 2022.
[35] Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewiez. Building a large annotated corpus of English: The Penn Treebank, Computational Linguistics, 19()):313-330 1993. URL https
//uuw. www.wa.cwebb..... wwweeebb.rrgg/aee www/ http//anhooogyyJ3331 J93-
[36] JS McCarley, Rishav Chakravarti, and Avirup Sil. Structured pruning of a bert-based question answering model. arXiv preprint arXiv: 1910009900003600, 2019.
[37] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
12

---

[38] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? dataset for open book question answering. In EMNLP, 2018.
[39] OpenAL. Gpt-4 technical report, 2023.
[40] Haojie Pan, Chengyu Wang. Minghui Qiu, Yichang Zhang, Yaliang Li, and Jun Huang. Meta-kd: A meta knowledge distillation framework for language model compression across domains. CoRR, ab2/201..016666.
2020. URL URL httts:://rziii /tstpe///aazav..orrab///222222 arg/aba//011.00226
[41] Haojie Pan. Chengyu Wang, Minghui Qiu. Yichang Zhang. Yaliang Li, and Jun Huang. Meta-kd: meta knowledge distillation framework for language model compression across domains. arXiv preprint arXiv:2012.0/266 2020.
[42] Ahmad Rashid, Vasileios Lioutas, Abbas Ghaddar, and Mehdi Rezagholizadeh. Towards zero-shot knowledge distillation for natural language processing, 2020.
[43] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.
[44] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili,, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter multilingual language model. arXiv preprint arXiv:2211.05100, 2022.
[45] Suraj Srinivas and R Venkatesh Babu. Data-free parameter pruning for deep neural networks. arXiv preprint or / 2015.
[46] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation bert model compression. arXiv preprint 19X.::1998.00355. 2019.
[47] Siqi Sum, Zhe Gan, Yuwei Fang, Cheng, Shuohang Wang, and Jingjing Liu. Contrastive distillation on intermediate representations for language model compression. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 498-508, Online, November 2020. Association for Computational Linguistics. doi: (16653//1/220.emmmpppm //aclanthology.org//
URL https
emnlp-main.33
[48] Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: compact tagk-sgiostic bert for resource-limited devices. arXiv preprint arXiv:2004.02990 2020.
[49] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An ceaructioo-followiii llama model. htttss://gitam com com/tatsu- lab/ s stanf arp_aapacaaaaa 2023.
[50] Romal Thoppilan, Daniel De Freitas, Jamic Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv: 2022.
[51] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Amn Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[52] Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov, Analyzing multi-head self- attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797-5808, 2019.
[53] Alex Wang, Amanprect Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 1804.0746), 2018.
[54] Chaoqi Wang, Roger Grosse, Sanja Fidler, and Guodong Zhang. Eigendamage: Structured pruning in the kronecker-factored eigenbasis. International conference on machine learning, pages 6566 6575, PMLR, 2019.
[55] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33:5776-5788, 2020.
[56] Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. arXiv preprint 17X1:11970044332. 2019.
13

---

[57] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama. Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. arXiv preprint arXiv:2206.06622. 2022.
[58] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting clicits reasoning in large language modcls, arXiv preprint arXiv:2201.1 //900, 2022.
[59]
Minghao Wu, Abdul Waheed, Chiyu Zhang. Muhammad Abdul-Mageed, and Alham Fikri Aji. Lamini-Im: A diverse herd of distilled models from large-scale instructions, 2023.
[60] Yiquan Wu, Kun Kuang, Yating Zhang, Xiaozhong Liu, Changlong Sun, Jun Xiao, Yueting Zhuang, Luo Si, and Fei Wu. De-biased court's view generation with causality. In Proceedingy of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 763-780, 2020.
[61] Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. arXiv preprint arXiv:2204. 00408, 2022.
[62] Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. DeeBERT: Dynamic early exiting for accelerating BERT inference. In Proceedings of the 58th Annual Meeting of the Association for Computa- tiomal Linguistics, pages 2246-2251, Online. July 2020. Association for Computational Linguistics. doi:
10. 1..18653/1/20220.cs 204. htttttts////ccaattiee www.aannoooooooooo///e/.22/.. org/2020.00.2.. acl main. 204
[63] Dongkuan Xu, Ian En-Hsu Yen, Jinxi Zhao, and Zhibin Xiao. Rethinking network muning-under the pre-train and fine-tune paradigm. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2376-2382, 2021.
[64] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010. 11934, 2020.
[65] Xingyi Yang. Jingwen Yc, and Xinchao Wang. Factorizing knowledge in neural networks. In European Conference on Computer Vision, 2022.
[66] Xingyi Yang, Daquan Zhou, Songhua Liu, Jingwen Ye, and Xinehao Wang. Deep model reassembly. In Advances in Neural Information Processing Systems, 2022.
[67] Yiding Yang, Jiayan Qiu, Mingli Song, Dacheng Tao, and Xinchao Wang. Distilling knowledge from graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[68] Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168-277
2022.
[69] Deming Ye, Yankai Lin, Yufci Huang, and Maosong Sun. TR-BERT: Dynamic token reduction for accelerating BERT inference. In Proceedings of the 2021 Conference the North American Chapter of the Association for Computational Linguistics: Human Longuage Technologies, pages 5798 5809, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v 10155//v112221.. .nacl- main.463. URL htcps://aclanttoll.g
org/2021 naacl-main.. 463
[70] Edouard Yvinec, Arnaud Dapogny, Matthieu Cord, and Kevin Bailly. Red++: Data-free pruning of deep neural networks via input splitting and output merging. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3664-360 2022.
[71] Ohr Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized Sbit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS NNerIPS Edition (EMC2-NIPS)) pages 36-39. IEFF, 2019.
[72] Ofir Zafrir, Ariel Larey, Guy Boudoukh, Haihao Shen, and Moshe Wasserblat. Prune once for all: Sparse pre-trained language models. arXiv preprint arXiv:2111.0575: 2021.
[73] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
[74] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, al. Glm-130b: An open bilingual pre-trained model. arXiv preprint 2200042210.02414, 2022.
[75] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story- like visual explanations by watching movies and reading books, In The IEEE International Conference on Computer Vision (ICCV), December 2015.
14

---

A Detailed Explanations for the Dependency Rules
Group B: Multi-head Attention
Node
Node
Node
Node
Node
Node
Node:
Quarry
Kay
Case 1: Node dependent Nods
Case Node deperdent Node
Case Node K dependent Nods
Figure 6: Illustrations of the two dependency rules. All the cases are extracted from the multi-head attention module.
We provide a detailed explanation of the two dependency rules. It is important to note that these dependency rules do not pertain solely to the forward computation. Instead, they represent directional relationships that exist in both directions. For instance, removing node in a subsequent layer may also result in the pruning of node in the preceding layer. Recall the two dependency rules follows:
NJ (N) A Deg (N₂) = 1 = N, dependent on N; In(N,) Deg (N,)
N; is dependent on N,
(8)
where N₁ and N, are two neurons. In (N,, and Out(NV, represents all the neurons that point towards or point from N, Deg (N;) and Deg + (N;) represents the in-degree and out-degree neuron N;. Figure serves an illustration of the two dependency rules:
In case 1, Node and Node J satisfy the rule stated in Eq Consequently, Node J depends on Node I. When Node is pruned, it is necessary prune Node J as well.
case 2, Node and Node satisfy the rule Eq Thus, Node I dependent on Node If Node is pruned, it becomes imperative to prune Node I as well.
In case 3, Node and Node K do not meet the requirement of Eq due to the mismatch in Deg (N&) # 1. Thus, with Node pruned, Node K would not be affected.
B
Implementation Details
B.1 For Pruning
For the baseline Given the lack of previous work the struetural pruning of Large Language Models a task-agnostic and low-resource setting, there is currently no existing baseline for our model. To provide a comprehensive demonstration of the effectiveness of LLM-Pruner, we employ two additional methods for evaluation, alongside the data-free pruning method. All of these methods built upon the dependent groups identified Section
L2: We assess the importance of each group based on the magnitude of its weight matrix.
Random: This method involves randomly selecting certain groups for pruning.
For the 'Block' Group. Based on the findings presented in Table 3 it is preferable to leave the first three layers and the final layer unchanged, modifying parameters in those layers significantly impacts the model. Within each module, such as the MLP or the Multi-head Attention, the discovered groups are pruned based on pre-set ratio. For instance, in the MLP layer of LLaMA-7B, we identified 11 11,008 groups, and with a 25% pruning ratio, the module would prune 2,752 groups It worth noting that the pruning rate for the selected groups higher than the pruning ratio for parameters, certain layers (e.g., the embedding layer and excluded layers mentioned) retain their parameters. When aiming for a parameter pruning ratio of 20%, we prune 25% from Layer to Layer 30, Similarly, for a 50% parameter removal, we prune 60% of the groups from Layer to Layer 30.
15

---

For the 'Channel' Group. The Group 'Channel' exhibits a resemblance to dimension pruning in the model, targeting the pruning of certain dimensions. the case of the Query, Key, and Value projection in MHA, only the input dimension is pruned, while for the Output projection in MHA, only the output dimension is pruned. is important note that the entire dependency established automatically, without any manual design involved. The 'Channel' group operates in complementary manner to the 'Block Group'. the 'Channel' Group, the pruning ratio the group equals to the pruning ratio of the parameters, all weight matrices, including the embedding matrix, undergo pruning. Therefore, 20% pruning ratio of parameters implies pruning 20% the groups, while 50% pruning ratio implies pruning 50% of the groups.
B.2 For Recovery Stage
We follow [19 in our recovery stage. We set the rank d to 8 in our experiment. The learning rate is set lc-4 with 100 warming steps. The batch size of training is selected from [64, 128 and the AdamW optimizer employed in our experiment. The best training epoch we found is 2 epochs, as training with more epochs even has a negative impact on the model performance. We run our experiment on single GPU with 24GB memory, using approximately 2.5 hours if RTX4090 utilized. All the linear module is taken into account for efficient tuning, An ablation experiment for this is shown in Table T2
Table 12: Ablation: Tuning different modules in the re- covery stage
Module
WikiText
PTB
ALL
17.36
29.99
MLP
17.64
30.63
MHA
17.62
30.23
C More Analysis
C.1 Pruning Quantization
Here, we conduct comparative analysis of different compression techniques and illustrate that these techniques can effectively combined with little performance degradation. We have chosen LLM.in880 [8] representative example of quantization methods. Our results show that LLM.int8) outperforms LLM--Punnt Pruner while LLM-Pruner enhances latency, reduces parameter size. When these two techniques are applied in tandem, they collectively reduce memory consumption and expedite inference, offering a balanced approach that combines the benefits of both methods.
Table 13: Pruning and Quantization on LLaMA-7B
Pruning Ratio
4Param
Memory
Lasency
BoolQ
PIQA
HellaSwag
WinoGrande
ARC
ARC-c
OBQA
Average
LLaMA-7B
6.74B
12884 SMiB
69.32s
73.18
78.35
72.90
67.01
67.45
41.38
42.40
63.25
6.74B
LLaMA-5.4B
5.17B
10458 AMiB
58.55
76.57
77.37
66
65.82
70.62
10.70
38.80
62.36
LLaMA-3.48 11-----10 LLMMin88 inn80
5.478
37MiB
63.10
76.39
76.71
66.62
66.46
70.54
40.19
39:20
62.30
C.2 Global Pruning Local Pruning
present comparative analysis between global pruning and local pruning, where the pruning ratio 20% and the base model is LLaMA-7B. Global pruning refers to ranking all groups in the model together, whereas local pruning involves only ranking groups within the same module for pruning. The outcome of global pruning leads to varying widths across different layers and modules, whereas local pruning ensures uniformity across all layers.
Based on our experimental findings, we observed slight advantage of local pruning over global pruning. We think this is because of the varying magnitudes in different layers modules, which makes the importance scores incomparable between groups across different layers.
Table 14: Results of global pruning and local pruning
Method
WikiText2
PTB.
BoolQ
PIQA
HellaSwag
WinoGrande
ARC-c
ARC-c
OBQA
Average
Element local
19.09
34.21
$7.06
75.68
66,80
59.83
60.94
36.52
40.00
$6.69
Element
global
20.84
32.86
63.15
73.23
63.31
66.38
55.85
35.49
38.00
56.49
16

---

C.3 Overfitting Phenomena in Post- pust-Training
We present a comprehensive analysis of the overfitting issue in the recovery stage, as previously mentioned in Figure [5] Here the results cover all 9 datasets across various training steps. Based on the findings presented in Table [5| noticeable trend emerges: the accuracy or generation quality initially shows improvement but subsequently experiences a slight decline. This pattern suggests that the recovery process completed within short period. And given that the training corpus is constainn, more training epochs can result in overfitting to the specific dataset while potentially compromising the original capabilities of the language model.
Table 15: The PPL. and Accuracy different training steps
Step
WikiText2
PTB|
BoolQ
PIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Average
19.09
34.21
57.06
75.68
66.80
59.83
60.94
36.52
40.00
56.69
18.10
30.66
64.62
77.20
68.80
63.14
64.31
36.77
39.80
59.24
400
17.69
30.26
63.00
76.66
68.75
63.54
64.39
37.20
40.60
59.16
600
17.69
30.57
66.24
76.28
68.52
63.85
64.48
37.37
41.00
59.68
800
17.64
30.57
65.05
76.22
68.38
63.77
63.64
37.29
40.80
59.31
1000
17.67
30.60
66.39
76.17
68.24
64.17
63.05
37.37
41.60
59.57
1200
17.74
30.75
65.75
76.28
68.28
63.77
63,30
37.63
41.20
59.46
1400
17.88
30.85
64.34
76.28
68.31
63.85
63.47
37.80
41.20
$9.32
C.4 Pruning with Large Rates
Additionally, we conducted tests on LLaMA-7B and Vicuna-7B with 50% parameters pruned. We observe significant decrease in performance compared to the base model. However, the recovery stage proved to be beneficial, resulting in an improvement of approximately 7.39%. Pruning Language Model with such a high pruning rate remains challenging task.
Table 16: The PPL and Accuracy on LLaMA-7B with 50% parameters pruned
Pruncd Model
Method
WikiTex22
PTBI
BoolQ
PIQA
HollaSwag
WineGrande
ARC-e
ARC-c
OBQA
Average
12
39266.42
48867 85
55.11
53.59
27.03
49.49
26.43
29.01
34.40
39.29
random
4337.27
27.90
37.75
Ratio 50%
Channel
13891.92
16114.91
40.37
52.18
25.72
48.86
25.72
28.24
30:40
35.93
w/o tune
Vector
141.06
236.24
62.17
55.11
27.25
49.88
29.00
25.77
34.00
40.45
Element
106.07
266.65
52.57
60.45
35.86
49.01
32,83
25,51
34.80
41.58
Flement
255.38
53.20
27.22
33.40
Channel
1122.15
1092.26
40.76
54.84
26.94
49.41
27.86
25:43
32.20
36.77
Ratio 50% w/ tune
Vector
43.47
68.51
51.54
28.33
32,40
46.61
Element
45.70
69.33
61.47
68.82
47.56
$5.09
46.46
28.24
35.20
48.98
Element
38.12
66.35
60.28
69.31
47.06
53.43
45.96
29.18
35.60
48.69
Table 17: The PPL and Accuracy on Vicuna- 7B with 50% parameters pruned
Pruned
Method
WaiTex12
PTB.
BoolQ
PIQA
HellaSwag
WinoGrande
ARC-c
OBQA
Average
Ratio = 0%
Vicusa-7B
16.11
61.37
76.57
77.75
70.64
67.40
65.11
41.21
40.80
62.78
12
54516.03
66274.6
45.99
53.48
26.55
47,83
27.53
28.58
30.40
37.19
random
17020.73
13676.54
48.17
53,43
27.31
50.43
26.30
29.78
30.20
37.95
Ratio 50% tume
Channel
8360.30
10681.30 30
38.69
53,10
26.42
50.20
25.97
29.52
29.60
36.22
Vector
189.87
409.75
26.99
Element
143.85
427.77
53.76
59.79
34.86
50.28
33.29
27.30
34,60
41. 18.98
Element
162.38
469.74
52.45
58.65
34.27
49,72
32.41
28.75
35,80
41.72
Channel
206.23
$26.89
62.11
58.38
32.29
51.22
33.08
24.06
30.20
41.62
Vector
46.11
147.52
62.14
64.91
39.80
50.91
47.77
27.30
32.60
46.49
Element 42.99
139.91
54.78
29.01
w/ tune
Element
41.64
143.74
62.08
69.48
46.79
54.54
48.15
29.78
36.20
49,57
D Generations From Compressed Model
Table 18 19 20 and 21 show more examples of the models pruned by L.MM-Pruner. We present
the generation results of both the pruned model with post-training and without post-training. The absence of post-training allows us to better understand the information retained in the model. We include the results of CharGL--6B in two languages as it is bilingual model.
17

---

Table 18: Generated Examples from the Compressed LLaMA-5.4B
Model
Generated Sentences
Neural Sequential Model, especially transformers, have become very popular due their impressive performance in different arens. The neural network model can recognize the human voice more efficiently than speech recognition, and speech can be better distinguished. However, its high energy consumption has been the biggesi problem. Many efforts have been made, such as minimizing the number layers and the length of the neural network, has had effect. The work is to develop low-dimensional sound model which is suitable for the high frequency band of the speech. After roducing the
can create a logo seconds or design a book cover, they need what are the implications that this has for branding design? Giving Creative Control Algorithms, The first wave of 'creative' branding has come from designers and agencies submitting their creations to algurithuss. They are algorithus that have been programmed to create, and are heing asked to replace designers the creative process. For example, in 1998 Google used random process select words and colours for their logo. Now,
much faster than humans can.
LLaMA w/ Tune
What's great about the holiday season, is it provides us with the excuse we need to reconnect with the ones we have grown apart from. There are many people who I only talk to a couple of times per year during the year and never interact with them anymore, and while I'm completely okay with that
they too should use Christmas as a great upporiunity lo get back touch with people in their lives that maybe they were lou busy to talk to before. If you are on good terms with family member or friend, and you haven't I been talking to them lately due to how
Noural Soquential Model, especially transformers, we designed them as a neural network that transforms input words coresponding ouiput. In our model. the 73 neural nodes arranged the form of L-shaped and used construct the woo-dimesiional lattice and form sentence. This sentence is then transferred the decoder to learn the words. Through 5.4 5.4.1 million repetitions, we can form a language model through the neural network we designed. With the help of a word level data correction method, we made strong progress in the recognition human language.
Al can create a logo in soconds. You'll Il see design in instant and you'l 'll be ?? that you' 'e getting one the best logo designs around. The process is quick and easy and if you need now you'l ll pay just a fraction of the cost for a professional logo designer. Why would I need logo? It helps a brand stand out from the competition. It's the visual cue that people respond to the logo. It's what makes you stop and look the brand when you'de 're walking by street. Why invest in logo?
LLaMA w/o Tune
What's great about the holiday season is being with the people that you love, and the belidays are the perfect occasion celebrate with the people that you love most. Being together with family and friends is one of those special occasions that can have remarkable eflect on low perceive our lives, which why they become some of the most cherished memories of our lifetimes. But holiday planning is more complex than a simple gathering. In fact, many families need professional advice to find the right way to handle the emotions involved. And that's why planning, prepuration.
Table 19: Generated Examples from the Compressed Vicuna-5.4B
Model
Generated Sentences
Last night we all danced together in the rain, our cares and worries forgotien, And as the sun began to rise. looked into your eyes and knew that everything was going to be okay. That's the way is now, forever and forever. love you. my baby, and I always will.
It fills me with such pride and joy to be working along side her, learning from her and sharing our experiences, joys, and challenges partners. Our bund growing stronger each day and I can't 't help but feel grateful for the beautiful life she has brought into my owa. To my loving wife, I love you more and more each day. I can't 't wait to spend the rest of my life with you and raising our future children logether proyecting the light of Christ into the world. And to all of you reading this, I encourage you to take scriously and
Vicuna w/ Tune
your relationship with the one
Meet me at midnight tonight at the Bell Tower in town. Bring torch, and we'll 'Il take the treasure off the church's 's hands, no muiter the cost.' With a deep sense of trepidation, Alaric finds himsell agreeing to the plan. He spends the day getting everything he can from the village, preparing for inevitable attack he'l have to launch. When the day arrives, he and his companions mount their steeds and set off towards the towering keep of the temple. As they draw closer, the wind whispers
Last night we all danced together in the rain. The water felt good on my face. The soog played over and over, but iid''t 'tantt care. I just kept duncing with you. I don't know what to say. I'm not good with words. But know that I've never felt this way before. With anyone. Not even myself. And I don't want to let that feeling go. I don't know what you feel, or what you want from me. But know that this is real. And I don't Ion't want to let go
It fills me with such pride and joy. it's like seeing my baby for the first time, and just want hold it close and never let it go. The thought that it mighi have to be put down one day breaks my heart. want to du everything my power to protect it, to keep it safe and give it the best life possible. But it's not mine. It's never been mine to keep. And I know that one day, sooner or later, it will have to be given away. can't 't help but feel a sense of responsibility that don't...
Vicuna w/o Tune
Meet me at midnight. I promise you'll 'll be the only one there. It's hard for me to helieve, but I'm scared. I don't 't want to be alone tonight, but know have to be brave. Hurry down the hill, I'm waiting for you, I'll take you to place that know is safe. We won't 't be soen, nobody'll know, It's almost time for you to go. don't 't want you to be late tonight.
18

---

Table 20: Generated Examples from the Compressed ChatGLM-5.66 (English)
Model
Generated Sentences
How buy cat? It's important understand that while understandaalle you might be looking todnle cat for yourself, adoption is a great way bring home furry friend without having to shop. Here are some things to consider before adoption: 1. First, make sure you know you' 'e interested in Jr. or Vioness. This determines whether you'l 're looking for a young adult cat or an older cat, 2. research your options: Look up reputable cat owners or organizations that foster cats your area. This will give you information on cat owners
CharGi w/ Tume
What is McDonald's? s McDonald's is a popular American fast food restaurant chain that offers a variety of food and heverage items. The company was founded in 1940 by Richard and Maurice Mc Fu Fu in San Franciscn, and the chain was originally called "McNuggun ugguggets." Today. McDonald's is known for its popular fast-food items such McDonald's s Drive-Thru, McChickens, and MeFlurryary. The company also offers wide range of other food and beverage items such as Happy Endings, Big Macs, and
What is Mean Square Emoo? The Mean Square Error (MSE) measure difference between a target value and the best possible estimate of its true value, based the data. is calculated by laking the average difference between the actual value and the estimated value over thes number of times the target value is repeated.
How to buy cat? It's important to understand that adopting a cat from a shelter or organization is a valuable and responsible adoption option, as well as one that may safer and more comfortable for you and your family. Additionally. it's important to read carefully before adopting ensure that you choose a Cat who suitable for you and your family.
What McDonald's? McDonald's is a popular fast food restaurant chain from United States known for its restaurant concepts such McNogguggets, McFluryyyy and Mc Mibibs. The chain started in 1955 as small operation operation called "Speedy McDouald's in San Antonio, Texas and it quickly expanded and became a widely recugmized and popular phenomenon in the country. Today, McDonald's global icon known around the world.
ChatGLM w/o Tune
What Mean Square Emror? Mcan Square Error (MSE) measure of the accuracy function or a measurement when it is repeated and the result is expected to be within certain range. It is calculated by taking the difference between two sets of measurements and measuring the difference squared. In simple terms, it means the mean sum of the ermis over a given period of time difference between the actual and expected values ...
Table 21: Generated Examples from the Compressed CharGLM-5.60 (Chinese)
Model
Generated Sentences
# W.
A9
M H0 HDKI
P0 AC Ay (i)
fL #o EAL
CharGLM w/ Tune
X = #|x i 75 A ji FHE
| AN) TTE HE KN) (P) dea at ME USL 1E TIR mn
my etl
$ f.
1E ie,
Al at
LX LA BAL: ALL AP
ARS FALREE SXFORR TOR HE HEE
HA
1E (REAP) THE E ARR
E RE
LA JA
A32 11:
#
ChatGLM
-
PE By AS LA BE
w/o Tune
1 04
A JOSSE THE
M di 57
Bn dud (9
AEE By
t. 63 A Je 14
Ba - FA HIT
19


Reviews:
[{"review_id": "50L7ZhM0n4", "reviewer": "Reviewer_ahc6", "summary": "This work proposes an LLM compression framework, LLM-Pruner, that employs structural pruning to compress LLMs in a task agnostic manner so that their multi-task solving is preserved. They achieve this without using the original training data. In addition, they employ adaptors (LoRA) during the tuning phase for parameter and training efficiency. They evaluate the models on zero-shot classification and text generation and demonstrate reasonable performance compared to the original model.", "questions": "1. LoRA and other adaptors are not necessarily faster to train. They reduce the number of trainable parameters but the backprop generally runs through the entire network. Can you compare the training time with and without LoRA (i.e. full network training).\n\n2. The Ablation Section at the beginning of line 227 explains a table but does not refer to one. I assumed it was Table 3 but if so some of the explanations were not clear.\n\n3. Weights and parameters are some times used interchangeably in deep learning. What is meant by weights in Section 3.2.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Although task agnostic knowledge distillation exists (TinyBert, DistilBERT etc.) and can be done without the original training data, it requires a significant training effort. This work can achieve upto 5 times compression with minimal training.\n\n2. The estimation of group importance estimation is important for gaining an inference speed advantage for pruning. However, the authors did not present an analysis on the impact of their method on inference speed.", "weaknesses": "1. Although the authors presents the reduction in parameters, they do not mention the inference speedup with and without LLM-pruner. What would be specially interesting would be the inference speed under different types of group pruning (e.g. channel vs block)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the reviewer for the constructive comments.\n\n>W1: Although the authors presents the reduction in parameters, they do not mention the inference speedup with and without LLM-pruner. What would be specially interesting would be the inference speed under different types of group pruning (e.g. channel vs block)\n\nWe thank the reviewer for the valuable suggestion. In the table below, we show the latency of different grouping strategies on LLaMA-7B. The latencies shown in the table correspond to runtime evaluations on the test set of WikiText2 and PTB. We test the result on A5000 with 24GB memory. Notably, both strategies accelerate the inference over LLaMA-7B. Nevertheless, a performance disparity persists between these two strategies, with the block strategy outpacing the channel strategy in speed. Such discrepancy arises from distinct pruning approaches, resulting in variations in the model structure.This presents an intriguing topic for further investigation, and we will add these results to the revised version.\n\n| Group   | Ratio | #Params | Memory     | Latency(WikiText2) | Latency(PTB) |\n|---------|-------|---------|------------|---------------|--------------|\n| /      | 0%     | 6.74B   | 12884.5MiB | 69.32s        | 20.47s       |\n| channel | 20\\%  | 5.47B   | 10488.4MiB | 61.50s        | 18.30s       |\n| block   | 20\\%  | 5.42B   | 10375.5MiB | 58.55s        | 17.55s       |\n| channel | 50\\%  | 3.37B   | 6556.3MiB  | 40.11s        | 11.96s       |\n| block   | 50\\%  | 3.35B   | 6533.9MiB  | 37.54s        | 11.27s       |\n\n> Q1: LoRA and other adaptors are not necessarily faster to train. They reduce the number of trainable parameters but the backprop generally runs through the entire network. Can you compare the training time with and without LoRA (i.e. full network training).\n\nThanks for the comment. The table below shows the contrast between training time and accuracy achieved through LoRA and full-parameter tuning obtained on A5000 GPUs with 24G memory, where LoRA indeed demonstrates faster training. \nThis is mainly due to two reasons: (1) Constraints in conducting full-parameter fine-tuning on a single 24G GPU result in increasing communication overhead across multiple GPUs. (2) The greater quantity of parameter updates, along with additional updates in the optimizer during full-parameter training, leads to prolonged training durations.\n\nApart from comparing the time, we also show the accuracy with and without LoRA.\nour results indicate LoRA's superior performance compared to full-parameter supervised fine-tuning in this case. This is attributed to the dataset's small size, making full-parameter training susceptible to overfitting, especially in the context of zero-shot evaluation of LLMs.\n\n| Pruning Ratio      | Time      | #GPUs | BoolQ | PIQA  | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA  | Average |\n|--------------------|-----------|-------|-------|-------|-----------|------------|-------|-------|-------|---------|\n| Full-Parameter SFT | 5.4 hours | 8     | 62.20 | 62.68 | 41.31     | 53.83      | 48.19 | 29.35 | 33.60 | 47.31   |\n| LoRA               | 3.5 hours | 1     | 61.56 | 68.72 | 46.62     | 52.64      | 47.94 | 29.27 | 35.40 | 48.88   |\n\n> Q2: The Ablation Section at the beginning of line 227 explains a table but does not refer to one. I assumed it was Table 3 but if so some of the explanations were not clear.\n\nWe thank the reviewer for the comments and truly apologize for the missing reference for the table. The ablation experiment mentioned in Line 227 refers to Table 3, with the line annotated with \"parameter\" or \"weight\" (The names would also be modified as described in the next question.)\n\n> Q3: Weights and parameters are sometimes used interchangeably in deep learning. What is meant by weights in Section 3.2.\n\nWe thank the reviewer for the comment. Indeed, the wording of `weights` and `parameters` could cause ambiguity, for which we apologize. In our revised version, we will change our terminologies to `vector-wise importance` (previously `weight`)\nand `element-wise importance` (previously `parameter`). The two represent different importance-assessment granularities: `Vector-wise importance` involves evaluating the entirety of weight parameters within the sample module as a whole through Taylor expansion, while `element-wise importance` pertains to assessing each individual weight in the same module distinctly.\n\n\n\n"}, {"review_id": "r0yCtl1FMV", "reviewer": "Reviewer_3DoF", "summary": "This paper present a method for LLM pruning, where some parameters are discarded by sacrificing the downstream accuracy.\nThe proposed method, LLM-Pruner, adopts structural pruning that selectively removes non-critical coupled structures based on gradient information.\nBy investigating different pruning settings with different LLMs, this paper finds the proposed method can help reduce the computations while retaining the zero-shot capabilities.", "questions": "Is the proposed method a general model compression method or designed for LLM? After reading this submission, my feeling is this is general method while the method's name is LLM-pruner.", "limitations": "n/a", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is task-agnostic, not relying on domain/downstream data.\n- Experimented with three different LLMs, with consistent observations.", "weaknesses": "- Missing the comparison between a pruned larger model and a smaller unpruned model. For example, when pruning 50% parameters for 7B model, how the results compared with a 3B model?\n- Examining the proposed method with different models is good. However, instead of experimenting with 3 LLMs in similar sizes (7B, 7B, and 6B parameters), pruning a larger one would be more important. For example, a 13B llama/Vicuna model.\n- Experiments only cover the proposed method, weakly connected to other model compression works. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the reviewer for the constructive comments.\n\n---\n\n> W1: Missing the comparison with the smaller models. For example, when pruning 50\\% parameters for 7B model, how the results compared with a 3B model?\n\nThanks for the constructive suggestion. As advised, we show the experimental results comparing the popular StableLM-3B and ours below. Specifically, for the sake of fairness, we selected the `stabilityai/stablelm-tuned-alpha-3b`, since it has also been tuned on Alpaca.  The performances of the two models are shown in the table below.  All results are obtained through the `lm-evaluation-harness`. LLaMA-3B obtained by LLM-Pruner demonstrates superior performance compared to StableLM-3B, incurring only a minor expense (with 7.3M vs. 800B tokens for training) to attain this compact LLM, as opposed to undertaking a complete retraining process. We would include these results in the revised version.\n\n| Pruning Ratio | #Param | Latency | #Tokens for Training | BoolQ | PIQA  | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA  | Average |\n|----|--------|------|------|-------|-------|-----|------|-------|-------|-------|---------|\n| StableLM-3B   | 3.6B   | 31.69s  | 800 Billion         | 48.78 | 69.48 | 44.52     | 54.62      | 50.93 | 25.17 | 27.40 | 45.84   |\n| LLaMA-3B  by LLM-Pruner    | 3.6B   | 37.96s  | 7.3 Million            | 61.41 | 70.08 | 51.01     | 55.01      | 46.80 | 30.38 | 37.40 | 50.30   |\n\n---\n\n> W2: Instead of experimenting with 3 LLMs in similar size (7B, 7B, and 6B parameters), pruning a larger one would be more important. For example, a 13B llama/Vicuna model.\n\nThanks. We show the results using LLM-Pruner to prune 20% parameters of LLaMA-13B here:\n\n| Pruning Ratio        | Method    | WikiText2 | PTB    | BoolQ | PIQA  | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA  | Average |\n|----------|---------|-----------|--------|-------|-------|-----------|------------|-------|-------|-------|---------|\n| Ratio = 0%            | LLaMA-13B | 11.58     | 20.24  | 68.47 | 78.89 | 76.24     | 70.09      | 74.58 | 44.54 | 42.00 | 64.97   |\n| Ratio = 20% w/o tune  | L2        | 61.15     | 91.43  | 61.50 | 67.57 | 52.90     | 57.54      | 50.13 | 31.14 | 36.80 | 51.08   |\n|                      | Random    | 19.24     | 31.84  | 63.33 | 73.18 | 63.54     | 60.85      | 64.44 | 36.26 | 38.00 | 57.09   |\n|                      | Channel   | 49.03     | 106.48 | 62.39 | 66.87 | 49.17     | 58.96      | 49.62 | 31.83 | 33.20 | 50.29   |\n|                      | Block     | 16.01     | 29.28  | 67.68 | 77.15 | 73.41     | 65.11      | 68.35 | 38.40 | 42.40 | 61.79   |\n| Ratio = 20% w/ tune  | L2        | 20.97     | 38.05  | 73.24 | 76.77 | 71.86     | 64.64      | 67.59 | 39.93 | 40.80 | 62.12   |\n|                      | Random    | 16.84     | 31.98  | 64.19 | 76.06 | 68.89     | 63.30      | 66.88 | 38.31 | 40.80 | 59.78   |\n|                      | Channel   | 17.58     | 29.76  | 69.20 | 76.55 | 68.89     | 66.38      | 62.08 | 38.99 | 39.60 | 60.24   |\n|                      | Block     | 15.18     | 28.08  | 70.31 | 77.91 | 75.16     | 67.88      | 71.09 | 42.41 | 43.40 | 64.02   |\n\nMost of the observations align with the experimental findings on LLaMA-7B. \nNotably, when contrasting the experimental outcomes of LLaMA-7B and LLaMA-13B, it becomes evident that the parameter redundancy in LLaMA-13B is higher than that in LLaMA-7B. Following the removal of 20% of the parameters, LLaMA-7B exhibited a reduction of 5.86%. By contrast, LLaMA-13B experienced a decrease of merely 3.18%. Furthermore, after undergoing fine-tuning, LLaMA-13B demonstrated a minor decline of only 0.95% (from 64.97 to 64.02), which significantly outperforms LLaMA-7B's decline of 3.43% (from 63.25 to 59.82).\n\n---\n\n> W3: Experiments only cover the proposed method with a weak connection to other model compression work.\n\nThanks. LLM-Pruner focuses on developing an efficient model compression algorithm for LLMs, with reduced dependence on data. The reason we don't extend previous approaches to LLM as a baseline is their high demand for large-scale datasets for compression. Previous methods, like those in knowledge distillation, require the training dataset to recover the model performance. For instance, LLaMA's training set has 1T tokens, which is often impractical for compressing due to high costs.\n\nBelow, we conducted an experiment utilizing DistilBERT as the baseline to demonstrate the performance when applying the same dataset to compress LLaMA-7B using both LLM-Pruner and DistilBERT. We employed the same dataset, Alpaca, as the dataset for the recovery stage. A comparison between a 50% pruned model through LLM-Pruner and DistilBERT highlights LLM-Pruner's superior performance. This advantage arises from LLM-Pruner's enhanced capability to preserve critical knowledge from the base model, resulting in reduced data dependency.\n\n| Pruning Ratio | #Param | BoolQ | PIQA  | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA  | Average |\n|---|---|---|---|------|------|----|----|---|-----|\n| DistilBert    | 3.50B  | 54.77 | 63.28 | 40.44     | 52.49      | 43.31 | 27.22 | 31.00 | 44.64   |\n| LLM-Pruner    | 3.35B  | 61.56 | 68.72 | 46.62     | 52.64      | 47.94 | 29.27 | 35.40 | 48.88   |\n\n---\n\n> Q1: Is the proposed method a general model compression method or designed for LLM? After reading this submission, my feeling is this is general method while the method's name is LLM-pruner\n\nWe thank the reviewer for the comment. The proposed strategy is indeed general, yet tailored for LLM with dedicated designs. For instance, we skip some initial and final layers, adapt different pruning strategies during the pruning process and employ LoRA for fine-tuning. These design choices are incorporated to ensure the model's performance is retained as much as possible when pruning and facilitate model recovery with limited data. These designs are unnecessary when data is abundant and model scale is less of a concern."}, {"review_id": "CTseNWKrTN", "reviewer": "Reviewer_VdKx", "summary": "The authors propose LLM-Pruner to compress large language models in a task-agnostic manner. Their method can achieve a fast recovery by tuning LoRA after pruning the model. Their experiment results show LLM-Pruner can achieve comparable performance with much smaller memory. ", "questions": "None", "limitations": "None", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Their motivation is interesting and the proposed method can be generalized to variable tasks. Their method design also makes sense.\n2. Their experiments are extensive, and their analyses also demonstrate the effectiveness of LLM-Pruner.\n", "weaknesses": "1. What's the performance if no post-tuning is applied? \n2. What is the performance of other compressing methods like quantization and task-specific pruning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the reviewer for the constructive comments.\n\n---\n\n> W1: What's the performance if no post-tuning is applied?\n\nThanks for the comment. In Table 1-3 in our submitted manuscript, we present the results without post-training (please refer to `Ratio=X% w/o tune`). Without post-training, the models can retain a large portion of their performance with 20% of parameters pruned. For instance, on LLaMA-7B, the model retains 90.7% of the original performance. From the qualitative experimental results of the generated results (see Tables 12-15 in the Appendix), the pruned model with no post-tuning also exhibits decent text generation capabilities.\n\n---\n\n> W2: What is the performance of other compressing methods like quantization and task-specific pruning?\n\nWe thank the reviewer for the comment. \n \n* Task-specific Pruning:\n\nWe conducted task-specific pruning on the ARC-easy dataset. In this experiment, both the pruning and tuning process utilized the ARC-easy dataset. The table below shows our experimental results. It can be observed that, compared to LLM-Pruner, there is an improvement in performance on ARC-easy. However, since task-specific pruning is optimized for specific datasets, if we were to perform tests on other datasets, such as BoolQ, the performance would be significantly inferior to that of LLM-Pruner (56.69 vs. 66.79), even without tuning (56.69 vs. 62.54). We believe this is primarily due to task-specific pruning being overfitted to specific domains, thereby not ensuring optimal performance on out-of-domain or zero-shot datasets.\n\n| Method                                                        | In-domain: ARC-easy | Out-of-domain: BoolQ |\n|---------------------------------------------------------------|---------------------|----------------------|\n| Task-specific Pruning + Task-specific Fine-tuning             | 68.35               | 56.69                |\n| LLM-Pruner: Task-agnostic Pruning                             | 61.57               | 62.54                |\n| LLM-Pruner: Task-agnostic Pruning + Task-agnostic Fine-tuning | 64.06               | 66.79                |\n\n* Quantization\n\nWe thank the reviewer for pointing out network quantization. By nature, quantization is indeed another line of model compression strategy, which is orthogonal to pruning and hence can be readily deployed on top of pruning to further reduce the network size. The reasons behind lie in that, pruning aims to reduce the number of parameters, while quantization strives for reducing the precision of the numerical values used to represent the parameters. In practice, these techniques are often used in combination to achieve even greater efficiency gains. For example, one can prune a neural network and then quantize the remaining weights and activations to further optimize the model.\n\nWe evaluate our techniques against LLM.int8()[1] on LLaMA-7B. We post-train the pruned model obtained by LLM-Pruner under `MBZUAI/LaMini-instruction`. From the results, both quantization and pruning have their own pros and cons: LLM.int8() performs slightly better than LLM-Pruner but LLM-Pruner is faster than LLM.int8() (the speedup ratio of LLM.int() on a 6.7B model is 0.86x, as mentioned in the paper [1]). The pruning of LLMs is still in its early stage,  and there is much that remains to be thoroughly investigated and studied.\n\n[1] LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale\n\n| Pruning Ratio | #Param | Memory     | Latency | Speedup | BoolQ | PIQA  | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA  | Average |\n|---------------|--------|------------|---------|---------|-------|-------|-----------|------------|-------|-------|-------|---------|\n| LLaMA-7B      | 6.74B  | 12884.5MiB | 69.32s  | 1x      | 73.18 | 78.35 | 72.99     | 67.01      | 67.45 | 41.38 | 42.40 | 63.25   |\n| LLM.int8()    | 6.74B  | 6777.7MiB  | 76.20s  | 0.91x   | 73.36 | 78.18 | 73.01     | 66.93      | 67.47 | 40.87 | 41.80 | 63.09   |\n| LLaMA-5.4B    | 5.47B  | 10488.4MiB | 58.55s  | 1.18x   | 76.57 | 77.37 | 66.60     | 65.82      | 70.62 | 40.70 | 38.80 | 62.36   |\n\n"}, {"review_id": "g3JU6rMwPk", "reviewer": "Reviewer_mPK3", "summary": "The paper is trying to present an resource efficient task agnostic technique for pruning large language models. The authors tackle two main problems, 1) in a setup where there is no access to the original dataset that was used to train the LLM, and 2) the pruning should be fast.\n\nThe paper presents a three step approach to pruning:\n1) Discovery -- finding the groups (structures) that will be evaluated for pruning.\n2) Estimation -- estimating the impact of removing a particular group on the performance of the model.\n3) Recovery -- post pruning training of the model to recover on the lost performance.", "questions": "Please see the weakness section. I would be curious to look into latency analysis and comparison with a model directly trained on the same corpus.", "limitations": "There isn't any discussion around deploying such a sparse model compared to a dense model, especially whether the theoretical guarantees in terms of speedups translate to the same when deploying.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper presents a novel approach to pruning, which is fast, efficient, and agnostic of the downstream task and operates without the availability of the pretraining dataset.\n\nThe paper is well written and clearly explains their approach, and performs a good number of ablation studies to provide a good understanding of the impact of inclusion/exclusion of each component of their approach.\n\nThe paper evaluates the work on a wide variety of tasks in zero shot setting.", "weaknesses": "The paper starts with the problem that deployment & inference are the key challenges faced by large language models, and presents a solution to it. However, no assessment of latency is provided in the paper. If we prune some parameters, the resulting model is sparse. Sparse models often to less theoretical computations, but at times do not benefit much on latency because the inefficiency introduced by sparse computations.\n\nThere is no baseline that is taken into consideration. It it possible to get two models of similar architecture, and prune the larger model to the smaller one, and then train both the pruned model, and the small model, on the Alpaca dataset used in the paper to understand the gain that this approach gives when compared to a model that has been directly trained? A latency analysis between these two models will be pretty useful to make a conclusive statement as well in order for it to be useful in production setups.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the reviewer for the constructive comments.\n\n---\n\n> W1: The paper starts with the problem that deployment \\& inference are the key challenges faced by large language models, and presents a solution to it. However, no assessment of latency is provided in the paper. If we prune some parameters, the resulting model is sparse. Sparse models often to less theoretical computations, but at times do not benefit much on latency because the inefficiency introduced by sparse computations.\n\nThanks for the comment. We show in the table below the latency of the pruned model across different pruning ratios, employing diverse grouping strategies. The latencies shown in the table correspond to runtime evaluations on the test set of WikiText2 and PTB. We test the result on A5000 with 24GB memory. \n\nRegarding sparsity, we would like to note that the proposed approach indeed goes beyond the practice of masking certain parameters to be zero, which is generally known as **unstructural pruning**. Our approach, as indicated in the title, focuses on **structural pruning** of LLM, meaning that we completely remove unnecessary connections and neurons alongside their associated weights, resulting in a **smaller** model rather than a **sparse** one. The model pruned by LLM-Pruner yields improved model acceleration without the need for specialized hardware. By removing unnecessary connections and neurons, structural pruning reduces the number of required multiplications and additions during forward-pass computations. This reduction in floating-point operations leads to faster inference time. \n\n| Group   | Ratio | #Params | Memory     | Latency(WikiText2) | Latency(PTB) |\n|---------|-------|---------|------------|---------------|--------------|\n| /      | 0%     | 6.74B   | 12884.5MiB | 69.32s        | 20.47s       |\n| channel | 20\\%  | 5.47B   | 10488.4MiB | 61.50s        | 18.30s       |\n| block   | 20\\%  | 5.42B   | 10375.5MiB | 58.55s        | 17.55s       |\n| channel | 50\\%  | 3.37B   | 6556.3MiB  | 40.11s        | 11.96s       |\n| block   | 50\\%  | 3.35B   | 6533.9MiB  | 37.54s        | 11.27s       |\n\n---\n\n> W2: There is no baseline that is taken into consideration. It it possible to get two models of similar architecture, and prune the larger model to the smaller one, and then train both the pruned model, and the small model, on the Alpaca dataset used in the paper to understand the gain that this approach gives when compared to a model that has been directly trained? A latency analysis between these two models will be pretty useful to make a conclusive statement as well in order for it to be useful in production setups.\n\nWe thank the reviewer for the comment. As advised, we added the StableLM-3B as a baseline. Specifically, we employed the `stabilityai/stablelm-tuned-alpha-3b` model, which has a similar number of parameters and is also tuned on Alpaca to ensure the fairness of comparison.\n\nFrom the perspective of model performance, the pruned LLaMA-3B model using LLM-Pruner outperforms StableLM-3B, incurring only a minor expense (7.3M vs. 800B tokens) to attain this compact LLM. It demonstrates significant advantages over StableLM on multiple datasets. However, due to the fact that the LLaMA model has a deeper depth (32 layers) but smaller width, while StableLM is shallower (16 layers) but wider, StableLM-3B is slightly quicker than LLaMA-3B pruned by LLM-Pruner. The shallow but wide network is easier for parallel processing of matrix calculation, leading to better inference speeds compared to a deep but narrow network, as also discussed in [1]. \n\n| Pruning Ratio | #Param | Latency | #Tokens for Training | BoolQ | PIQA  | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA  | Average |\n|---------------|--------|---------|----------------------|-------|-------|-----------|------------|-------|-------|-------|---------|\n| StableLM-3B   | 3.6B   | 31.69s  | 800 Billion         | 48.78 | 69.48 | 44.52     | 54.62      | 50.93 | 25.17 | 27.40 | 45.84   |\n| LLaMA-3B  by LLM-Pruner   | 3.6B   | 37.96s  | 7.3 Million            | 61.41 | 70.08 | 51.01     | 55.01      | 46.80 | 30.38 | 37.40 | 50.30   |\n\n[1]DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter\n\n"}, {"review_id": "PkLeIglL22", "reviewer": "Reviewer_AFsw", "summary": "This paper tackles the problem of reducing the parameter number of large language models with pruning. \nIt identifies structures within an LLM and uses them to compose a dependency graph-based. \nThe authors provide experiments where they prune the parameters of LLMS and evaluate them on zero-shoot tasks.\nNamely, they consider LLaMA (7B), VICUNA (7B), and ChatGLM (6B).\nThe results show that pruning can reduce 20% of the parameters and retain 90-94% of the original performance.", "questions": "- How do you compare your approach with a distilling method (like Distilbert), where general competencies (MLM) are actually distilled from bigger models?\n", "limitations": " ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper addresses a crucial aspect of a more efficient language model - especially when models get enormous. \n- It employs efficient methods and evaluates the pruned models on a broad set of tasks. \n- The results show that more performance can be preserved than parameters removed during pruning. ", "weaknesses": "- As the authors show, pruning effectively reduces the parameter of LLMs. But how does the pruned model compare to the next smaller model variation? Showing that with the proposed method, we can get a better-pruned model (7B to 3B) compared to the next smaller model (3N) could make a strong point for this paper. \n- This paper considers only the same-sized LLMs, but from recent work, we know that emergent capabilities (when we can call them that) emerges with even bigger models. However, with this proposed approach, we are still determining how it scales to bigger models. Do you have more results on bigger models (13B, 20B, 40B) or insights on how efficient your approach is with a growing number of parameters? \n- While referring to efficiency gains in terms of GPU hours, the paper does not provide any further information about which GPU is used. Are the 14 GPU days of TinyBERT and the 3 GPU hours of the proposed approach comparable in terms of GPT hardware?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the reviewer for the constructive comments.\n\n---\n\n> As the authors show, pruning effectively reduces the parameter of LLMs. But how does the pruned model compare to the next smaller model variation? Showing that with the proposed method, we can get a better-pruned model (7B to 3B) compared to the next smaller model (3N) could make a strong point for this paper.\n\nWe thank the reviewer for the constructive suggestion.  We chose a well-known open-source 3B LLM, StableLM-3B, and compared it with a pruned LLaMA model with a similar parameter count. To ensure fairness, both models were fine-tuned on the Alpaca dataset. The experimental results of these two models are shown in the table below. As evident from the table, our model outperforms StableLM-3B. Additionally, we would like to note that the advantage of our approach lies in the ability to achieve a lightweight LLM with minimal overhead (7.3M vs. 800B tokens for training), which removes the necessity of training a full 3B LLM from scratch. We would include these results in the revised version.\n\n| Pruning Ratio | #Param | Latency | #Tokens for Training | BoolQ | PIQA  | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA  | Average |\n|-------|--------|---------|--------|-------|-------|--------|------|-------|-------|-------|---------|\n| StableLM-3B   | 3.6B   | 31.69s  | 800 Billion         | 48.78 | 69.48 | 44.52     | 54.62      | 50.93 | 25.17 | 27.40 | 45.84   |\n| LLaMA-3B by LLM-Pruner    | 3.6B   | 37.96s  | 7.3 Million            | 61.41 | 70.08 | 51.01     | 55.01      | 46.80 | 30.38 | 37.40 | 50.30   |\n\n---\n\n> This paper considers only the same-sized LLMs, but from recent work, we know that emergent capabilities (when we can call them that) emerges with even bigger models. However, with this proposed approach, we are still determining how it scales to bigger models. Do you have more results on bigger models (13B, 20B, 40B) or insights on how efficient your approach is with a growing number of parameters?\n\nThanks for the comment. We show the pruning results on LLaMA-13B in the table below. The training of the pruned LLaMA-13B takes 7 hours on a single A5000 with 24GB memory, while training a pruned LLaMA-7B takes ~4.5 hours on the same device. Most of the experimental results of LLaMA-13B are consistent with the results on LLaMA-7B. Notably, when comparing the outcomes of experiments conducted on 7B and 13B models, a distinction in parameter redundancy becomes very evident. After pruning 20% of the parameters, LLaMA-13B experiences a relatively minor drop of 3.18% in comparison to LLaMA-7B's decline of 5.86%. Following fine-tuning, the 13B model only exhibits a decrease of 0.95% in performance (from 64.97 to 64.02), outperforming the 3.43% decrease observed in the 7B model (from 63.25 to 59.82).\n\n| Pruning Ratio        | Method    | WikiText2 | PTB    | BoolQ | PIQA  | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA  | Average |\n|----------------------|-----------|-----------|--------|-------|-------|-----------|------------|-------|-------|-------|---------|\n| Ratio = 0%            | LLaMA-13B | 11.58     | 20.24  | 68.47 | 78.89 | 76.24     | 70.09      | 74.58 | 44.54 | 42.00 | 64.97   |\n| Ratio = 20% w/o tune | L2        | 61.15     | 91.43  | 61.50 | 67.57 | 52.90     | 57.54      | 50.13 | 31.14 | 36.80 | 51.08   |\n|                      | Random    | 19.24     | 31.84  | 63.33 | 73.18 | 63.54     | 60.85      | 64.44 | 36.26 | 38.00 | 57.09   |\n|                      | Channel   | 49.03     | 106.48 | 62.39 | 66.87 | 49.17     | 58.96      | 49.62 | 31.83 | 33.20 | 50.29   |\n|                      | Block     | 16.01     | 29.28  | 67.68 | 77.15 | 73.41     | 65.11      | 68.35 | 38.40 | 42.40 | 61.79   |\n| Ratio = 20% w/ tune  | L2        | 20.97     | 38.05  | 73.24 | 76.77 | 71.86     | 64.64      | 67.59 | 39.93 | 40.80 | 62.12   |\n|                      | Random    | 16.84     | 31.98  | 64.19 | 76.06 | 68.89     | 63.30      | 66.88 | 38.31 | 40.80 | 59.78   |\n|                      | Channel   | 17.58     | 29.76  | 69.20 | 76.55 | 68.89     | 66.38      | 62.08 | 38.99 | 39.60 | 60.24   |\n|                      | Block     | 15.18     | 28.08  | 70.31 | 77.91 | 75.16     | 67.88      | 71.09 | 42.41 | 43.40 | 64.02   |\n\n---\n\n> W3: While referring to efficiency gains in terms of GPU hours, the paper does not provide any further information about which GPU is used. Are the 14 GPU days of TinyBERT and the 3 GPU hours of the proposed approach comparable in terms of GPT hardware?\n\nWe apologize for the imprecise expression here. We will use the size of the dataset instead of GPU training duration to compare the training costs of these two methods. The revision is as follows: In the case of TinyBERT, it was trained for 3 epochs on the Wikipedia dataset with 2500M words, whereas our LLM-Pruner needs only 2 epochs of training on the alpaca dataset with only 5M words.\n\n---\n\n> Q1: How do you compare your approach with a distilling method (like Distilbert), where general competencies (MLM) are actually distilled from bigger models?\n\nThanks. We show the comparison results of DistilBERT and LLM-Pruner on LLaMA-7B here. LLM-Pruner outperforms DistilBERT by 4.24% on average, with even a smaller size. The reason lies in  that our approach aims to minimize model disruption during pruning, whereas DistilBERT merely selects one layer out of two. As a result, the model pruned by LLM-Pruner demands less data to recover its performance compared with DistilBERT, consequently achieving superior performance.\n\n| Pruning Ratio | #Param | BoolQ | PIQA  | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA  | Average |\n|---------------|--------|-------|-------|-----------|------------|-------|-------|-------|---------|\n| DistilBert    | 3.50B  | 54.77 | 63.28 | 40.44     | 52.49      | 43.31 | 27.22 | 31.00 | 44.64   |\n| LLM-Pruner    | 3.35B  | 61.56 | 68.72 | 46.62     | 52.64      | 47.94 | 29.27 | 35.40 | 48.88   |\n\n\n\n\n"}]

Author rebuttal:

Meta-review: The paper addresses a very important problem: pruning an LLM to a smaller, more efficient, and yet well performing model. The method is certainly useful, and the experiments are generally comprehensive. However, there are some important baselines that are not included in the submitted version of the paper. 

The authors did an amazing job in their rebuttals, conducting many new experiments to address the reviewers concerns, and do that satisfactorily in my opinion. But these are the results not currently in the paper. If they are added, the paper will present a comprehensive evaluation. I, therefore, recommend conditional acceptance of this paper provided that the new experiments from the rebuttal are adequately incorporated.
Decision: Accept (poster)