SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xinyin Ma Gongfan Fang Xinchao Wang*
maxinyin@u.nus.edu, gongfan@u.nus.edu, xinchao@nus.edu.sg
Large language models (LLMs) have shown remarkable capabilities in language understanding and generation. However, such impressive capability typically comes with a substantial model size, which presents significant challenges in both the deployment, inference, and training stages. With LLM being a general-purpose task solver, we explore its compression in a task-agnostic manner, which aims to preserve the multi-task solving and language generation ability of the original LLM. One challenge to achieving this is the enormous size of the training corpus of LLM, which makes both data transfer and model post-training over-burdensome. Thus, we tackle the compression of LLMs within the bound of two constraints: being task-agnostic and minimizing the reliance on the original training dataset. Our method, named LLM-Pruner, adopts structural pruning that selectively removes non-critical coupled structures based on gradient information, maximally preserving the majority of the LLM's functionality. To this end, the performance of pruned models can be efficiently recovered through tuning techniques, LoRA, in merely 3 hours, requiring only  $50K$  data. We validate the LLM-Pruner on three LLMs, including LLaMA, Vicuna, and ChatGLM, and demonstrate that the compressed models still exhibit satisfactory capabilities in zero-shot classification and generation. The code is available at: https://github.com/horseee/LLM-Pruner
Recently, Large Language Models (LLMs) [39, 51, 50, 44, 64, 4, 74] have demonstrated remarkable proficiency in language understanding and generation. With the increase in model size, they are better equipped to handle complex tasks [3, 5, 58, 60] and even exhibit emergent abilities [57]. However, notwithstanding their impressive performance, LLMs pose challenges in deployment and inference. Their extensive scale engenders substantial computational demands, and the multitude of parameters involved can induce long latencies and other related issues. Several techniques are proposed to solve these problems, like model pruning [56, 61, 72, 21], knowledge distillation [46, 41, 47], quantization [1, 13] within the context of pre-trained language model (PLM).
While previous methods have effectively maintained model performance amidst parameter reduction, they primarily target compression within specialized domains or for designated tasks in the context of task-specific compression. For instance, a PLM is fine-tuned on a particular dataset, such as one of the classification tasks in the GLUE benchmark [53], after which these models are distilled into a smaller classification model [46, 18]. Although this paradigm could potentially be employed for LLM compression, it compromises the LLM's capacity as a versatile task solver, rendering it suited to a single task exclusively.
Figure 1: Illustration of LLM-Pruner. (i) Task-specific compression: the model is fine-tuned then compressed on a specific task. (ii) TinyBERT: First distill the model on unlabeled corpus and then fine-tune it on the specific task. (iii) LLM-Pruner: Task-agnostic compression within 3 hours.
LLaMA-5.4B by LLM-Pruner
The Leaning Tower of Pisa is known for its unusual tilt, which is a result of a number of factors. When the tower was built in the twelfth century, the soil beneath it was extremely soft, allowing the buttresses to settle unevenly. This resulted in a tilt towards one side.
The Leaning Tower of Pisa is known for being tilted and unstable. However, its story is much more fascinating. Although construction began in 1173, the tower was never meant to be tilted. It simply became that way because it was built on unstable ground.
Thus, we strive to compress the LLM in a new setting: to reduce the LLM's size while preserving its diverse capabilities as general-purpose task solvers, as depicted in Figure 1. This introduces the task-agnostic compression of LLMs, which presents two key challenges:
- The size of the training corpus of the LLM is enormous. Previous compression methods heavily depend on the training corpus. The LLM has escalated the corpus scale to 1 trillion tokens or more [17, 51]. The extensive storage needs and protracted transmission times make the dataset difficult to acquire. Furthermore, if the dataset is proprietary, acquisition of the training corpus verges on impossibility, a situation encountered in [74, 39].
- The unacceptably long duration for the post-training of the pruned LLM. Existing methods require a substantial amount of time for post-training the smaller model [55, 28]. For instance, the general distillation in TinyBERT takes around 14 GPU days [20]. Even post-training a task-specific compressed model of BERT demands around 33 hours [61, 22]. As the size of both the model and corpus for LLMs increases rapidly, this step will invariably consume an even more extensive time.
To tackle the aforementioned challenges associated with the task-agnostic compression of LLMs, we introduce a novel approach called LLM-Pruner. Since our goal is to compress LLMs with reduced data dependency and expedited post-training, how to prune model with the minimal disruption to the origin is crucial. To accomplish this, we propose a dependency detection algorithm that identifies all the dependent structures within the model. Once the coupled structure is identified, we employ an efficient importance estimation strategy to select the optimal group for pruning under the task-agnostic setting, where the first-order information and an approximated hessian information is taken into account. Finally, a rapid recovery stage is executed to post-train the pruned model with limited data.
Contribution. In this paper, we propose a novel framework, LLM-Pruner, for the task-agnostic compression of the large language model. To the best of our knowledge, LLM-Pruner is the first framework designed for structured pruning of LLMs. We conclude the advantages of the LLM-Pruner as (i) Task-agnostic compression, where the compressed language model retains its ability to serve as a multi-task solver. (ii) Reduced demand for the original training corpus, where only 50k publicly available samples are needed for compression, significantly reducing the budget for acquiring the training data (iii) Quick compression, where the compression process ends up in three hours. (iv) An automatic structural pruning framework, where all the dependent structures are grouped without the need for any manual design. To evaluate the effectiveness of LLM-Pruner, we conduct extensive experiments on three large language models: LLaMA-7B, Vicuna-7B, and ChatGLM-6B. The compressed models are evaluated using nine datasets to assess both the generation quality and the zero-shot classification performance of the pruned models. The experimental results demonstrate that even with the removal of  $20\%$  of the parameters, the pruned model maintains  $94.97\%$  of the performance of the original model.
Compression of Language Model. Language models [9, 31, 25] have gained much attention and increase the need to reduce the size of parameters and reduce the latency [23, 48]. To compress the language model, previous works can be divided into several categories: network pruning [21, 63, 32, 15], knowledge distillation [46, 47, 40], quantization [68, 1, 71] and other techniques, like early exit [62] or dynamic token reduction [69]. We focus on the pruning of the language models, especially structural pruning [26]. Structural pruning removes the entire filter from the neural network, which is more hardware friendly. There are several ways to remove the structure, such as 11-dependent pruning [16, 72], first-order importance estimation [18], hessian-based estimation [21, 54] or the optimal brain surgeon [24, 21]. As for the pruning unit in structural pruning, some works adopt the entire layer [10] as the minimal unit, and others take the multi-head attention [52] or the feed-forward layers [18, 36] as the basic structure to prune. CoFi [61] studies the pruning unit in different granularity.
Efficient and Low Resource Compression. With the growing size of neural network models, there is an increasing demand for efficient and low-resource compression [67, 66, 30, 29, 65]. As for the efficient compression, [22] accelerate the post-training by defining the reconstruction error as a linear least squares problem. [13, 12] propose the layer-wise optimal brain surgeon. As for the constraint of availability of the training corpus, data-free pruning [45, 70] come up with several strategies to prune the model by measuring neurons' similarity. Besides, [34, 33, 42] proposes methods that distill the model without reliance on the training corpus of the model. However, those methods are too time-consuming, involving synthesizing samples by backpropagating the pre-trained models.
In this section, we provide a detailed explanation of LLM-Pruner. Following the conventional model compression pipeline[22], LLM-Pruner consists of three steps: (1) Discovery Stage (Section 3.1). This step focuses on identifying groups of interdependent structures within LLMs. (2) Estimation Stage (Section 3.2). Once the coupled structures are grouped, the second step entails estimating the contribution of each group to the overall performance of the model and deciding which group to be pruned. (3) Recover Stage (Section 3.3). This step involves fast post-training that alleviates potential performance degradation caused by the removal of structures.
3.1 Discover All Coupled Structure in LLMs
In light of the limited availability of data for post-training, it becomes imperative to prioritize the removal of structures with minimal damage when compressing the model. This underscores the dependency-based structural pruning, which ensures coupled structures are pruned in unison. We provide an experiment in Section 4.3 to show the importance of dependency-based structural pruning when compressing the large language model.
Structure Dependency in LLMs. Similar to [11], the pruning begins by building the dependency for LLMs. Assume  $N_{i}$  and  $N_{j}$  are two neurons in the model,  $\operatorname{In}(N_i)$  and  $\operatorname{Out}(N_i)$  represents all the neurons that point towards or point from  $N_{i}$ . The dependency between structures can be defined as:
$$
N _ {j} \in \operatorname {O u t} \left(N _ {i}\right) \wedge \operatorname {D e g} ^ {-} \left(N _ {j}\right) = 1 \Rightarrow N _ {j} \text {i s d e p e n d e n t o n} N _ {i} \tag {1}
$$
where  $\mathrm{Deg}^{-}(N_{j})$  represents the in-degree of neuron  $N_{j}$ . Noting that this dependency is directional, we can therefore correspondingly obtain another dependency:
$$
N _ {i} \in \operatorname {I n} \left(N _ {j}\right) \wedge \operatorname {D e g} ^ {+} \left(N _ {i}\right) = 1 \Rightarrow N _ {i} \text {i s d e p e n d e n t o n} N _ {j} \tag {2}
$$
where  $\mathrm{Deg}^{+}(N_{i})$  represents the out-degree of neuron  $N_{i}$ . The principle of dependency here is, if a current neuron (e.g.,  $N_{i}$ ) depends solely on another neuron (e.g.,  $N_{j}$ ), and the neuron  $N_{j}$  is subjected to pruning, it follows that the neuron  $N_{i}$  must also undergo pruning. We provide a detailed case in Appendix A.
Trigger the Dependency Graph. By having the definition of dependency, the coupled structures in the LLM can be analyzed automatically. Considering any neuron within the LLM as the initial
Figure 2: Illustration of the coupled structures in LLaMA. We simplify the neurons in each layer to make the dependent group clear. The trigger neuron, marked as a circle with a bell, cause weights with dependency pruned (dashed lines), which may propagate (red dashed lines) to coupled neurons (dashed circles). A group can be triggered by a variety of trigger neurons. Taking Group Type B as an example, the trigger for this group involves (i) the attention head, (ii) the output neuron in Query, Key or Value, and (iii) the input neuron in the final output projection.
trigger, it possesses the capability to activate neurons that depend on it. Subsequently, these newly triggered neurons can serve as the subsequent triggers to identify the dependency and activate their respective dependent neurons. This iterative process continues until no new neurons are detected. Those neurons then form a group for further pruning. Taking LLaMA as an example, by searching over all the neurons as the initial trigger, we can locate all the coupled structures, as shown in Figure2.
Given the diversity in the structure of different LLMs, manual analysis and removal of coupled structures in each LLM could be extremely time-consuming. However, by employing LLM-Pruner, all coupled structures can be automatically identified and extracted.
3.2 Grouped Importance Estimation of Coupled Structure
Till now, all coupled structures within the model are grouped. Weights within the same group should be pruned simultaneously, as partial pruning not only increases parameter size but also introduces misaligned intermediate representations. Therefore, we estimate the importance of the group as a whole, as opposed to evaluating the importance of modules. Given the limited access to the training dataset, we explore the use of public datasets or manually created samples as alternative resources. Although the domains of these datasets may not perfectly align with the training set, they still provide valuable information for assessing the importance.
Vector-wise Importance. Suppose that given a dataset  $\mathcal{D} = \{x_i, y_i\}_{i=1}^N$ , where  $\mathbf{N}$  is the number of samples. In our experiments, we set  $\mathbf{N}$  equal to 10 and we use some public datasets as the source of  $\mathcal{D}$ . A group (as previously defined as a set of coupled structures) can be defined as  $\mathcal{G} = \{W_i\}_{i=1}^M$ , where  $\mathbf{M}$  is the number of coupled structures in one group and  $W_i$  is the weight for each structure. While pruning, our goal is to remove the group that has the least impact on the model's prediction, which can be indicated by the deviation in the loss. Specially, to estimate the importance of  $W_i$ , the change in loss can be formulated as [24]:
$$
I _ {W _ {i}} = \left| \Delta \mathcal {L} (\mathcal {D}) \right| = \left| \mathcal {L} _ {W _ {i}} (\mathcal {D}) - \mathcal {L} _ {W _ {i} = 0} (\mathcal {D}) \right| = \left| \underbrace {\frac {\partial \mathcal {L} ^ {\top} (\mathcal {D})}{\partial W _ {i}}} _ {\neq 0} W _ {i} \right. - \frac {1}{2} W _ {i} ^ {\top} H W _ {i} + \mathcal {O} \left(\left\| W _ {i} \right\| ^ {3}\right) | \tag {3}
$$
where  $H$  is the hessian matrix. Here,  $\mathcal{L}$  represents the next-token prediction loss. The first term is typically neglected in prior work [24, 54, 12], as the model has already converged on the training dataset, where  $\partial \mathcal{L}^{\top} / \partial W_{i} \approx 0$ . However, since  $\mathcal{D}$  here is not extracted from the original training data, which means that  $\partial \mathcal{L}^{\top} / \partial W_{i} \not\approx 0$ . This presents a desirable property for determining the importance of  $W_{i}$  by the gradient term under LLMs, since computation of the second term, the Hessian matrix, on the LLM is impractical with  $\mathcal{O}(N^{2})$  complexity.
Element-wise Importance. The above can be considered as an estimate for the weight  $W_{i}$ . We can derive another measure of importance at a finer granularity, where each parameter within  $W_{i}$  is assessed for its significance:
$$
I _ {W _ {i} ^ {k}} = \left| \Delta \mathcal {L} (\mathcal {D}) \right| = \left| \mathcal {L} _ {W _ {i} ^ {k}} (\mathcal {D}) - \mathcal {L} _ {W _ {i} ^ {k} = 0} (\mathcal {D}) \right| = \left| \frac {\partial \mathcal {L} (\mathcal {D})}{\partial W _ {i} ^ {k}} W _ {i} ^ {k} - \frac {1}{2} W _ {i} ^ {k} H _ {k k} W _ {i} ^ {k} + \mathcal {O} \left(\left\| W _ {i} ^ {k} \right\| ^ {3}\right) \right| \tag {4}
$$
Here,  $k$  represents the k-th parameter in  $W_{i}$ . The diagonal of the hessian  $H_{kk}$  can be approximated by the Fisher information matrix, and the importance can be defined as:
$$
I _ {W _ {i} ^ {k}} = \left| \mathcal {L} _ {W _ {i} ^ {k}} (\mathcal {D}) - \mathcal {L} _ {W _ {i} ^ {k} = 0} (\mathcal {D}) \right| \approx \left| \frac {\partial \mathcal {L} (\mathcal {D})}{\partial W _ {i} ^ {k}} W _ {i} ^ {k} - \frac {1}{2} \sum_ {j = 1} ^ {N} \left(\frac {\partial \mathcal {L} (\mathcal {D} _ {j})}{\partial W _ {i} ^ {k}} W _ {i} ^ {k}\right) ^ {2} + \mathcal {O} \left(\left\| W _ {i} ^ {k} \right\| ^ {3}\right) \right| \tag {5}
$$
Group Importance. By utilizing either  $I_{W_i^k}$  or  $I_{W_i}$ , we estimate the importance at the granularity of either a parameter or a vector of weight. Remembering that our goal is to estimate the importance of  $\mathcal{G}$ , we aggregate the importance scores in four ways: (i) Summation:  $I_{\mathcal{G}} = \sum_{i=1}^{M} I_{W_i}$  or  $I_{\mathcal{G}} = \sum_{i=1}^{M} \sum_{k} I_{W_i^k}$ , (ii) Production:  $I_{\mathcal{G}} = \prod_{i=1}^{M} I_{W_i}$  or  $I_{\mathcal{G}} = \prod_{i=1}^{M} \sum_{k} I_{W_i^k}$ , (iii) Max:  $I_{\mathcal{G}} = \max_{i=1}^{M} I_{W_i}$  or  $I_{\mathcal{G}} = \max_{i=1}^{M} \sum_{k} I_{W_i^k}$ ; (iv) Last-Only: Since deleting the last executing structure in a dependency group is equivalent to erasing all the computed results within that group, we assign the importance of the last executing structure as the importance of the group:  $I_{\mathcal{G}} = I_{W_l}$  or  $I_{\mathcal{G}} = \sum_{k} I_{W_l^k}$ , where  $l$  is the last structure. After assessing the importance of each group, we rank the importance of each group and prune the groups with lower importance based on a predefined pruning ratio.
3.3 Fast Recovery with Low-rank Approximation
In order to expedite the model recovery process and improve its efficiency under limited data, it is crucial to minimize the number of parameters that need optimization during the recovery phase. To facilitate this, we employ the low-rank approximation, LoRA[19], to post-train the pruned model. Each learnable weight matrix in the model, denoted as  $W$ , encompassing both pruned and unpruned linear projection in the LLM, can be represented as  $W$ . The update value  $\Delta W$  for  $W$  can be decomposed as  $\Delta W = PQ \in \mathbb{R}^{d^{-} \times d^{+}}$ , where  $P \in \mathbb{R}^{d^{-} \times d}$  and  $Q \in \mathbb{R}^{d \times d^{+}}$ . The forward computation can now be expressed as:
$$
f (x) = (W + \Delta W) X + b = (W X + b) + (P Q) X \tag {6}
$$
where  $b$  is the bias in the dense layer. Only training  $P$  and  $Q$  reduces the overall training complexity, reducing the need for large-scale training data. Besides, the extra parameters  $P$  and  $Q$  can be reparameterized into  $\Delta W$ , which would not cause extra parameters in the final compressed model.
Foundation Large Language Model. To showcase the effectiveness and versatility of LLM-Pruner, we test it over three open-source large language models with two kinds of structure: LLaMA-7B [51], Vicuna-7B [4]² and ChatGLM-6B [74].
Evaluation and Datasets. To assess the performance of the model in the task-agnostic setting, we follow LLaMA's evaluation to perform zero-shot task classification on common sense reasoning datasets: BoolQ [6], PIQA [2], HellaSwag [73], WinoGrande [43], ARC-easy [7], ARC-challenge [7] and OpenbookQA [38]. Follow [14], the model ranks the choices in the multiple choice tasks or generates the answer in the open-ended generation  $^{3}$ . Additionally, we complement our evaluation with a zero-shot perplexity (PPL) analysis on WikiText2 [37] and PTB [35].
$^{2}$ https://huggingface.co/lmsys/ Vicuna-7b-delta-v0
<sup>3</sup>https://github.com/EleutherAI/lm-evaluation-harness
Table 1: Zero-shot performance of the compressed LLaMA-7B. The average is calculated among seven classification datasets. 'Underline' indicates the best pruning-only performance, while 'bold' represents the overall best performance with the same pruning ratio, considering both pruning and post-training. The 'Channel' strategy only prunes the dependent group of Type C, while all other methods employ the 'Block' strategy to prune dependent groups in both Type A and Type B. Since [51] did not provide its prompt, the evaluation of the result with  $\star$  is performed under different prompts, which is lower than the official results.
Pruning Ratio	Method	WikiText2↓	PTB↓	BoolQ	PIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Average
Ratio = 0%	LLaMA-7B[51]	-	-	76.5	79.8	76.1	70.1	72.8	47.6	57.2	68.59
LLaMA-7B*	12.62	22.14	73.18	78.35	72.99	67.01	67.45	41.38	42.40	63.25
Ratio = 20% w/o tune	L2	582.41	1022.17	59.66	58.00	37.04	52.41	33.12	28.58	29.80	42.65
Random	27.51	43.19	61.83	71.33	56.26	54.46	57.07	32.85	35.00	52.69
Channel	74.63	153.75	62.75	62.73	41.40	51.07	41.38	27.90	30.40	45.38
Vector	22.28	41.78	61.44	71.71	57.27	54.22	55.77	33.96	38.40	53.25
Element2	19.77	36.66	59.39	75.57	65.34	61.33	59.18	37.12	39.80	56.82
Element1	19.09	34.21	57.06	75.68	66.80	59.83	60.94	36.52	40.00	56.69
Ratio = 20% w/ tune	Channel	22.02	38.67	59.08	73.39	64.02	60.54	57.95	35.58	38.40	55.57
Vector	18.84	33.05	65.75	74.70	64.52	59.35	60.65	36.26	39.40	57.23
Element2	17.37	30.39	69.54	76.44	68.11	65.11	63.43	37.88	40.00	60.07
Element1	17.58	30.11	64.62	77.20	68.80	63.14	64.31	36.77	39.80	59.23
Table 2: Zero-shot performance of the compressed LLaMA-13B. Here we adopt Element<sup>1</sup> as the importance estimation for 'Channel' and 'Block'.
Pruning Ratio	Method	WikiText2↓	PTB↓	BoolQ	PIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Average
Ratio = 0%	LLaMA-13B*	11.58	20.24	68.47	78.89	76.24	70.09	74.58	44.54	42.00	64.97
Ratio = 20%	L2	61.15	91.43	61.50	67.57	52.90	57.54	50.13	31.14	36.80	51.08
w/o tune	Random	19.24	31.84	63.33	73.18	63.54	60.85	64.44	36.26	38.00	57.09
Channel	49.03	106.48	62.39	66.87	49.17	58.96	49.62	31.83	33.20	50.29
Block	16.01	29.28	67.68	77.15	73.41	65.11	68.35	38.40	42.40	61.79
Ratio = 20%	L2	20.97	38.05	73.24	76.77	71.86	64.64	67.59	39.93	40.80	62.12
w/ tune	Random	16.84	31.98	64.19	76.06	68.89	63.30	66.88	38.31	40.80	59.78
Channel	17.58	29.76	69.20	76.55	68.89	66.38	62.08	38.99	39.60	60.24
Block	15.18	28.08	70.31	77.91	75.16	67.88	71.09	42.41	43.40	64.02
Implementation Details. In the model pruning process, we use 10 randomly selected samples from Bookcorpus [75], each truncated to a sequence length of 128, as the calibration samples for establishing dependency and calculating the gradient for both LLaMA and Vicuna. For ChatGLM, we select 10 random samples from DailyDialog [27]. During the recovery phase, we utilize the cleaned version of Alpaca [49], which comprises approximately 50k samples. Remarkably, tuning these samples requires merely 3 hours on a single GPU with only 2 epochs. More hyper-parameters of pruning and training can be found in Appendix B.
Statistics of the Compressed Model. Table 3 presents the statistic of the 7B models that are used in our experiments: the parameter count, MACs, memory requirements and latency for running each model. The statistical evaluation is conducted using the infer
Table 3: Statistics of the base model and the compressed model.
Model	Strategy	Ratio	#Params	#MACs	Memory	Latency
LLaMA-7B	-	-	6.74B	424.02G	12884.5MiB	69.32s
Channel	20%	5.39B	339.36G	10363.6MiB	61.50s
Block	20%	5.42B	339.60G	10375.5MiB	58.55s
Vicuna-7B	Channel	50%	3.37B	212.58G	6556.3MiB	40.11s
Block	50%	3.35B	206.59G	6533.9MiB	37.54s
ence mode, where the model is fed a sentence consisting of 64 tokens. The latency is tested under the test set of WikiText2 on a single A5000. Here, the 'Block' strategy implies that the pruned unit in the model consists of Group Type A and Group Type B as illustrated in Figure 2, whereas 'Channel' indicates that the unit to be pruned is Group Type C. We delve into an analysis of these two choices in Section 4.2(Channel Strategy vs. Block Strategy). The pruning ratio stated here denotes the approximate ratio of parameters to be pruned since the number of parameters within each pruned structure does not perfectly match the total number of pruned parameters.
4.2 Zero-shot Performance