SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1,2,4 and 5 shows the zero-shot performance of the pruned model. Based on the evaluation conducted on LLaMA, employing a  $20\%$  parameter reduction without post-training, the pruned
Table 4: Zero-shot performance of the compressed Vicuna-7B
Pruned Model	Method	WikiText2↓	PTB↓	BoolQ	PIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Average
Ratio = 0%	Vicuna-7B	16.11	61.37	76.57	77.75	70.64	67.40	65.11	41.21	40.80	62.78
Ratio = 20% w/o tune	l2	3539.98	5882.21	55.90	56.15	32.37	51.85	30.01	28.41	28.20	40.41
random	34.63	112.44	61.47	70.89	54.67	56.27	55.60	31.74	34.60	52.18
Channel	71.75	198.88	51.77	63.93	42.58	55.17	43.94	29.27	33.40	45.72
Vector	27.03	92.51	62.17	71.44	55.80	53.43	55.77	33.28	37.80	52.81
Element2	24.70	94.34	62.87	75.41	64.00	58.41	60.98	37.12	39.00	56.83
Element1	25.74	92.88	61.70	75.30	63.75	56.20	63.22	36.60	37.00	56.25
Ratio = 20% w/ tune	Vector	19.94	74.66	63.15	74.59	61.95	60.30	60.48	36.60	39.40	56.64
Element2	18.97	76.78	60.40	75.63	65.45	63.22	63.05	37.71	39.00	57.78
Element1	19.69	78.25	63.33	76.17	65.13	60.22	62.84	37.12	39.20	57.71
model manages to retain  $89.8\%$  of the performance exhibited by the unpruned model. Furthermore, through the efficient post-training, the classification accuracy further improves to  $60.07\%$ , achieving  $94.97\%$  of the accuracy attained by the original model. This demonstration proves the feasibility of using LLM-Pruner to effectively compress the model, even without relying on training data, and within a remarkably short period of time. Surprisingly, we discover that on most datasets, the pruned model with 5.4B LLaMA even outperformed chatGLM-6B. This highlights the superiority of the LLM-Pruner: if a smaller model with a customized size is required, LLM-Pruner is more cost-effective compared to retraining another model with a satisfying performance. However, with  $50\%$  parameters pruned, a large accuracy degradation is observed (see Appendix C.4). Compressing LLMs under high compression rates still remains a large challenge.
The compression results of Vicuna-7B align with those of LLaMA, as pruning  $20\%$  of parameters on Vicuna-7B maintains performance at  $92.03\%$  of the original model. We test a smaller pruning rate of  $10\%$  on chatGLM-7B, where the pruned model only experiences a marginal performance decrease of  $0.89\%$ , which can be recovered through post-training. Despite the pruned model outperforming the uncompressed model, we don't assert it is better than the original model. This is largely because chatGLM-6B, a bilingual model, has limited English pre-training exposure. Post-training, however, introduces it to more English corpus, albeit limited, improving its English comprehension.
Ablation: Impact of Importance Estimation. We conduct tests on all proposed importance estimation techniques mentioned in Section 3.2. The results can be found in Table 1 and 4. Here,  $\text{Element}^{\text{n}}$  represents the importance evaluation utilizing the n-th order term in Eq.5. Vector represents the result corresponding to Eq.3. Based on the results obtained from LLaMA-7B and Vicuna-7B, pruning algorithms achieved the best average performance mostly by leveraging the second-order derivatives for each parameter. Nonetheless, given that first-order derivatives are considerably more efficient than second-order derivatives, though yielding slightly inferior results, we still vote for the first-order term as a competitive method. Besides, the results on chatGLM-7B differed significantly from these findings. The importance estimation on each parameter fails, performing even worse than 12, while the importance estimation on the weight matrix reaches the best performance.
Ablation: Channel Strategy vs. Block Strategy. From the results presented in Table 2, it is evident that pruning 'Channel' significantly deteriorates performance compared to pruning 'Block'. This discrepancy arises because the layers within the stacked transformer do not evenly distribute their importance. As shown in Figure 3, the first and last layers have a profound impact on the model's performance, and pruning them results in more substantial performance degradation compared to other layers. However, due to the uniform treatment of the 'Channel' group across all layers, it becomes inevitable to prune the first and last layers, leading to a significant decline in performance.
Figure 3: Layer sensitivity for Pruning: Removing Groups in only one layer.
Table 5: Zero-shot Performance of the compressed ChatGLM-6B
Pruned Model	Method	PIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Average
Ratio = 0%	ChatGLM-6B	67.95	46.37	52.33	48.36	29.95	37.40	47.05
Ratio = 10%	L2	61.97	37.22	49.72	42.05	28.24	35.40	42.43
w/o tune	Random	65.29	43.18	51.30	47.52	29.52	34.60	45.24
Vector	66.32	43.51	53.04	47.56	30.72	35.80	46.16
Element1	54.35	28.07	50.59	27.82	24.66	33.20	36.45
w/ tune	Vector	67.74	46.35	53.99	51.01	29.95	35.00	47.34
Figure 4: The pruning results on LLaMA-7B (left) and Vicuna-7B (right) with different pruning rates.
Figure 5: Perplexity on zero-shot datasets across varying steps.
4.3 More Analysis
Impact of Different Pruning Rates. We investigate the impact of pruning the LLM at various pruning ratios in Figure 4. We compare our pruning results with the L2 strategy because L2 is also a data-free pruning algorithm. It is observed in the experiment of LLaMA that when the pruning ratio reaches approximately  $20\%$ , the magnitude-dependent algorithm experiences a rapid collapse, leading to the loss of information. Conversely, by employing LLM-Pruner, we are able to increase the pruning ratio to around  $60\%$  while achieving an equivalent perplexity level. Furthermore, in the case of Vicuna-7B, removing  $10\%$  parameters results in a performance decline equivalent to that of LLM-Pruner with  $60\%$ . The utilization of LLM-Pruner enables a significant increase in the number of model parameters that can be pruned, thereby substantially reducing computational overhead.
Tuning on the External Dataset. To tune the pruned model, we utilize the external dataset Alpaca [49]. The evaluation curves of the pruned model on two zero-shot datasets during the post-training process are depicted in Figure 5. The results demonstrate a rapid decrease in the perplexity of the pruned model within 300 steps, followed by a gradual increase. We provide a more comprehensive evaluation in Appendix C.3. It is important to note that if the model is trained for an excessive number of steps, it runs the risk of overfitting the external dataset, potentially compromising its performance in other general-purpose tasks.
Impact of Dependency-based Structured Pruning. To study the importance of dependency-based structural pruning, we conduct an experiment to disrupt dependencies within groups, where each weight matrix  $W_{i}$  is pruned solely based on the importance score estimated on itself. Table 6 presents the results demonstrating the impact of dependencies in structural pruning. In the absence of dependencies, the model nearly fails in the zero-shot generation and classification tasks. Even with tuning, the model fails to recover, showing a substantial difference compared to the results in dependency-based pruning.
Impact of Different Aggregation Strategies. We conduct tests on the aggregation algorithms proposed in Section 3.2. Our experimental results unveil notable discrepancies in model performance across different aggregation strategies, with particular emphasis on the 'Last-only' strategy. Among the evaluated approaches, the 'Max' strategy attains the most favorable outcomes in terms of perplexity, signifying enhanced coherence and fluency in sentence generation. However, it is important to note that the 'Max' strategy exhibits the poorest zero-shot classification results compared to all four strategies. Conversely, the 'Last-only' strategy showcases superior classification performance but
Table 6: Effect of the dependency-based structural pruning. Average represents the average performance on 7 classification datasets.
Method	WikiText2↓	PTB↓	Average↑
w/o Tuning	w/o dependency	68378.42	79942.47	38.32
w/ dependency	19.09	34.21	56.69
w/ Tuning	w/o dependency	13307.46	13548.08	38.10
w/ dependency	17.58	30.11	59.23
Table 7: Impact of different aggregation strategies on group importance estimation. Experiments are performed on LLaMA-7B.
Method	WikiText2↓	PTB↓	ARC-c↑	PIQA↑	OBQA↑
Summation	66.13	164.25	40.70	63.49	34.80
Max	62.59	144.38	39.60	63.71	34.60
Production	77.63	192.88	37.84	62.08	35.00
Last-only	130.00	170.88	41.92	64.75	35.20
suffers from the poorest generation quality. In our experiments, we make a trade-off by selecting the 'Summation' strategy since it shows both good generalization quality and classification performance.
Comparison with DistilBERT We show the comparison results of DistilBERT and LLM-Pruner on LLaMA-7B in Table 8. LLM-Pruner outperforms DistilBERT by  $4.24\%$  on average with even a smaller size. The reason lies in that LLM-Pruner minimizes model disruption during pruning, whereas DistilBERT merely selects one layer out of two. As a result, the model pruned by LLM-Pruner demands less data to recover its performance compared with DistilBERT, consequently achieving superior performance.
Table 8: DistilBert vs. LLM-Pruner. The average here means the average score on the above seven datasets.
Pruning Ratio	#Param	Average
DistilBert	3.50B	44.64
LLM-Pruner	3.35B	48.88
Scratch Training vs. Pruning. We compare LLM-Pruner with StableLM-3B<sup>4</sup> with a similar parameter size. To ensure fairness, both models are fine-tuned on the Alpaca dataset. The experimental results of these two models are shown in the Table 9. LLM-Pruner crafts lightweight LLMs with low resources, and even can sometimes achieve better performance than LLMs from scratch training. However, we also acknowledge that the LLaMA-3B obtained by LLM-Pruner will not always outperform other 3B models from scratch training, due to the huge gap in the size of training corpus.
Table 9: Scratch Training (StableLM-3B) vs. Pruning (LLaMA-3B by LLM-Pruner)
Pruning Ratio	#Param	Latency	BoolQ	PIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Average
StableLM-3B	3.6B	31.69s	48.78	69.48	44.52	54.62	50.93	25.17	27.40	45.84
LLaMA-3B	3.6B	37.96s	61.41	70.08	51.01	55.01	46.80	30.38	37.40	50.30
More Data for Recovery Despite our primary experiments being conducted using 50k samples, we remain convinced that the inclusion of additional data could substantially enhance the recovery process, albeit at a considerably higher computational cost. Consequently, we conduct an experiment aimed at model recovery with more data, employing a dataset comprising 2.59 million samples [59]. The results are detailed in Table 10. From the results, it is evident that the performance of the compressed model closely approximates that of the base model, exhibiting only a marginal performance decrease of  $0.89\%$ .
Table 10: Model Recovery:  ${50}\mathrm{k}$  samples vs.  ${2.59}\mathrm{M}$  samples
Model	#Samples	BoolQ	PIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Average
LLaMA-7B	-	73.18	78.35	72.99	67.01	67.45	41.38	42.40	63.25
LLaMA-5.4B	50k [49]	64.62	77.20	68.80	63.14	64.31	36.77	39.80	59.23
LLaMA-5.4B	2.59M [59]	76.57	77.37	66.60	65.82	70.62	40.70	38.80	62.36
Case Study. We provide some examples of sentences generated by the model compressed using LLM-Pruner in Table 11. We made efforts to ensure a minimal overlap between these generated sentences and the information contained in the tuning corpus, which demonstrates that the information originates from the original model rather than the tuning corpus. We provide additional examples in the Appendix, including the generated sentences of the model without post-training. From the
4https://huggingface.co/stabilityai/stablelm-tuned-alpha-3b
Table 11: Some examples of generations obtained with the original model and the pruned model.
Model	Original Model	Pruned Model
LLaMa	The universe is the entirety of space, time, matter, and energy that exists. It includes all matter and energy, whether visible or not, and in fact the whole universe is invisible. There are many theories regarding its origin and ultimate fate...	The universe is the entirety of space, time, matter, and energy that exists. The laws that make up physics — electrons orbiting atoms, magnets attracting other objects, planets orbiting the Sun—have been consistent over ...
Vicuna	10 steps to build an ios app: 1. Research and develop the idea. 2. Decide on the platform you want to use (iOS, Android, or both). 3. Create a plan and budget. 4. Hire a team of developers. 5. Design the user interface (UI). 6. Develop the user interface (UI). 7. Test and iterate on the app. 8. Submit the app to the app store. 9. Optimize the app for performance and user experience (UX). 10. Market and promote the app.	10 steps to build an ios app: 1. Plan your app: decide what it should do, what it will look like, and what it will cost to build it. 2. Hire a developer: A good iOS developer will help you design the user interface, create the app, and make sure it's bug-free. 3. Create a user interface: This is where the magic happens. Your app's user interface will be designed and mocked up at this point...
ChatGLM	Write a poetry about love and peace. In the midst of life's hustle and bustle, I find your love a soothing peace. It is a feeling that cannot be conveyed, A connection that cannot be distanced. With your smile, your eyes, your embrace, I feel a sense of belonging that only you can give. It is a love that lasts forever, A love that cannot be defeat...	Write a poetry about love and peace. Love and peace, two such wonderful things\A pure and endless desire in my heart\And both of them, I must seek for\A long, long time, I know.\Love, I know, is a feeling of being\A perfect partner, in every sense\And peace, I need it, so much, one day\A long, long way, my heart will go...
cases in Table 11, it is evident that the sentences generated by the compressed model are comparable to those produced by the original model. They exhibit fluency, relevance, and informativeness regarding the given topic. Nevertheless, during our experiments, we observed that the pruned model's performance deviates from that of the original model, particularly when generating lengthy sentences. Occasionally, it may generate sentences that are meaningless or contain repetitive tokens.
In this paper, we propose LLM-Pruner, a structured pruning approach for large language models. LLM-Pruner aims to compress sizable language models in a task-agnostic manner while minimizing the dependency on the original training corpus and preserving the linguistic capabilities of LLMs. LLM-Pruner accomplishes this by iteratively examining each neuron within the model as a trigger for identifying dependency groups, thereby constructing the LLM's dependency graph. Subsequently, LLM-Pruner assesses the importance of these groups using both parameter-wise and weight-wise estimation. Finally, we utilize LoRA for fast recovery and adjustment of the pruned model. We evaluate the efficacy of LLM-Pruner on three distinct models—LLaMA, Vicuna, and ChatGLM—utilizing various zero-shot datasets. Our experimental results indicate that LLM-Pruner successfully prunes the model, reducing computational burden while retaining its zero-shot capabilities. Nevertheless, considerable performance degradation occurs when employing high pruning rates, such as the removal of  $50\%$  of LLaMA's parameters, resulting in a substantial decline in model performance. Additionally, we observe instances in which the model generates incoherent sentences. Addressing the challenges associated with compressing LLMs at higher pruning rates remains a challenging task.
This project is supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (Award Number: MOE-T2EP20122-0006), and the National Research Foundation, Singapore, under its Medium Sized Center for Advanced Robotics Technology Innovation.
A Detailed Explanations for the Dependency Rules
Figure 6: Illustrations of the two dependency rules. All the cases are extracted from the multi-head attention module.
Group B: Multi-head Attention
We provide a detailed explanation of the two dependency rules. It is important to note that these dependency rules do not pertain solely to the forward computation. Instead, they represent directional relationships that exist in both directions. For instance, removing a node in a subsequent layer may also result in the pruning of a node in the preceding layer. Recall the two dependency rules as follows:
$$
N _ {j} \in \operatorname {O u t} \left(N _ {i}\right) \wedge \operatorname {D e g} ^ {-} \left(N _ {j}\right) = 1 \Rightarrow N _ {j} \text {i s d e p e n d e n t o n} N _ {i} \tag {7}
$$
$$
N _ {i} \in \operatorname {I n} \left(N _ {j}\right) \wedge \operatorname {D e g} ^ {+} \left(N _ {i}\right) = 1 \Rightarrow N _ {i} \text {i s d e p e n d e n t o n} N _ {j} \tag {8}
$$
where  $N_{i}$  and  $N_{j}$  are two neurons.  $\operatorname {In}(N_i)$  and  $\mathrm{Out}(N_i)$  represents all the neurons that point towards or point from  $N_{i}$ .  $\mathrm{Deg}^{-}(N_{i})$  and  $\mathrm{Deg}^{+}(N_{i})$  represents the in-degree and out-degree of neuron  $N_{i}$ . Figure 6 serves as an illustration of the two dependency rules:
- In case 1, Node I and Node J satisfy the rule stated in Eq.7. Consequently, Node J depends on Node I. When Node I is pruned, it is necessary to prune Node J as well.
- In case 2, Node I and Node J satisfy the rule Eq.8. Thus, Node I is dependent on Node J. If Node J is pruned, it becomes imperative to prune Node I as well.
- In case 3, Node J and Node K do not meet the requirement of Eq.7 due to the mismatch in  $\mathrm{Deg}^{-}(N_{k})\neq 1$ . Thus, with Node J pruned, Node K would not be affected.
B.1 For Pruning
For the baseline Given the lack of previous work on the structural pruning of Large Language Models in a task-agnostic and low-resource setting, there is currently no existing baseline for our model. To provide a comprehensive demonstration of the effectiveness of LLM-Pruner, we employ two additional methods for evaluation, alongside the data-free pruning method. All of these methods are built upon the dependent groups identified in Section 3.1:
- L2: We assess the importance of each group based on the magnitude of its weight matrix.
- Random: This method involves randomly selecting certain groups for pruning.
For the 'Block' Group. Based on the findings presented in Table 3, it is preferable to leave the first three layers and the final layer unchanged, as modifying parameters in those layers significantly impacts the model. Within each module, such as the MLP or the Multi-head Attention, the discovered groups are pruned based on a pre-set ratio. For instance, in the MLP layer of LLaMA-7B, we identified 11,008 groups, and with a  $25\%$  pruning ratio, the module would prune 2,752 groups. It is worth noting that the pruning rate for the selected groups is higher than the pruning ratio for the parameters, as certain layers (e.g., the embedding layer and excluded layers mentioned) retain their parameters. When aiming for a parameter pruning ratio of  $20\%$ , we prune  $25\%$  from Layer 5 to Layer 30. Similarly, for a  $50\%$  parameter removal, we prune  $60\%$  of the groups from Layer 4 to Layer 30.
For the 'Channel' Group. The Group 'Channel' exhibits a resemblance to dimension pruning in the model, targeting the pruning of certain dimensions. In the case of the Query, Key, and Value projection in MHA, only the input dimension is pruned, while for the Output projection in MHA, only the output dimension is pruned. It is important to note that the entire dependency is established automatically, without any manual design involved. The 'Channel' group operates in a complementary manner to the 'Block Group'. In the 'Channel' Group, the pruning ratio of the group equals to the pruning ratio of the parameters, as all weight matrices, including the embedding matrix, undergo pruning. Therefore, a  $20\%$  pruning ratio of parameters implies pruning  $20\%$  of the groups, while a  $50\%$  pruning ratio implies pruning  $50\%$  of the groups.
B.2 For Recovery Stage
We follow [19] in our recovery stage. We set the rank  $d$  to 8 in our experiment. The learning rate is set to 1e-4 with 100 warming steps. The batch size of training is selected from \{64, 128\} and the AdamW optimizer is employed in our experiment. The best training epoch we found is 2 epochs, as training with more epochs even has a negative impact on the model performance. We run our experiment on a single GPU with 24GB memory, using approximately 2.5 hours if RTX4090 is utilized. All the linear module is taken into account for efficient tuning. An ablation experiment for this is shown in Table 12.
Table 12: Ablation: Tuning different modules in the recovery stage
Module	WikiText ↓	PTB ↓
ALL	17.36	29.99
- MLP	17.64	30.63
- MHA	17.62	30.23
C.1 Pruning vs. Quantization
Here, we conduct a comparative analysis of different compression techniques and illustrate that these techniques can be effectively combined with little performance degradation. We have chosen LLM.int8() [8] as a representative example of quantization methods. Our results show that LLM.int8() outperforms LLM-Pruner while LLM-Pruner enhances latency, reduces parameter size. When these two techniques are applied in tandem, they collectively reduce memory consumption and expedite inference, offering a balanced approach that combines the benefits of both methods.
Table 13: Pruning and Quantization on LLaMA-7B
Pruning Ratio	#Param	Memory	Latency	BoolQ	PIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Average
LLaMA-7B	6.74B	12884.5MiB	69.32s	73.18	78.35	72.99	67.01	67.45	41.38	42.40	63.25
LLM.int8()	6.74B	6777.7MiB	76.20s	73.36	78.18	73.01	66.93	67.47	40.87	41.80	63.09
LLaMA-5.4B	5.47B	10488.4MiB	58.55s	76.57	77.37	66.60	65.82	70.62	40.70	38.80	62.36
LLaMA-5.4B + LLM.int8()	5.47B	5444.37MiB	63.10s	76.39	76.71	66.62	66.46	70.54	40.19	39.20	62.30
C.2 Global Pruning vs. Local Pruning
we present a comparative analysis between global pruning and local pruning, where the pruning ratio is  $20\%$  and the base model is LLaMA-7B. Global pruning refers to ranking all groups in the model together, whereas local pruning involves only ranking groups within the same module for pruning. The outcome of global pruning leads to varying widths across different layers and modules, whereas local pruning ensures uniformity across all layers.
Based on our experimental findings, we observed a slight advantage of local pruning over global pruning. We think this is because of the varying magnitudes in different layers or modules, which makes the importance scores incomparable between groups across different layers.
Table 14: Results of global pruning and local pruning
Method	WikiText2 ↓	PTB↓	BoolQ	PIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Average
Element1 - local	19.09	34.21	57.06	75.68	66.80	59.83	60.94	36.52	40.00	56.69
Element1 - global	20.84	32.86	63.15	73.23	63.31	66.38	55.85	35.49	38.00	56.49
C.3 Overfitting Phenomena in Post-Training
We present a comprehensive analysis of the overfitting issue in the recovery stage, as previously mentioned in Figure 5. Here the results cover all 9 datasets across various training steps. Based on the findings presented in Table 15, a noticeable trend emerges: the accuracy or generation quality initially shows improvement but subsequently experiences a slight decline. This pattern suggests that the recovery process is completed within a short period. And given that the training corpus is domain-constrained, more training epochs can result in overfitting to the specific dataset while potentially compromising the original capabilities of the language model.