SlimGPT: Layer-wise Structured Pruning for Large Language Models
Authors: Gui Ling
Ziyang Wang
YuliangYan
Qingwen Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: Model Compression
Structured Pruning
Layer-wise Pruning
Large Language Model
Primary area: natural_language_processing

Abstract: Large language models (LLMs) have garnered significant attention for their remarkable capabilities across various domains, whose vast parameter scales present challenges for practical deployment. Structured pruning is an effective method to balance model performance with efficiency, but performance restoration under computational resource constraints is a principal challenge in pruning LLMs. Therefore, we present a low-cost and fast structured pruning method for LLMs named SlimGPT based on the Optimal Brain Surgeon framework. We propose Batched Greedy Pruning for rapid and near-optimal pruning, which enhances the accuracy of head-wise pruning error estimation through grouped Cholesky decomposition and improves the pruning efficiency of FFN via Dynamic Group Size, thereby achieving approximate local optimal pruning results within one hour. Besides, we explore the limitations of layer-wise pruning from the perspective of error accumulation and propose Incremental Pruning Ratio, a non-uniform pruning strategy to reduce performance degradation. Experimental results on the LLaMA benchmark show that SlimGPT outperforms other methods and achieves state-of-the-art results.

SlimGPT: Layer-wise Structured Pruning for Large Language Models
Gui Ling, Ziyang Wang, Yuliang Yan', Qingwen Liu
Alibaba Group
Flinggui.1g shanyl.way wzy, yuliang yyl, xiangsheng 1twl ierabba-nn...o inc. com
Abstract
Large language models (LLMs) have garnrred significant attention for their markable capabilities across various domains, whose vast parameter scales present challenges for practical deployment. Structured pruning is effective method to balance model performance with efficiency, but performance restoration un- der computational resource constraints is principal challenge in pruning LLMs. Therefore, we present low-cost and fast structured pruning method for LLMs named SlimGPT based on the Optimal Brain Surgeon framework. We propose Batched Greedy Pruning for rapid and near-optimal pruning, which enhances the accuracy of head-wise pruning error estimation through grouped Cholesky decomposition and improves the pruning efficiency of FFN via Dynamic Group Size, thereby achieving approximate local optimal pruning results within one hour. Besides, we explore the limitations of layer-wise pruning from the perspective of error accumulation and propose Incremental Pruning Ratio, non-uniform pruning strategy to reduce performance degradation. Experimental results on the LLaMA benchmark show that SlimGPT outperforms other methods and achieves state-of-the-ane results.
Introduction
Large Language Models (LLMs) 3 have made significant strides in various natural language processing tasks, leading to the emergence of novel applications such as AI agents [4]. One of the factors contributing to the exceptional capabilities of LLMs is their massive parameter scales. However, these extensive parameters also introduce increased inference costs and deployment challenges, hindering the widespread application and adoption of LLMs. Accelerating inference for LLMs has become focal point of current research. Model compression (5), as one of the strategies for inference acceleration, including techniques like pruning and quantization 7 has been extensively researched. Nevertheless, earlier model compression techniques, particularly model pruning, typically rely on heavy post-training to recover the model's 's capabilities, which typically involves retraining with the entire training dataset. Given the constraints of current computational resources, the above approaches are not feasible for LLMs.
In the domain of LLM pruning, recent studies have largely focused on unstructured semi- structured) pruning [ a method that shrinks models by selectively zeroing out weights considered non-critical. Despite its advancements, unstructured pruning falls short in substantially reducing parameter count, which is crucial for accelerating LLM inference as it is often bottlenecked on memory bandwidth and communication [ To accelerate inference speed, unstructured pruning models are often paired with specialized frameworks or hardware solutions. Conversely, structured pruning [10 I1 effectively decreases the model's parameter count by systematically eliminating columns or rows from weight matrices, enabling significant improvements in inference speed, and
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

reduce deployment cost on conventional hardware. Yet, structured pruning often entails more pronounced compromises in model performance, which poses a greater challenge.
Recently, researchers have applied the classic Optimal Brain Surgeon (OBS) framework to the compression of L.Mss. This approach includes parameter compensation which can mitigate the loss incurred during compression and reduce the dependence on post-training. The OBS framework is currently applied in the areas of unstructured pruning [12] and quantization [3 for LLMs. However, there exist some challenges in its application to structured pruning:
The OBS is fine-grained compression framework that compresses one parameter at each iteration, whereas structured pruning has a minimum granularity of either a column or head. Directly applying the OBS framework will result in high numerical errors, impairing model performance.
The OBS is essentially a layer-wise compression method. It focuses on each individual layer, thus failing allocate pruning ratios for each layer rationally using global information (such global gradients). This is crucial for LLM structured pruning, which relies on a non-uniform strategy to reduce the impact on performance.
To address these issues, we propose a new structured pruning method for LLMs. We introduce Batched Greedy Pruning to achieve low-cost and rapid pruning for LLMs. Specifically, for attention heads, we propose grouped Cholesky decomposition to select nearly optimal heads for pruning in each iteration, thereby maintaining Networks (FFNs), we achieve near-optimal and efficient pruning results through Dynamic Group Size. Furthermore, since the OBS is essentially layer the error accumulation phenomenon in layer-wise pruning and propose pruning by Incremental Pruning Ratio, straightforward non-uniform strategy control the pruning rate of each layer, further mitigating performance loss under a given overall pruning ratio.
approximately locally optimal pruning result. For Ferd-Forward
waiee-wise wise compression framework, we investigate
Contribution. In this paper, we propose SlimGPT, layer-wist pruning approach that extends the classical OBS framework to structured pruning for LLMs. The characteristics of SlimGPT can be summarized as follows: (i) Task-agnostic pruning scheme. Only random sample of data from generic pre-training corpora needed as a calibration set, and we can obtain a compressed model with most performance preserved; (ii) Low-cost, low- oowrresurce, and time-efficient compression scheme. The model can be compressed using just a single GPU, a few hundred of calibration data, and about one hour: (iii) A universal pruning method for Transformer-baut based models. It has good transferability and, theoretically, is applicable to all large models based on the conventional Transformer architecture. We employ ILLMAA models for pruning and conduct evaluations wikitext2 and Commonsense Reasoning tasks. The results indicate that SlimGPT substantially retains the performance of the pruned models, surpassing state-of-the-ar methods.
2 Related Work
Compression methods with regularization. Before the era of LLMs, using the scaling factors from Batch Normalization layers as indicators of channel importance made pruning based regularization a very popular method [14 15 Notably, Louizos et al. [16] implemented the non-differentiable LO penalty in differentiable form, technique frequently used for pruning in large models. Compresso [77 combines LO regularization with LoRA training [18), effectively preserving model performance
at low cost. In similar vein, Sheared LLaMA [19 employs augmented LO regularization inserted masks for structured pruning, using extensive data to restore performance and deliver compact yet powerful pruned models.
Global gradient- based compression methods. NVIDIA's works 200 21l involve a Taylor expansion of the global loss. By eliminating higher-order terms, it is revealed that the impact of weight on the loss can be assessed using the magnitude the weight combined with gradient information. Based on this, LLM-Pruner [T]] employs first-order importance estimation to gauge the importance of weights. LORAPrune 22 measures the importance of weights based on the gradients of the LORA parameters rather than the model's parameters, achieving commendable results.
Outliers-dependent compression methods. Dettmers et 23 identifies an attribute unique to LLMs, where a small subset of activation values in the data features have magnitudes significantly larger than the others. And removing corresponding weights impacts model performance substantially. Building upon this, Wanda 24 proposes simple yet effective unstructured pruning method, using

---

the product of a weight's LI norm and the L2 norm of cigenvalues to gauge its importance, achieving impressive pruning results. OWL [25] determines layer- wise sparsity ratios based on Layerwise Outlier Distribution (LOD), obtaining substantial performance gains at high sparsity levels.
provide layer-wise compression framework with a locally optimal solution named Optimal Brain Surgeon (OBS). And then OBC (28) reduces the computational burden by converting layer-wise pruning into row-wise pruning and updating the inverse Hessian using a proposed formula. Furthermore, GPTQ 33 accelerates the process with Lazy Batch- Suact-spdates and Cholesky Reformulation, enabling the application of this method to the quantization of LLMs. SparseGPT [12] also adapts this approach for unstructured pruning of LLMs. However, there appears be no existing research that has implemented OBS in structured pruning for LLMs.
Layer-wise compression methods. The early works 226 27
Structured Pruning vs. Other Techniques. Given that OBS has previously been used in both quantization and unstructured pruning, and is now being applied to structured pruning, there is an inherent consistency across these three compression schemes. These methods actually compress the model at varying levels of granularity. Quantization, which "trims" floating-pons precision, represents the finest granularity and delivers excellent compression outcomes. Structured pruning, on the other hand, involves trimming weight vectors and represents the coarsest granularity, naturally resulting in higher performance losses compared to other methods, which poses significant challenges. For small models, it is possible to recover most of the performance with post-training. but this is challenging to achieve in LLMs due resource constraints. Nonetheless, structured pruning effectively reduces the number of parameters without needing special inference framework support and is compatible with the other two methods, thus still holding considerable potential for application.
3 Preliminary
Layer- Wise Pruning. Consider the scenario of pruning on well-optimized model, known as post-training pruning, a prevalent approach involves decomposing the global model pruning challenge into layer-wise subproblems (i.e., Layer-wise pruning), which are typically modeled as issues of minimizing L.2 error. Specifically, let W, represent the weights of the i-th layer of pretrained model and X, be the input features for layer I. The goal to determine pruned weights W, that achieve a predefined pruning ratio while minimizing the squared error:
argminw, || - W
Optimal Brain Surgeon (OBS) Framework. As Equation I can be rewritten as the sum of square error of each row of the weights to be pruned, the layer-wise pruning can be further split into row-wise pruning (28) Consider the removal of a single weight from a row in Wi, Equation [ Thas a closed of Wi, and let p be its corresponding index. Given that our optimization objective is to minimize row-wise squared error, the Hessian of this objective with respect to the weight row of layer / is given by Hi, XX,'7". The weight to be pruned, Wp. as well as the necessary update 5, applied to the remaining weights of the same row to counterbalance the removal, can be determined through the following calculation:
solution (27] Let denote specific weight in
Wp = argmin
P Hp.
6p
Wp Hp
H
(2)
where H-1 denotes the th diagonal entry of the inverse Hessian, and H is its p th column. By iteratively using Equation to remove one weight and update the remaining weights the same row, one can obtain a locally optimal compressed model. After each iteration, H will be updated by removing the p row and column, which is represented by Hi-p), here we use [-p] to indicate the removal of p row and column of the matrix. As H cannot be updated by simple removal as (H(-)))- * (H ()-))))) to avoid the expensive full recomputations of H-1, the following formula proposed to quickly update H-1 28
(H(-))-- (H-1
H H .p
[--)
Hp.
This framework can be practically applied to medium-sized models. However, for models with billions of weights, the iterative pruning becomes exceedingly time-consuming.
3

---

Methodology
In this section, by extending the OBS framework to structured pruning, we introduce SlimGPT from two aspects: (1) By employing Batched Greedy Pruning to reduce error computation, minimize the performance degradation caused by pruning while also accelerating the pruning speed: (2) By analyzing the limitation of layer-wise pruning from the perspective of error accumulation, we introduce Incremental Pruning Ratio, a non-uniform pruning strategy.
4.1 Structured Pruning with OBS Framework
As mentioned above, the pruning between different rows is independent, making it possible to prune all rows simultancously [29]. We extend the OBS framework to structured column pruning, i.e. pruning column time and compensating the rest columns using the following formula:
W SpP
W R Hp.
W argminw,,,
A
H Hp
Hp.
where H denotes the p-th row of H and the obtained A is compensation matrix of the same size as W. We following previous works employ attention blocks and FFNs as the smallest units for pruning. By pruning the columns of the output matrix in attention blocks and the dimensionality reduction matrix FFN blocks, we reduce the number of attention heads and FFN channels, thereby decreasing the model's parameter count.
However, the above formula cannot be applied directly, as iteratively finding and pruning the column with the minimum error is time-consuming More critically, the structural dependency in attention blocks imposes additional constraints on column pruning, making it impossible to evaluate the importance of a head based solely information from single column.
4.2 Batched Greedy Pruning
Given that the calculation the pruming error re- quires only the diagonal elements of H (see Equation 4. which are updated after each it- eration, computing these elements in advance allows for calculating the head-wise error. With the observation that the sequential row removal via Equation
1
Caluma Receder
3 for the symmetric H-1 essen- tially corresponds to taking Cholesky decom- position [3], we can obtain the elements in advance with Cholesky decomposition.
88
69
55
praned
Grouped Cholesky Decomposition
BSi
Hoewever, the matrix obtained by Cholesky decomposition is triangular, and the elements of the current row (column) are calculated based on the elements of all the previous rows (columns), which means the Cholesky decom- position breaks the comparability between rows (columns). So it hard to obtain all the required information in advance through the Cholesky decomposition like [2] whose eиrоr com- parison is usually within the same column but structured pruning requires the comparison of different columns.
Reardered A-11
All
3.9
B4
Ead
L:91
8
As
A12
630
6.3
1.11
613
Cholesky Decemposition
Column Restere
La1
Compenvate
Since structured pruning only requires travers- ing the columns that need to be removed, by rearranging the rows and columns correspond- ing to a head that is to be pruned in H to the front, and then invert the matrix followed by Cholesky decomposition,
The figure illustrates Batched Greedy Pruning on attention blocks, where W is output matrix and H is the corresponding Hessian. Dif- ferent colors represent distinct attention heads and gray indicates the pruned weights.
Figure 1:
calculate the

---

Algorithm 1 Batched Greedy Pruming for Attention Heads Given Weight matrix W, inverse Hessian H head size d and head count n.
// Step 1: calculate head-wiss enror
H GroupedCholesky(HH
E 2 /Diag )2 error matrix
E Σ o.d Σ E. d..d::2.-.. > E..ln-1)ddii If head
f Hcad2ColumnIdks Argson((E)) I/ remodered column index
W + A[;,A), H - H [A, [B, // reorder
// Step 2: prume a head column-wise
H +- Cholesky(H ()" [[]:d d
E + Xd xd
for i in O, 1, 2.dd
B
W H // pruning error
W Wind < Wiid
B x H.ddd
// local update, column i is zerved
end for
W :.d: Wid:: E H 11 global update
W Argsor(A))
I/ restore
head error column-wise. However, repeated rearrangement followed by matrix inversion and Cholesky decomposition is highly time-consuming, and this is just to find one head to be pruned.
We accelerate the above process through two common lemmas (proofs are provided in the Appendix): (i) For symmetric H, the inverse matrix after permutation can be obtained by the same permutation of H : (ii) The principal submatrix of symmetric H after Cholesky decomposition is equivalent to the Cholesky decomposition of its principal submatrix. Thus we can calculate the pruning error of all the heads split it into Rhedd matrices along the main diagonal, with each remains definite and symmetric, and decompose them in parallel:
through grouped Cholesky decomposition. Specifically, inverse H once and
H-1 H Cholesky(Stcck(FFod..o H d:2d, (n- (id-1)dddd (n- (11))dna)))
(5)
where decomposed H is matrix of size Nhead dread dread, Nhead and dhead represent the head number and head dimension, respectively. Utilizing GPU acceleration, we can quickly calculate the value of the diagonal element in advance and calculate the head-wiss error. Note that during error computation, we only update the diagonal elements of H and skip the update of W, which is small and does not dominate the ordering of errors.
After determining the head to pruned, we rearrange the corresponding columns W and the cor- responding and columns of H to the front, and again use the global Cholesky decomposition on reordered -1 to prune the head column by column until the first head pruned. In this way, avoid traversing columns that do not need pruning and only traverse necessary columns to improve pruning efficiency further. Figure T shows the process of Batched Greedy Pruning applied to attention blocks, and Algorithm i a pseudocode illustrating how prune head with two steps: calculating head-wise error and pruning a head column- wiie.
For FFNs, since there is no block constraint similar to attention heads, we can achieve local numerical optimality by pruning columns individually using Equation 4 However, column-wise pruning is time-consuming because of the substantial intermediate dimensions of FFN. We thus prune group of columns at a time and select the top-k columns with the most minor errors for pruning at each iteration. Considering that the compensation each iteration may lead to local reshuffling of column errors, we adopt dynamic grouping strategy for pruning FFN blocks. We start with larger group size such 1024 for pruning and gradually decrease the group size to small number like 8, which allows us to enhance pruning efficiency while approaching an approximate optimal solution.
4.3 Incremental Pruning Ratio
Through Batched Greedy Pruning, we can obtain near-optimal structured pruning results for each layer. However, finding a suitable pruning ratio for each layer is difficult, as considering global information is quite challenging for layer-wise pruning, which only provides optimal pruning results
5

---

for the currett layer. Maintaining a uniform pruning ratio across all layers is unreasonable and will impact model performance, especially when the pruning ratio is high. Existing works have different approaches to the problem. For example, L.LM-Pruner [1]] avoids pruning in the initial and final layers while maintaining a consistent ratio in the intermediate layers to manually implement non-uniform pruning. OWL [25] adjusts sparse ratios dynamically for each layer based on the proportion of feature outliers, which is applied to unstructured pruning.
We find that layer-wise pruning, particularly structured layer-wise pruning. suffers from error accumulation due to its locality. Errors intro- duced during pruning in one layer can be ampli- fied in subsequent layers, resulting in significant discrepancies between the final model output presents the per-layer output error of FFN between the original model and three distinct pruned models. The pruned models each implement a first-layer pruning of 25%, 50%, and 75%, respectively. The error increases with model depth and accumulates at rate exceeding linear progression as the initial layer's pruning ratio increases. Based on this observation, propose a straightforward prun- ing strategy for layer-wise pruning, termed In- cremental Pruning Ratio, which can effectively minimize pruning losses without any additional operation.
0.010
Layerr__P.22:PPL_ P0.25:PPL-12.64
Layero_Po.5:PPM
P0.5:PPL=13.98
Squared error
0.008
LayerO P0775:PPL=21.49
0.006
and the original. Figure 2
0.000
10 15
20 25
Layer index
Figure 2: Per-layer FFN output error between the original LLaMAA-B and three distinct pruned mod- cls. The pruned models each implement a first- layer reduction of 25%. 50%, and 75%, respec- tively. The PPL of original model is 12.63. For ease of visualization, the layer index has been trun- cated to 25.
In Incremental Pruning Ratio, without loss of generality, we employ a logarithmically increas- ing strategy to control the layer-wise wise pruning ratio. Specifically, for an n-layer model with the first and last layer pruning ratios denoted ro and rn-1 respectively, the pruning ratio for the i-th layer is defined as follows:
T3 = To + (In-1 - mo) log(i + +1)
(0 s
(6)
log(n)
where r; represents the pruning ratio for the i-th layer. This formula ensures that the pruning ratio from the first layer to the last layer transitions smoothly as a logarithmic curve. The strategy mitigates the pruning error accumulation in shallow layers while avoiding the issue of excessive pruning in the deeper layers, allowing for further reduction in performance loss.
Experiment
5.1 Experimental Settings
Implementation details. We use C4 dataset (30 as the calibration set. From the first shard of C4, we randomly select 256 1048-token sequences for pruning. To restore performance, we following LLM--runer [1 finetune the pruned model with LORA [18] We tune with Alpaca datsets [3] for one epoch and utilize the AdamW optimizer with initial learning rate set to le-4, coupled with cosine annealing schedule for the learning rate. The global batch size set to 64 and the sequence length is truncated to 256. All pruning experiments are conducted on single A A100, while finetuning is performed using two A.00s.
Models and Metrics. To assess the effectiveness and generality of SlimGPT, We carry out a series of experiments on the LLaMA families 2 And measure the effectiveness of our pruned models in the task-agnostic setting, we follow previous pruning works to evaluate language modeling performance and commonsense reasoning capabilities. The language modeling performance is evaluated on the WikiText2 [32] validation set with sequence length truncated to 128, and the commonsense reasoning capabilities is carried out under a zero-shot setting on the Commonsense Reasoning datasets, which encompass seven diverse subtasks: BoolQ 33 PIQA (34] HellaSwag [5), WinoGrande (36). ARC- We utilize the on-eval-harnes framework
easy [37] ARC-challenge [37], and OpenbookQA
39 to conduct these evaluations.

---

Table 1: PPL & Commonsense Reasoning zero-shot performance of the pruned LLaMA-7B. The average score is computed across seven datasets. The bolded results represent the optimal results,
while the underlined ones is the sub-optimal results. The asterisk-marked (*) results those
replicated within a consistent experimental framework, which slightly differ from the original source.
Prune%
Method
#Params
PPI.]
BoolQ
PIQA
HellaS
WinoG
ARC-e
ARC-c
OBQA
Avg.
6.7B
12.63
75.08
79.16
76.20
70.00
72.89
44.88
44.40
66.09
LLM-Pruner*
18.01
66.76
78.45
71.44
63.77
66.41
39.85
43.80
61.50
Compresso
79.08
75.46
53.44
67.80
68.64
37.97
34.20
59.51
20%
LoraPrune
5.4B
16.80
65.62
79.31
70.00
62.76
65.87
37.69
39.14
60.06
SlimGPT w/o tine
16.99
75.93
77.58
73.07
67.96
68.60
41.72
41.80
63.81
SlimGPT
16.68
74.59
78.94
74.40
68.43
70.50
43.26
45,40
65.07
LLM-PPrnne*#
20.57
62.81
76.93
69.21
60.46
63.34
38.14
39.80
58.67
Compresso
73.55
73.07
49.16
64.80
66.20
37.20
29.80
56.25
25%
5.0B
SlimGPT w/o tune
19.11
75.11
76.77
70.60
67.25
66.75
40.40
40.40
62.47
SlimGPT
18.45
73.46
77.42
72.07
65.51
67.17
41.13
40.40
62.45
I.IM-Pruner*
24.50
62.02
74.92
64.41
61.80
53.79
32.00
38.80
55.39
Compresso
68.69
72.85
47.18
63.38
65.99
35.07
29.00
54.59
33%
4.5B
SlimGPT w/o tune SlimGPT
24.55
72.72
75.68
68.10
66.54
62,29
37.03
40.20
60.37
22.43
71.53
76.66
70.55
66.06
64.35
39.33
41.40
61.41
LLM-PPrnne*
40.64
60.21
68.88
47.86
54.62
43.94
27.73
35.20
48.35
LoraPrune
30.12
61.88
71.53
47.86
55.01
45.13
31.62
34.98
49.72
50%
3.4B
SlimGPT w/o
38.83
65.87
70.35
54.62
59.59
49.71
31.06
34.40
52.23
SlimGPT
31.07
65.11
71.60
59.94
59.27
53.37
31.83
35.20
53.76
To validate the universality of SlimGPT, we conduct experiments on additional models and supple- mentary evaluation datasets. The results of these experiments can be found in the Appendix. We conduct further pruning experiments on Vicuna 40), LLaMA2 [41), and Baichuan [42]. which yield results consistent with those observed using the LLaMA model. In addition, we engage in prelimi-
nary evaluations on more complex tasks, specifically MMILU [4) and LongBench +4 Although
SlimGPT exhibits slightly larger performance losses on these datasets, it still retains a significant advantage over the bascline models.
Baselines. We compare SlimGPT with the following recent SOTA works on structured pruning, which we could find during our experiments:
LLM-Pruner [. a gradient-based pruning approach, serves benchmark. This method involves a two-step process: a one-shot pruning followed by performance restoration through LORA fine-tuning.
Compresso [17] is pruning method based on sparse training, applying LO penalty to manually inserted masks during the LORA fine-tuning phase and employing a cubic sparsity schedule to iteratively prune the model until the desired pruning ratio is achieved.
LoRAPrune [22) utilizes gradients from the LORA module's parameters determine the impor- tance of the original model's parameters, thus requiring only gradient information from the LORA module, which significantly reduces computational demands.
5.2
Main Result
5.2.1
Performance Evaluation
To facilitate a more effective comparison of the evaluated results with prior works, we prune the LLaMA-7B model using four distinct pruning ratios 20%, 25%, 33%, and s0suuiing in four smaller models with parameter counts of 5.4B, 5B, 4.5B, and 3.4B, respectively. Table [ shows the detailed perplexity and zero-shot performance of pruned LLaMA-7B with four different sizes. Compared to other approaches, SlimGPT demonstrates superior performance in language modeling and commonsense reasoning across most subtasks. Under a pruning condition of 20%, SlimGPT achieves a slightly better perplexity score than the best existing results (16.68 vs. 16.80) and shows
7

---

Table PPL Commonsense Reasoning zero-shot performance of the pruned LLaMA-13B/305 The perplexity is evaluated on Wikitext2 and the zero-shot average is computed across seven Com- monsense Reasoning datasets. The bolded results represent the optimal results. The asterisk-marked (+) results are those replicated within consistent experimental framework, which slightly differ from the original source. Detailed results are available the Appendix.
Prunc%
Method
#Params
PPL4
Zero-shot Avg.t #Params PPLJ Zero-shot Avg.t
13.0B
11.58
68.16
32.5B
9.78
71.92
LLM-Pruner*
16.62
65.68
12.06
69.99
20%
SlimGPT w/o tune SlimGPT
10.4B
14.87
66.37
26.0B
11.59
71.13
14.73
68.06
11.69
72.56
LLM-Prunerr
74.62
53.22
22.33
59.47
50%
SlimGPT w/o tune
6.5B
31.05
57.82
16.3B
18.61
65,50
SlimGPT
26.38
59.49
17.17
66.79
Table 4: Inference Latency and Memory Usage
Table 3: Pruning Runtime and Memory Usage
Prune #Params
Latency
Memory
#Params
Runtime (20%)
Runtime (50%)
Memory
6.7B
13.51ms
27737MB
7b
678s
1074s
7375M
20%
5.4B
11.89ms
22497MB
13b
1417s
2475s
11601M
50%
3.4B
9.21ms 14297MB
3.6-point improvement in zero-shot average (65.07 61.50). As the pruning ratio increases to 50%, the advantages of SlimGPT become even more pronounced. SlimGPT without post-training represents approximately 8% improvement over the baseline LLM-Pruner in average performance (52.23 vs. 48.35), and with post-training, the average performance improvement reaches up to 11 1% (53.76 vs. 48.35). Specifically, on a dataset like Hellaswag, the improvement soars up 25% (59.94 vs. 47.86).
Moreover, we observe that although SlimGPT affects different subtasks varying degrees, its impact is relatively balanced across different tasks, eliminating the occurrence of disproportionately large losses in partieular tasks. At lower pruning ratios, some tasks such BoolQ can even outperform the original unpruned model. Additionally, the effects of fine-tuning also differ among tasks, significantly improving tasks like HellaSwag and ARC-easy, while potentially causing negative side effects for tasks such as BoolQ and WinoGrande. This phenomenon is likely closely associated with the datasets used for fine-tuning.
For larger-scale models such as LLaMA-13B and LLaMA-30B. previous works have not provided pruning results for these models. Therefore, solely compare our results to the LLM-Prunc baseline, concentrating on two specific pruning settings: a lower pruning ratio (20%) and a higher pruning ratio (50%). The replication of LLM-Prunnr is consistent paper, where the prumed models by L.M-Pruner are finetuned with LORA.
with the method described in the
Table 2 presents the pruning results of LLaMA-13B and LLaMA-30B. and we can draw similar conclusions: SlimGPT outperforms L.rnrr in terms of both PPL and zero-shot average scores even without post-training. Note that as the scale of the model increases, the performance loss due to pruning becomes smaller, suggesting a higher degree of parameter redundancy in larger models. At a low pruning ratio of 20%, the LLaMA-133 model's average performance in commonsense reasoning is nearly on par with that of the original, unpruned model (68.06 vs. 68. 16). Similarly, the pruned LLaMA-30B model slightly outperforms the unpruned version (72.56 vs. 71.92). For the perplexity task, even though SlimGPT exhibits gaps compared to the original model, still performs better than baseline,
low pruning ratios.
Besides, we can find that the performance of LLaMA-13B pruned by 50% falls short compared to LLaMA-7B pruned by 20%. This highlights the limitations of low-cost fine-tuning, where resource constraints and training with techniques like LoRA result in limited performance recovery for the model. Therefore, using lower pruning ratios compress smaller LLMs yields better returns.
8

---

Table 5: Pruning results under different strategies of SlimGPT. '-DGS' means removing Dynamic Group Size for FFN while '-GDD' means remov- ing grouped Cholesky decomposition for attention blocks.
PPT.
Zero-shot Avg.
SlimGPT
38.83
52.23
DGS
39.73 (+0.90)
51.63 (-0.60)
GCD
54.94 (+16.11)
51.59 (-0.64)
Table 6: Pruning results with different pruning ratio strategies.
Model Size
PPL.L
Zero-shot Avg-t
log increase (SlimGPT)
3,40b
38,83
52.23
linear increase
3.34b
46.57 (+7.74)
53.45 (+1.22)
uniform
3.50b
123.05 (+84.22)
44.34 (-7.89)
log decrease
3,40b
380.69 (+341.86)
36.73 (-15,50)
linear decrease
3.34b
932.64 (+893.81)
35.62 (-16.61)
5.2.2 Efficiency Analysis
The pruning runtime and memory usage for LLaMA-7B and LLaMA-13B are detailed in Table 3 Memory usage fluctuates based on the model size and the calibration scale, while the pruning speed additionally affected by the pruning ratio. We demonstrate the pruning efficiency results derived from our experimental setup. Utilizing SlimGPT, which operates need load the entire model at once. Instead, we only load the parameters of the current layer along with the corresponding input features, significantly reducing memory consumption. For instance, to prune the 7B model by 20%, approximately 7 GB GPU memory and 18 minutes are required to complete the process. Similarly, pruning the 13B model by 50% necessitates around 12 GB of GPU memory and 41 minutes to finalize.
layer-wise basis, there is no
Table Iillustrates the inference latency and memory usage of the pruned LLaMA-7b models. We prune LLaMA-7b by 20% and 50% respectively. The maximum output limit is set to 512 and the presented values are the average derived from 50 inference trials. When pruning 50% of the parameters, the memory usage of the model during inference decreases to approximately 51% (14297MB vs. 27737MB), and the inference latency is reduced to about 69% (9.21 vs. 13.51ms).
5.3 Ablation Study
We systematically analyze the influence of several key parts of SlimGPT on the pruning effect, including the Batched Greedy Pruning and Incremental Pruning Ratio strategy. Within the calibration dataset, conduct thorough experiments with sample sizes and sequence lengths. Unless specifically stated otherwise, all the following experiments are conducted under the condition of pruning 50% of LLaMA-7b 7b without further post-training, eliminate potential confounding effects. Supplementary ablation experiments can be found in the Appendix.
5.3.1 Impact of Batched Greedy Pruning Strategy
We leverage grouped Cholesky decomposition to enhance the accuracy of head-wise error computation in attention blocks. Similarly, for FFNs, our proposed Dynamic Group Size substantially increases pruning efficiency while preserving near-optimal pruning results. To validate the effectiveness of these two strategies, we start with the complete SlimGPT algorithm and first remove the Dynamic Group Size (denoted as *DGG'', ), setting the group size for FFN pruning to fixed value of 128. Then, we remove the grouped Cholesky decomposition (denoted as '-GDD") and use the initial H-1 to calculate head-wise errors. The experimental results are shown in Table For attention blocks. the grouped Cholesky decomposition strategy plays key role in language modeling capabilities by improving the accuracy of error compensation. Replacing with ordinary Cholesky decomposition results in significant increase in PPL. (38.83 vs 54.94). In comparison to the naive fixed group size scheme for FFNs, the Dynamic Group Size strategy proposed contributes to maintaining the model's commonsense reasoning performance (52.23 51.63).

---

53.5
65
PPL1
Zero shot
PPL
Zero-shot Avg.
54
48-
53.0
Kero Avg
60
Tero-shot Avg.
52.5
PPL
52.0
52
50
44
51
45
42
51.0
50
40
40
50.5
126
512
1024
2048
512
1024
2048
Sample size
Seqence length
(a)
(b)
Figure 3: Effects of Calibration Sample Size & Sequence Length.
5.3.2 Impact of Incremental Pruning Ratio Strategy
The Incremental Pruning Ratio a strategy specifically proposed for addressing the issue of layer-wise pruning. To maintain generality, we selected various common non-uniform strategies for comparative experiments, including logarithmic and linear increase strategies, well as their corresponding decrease strategies. Among these, the logarithmic increase strategy is the default configuration for SlimGPT, Additionally, we conduct experiments under the setting uniform pruning. deblel the results under the different settings. From an overall perspective, the increase strategy for the pruning ratio has a clear advantage over uniform, and likewise, uniform shows distinct advantage over decrease. Such results further verify the phenomenon of layer-wize error accumulation. As for the increase strategies of logarithmic and linear changes, due to disparities model sizes, their results are not entirely comparable. The former performs best in language modeling (38.83), while the latter shows better performance in common sense reasoning tasks (53.45).
5.3.3 Effects of Calibration Samples & Sequence Length
We delve further into the impact of calibration samples and sequence length, and choose C4 dataset for our experiments as has longer average sequence length. exploring the effects of the sample scale, we fix the sequence length at 256 and test five scales ranging from 128 to 2048; similarly, when investigating the impact of sequence length, the sample scale set to 256, with choices of sequence length varying from 64 to 2048. Figure presents the perplexity result and zero-shot performance with different calibration samples and sequence lengths. As the number of samples increases, the PPL. and zero-shot averages show positive overall trend. Furthermore, after the sample count reaches 2048, the PPI. does not bottom out, and there is room for further reduction. Similar phenomena be observed in experiments sequence length. With sufficiently high-quality datasets with longer sequences, we believe SlimGPT can achieve better pruning effects.
Conclusion
In this work, we introduce a fast, structured pruning method for large-scale models within resource- constrained scenarios, based on the OBS framework, termed SlimGPT. Leveraging the novel Batched Greedy Pruning, we enhance the accuracy of pruning error estimation, thereby minimizing perfor- mance degradation from pruning. Moreover, we analyze the limitations of layer-wiee wise pruning from the perspective error accumulation and propose non-uniform strategy named Incremental Pruning Ratio, which effectively improves the pruned model's performance. Evidence from open-source experiments affirms the efficacy of our approach.
Limitations. Even though SlimGPT achieves SOTA results in the structured pruning of LLMs, the model performance degradation high pruning ratios (e.g., 50%) or on more complex tasks (e.g.. LongBench) is still significant. How to enhance the model compression effectiveness under low- resource conditions remains a challenge. Moreover, we utilized naive logarithmic change strategy in the Incremental Pruning Ratio, which, while ensuring generality, is not the optimal solution. The most suitable non- non-unifee con-unffomm approach requires further exploration. Lastly, similar many large-scabe scale open-source models available today, the model obtained through pruning by SlimGPT poses risks in terms of ethical safety and requires cautious handling.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr. Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
[2] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
[3] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprinl arXiv:2309. 16609, 2023.
[4] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. arXiv preprint (.Xiv:3309007866 2023.
[5] Tejalal Choudhary, Vipul Mishra, Anurag Goswami, and Jagannathan Sarangapani. A com- prehensive survey on model compression and acceleration. Artificial Intelligence Review, 53:5113-5155 2020.
[6] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruming and growth for efficient inference and training in neural networks. The Journal of Machine Learning Research, 22(1):00822-1100 2021.
[7] Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. survey of quantization methods for efficient neural network inference. In Low-Power Com- puter Vision, pages 291-326. Chapman and Hal//CRC, 2022.
[8] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. arXiv preprint 1X0::0110000149, 2015.
[9] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. International Conference on Machine Learning, pages 19274-19286. PMLR, 2023.
[10] Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. ACM Journal Emerging Technologies in Computing Systems (JETC),
13(3):1-88, 2017.
[11] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. arXiv preprint arXiv:2305. 11677, 2023.
[12] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323-10337. PMLR, 2023.
[13] Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210. 17323, 2022.
[14] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision, pages 2736-2744, 2017.
[15] Tao Zhuang, Zhixuan Zhang, Yuheng Huang, Xiaoyi Zeng, Kai Shuang, and Xiang Li. Neuron- level structured pruning using polarization regularizer. Advances in neural information process- ing systems, 33:9865- 9877, 2020.
[16] Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_0 regularization. arXiv preprint arXiv:1712.013112 2017.
[17] Song Guo, Jiahang Xu, Li Lyna Zhang, and Mao Yang. Compresso: Structured pruning with col- laborative prompting learns compact large language models. arXiv preprint 10.00:2310.05015 2023.
11

---

[18] Edward Hu. yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[19] Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared Ilama: Accelerating language model pre-training via structured pruning. In Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ NeurlPS 2023), 2023.
[20] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Representations, 2017.
[21] Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11264-11272, 2019.
[22] Mingyang Zhang, Chunhua Shen, Zhen Yang. Linlin Ou, Xinyi Yu, Bohan Zhuang, et al. Pruning meets low-rank parameter-efficient fine-tuning. arXiv preprint 23Xiv:2305.184000 2023.
[23] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 O: 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07333, 2022.
[24] Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 11695, 2023.
[25] AMissing SECRET SAUCE. Outlier weighed layerwise sparsity (owl): Amissing secret sauce for pruning Iims to high sparsity. 2023.
[26] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
[27] Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, pages 293-299, IEEE, 1993.
[28] Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post- training quantization and pruning, Advances in Neural Information Processing Systems, 35:4475- 4488, 2022.
[29] Eldar Kurtié, Elias Frantar, and Dan Alistarh. Ziplm: Inference-aware structured pruning of language models. Advances in Neural Information Processing Systems, 36, 2024.
[30] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, (11555585-5551, 2020.
[31] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-followin llama model, 2023.
[32] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2016.
[33] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers),
pages 2936, 2019.
[34] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piga: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
12

---

[35] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, 2019.
[36] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
[37] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint vrXiv:/803.054 2018.
[38] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2391, 2018.
[39] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version vO. 0.I. Sept, 2021.
[40] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023. URL https://lmsys. org/blog/2023-03 30-vicuna, 3(5), 2023.
[41] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307. 09288, 2023.
[42] Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309. 10305, 2023.
[43] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint WwXiv:2009.03300 2020.
[44] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint 20X1::2388.44508 2023.
[45] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b, arXiv preprint 100.62,3 2023.
13

---

A Proof of Lemmas
A.1 Proof of Lemma (i)
Lemma. For symmetric matrix M, the inverse matrix after permutation can he obtained by the same permutation of M-
Proof. The lemma can be easily proven through elementary matrix transformations. Let P be permutation matrix. We have pt p I. And since M is symmetric, M = M We wish to prove that the inverse of the permuted matrix M' = pd MP (M') = PT P- P. By the following transformations:
M- = p (M
MIH-1P - (7)
we can demonstrates that (M')-= M-
A.2 Proof of Lemma (ii)
Lemma. The principal submatrix of symmetric M after Cholesky decomposition is equivalent the Cholesky decomposition its principal submatrix.
Proof. Consider symmetric matrix M. Without loss of generality, let's consider are removing the last row and column. block form:
B
M
(8)
BTT
C
its Cholesky decomposition can be expressed as:
LA 0 LA
47
M
(9)
LB
where LA is the Cholesky decomposition of A, and a scalar value. Here, A I. I. and this matches the definition of the Cholesky decomposition for the principal submatrix A of M. Thus the statement demonstrated through the uniqueness of the Cholesky decomposition.
Table 7: PPL & Commonsense Reasoning zero-shot performance of the pruned LLaMA-13B/30B
Prune%
Method
#aaams
PPI.]
BoolQ
PIQA
HellaS
WinoG
ARC-e
ARC-c
OBQA
Avg.
13.0B
11.58
77.89
80.14
79.06
72.85
74.75
47.61
44.80
68.16
LLM-Pruner
16.62
79.38
77.36
71.47
70.32
70.54
44.88
45,80
65.68
20%
SlimGPT w/o tune 10.4B
14.87
77.06
79.82
76.94
72.61
69.78
44.80
43.60
66.37
SlimGPT
14.73
80.00
80.47
78.44
72.69
71.59
47.61
45,60
LLM-Pruner
74.62
62.35
72.74
58.43
55.88
51.89
33.02
38.20
53.22
50%
SlimGPT w/o tune
6.5B
31.05
69.14
14
74.32
64.57
65.82
57.74
35.15
38.00
57.82
SlimGPT
26.38
71.44
75.57
68.08
64.96
61.78
36.77
37.80
59.49
32.5B
9.78
82,69
82.26
82.60
75.85
78.91
52.90
48.20
71.92
LLM-Pruner
12.06
81.28
80.96
80.66
73.16
76.98
49.49
47.40
69.99
20%
SlimGPT w/o tune SlimGPT
26.0B
11.59
82.87
81.28
81.01
76.09
76.98
51.28
48.40
71.13
11.69
84.01
82.37
81.94
76.01
80.81
54.01
48.80
72.56
LLM-Pruner
22.33
66.21
76.44
69.46
64.56
60.98
37.63
41,00
59.47
50%
SlimGPT w/o tune
16.3B
18.61
75.08
77.20
75.01
74.11
68.43
43.26
45,40
65.50
SlimGPT
17.17
75.93
77.91
77.43
73.80
70.62
44.45
47.40
66.79
B More Detailed Evaluation Results
Detailed evaluation results of pruned I.L.MA-3BB/30F Table details the experimental results for LL The evaluation results in this table represent a detailed version of Table 2 listing scores for each specific commonsense task to provide more detailed comparison.
14

---

Table 8: PPL & Commonsense Reasoning zero-shot performance of the pruned Vicuna-7B
Prune%
Method
#aarams
PPT.
BoolQ
PIQA
HellaS
WinoG
ARC-e
ARC-c
OBQA
Avg.
6.7B
16.11
78.41
78.56
74.68
70.09
72.01
43.77
43.40
65.85
L.MM-rruner*
19.11
61.96
76.88
69.18
63.30
61.83
37.88
39.40
58.63
20%
SlimGPT w/o tune SlimGPT
5.4B
21.14
75.41
77.09
72.34
68.43
69.23
41.47
43,40
63.91
17.73
74.98
77.42
72.19
67.88
68.31
41.47
42,60
63.55
LLM-Pruner*
43,96
40.76
67.08
46.64
53.28
43.98
27.56
34,00
44.76
50%
SlimGPT w/o tune 3.4B
42.90
65.84
71.22
54.08
56.83
54.71
31.40
35,60
52.81
SlimGPT
31.41
61.04
71.33
58.87
57.85
55.64
32.08
36.60
53.34
PPL & Commonsense Reasoning evaluations of pruned Vicuna-7B. Table&Sdetann the experi- mental results for Vicuna-7B. We observe that, the Wikitext2 dataset, SlimGPT without finetuning exhibits comparable or higher PPL than LLM-rrruner, a result that diverges from findings in experi- ments with LLaMA models. The parameter compensation of SlimGPT makes it more dependent on the distribution of the calibration set compared to LLM--runer, while Vicuna is a model finetuned on general instructions, and this point, pretrained data not the most appropriate calibration set. Using an instruction dataset for pruning might yield better results, which remains to be verified. However, SlimGPT with finetuning still leads on most of the tasks.
Table 9: PPL & Commonsense Reasoning zero-shot performance of the pruned LLaMA2-7B
Prune
Method
#Params
PPL,
BoolQ
PIQA
HellaS
WinoG
ARC-e
ARC-c
OBQA
Avg.
6.7B
12.19
77.71
79.05
76.00
68.98
74.58
46.33
44.20
66.69
LLM--Prnne*
17.00
67.95
77.58
71.43
64.01
63.51
38.05
39.80
60.33
20%
SlimGPT w/o tine 5.4B
16.49
73.43
77.58
72.62
68.82
69.99
42.32
42.00
63.82
SlimGPT
16.02
76.06
78.73
74.94
69.30
72.73
45.14
43.40
65.75
Table 10: MMLU 5-shot performance of the pruned LLaMA2-7b
Prunc
Method
#Params
Humanitics
Social Sciences
STEM
Other
Avg
6.7B
43.3
51.6
36.3
52.1
45.6
LLM-Pruner
25.7
23.6
24.2
26.8
25.2
20%
SlimGPT w/o tune SlimGPT
5.4B
36.0
45,2
33.5
44.1
39.4
35.3
42.2
31.5
43.0
37.8
PPL & Commonsense Reasoning & MMLU evaluations of pruned LLaMA2-7B. LLaMA2-7B is new generation model with completely different parameters, exhibiting better overall perfor- mance compared to the first generation LLaMA-7B. In addition to the Perplexity and Commonsense Reasoning assessments, we also supplement evaluation on the Massive Multitask Language Under- standing (MMLU) task. MMLU is a quiz bank covering 57 subjects, presenting a greater challenge compared to the Commonsense Reasoning datasets. We evaluate using LLaMA2-7B with 20% of its parameters pruned, under 5-shot settings. The evaluation results for PPL and Commonsense Reasoning are shown in Table 9 while the results the MMLU task are presented in Table TT] In the Commonsense Reasoning task, SlimGPT significantly outperforms the baseline and closely approaches the performance of the original unpruned model. In the MMLU task, although SlimGPT still substantially leads over the baseline, exhibits noticeable gap compared to the unprumed model and shows slight decline after finetuning. For such challenging tasks, full post-training is required to restore performance, rather than relying solely on lightweight LoRA finetuning.
PPL & Commonsense Reasoning & MMLU evaluations of pruned Baichuan-7B. We conduct pruning experiments on the Baichuan-7b model and perform evaluations on the Wikitext2, Common- sense Reasoning datasets, and MMLU datasets. Tables 11 and Table 12 present the evaluation results for commonsense reasoning and MMLU, respectively. Similar to the findings with LI aMM-77b,, under the same LoRA finetuning settings, SlimGPT shows clear improvement over the baseline.
15

---

Table 11: PPL & Commonsense Reasoning zero-shot performance of the pruned Baichuan- 7B
Prune%
Method
#Params
PPT.
BoolQ
PIQA
HellaS
WinoG
ARC-e
ARC-c
OBQA
Avg
_
7B
13.25
70.09
76.93
70.06
64.09
67.05
40.53
38.20
60.99
I.MM-Prnner*
19.85
62.87
74.48
63.03
60.93
60.31
36.86
36.20
56.38
20%
SlimGPT w/o tune 5.7B
20.01
69.17
75.03
65.25
61.25
64.94
35.15
36.60
58.20
SlimGPT
19.73
66.21
75.03
66.74
63.85
62.84
38,91
38.00
58.80
Table 12: MMLU 5-shot performance of the pruned Baichuan-77
Prune%
Method
#Params
Humanities
Social Sciences
STEM
Other
Avg
7B
38.1
49.2
35.2
47.7
42.1
-LMn-ru
24.8
23.4
21.9
26.8
24.3
20%
SlimGPT w/o tune
5.7B
29.7
38.0
31.3
38.9
34.0
SlimGPT
32.2
39.2
31.1
40.2
35.4
Table 13: LongBench evaluation results of the pruned Daatall-7B-Isttuutt--...
Prune%
Method
#Params
Single-Doc QA
Multi-Doc QA
Summarization*
Few-shot
6.7B
35.8
29.4
24.2
65.8
20%
SlimGPT
5.4B
30.8
27.2
21.4
60.0
Long Context Understanding evaluation results. To further explore the impact of SlimGPT on the understanding of long-context texts, select the Mistral-7B- 177B-Instruct-22. model [45] for experiments, which supports up to 32k context windows. We prune it by 20% and conduct an evaluation the LongBench task. Table 13 presents the evaluation results of the model before and after pruning. Note that we skip the evaluation of the GovReport datasets and thus the average score on Summarization tasks does not include that dataset. Under the LoRA finetuning settings, using SlimGPT with 20% of its parameters pruned can retain 90% of its long-text texttext comprehension capabilities.
C
Supplementary Ablation Experiments
C.1 Influence of Calibration Data Category.
As SlimGPT updates the remaining parameters to mitigate the effects of pruning, which dependent on the calibration data, it underscores the importance of investigating the impact of various calibration dataset categories. We conduct experiments on three general datasets:
C4 subset: A commonly used pre-training corpus, which the default calibration set for SlimGPT. We sample 512 sentences with 512 tokens from the first 20,000 corpus.
Alpaca dataset: high quality generic domain dataset used for supervised finetuning, generated by GPT3.5. We randomly sample 512 sentences with 512 tokens.
GPT4-Alpaca dataset: A high quality dataset similar to Alpaca generated by GPT4. We randomly sample 512 sentences with 512 tokens.
We maintain consistency in pruning strategies across all models, differing only in the dataset used. Each model pruned by 50% We assess performance directly on these pruned models without any post-training. Table 14 presents the pruning results across various datasets. The three datasets can be categorized into pre-training datasets (C4) and eustuutiionfolllwiing datasets (Alpaca, GPT4 -AT__a_ Models pruned on C4 exhibit better PPL results on Wikitext2, whereas models pruned on Alpaca series perform better on the Commonsense Reasoning dataset. Different types of datasets have varying impacts SlimGPT. instruction-followin datasets is more favorable for retaining the model's commonsense knowledge, whereas using pre-training datasets can achieve a balance between language modeling capabilities and commonsense abilities.
16

---

Table 14: Pruning results with various calibration datasets
Dataset
PPL.
Zero-shot Avg-t
C4 (SlmGGPT)
42.06
52.70
Alpaca
48.26
54.44
GPT4-Alpaca
47.06
54.66
D More Analysis
D.1 About Structural Dependency
The structural dependency problem attention blocks happens when a column of weights in an attention head is removed, elements in other positions in the attention matrix are also affected because of the softmaz function. Directly summing the errors across all columns of head may result in significant numerical inaccuracies, Equation applies only to single-column pruning instead of multiple columns. To achieve multi-column pruning, we need to iterate using Equation 4 and update H-1 with Equation which makes it difficult to assess the pruning error of total attention head in advance.
D.2 Layer-wise Pruning Ratio at Pruning Stage
60%
Pruning Ratio
50%
40%
20%
LLM-Pruner
SlimGPT
0
24
Layer Index
Figure 4: Layer-wise pruning ratio on LLaMA-7B with total pruning ratio 50%.
To conveniently present the details of the logarithmic increase variation in Incremental Pruning Ratio, we illustrate the layer-wise pruning ratios for SlimGPT's logarithmic increase and I.IM-- Pruner's heuristic setting 50% pruning rate in Figure SlimGPT starts with lower initial pruning rate, with a rapid increase in the shallower layers followed by slower change in the deeper layers, eventually approximating the fixed pruning ratio of LLM-Pruner. Their biggest difference lies in the handling of the last two layers. LLM-Pruner lacks parameter compensation, so the layers pruned the output end have a larger impact the final results, whereas SlimGPT reduces their impact on the model through parameter compensation.
D.3 Training Loss Recovery Stage
3.0
2.5
2.0
Wikitext2 Eval Loss
Alpaca Train Loss
1.5
1.0
0.2
0.4
0.6
0.8
1.0
Epoch
Figure 5: Alpaca train loss & Wikitext2 evaluation loss.
17

---

Table 15: Generated Examples from the LLaMA-7B and Pruned LLaMA-5.4B
Model
Generated Sentences
AI can create a logo seconds, but it's ready for the real world. By Dyllan Furness @dyl- lonline Posted on July 12. 2017 9:35AM 07. 12.17 9:35AM. A new artificial intelligence system has been developed that can generate logos from scratch with just one click of a mouse button. The technology is impressive and could have some practical applications down the line, but there are still many hurdles to overcome before see this kind of software being used by designers or marketers....
LLaMA-7b
Neural Sequential Model, especially transformers, have been the most successful model in re- cent years. Their success is due to their ability to learn long-range dependencies and capture complex interactions between words. However, they are still far from perfect as they struggle with some of the challenges such as low performance on out- In this talk we will discuss how these models can be improved by using combination of pretraining techniques... ...
data or noisy labels.
What's great about the holiday season, is that it gives us a chance to reflect on what thankful for. I am so grateful and blessed with my family, friends, clients, colleagues, mentors, teachers, students... and all of you! I have been given an amazing opportunity in life to do something I love every day help people find their voice through music. It has truly changed me as person and helped me grow into who want to be. Thank you for heing part of this journey with me.
Al can create logo in seconds, but it's not the same as having an expert design one for you. AT is great at creating logos that are visually appealing and look good on paper or screen. But if your company needs to have a unique brand identity, then there's no substitute for working with a professional designer who understands what makes a strong visual statement.
LLaMA-5.4b w/o tune
Neural Sequential Model, especially transformers, have been widely used in many tasks. How- ever, the performance of these models is still limited by their lack of generalization ability and high dependence
data. In this paper propose novel method to improve the generalizabil- ity of neural sequential model: using an auxiliary task as a pre-training step for the main task. We use a simple image classification problem with 10 classes (cars) as our auxiliary task. ...
What's great about the holiday season, is that it gives us a chance to spend time with our family and friends. It also allows for some much needed downtime from work or school. But what happens when you have no one to celebrate with? Or maybe your loved ones are far away and can't make it home this year? Well, don't worry! There plenty of ways enjoy the holidays even if you aren't It surrounded by people who love you as much as you do them. Here are tips on how to survive being alone during Christmas:
Al can create a logo in seconds, but it's not the same as having an experienced designer. Al is great for creating logos and other visual elements that are simple or repetitive, like icons or illustrations. But when you need something more complex-like custom design with unique typography, color schemes, and layout- human creativity comes into play. That said, there some tools out there that make it easier to work with Al Al-generated designs: Adobe has released its own tool called Creative Intelligence (ACI).... ...
LJaMA-5.4b w/ tune
Neural Sequential Model, especially transformers, have been widely used in natural language processing (NLP) tasks. However, the performance of these models is still limited by their lack of robustness to noisy data and -iit----iitiibttion samples. In this paper we propose novel approach for improving the generalization ability of neural sequential model through incorporating adversarial training into the pretraining process. We show that our method can significantly improve the performance on various NLP tasks
What's s great about the holiday season, is that it gives us a chance to spend time with our loved ones. But what if you don't have any family or friends around? Don't worry! There are plenty of ways for you to enjoy your own little Christmas celebration and make this year even more memorable than ever before. Here some ideas: 1) Have an intimate dinner party at home- Invite close friends over for a cozy evening in where everyone can share their favorite dishes and stories from the past year. ... 2)
To figure out whether overfitting has occurred during the finetuning phase, potentially affecting the performance evaluation of the pruned models, we plot the loss curve of the model during the fine-tuning stage, as shown in Figure 3 We train for one epoch on the Alpaca dataset while using Wikitext2 as the evaluation set. The figure illustrates the train loss on Alpaca and the evaluation loss on Wikitext2. As is shown, the training loss is decreasing and converging normally, with no
18

---

significant fluctuations in the evaluation loss on Wikitext2, indicating that fine-tuning teniin is conducted on general data without specific optimization for Wikitex12, and there no occurrence of overfitting.
E
Generation Cases from Pruned Model
Table 15 shows the generation cases of the original LLaMA-7B model, the pruned LLaMA-5,44 model, and the pruned and finetuned LLaMA-5.4B model. All inference parameters are kept consistent. To avoid data contamination from the fine-tuning process, we following LLMMPrunne select three input cases. From a qualitative analysis perspective, the model post-pruning by 20% shows little difference from the original LLaMA-7B. After fine-tuning, the model's output tends to offer suggestions more, likely due to the influence of the Alpaca dataset, but it still maintains a high standard in terms of generation quality.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper clearly articulate the contributions in the Abstract and in the Introduction section, we highlight the advantages and application scenarios of our method.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of our proposed method in the Conclusion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer: [Yes]
Justification: We have provided a proof of mathematical process concerning a specific detail of the algorithm proposed in paper, with the proof process included in the Appendix. Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We submit our source code as an anonymized zip file, accompanied by a detailed replication guide.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
21

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: submit our source code as an anonymized zip file, accompanied by detailed replication guide.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
ctaaps..c/public/guue
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In the Experimental Settings section the Experiment chapter, we provide detailed descriptions of the models used, data sampling methods, and training details. Addi- tionally, more comprehensive configuration information is available the accompanying source code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The paper does not include error bars because of limited computational resources that prevented extensive experimentation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide all necessary running environments required for the experiments in the Experimental Settings section of the Experiment chapter.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have read the NeurIPS Code of Ethics and ensure that our paper strictly adheres to these guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In our conclusion section, we briefly discuss the potential adverse effects of large language models.
Guidelines:
23

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks as we do NOT release data models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In the paper, have duly cited all utilized models, datasets, and algorithms in an appropriate manner.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our paper identifies source code as crucial assets pertinent to our work, and is provided part of our submission in the form of anonymized zip file.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
25

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "bNTZu9269X", "reviewer": "Reviewer_jqTK", "summary": "The authors proposed a layer-wise pruning approach called SlimGPT that follows the Optimal Brain Surgeon framework but with a batched pruning procedure utilized to make it feasible on large models while remaining structured. The authors claim near-optimal pruning performance on commonsense reasoning datasets and wikitext ppl.", "questions": "1. Where is the plotting for the unpruned model in Figure 2? What pruning method is applied here? How much performance can fine-tuning recover in this setup?\n\n2. Is a pruned model superior to a smaller pretrained model? Like can a SlimGPT-pruned 13B model with a pruning ratio set around 45% be more performant than a 7/8B model? From the look of Tables 1 & 2, it seems a 50% pruned 13B model is significantly inferior to a 7B, and a 50% pruned 30B model is required to provide similar zero-shot task performance to an unpruned 7B, so this is unlikely. If confirmed, I am afraid this massively discounts the contribution of this work, as one can often just adopt a smaller pretrained model with no pruning or calibration necessary. Though I understand different application scenarios may call for models with different sizes, where the pretrained models can't cover them all.", "limitations": "The authors have provided a limitation and checklist section.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Models pruned via structured pruning approaches can naturally gain efficiency benefits, and the proposed method supposedly inherits this important property (though missing some efficiency reports). The task- and relatively architecture-agnostic nature of SlimGPT also makes it score well in terms of adaptability. The experiment reports indicate SlimGPT beats three other SOTA methods by a healthy margin (though its alignment requires some additional polish).", "weaknesses": "The main weakness of the paper is its eval, both in terms of alignment and coverage.\n\n1. Unaligned evaluation: Many of the compared baselines utilized a different calibration dataset and procedure, but it looks like only the LLM-Pruner is replicated in an aligned setup, while all results for other methods are copied from their original literature. This needs to be controlled, especially with only three methods to compare.\n\n2. Overly emphasis on the Llama family: The presented evaluation is solely conducted on various Llama 1/2 and Vicuna models, which are all Llama family-based. More coverage on other popular LLMs should be included. I'd recommend a healthy selection from Mistral, Phi, Gemma, Qiwen, and something along the line.\n\n3. Only on zero-shot commonsense reasoning tasks: As mentioned around line 243, the real task evaluation of this paper is conducted *\"under a zero-shot setting on the Commonsense Reasoning datasets, which encompass seven diverse subtasks...\"* This is not comprehensive enough. Common intelligence datasets like MMLU and GSM8k in typical few-shot setups should also be reported. Additionally, given the weak generation/long context performance in some recent layer pruning (but not layer-wise structured pruning in finer granularity) work like ShortGPT, I'd like to see SlimGPT evaluated on tasks like LongBench, InfinityBench, Needle-In-A-Haystack retrieval, and HumanEval on models with longer context window (e.g., mistral 32k).\n\n4. Incomplete efficiency report: There are no throughput or latency results, which are key efficiency metrics for conducting structured pruning in the first place and must be reported in efficiency literature, especially because different structured pruning methods can reflect different inference efficiencies, leading to work like ShearedLlaMA proposing targeted structured pruning. Also, the authors claim that *\"low-cost, low-resource, and time-efficient compression scheme\"* as their contribution in line 65, but there is no runtime or memory report on its pruning procedure.\n\n5. Not really a weakness, but the authors should consider giving a more detailed introduction of the compared baselines either in related work or around line 246.\n\nDespite my score of 4, I believe many of the raised concerns are addressable as there are mostly just more evals, and I am open to improving my rating upon proper rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the reviewer for the constructive comments. Due to text limitations, we try to answer your questions concisely and convincingly.\n\n> W1: Unaligned evaluation.\n\nWe acknowledge your concerns. Achieving a fully aligned experimental setup is challenging since pruning tasks differ from tasks like model optimization, and various pruning methods have unique principles and data requirements. For instance, `LLM-Pruner` has two stages: pruning and fine-tuning, while `Compresso` and `LoRAPrune` involve only one stage of sparse training. Our method, `SlimGPT`, also follows a two-stage process, aligning it with `LLM-Pruner`. Given the differing principles of `Compresso` and `LoRAPrune`, we make a compromise in terms of aligning the experimental environment. However, we believe that this does not affect the conclusions of our paper. In fact, we achieve better pruning results using fewer data resources or fewer iterations.\n\n> W2: Overly emphasis on the Llama family.\n\nThank you for your valuable comments. In our model selection, we reference previous works and the influence of LLama and Vicuna models but overlooked Vicuna's derivation from LLama. Due to time constraints, we choose Baichuan-7b, a prominent model in the Chinese community, for our experiments. This model includes a comprehensive MMLU and C-Eval evaluation script and is supported by LLM-Pruner, aiding our quick verification process.\n\nUsing the same setup as the paper, we prune Baichuan-7b by 20% with LLM-Pruner and SlimGPT. SlimGPT achieves slightly better PPL results (19.73 vs. 19.85) and significantly outperforms LLM-Pruner in all commonsense reasoning tasks (58.8 vs. 56.38).\n\nPrune%|Method|#Params|PPL↓|BoolQ|PIQA|HellasW|WinoG|ARC-e|ARC-c|OBQA|Avg\n-|-|-|-|-|-|-|-|-|-|-|-\n-|-|7B|13.25|70.09|76.93|70.06|64.09|67.05|40.53|38.20|60.99\n20%|LLM-Pruner|5.7B|19.85|62.87|74.48|63.03|60.93|60.31|36.86|36.20|56.38\n20%|SlimGPT w/o tune|5.7B|20.01|69.17|75.03|65.25|61.25|64.94|35.15|36.60|58.20\n20%|SlimGPT|5.7B|19.73|66.21|75.03|66.74|63.85|62.84|38.91|38.00|58.80\n\n> W3: Only on zero-shot commonsense reasoning tasks.\n\nThank you for your suggestions. We evaluated our model based on the official LLama report and previous research, focusing on language modeling performance (PPL) and zero-shot commonsense reasoning. While we believe these tasks are convincing, we recognize they may be insufficient for a complete assessment. To improve our evaluation, we perform 5-shot tests on Baichuan-7b using MMLU and cross-lingual task C-Eval, as seen in the table below.\n\nThe 5-shot evaluation results for MMLU show SlimGPT outperforming the baseline (35.4 vs. 24.3), and it also excels in the cross-lingual C-Eval dataset (28.7 vs. 22.4), despite reduced performance after finetuning on Alpaca.\n\nDataset|Prune%|Method|#Params|Humanities|Social Sciences|STEM|Other|Avg\n-|-|-|-|-|-|-|-|-\nMMLU|-|-|7B|38.1|49.2|35.2|47.7|42.1\n||20%|LLM-Pruner|5.7B|24.8|23.4|21.9|26.8|24.3\n||20%|SlimGPT w/o tune|5.7B|29.7|38.0|31.3|38.9|34.0\n||20%|SlimGPT|5.7B|32.2|39.2|31.1|40.2|35.4\nC-Eval|-|-|7B|47.2|50.5|36.4|45.5|43.3\n||20%|LLM-Pruner|5.7B|23.5|24.9|21.1|21.2|22.4\n||20%|SlimGPT w/o tune|5.7B|32.9|34.3|28.3|30.9|31.0\n||20%|SlimGPT|5.7B|31.8|33.1|24.2|29.9|28.7\n\nRegarding your suggestion for experiments on long-context performance, we recognize their importance but, due to time constraints, we won’t be able to conduct them at this moment. We plan to include them in future work for a more comprehensive study.\n\n> W4: Incomplete efficiency report.\n\nThank you for your suggestion. We include the experimental results and analysis on inference speed and pruning efficiency in the \"global response\" section, as other reviewers have similar questions. Please refer to that for detailed information.\n\n> W5: Detailed introduction of the compared baselines.\n\nDue to space constraints, we have omitted the baseline introduction in the final paper version. We apologize for any inconvenience and will include it in the updated version.\n\n> Q1: Further explanation on Figure 2. \n\nFigure 2 shows the output errors of various pruned models relative to the unpruned model, which has an error of zero and is not displayed (its PPL is 12.63). In our pruning approach, we utilize SlimGPT with the Alpaca dataset, pruning only the first layer to minimize output errors.\n\nTo assess performance recovery after finetuning, we present the PPL results below, where all models are finetuned with LoRA for one epoch using Alpaca. We observe that the PPL does not improve after finetuning, even for the unpruned model. We believe this may be due to the distinct distribution bwtween the instruction-following dataset Alpaca and test dataset Wikitext2, potentially causing overfitting of the unpruned weights and poorer performance on the test data, which requires further validation.\n\nModel|PPL (w/o tune)|PPL (w/ tune)\n-|-|-\nLLama-7B|12.63|15.63\nLayer0-prune-25%|12.86|16.86\nLayer0-prune-50%|13.98|17.31\nLayer0-prune-75%|21.49|31.27\n\n>  Q2: Is a pruned model superior to a smaller pretrained model?\n\nThis topic deserves discussion. Under **low-resource conditions**,  pruned models after finetuning generally perform worse than smaller pre-trained ones, as shown in our works and previous research(LLM-Pruner). However, with **full training**, pruned models can outperform their smaller versions, like in Sheared-LLama[1]. Thus, pruning may serve as a high-benchmark initialization method to lessen the need for extensive training. \n\nOur aim is to focus specifically on the task of LLM pruning itself. Under constrained resource conditions, such as when a more compact version of the model (e.g., less than 1B parameters) is required for edge-side deployment, LLM pruning and compression provide a cost-effective solution.\n\n[1]. Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ee93PA4E4E", "reviewer": "Reviewer_QvFZ", "summary": "This paper presents SlimGPT, a method for structured pruning of LLMs to balance performance with efficiency. The method is based on the OBS framework and introduces Batched Greedy Pruning to enhance pruning accuracy and speed. The authors also propose the Incremental Pruning Ratio strategy to mitigate performance loss due to error accumulation. Experimental results on LLaMA and other models demonstrate that SlimGPT achieves state-of-the-art results with significant improvements in performance retention compared to existing methods.", "questions": "1. In Figure 1., from my view, it seems that the output elements of the third head are all the smallest, so they are all reordered to the first head for pruning. My question is what if not all elements of a head are in the same magnitude order? In this case, how to conduct batched greedy pruning?\n2. Can you provide some insights about the reason why the pruning results with 2048 samples and 2048 sequence length start to decrease in zero-shot average metric?", "limitations": "Not applicable.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a novel method for structured pruning based on the OBS framework to accelerate large language models.\n2. The method is validated through comprehensive experiments on various models, showing improved performance and efficiency.", "weaknesses": "1. Since Table 9 illustrates the significant impact of the calibration dataset on the pruning performance, I doubt whether the selection of calibration samples is more important than the design of the pruning criterion. The experimental results compared in the paper are with different sampling strategies of the calibration dataset, so it is hard to evaluate the superiority of the proposed method.\n2. The design of the layer-wise sparsity is empirical with no theoretical analysis. Since the pruning within a layer is a greedy pipeline, it is unclear why the layer-wise sparsity design is not in a greedy pipeline.\n3. The inference speed should be provided for comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the reviewer for the constructive comments.\n\n> W1: The impact of the calibration dataset on the pruning performance.\n\nThank you for the valuable comment. Table 9 displays the pruning results of SlimGPT **without fine-tuning** (for convenience, Table 9 is reproduced below). When the calibration data is switched from C4 to Alpaca/GPT4-Alpaca, the PPL results indeed degrade (48.26/47.06 vs. 42.06). However, the Zero-shot Avg scores improve significantly (54.44/54.66 vs. 52.70), surpassing the **fine-tuned results** of all baselines. \n\n| Dataset      | PPL↓  | Zero-shot Avg.↑ |\n| ------------ | ----- | --------------- |\n| C4 (SlimGPT) | 42.06 | 52.70           |\n| Alpaca       | 48.26 | 54.44           |\n| GPT4-Alpaca  | 47.06 | 54.66           |\n\n\nThis observation highlights a trade-off between PPL and Zero-shot Avg. Due to SlimGPT's inherent parameter compensation mechanism, it is sensitive to the quality of the input data, so there is a distinct difference in impact between pre-trained data and instruction-following data. Additionally, our experiments show that random open-source pre-trained data can achieve SOTA results. Therefore, we believe that using higher-quality data tailored to specific domains provides SlimGPT with greater potential for improvement compared to other methods.\n\n> W2: The design of the layer-wise sparsity is empirical.\n\nThe core concept of SlimGPT is derived from the OBS framework, which addresses global model pruning by breaking it down into layer-wise subproblems. Each layer is optimized **sequentially** from shallow to deep. Previous studies, such as OBC[1] and GPTQ[2], have demonstrated that this method yields excellent results across various domains. And our primary objective is to apply this framework to the structured pruning of LLMs.\n\nWe would like to discuss the viability of the layer-wise greedy pipeline. Due to the unidirectional influence between layers, the current layer is impacted solely by the preceding layer and remains unaffected by subsequent layers. If the pruning process is not executed sequentially, the local optimality at each step would be compromised. We believe this would complicate the task and significantly increase the computational time required.\n\n[1]. Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning.\n[2]. GPTQ: Accurate Post-training Quantization of Generative Pretrained Transformers.\n\n> W3: The inference speed should be provided.\n\nThank you for your valuable suggestion. About the additional experiments on inference speed, we provide the experimental results and analysis in \"global response\" at the top. Please refer to that reponse for detailed information.\n\n> Q1: Further explanation on Figure 1.\n\nAs a structured pruning scheme, SlimGPT prunes attention blocks with the smallest pruning granularity at the head level. Consequently, heads are treated as indivisible units. We evaluate each head by summing the errors of all columns within it. For those interested in the specifics, our detailed algorithmic process is outlined in Algorithm 1.\n\n> Q2: Can you provide some insights about the reason why the pruning results with 2048 samples and 2048 sequence length start to decrease in zero-shot average metric? \n\nThank you for the comment. The Zero-shot Avg is essentially a mean value, which can be easily influenced by sub-tasks with significant fluctuations. To facilitate analysis, we provide detailed results for the commonsense reasoning tasks below.\n\n| Experiment | Sample Size | Sequence Length | BoolQ | PIQA  | HellasW | WinoG | ARC-e | ARC-c | OBQA  | Avg.  |\n| ---------- | ----------- | --------------- | ----- | ----- | ------- | ----- | ----- | ----- | ----- | ----- |\n| (a)        | 512         | 64              | 63.12 | 68.82 | 51.34   | 57.62 | 46.17 | 29.35 | 35.00 | 50.20 |\n|            | 512         | 1024            | 67.22 | 71.06 | 55.23   | 59.27 | 53.45 | 31.14 | 36.60 | 53.42 |\n|            | 512         | 2048            | 66.73 | 71.60 | 55.35   | 57.30 | 53.11 | 32.42 | 35.60 | 53.16 |\n| (b)        | 128         | 512             | 65.66 | 69.37 | 54.07   | 58.33 | 50.04 | 30.55 | 35.2  | 51.89 |\n|  | 256        | 512         | 63.76           | 70.29 | 54.47 | 58.88   | 53.07 | 30.89 | 34.6  | 52.28 |\n|            | 1024        | 512             | 65.60 | 71.65 | 54.35   | 57.46 | 53.07 | 30.63 | 35.40 | 52.59 |\n|            | 2048        | 512             | 63.30 | 71.71 | 54.79   | 57.38 | 53.2  | 31.66 | 34.60 | 52.38 |\n\nAs the sample length increases from 64 to 1024, there is a noticeable improvement in the metrics across various tasks. However, when the length is further increased to 2048, the rate of improvement slows down, and some metrics even decline, particularly for WinoGrande (59.27 vs. 57.3). We believe that since most commonsense reasoning tasks involve short text data, the impact of SlimGPT on these tasks may diminish when the sequence length of the calibration set exceeds a certain threshold. In fact, it could even weaken the model's understanding of short texts.\n\nWhen the sample size increases, the changes across the various subtasks are less consistent and exhibit smaller magnitudes. As shown in Figure 3 of the paper, the fluctuations in Zero-shot Avg in subplot (a) are significantly smaller than those in subplot (b), and there are two instances of decline. Both of these declines result from fluctuations in the BoolQ dataset (refer to the table above). Thus, we hypothesize that the BoolQ dataset is particularly sensitive to random sampling in the calibration set, which may result in the drop in Zero-shot Avg.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZNovR45Ld6", "reviewer": "Reviewer_8hG2", "summary": "This paper presents a novel SlimGPT framework to conduct structured pruning for LLMs in a fast and low-cost way. Specifically, SlimGPT modifies the Optimal Brain Surgeon (OBS) framework, and proposes a Batched Greedy Pruning to  enhance the performance of head-wise pruning through Cholesky decomposition. SlimGPT also improves the FFN pruning efficiency via Dynamic Group Size. Besides, SlimGPT employs an Incremental Pruning Ratio in order to mitigate the error accumulation problem in layer-wise pruning. Experiments on the LLaMA, LLaMA-2, and other popular LLMs demonstrate that SlimGPT achieves a new SOTA on LLM pruning.", "questions": "Under the same total sparsity value, does different increasing schedules affect the model inference speedup?", "limitations": "See Weaknesses and Questions. I do appreciate the theoretical contributions. So my rating may be adjusted after carefully checking the replies and other reviews.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper presents a proper way to extend the OBS framework to structured pruning with strong theoretical foundation. The technical details are thorough and convincing.\n2. Extensive experiment results demonstrate that SlimGPT successfully achieves a new SOTA, surpassing all related works in this field.", "weaknesses": "1. SlimGPT employs an Incremental Pruning Ratio strategy. In L220, the article specifies that a logarithmic increasing strategy performs well and is employed in all experiments. Actually, this should be considered more carefully. Pruning can be viewed as a method to get rid of unnecessary information in the activations and only preserve necessary components for later layers. From this perspective, it resembles that of Token Merging [1,2]. It is already demonstrated in [1,2] that the performance loss caused by an aggressive pruning schedule in the first layers  can be mitigated by re-training. Therefore, I suggest the authors test more increasing strategies, instead of the logarithmic strategy, to fully utilize the power of SlimGPT.\n2. Pruning at a $p\\\\%$ sparsity does not usually lead to an $\\frac{1}{1-p\\\\%}$ throughput speedup [2]. To demonstrate that SlimGPT actually helps to deploy LLMs, the authors should carefully compare the **throughput** (eg., tokens per sec.) of pruned models generated by SlimGPT and the competing baselines.\n\n[1] Token Merging: Your ViT But Faster (ICLR 2023)\n\n[2] PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptation (ECCV 2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the reviewer for the constructive comments.\n\n> W1: Further explanation about Incremental Pruning Ratio strategy.\n\nIn Section 5.3.2, we have discussed the impact of different pruning ratio strategies on performance, with detailed results presented in Table 5 (for convenience, Table 5 is reproduced below). Specifically, for the Incremental Pruning Ratio strategy, both logarithmic and linear approaches are employed. Each of these strategies offers distinct benefits: the linear approach achieves better Zero-shot Avg results but incurs a slight loss in PPL. Nevertheless, overall, whether the strategy is linear or logarithmic, the incremental scheme significantly outperforms uniform pruning or decrementing strategies.\n\n|                        | Model Size | PPL    | Zero-shot Avg. |\n| ---------------------- | ---------- | ------ | -------------- |\n| log increase (SlimGPT) | 3.40b      | 38.83  | 52.23          |\n| linear increase        | 3.34b      | 46.57  | 53.45          |\n| uniform                | 3.50b      | 123.05 | 44.34          |\n| log decrease           | 3.40b      | 380.69 | 36.73          |\n| linear decrease        | 3.34b      | 932.64 | 35.62          |\n\nPlease note that our experiments are conducted under **low-resource conditions**. After extensive fine-tuning on large-scale data, the differences in performance resulting from various pruning ratio strategies will diminish further (Sheared-LLama[1], even with uniform pruning, mitigated performance impacts through subsequent training on a large data scale). However, under resource-limited conditions, selecting an appropriate pruning ratio strategy can reduce the reliance on subsequent training, thus minimizing performance loss. This is particularly crucial for LLMs, as a complete training cycle demands substantial resources and time.\n\n[1]. Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning.\n\n> W2: Compare the throughput (eg., tokens per sec.) of pruned models generated by SlimGPT and the competing baselines.\n\nThank you for your valuable suggestion. About the additional experiments on inference speed, we provide the experimental results and analysis in \"global response\" at the top. Please refer to that reponse for detailed information.\n\n> Q: Under the same total sparsity value, does different increasing schedules affect the model inference speedup?\n\nIn general, under the same number of parameters, deeper models (with more layers) tend to have slower inference speeds. On the other hand, models with the same number of layers but different widths primarily experience variations in instantaneous computational load, which impacts speed differently depending on the performance and optimization of the GPU used. \n\nSince our current method typically does not affect the number of layers, as long as there isn't an extreme variation in width distribution, inference speed should not be significantly impacted. This is supported by the experimental results from the previous question.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We sincerely appreciate your valuable and insightful comments. Here I would like to address the concerns regarding inference speed or  pruning efficiency raised by all reviewers.

> Inference speed and memory usage report.

As the inference speed is primarily influenced by the final model structure and is not specifically tied to the pruning algorithm used (typically, the number of layers does not decrease), we initially omitted the inference runtime report. To demonstrate that SlimGPT actually helps to deploy LLMs, we provide the inference speeds for LLama-7b with 20% and 50% pruning, as shown in the table below. The batch size is set to 1, the maximum output limit is 512, and the average value is taken from 50 inference results. Additionally, we examine the impact of two different pruning ratio strategies on inference speed: uniform pruning and the Incremental Pruning Ratio strategy. All supplementary experiments were conducted in an environment with NVIDIA H20.

In the case of pruning 50% of the parameters using the log increase strategy, the model's memory usage during inference is reduced to 51% (14297MB vs. 27737MB), and the inference latency decreases to 63% (9.21ms vs. 13.51ms). When employing uniform pruning, both memory usage and latency experience slight reductions, although it is important to note that the parameter counts are not entirely equivalent between the two methods.

| Prune% | Strategy     | #Params | Memory  | Avg Latency (per token) |
| ------ | ------------ | ------- | ------- | ----------------------- |
| -      | -            | 6.7B    | 27737MB | 13.51ms                 |
| 20%    | log-increase | 5.4B    | 22497MB | 11.89ms                 |
| 50%    | log-increase | 3.4B    | 14297MB | 9.21ms                  |
| 50%    | uniform      | 3.4B    | 13793MB | 9.05ms                  |

> Pruning runtime and memory usage report.

Regarding the runtime and memory usage during the pruning procedure, we have mentioned briefly that all pruning processes can be completed within 1 GPU hour (using A100 hardware). Specifically, the memory usage varies depending on the model size and the calibration size, and the pruning speed is additionally influenced by the pruning ratio. We present the pruning efficiency results from our paper experimental setup. 

Since SlimGPT operates in a layer-wise manner, we don't need to load the entire model but only load the parameters of the current layer and the corresponding input features at one time, which significantly reduces memory usage. For the task of pruning the 13B model by 50%, we only require 12 GB of GPU memory and 41 minutes to complete the process.

| Model     | memory | prune-20%-runtime | prune-50%-runtime |
| --------- | ------ | ----------------- | ----------------- |
| LLama-7b  | 7375M  | 678.4s            | 1073.9s           |
| LLama-13b | 11601M | 1417.1s           | 2475.3s           |

Meta-review: The authors introduced a low-cost and fast structured pruning method named SlimGPT. Experiments results show that SlimGPT outperforms other methods and achieves state-of-the-art performances.

We appreciate the responses from authors and the discussions with reviewers. Please add all the mentioned experiments and analysis into the final version.
Decision: Accept (poster)