Title: SlimGPT: Layer-wise Structured Pruning for Large Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Model Compression

Structured Pruning

Layer-wise Pruning

Large Language Model
Abstract: Large language models (LLMs) have garnered significant attention for their remarkable capabilities across various domains, whose vast parameter scales present challenges for practical deployment. Structured pruning is an effective method to balance model performance with efficiency, but performance restoration under computational resource constraints is a principal challenge in pruning LLMs. Therefore, we present a low-cost and fast structured pruning method for LLMs named SlimGPT based on the Optimal Brain Surgeon framework. We propose Batched Greedy Pruning for rapid and near-optimal pruning, which enhances the accuracy of head-wise pruning error estimation through grouped Cholesky decomposition and improves the pruning efficiency of FFN via Dynamic Group Size, thereby achieving approximate local optimal pruning results within one hour. Besides, we explore the limitations of layer-wise pruning from the perspective of error accumulation and propose Incremental Pruning Ratio, a non-uniform pruning strategy to reduce performance degradation. Experimental results on the LLaMA benchmark show that SlimGPT outperforms other methods and achieves state-of-the-art results.

SlimGPT: Layer-wise Structured Pruning for Large Language Models

Abstract

Large language models (LLMs) have garnered significant attention for their remarkable capabilities across various domains, whose vast parameter scales present challenges for practical deployment. Structured pruning is an effective method to balance model performance with efficiency, but performance restoration under computational resource constraints is a principal challenge in pruning LLMs. Therefore, we present a low-cost and fast structured pruning method for LLMs named SlimGPT based on the Optimal Brain Surgeon framework. We propose Batched Greedy Pruning for rapid and near-optimal pruning, which enhances the accuracy of head-wise pruning error estimation through grouped Cholesky decomposition and improves the pruning efficiency of FFN via Dynamic Group Size, thereby achieving approximate local optimal pruning results within one hour. Besides, we explore the limitations of layer-wise pruning from the perspective of error accumulation and propose Incremental Pruning Ratio, a non-uniform pruning strategy to reduce performance degradation. Experimental results on the LLaMA benchmark show that SlimGPT outperforms other methods and achieves state-of-the-art results.

1 Introduction

Large Language Models (LLMs) [1, 2, 3] have made significant strides in various natural language processing tasks, leading to the emergence of novel applications such as AI agents [4]. One of the factors contributing to the exceptional capabilities of LLMs is their massive parameter scales. However, these extensive parameters also introduce increased inference costs and deployment challenges, hindering the widespread application and adoption of LLMs. Accelerating inference for LLMs has become a focal point of current research. Model compression [5], as one of the strategies for inference acceleration, including techniques like pruning and quantization [6, 7], has been extensively researched. Nevertheless, earlier model compression techniques, particularly model pruning, typically rely on heavy post-training to recover the model's capabilities, which typically involves retraining with the entire training dataset. Given the constraints of current computational resources, the above approaches are not feasible for LLMs.

In the domain of LLM pruning, recent studies have largely focused on unstructured (or semistructured) pruning [8], a method that shrinks models by selectively zeroing out weights considered non-critical. Despite its advancements, unstructured pruning falls short in substantially reducing parameter count, which is crucial for accelerating LLM inference as it is often bottlenecked on memory bandwidth and communication [9]. To accelerate inference speed, unstructured pruning models are often paired with specialized frameworks or hardware solutions. Conversely, structured pruning [10, 11] effectively decreases the model's parameter count by systematically eliminating columns or rows from weight matrices, enabling significant improvements in inference speed, and

reduce deployment cost on conventional hardware. Yet, structured pruning often entails more pronounced compromises in model performance, which poses a greater challenge.

Recently, researchers have applied the classic Optimal Brain Surgeon (OBS) framework to the compression of LLMs. This approach includes parameter compensation which can mitigate the loss incurred during compression and reduce the dependence on post-training. The OBS framework is currently applied in the areas of unstructured pruning [12] and quantization [13] for LLMs. However, there exist some challenges in its application to structured pruning:

The OBS is a fine-grained compression framework that compresses one parameter at each iteration, whereas structured pruning has a minimum granularity of either a column or head. Directly applying the OBS framework will result in high numerical errors, impairing model performance.

The OBS is essentially a layer-wise compression method. It focuses on each individual layer, thus failing to allocate pruning ratios for each layer rationally using global information (such as global gradients). This is crucial for LLM structured pruning, which relies on a non-uniform strategy to reduce the impact on performance.

To address these issues, we propose a new structured pruning method for LLMs. We introduce Batched Greedy Pruning to achieve low-cost and rapid pruning for LLMs. Specifically, for attention heads, we propose grouped Cholesky decomposition to select nearly optimal heads for pruning in each iteration, thereby maintaining an approximately locally optimal pruning result. For Feed-Forward Networks (FFNs), we achieve near-optimal and efficient pruning results through Dynamic Group Size. Furthermore, since the OBS is essentially a layer-wise compression framework, we investigate the error accumulation phenomenon in layer-wise pruning and propose pruning by Incremental Pruning Ratio, a straightforward non-uniform strategy to control the pruning rate of each layer, further mitigating performance loss under a given overall pruning ratio.

Contribution. In this paper, we propose SlimGPT, a layer-wise pruning approach that extends the classical OBS framework to structured pruning for LLMs. The characteristics of SlimGPT can be summarized as follows: (i) Task-agnostic pruning scheme. Only a random sample of data from generic pre-training corpora is needed as a calibration set, and we can obtain a compressed model with most performance preserved; (ii) Low-cost, low-resource, and time-efficient compression scheme. The model can be compressed using just a single GPU, a few hundred of calibration data, and about one hour; (iii) A universal pruning method for Transformer-based models. It has good transferability and, theoretically, is applicable to all large models based on the conventional Transformer architecture. We employ LLaMA models for pruning and conduct evaluations on wikitext2 and Commonsense Reasoning tasks. The results indicate that SlimGPT substantially retains the performance of the pruned models, surpassing state-of-the-art methods.

2 Related Work

Compression methods with regularization . Before the era of LLMs, using the scaling factors from Batch Normalization layers as indicators of channel importance made pruning based on regularization a very popular method [14, 15]. Notably, Louizos et al. [16] implemented the non-differentiable L0 penalty in a differentiable form, a technique frequently used for pruning in large models. Compresso [17] combines L0 regularization with LoRA training [18], effectively preserving model performance at a low cost. In a similar vein, Sheared LLaMA [19] employs augmented L0 regularization on inserted masks for structured pruning, using extensive data to restore performance and deliver compact yet powerful pruned models.

Global gradient-based compression methods . NVIDIA's works [20, 21] involve a Taylor expansion of the global loss. By eliminating higher-order terms, it is revealed that the impact of a weight on the loss can be assessed using the magnitude of the weight combined with gradient information. Based on this, LLM-Pruner [11] employs a first-order importance estimation to gauge the importance of weights. LORAPrune [22] measures the importance of weights based on the gradients of the LORA parameters rather than the model's parameters, achieving commendable results.

Outliers-dependent compression methods . Dettmers et al. [23] identifies an attribute unique to LLMs, where a small subset of activation values in the data features have magnitudes significantly larger than the others. And removing corresponding weights impacts model performance substantially. Building upon this, Wanda [24] proposes a simple yet effective unstructured pruning method, using

the product of a weight's L1 norm and the L2 norm of eigenvalues to gauge its importance, achieving impressive pruning results. OWL [25] determines layer-wise sparsity ratios based on Layerwise Outlier Distribution (LOD), obtaining substantial performance gains at high sparsity levels.

Layer-wise compression methods . The early works [26, 27] provide a layer-wise compression framework with a locally optimal solution named Optimal Brain Surgeon (OBS). And then OBC [28] reduces the computational burden by converting layer-wise pruning into row-wise pruning and updating the inverse Hessian using a proposed formula. Furthermore, GPTQ [13] accelerates the process with Lazy Batch-Updates and Cholesky Reformulation, enabling the application of this method to the quantization of LLMs. SparseGPT [12] also adapts this approach for unstructured pruning of LLMs. However, there appears to be no existing research that has implemented OBS in structured pruning for LLMs.

Structured Pruning vs. Other Techniques . Given that OBS has previously been used in both quantization and unstructured pruning, and is now being applied to structured pruning, there is an inherent consistency across these three compression schemes. These methods actually compress the model at varying levels of granularity. Quantization, which "trims" floating-point precision, represents the finest granularity and delivers excellent compression outcomes. Structured pruning, on the other hand, involves trimming weight vectors and represents the coarsest granularity, naturally resulting in higher performance losses compared to other methods, which poses significant challenges. For small models, it is possible to recover most of the performance with post-training, but this is challenging to achieve in LLMs due to resource constraints. Nonetheless, structured pruning effectively reduces the number of parameters without needing special inference framework support and is compatible with the other two methods, thus still holding considerable potential for application.

3 Preliminary

Layer-Wise Pruning. Consider the scenario of pruning on a well-optimized model, known as post-training pruning, a prevalent approach involves decomposing the global model pruning challenge into layer-wise subproblems ( i.e., Layer-wise pruning), which are typically modeled as issues of minimizing L2 error. Specifically, let W l represent the weights of the l -th layer of a pretrained model and X l be the input features for layer l . The goal is to determine pruned weights ˆ W l that achieve a predefined pruning ratio while minimizing the squared error:

Optimal Brain Surgeon (OBS) Framework. As Equation 1 can be rewritten as the sum of square error of each row of the weights to be pruned, the layer-wise pruning can be further split into row-wise pruning [28]. Consider the removal of a single weight from a row in W l , Equation 1 has a closed-form solution [27]. Let w denote a specific weight in a row of W l , and let p be its corresponding index. Given that our optimization objective is to minimize row-wise squared error, the Hessian of this objective with respect to the weight row of layer l is given by H l = 2 X l X T l . The weight to be pruned, w p , as well as the necessary update δ p applied to the remaining weights of the same row to counterbalance the removal, can be determined through the following calculation:

where H -1 p,p denotes the p th diagonal entry of the inverse Hessian, and H -1 : ,p is its p th column. By iteratively using Equation 2 to remove one weight and update the remaining weights in the same row, one can obtain a locally optimal compressed model. After each iteration, H will be updated by removing the p row and column, which is represented by H [ -p ] , here we use [ -p ] to indicate the removal of p row and column of the matrix. As H -1 cannot be updated by simple removal as ( H [ -p ] ) -1 = ( H -1 ) [ -p ] , to avoid the expensive full recomputations of H -1 , the following formula is proposed to quickly update H -1 [28]:

This framework can be practically applied to medium-sized models. However, for models with billions of weights, the iterative pruning becomes exceedingly time-consuming.

̸

4 Methodology

In this section, by extending the OBS framework to structured pruning, we introduce SlimGPT from two aspects: (1) By employing Batched Greedy Pruning to reduce error computation, we minimize the performance degradation caused by pruning while also accelerating the pruning speed; (2) By analyzing the limitation of layer-wise pruning from the perspective of error accumulation, we introduce Incremental Pruning Ratio , a non-uniform pruning strategy.

4.1 Structured Pruning with OBS Framework

As mentioned above, the pruning between different rows is independent, making it possible to prune all rows simultaneously [29]. We extend the OBS framework to structured column pruning, i.e., pruning one column at a time and compensating the rest columns using the following formula:

where H -1 p, : denotes the p -th row of H -1 , and the obtained ∆ is a compensation matrix of the same size as W . We following previous works employ attention blocks and FFNs as the smallest units for pruning. By pruning the columns of the output matrix in attention blocks and the dimensionality reduction matrix in FFN blocks, we reduce the number of attention heads and FFN channels, thereby decreasing the model's parameter count.

However, the above formula cannot be applied directly, as iteratively finding and pruning the column with the minimum error is time-consuming. More critically, the structural dependency in attention blocks imposes additional constraints on column pruning, making it impossible to evaluate the importance of a head based solely on information from a single column.

4.2 Batched Greedy Pruning

Given that the calculation of the pruning error requires only the diagonal elements of H -1 (see Equation 4), which are updated after each iteration, computing these elements in advance allows for calculating the head-wise error. With the observation that the sequential row removal via Equation 3 for the symmetric H -1 essentially corresponds to taking a Cholesky decomposition [13], we can obtain the elements in advance with Cholesky decomposition.

Hoewever, the matrix obtained by Cholesky decomposition is triangular, and the elements of the current row (column) are calculated based on the elements of all the previous rows (columns), which means the Cholesky decomposition breaks the comparability between rows (columns). So it is hard to obtain all the required information in advance through the Cholesky decomposition like [12, 13], whose error comparison is usually within the same column but structured pruning requires the comparison of different columns.

Since structured pruning only requires traversing the columns that need to be removed, by rearranging the rows and columns corresponding to a head that is to be pruned in H to the front, and then invert the matrix followed by Cholesky decomposition, we can calculate the

…

0.9 …

…

…

0

0.8

0

0

…

0.6 …

…

…

0

0.4

0

0

…

0.3 …

…

…

0

0.2

0

0

…

0.9 …

…

…

0

0.8

0

0

Column Reorder

𝑾

𝑯

!𝟏

0.7

…

0.4

1.1

…

0.5

0.2

…

0.3

…

…

…

…

…

…

…

…

…

0.4

…

0.5

0.9

…

0.8

0.1

…

0.2

… …

0.72

…

0.42

1.08

…

0.51

0.21

…

0

…

…

…

…

…

…

…

…

0

0.41

…

0.51

0.89

…

0.82

0.11

…

0

… …

0.71

…

0.41

1.09

…

0.52

0.20

0

0

…

…

…

…

…

…

…

0

0

0.42

…

0.50

0.91

…

0.83

0.12

0

0

… …

0.73

…

0.43

1.11

…

0.53

0

0

0

…

…

…

…

…

…

0

0

0

0.41

…

0.52

0.92

…

0.84

0

0

0

0

0

0

1.11

…

0.53

0.73

…

0.43

0

0

0

…

…

…

…

…

…

0

0

0

0.92

…

0.84

0.41

…

0.52

Column Restore

Grouped Cholesky Decomposition

Cholesky Decomposition

Reordered

𝑯

!𝟏

0.2

…

0.3

1.1

…

0.5

0.7

…

0.4

…

…

…

…

…

…

…

…

…

0.1

…

0.2

0.9

…

0.8

0.4

…

0.5

head 0

head 1

head 2

to be pruned

Compensate

Algorithm 1 Batched Greedy Pruning for Attention Heads Given Weight matrix W , inverse Hessian H -1 , head size d and head count n .

head error column-wise. However, repeated rearrangement followed by matrix inversion and Cholesky decomposition is highly time-consuming, and this is just to find one head to be pruned.

We accelerate the above process through two common lemmas (proofs are provided in the Appendix): (i) For symmetric H , the inverse matrix after permutation can be obtained by the same permutation of H -1 ; (ii) The principal submatrix of symmetric H -1 after Cholesky decomposition is equivalent to the Cholesky decomposition of its principal submatrix. Thus we can calculate the pruning error of all the heads at once through grouped Cholesky decomposition . Specifically, we inverse H once and split it into n head matrices along the main diagonal, with each remains definite and symmetric, and decompose them in parallel:

where decomposed ˆ H -1 is a matrix of size n head × d head × d head , n head and d head represent the head number and head dimension, respectively. Utilizing GPU acceleration, we can quickly calculate the value of the diagonal element in advance and calculate the head-wise error. Note that during error computation, we only update the diagonal elements of H -1 and skip the update of W , which is small and does not dominate the ordering of errors.

After determining the head to be pruned, we rearrange the corresponding columns of W and the corresponding rows and columns of H -1 to the front, and again use the global Cholesky decomposition on reordered H -1 to prune the head column by column until the first head is pruned. In this way, we can avoid traversing columns that do not need pruning and only traverse necessary columns to improve pruning efficiency further. Figure 1 shows the process of Batched Greedy Pruning applied to attention blocks, and Algorithm 1 is a pseudocode illustrating how to prune a head with two steps: calculating head-wise error and pruning a head column-wise.

For FFNs, since there is no block constraint similar to attention heads, we can achieve local numerical optimality by pruning columns individually using Equation 4. However, the column-wise pruning is time-consuming because of the substantial intermediate dimensions of FFN. We thus prune a group of columns at a time and select the top-k columns with the most minor errors for pruning at each iteration. Considering that the compensation at each iteration may lead to a local reshuffling of column errors, we adopt a dynamic grouping strategy for pruning FFN blocks. We start with larger group size such as 1024 for pruning and gradually decrease the group size to a small number like 8, which allows us to enhance pruning efficiency while approaching an approximate optimal solution.

4.3 Incremental Pruning Ratio

Through Batched Greedy Pruning, we can obtain near-optimal structured pruning results for each layer. However, finding a suitable pruning ratio for each layer is difficult, as considering global information is quite challenging for layer-wise pruning, which only provides optimal pruning results

for the current layer. Maintaining a uniform pruning ratio across all layers is unreasonable and will impact model performance, especially when the pruning ratio is high. Existing works have different approaches to the problem. For example, LLM-Pruner [11] avoids pruning in the initial and final layers while maintaining a consistent ratio in the intermediate layers to manually implement non-uniform pruning. OWL [25] adjusts sparse ratios dynamically for each layer based on the proportion of feature outliers, which is applied to unstructured pruning.

We find that layer-wise pruning, particularly structured layer-wise pruning, suffers from error accumulation due to its locality. Errors introduced during pruning in one layer can be amplified in subsequent layers, resulting in significant discrepancies between the final model output and the original. Figure 2 presents the per-layer output error of FFN between the original model and three distinct pruned models. The pruned models each implement a first-layer pruning of 25%, 50%, and 75%, respectively. The error increases with model depth and accumulates at a rate exceeding linear progression as the initial layer's pruning ratio increases. Based on this observation, we propose a straightforward pruning strategy for layer-wise pruning, termed Incremental Pruning Ratio, which can effectively minimize pruning losses without any additional operation.

In Incremental Pruning Ratio, without loss of generality, we employ a logarithmically increas-

0

5

10

15

20

25

Layer index

0.000

0.002

0.004

0.006

0.008

0.010

Squared error

Layer0_P0.25:PPL=12.64

Layer0_P0.5:PPL=13.98

Layer0_P0.75:PPL=21.49

ing strategy to control the layer-wise pruning ratio. Specifically, for an n -layer model with the first and last layer pruning ratios denoted as r 0 and r n -1 respectively, the pruning ratio for the i -th layer is defined as follows:

where r i represents the pruning ratio for the i -th layer. This formula ensures that the pruning ratio from the first layer to the last layer transitions smoothly as a logarithmic curve. The strategy mitigates the pruning error accumulation in shallow layers while avoiding the issue of excessive pruning in the deeper layers, allowing for further reduction in performance loss.

5 Experiment

5.1 Experimental Settings

Implementation details. We use C4 dataset [30] as the calibration set. From the first shard of C4, we randomly select 256 2048-token sequences for pruning. To restore performance, we following LLM-Pruner [11] finetune the pruned model with LORA [18]. We tune with Alpaca datsets [31] for one epoch and utilize the AdamW optimizer with an initial learning rate set to 1e-4, coupled with a cosine annealing schedule for the learning rate. The global batch size is set to 64 and the sequence length is truncated to 256. All pruning experiments are conducted on a single A100, while finetuning is performed using two A100s.

Models and Metrics. To assess the effectiveness and generality of SlimGPT, We carry out a series of experiments on the LLaMA families [2]. And to measure the effectiveness of our pruned models in the task-agnostic setting, we follow previous pruning works to evaluate language modeling performance and commonsense reasoning capabilities. The language modeling performance is evaluated on the WikiText2 [32] validation set with sequence length truncated to 128, and the commonsense reasoning capabilities is carried out under a zero-shot setting on the Commonsense Reasoning datasets, which encompass seven diverse subtasks: BoolQ [33], PIQA [34], HellaSwag [35], WinoGrande [36], ARCeasy [37], ARC-challenge [37], and OpenbookQA [38]. We utilize the lm-eval-harness framework [39] to conduct these evaluations.

To validate the universality of SlimGPT, we conduct experiments on additional models and supplementary evaluation datasets. The results of these experiments can be found in the Appendix. We conduct further pruning experiments on Vicuna [40], LLaMA2 [41], and Baichuan [42], which yield results consistent with those observed using the LLaMA model. In addition, we engage in preliminary evaluations on more complex tasks, specifically MMLU [43] and LongBench [44]. Although SlimGPT exhibits slightly larger performance losses on these datasets, it still retains a significant advantage over the baseline models.

Baselines. We compare SlimGPT with the following recent SOTA works on structured pruning, which we could find during our experiments:

LLM-Pruner [11], a gradient-based pruning approach, serves as our benchmark. This method involves a two-step process: a one-shot pruning followed by performance restoration through LORA fine-tuning.

Compresso [17] is a pruning method based on sparse training, applying L0 penalty to manually inserted masks during the LORA fine-tuning phase and employing a cubic sparsity schedule to iteratively prune the model until the desired pruning ratio is achieved.

LoRAPrune [22] utilizes gradients from the LORA module's parameters to determine the importance of the original model's parameters, thus requiring only gradient information from the LORA module, which significantly reduces computational demands.

5.2 Main Result

5.2.1 Performance Evaluation

To facilitate a more effective comparison of the evaluated results with prior works, we prune the LLaMA-7B model using four distinct pruning ratios-20%, 25%, 33%, and 50%-resulting in four smaller models with parameter counts of 5.4B, 5B, 4.5B, and 3.4B, respectively. Table 1 shows the detailed perplexity and zero-shot performance of pruned LLaMA-7B with four different sizes. Compared to other approaches, SlimGPT demonstrates superior performance in language modeling and commonsense reasoning across most subtasks. Under a pruning condition of 20%, SlimGPT achieves a slightly better perplexity score than the best existing results (16.68 vs. 16.80) and shows

a 3.6-point improvement in zero-shot average (65.07 vs. 61.50). As the pruning ratio increases to 50%, the advantages of SlimGPT become even more pronounced. SlimGPT without post-training represents an approximately 8% improvement over the baseline LLM-Pruner in average performance (52.23 vs. 48.35), and with post-training, the average performance improvement reaches up to 11% (53.76 vs. 48.35). Specifically, on a dataset like Hellaswag, the improvement soars up to 25% (59.94 vs. 47.86).

Moreover, we observe that although SlimGPT affects different subtasks to varying degrees, its impact is relatively balanced across different tasks, eliminating the occurrence of disproportionately large losses in particular tasks. At lower pruning ratios, some tasks such as BoolQ can even outperform the original unpruned model. Additionally, the effects of fine-tuning also differ among tasks, significantly improving tasks like HellaSwag and ARC-easy, while potentially causing negative side effects for tasks such as BoolQ and WinoGrande. This phenomenon is likely closely associated with the datasets used for fine-tuning.

For larger-scale models such as LLaMA-13B and LLaMA-30B, previous works have not provided pruning results for these models. Therefore, we solely compare our results to the LLM-Pruner baseline, concentrating on two specific pruning settings: a lower pruning ratio (20%) and a higher pruning ratio (50%). The replication of LLM-Pruner is consistent with the method described in the paper, where the pruned models by LLM-Pruner are finetuned with LORA.

Table 2 presents the pruning results of LLaMA-13B and LLaMA-30B, and we can draw similar conclusions: SlimGPT outperforms LLM-Pruner in terms of both PPL and zero-shot average scores even without post-training. Note that as the scale of the model increases, the performance loss due to pruning becomes smaller, suggesting a higher degree of parameter redundancy in larger models. At a low pruning ratio of 20%, the LLaMA-13B model's average performance in commonsense reasoning is nearly on par with that of the original, unpruned model (68.06 vs. 68.16). Similarly, the pruned LLaMA-30B model slightly outperforms the unpruned version (72.56 vs. 71.92). For the perplexity task, even though SlimGPT exhibits gaps compared to the original model, it still performs better than baseline, even at low pruning ratios.

Besides, we can find that the performance of LLaMA-13B pruned by 50% falls short compared to LLaMA-7B pruned by 20%. This highlights the limitations of low-cost fine-tuning, where resource constraints and training with techniques like LoRA result in limited performance recovery for the model. Therefore, using lower pruning ratios to compress smaller LLMs yields better returns.

5.2.2 Efficiency Analysis

The pruning runtime and memory usage for LLaMA-7B and LLaMA-13B are detailed in Table 3. Memory usage fluctuates based on the model size and the calibration scale, while the pruning speed is additionally affected by the pruning ratio. We demonstrate the pruning efficiency results derived from our experimental setup. Utilizing SlimGPT, which operates on a layer-wise basis, there is no need to load the entire model at once. Instead, we only load the parameters of the current layer along with the corresponding input features, significantly reducing memory consumption. For instance, to prune the 7B model by 20%, approximately 7 GB of GPU memory and 18 minutes are required to complete the process. Similarly, pruning the 13B model by 50% necessitates around 12 GB of GPU memory and 41 minutes to finalize.

Table 4 illustrates the inference latency and memory usage of the pruned LLaMA-7b models. We prune LLaMA-7b by 20% and 50% respectively. The maximum output limit is set to 512 and the presented values are the average derived from 50 inference trials. When pruning 50% of the parameters, the memory usage of the model during inference decreases to approximately 51% (14297MB vs. 27737MB), and the inference latency is reduced to about 69% (9.21ms vs. 13.51ms).

5.3 Ablation Study

We systematically analyze the influence of several key parts of SlimGPT on the pruning effect, including the Batched Greedy Pruning and Incremental Pruning Ratio strategy. Within the calibration dataset, we conduct thorough experiments with sample sizes and sequence lengths. Unless specifically stated otherwise, all the following experiments are conducted under the condition of pruning 50% of LLaMA-7b without further post-training, to eliminate potential confounding effects. Supplementary ablation experiments can be found in the Appendix.

5.3.1 Impact of Batched Greedy Pruning Strategy

Weleverage grouped Cholesky decomposition to enhance the accuracy of head-wise error computation in attention blocks. Similarly, for FFNs, our proposed Dynamic Group Size substantially increases pruning efficiency while preserving near-optimal pruning results. To validate the effectiveness of these two strategies, we start with the complete SlimGPT algorithm and first remove the Dynamic Group Size (denoted as '-DGS'), setting the group size for FFN pruning to a fixed value of 128. Then, we remove the grouped Cholesky decomposition (denoted as '-GCD') and use the initial H -1 to calculate head-wise errors. The experimental results are shown in Table 5. For attention blocks, the grouped Cholesky decomposition strategy plays a key role in language modeling capabilities by improving the accuracy of error compensation. Replacing it with ordinary Cholesky decomposition results in a significant increase in PPL (38.83 vs 54.94). In comparison to the naive fixed group size scheme for FFNs, the Dynamic Group Size strategy proposed contributes to maintaining the model's commonsense reasoning performance (52.23 vs 51.63).

128

256

512

1024

2048

Sample size

40

42

44

46

48

50

PPL

PPL

50.5

51.0

51.5

52.0

52.5

53.0

53.5

Zero-shot Avg.

Zero-shot Avg.

(a)

64

128

256

512

1024

2048

Seqence length

40

45

50

55

60

65

PPL

PPL

49

50

51

52

53

54

Zero-shot Avg.

Zero-shot Avg.

(b)

5.3.2 Impact of Incremental Pruning Ratio Strategy

The Incremental Pruning Ratio is a strategy specifically proposed for addressing the issue of layer-wise pruning. To maintain generality, we selected various common non-uniform strategies for comparative experiments, including logarithmic and linear increase strategies, as well as their corresponding decrease strategies. Among these, the logarithmic increase strategy is the default configuration for SlimGPT. Additionally, we conduct experiments under the setting of uniform pruning. Table 6 details the results under the different settings. From an overall perspective, the increase strategy for the pruning ratio has a clear advantage over uniform, and likewise, uniform shows a distinct advantage over decrease. Such results further verify the phenomenon of layer-wize error accumulation. As for the increase strategies of logarithmic and linear changes, due to disparities in model sizes, their results are not entirely comparable. The former performs best in language modeling (38.83), while the latter shows better performance in common sense reasoning tasks (53.45).

5.3.3 Effects of Calibration Samples & Sequence Length

We delve further into the impact of calibration samples and sequence length, and we choose C4 dataset for our experiments as it has a longer average sequence length. In exploring the effects of the sample scale, we fix the sequence length at 256 and test five scales ranging from 128 to 2048; similarly, when investigating the impact of sequence length, the sample scale is set to 256, with choices of sequence length varying from 64 to 2048. Figure 3 presents the perplexity result and zero-shot performance with different calibration samples and sequence lengths. As the number of samples increases, the PPL and zero-shot averages show a positive overall trend. Furthermore, after the sample count reaches 2048, the PPL does not bottom out, and there is room for further reduction. Similar phenomena can be observed in experiments on sequence length. With more sufficiently high-quality datasets with longer sequences, we believe SlimGPT can achieve better pruning effects.

6 Conclusion

In this work, we introduce a fast, structured pruning method for large-scale models within resourceconstrained scenarios, based on the OBS framework, termed SlimGPT. Leveraging the novel Batched Greedy Pruning, we enhance the accuracy of pruning error estimation, thereby minimizing performance degradation from pruning. Moreover, we analyze the limitations of layer-wise pruning from the perspective of error accumulation and propose a non-uniform strategy named Incremental Pruning Ratio, which effectively improves the pruned model's performance. Evidence from open-source experiments affirms the efficacy of our approach.

Limitations. Even though SlimGPT achieves SOTA results in the structured pruning of LLMs, the model performance degradation at high pruning ratios ( e.g., 50%) or on more complex tasks ( e.g., LongBench) is still significant. How to enhance the model compression effectiveness under lowresource conditions remains a challenge. Moreover, we utilized a naive logarithmic change strategy in the Incremental Pruning Ratio, which, while ensuring generality, is not the optimal solution. The most suitable non-uniform approach requires further exploration. Lastly, similar to many large-scale open-source models available today, the model obtained through pruning by SlimGPT poses risks in terms of ethical safety and requires cautious handling.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609 , 2023.

Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864 , 2023.

Tejalal Choudhary, Vipul Mishra, Anurag Goswami, and Jagannathan Sarangapani. A comprehensive survey on model compression and acceleration. Artificial Intelligence Review , 53:5113-5155, 2020.

Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. The Journal of Machine Learning Research , 22(1):10882-11005, 2021.

Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. In Low-Power Computer Vision , pages 291-326. Chapman and Hall/CRC, 2022.

Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 , 2015.

Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning , pages 19274-19286. PMLR, 2023.

Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies in Computing Systems (JETC) , 13(3):1-18, 2017.

Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. arXiv preprint arXiv:2305.11627 , 2023.

Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning , pages 10323-10337. PMLR, 2023.

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323 , 2022.

Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision , pages 2736-2744, 2017.

Tao Zhuang, Zhixuan Zhang, Yuheng Huang, Xiaoyi Zeng, Kai Shuang, and Xiang Li. Neuronlevel structured pruning using polarization regularizer. Advances in neural information processing systems , 33:9865-9877, 2020.

Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through l _ 0 regularization. arXiv preprint arXiv:1712.01312 , 2017.

Song Guo, Jiahang Xu, Li Lyna Zhang, and Mao Yang. Compresso: Structured pruning with collaborative prompting learns compact large language models. arXiv preprint arXiv:2310.05015 , 2023.

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022.

Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. In Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ NeurIPS 2023) , 2023.

Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Representations , 2017.

Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11264-11272, 2019.

Mingyang Zhang, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, Bohan Zhuang, et al. Pruning meets low-rank parameter-efficient fine-tuning. arXiv preprint arXiv:2305.18403 , 2023.

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339 , 2022.

Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695 , 2023.

AMissing SECRET SAUCE. Outlier weighed layerwise sparsity (owl): Amissing secret sauce for pruning llms to high sparsity. 2023.

Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems , 2, 1989.

Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks , pages 293-299. IEEE, 1993.

Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate posttraining quantization and pruning. Advances in Neural Information Processing Systems , 35:44754488, 2022.

Eldar Kurti´ c, Elias Frantar, and Dan Alistarh. Ziplm: Inference-aware structured pruning of language models. Advances in Neural Information Processing Systems , 36, 2024.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research , 21(1):5485-5551, 2020.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations , 2016.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 2924-2936, 2019.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 7432-7439, 2020.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4791-4800, 2019.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM , 64(9):99-106, 2021.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 , 2018.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 2381-2391, 2018.

Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version v0. 0.1. Sept , 2021.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023. URL https://lmsys. org/blog/2023-0330-vicuna , 3(5), 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305 , 2023.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2020.

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508 , 2023.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 , 2023.

A Proof of Lemmas

A.1 Proof of Lemma (i)

Lemma. For symmetric matrix M , the inverse matrix after permutation can be obtained by the same permutation of M -1 ;

Proof. The lemma can be easily proven through elementary matrix transformations. Let P be a permutation matrix. We have P T P = I . And since M is symmetric, M = M T . We wish to prove that the inverse of the permuted matrix M ′ = P T MP is ( M ′ ) -1 = P T M -1 P . By the following transformations:

we can demonstrates that ( M ′ ) -1 = P T M -1 P .

A.2 Proof of Lemma (ii)

Lemma. The principal submatrix of symmetric M after Cholesky decomposition is equivalent to the Cholesky decomposition of its principal submatrix.

Proof. Consider a symmetric matrix M . Without loss of generality, let's consider we are removing the last row and column. In block form:

its Cholesky decomposition can be expressed as:

where L A is the Cholesky decomposition of A , and l is a scalar value. Here, A = L A L T A , and this matches the definition of the Cholesky decomposition for the principal submatrix A of M . Thus the statement is demonstrated through the uniqueness of the Cholesky decomposition.

B More Detailed Evaluation Results

Detailed evaluation results of pruned LLaMA-13B/30B. Table 7 details the experimental results for LLaMA-13B/30B. The evaluation results in this table represent a detailed version of Table 2, listing scores for each specific commonsense task to provide a more detailed comparison.

PPL & Commonsense Reasoning evaluations of pruned Vicuna-7B. Table 8 details the experimental results for Vicuna-7B. We observe that, on the Wikitext2 dataset, SlimGPT without finetuning exhibits comparable or higher PPL than LLM-Pruner, a result that diverges from findings in experiments with LLaMA models. The parameter compensation of SlimGPT makes it more dependent on the distribution of the calibration set compared to LLM-Pruner, while Vicuna is a model finetuned on general instructions, and at this point, pretrained data is not the most appropriate calibration set. Using an instruction dataset for pruning might yield better results, which remains to be verified. However, SlimGPT with finetuning still leads on most of the tasks.

PPL & Commonsense Reasoning & MMLU evaluations of pruned LLaMA2-7B. LLaMA2-7B is a new generation model with completely different parameters, exhibiting better overall performance compared to the first generation LLaMA-7B. In addition to the Perplexity and Commonsense Reasoning assessments, we also supplement evaluation on the Massive Multitask Language Understanding (MMLU) task. MMLU is a quiz bank covering 57 subjects, presenting a greater challenge compared to the Commonsense Reasoning datasets. We evaluate using LLaMA2-7B with 20% of its parameters pruned, under 5-shot settings. The evaluation results for PPL and Commonsense Reasoning are shown in Table 9, while the results on the MMLU task are presented in Table 10. In the Commonsense Reasoning task, SlimGPT significantly outperforms the baseline and closely approaches the performance of the original unpruned model. In the MMLU task, although SlimGPT still substantially leads over the baseline, it exhibits a noticeable gap compared to the unpruned model and shows a slight decline after finetuning. For such challenging tasks, full post-training is required to restore performance, rather than relying solely on lightweight LoRA finetuning.

PPL & Commonsense Reasoning & MMLU evaluations of pruned Baichuan-7B. We conduct pruning experiments on the Baichuan-7b model and perform evaluations on the Wikitext2, Commonsense Reasoning datasets, and MMLU datasets. Tables 11 and Table 12 present the evaluation results for commonsense reasoning and MMLU, respectively. Similar to the findings with LLaMA2-7b, under the same LoRA finetuning settings, SlimGPT shows a clear improvement over the baseline.

Long Context Understanding evaluation results. To further explore the impact of SlimGPT on the understanding of long-context texts, we select the Mistral-7B-Instruct-V2.0 model [45] for experiments, which supports up to 32k context windows. We prune it by 20% and conduct an evaluation on the LongBench task. Table 13 presents the evaluation results of the model before and after pruning. Note that we skip the evaluation of the GovReport datasets and thus the average score on Summarization tasks does not include that dataset. Under the LoRA finetuning settings, using SlimGPT with 20% of its parameters pruned can retain 90% of its long-text comprehension capabilities.

C Supplementary Ablation Experiments

C.1 Influence of Calibration Data Category.

As SlimGPT updates the remaining parameters to mitigate the effects of pruning, which is dependent on the calibration data, it underscores the importance of investigating the impact of various calibration dataset categories. We conduct experiments on three general datasets:

C4 subset: A commonly used pre-training corpus, which is the default calibration set for SlimGPT. We sample 512 sentences with 512 tokens from the first 20,000 corpus.

Alpaca dataset: A high quality generic domain dataset used for supervised finetuning, generated by GPT3.5. We randomly sample 512 sentences with 512 tokens.

GPT4-Alpaca dataset: A high quality dataset similar to Alpaca generated by GPT4. We randomly sample 512 sentences with 512 tokens.

We maintain consistency in pruning strategies across all models, differing only in the dataset used. Each model is pruned by 50% . We assess performance directly on these pruned models without any post-training. Table 14 presents the pruning results across various datasets. The three datasets can be categorized into pre-training datasets (C4) and instruction-following datasets (Alpaca, GPT4_Alpaca). Models pruned on C4 exhibit better PPL results on Wikitext2, whereas models pruned on Alpaca series perform better on the Commonsense Reasoning dataset. Different types of datasets have varying impacts on SlimGPT. Instruction-following datasets is more favorable for retaining the model's commonsense knowledge, whereas using pre-training datasets can achieve a balance between language modeling capabilities and commonsense abilities.

D More Analysis

D.1 About Structural Dependency

The structural dependency problem in attention blocks happens when a column of weights in an attention head is removed, elements in other positions in the attention matrix are also affected because of the softmax function. Directly summing the errors across all columns of a head may result in significant numerical inaccuracies, as Equation 4 applies only to single-column pruning instead of multiple columns. To achieve multi-column pruning, we need to iterate using Equation 4 and update H -1 with Equation 3, which makes it difficult to assess the pruning error of a total attention head in advance.

D.2 Layer-wise Pruning Ratio at Pruning Stage

0

8

16

24

32

Layer Index

0

20%

40%

50%

60%

Pruning Ratio

LLM-Pruner

SlimGPT

To more conveniently present the details of the logarithmic increase variation in Incremental Pruning Ratio, we illustrate the layer-wise pruning ratios for SlimGPT's logarithmic increase and LLMPruner's heuristic setting at a 50% pruning rate in Figure 4. SlimGPT starts with a lower initial pruning rate, with a rapid increase in the shallower layers followed by a slower change in the deeper layers, eventually approximating the fixed pruning ratio of LLM-Pruner. Their biggest difference lies in the handling of the last two layers. LLM-Pruner lacks parameter compensation, so the layers pruned at the output end have a larger impact on the final results, whereas SlimGPT reduces their impact on the model through parameter compensation.

D.3 Training Loss at Recovery Stage

0.2

0.4

0.6

0.8

1.0

Epoch

1.0

1.5

2.0

2.5

3.0

Loss

Wikitext2 Eval Loss

Alpaca Train Loss

To figure out whether overfitting has occurred during the finetuning phase, potentially affecting the performance evaluation of the pruned models, we plot the loss curve of the model during the fine-tuning stage, as shown in Figure 5. We train for one epoch on the Alpaca dataset while using Wikitext2 as the evaluation set. The figure illustrates the train loss on Alpaca and the evaluation loss on Wikitext2. As is shown, the training loss is decreasing and converging normally, with no

significant fluctuations in the evaluation loss on Wikitext2, indicating that fine-tuning is conducted on general data without specific optimization for Wikitext2, and there is no occurrence of overfitting.

E Generation Cases from Pruned Model

Table 15 shows the generation cases of the original LLaMA-7B model, the pruned LLaMA-5.4B model, and the pruned and finetuned LLaMA-5.4B model. All inference parameters are kept consistent. To avoid data contamination from the fine-tuning process, we following LLM-Pruner select three input cases. From a qualitative analysis perspective, the model post-pruning by 20% shows little difference from the original LLaMA-7B. After fine-tuning, the model's output tends to offer suggestions more, likely due to the influence of the Alpaca dataset, but it still maintains a high standard in terms of generation quality.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The paper clearly articulate the contributions in the Abstract and in the Introduction section, we highlight the advantages and application scenarios of our method. Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper discusses the limitations of our proposed method in the Conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have provided a proof of a mathematical process concerning a specific detail of the algorithm proposed in our paper, with the proof process included in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We submit our source code as an anonymized zip file, accompanied by a detailed replication guide.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We submit our source code as an anonymized zip file, accompanied by a detailed replication guide.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In the Experimental Settings section of the Experiment chapter, we provide detailed descriptions of the models used, data sampling methods, and training details. Additionally, more comprehensive configuration information is available in the accompanying source code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The paper does not include error bars because of limited computational resources that prevented extensive experimentation.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide all necessary running environments required for the experiments in the Experimental Settings section of the Experiment chapter.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the NeurIPS Code of Ethics and ensure that our paper strictly adheres to these guidelines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: In our conclusion section, we briefly discuss the potential adverse effects of large language models.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks as we do NOT release data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: In the paper, we have duly cited all utilized models, datasets, and algorithms in an appropriate manner.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our paper identifies source code as crucial assets pertinent to our work, and it is provided as part of our submission in the form of an anonymized zip file.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel SlimGPT framework to conduct structured pruning for LLMs in a fast and low-cost way. Specifically, SlimGPT modifies the Optimal Brain Surgeon (OBS) framework, and proposes a Batched Greedy Pruning to  enhance the performance of head-wise pruning through Cholesky decomposition. SlimGPT also improves the FFN pruning efficiency via Dynamic Group Size. Besides, SlimGPT employs an Incremental Pruning Ratio in order to mitigate the error accumulation problem in layer-wise pruning. Experiments on the LLaMA, LLaMA-2, and other popular LLMs demonstrate that SlimGPT achieves a new SOTA on LLM pruning.\n\nSTRENGTHS:\n1. This paper presents a proper way to extend the OBS framework to structured pruning with strong theoretical foundation. The technical details are thorough and convincing.\n2. Extensive experiment results demonstrate that SlimGPT successfully achieves a new SOTA, surpassing all related works in this field.\n\nWEAKNESSES:\n1. SlimGPT employs an Incremental Pruning Ratio strategy. In L220, the article specifies that a logarithmic increasing strategy performs well and is employed in all experiments. Actually, this should be considered more carefully. Pruning can be viewed as a method to get rid of unnecessary information in the activations and only preserve necessary components for later layers. From this perspective, it resembles that of Token Merging [1,2]. It is already demonstrated in [1,2] that the performance loss caused by an aggressive pruning schedule in the first layers  can be mitigated by re-training. Therefore, I suggest the authors test more increasing strategies, instead of the logarithmic strategy, to fully utilize the power of SlimGPT.\n2. Pruning at a $p\\\\%$ sparsity does not usually lead to an $\\frac{1}{1-p\\\\%}$ throughput speedup [2]. To demonstrate that SlimGPT actually helps to deploy LLMs, the authors should carefully compare the **throughput** (eg., tokens per sec.) of pruned models generated by SlimGPT and the competing baselines.\n\n[1] Token Merging: Your ViT But Faster (ICLR 2023)\n\n[2] PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptation (ECCV 2024)\n\nQUESTIONS:\nUnder the same total sparsity value, does different increasing schedules affect the model inference speedup?\n\nLIMITATIONS:\nSee Weaknesses and Questions. I do appreciate the theoretical contributions. So my rating may be adjusted after carefully checking the replies and other reviews.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents SlimGPT, a method for structured pruning of LLMs to balance performance with efficiency. The method is based on the OBS framework and introduces Batched Greedy Pruning to enhance pruning accuracy and speed. The authors also propose the Incremental Pruning Ratio strategy to mitigate performance loss due to error accumulation. Experimental results on LLaMA and other models demonstrate that SlimGPT achieves state-of-the-art results with significant improvements in performance retention compared to existing methods.\n\nSTRENGTHS:\n1. The paper introduces a novel method for structured pruning based on the OBS framework to accelerate large language models.\n2. The method is validated through comprehensive experiments on various models, showing improved performance and efficiency.\n\nWEAKNESSES:\n1. Since Table 9 illustrates the significant impact of the calibration dataset on the pruning performance, I doubt whether the selection of calibration samples is more important than the design of the pruning criterion. The experimental results compared in the paper are with different sampling strategies of the calibration dataset, so it is hard to evaluate the superiority of the proposed method.\n2. The design of the layer-wise sparsity is empirical with no theoretical analysis. Since the pruning within a layer is a greedy pipeline, it is unclear why the layer-wise sparsity design is not in a greedy pipeline.\n3. The inference speed should be provided for comparisons.\n\nQUESTIONS:\n1. In Figure 1., from my view, it seems that the output elements of the third head are all the smallest, so they are all reordered to the first head for pruning. My question is what if not all elements of a head are in the same magnitude order? In this case, how to conduct batched greedy pruning?\n2. Can you provide some insights about the reason why the pruning results with 2048 samples and 2048 sequence length start to decrease in zero-shot average metric?\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a layer-wise pruning approach called SlimGPT that follows the Optimal Brain Surgeon framework but with a batched pruning procedure utilized to make it feasible on large models while remaining structured. The authors claim near-optimal pruning performance on commonsense reasoning datasets and wikitext ppl.\n\nSTRENGTHS:\nModels pruned via structured pruning approaches can naturally gain efficiency benefits, and the proposed method supposedly inherits this important property (though missing some efficiency reports). The task- and relatively architecture-agnostic nature of SlimGPT also makes it score well in terms of adaptability. The experiment reports indicate SlimGPT beats three other SOTA methods by a healthy margin (though its alignment requires some additional polish).\n\nWEAKNESSES:\nThe main weakness of the paper is its eval, both in terms of alignment and coverage.\n\n1. Unaligned evaluation: Many of the compared baselines utilized a different calibration dataset and procedure, but it looks like only the LLM-Pruner is replicated in an aligned setup, while all results for other methods are copied from their original literature. This needs to be controlled, especially with only three methods to compare.\n\n2. Overly emphasis on the Llama family: The presented evaluation is solely conducted on various Llama 1/2 and Vicuna models, which are all Llama family-based. More coverage on other popular LLMs should be included. I'd recommend a healthy selection from Mistral, Phi, Gemma, Qiwen, and something along the line.\n\n3. Only on zero-shot commonsense reasoning tasks: As mentioned around line 243, the real task evaluation of this paper is conducted *\"under a zero-shot setting on the Commonsense Reasoning datasets, which encompass seven diverse subtasks...\"* This is not comprehensive enough. Common intelligence datasets like MMLU and GSM8k in typical few-shot setups should also be reported. Additionally, given the weak generation/long context performance in some recent layer pruning (but not layer-wise structured pruning in finer granularity) work like ShortGPT, I'd like to see SlimGPT evaluated on tasks like LongBench, InfinityBench, Needle-In-A-Haystack retrieval, and HumanEval on models with longer context window (e.g., mistral 32k).\n\n4. Incomplete efficiency report: There are no throughput or latency results, which are key efficiency metrics for conducting structured pruning in the first place and must be reported in efficiency literature, especially because different structured pruning methods can reflect different inference efficiencies, leading to work like ShearedLlaMA proposing targeted structured pruning. Also, the authors claim that *\"low-cost, low-resource, and time-efficient compression scheme\"* as their contribution in line 65, but there is no runtime or memory report on its pruning procedure.\n\n5. Not really a weakness, but the authors should consider giving a more detailed introduction of the compared baselines either in related work or around line 246.\n\nDespite my score of 4, I believe many of the raised concerns are addressable as there are mostly just more evals, and I am open to improving my rating upon proper rebuttal.\n\nQUESTIONS:\n1. Where is the plotting for the unpruned model in Figure 2? What pruning method is applied here? How much performance can fine-tuning recover in this setup?\n\n2. Is a pruned model superior to a smaller pretrained model? Like can a SlimGPT-pruned 13B model with a pruning ratio set around 45% be more performant than a 7/8B model? From the look of Tables 1 & 2, it seems a 50% pruned 13B model is significantly inferior to a 7B, and a 50% pruned 30B model is required to provide similar zero-shot task performance to an unpruned 7B, so this is unlikely. If confirmed, I am afraid this massively discounts the contribution of this work, as one can often just adopt a smaller pretrained model with no pruning or calibration necessary. Though I understand different application scenarios may call for models with different sizes, where the pretrained models can't cover them all.\n\nLIMITATIONS:\nThe authors have provided a limitation and checklist section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}