SlimGPT: Layer-wise Structured Pruning for Large Language Models

Authors: gui  ling
ziyang  wang
yuliang  yan
qingwen  liu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 05:53:13

Abstract: Large language models (LLMs) have garnered significant attention for their remarkable capabilities across various domains, whose vast parameter scales present challenges for practical deployment. Structured pruning is an effective method to balance model performance with efficiency, but performance restoration under computational resource constraints is a principal challenge in pruning LLMs. Therefore, we present a low-cost and fast structured pruning method for LLMs named SlimGPT based on the Optimal Brain Surgeon framework. We propose Batched Greedy Pruning for rapid and near-optimal pruning, which enhances the accuracy of head-wise pruning error estimation through grouped Cholesky decomposition and improves the pruning efficiency of FFN via Dynamic Group Size, thereby achieving approximate local optimal pruning results within one hour. Besides, we explore the limitations of layer-wise pruning from the perspective of error accumulation and propose Incremental Pruning Ratio, a non-uniform pruning strategy to reduce performance degradation. Experimental results on the LLaMA benchmark show that SlimGPT outperforms other methods and achieves state-of-the-art results.

Main paper: 
 introduction: 
 Large Language Models (LLMs) [1, 2, 3] have made significant strides in various natural language processing tasks, leading to the emergence of novel applications such as AI agents [4] . One of the factors contributing to the exceptional capabilities of LLMs is their massive parameter scales. However, these extensive parameters also introduce increased inference costs and deployment challenges, hindering the widespread application and adoption of LLMs. Accelerating inference for LLMs has become a focal point of current research. Model compression [5] , as one of the strategies for inference acceleration, including techniques like pruning and quantization [6, 7] , has been extensively researched. Nevertheless, earlier model compression techniques, particularly model pruning, typically rely on heavy post-training to recover the model's capabilities, which typically involves retraining with the entire training dataset. Given the constraints of current computational resources, the above approaches are not feasible for LLMs.In the domain of LLM pruning, recent studies have largely focused on unstructured (or semistructured) pruning [8] , a method that shrinks models by selectively zeroing out weights considered non-critical. Despite its advancements, unstructured pruning falls short in substantially reducing parameter count, which is crucial for accelerating LLM inference as it is often bottlenecked on memory bandwidth and communication [9] . To accelerate inference speed, unstructured pruning models are often paired with specialized frameworks or hardware solutions. Conversely, structured pruning [10, 11] effectively decreases the model's parameter count by systematically eliminating columns or rows from weight matrices, enabling significant improvements in inference speed, and the product of a weight's L1 norm and the L2 norm of eigenvalues to gauge its importance, achieving impressive pruning results. OWL [25] determines layer-wise sparsity ratios based on Layerwise Outlier Distribution (LOD), obtaining substantial performance gains at high sparsity levels.Layer-wise compression methods. The early works [26, 27] provide a layer-wise compression framework with a locally optimal solution named Optimal Brain Surgeon (OBS). And then OBC [28] reduces the computational burden by converting layer-wise pruning into row-wise pruning and updating the inverse Hessian using a proposed formula. Furthermore, GPTQ [13] accelerates the process with Lazy Batch-Updates and Cholesky Reformulation, enabling the application of this method to the quantization of LLMs. SparseGPT [12] also adapts this approach for unstructured pruning of LLMs. However, there appears to be no existing research that has implemented OBS in structured pruning for LLMs.Structured Pruning vs. Other Techniques. Given that OBS has previously been used in both quantization and unstructured pruning, and is now being applied to structured pruning, there is an inherent consistency across these three compression schemes. These methods actually compress the model at varying levels of granularity. Quantization, which "trims" floating-point precision, represents the finest granularity and delivers excellent compression outcomes. Structured pruning, on the other hand, involves trimming weight vectors and represents the coarsest granularity, naturally resulting in higher performance losses compared to other methods, which poses significant challenges. For small models, it is possible to recover most of the performance with post-training, but this is challenging to achieve in LLMs due to resource constraints. Nonetheless, structured pruning effectively reduces the number of parameters without needing special inference framework support and is compatible with the other two methods, thus still holding considerable potential for application.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The paper clearly articulate the contributions in the Abstract and in the Introduction section, we highlight the advantages and application scenarios of our method.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The paper discusses the limitations of our proposed method in the Conclusion section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminary: 
 Layer-Wise Pruning. Consider the scenario of pruning on a well-optimized model, known as post-training pruning, a prevalent approach involves decomposing the global model pruning challenge into layer-wise subproblems (i.e., Layer-wise pruning), which are typically modeled as issues of minimizing L2 error. Specifically, let W l represent the weights of the l-th layer of a pretrained model and X l be the input features for layer l. The goal is to determine pruned weights Ŵl that achieve a predefined pruning ratio while minimizing the squared error: argmin Ŵl ∥W l X l -Ŵl X l ∥ 2 2 .(EQUATIONOptimal Brain Surgeon (OBS) Framework. As Equation 1 can be rewritten as the sum of square error of each row of the weights to be pruned, the layer-wise pruning can be further split into row-wise pruning [28] . Consider the removal of a single weight from a row in W l , Equation 1 has a closed-form solution [27] . Let w denote a specific weight in a row of W l , and let p be its corresponding index. Given that our optimization objective is to minimize row-wise squared error, the Hessian of this objective with respect to the weight row of layer l is given by H l = 2X l X T l . The weight to be pruned, w p , as well as the necessary update δ p applied to the remaining weights of the same row to counterbalance the removal, can be determined through the following calculation:EQUATIONwhere H -1 p,p denotes the p th diagonal entry of the inverse Hessian, and H -1 :,p is its p th column. By iteratively using Equation 2 to remove one weight and update the remaining weights in the same row, one can obtain a locally optimal compressed model. After each iteration, H will be updated by removing the p row and column, which is represented by H [-p] , here we use [-p] to indicate the removal of p row and column of the matrix. As H -1 cannot be updated by simple removal as (H [-p] ) -1 ̸ = (H -1 ) [-p] , to avoid the expensive full recomputations of H -1 , the following formula is proposed to quickly update H -1 [28] :EQUATIONThis framework can be practically applied to medium-sized models. However, for models with billions of weights, the iterative pruning becomes exceedingly time-consuming.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: We have read the NeurIPS Code of Ethics and ensure that our paper strictly adheres to these guidelines.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
methodology: 
 In this section, by extending the OBS framework to structured pruning, we introduce SlimGPT from two aspects: (1) By employing Batched Greedy Pruning to reduce error computation, we minimize the performance degradation caused by pruning while also accelerating the pruning speed;(2) By analyzing the limitation of layer-wise pruning from the perspective of error accumulation, we introduce Incremental Pruning Ratio, a non-uniform pruning strategy.As mentioned above, the pruning between different rows is independent, making it possible to prune all rows simultaneously. We extend the OBS framework to structured column pruning, i.e., pruning one column at a time and compensating the rest columns using the following formula:EQUATIONwhere H -1 p,: denotes the p-th row of H -1 , and the obtained ∆ is a compensation matrix of the same size as W . We following previous works employ attention blocks and FFNs as the smallest units for pruning. By pruning the columns of the output matrix in attention blocks and the dimensionality reduction matrix in FFN blocks, we reduce the number of attention heads and FFN channels, thereby decreasing the model's parameter count. However, the above formula cannot be applied directly, as iteratively finding and pruning the column with the minimum error is time-consuming. More critically, the structural dependency in attention blocks imposes additional constraints on column pruning, making it impossible to evaluate the importance of a head based solely on information from a single column. Given that the calculation of the pruning error requires only the diagonal elements of H -1 (see Equation 4), which are updated after each iteration, computing these elements in advance allows for calculating the head-wise error. With the observation that the sequential row removal via Equation 3 for the symmetric H -1 essentially corresponds to taking a Cholesky decomposition [13] , we can obtain the elements in advance with Cholesky decomposition.… … 0.9 … … 0 0.8 0 0 … … 0.6 … … 0 0.4 0 0 … … 0.3 … … 0 0.2 0 0 … … 0.9 … … 0 0.8 0 0 Column Reorder 𝑾 𝑯 !𝟏 0.7 … 0.4 1.1 … 0.5 0.2 … 0.3 … … … … … … … … … 0.4 … 0.5 0.9 … 0.8 0.1 … 0.2 … … 0.72 … 0.42 1.08 … 0.51 0.21 … 0 … … … … … … … … 0 0.Hoewever, the matrix obtained by Cholesky decomposition is triangular, and the elements of the current row (column) are calculated based on the elements of all the previous rows (columns), which means the Cholesky decomposition breaks the comparability between rows (columns). So it is hard to obtain all the required information in advance through the Cholesky decomposition like [12, 13] , whose error comparison is usually within the same column but structured pruning requires the comparison of different columns.Since structured pruning only requires traversing the columns that need to be removed, by rearranging the rows and columns corresponding to a head that is to be pruned in H to the front, and then invert the matrix followed by Cholesky decomposition, we can calculate the E ← 0 drow ×d for i in 0, 1, 2..d do E:,i:i+1 ← W:,i:i+1/ Ĥ-1 i,i // pruning error W :,i:d ← W :,i:d -E:,i:i+1 × Ĥ-1 i,i:d // local update, column i is zeroed end for W :,d: ← W :,d: -E × Ĥ-1 :,d: // global update W ← W[:, Argsort(A)] // restorehead error column-wise. However, repeated rearrangement followed by matrix inversion and Cholesky decomposition is highly time-consuming, and this is just to find one head to be pruned. We accelerate the above process through two common lemmas (proofs are provided in the Appendix): (i) For symmetric H, the inverse matrix after permutation can be obtained by the same permutation of H -1 ; (ii) The principal submatrix of symmetric H -1 after Cholesky decomposition is equivalent to the Cholesky decomposition of its principal submatrix. Thus we can calculate the pruning error of all the heads at once through grouped Cholesky decomposition. Specifically, we inverse H once and split it into n head matrices along the main diagonal, with each remains definite and symmetric, and decompose them in parallel: Ĥ-1 =where decomposed Ĥ-1 is a matrix of size n head × d head × d head , n head and d head represent the head number and head dimension, respectively. Utilizing GPU acceleration, we can quickly calculate the value of the diagonal element in advance and calculate the head-wise error. Note that during error computation, we only update the diagonal elements of H -1 and skip the update of W , which is small and does not dominate the ordering of errors.After determining the head to be pruned, we rearrange the corresponding columns of W and the corresponding rows and columns of H -1 to the front, and again use the global Cholesky decomposition on reordered H -1 to prune the head column by column until the first head is pruned. In this way, we can avoid traversing columns that do not need pruning and only traverse necessary columns to improve pruning efficiency further. Figure 1 shows the process of Batched Greedy Pruning applied to attention blocks, and Algorithm 1 is a pseudocode illustrating how to prune a head with two steps: calculating head-wise error and pruning a head column-wise.For FFNs, since there is no block constraint similar to attention heads, we can achieve local numerical optimality by pruning columns individually using Equation 4. However, the column-wise pruning is time-consuming because of the substantial intermediate dimensions of FFN. We thus prune a group of columns at a time and select the top-k columns with the most minor errors for pruning at each iteration. Considering that the compensation at each iteration may lead to a local reshuffling of column errors, we adopt a dynamic grouping strategy for pruning FFN blocks. We start with larger group size such as 1024 for pruning and gradually decrease the group size to a small number like 8, which allows us to enhance pruning efficiency while approaching an approximate optimal solution.Through Batched Greedy Pruning, we can obtain near-optimal structured pruning results for each layer. However, finding a suitable pruning ratio for each layer is difficult, as considering global information is quite challenging for layer-wise pruning, which only provides optimal pruning results for the current layer. Maintaining a uniform pruning ratio across all layers is unreasonable and will impact model performance, especially when the pruning ratio is high. Existing works have different approaches to the problem. For example, LLM-Pruner [11] avoids pruning in the initial and final layers while maintaining a consistent ratio in the intermediate layers to manually implement non-uniform pruning. OWL [25] adjusts sparse ratios dynamically for each layer based on the proportion of feature outliers, which is applied to unstructured pruning. In Incremental Pruning Ratio, without loss of generality, we employ a logarithmically increasing strategy to control the layer-wise pruning ratio. Specifically, for an n-layer model with the first and last layer pruning ratios denoted as r 0 and r n-1 respectively, the pruning ratio for the i-th layer is defined as follows:EQUATIONwhere r i represents the pruning ratio for the i-th layer. This formula ensures that the pruning ratio from the first layer to the last layer transitions smoothly as a logarithmic curve. The strategy mitigates the pruning error accumulation in shallow layers while avoiding the issue of excessive pruning in the deeper layers, allowing for further reduction in performance loss. 
experimental settings: 
 Implementation details. We use C4 dataset [29] as the calibration set. From the first shard of C4, we randomly select 256 2048-token sequences for pruning. To restore performance, we following LLM-Pruner [11] finetune the pruned model with LORA [18] . We tune with Alpaca datsets [30] for one epoch and utilize the AdamW optimizer with an initial learning rate set to 1e-4, coupled with a cosine annealing schedule for the learning rate. The global batch size is set to 64 and the sequence length is truncated to 256. All pruning experiments are conducted on a single A100, while finetuning is performed using two A100s.Models and Metrics. To assess the effectiveness and generality of SlimGPT, We carry out a series of experiments on the LLaMA families [2] . And to measure the effectiveness of our pruned models in the task-agnostic setting, we follow previous pruning works to evaluate language modeling performance and commonsense reasoning capabilities. The language modeling performance is evaluated on the WikiText2 [31] validation set with sequence length truncated to 128, and the commonsense reasoning capabilities is carried out under a zero-shot setting on the Commonsense Reasoning datasets, which encompass seven diverse subtasks: BoolQ [32] , PIQA [33] , HellaSwag [34] , WinoGrande [35] , ARCeasy [36] , ARC-challenge [36] , and OpenbookQA [37] . We utilize the lm-eval-harness framework [38] to conduct these evaluations. To validate the universality of SlimGPT, we conduct experiments on additional models and supplementary evaluation datasets. The results of these experiments can be found in the Appendix. We conduct further pruning experiments on Vicuna [39] , LLaMA2 [40] , and Baichuan [41] , which yield results consistent with those observed using the LLaMA model. In addition, we engage in preliminary evaluations on more complex tasks, specifically MMLU [42] and LongBench [43] . Although SlimGPT exhibits slightly larger performance losses on these datasets, it still retains a significant advantage over the baseline models.Baselines. We compare SlimGPT with the following recent SOTA works on structured pruning, which we could find during our experiments:• LLM-Pruner [11] , a gradient-based pruning approach, serves as our benchmark. This method involves a two-step process: a one-shot pruning followed by performance restoration through LORA fine-tuning.• Compresso [17] is a pruning method based on sparse training, applying L0 penalty to manually inserted masks during the LORA fine-tuning phase and employing a cubic sparsity schedule to iteratively prune the model until the desired pruning ratio is achieved.• LoRAPrune [22] utilizes gradients from the LORA module's parameters to determine the importance of the original model's parameters, thus requiring only gradient information from the LORA module, which significantly reduces computational demands.To facilitate a more effective comparison of the evaluated results with prior works, we prune the LLaMA-7B model using four distinct pruning ratios-20%, 25%, 33%, and 50%-resulting in four smaller models with parameter counts of 5.4B, 5B, 4.5B, and 3.4B, respectively. Table 1 shows the detailed perplexity and zero-shot performance of pruned LLaMA-7B with four different sizes.Compared to other approaches, SlimGPT demonstrates superior performance in language modeling and commonsense reasoning across most subtasks. Under a pruning condition of 20%, SlimGPT achieves a slightly better perplexity score than the best existing results (16.68 vs. 16.80) and shows Moreover, we observe that although SlimGPT affects different subtasks to varying degrees, its impact is relatively balanced across different tasks, eliminating the occurrence of disproportionately large losses in particular tasks. At lower pruning ratios, some tasks such as BoolQ can even outperform the original unpruned model. Additionally, the effects of fine-tuning also differ among tasks, significantly improving tasks like HellaSwag and ARC-easy, while potentially causing negative side effects for tasks such as BoolQ and WinoGrande. This phenomenon is likely closely associated with the datasets used for fine-tuning.For larger-scale models such as LLaMA-13B and LLaMA-30B, previous works have not provided pruning results for these models. Therefore, we solely compare our results to the LLM-Pruner baseline, concentrating on two specific pruning settings: a lower pruning ratio (20%) and a higher pruning ratio (50%). The replication of LLM-Pruner is consistent with the method described in the paper, where the pruned models by LLM-Pruner are finetuned with LORA.Table 2 presents the pruning results of LLaMA-13B and LLaMA-30B, and we can draw similar conclusions: SlimGPT outperforms LLM-Pruner in terms of both PPL and zero-shot average scores even without post-training. Note that as the scale of the model increases, the performance loss due to pruning becomes smaller, suggesting a higher degree of parameter redundancy in larger models. At a low pruning ratio of 20%, the LLaMA-13B model's average performance in commonsense reasoning is nearly on par with that of the original, unpruned model (68.06 vs. 68.16). Similarly, the pruned LLaMA-30B model slightly outperforms the unpruned version (72.56 vs. 71.92). For the perplexity task, even though SlimGPT exhibits gaps compared to the original model, it still performs better than baseline, even at low pruning ratios.Besides, we can find that the performance of LLaMA-13B pruned by 50% falls short compared to LLaMA-7B pruned by 20%. This highlights the limitations of low-cost fine-tuning, where resource constraints and training with techniques like LoRA result in limited performance recovery for the model. Therefore, using lower pruning ratios to compress smaller LLMs yields better returns. The pruning runtime and memory usage for LLaMA-7B and LLaMA-13B are detailed in Table 3 .Memory usage fluctuates based on the model size and the calibration scale, while the pruning speed is additionally affected by the pruning ratio. We demonstrate the pruning efficiency results derived from our experimental setup. Utilizing SlimGPT, which operates on a layer-wise basis, there is no need to load the entire model at once. Instead, we only load the parameters of the current layer along with the corresponding input features, significantly reducing memory consumption. For instance, to prune the 7B model by 20%, approximately 7 GB of GPU memory and 18 minutes are required to complete the process. Similarly, pruning the 13B model by 50% necessitates around 12 GB of GPU memory and 41 minutes to finalize.Table 4 illustrates the inference latency and memory usage of the pruned LLaMA-7b models. We prune LLaMA-7b by 20% and 50% respectively. The maximum output limit is set to 512 and the presented values are the average derived from 50 inference trials. When pruning 50% of the parameters, the memory usage of the model during inference decreases to approximately 51% (14297MB vs. 27737MB), and the inference latency is reduced to about 69% (9.21ms vs. 13.51ms).We systematically analyze the influence of several key parts of SlimGPT on the pruning effect, including the Batched Greedy Pruning and Incremental Pruning Ratio strategy. Within the calibration dataset, we conduct thorough experiments with sample sizes and sequence lengths. Unless specifically stated otherwise, all the following experiments are conducted under the condition of pruning 50% of LLaMA-7b without further post-training, to eliminate potential confounding effects. Supplementary ablation experiments can be found in the Appendix.We leverage grouped Cholesky decomposition to enhance the accuracy of head-wise error computation in attention blocks. Similarly, for FFNs, our proposed Dynamic Group Size substantially increases pruning efficiency while preserving near-optimal pruning results. To validate the effectiveness of these two strategies, we start with the complete SlimGPT algorithm and first remove the Dynamic Group Size (denoted as '-DGS'), setting the group size for FFN pruning to a fixed value of 128. Then, we remove the grouped Cholesky decomposition (denoted as '-GCD') and use the initial H -1 to calculate head-wise errors. The experimental results are shown in Zero-shot Avg.Zero-shot Avg. The Incremental Pruning Ratio is a strategy specifically proposed for addressing the issue of layer-wise pruning. To maintain generality, we selected various common non-uniform strategies for comparative experiments, including logarithmic and linear increase strategies, as well as their corresponding decrease strategies. Among these, the logarithmic increase strategy is the default configuration for SlimGPT. Additionally, we conduct experiments under the setting of uniform pruning. Table 6 details the results under the different settings. From an overall perspective, the increase strategy for the pruning ratio has a clear advantage over uniform, and likewise, uniform shows a distinct advantage over decrease. Such results further verify the phenomenon of layer-wize error accumulation. As for the increase strategies of logarithmic and linear changes, due to disparities in model sizes, their results are not entirely comparable. The former performs best in language modeling (38.83), while the latter shows better performance in common sense reasoning tasks (53.45).We delve further into the impact of calibration samples and sequence length, and we choose C4 dataset for our experiments as it has a longer average sequence length. In exploring the effects of the sample scale, we fix the sequence length at 256 and test five scales ranging from 128 to 2048; similarly, when investigating the impact of sequence length, the sample scale is set to 256, with choices of sequence length varying from 64 to 2048. Figure 3 presents the perplexity result and zero-shot performance with different calibration samples and sequence lengths. As the number of samples increases, the PPL and zero-shot averages show a positive overall trend. Furthermore, after the sample count reaches 2048, the PPL does not bottom out, and there is room for further reduction. Similar phenomena can be observed in experiments on sequence length. With more sufficiently high-quality datasets with longer sequences, we believe SlimGPT can achieve better pruning effects. 
conclusion: 
 In this work, we introduce a fast, structured pruning method for large-scale models within resourceconstrained scenarios, based on the OBS framework, termed SlimGPT. Leveraging the novel Batched Greedy Pruning, we enhance the accuracy of pruning error estimation, thereby minimizing performance degradation from pruning. Moreover, we analyze the limitations of layer-wise pruning from the perspective of error accumulation and propose a non-uniform strategy named Incremental Pruning Ratio, which effectively improves the pruned model's performance. Evidence from open-source experiments affirms the efficacy of our approach.Limitations. Even though SlimGPT achieves SOTA results in the structured pruning of LLMs, the model performance degradation at high pruning ratios (e.g., 50%) or on more complex tasks (e.g., LongBench) is still significant. How to enhance the model compression effectiveness under lowresource conditions remains a challenge. Moreover, we utilized a naive logarithmic change strategy in the Incremental Pruning Ratio, which, while ensuring generality, is not the optimal solution. The most suitable non-uniform approach requires further exploration. Lastly, similar to many large-scale open-source models available today, the model obtained through pruning by SlimGPT poses risks in terms of ethical safety and requires cautious handling. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: In our conclusion section, we briefly discuss the potential adverse effects of large language models.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks as we do NOT release data or models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: In the paper, we have duly cited all utilized models, datasets, and algorithms in an appropriate manner.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: Our paper identifies source code as crucial assets pertinent to our work, and it is provided as part of our submission in the form of an anonymized zip file.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Proof of Lemma (i)Lemma. For symmetric matrix M , the inverse matrix after permutation can be obtained by the same permutation of M -1 ;Proof. The lemma can be easily proven through elementary matrix transformations. Let P be a permutation matrix. We have P T P = I. And since M is symmetric, M = M T . We wish to prove that the inverse of the permuted matrix M ′ = P T M P is (M ′ ) -1 = P T M -1 P . By the following transformations:M ′ (P T M -1 P ) = (P T M P )(P T M -1 P ) = P T (M (P P T ))M -1 P = P T M IH -1 P = I (7)we can demonstrates that (M ′ ) -1 = P T M -1 P .Lemma. The principal submatrix of symmetric M after Cholesky decomposition is equivalent to the Cholesky decomposition of its principal submatrix.Proof. Consider a symmetric matrix M . Without loss of generality, let's consider we are removing the last row and column. In block form:EQUATIONits Cholesky decomposition can be expressed as:EQUATIONwhere L A is the Cholesky decomposition of A, and l is a scalar value. Here, A = L A L T A , and this matches the definition of the Cholesky decomposition for the principal submatrix A of M . Thus the statement is demonstrated through the uniqueness of the Cholesky decomposition. Reasoning assessments, we also supplement evaluation on the Massive Multitask Language Understanding (MMLU) task. MMLU is a quiz bank covering 57 subjects, presenting a greater challenge compared to the Commonsense Reasoning datasets. We evaluate using LLaMA2-7B with 20% of its parameters pruned, under 5-shot settings. The evaluation results for PPL and Commonsense Reasoning are shown in Table 9 , while the results on the MMLU task are presented in Table 10 .In the Commonsense Reasoning task, SlimGPT significantly outperforms the baseline and closely approaches the performance of the original unpruned model. In the MMLU task, although SlimGPT still substantially leads over the baseline, it exhibits a noticeable gap compared to the unpruned model and shows a slight decline after finetuning. For such challenging tasks, full post-training is required to restore performance, rather than relying solely on lightweight LoRA finetuning.Reasoning & MMLU evaluations of pruned Baichuan-7B. We conduct pruning experiments on the Baichuan-7b model and perform evaluations on the Wikitext2, Commonsense Reasoning datasets, and MMLU datasets. Tables 11 and Table 12 present the evaluation results for commonsense reasoning and MMLU, respectively. Similar to the findings with LLaMA2-7b, under the same LoRA finetuning settings, SlimGPT shows a clear improvement over the baseline. Long Context Understanding evaluation results. To further explore the impact of SlimGPT on the understanding of long-context texts, we select the Mistral-7B-Instruct-V2.0 model [44] for experiments, which supports up to 32k context windows. We prune it by 20% and conduct an evaluation on the LongBench task. Table 13 presents the evaluation results of the model before and after pruning. Note that we skip the evaluation of the GovReport datasets and thus the average score on Summarization tasks does not include that dataset. Under the LoRA finetuning settings, using SlimGPT with 20% of its parameters pruned can retain 90% of its long-text comprehension capabilities.C.1 Influence of Calibration Data Category.As SlimGPT updates the remaining parameters to mitigate the effects of pruning, which is dependent on the calibration data, it underscores the importance of investigating the impact of various calibration dataset categories. We conduct experiments on three general datasets:• C4 subset: A commonly used pre-training corpus, which is the default calibration set for SlimGPT. We sample 512 sentences with 512 tokens from the first 20,000 corpus.• Alpaca dataset: A high quality generic domain dataset used for supervised finetuning, generated by GPT3.5. We randomly sample 512 sentences with 512 tokens.• GPT4-Alpaca dataset: A high quality dataset similar to Alpaca generated by GPT4. We randomly sample 512 sentences with 512 tokens.We maintain consistency in pruning strategies across all models, differing only in the dataset used. Each model is pruned by 50% . We assess performance directly on these pruned models without any post-training. Table 14 presents the pruning results across various datasets. The three datasets can be categorized into pre-training datasets (C4) and instruction-following datasets (Alpaca, GPT4_Alpaca). Models pruned on C4 exhibit better PPL results on Wikitext2, whereas models pruned on Alpaca series perform better on the Commonsense Reasoning dataset. Different types of datasets have varying impacts on SlimGPT. Instruction-following datasets is more favorable for retaining the model's commonsense knowledge, whereas using pre-training datasets can achieve a balance between language modeling capabilities and commonsense abilities. The structural dependency problem in attention blocks happens when a column of weights in an attention head is removed, elements in other positions in the attention matrix are also affected because of the sof tmax function. Directly summing the errors across all columns of a head may result in significant numerical inaccuracies, as Equation 4applies only to single-column pruning instead of multiple columns. To achieve multi-column pruning, we need to iterate using Equation 4and update H -1 with Equation 3, which makes it difficult to assess the pruning error of a total attention head in advance. To more conveniently present the details of the logarithmic increase variation in Incremental Pruning Ratio, we illustrate the layer-wise pruning ratios for SlimGPT's logarithmic increase and LLM-Pruner's heuristic setting at a 50% pruning rate in Figure 4 . SlimGPT starts with a lower initial pruning rate, with a rapid increase in the shallower layers followed by a slower change in the deeper layers, eventually approximating the fixed pruning ratio of LLM-Pruner. Their biggest difference lies in the handling of the last two layers. LLM-Pruner lacks parameter compensation, so the layers pruned at the output end have a larger impact on the final results, whereas SlimGPT reduces their impact on the model through parameter compensation. AI can create a logo in seconds, but it's not ready for the real world. By Dyllan Furness @dyllonline -Posted on July 12, 2017 -9:35AM 07.12.17 -9:35AM. A new artificial intelligence system has been developed that can generate logos from scratch with just one click of a mouse button. The technology is impressive and could have some practical applications down the line, but there are still many hurdles to overcome before we see this kind of software being used by designers or marketers. ... Neural Sequential Model, especially transformers, have been the most successful model in recent years. Their success is due to their ability to learn long-range dependencies and capture complex interactions between words. However, they are still far from perfect as they struggle with some of the challenges such as low performance on out-of-domain data or noisy labels.In this talk we will discuss how these models can be improved by using a combination of pretraining techniques ... What's great about the holiday season, is that it gives us a chance to reflect on what we are thankful for. I am so grateful and blessed with my family, friends, clients, colleagues, mentors, teachers, students. . . and all of you! I have been given an amazing opportunity in life to do something I love every day -help people find their voice through music. It has truly changed me as a person and helped me grow into who I want to be. Thank you for being part of this journey with me.LLaMA-5.4b w/o tune AI can create a logo in seconds, but it's not the same as having an expert design one for you.AI is great at creating logos that are visually appealing and look good on paper or screen. But if your company needs to have a unique brand identity, then there's no substitute for working with a professional designer who understands what makes a strong visual statement. Neural Sequential Model, especially transformers, have been widely used in many tasks. However, the performance of these models is still limited by their lack of generalization ability and high dependence on data. In this paper we propose a novel method to improve the generalizability of neural sequential model: using an auxiliary task as a pre-training step for the main task. We use a simple image classification problem with 10 classes (cars) as our auxiliary task. ... What's great about the holiday season, is that it gives us a chance to spend time with our family and friends. It also allows for some much needed downtime from work or school. But what happens when you have no one to celebrate with? Or maybe your loved ones are far away and can't make it home this year? Well, don't worry! There are plenty of ways to enjoy the holidays even if you aren't surrounded by people who love you as much as you do them. Here are 5 tips on how to survive being alone during Christmas: ...AI can create a logo in seconds, but it's not the same as having an experienced designer. AI is great for creating logos and other visual elements that are simple or repetitive, like icons or illustrations. But when you need something more complex-like a custom design with unique typography, color schemes, and layout -human creativity comes into play. That said, there are some tools out there that make it easier to work with AI-generated designs: Adobe has released its own tool called Creative Intelligence (ACI), ... Neural Sequential Model, especially transformers, have been widely used in natural language processing (NLP) tasks. However, the performance of these models is still limited by their lack of robustness to noisy data and out-of-distribution samples. In this paper we propose a novel approach for improving the generalization ability of neural sequential model through incorporating adversarial training into the pretraining process. We show that our method can significantly improve the performance on various NLP tasks ... What's great about the holiday season, is that it gives us a chance to spend time with our loved ones. But what if you don't have any family or friends around? Don't worry! There are plenty of ways for you to enjoy your own little Christmas celebration and make this year even more memorable than ever before. Here are some ideas: 1) Have an intimate dinner party at home -Invite close friends over for a cozy evening in where everyone can share their favorite dishes and stories from the past year. ... 2) ...To figure out whether overfitting has occurred during the finetuning phase, potentially affecting the performance evaluation of the pruned models, we plot the loss curve of the model during the fine-tuning stage, as shown in Figure 5 . We train for one epoch on the Alpaca dataset while using Wikitext2 as the evaluation set. The figure illustrates the train loss on Alpaca and the evaluation loss on Wikitext2. As is shown, the training loss is decreasing and converging normally, with no significant fluctuations in the evaluation loss on Wikitext2, indicating that fine-tuning is conducted on general data without specific optimization for Wikitext2, and there is no occurrence of overfitting.Table 15 shows the generation cases of the original LLaMA-7B model, the pruned LLaMA-5.4B model, and the pruned and finetuned LLaMA-5.4B model. All inference parameters are kept consistent. To avoid data contamination from the fine-tuning process, we following LLM-Pruner select three input cases. From a qualitative analysis perspective, the model post-pruning by 20% shows little difference from the original LLaMA-7B. After fine-tuning, the model's output tends to offer suggestions more, likely due to the influence of the Alpaca dataset, but it still maintains a high standard in terms of generation quality.NeurIPS Paper Checklist 


Decision: The authors introduced a low-cost and fast structured pruning method named SlimGPT. Experiments results show that SlimGPT outperforms other methods and achieves state-of-the-art performances.

We appreciate the responses from authors and the discussions with reviewers. Please add all the mentioned experiments and analysis into the final version.

Reviews: review: paper_summary: paper_summary: The authors proposed a layer-wise pruning approach called SlimGPT that follows the Optimal Brain Surgeon framework but with a batched pruning procedure utilized to make it feasible on large models while remaining structured. The authors claim near-optimal pruning performance on commonsense reasoning datasets and wikitext ppl.
strength_weakness: strength_weakness: The main weakness of the paper is its eval, both in terms of alignment and coverage.

1. Unaligned evaluation: Many of the compared baselines utilized a different calibration dataset and procedure, but it looks like only the LLM-Pruner is replicated in an aligned setup, while all results for other methods are copied from their original literature. This needs to be controlled, especially with only three methods to compare.

2. Overly emphasis on the Llama family: The presented evaluation is solely conducted on various Llama 1/2 and Vicuna models, which are all Llama family-based. More coverage on other popular LLMs should be included. I'd recommend a healthy selection from Mistral, Phi, Gemma, Qiwen, and something along the line.

3. Only on zero-shot commonsense reasoning tasks: As mentioned around line 243, the real task evaluation of this paper is conducted *"under a zero-shot setting on the Commonsense Reasoning datasets, which encompass seven diverse subtasks..."* This is not comprehensive enough. Common intelligence datasets like MMLU and GSM8k in typical few-shot setups should also be reported. Additionally, given the weak generation/long context performance in some recent layer pruning (but not layer-wise structured pruning in finer granularity) work like ShortGPT, I'd like to see SlimGPT evaluated on tasks like LongBench, InfinityBench, Needle-In-A-Haystack retrieval, and HumanEval on models with longer context window (e.g., mistral 32k).

4. Incomplete efficiency report: There are no throughput or latency results, which are key efficiency metrics for conducting structured pruning in the first place and must be reported in efficiency literature, especially because different structured pruning methods can reflect different inference efficiencies, leading to work like ShearedLlaMA proposing targeted structured pruning. Also, the authors claim that *"low-cost, low-resource, and time-efficient compression scheme"* as their contribution in line 65, but there is no runtime or memory report on its pruning procedure.

5. Not really a weakness, but the authors should consider giving a more detailed introduction of the compared baselines either in related work or around line 246.

Despite my score of 4, I believe many of the raised concerns are addressable as there are mostly just more evals, and I am open to improving my rating upon proper rebuttal.
questions: questions: 1. Where is the plotting for the unpruned model in Figure 2? What pruning method is applied here? How much performance can fine-tuning recover in this setup?

2. Is a pruned model superior to a smaller pretrained model? Like can a SlimGPT-pruned 13B model with a pruning ratio set around 45% be more performant than a 7/8B model? From the look of Tables 1 & 2, it seems a 50% pruned 13B model is significantly inferior to a 7B, and a 50% pruned 30B model is required to provide similar zero-shot task performance to an unpruned 7B, so this is unlikely. If confirmed, I am afraid this massively discounts the contribution of this work, as one can often just adopt a smaller pretrained model with no pruning or calibration necessary. Though I understand different application scenarios may call for models with different sizes, where the pretrained models can't cover them all.
limitations: limitations: The authors have provided a limitation and checklist section.
paper_summary: paper_summary: This paper presents SlimGPT, a method for structured pruning of LLMs to balance performance with efficiency. The method is based on the OBS framework and introduces Batched Greedy Pruning to enhance pruning accuracy and speed. The authors also propose the Incremental Pruning Ratio strategy to mitigate performance loss due to error accumulation. Experimental results on LLaMA and other models demonstrate that SlimGPT achieves state-of-the-art results with significant improvements in performance retention compared to existing methods.
strength_weakness: strength_weakness: 1. Since Table 9 illustrates the significant impact of the calibration dataset on the pruning performance, I doubt whether the selection of calibration samples is more important than the design of the pruning criterion. The experimental results compared in the paper are with different sampling strategies of the calibration dataset, so it is hard to evaluate the superiority of the proposed method.
2. The design of the layer-wise sparsity is empirical with no theoretical analysis. Since the pruning within a layer is a greedy pipeline, it is unclear why the layer-wise sparsity design is not in a greedy pipeline.
3. The inference speed should be provided for comparisons.
questions: questions: 1. In Figure 1., from my view, it seems that the output elements of the third head are all the smallest, so they are all reordered to the first head for pruning. My question is what if not all elements of a head are in the same magnitude order? In this case, how to conduct batched greedy pruning?
2. Can you provide some insights about the reason why the pruning results with 2048 samples and 2048 sequence length start to decrease in zero-shot average metric?
limitations: limitations: Not applicable.
paper_summary: paper_summary: This paper presents a novel SlimGPT framework to conduct structured pruning for LLMs in a fast and low-cost way. Specifically, SlimGPT modifies the Optimal Brain Surgeon (OBS) framework, and proposes a Batched Greedy Pruning to  enhance the performance of head-wise pruning through Cholesky decomposition. SlimGPT also improves the FFN pruning efficiency via Dynamic Group Size. Besides, SlimGPT employs an Incremental Pruning Ratio in order to mitigate the error accumulation problem in layer-wise pruning. Experiments on the LLaMA, LLaMA-2, and other popular LLMs demonstrate that SlimGPT achieves a new SOTA on LLM pruning.
strength_weakness: strength_weakness: 1. SlimGPT employs an Incremental Pruning Ratio strategy. In L220, the article specifies that a logarithmic increasing strategy performs well and is employed in all experiments. Actually, this should be considered more carefully. Pruning can be viewed as a method to get rid of unnecessary information in the activations and only preserve necessary components for later layers. From this perspective, it resembles that of Token Merging [1,2]. It is already demonstrated in [1,2] that the performance loss caused by an aggressive pruning schedule in the first layers  can be mitigated by re-training. Therefore, I suggest the authors test more increasing strategies, instead of the logarithmic strategy, to fully utilize the power of SlimGPT.
2. Pruning at a $p\\%$ sparsity does not usually lead to an $\frac{1}{1-p\\%}$ throughput speedup [2]. To demonstrate that SlimGPT actually helps to deploy LLMs, the authors should carefully compare the **throughput** (eg., tokens per sec.) of pruned models generated by SlimGPT and the competing baselines.

[1] Token Merging: Your ViT But Faster (ICLR 2023)

[2] PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptation (ECCV 2024)
questions: questions: Under the same total sparsity value, does different increasing schedules affect the model inference speedup?
limitations: limitations: See Weaknesses and Questions. I do appreciate the theoretical contributions. So my rating may be adjusted after carefully checking the replies and other reviews.

Comments: title: Thanks. Bumping to 6 but please include the additional results, as well as tone down your claim a little.
What's slowing things down from obtaining a 30% pruned SlimGPT?
Why can't you prune SlimGPT 30% to be comparable with your APT report?
Bumping the score to 5, but would need more evals and comparsions to keep increasing my rating.
comment: Dear Reviewer jqTK:

Thanks for your valuable feedback. We sincerely appreciate your time to review our submission and response. We will revise the paper accordingly and incorporate the above results into the updated version.

Best Regards,  
Authors of submission 8375
Thank you for being resourceful and adding many requested experiments during the rebuttal time. **The added result confirms my intuition: that cheap, non-ShreadLlama-like LLM pruning techniques do not perform well under rigorous evaluation.** Your added results on GSM8k and LongBench confirm that, as there are visible drops with just 20% pruned. Note that we usually don't observe such a performance drop with techniques like weight-only quantization at a much more aggressive rate; even with vanilla group-wise quantization with no finetuning.

That being said, I recognize that pruning LLMs is much harder than quantifying LLMs. There surely are some benefits unique to the pruning way, and overall, pruning is without a doubt a school of efficiency worth developing; especially knowing its gap with quantization. **The proposed method is better or at least on par with the established/recent baselines, so I recommend an acceptance with score 6 & confidence 5.** But I urge the authors to:

* Tone down the claim a bit, e.g., the "98% performance" claim in your abstract is slightly misleading. Most common-sense reasoning tasks are easy and do not represent LLM's true capability, so it is almost an overstatement based on cherry-picked results.
* Highlight the results that are not perfect (MMLU, GSM8k, LongBench, etc.) so that future works will have a clear direction for improvement, instead of always muddling those easy tasks.
* Add a proper section to discuss the pros/cons of pruning compared to other efficiency techniques (e.g., quantization) and their unique challenges.
Dear Reviewer jqTK:

In fact, all of our hardware resources were dedicated to running the LongBench experiments, which exceeded our expectations in terms of resource demands. Just before our previous response, we completed the LongBench experiments. Now we finish the pruning experiment for LLaMA2-7B at 30% and the subsequent evaluation. Below, we present our experimental results (with APT data referenced from the paper). We primarily focus on comparing the HellaSwag and MMLU tasks, as these two datasets are used for evaluation in both papers. We train for 3 epochs during fine-tuning. SlimGPT performs slightly lower than APT on MMLU but demonstrates better performance on HellaSwag. And it is important to note that SlimGPT requires fewer iterations and less trainable parameters for LoRA during the fine-tuning phase.

| Method  | Prune% | Tuning Data | Tuning Epoch | LORA rank | HellaSwag Eval | MMLU Eval |
| ------- | ------ | ----------- | ------------ | --------- | -------------- | --------- |
| APT     | **30%**    | Alpaca      | 15           | 8-256     | 71.10           | 36.9      |
| SlimGPT | **30%**    | Alpaca      | 3            | 8         | 72.42           | 35.2      |

Best Regards,  
Authors of submission 8375
Per your global rebuttal, pruning a Llama2-7B to 50% only takes a little over 1,000 seconds, which is almost a trivial cost, and a few epochs of Alpaca are often pretty fast too. So what is actually slowing things down to the point that you *"do not have sufficient time to conduct additional experiments to prune the model by 30% for comparison now (The experiment is ongoing)"*?

Just trying to make sure I am not missing any major things.
Dear Reviewer jqTK:

We would like to thank the reviewer for the comments.

> Why can't you prune SlimGPT 30% to be comparable with your APT report?

Our analysis is based on the results provided in the APT paper and the existing results from the SlimGPT paper. Unfortunately, we do not have sufficient time to conduct additional experiments to prune the model by 30% for comparison now (The experiment is ongoing). We hope you can understand this limitation. Furthermore, the APT paper do not provide evaluation results for GSM8K, so we are currently unable to present GSM8K comparison results.

> More evaluation of long-context tasks on LongBench.

We update our evaluation of long-context tasks on LongBench, which includes 4 task types, as shown in the table below. The current version of the model evaluation has fixed bugs related to input format compared to the previous version, leading to a slight change in performance. 

According to the table, we find that the performance of the pruned model varies on **English** tasks; in some cases, it even exceeds the results of the original model (2WikiMQA: 28.34 vs. 26.32). The majority of tasks maintain over 80% of the original performance.

- Evaluations on Single-Doc QA tasks

| Prune% | Method           | #Params | NarrativeQA | Qasper | MultiFieldQA-en | MultiFieldQA-zh |
| ------ | ---------------- | ------- | ----------- | ------ | --------------- | --------------- | 
| -      | -                | 6.7B    | 27.32       | 31.47  | 48.57           | 49.06           |
| 20%    | SlimGPT          | 5.4B    | 19.33       | 30.37  | 42.64           | 26.65           |

- Evaluations on Multi-Doc QA tasks

| Prune% | Method           | #Params | HotpotQA | 2WikiMQA|Musique|DuReader (zh)|
| ------ | ---------------- | ------- | ----------- | ------ | --------------- | --------------- |
| -      | -                | 6.7B    | 43.11       | 26.32  | 18.81           | 30.57           |
| 20%    | SlimGPT          | 5.4B    | 38.13       | 28.34  | 15.16           | 13.98           |

- Evaluations on Summarization tasks

| Prune% | Method           | #Params |GovReport|QMSum|MultiNews|VCSUM (zh)|
| ------ | ---------------- | ------- | ----------- | ------ | --------------- | --------------- |
| -      | -                | 6.7B    | -           | 22.92  | 25.46           | 14.91           |
| 20%    | SlimGPT          | 5.4B    | -           | 19.95  | 22.78           | 12.49           |

- Evaluations on Few-shot Learning tasks

| Prune% | Method           | #Params |TREC|TriviaQA|SAMSum|LSHT (zh)|
| ------ | ---------------- | ------- | ----------- | ------ | --------------- | --------------- |
| -      | -                | 6.7B    | 68.50       | 86.98  | 42.00           | 39.00           |
| 20%    | SlimGPT          | 5.4B    | 56.00       | 82.59  | 41.46           | 22.75           |

Best Regards,  
Authors of submission 8375
I'll take a closer look at the rest soon, but may the author please address the question as titled? Just want to post earlier so that you can have a chance to reply. I'd also like to see APT on GSM8k under a comparable setting.
Dear Reviewer 8hG2:

We sincerely appreciate your valuable and insightful comments. With less than 24 hours remaining in the discussion period, we anticipate further opinion from you.

We would like to further discuss the LLM inference speed. Since most operators are computed in parallel, the acceleration from pruning does not arise from a reduction in computational load but rather from a decrease in parameter access time, which is a significant bottleneck in large model inference. Therefore, the inference time is not linearly related to the parameter number. In our experiments, we find that pruning 50% of the parameters result in an inference speed that is 63% of the original.

Best Regards,  
Authors of submission 8375
Dear Reviewer QvFZ:

As the discussion period comes to a close, we sincerely look forward to your feedback on our rebuttal. Your further opinions would be essential for us to improve our work.

Regarding your concern that the calibration set is more important than the pruning method, we include additional experiments with LLM-Pruner (our baseline). We perform pruning without fine-tuning at the same pruning ratio, yielding an evaluation result of `PPL=136.19`. In contrast, the worst result for SlimGPT is `PPL=48.26`. Therefore, we believe that while the calibration set may have a slight influence on the bias of model pruning, it does not fundamentally affect the pruning results.

Best Regards,  
Authors of submission 8375
Dear Reviewer jqTK:

We truly appreciate the reviewer's recognition of our work. In response to your suggestions regarding extra experiments, we add the following evaluations:

> Evaluation of LLama2-7b on MMLU and mathematical tasks GSM8k.

| Prune% | Method           | #Params | Humanities | Social Sciences | STEM | Other | Avg  | GSM-8k-Acc  |
| ------ | ---------------- | ------- | ---------- | --------------- | ---- | ----- | ---- | ---- |
| -      | -                | 6.7B    | 43.3       | 51.6            | 36.3 | 52.1  | 45.6 | 13.8 |
| 20%    | LLM-Pruner       | 5.4B    | 25.7       | 23.6            | 24.2 | 26.8  | 25.2 | 2.3  |
| 20%    | SlimGPT w/o tune | 5.4B    | 36.0       | 45.2            | 33.5 | 44.1  | 39.4 | 4.2  |
| 20%    | SlimGPT          | 5.4B    | 35.3       | 42.2            | 31.5 | 43.0  | 37.8 | 6.0  |

The first few columns display the results for MMLU, while the last column shows the evaluation results for GSM-8k. SlimGPT demonstrates significant improvements over the baseline in both tasks. Notably, for the challenging task of GSM-8k, LLM-Pruner retains 16.7% of the performance (2.3 vs. 13.8) after pruning, whereas SlimGPT retains 43% of the performance (6.0 vs. 13.8), achieving more than a twofold increase. Since we employ a very basic and low-cost fine-tuning method, we believe there is room for further performance enhancement.

> Evaluation of Mistral-7B-Instruct-V2.0 on LongBench

Due to the more representative 32k context of Mistral, we will present the evaluation results for Mistral-7B-Instruct-V2.0 on LongBench, hoping this will serve as a valuable reference. 

The model evaluation takes longer than we anticipate, so we primarily provide results for the single-document QA task, as shown in the table below. The performance of the pruned model varies on English tasks, with the capability retaining as much as 97% (30.54 vs. 31.47) in some instances. However, its performance on the cross-language task MultiFieldQA-zh is comparatively modest. We find that this can be largely attributed to the Alpaca fine-tuning, which results in a bias towards answering questions in English, consequently lowering the overall scores. For example, consider the following case.

`{'pred': 'The appeals court determined that the defendant should pay a compensation amount of RMB 57,081.86.', 'answers': ['人民币57081.86元。'], 'all_classes': None, 'length': 2975}`

| Prune% | Method  | #Params | NarrativeQA | Qasper | MultiFieldQA-en | MultiFieldQA-zh |
| ------ | ------- | ------- | ----------- | ------ | --------------- | --------------- |
| -      | -       | 6.7B    | 27.33       | 31.47  | 48.59           | 58.17           |
| 20%    | SlimGPT | 5.4B    | 18.89       | 30.54  | 40.47           | 24.75           |


> SlimGPT vs. APT.

The APT paper actually describes a sparse training pruning method. This approach manually inserts masks before and after specific modules and measures the importance of channels/heads using outliers. Its advantage lies in an end-to-end design; however, its task-specific nature and strong coupling with LoRA complicate its application to other tasks.  

Based on the results provided in the paper, we conduct a rough comparative analysis. While the different pruning ratios make it difficult to directly judge the evaluation results, it is evident that SlimGPT, with 1 epoch naive LORA tuning, offers a more lightweight and straightforward setup without requiring alterations to the model structure.

| Method  | Prune% | Tuning Data | Tuning Epoch | LORA rank | HellaSwag Eval | MMLU Eval |
| ------- | ------ | ----------- | ------------ | --------- | -------------- | --------- |
| APT     | _30%_    | Alpaca      | 15           | 8-256     | 71.1           | 36.9      |
| SlimGPT | _20%_    | Alpaca      | **1**            | **8**         | 74.9           | 37.8      |  


Best Regards,  
Authors of submission 8375
I thank the authors for the detailed rebuttal. The new results — especially on the efficiency front — look decent and thus warrant a slight bump to 5. However, to fully convince me, I'd like to see SlimGPT applied on truly challenging tasks like GSM8k, as well as long context tasks like LongBench and Needle-in-a-Haystack (on llama3 and mistral v0.2). I am particularly interested in the long context front due to the known drawbacks of ShortGPT. 

With the unaligned nature discussed in W1, it looks like the proposed method is mostly compared to LLMPruner. While I recognize that LLMPruner is an established method, I wonder how SlimGPT would perform against some of the more advanced methods, like APT [2].

p.s. While I appreciate the addition of Baichuan, I would still like to see the MMLU report on a more mainstream model, like llama2-7b, for better cross-referencing needs. May the authors supply that? 

---

[2] APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference