Title: AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Pruning

large language models

heavy tails
Abstract: Recent work on pruning large language models (LLMs) has shown that one can eliminate a large number of parameters without compromising performance, making pruning a promising strategy to reduce LLM model size. Existing LLM pruning strategies typically assign uniform pruning ratios across layers, limiting overall pruning ability; and recent work on layerwise pruning of LLMs is often based on heuristics that can easily lead to suboptimal performance. In this paper, we leverage Heavy-Tailed Self-Regularization (HT-SR) Theory, in particular the shape of empirical spectral densities (ESDs) of weight matrices, to design improved layerwise pruning ratios for LLMs. Our analysis reveals a wide variability in how well-trained, and thus relatedly how prunable, different layers of an LLM are. Based on this, we propose AlphaPruning, which uses shape metrics to allocate layerwise sparsity ratios in a more theoretically-principled manner. AlphaPruning can be used in conjunction with multiple existing LLM pruning methods. Our empirical results show that AlphaPruning prunes LLaMA-7B to 80% sparsity while maintaining reasonable perplexity, marking a first in the literature on LLMs.

AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models

Abstract

Recent work on pruning large language models (LLMs) has shown that one can eliminate a large number of parameters without compromising performance, making pruning a promising strategy to reduce LLM model size. Existing LLM pruning strategies typically assign uniform pruning ratios across layers, limiting overall pruning ability; and recent work on layerwise pruning of LLMs is often based on heuristics that can easily lead to suboptimal performance. In this paper, we leverage Heavy-Tailed Self-Regularization (HT-SR) Theory, in particular the shape of empirical spectral densities (ESDs) of weight matrices, to design improved layerwise pruning ratios for LLMs. Our analysis reveals a wide variability in how well-trained, and thus relatedly how prunable, different layers of an LLM are. Based on this, we propose AlphaPruning , which uses shape metrics to allocate layerwise sparsity ratios in a more theoretically-principled manner. AlphaPruning can be used in conjunction with multiple existing LLM pruning methods. Our empirical results show that AlphaPruning prunes LLaMA-7B to 80% sparsity while maintaining reasonable perplexity, marking a first in the literature on LLMs. We have open-sourced our code. 2

1 Introduction

Recent work on pruning large language models (LLMs) [Jaiswal et al., 2023, Frantar and Alistarh, 2023a, Sun et al., 2023] has shown the ability to reduce the number of parameters significantly, without compromising performance, resulting in notable savings in memory footprint, computing time, and energy consumption. Unlike pre-LLM pruning methods [Sanh et al., 2020, Kurtic et al., 2022], existing LLM pruning approaches typically allocate the 'sparsity budget' (i.e., the number of pruned parameters or pruning ratios) uniformly across layers, making it difficult to increase sparsity to very high levels. Relatively little effort has been put into developing theoretically-principled ways to compute layerwise pruning ratios. For example, the Outlier Weighed Layerwise sparsity (OWL) method [Yin et al., 2023] uses a nonuniform layerwise sparsity based on the distribution of outlier activations. However, OWL relies on heuristics related to the presence of outliers [Kovaleva et al., 2021, Puccetti et al., 2022, Dettmers et al., 2022]. This can lead to suboptimal performance in the absence of outliers, and this can make it difficult to achieve very aggressive levels of sparsity. For example, Yin et al. [2023] shows that pruning LLMs to 80% sparsity often significantly degrades the prediction performance of LLMs.

In developing a principled approach to allocate sparsity budgets across layers, we draw inspiration from Heavy-Tailed Self-Regularization (HT-SR) Theory [Martin et al., 2021, Martin and Mahoney,

Weight matrices

ESDs

Metric values

Pruning ratios

Linear

mapping

LLM

. . .

N

More heavy-tailed

u1

Less heavy-tailed

u2

V2

PLfitting

u3

V3

ESD

2021a,b, 2019, 2017, 2020, Yang et al., 2023, Zhou et al., 2023, Qing et al., 2024]. HT-SR theory analyzes the weight matrices of models to derive quantities (related to the shape of the weight matrix eigenspectrum), which help characterize model capacity and quality. Applications of HT-SR to model selection [Martin and Mahoney, 2019, 2020, 2021a, Martin et al., 2021, Yang et al., 2023] and layerwise adaptive training [Zhou et al., 2023] demonstrate the effectiveness of the theory in estimating model and layer quality. Furthermore, in the context of pruning memory/computation-efficient LLMs, using this kind of low-cost weight analysis is advantageous because it requires no data or gradient backpropagation.

Our study consists of two main parts. In the first part, we evaluate the effectiveness of various weight matrix-based metrics for allocating layer-wise sparsity. Our primary finding indicates that shape metrics generally outperform scale metrics in determining layer importance for pruning. Shape metrics capture the shape properties of the empirical spectral densities (ESDs) of layer weight matrices, whereas scale metrics, like matrix norms, reflect the size or scale of ESDs. This offers a novel perspective since shape metrics are less frequently used in the literature than scale metrics, which have been used to create regularizers [Yoshida and Miyato, 2017] and inform pruning [Han et al., 2015].

In the second part of the paper, we introduce a theoretically principled layer-wise sparsity allocation method, AlphaPruning , based on metrics that quantify a unique heavy-tailed (HTed) shape of ESDs. According to HT-SR theory, well-trained models display strong correlations among the weight matrix elements, leading to HT structures in layer weight matrices' ESDs [Martin and Mahoney, 2019, 2020, 2021a]. Moreover, layers with more pronounced HT properties are typically better trained than others. We quantify the HT properties by fitting a power law (PL) distribution [Alstott et al., 2014, Clauset et al., 2009] to ESD and using the PL exponent as the HT metric PL Alpha Hill . 3 The principle of AlphaPruning is to allocate less sparsity to more well-trained (more HTed) layers, as indicated by lower PL Alpha Hill values, thereby preserving their quality during pruning. Figure 1 illustrates the pipeline of AlphaPruning .

We conducted a comprehensive empirical evaluation to assess the generalizability of AlphaPruning in LLM pruning. This evaluation involved comparisons with various baseline methods, integration with existing techniques, and testing across different architectures. We also conducted sanity checks to ensure that AlphaPruning indeed reduces the variation of PL Alpha Hill values across layers. Our key contributions are summarized below.

This paper is the first to study principled layer-wise sparsity allocation from the HT-SR perspective. We systematically evaluate multiple weight matrix-based metrics to compute sparsity based on their effectiveness in estimating layer quality, discovering an interesting finding: shape metrics outperform scale metrics in allocating sparsity, despite the latter being more commonly used.

We introduce a novel sparsity allocation method, AlphaPruning , inspired by HT-SR theory, which demonstrates superior performance in LLM pruning. This method assigns sparsities based on the heavy-tailed shape of ESDs in layer weight matrices, a previously unexplored concept. Our empirical evaluations span a range of LLM architectures, including the LLaMA V1-3 families [Touvron et al., 2023a,b], OPT families [Zhang et al., 2023], Vicuna-7B [Zheng et al., 2024], and Mistral-7B [Jiang et al., 2023]. The results show that AlphaPruning outperforms OWL [Yin et al., 2023], the current SOTA non-uniform sparsity allocation method for LLMs, reducing perplexity by 304.31 and achieving an average accuracy gain of 4.6% over 7 zero-shot tasks at 80% sparsity, while providing a 3.06 × end-to-end speedup on CPUs for LLaMA-7B on the DeepSparse [NeuralMagic, 2021] inference engine. AlphaPruning also outperforms six layer-wise allocation methods, including global thresholding [Frankle and Carbin, 2018], ER [Mocanu et al., 2018], ER-Plus [Liu et al., 2022a], LAMP [Lee et al., 2020], rank selection [Kuzmin et al., 2019, El Halabi et al., 2022], and layer-wise error thresholding [Ye et al., 2020, Zhuang et al., 2018].

AlphaPruning provides layer-wise budget allocation (e.g., sparsity), demonstrating remarkable generalizability and can be integrated with multiple LLM compression techniques to enhance performance. This includes unstructured pruning (Wanda [Sun et al., 2023], SparseGPT [Frantar and Alistarh, 2023b]), with or without fine-tuning [Hu et al., 2021], semi-structured (DominoSearch [Sun et al., 2021]), structured pruning (LLMPruner [Ma et al., 2023], OSSCAR [Meng et al., 2024]), mixed-precision quantization [Tang et al., 2022]. Additionally, we have extended this method to large Computer Vision (CV) architectures, such as Vision Transformers (ViT) [Dosovitskiy et al., 2020], and ConvNext [Liu et al., 2022b]. In this case, OWL significantly underperforms AlphaPruning due to the lack of outlier features. This indicates HT metrics are more general than outlier metrics.

AlphaPruning is theoretically driven, and its improvements can be interpreted by HT-SR metrics. We demonstrate that model performance correlates with model-wise and layer-wise changes in PL Alpha Hill before and after pruning. Furthermore, compared to baseline pruning methods, AlphaPruning not only improves model performance but also achieves a lower mean of layer-wise PL Alpha Hill . HT-SR theory suggests that AlphaPruning preserves the quality of model layers 'on average', minimizing the damage caused by pruning.

We provide an overview of related work in Appendix C.

2 Background and Setup

2.1 Notation

Consider a NN with L layers, W i is one of the weight matrices extracted from the i -th layer with shape m × n ( m ≥ n ). We note that the 'layer' used in this work refers to the transformer block (layer), and each block contains multiple weight matrices, such as the attention layer weight matrix, and projection layer weight matrix. The correlation matrix X i = W ⊤ i W i is an n × n symmetric matrix, and the ESD of X i is formulated as

where λ 1 ( X i ) ≤ . . . ≤ λ n ( X i ) are the eigenvalues of X i and δ is the Dirac delta function. The ESD is a probability measure, which can be viewed as a distribution of the eigenvalues of X i .

2.2 HT-SR theory and metrics

Here, we provide a brief overview of HT-SR theory. HR-SR theory originated as a semi-empirical theory, with early seminal work [Martin and Mahoney, 2019, Martin et al., 2021] examining the empirical spectral density (ESD) of weight matrices, specifically the eigenspectrum of the correlation matrix X i = W ⊤ i W i . This research found that the structures of the ESDs strongly correlate with training quality. These findings are rooted in statistical physics and Random Matrix Theory [Couillet and Liao, 2022], as detailed in Table 1 of Martin and Mahoney [2019]. It is well-known [Wang et al., 2024, Couillet and Liao, 2022] that spikes in ESD represent 'signals,' while the bulk represents noise, which follows the Marchenko-Pastur law. In the theoretical setting of Wang et al. [2024], the signal or the spike aligns with ground-truth features from the teacher model, and that corresponds to increased correlations in weight elements. Furthermore, Kothapalli et al. [2024] show that heavy tails in ESD

originate from the interaction between spikes and bulk, which can be quantified precisely using recent advances in the free-probability theory [Landau et al., 2023], and the interaction characterizes the 'bulk-decay' phase in the five-plus-one phase model in Martin and Mahoney [2019], a critical phase between classical 'bulk+spike' model and heavy-tail models.

To quantify the structure of ESDs, HT-SR theory provides several metrics, collectively known as HTSR metrics. These metrics are typically categorized into two groups: scale metrics and shape metrics.

Scale metrics. Scale metrics refer to those obtained from measuring various norms of weight matrices. As demonstrated empirically in Yang et al. [2023], these metrics are often strongly correlated with the generalization gap (which is the gap between training and test performance), instead of the quality of the models. In this paper, we mainly study two scale metrics, Frobenius Norm and Spectral Norm . The Frobenius Norm metric is calculated by the squared Frobenius norm of the weight matrix ∥ W ∥ 2 F ; and the Spectral Norm metric can be calculated by the square of the spectral radius of the weight matrix ∥ W ∥ 2 2 .

Shape metrics. Drawing analytic methods from Random Matrix Theory, HT-SR work analyzes poorly trained and well-trained models and concludes that the performance of these models usually correlates with shapes emerging in their ESDs, such as 'bulk+spike' shape or 'heavy-tailed' shape. The metrics used to characterize these ESD shapes are called shape metrics, and we mainly studied four of them: PL Alpha Hill , Alpha Hat , Stable Rank , and Entropy . PL Alpha Hill is the main metric used in our method, and we define it in Section 3.2. The definitions of other shape metrics (including Alpha Hat , Stable Rank , Entropy ) can be found in Appendix D.

3 Alpha-Pruning

In this section, we first outline the motivation behind AlphaPruning , followed by an introduction to the layer-wise importance metric, PL Alpha Hill , and the sparsity allocation function, as shown in Figure 1. Our empirical analysis reveals that shape metrics generally outperform scale metrics in guiding sparsity allocation with the same function. Notably, PL Alpha Hill , the shape metric used in AlphaPruning , achieves the best results in preliminary evaluations.

3.1 Rationale

HT-SR theory, introduced in Section 2.2, examines the ESD of weight matrices and finds a strong correlation between heavy-tailed structures in the ESD and training quality. It suggests that heavy-tailed structures emerge from feature learning, where useful correlations are extracted during optimization. Layers with more heavy-tailed ESDs tend to capture more signals, indicating better training quality. Inspired by these findings, we propose to assign sparsity based on the heavy-tailed properties of each layer's ESD. Layers with more heavy-tailed ESDs, which contain more learned signals, are assigned lower sparsity, while layers with light-tailed ESDs are assigned higher sparsity. In practice, the heavy-tailed structure is measured by fitting a PL distribution to the ESD, and extracting the PL exponent α as the indicator. This is why our method is named AlphaPruning .

3.2 Estimating layer quality by HT metric

AlphaPruning relies on estimating the layer quality based on the HT characteristic of the layer ESDs, which is quantified by HT metric PL Alpha Hill . Given an ESD µ X i of a weight matrix's correlation matrix, we fit a PL density function p ( λ ) on it, taking values within an interval ( λ min, λ max), formally defined as:

The estimated exponent α is then used as a metric to characterize the HT extent of the ESD, with a lower value means more HTed. We estimate the PL coefficient using the Hill estimator [Hill, 1975, Xiao et al., 2023b, Zhou et al., 2023], and we refer to it as the PL Alpha Hill metric. The Hill estimator is defined as:

where { λ i } n i =1 is sorted in ascending order, and k is a tunable parameter that adjusts the lower eigenvalue threshold λ min for (truncated) PL estimation. We adopt the Fix-finger method [Yang

et al., 2023] to select the k , which sets k such that λ min aligns with the peak of the ESD. Note that PL Alpha Hill and other scale/shape metrics are calculated for each weight matrix individually.

3.3 Allocating sparsity based on the layer quality

AlphaPruning allocates sparsity for each layer (transformer block) by using a mapping function ϕ : R L → R L to map a sequence of layer quality measures q = ( q 1 , q 2 , .., q L ) into corresponding sparsities ϕ ( q ) .

Here, ϕ ( q ) i represents the i -th element of the resulting vector ϕ ( q ) , q i represents the i -th element of the input vector q , and q min, q max represent the minimum and maximum values of q . The normalization factor η adjusts the sparsity levels to achieve the target global sparsity S . Each layer's sparsity is normalized within the interval [ ηs 1 , ηs 2 ] . η is calculated using the equation ∑ L i =1 ϕ ( q ) i d i = S · ∑ L i =1 d i , in which d i is the number of parameters of W i . Both sides of the equation represent the total number of remaining parameters. The ( s 1 , s 2 ) are tunable hyperparameters that adjust the non-uniformity of the sparsity distribution. We note that sparsity allocation is executed on a per-block basis, averaging the HT-SR metric across all matrices within a block to determine q i . This design is supported by an ablation study, presented in Appendix F, which shows that it yields superior performance over a per-matrix allocation. Hyperparameter settings for all experiments are provided in Appendix G.

3.4 Shape vs. scale metrics for sparsity allocation

In this study, we evaluated various HT-SR metrics, as defined in Section 2.2, to assess their effectiveness in estimating layer quality q for computing layer-wise sparsity. Our preliminary experiments involved pruning the LLaMA-7B model to 70% sparsity and assessing its performance through WikiText perplexity and accuracy across seven zero-shot tasks, with results presented in Table 1. We applied each HT-SR metric in conjunction with three intra-layer pruning techniques (which only determine which matrix elements to prune), such as Magnitude, Wanda, and SparseGPT, to thoroughly evaluate their efficacy. Further experiments on Vision Transformers (ViT) are described in Appendix I.1. Across all tests, shape metrics consistently outperformed scale metrics in assigning layer-wise sparsities. This finding suggests that shape metrics are more robust and yield more reliable predictions of layer quality, which extends previous research [Yang et al., 2023, Zhou et al., 2023, Martin et al., 2021] on estimating model quality. Notably, the shape metric PL Alpha Hill that focuses on estimating the HT shape, proved to be the most effective. Consequently, we have adopted PL Alpha Hill as the primary metric in our proposed method, AlphaPruning .

LLMs. Shape metrics are obtained from the shapes of the ESDs. Scale metrics are norm-based metrics measuring the scale of weights matrices (which can also be obtained from the ESD). The results are conducted on LLaMA-7B at 70% sparsity. We show WikiText validation perplexity and average accuracy on seven different zero-shot tasks as evaluation metrics. We observe that shape metrics outperform scale metrics and PL Alpha Hill performs the best.

4 Empirical results

In this section, we evaluate the performance, generalizability, and interpretability of AlphaPruning . Section 4.1 outlines our experimental setup. In Section 4.2, we evaluate AlphaPruning 's performance by comparing it to the SOTA method OWL and five other baseline methods, and we analyze the efficiency of LLMs pruned by AlphaPruning using practical metrics such as FLOPs and latency. Section 4.3 evaluates the generalizability of AlphaPruning by integrating it with various LLM compression techniques including post-pruning fine-tuning, semi-structured pruning, structured pruning, and mixed-precision quantization. Additionally, we extend its application to CV tasks. Section 4.4 provides an analysis of the layer-wise sparsities and the PL Alpha Hill distribution to further elucidate the effectiveness and implications of AlphaPruning .

4.1 Experimental setup

Models and Evaluation. We evaluate AlphaPruning on the three most widely adopted LLM model families: LLaMA 7B/13B/30B/65B [Touvron et al., 2023a], LLaMA-2 7B/13B/70B [Touvron et al., 2023b], OPT 125M/350M/2.7B/6.7B, and other advanced LLMs: LLaMA-3-8B, Vicuna-7B, Mistral-7B. Our evaluation protocol aligns with established methodologies for LLM pruning [Xiao et al., 2023a], including assessments of language modeling proficiency and zero-shot capabilities. Specifically, we evaluate the perplexity on the held-out WikiText [Merity et al., 2016] validation set, and use seven tasks, including BoolQ [Clark et al., 2019], RTE [Wang et al., 2018], HellaSwag [Zellers et al., 2019], WinoGrande [Sakaguchi et al., 2021], ARC Easy and Challenge [Clark et al., 2018] and OpenbookQA [Mihaylov et al., 2018] for downstream zero-shot evaluation [Gao et al., 2023].

Baselines. We apply the layer-wise sparsities determined by AlphaPruning to three LLM pruning methods, including Magnitude [Han et al., 2015], SparseGPT [Frantar and Alistarh, 2023b] and Wanda [Sun et al., 2023]. Magnitude-based pruning is a simple and strong baseline in which weights are discarded based on their magnitudes. Wanda and SparseGPT are two strong LLM pruning baselines due to their capability to sustain reasonable performance even at relatively high sparsity levels (around 50%). All these methods originally used uniform layerwise sparsity. We incorporate AlphaPruning directly into these baselines, and we demonstrate that this results in improved performance. Besides, we also compare AlphaPruning with OWL [Yin et al., 2023], a recently proposed non-uniform LLM pruning method and six layer-wise pruning methods, including global thresholding [Frankle and Carbin, 2018], ER [Mocanu et al., 2018], ER-Plus [Liu et al., 2022a], LAMP [Lee et al., 2020], rank selection [Kuzmin et al., 2019, El Halabi et al., 2022], and layer-wise error thresholding [Ye et al., 2020, Zhuang et al., 2018].

4.2 Main results

Language Modeling. In Table 2, we report the perplexity of the pruned LLaMA and LLaMA-2 models at 70% sparsity. We provide results for more sparsity levels in Figure 2 and Appendix H. AlphaPruning , as a general layerwise sparsity method, consistently demonstrates performance improvements when used in conjunction with various pruning methods. For example, in the case of LLaMA-7B with a sparsity of 70%, AlphaPruning produces sparse networks with a perplexity of 231.01, significantly outperforming the Magnitude-based pruning baseline of 48419.13. Notably, when applied to Wanda and SparseGPT, two robust LLM pruning methods, AlphaPruning still achieves substantial perplexity reductions, evidenced by a decrease of 61.91 for Wanda and 7.76 for SparseGPT, in the case of LLaMA-7B with a sparsity of 70%.

Zero-shot tasks. We conducted empirical evaluations to determine the zero-shot ability of pruned LLMs on diverse zero-shot downstream tasks with prompting. The results are shown in Table 3, where we show the mean zero-shot accuracy on 7 zero-shot tasks of pruned LLaMA and LLaMA-2 models at sparsity of 70%. AlphaPruning consistently improves accuracy across all settings. For example, AlphaPruning achieves an average accuracy gain of 8.79, 6.05, and 2.61 over 7 tasks and 7 models compared to Magnitude, Wanda, and SparseGPT alone, respectively. These results highlight the promise of AlphaPruning for more challenging zero-shot downstream tasks.

More baseline comparison. For allocating layerwise sparsity ratios, we compare AlphaPruning with other allocation methods. The experiments involve pruning the LLaMA-7B model and LLaMA13B to various sparsities. We use Wanda as the basic pruning method, with results presented in Figure 2 and Figure 3. Results indicate that AlphaPruning significantly outperforms all baseline

methods in relatively high-sparsity regimes. For LLaMA-7B and LLaMA-13B pruned to 80% sparsity, AlphaPruning reduces perplexity by 304.31 and 200.54 compared to OWL, respectively. Achieving high levels of sparsity is crucial for unstructured sparsity to yield significant speedups on GPUs by leveraging existing sparse kernels. Sparse kernels such as Flash-LLM [Xia et al., 2023] and Sputnik [Gale et al., 2020] have shown that unstructured sparsity outperforms dense computation in terms of performance, but only when sparsity levels reach 60% and 71%, respectively. The importance of achieving high sparsity is further substantiated in the following section, which demonstrates that high sparsity levels facilitate significant end-to-end inference speedup. Additional comparisons with rank selection and layer-wise error thresholding, as well as results using SparseGPT as the pruning method and low sparsity results, can be found in Appendices H.

65

70

75

80

Sparsity (%)

100

500

900

Perplexity

65

70

75

80

Sparsity (%)

100

200

300

Perplexity

65

70

75

80

Sparsity (%)

30

40

50

Accuracy (%)

Efficiency measure. To verify the sparse LLM pruned by our method can indeed achieve speedups when deployed on the CPU, we provide new results in Table 4. We apply our method to Llama27B-Chat-hf, prune it to different sparsities, and then test its end-to-end decode latency using DeepSparse [Kurtic et al., 2023] inference engine on an Intel Xeon Gold 6126 CPU with 24 cores. The results indicate that when the global sparsity reaches 80%, the speedup reaches 3.06 × .

In Appendix I.2, we evaluate the pruned LLM by efficiency metrics other than sparsity, such as FLOPs, Compared with uniform sparsity ratios, our approach is able to achieve better performance-FLOPs trade-off. In Appendix I.3, we show that AlphaPruning can control the minimum layer sparsity without losing the performance advantage to meet the hardware requirements of a memory-limited device. In Appendix I.4, we report the runtime of AlphaPruning to show that the computational overhead is reasonable. The computational complexity is not large because the most computationintensive aspect of our method involves performing SVD decomposition on weight matrices.

4.3 Corroborating results

To demonstrate the generalizability of AlphaPruning , we first evaluate if the performance of the model pruned by AlphaPruning can be well recovered by fine-tuning. Then we apply AlphaPruning to other LLM compression techniques (semi-structured, structured pruning, and quantization), and CV models.

Fine-tuning. We show the performance of LLMs pruned by AlphaPruning can be well recovered by fine-tuning. We investigate the parameter-efficient strategies for fine-tuning LLMs: LoRA finetuning [Hu et al., 2021]. Fine-tuning is conducted on the C4 training set [Raffel et al., 2020] with the pre-training auto-regressive loss. The pruned mask is fixed during fine-tuning. The low-rank ( r = 8) adapter is applied to the query and value projection matrices in the attention layers. We fine-tune LLaMA-7B pruned by SparseGPT at various sparsities. Table 6 summarizes the results for perplexity and mean zero-shot accuracy after fine-tuning pruned LLaMA-7B models. We can see the performance of pruned LLMs can be notably improved with very light LoRA fine-tuning. In Appendix I.8, we further compare AlphaPruning with baselines. The results show that the advantages of AlphaPruning don't diminish after fine-tuning.

More LLM architectures. To demonstrate that the effectiveness of AlphaPruning is robust across various more advanced LLMs, we also apply AlphaPruning to LLaMA-3-7B, Vicuna-7B [Zheng et al., 2024], and Mistral-7B [Jiang et al., 2023]. The results in Table 5 show that, as a general method, AlphaPruning can consistently achieve performance improvement across different architectures. Results for OPT families can be found in Appendix I.5.

Integration with other compression techniques. To demonstrate the generalizability of our non-uniform layerwise sparsity, we integrated AlphaPruning with three prominent LLM compression methods: N:M sparsity, structured pruning, and quantization. In Appendix I.6, we examine a mixed N:8 sparsity setup using DominoSearch [Sun et al., 2021] as well as combine AlphaPruning with structured pruning methods LLMPruner [Ma et al., 2023] and OSSCAR [Meng et al., 2024]. In Appendix I.7, we merge our method with mixedprecision quantization as [Tang et al., 2022]. Across these configurations, AlphaPruning consistently enhances the performance of baseline methods.

Vision models. To illustrate AlphaPruning in a broader context, we study how it performs against other methods for determining layerwise sparsity on CV tasks, where non-uniform layerwise sparsity has been widely used. We consider two modern CV architectures:

50

60

70

80

Sparsity (%)

20

40

60

80

Accuracy (%)

Uniform

OWL

Ours

ViT [Dosovitskiy et al., 2020] and ConvNext [Liu et al., 2022b]. We adopt Wanda as the pruning approach and compare AlphaPruning with uniform layerwise sparsity and OWL. We present the results on ConvNext in Figure 4 and provide more results on DeiT and ViT models in Appendix I.9. These results affirm that AlphaPruning effectively allocates layerwise sparsities in CV tasks as well, where OWL significantly underperforms AlphaPruning due to the lack of outlier features. This indicates HT metrics are more general than outlier metrics.

4.4 Analyzing LLM pruning via HT-SR perspective

We study how the PL Alpha Hill metric as a measure of model quality (based on HT-SR theory) changes before and after pruning. In particular, we show that the proposed method, AlphaPruning , effectively controls the damage of pruning to model quality, resulting in a more favorable distribution (lower mean) of PL Alpha Hill among the model layers compared to the baseline pruning method.

Analyzing PL Alpha Hill affected by pruning. We investigate how PL Alpha Hill values change before and after pruning. According to HT-SR Theory [Yang et al., 2023, Martin et al., 2021], models or layers of higher quality typically exhibit lower PL Alpha Hill values. As observed in Figure 5a, dense LLaMA models (depicted in gray) consistently show a lower mean PL Alpha Hill , with larger models demonstrating an even lower mean. Furthermore, we can see both pruning methods lead to increments of the metric value, as pruning is often seen as damaging the quality of the model, the changes are well correlated with the perplexity. More interestingly, AlphaPruning not only outperforms the Uniform pruning in perplexity, but it also leads to a smaller mean of PL Alpha Hill . Figure 5b delves deeper into this phenomenon by visualizing the metric value in a layer-wise manner. It is noticeable that AlphaPruning leads to lower PL Alpha Hill than the uniform pruning among the first several blocks. This is due to the mechanism (Figure 8) by which our method prunes the model based on the layer-wise PL Alpha Hill , and prunes less on these more heavy-tailed layers.

Based on these results, we can conclude that: (1) pruning a model to a larger sparsity generally hurts the model's task performance (e.g., perplexity), and this coincides with decreased model quality, as measured by PL Alpha Hill ; and (2) a better pruning method such as AlphaPruning can obtain a sparse model with a smaller mean of layer-wise PL Alpha Hill , and this is achieved by pruning less aggressively in the dense layers with lower PL Alpha Hill layers.

Here we provide more analytical results in Appendix E. We show analyses on other LLMs provided in Appendix E.1. We compare our HT-SR metrics with other layer quality metrics in Appendix E.2. Additionally, in Appendix E.3, we investigate the distribution of layerwise sparsities allocated by the heavy-tailed metric PL Alpha Hill . In Appendix E.4, we also examine the apparent connection between our method, AlphaPruning , and Low-Rank Approximation (LRA) [Zhang et al., 2015, Wen et al., 2017, Xu et al., 2019, Barsbey et al., 2021] from two perspectives. First, we study the relationship between the ESD used in our method and the low-rank properties often used in LRA, where we use PL Alpha Hill to measure HT and Stable Rank to measure low-rank properties. Second, we study the differing layer-wise assignment strategies adopted by the two methods. Finally, we discuss

Dense

Uniform

AlphaPruning

LLaMA-7B

LLaMA-13B LLaMA-30B

3.0

3.2

3.4

3.6

PL_Alpha_Hill

0

10

20

Perplexity

(a) Model-wise heavy-tail analysis

(b) Layer-wise heavy-tail analysis

how our findings relate to those of Barsbey et al. [2021], a paper closely related to ours, showing that the results from both studies complement each other, offering different yet compatible insights.

5 Conclusion

We have used methods from HT-SR Theory to develop improved methods for pruning LLMs. The basic idea is to analyze the ESDs of trained weight matrices and to use shape metrics from these ESDs to measure how much to prune a given layer, with less well-trained layers, as measured by these shape metrics from HT-SR Theory, being pruned more aggressively. Our extensive empirical evaluation demonstrates that AlphaPruning offers a straightforward yet effective way of determining the layer-wise sparsity ratios. Our analysis reveals that different layers of an LLM are not equally trained (typically, the ESDs of early layers are more HT and thus are more well-trained, compared to later layers), and that shape-based ESD metrics work better for layer quality prediction in pruning pipelines than scale-based ESD metrics. AlphaPruning achieves higher sparsity, without severely hurting performance, and also smaller values of PL Alpha Hill after pruning. AlphaPruning is also compatible with multiple existing LLM pruning methods and is expected to be integrated with future ones, as long as the methods allow specifying layerwise ratios.

References

Jeff Alstott, Ed Bullmore, and Dietmar Plenz. powerlaw: a python package for analysis of heavy-tailed distributions. PloS one , 9(1):e85777, 2014.

Melih Barsbey, Milad Sefidgaran, Murat A Erdogdu, Gael Richard, and Umut Simsekli. Heavy tails in sgd and compressibility of overparametrized neural networks. Advances in neural information processing systems , 34:29364-29378, 2021.

Srinadh Bhojanapalli, Ayan Chakrabarti, Andreas Veit, Michal Lukasik, Himanshu Jain, Frederick Liu, Yin-Wen Chang, and Sanjiv Kumar. Leveraging redundancy in attention with reuse transformers. Technical Report Preprint: arXiv:2110.06821, 2021.

Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of Machine Learning and Systems , 2:129-146, 2020.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems , 33:1877-1901, 2020.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. Technical Report Preprint: arXiv:1905.10044, 2019.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. Technical Report Preprint: arXiv:1803.05457, 2018.

Aaron Clauset, Cosma Rohilla Shalizi, and Mark EJ Newman. Power-law distributions in empirical data. SIAM review , 51(4):661-703, 2009.

Romain Couillet and Zhenyu Liao. Random Matrix Methods for Machine Learning . Cambridge University Press, 2022.

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. Technical Report Preprint: arXiv:2208.07339, 2022.

Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 293-302, 2019.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. Technical Report Preprint: arXiv:2010.11929, 2020.

Marwa El Halabi, Suraj Srinivas, and Simon Lacoste-Julien. Data-efficient structured pruning via submodular optimization. Advances in Neural Information Processing Systems , 35:36613-36626, 2022.

Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning (ICML) , pages 2943-2952, 2020.

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. Technical Report Preprint: arXiv:1803.03635, 2018.

Elias Frantar and Dan Alistarh. Massive language models can be accurately pruned in one-shot. Technical Report Preprint: arXiv:2301.00774, 2023a.

Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning , pages 10323-10337, 2023b.

Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. Technical Report Preprint: arXiv:1902.09574, 2019.

Trevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pages 1-14. IEEE, 2020.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023.

Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403.17887 , 2024.

Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems , pages 1135-1143, 2015.

Bruce M Hill. A simple general approach to inference about the tail of a distribution. The annals of statistics , pages 1163-1174, 1975.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. Technical Report Preprint: arXiv:2106.09685, 2021.

Ajay Jaiswal, Shiwei Liu, Tianlong Chen, and Zhangyang Wang. The emergence of essential sparsity in large pre-trained models: The weights that matter. Technical Report Preprint: arXiv:2306.03805, 2023.

Steven A Janowsky. Pruning versus clipping in neural networks. Physical Review A , 39(12):6600, 1989.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. (Preprint: arXiv:2310.06825), 2023.

Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. Technical Report Preprint: arXiv:2306.07629, 2023.

Vignesh Kothapalli, Tianyu Pang, Shenyang Deng, Zongmin Liu, and Yaoqing Yang. Crafting heavy-tails in weight matrix spectrum without gradient noise. arXiv preprint arXiv:2406.04657 , 2024.

Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. Bert busters: Outlier dimensions that disrupt transformers. Technical Report Preprint: arXiv:2105.06990, 2021.

Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. Technical Report Preprint: arXiv:2203.07259, 2022.

Eldar Kurtic, Denis Kuznedelev, Elias Frantar, Michael Goin, and Dan Alistarh. Sparse fine-tuning for inference acceleration of large language models, 2023. URL https://arxiv.org/abs/ 2310.06927 .

Andrey Kuzmin, Markus Nagel, Saurabh Pitre, Sandeep Pendyam, Tijmen Blankevoort, and Max Welling. Taxonomy and evaluation of structured compression of convolutional neural networks. arXiv preprint arXiv:1912.09802 , 2019.

Itamar D Landau, Gabriel C Mel, and Surya Ganguli. Singular vectors of sums of rectangular random matrices and optimal estimation of high-rank signals: The extensive spike model. Physical Review E , 108(5):054129, 2023.

Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in Neural Information Processing Systems , pages 598-605, 1990.

Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer-adaptive sparsity for the magnitude-based pruning. Technical Report Preprint arXiv:2010.07611, 2020.

Mingbao Lin, Rongrong Ji, Yuxin Zhang, Baochang Zhang, Yongjian Wu, and Yonghong Tian. Channel pruning via automatic structure search. Technical Report Preprint: arXiv:2001.08565, 2020.

Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and Mykola Pechenizkiy. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. Technical Report Preprint: arXiv:2202.02643, 2022a.

Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11976-11986, 2022b.

Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. Technical Report Preprint: arXiv:2305.11627, 2023.

Charles H Martin and Michael W Mahoney. Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior. Technical Report Preprint: arXiv:1710.09553, 2017.

Charles H Martin and Michael W Mahoney. Traditional and heavy tailed self regularization in neural network models. In International Conference on Machine Learning , 2019.

Charles H Martin and Michael W Mahoney. Heavy-tailed universality predicts trends in test accuracies for very large pre-trained deep neural networks. In SIAM International Conference on Data Mining , 2020.

Charles H Martin and Michael W Mahoney. Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metrics. (Preprint: arXiv:2106.00734), 2021a.

Charles H Martin and Michael W Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research , 22(165):1-73, 2021b.

Charles H Martin, Tongsu Serena Peng, and Michael W Mahoney. Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data. Nature Communications , 12(1):1-13, 2021.

Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853 , 2024.

Xiang Meng, Shibal Ibrahim, Kayhan Behdin, Hussein Hazimeh, Natalia Ponomareva, and Rahul Mazumder. Osscar: One-shot structured pruning in vision and language models with combinatorial optimization. arXiv preprint arXiv:2403.12983 , 2024.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. Technical Report Preprint: arXiv:1609.07843, 2016.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. Technical Report Preprint: arXiv:1809.02789, 2018.

Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications , 9(1):2383, 2018.

Michael C Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from a network via relevance assessment. Advances in Neural Information Processing Systems , 1, 1988.

NeuralMagic. Neuralmagic deepsparse inference engine. 2021. URL https://github.com/ neuralmagic/deepsparse .

Giovanni Puccetti, Anna Rogers, Aleksandr Drozd, and Felice Dell'Orletta. Outliers dimensions that disrupt transformers are driven by frequency. Technical Report Preprint: arXiv:2205.11380, 2022.

Peijun Qing, Chongyang Gao, Yefan Zhou, Xingjian Diao, Yaoqing Yang, and Vosoughi Soroush. Alphaexpert: Assigning lora experts based on layer training quality. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , 2024.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research , 21(1):5485-5551, 2020.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM , 64(9):99-106, 2021.

Victor Sanh, Thomas Wolf, and Alexander M Rush. Movement pruning: Adaptive sparsity by fine-tuning. Technical Report Preprint: arXiv:2005.07683, 2020.

Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 8815-8821, 2020.

Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. Technical Report Preprint: arXiv:2306.11695, 2023.

Wei Sun, Aojun Zhou, Sander Stuijk, Rob Wijnhoven, Andrew O Nelson, Henk Corporaal, et al. Dominosearch: Find layer-wise fine-grained n: M sparse schemes from dense neural networks. Advances in Neural Information Processing Systems , 34:20721-20732, 2021.

Chen Tang, Kai Ouyang, Zhi Wang, Yifei Zhu, Wen Ji, Yaowei Wang, and Wenwu Zhu. Mixedprecision neural network quantization via learned layer-wise importance. In European Conference on Computer Vision , pages 259-275. Springer, 2022.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ ee Lacroix, Baptiste Rozi` ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. Technical Report Preprint: arXiv:2302.13971, 2023a.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Technical Report Preprint: arXiv:2307.09288, 2023b.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. Technical Report Preprint: arXiv:1804.07461, 2018.

Wenxuan Wang and Zhaopeng Tu. Rethinking the value of transformer components. Technical Report Preprint: arXiv:2011.03803, 2020.

Zhichao Wang, Andrew Engel, Anand D Sarwate, Ioana Dumitriu, and Tony Chiang. Spectral evolution and invariance in linear-width neural networks. Advances in Neural Information Processing Systems , 36, 2024.

Wei Wen, Cong Xu, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Coordinating filters for faster deep neural networks. In Proceedings of the IEEE international conference on computer vision , pages 658-666, 2017.

Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang, Zhongzhu Zhou, Xiafei Qiu, Yong Li, Wei Lin, and Shuaiwen Leon Song. Flash-llm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity. arXiv preprint arXiv:2309.10285 , 2023.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning , pages 38087-38099, 2023a.

Xuanzhe Xiao, Zeng Li, Chuanlong Xie, and Fengwei Zhou. Heavy-tailed regularization of weight matrices in deep neural networks. Technical Report Preprint: arXiv:2304.02911, 2023b.

Yuhui Xu, Yuxi Li, Shuai Zhang, Wei Wen, Botao Wang, Wenrui Dai, Yingyong Qi, Yiran Chen, Weiyao Lin, and Hongkai Xiong. Trained rank pruning for efficient deep neural networks. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS) , pages 14-17. IEEE, 2019.

Yaoqing Yang, Ryan Theisen, Liam Hodgkinson, Joseph E Gonzalez, Kannan Ramchandran, Charles H Martin, and Michael W Mahoney. Test accuracy vs. generalization gap: Model selection in nlp without accessing training or testing data. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 3011-3021, 2023.

MaoYe, Chengyue Gong, Lizhen Nie, Denny Zhou, Adam Klivans, and Qiang Liu. Good subnetworks provably exist: Pruning via greedy forward selection. In International Conference on Machine Learning , pages 10820-10830. PMLR, 2020.

Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Mykola Pechenizkiy, Yi Liang, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity. Technical Report Preprint: arXiv:2310.05175, 2023.

Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning. arXiv preprint arXiv:1705.10941 , 2017.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? Technical Report Preprint: arXiv:1905.07830, 2019.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models, 2022. (Preprint: arXiv:2205.01068), 2023.

Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. IEEE transactions on pattern analysis and machine intelligence , 38(10):1943-1955, 2015.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36, 2024.

Yefan Zhou, Tianyu Pang, Keqin Liu, Charles H. Martin, Michael W. Mahoney, and Yaoqing Yang. Temperature balancing, layer-wise weight analysis, and neural network training. In Advances in Neural Information Processing Systems , 2023.

Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. Technical Report Preprint: arXiv:1710.01878, 2017.

Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discrimination-aware channel pruning for deep neural networks. Advances in neural information processing systems , 31, 2018.

Appendix

A Impact Statements

This paper leverages HT-SR Theory to design improved layerwise pruning ratios for LLMs. Although the proposed method could be applied to compress models with adverse applications, we do not see any immediate negative societal impacts stemming from the algorithm itself. Indeed, we see a lot of societal value in proposing our method to the community. Through the implementation of effective layerwise sparsity, we can achieve substantial reductions in the parameters of LLMs while retaining their functionality. Consequently, this advancement facilitates the deployment of LLMs in resource-constrained devices, accelerates the predictions for resource-limited LLM services, and contributes to the sustainability of LLM technologies.

B Limitation

This paper's empirical evaluation focuses on post-training pruning methods without fine-tuning, in alignment with recent research in LLM pruning, such as OWL, Wanda, and SparseGPT. This is due to the substantial computational resources required to restore heavily pruned LLMs to their original performance. Nonetheless, our experiments with limited fine-tuning have demonstrated that the proposed pruning method can achieve only a mild performance drop while yielding significant efficiency improvements.

C Related work

Pruning. Removing weights or connections in a trained neural network (NN) to generate an efficient, compressed model has a long history [LeCun et al., 1990, Mozer and Smolensky, 1988, Janowsky, 1989, Mocanu et al., 2018]. Modern NNs are frequently over-parameterized [Wang and Tu, 2020, Bhojanapalli et al., 2021], and thus removing redundancies improves computation and memory efficiency. A common approach is weight-magnitude-based pruning [Han et al., 2015], which zeros out connections with weights smaller than a specified threshold. However, when it comes to pruning LLMs [Brown et al., 2020, Touvron et al., 2023a], progress has been limited. Conventional pruning typically requires a round of retraining to restore performance [Blalock et al., 2020], which can be challenging for LLMs. To address the difficulty in retraining, researchers have developed specially tailored pruning algorithms for LLMs. For example, Ma et al. [2023] explored sparse structured LLMs, using Taylor pruning to remove entire weight rows, followed by LoRA fine-tuning. More recent research has shifted towards unstructured pruning without the need for fine-tuning, showing substantial advancements. In particular, SparseGPT [Frantar and Alistarh, 2023b] uses the Hessian inverse for pruning and subsequent weight updates to reduce the reconstruction error of dense and sparse weights, while Wanda [Sun et al., 2023] uses a criterion that incorporates weight magnitudes and input activations, in order to preserve outlier features [Kovaleva et al., 2021, Puccetti et al., 2022, Dettmers et al., 2022]. Our work allocates parameters in a more theoretically-principled manner, enabling pruning LLMs to higher sparsity levels.

Layerwise sparsity budgets. Although layerwise sparsity has been widely studied in pre-LLM pruning [Mocanu et al., 2018, Evci et al., 2020, Liu et al., 2022a, Gale et al., 2019, Lee et al., 2020], relatively little attention has been devoted to determining the pruning ratios for each layer in LLMs. (Interestingly, this layer-wise approach has been applied to model quantization [Kim et al., 2023, Shen et al., 2020, Dong et al., 2019].) Frantar and Alistarh [2023b] and Sun et al. [2023] apply a uniform pruning ratio across all layers, and Yin et al. [2023] computes the sparsity budgets using the outlier ratio observed within each layer's token feature distribution. Existing work on sparsity budgets has generally used heuristics, such as different forms of size or scale metrics (such as norm-based metrics), to determine sparsity budgets per layer. For instance, ABCPruner [Lin et al., 2020] reduces the number of combinations of layer sparsities to search over, but it still requires training to determine the empirical validity of its suggested layer sparsities. Lee et al. [2020] modifies Magnitude-based pruning by rescaling the importance scores in a layer by a factor dependent on the magnitude of surviving connections in that layer. However, these methods are suboptimal in allocating layerwise sparsities for pruning LLMs. Recently, Outlier Weighed Layerwise sparsity (OWL) [Yin et al., 2023] designs a nonuniform layerwise sparsity based on the distribution of outlier activations in LLMs.

However, OWL heuristically relies on the emergence of outliers, and this can lead to suboptimal performance when outliers are absent from models. Our work uses HT-SR shape metrics such as PL Alpha Hill to predict layer importance, and it allocates parameters in a more theoretically principled manner, allowing pruning LLMs to higher sparsity levels than has ever been achieved before.

D Definitions of HT-SR metrics

Here, we define the shape metrics, beyond PL Alpha Hill , that we use in our analysis.

( Alpha Hat ) The Alpha Hat metric [Martin et al., 2021] has been shown to be effective at predicting generalization. It is the variant of PL exponent α ( PL Alpha ) weighted by the log maximum eigenvalue log λ max ( log spectral norm ):

( Stable Rank ) The Stable Rank metric is a norm-adjusted measure of the scale of the ESD, and previous work [Yang et al., 2023] has shown its strong correlation with PL Alpha . For a weight matrix W , it can be calculated as:

( Entropy ) For a weight matrix W , the Entropy metric is defined as

where p i = v 2 i / ∑ i v 2 i , v i is the i -th singular value of W , and R ( W ) refers to the rank of W .

E Further analysis of results

E.1 Post-pruning layer-wise heavy-tail analysis

We investigate layer-wise PL Alpha Hill values after pruning by Uniform pruning and AlphaPruning on more advanced LLMs (LLaMA-V3-8B, Vicuna-7B, Mistral-7B). According to HT-SR Theory, models or layers of higher quality typically exhibit lower PL Alpha Hill values. As observed in Figure 6, AlphaPruning leads to a smaller layer-wise PL Alpha Hill . This is due to the mechanism (1) by which our method prunes the model based on the layer-wise PL Alpha Hill , and prunes less on these more heavy-tailed layers.

Uniform

AlphaPruning

2

4

6

8

PL_Alpha_Hill

0.0

0.1

0.2

0.3

0.4

Density

(a) LLaMA-V3-8B

2

4

6

PL_Alpha_Hill

0.0

0.2

0.4

0.6

Density

(b) Vicuna-7B

2.5

5.0

7.5

PL_Alpha_Hill

0.0

0.2

0.4

Density

(c) Mistral-7B

E.2 Comparison with other LLM layer quality metrics

In addition to AlphaPruning proposed in our work, Gromov et al. [2024], Men et al. [2024] are other studies that investigated methods that measure whether a layer is well-trained or not, demonstrating

0

10

20

30

Layer (block) Index

0.6

0.7

0.8

0.9

Sparsity

LLaMA-V1-7B

0

10

20

30

40

Layer (block) Index

0.6

0.7

0.8

0.9

Sparsity

LLaMA-V1-13B

0

10

20

30

40

50

60

Layer (block) Index

0.6

0.7

0.8

0.9

Sparsity

LLaMA-V1-30B

LLMs layers are not equally well-trained. Gromov et al. [2024] developed a method that assesses the similarity between the representations at different layers, defined as the angular distance between feature vectors. They found that deeper layers are more similar to their neighboring layers than shallow layers, suggesting that LLMs may not fully use the parameters in these deeper layers, indicating these layers are not well-trained. Similarly, Men et al. [2024] introduced a metric called Block Influence, which measures the impact of each transformer block on hidden states to gauge layer significance. Their findings showed varying degrees of ineffectiveness/redundancy across layers, suggesting that these layers are not well-trained.

Besides, in Figure 7, we compare the sparsity allocation of AlphaPruning with OWL. We show that the general trends of sparsity distribution generated by the two methods are similar, with lower sparsities allocated to earlier layers and higher sparsities allocated to deeper layers. However, our method produces a more granular distribution with clearer distinctions between consecutive deep layers, resulting in improved pruning performance.

0

10

20

30

Layer Index

2.5

3.5

4.5

PL_Alpha_Hill

PL_Alpha_Hill

0.4

0.8

Sparsity

AlphaPruning

Uniform

E.3 Analyzing the layerwise sparsity distribution.

We elaborate on the layerwise sparsities assigned by AlphaPruning and heavy-tailed metric PL Alpha Hill . The metric and sparsities results of LLaMA-7B are presented in Figure 8. We can observe that, the blue bar histograms demonstrate that different layers of one model show diverse PL Alpha Hill values, this indicates that these layers are not equally well-trained. This conclusion is based on prior research on heavy-tails in weight matrices [Martin et al., 2021, Martin and Mahoney, 2019], this metric measures the heavy-tailed structure within the correlation matrix. This measurement indicates the amount of correlation among the weight matrix elements, with strong correlations leading to a more heavy-tailed empirical spectral density. Such a structure is often seen as a result [Wang et al., 2024] of extracting various useful correlations (or features) from data during optimization.

Consequently, smaller PL Alpha Hill layers (more heavy-tailed) contain a greater number of learned correlations. The larger PL Alpha Hill layers (less heavy-tailed) tend to retain fewer learned correlations, remaining closer to the random initialization state. Figure 8 shows that AlphaPruning suggests that large PL Alpha Hill layers with fewer learned correlations should be allocated with larger sparsity, or being pruned more, while those small PL Alpha Hill layers should be allocated with lower sparsity.

E.4 Connections with Low-rank Approximation.

A commonly used compression technique that also involves analyzing the eigenspectrum of the weights is Low-Rank Approximation (LRA) [Zhang et al., 2015, Wen et al., 2017, Xu et al., 2019, Barsbey et al., 2021]. We examine the apparent connection between our method, AlphaPruning , and LRA from two perspectives. First, we study the relationship between the ESD used in our method and the low-rank properties often used in LRA, where we use PL Alpha Hill to measure HT and Stable Rank to measure low-rank properties. Second, we study the differing layer-wise assignment strategies adopted by the two methods. Finally, we discuss how our findings relate to those of Barsbey et al. [2021], a paper closely related to ours, showing that the results from both studies complement each other, offering different yet compatible insights.

1

2

3

4

Tail indice (

)

0

100

200

300

Stable rank

matrix_size=1000

matrix_size=2000

matrix_size=3000

matrix_size=5000

0

10

20

30

Layer index

3.0

3.5

4.0

PL_Alpha_Hill

PL_Alpha_Hill

100

200

300

Stable rank

Stable rank

5

10

15

20

Reduction ratio

10

15

20

Perplexity

Compress more on HTed layers

Compress less on HTed layers

Uniform

Figure 9: Analyzing ESD properties and assignment strategies for LRA. (a) Stable Rank and PL Alpha Hill show a similar pattern (the more heavy-tailed, the more low-ranked) across different ESDs sampled from Pareto distribution. (b) The layer-wise PL Alpha Hill and Stable Rank of the LLaMA-7B model exhibit a similar trend. (c) Comparing two assignment strategies for LRA, and 'Compress more on HTed layers' is better. This finding is opposite to pruning-based methods, which find that 'Compress less on HTed layers' is better.

For the first point, we find a strong positive correlation between the HTed properties and the low-rank properties used in LRA: as the ESD becomes more HTed, it also becomes more low-ranked. To demonstrate this, we sampled eigenvalues from an IID Pareto distribution with varying tail indices to form ESDs and then measured Stable Rank on these ESDs. A lower PL Alpha Hill indicates a more HTed distribution, while a lower Stable Rank indicates a more low-ranked structure. Figure 9a shows that Stable Rank and PL Alpha Hill are positively correlated across different matrix sizes, suggesting a relationship between HTed and low-ranked properties. In Figure 9b, we further verify this by measuring PL Alpha Hill and Stable Rank across different layers of the pre-trained LLaMA-7B model. The results reveal that both metrics follow a similar trend: shallow layers show lower values, while deeper layers exhibit higher values, indicating consistent behavior across layers. Therefore, we can see that the key metrics used in AlphaPruning and LRA are highly correlated.

For the second point, we clarify that the strategies for assigning higher or lower compression to HTed or low-ranked layers differ between AlphaPruning and LRA. AlphaPruning assigns lower compression ratios (less sparsity) to HTed layers. In contrast, LRA assigns higher compression to low-ranked layers [Zhang et al., 2015, Wen et al., 2017, Xu et al., 2019], which, as established earlier, tend to be HTed. Therefore, the two methods seemingly use different procedures in assigning layerwise compression ratios. In Figure 9c, we empirically verify that, for LRA, assigning higher compression to HTed layers is indeed more beneficial than its opposite for LLMs, as shown by the results of applying both to LLaMA-7B models. This finding is opposite to the assignment method used in AlphaPruning . We hypothesize that these differences arise from the distinct mechanisms and principles of each method. In more detail, AlphaPruning , which is pruning-based,

removes elements from the weight matrices, affecting the entire eigenspectrum. LRA, however, removes only the smallest eigenvalues, leaving the larger eigenvalues intact. The guiding principle of AlphaPruning is also different from LRA. It aims to make the model more deterministic and less random by preserving weights corresponding to well-trained layers. It does this by preserving HTed layers that contain more signals and removing light-tailed layers that, according to HT-SR theory, resemble randomly initialized weight matrices. In some sense, it is similar to how decision trees choose branches to reduce entropy. LRA, on the other hand, focuses on applying more compression on low-rank matrices where the largest eigenvalues dominate. This allows for minimal impact on reconstruction loss when removing small eigenvalues.

Lastly, Barsbey et al. [2021] concluded that models with lower HT measures are generally more compressible than other models, focusing on cross-model comparisons. In contrast, our work examines compressibility within a single model and suggests that, within a model, layers with lower HT values are less compressible than other layers. Our insight supports layer-wise pruning strategies, such as AlphaPruning , which applies less compression to more HTed layers.

0.3

0.6

0.9

Pruning ratio

0.85

0.90

0.95

1.00

Relative test acc

Alpha=2.00

Alpha=1.98

Alpha=1.94

(a) Uniform Pruning

0.3

0.6

0.9

Pruning ratio

0.85

0.90

0.95

1.00

Relative test acc

Alpha=2.00

Alpha=1.98

Alpha=1.94

(b)

AlphaPruning

Compress less on HTed layer

Compress more on HTed layer

Uniform

0.3

0.6

0.9

Pruning ratio

50.0

52.5

55.0

57.5

Test accuracy

(a) Alpha=2.00

0.3

0.6

0.9

Pruning ratio

50.0

52.5

55.0

57.5

Test accuracy

(b) Alpha=1.98

0.3

0.6

0.9

Pruning ratio

50.0

52.5

55.0

57.5

Test accuracy

0.3

0.6

0.9

Pruning ratio

50.0

52.5

55.0

57.5

Test accuracy

(d) Alpha=1.94

We adopt the experimental setup of Barsbey et al. [2021], training fully connected networks (FCNs) with six hidden layers on the CIFAR-10 dataset. The HT measure used in their study referred to as Alpha, quantifies the HTed structure of the weight parameters. The model-wise HT measure is influenced by the 'temperature', defined as the ratio of learning rate to batch size. Higher temperatures yield models with lower HT measures. Our first experiment reproduces the finding of Barsbey et al. [2021], comparing the compressibility of models with varying model-wise HT measures. The results verified that models with lower HT metrics are more compressible. It also tries to verify if the conclusions hold when using our proposed layer-wise strategies. The results shown in Figure 10 support our hypotheses. The second experiment compares the effectiveness of these layer-wise strategies and also checks if the findings are consistent across models with different model-wise HT

measures. The results shown in Figure 11 confirmed that HTed layers are less compressible and that pruning HTed layers less (which AlphaPruning does) is more effective.

F Ablation study of sparsity allocation

In Section 3.3, we introduced the range hyperparameters s 1 and s 2 to control the non-uniformity of layer-wise sparsities. To simplify, we define τ such that s 1 = 1 -τ, s 2 = 1 + τ . AlphaPruning allocates sparsity on a 'per-block' basis, where all matrices within a block receive the same sparsity, determined by averaging the PL Alpha Hill values across matrices within that block. Alternatively, sparsity can be allocated on a 'per-matrix' basis, allowing different sparsities for individual matrices based on their PL Alpha Hill values. The ablation study on comparing per-matrix and per-block choices is presented in F.1. The ablation study on comparing different mapping functions is shown in F.2.

per-block

per-matrix

0

0.5

hyperparameter

5.6950

5.6975

5.7000

perplexity

Sparsity 10%

0

0.5

hyperparameter

5.805

5.810

5.815

perplexity

Sparsity 20%

0

0.2

hyperparameter

6.000

6.005

perplexity

Sparsity 30%

0

0.2

hyperparameter

6.38

6.40

perplexity

Sparsity 40%

0

0.2

hyperparameter

7.175

7.200

7.225

7.250

perplexity

Sparsity 50%

0

0.2

hyperparameter

9.5

10.0

10.5

perplexity

Sparsity 60%

0

0.2

0.4

hyperparameter

40

60

80

perplexity

Sparsity 70%

0

0.2

hyperparameter

2000

4000

6000

perplexity

Sparsity 80%

F.1 Per-matrix vs. per-block

In Figure 12, we compared per-matrix and per-block sparsity allocation across different values of the hyperparameter τ , using Wanda. The results show a regime transition in pruning effectiveness between the two methods. For sparsity levels below 50%, the per-matrix approach (gray lines) achieves lower perplexity, indicating better performance. However, for sparsity values of 50% and above, the per-block method (blue line) performs better. We further focused on high sparsity 70%, as higher sparsity is more important to provide efficiency improvements. Specifically, we evaluated both per-matrix and per-block methods in combination with various intra-layer pruning techniques. Table 7 shows that the per-block method consistently outperforms the per-matrix method when used in conjunction with three intra-layer pruning techniques.

Additionally, we analyzed the differences between MLP and Attention matrices by visualizing average PL Alpha Hill values for seven types of weight matrices in LLaMA-7B, as shown in Figure 13. The results indicate that query and key matrices have lower PL Alpha Hill values, suggesting they are less prunable. Based on this insight, we developed a new allocation strategy called 'Mixed', which combines per-block and per-matrix approaches. As defined, Mixed first assigns a block-wise pruning ratio using the per-block method, then refines it within each block using per-matrix allocation. Table 7 demonstrates that this Mixed approach provides further marginal improvements over both per-matrix and per-block methods.

F.2 Different mapping functions

We provide ablation studies on the choices of sparsity assignment function. We implemented the logarithmic method and compared it with the linear mapping function used in our current approach, as shown in Figure 14. The results show that both methods perform similarly when combined with Wanda, but linear mapping slightly outperforms the proposed logarithmic method when combined with SparseGPT.

65

70

75

80

Sparsity (%)

100

300

500

700

Perplexity

Logarithmic

Linear

65

70

75

80

Sparsity (%)

50

100

150

Perplexity

Logarithmic

Linear

(b) SparseGPT

G Hypterparamter setting

Here, we provide the values of τ used in the experiments, as shown in Table 8.

H More baseline comparison

For assigning layerwise sparsity ratios, we compare AlphaPruning with other methods. In this section, we provide definitions and details of these methods:

Uniform. [Zhu and Gupta, 2017] Every layer pruned with the same target sparsity.

Global [Frankle and Carbin, 2018]. A global threshold uniformly applied to all layers to satisfy the overall sparsity requirement. The specific layerwise sparsity is automatically adjusted based on this threshold.

ER [Mocanu et al., 2018]. The sparsity of the convolutional layer is scaled proportionally to 1 -h l -1 + h l h l -1 × h l where h l refers to the number of neurons/channels in layer l .

ER-Plus [Liu et al., 2022a]. ER-Plus modifies ER by forcing the last layer as dense if it is not, while keeping the overall parameter count the same.

LAMP [Lee et al., 2020]. This method modifies Magnitude-based pruning by rescaling the importance scores in a layer by a factor dependent on the magnitude of surviving connections in that layer.

OWL [Yin et al., 2023]. A non-uniform layerwise sparsity based on the distribution of outlier activations in LLMs, probing the possibility of pruning LLMs to high sparsity levels.

We adopt Wanda and SparseGPT as the pruning approach. The results are presented in Table 9 and Table 10, which indicate that AlphaPruning significantly outperforms all baseline methods in relatively high-sparsity regimes. Besides, we have conducted additional experiments using the 'layerwise error thresholding' method, where each layer is pruned sequentially as specified in [Zhuang et al., 2018, Ye et al., 2020]. We have also implemented the rank selection method as specified in Section 5.2 of [El Halabi et al., 2022]. We present the updated experimental results in Table 11. We observe that our method outperforms all the other baselines.

I Complementary Experimental Results

I.1 Shape metrics versus scale metrics on Vision Transformers

We further evaluate different metrics for computing layerwise sparsity on Vision Transformers. Shape metrics, including Alpha Hat , Entropy , PL Alpha Hill , and Stable Rank , are obtained from the shapes of the ESDs. Scale metrics, including Frobenius Norm and Spectral Norm , are norm-based metrics measuring the scale of weights matrices (which can also be obtained from the ESD). The results shown in Table 12 align with the results in LLMs that shape metrics outperform scale metrics on allocating layerwise sparsity and PL Alpha Hill performs the best.

Vision Transformers. Shape metrics are obtained from the shapes of the ESDs. Scale metrics are norm-based metrics measuring the scale of weights matrices (which can also be obtained from the ESD). We choose two models, ViT-L and DeiT-S, and the results are shown on ImageNet-1K accuracy without fine-tuning. We observe that shape metrics outperform scale metrics and PL Alpha Hill performs the best.

I.2 More results on other efficiency metrics

To further demonstrate the benefits of our approach, we provide results in other practical efficiency metrics such as FLOPs. Compared with uniform sparsity ratios, our approach is able to achieve a better performance-FLOPs trade-off. We have provided new results of FLOPs in Figure 15.

Table 13 summarizes the results from Figure 15. We show that, compared to uniform pruning, our method can achieve significant FLOPs reduction when pruned LLMs are compared at similar perplexity.

400

500

600

700

FLOPs

0

25

50

75

100

125

150

Perplexity

LLaMA-7B

Uniform Pruning

AlphaPruning (ours)

650

700

750

800

850

FLOPs (G)

8

9

10

11

12

Perplexity

LLaMA-7B

Uniform Pruning

OWL

AlphaPruning (ours)

(a) Broader perplexity range

(b) Lower perplexity range

I.3 Controlling the minimum layer sparsity for memory-limited hardware

To enhance the adaptability of our method to hardware, we demonstrate that our method allows for controlling the minimum sparsity, by adjusting the values of s 1 and s 2 . Additionally, Table 14 demonstrates that increasing the minimum sparsity by changing s 1 and s 2 doesn't diminish the advantages of our method, compared to the uniform pruning baseline.

Recall that the sparsity of the layer of the model can be determined by Eqn. 4. Increasing s 1 leads to a higher minimum sparsity ηs 1 , while maintaining the same global sparsity S . Table 14 displays the results of adjusting s 1 and s 2 to increase the minimum sparsity while maintaining the global sparsity at 70%. At a minimum sparsity of 57%, our method achieves the lowest perplexity. Even when the minimum sparsity is raised to 67%, nearing the uniform pruning baseline, our method still attains a perplexity of 49.6, which is 36.17 points lower than that of uniform pruning (85.77).

I.4 Computational complexity of AlphaPruning

The computational complexity of AlphaPruning is not large because the most computation-intensive aspect of our method involves performing SVD decomposition on weight matrices, which can be further optimized through parallel processing. Table 15 presents the runtime of our AlphaPruning and an optimized version that uses parallel processing. The increase in runtime is reasonable, at 32.44% with Wanda and 8.2% with SparseGPT. These experiments were conducted on pruning LLaMA-7B to 70% sparsity. The testing platform used A40 GPUs and an AMD EPYC 7713 64-Core CPU.

I.5 OPT family

In addition to LLaMA and LLaMA-2, we conduct experiments with OPT [Zhang et al., 2023]. Table 16 shows the results of comparing our method to uniform sparsity when both are combined with magnitude pruning. Table 17 shows the same comparison when both combined with Wanda [Sun et al., 2023] /SparseGPT [Frantar and Alistarh, 2023b]. In most of the cases, our method outperforms the baseline method and achieves lower perplexity.

I.6 More results on semi-structured and structured pruning

To assess the potential of our non-uniform layerwise sparsity for hardware-friendly applications, we investigate AlphaPruning across two distinct hardware-friendly pruning regimes: N:M sparsity and structured pruning. Following DominoSearch [Sun et al., 2021], we study the mixed N:8 sparsity configuration. Instead of using a uniform N value across all layers, we allow individual layers to possess distinct N values while maintaining the same parameter count. We adopt AlphaPruning to determine the optimal value of N for individual layers. The results shown in Table 18 demonstrate that AlphaPruning consistently outperforms the baselines.

Furthermore, instead of pruning weights, we follow the recent methodology introduced in LLM Pruner [Ma et al., 2023], wherein entire neurons and attention heads are removed. This action facilitates the direct acceleration of pruned LLMs on GPUs or TPUs. We replace the uniform layerwise sparsity used by the LLM pruner with a non-uniform layerwise sparsity using AlphaPruning . The

results, shown in Table 19, demonstrate that AlphaPruning can improve model performance at various sparsity levels.

Another structured pruning method in LLM is OSSCAR [Meng et al., 2024], which formulates the structured pruning problem as a quadratic program with combinatorial constraints. We integrated AlphaPruning with OSSCAR, and we provide the results in Figure 16. OSSCAR prunes only the linear sublayer of multi-head attention and the second sublayer of the feed-forward network, applying uniform pruning across each transformer block. By incorporating AlphaPruning 's layerwise sparsity allocation, we achieved non-uniform block-wise pruning ratios while keeping the global pruning ratio the same. The results show that integrating AlphaPruning with OSSCAR can reduce perplexity at different sparsities.

0.5

0.6

0.7

Pruning ratio

0

500

1000

Perplexity

OSSACR+AlphaPruning

OSSCAR

I.7 Mixed-Precision Quantization

We provide additional results to show that our method can enhance mixed-precision quantization by allocating precision to layers. We still use the PL Alpha Hill metric to estimate the heavy-tail extent of each layer, and more heavy-tailed layers are then allocated with higher precision.

Table 20 shows that, compared to three baselines (random assignment, assigned by the norm of weights, and OWL), our AlphaPruning method can always achieve the lowest perplexity under three types of mixed-precision quantization. Our experimental setup follows Tang et al. [2022].

I.8 More fine-tuning results

Here, we provide results for LoRA fine-tuning pruned LLaMA-7B with a sparsity of 70% using SparseGPT. We compare AlphaPruning with Uniform and OWL. The experiment settings align with Section 4.3. Table 21 summarizes the results for perplexity and mean zero-shot accuracies after fine-tuning pruned LLaMA-7B models, which show that the performance improvement achieved by our method doesn't diminish after fine-tuning.

I.9 More results on vision models

Here, we present more CV task results. We choose three widely used non-uniform layer-wise sparsity methods in the CV context including Global [Frankle and Carbin, 2018], ERK [Mocanu et al., 2018], LAMP [Lee et al., 2020]. Here, we use four ImageNet-1K pre-trained models (ViT-L, ViT-B, DeiT-B, DeiT-S), and we prune them to different sparsities.

I.10 Zero-shot tasks performance

For zero-shot results in Section 4.2, the 7 evaluated zero-shot tasks are: BoolQ [Clark et al., 2019], RTE [Wang et al., 2018], HellaSwag [Zellers et al., 2019], WinoGrande [Sakaguchi et al., 2021], ARC Easy and Challenge [Clark et al., 2018] and OpenbookQA [Mihaylov et al., 2018]. We show the task-wise performance in Table 23 and Table 24.

J Experiments Compute Resources

We conducted all our experiments using NVIDIA L40 (40GB) GPUs. Specifically, we used a single NVIDIA L40 GPU for pruning the 7B and 13B models, 4 GPUs for the 30B models, and 8 GPUs for the 65B models. For the LoRA fine-tuning, we operated under a constrained computational budget, employing 2x 40GB GPUs for the 7B models. Detailed information on the computational complexity of AlphaPruning is provided in Appendix I.4.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our main claims in the abstract and listed contributions in Section 1 reflect the main contributions made in the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitation of the paper in Appendix B.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Although the paper have no direct theoretical results, our work is theoreticallyprincipled, which heavily relies on the HT-SR theory.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided the code for reproducing the main experimental results and detailed the experimental settings in the Section 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our anonymized version of code is available here

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental setting is presented in section 4.1 and the hyperparameters setting can be seen in Appendix G.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We have conducted tests for the statistical significance of the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The information on the computer resources for experiments can be seen in Appendix J.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We make sure to conform the NeurIPS Code of Ethics in every respect of the paper.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed the societal impacts, which can be seen in Appendix A Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the original paper that produced the code package in our paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We have communicated the details of the code as part of our submissions and anonymized our assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces AlphaPruning, a novel framework for unstructured LLM pruning. The framework leverages HT-SR theory that utilizes the heavy-tailed shape of ESDs in layer-weight matrices to allocate layer-wise sparsity more effectively. By focusing on shape metrics rather than scale metrics, AlphaPruning demonstrates superior performance in maintaining model accuracy and reducing complexity. The method has been empirically validated across various architectures, showing significant improvements in performance and efficiency compared to existing methods, with strong generalizability to other compression techniques and architectures.\n\nSTRENGTHS:\n+ This paper introduces a novel sparsity allocation method that leverages the heavy-tailed shape of ESDs in layer weight matrices, a concept previously unexplored in the literature.\n\n+ The method is extensively validated with a range of LLM architectures, demonstrating significant improvements over state-of-the-art methods in terms of reducing perplexity, increasing accuracy, and achieving computational efficiency.\n\n+ AlphaPruning exhibits remarkable adaptability, integrating well with various other model compression techniques and extending beyond LLMs to include large vision model architectures, proving its versatility and broad applicability.\n\nWEAKNESSES:\n+ The novelty of the method for allocating sparsity based on layer quality (Section 3.2) is incremental, as a similar idea has previously been proposed in [1].\n\n[1] Zhou, Yefan, et al. Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Training.\n\nQUESTIONS:\n+ Section 3.2 proposes a linear mapping to get sparsity from layer quality. Would other mappings, such as first compute the logarithmic of the m and then perform linear mapping, yield better outcomes?\n\n+ I would like to see results on using AlphaPruning to determine layerwise sparsity for other structured pruning methods, such as OSSCAR [1].\n\n[1] Meng, X., Ibrahim, S., Behdin, K., Hazimeh, et al. OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimization.\n\nLIMITATIONS:\nThe authors addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents AlphaPruning, which prunes weight matrices of LLM models with different layer-wise sparsity levels based on the Heavy-tailed self-regularization theory.\nCompared to pruning with uniform sparsity among layers, AlphaPruning alleviates performance degeneration when the sparsity level is high and fine-tuning is not applied.\nAlphaPruning is composable with existing pruning methods.\nExperiments with various LLM models, such as Llama and Llama2, demonstrate the effectiveness.\n\nSTRENGTHS:\n- The presented method can be composed with other pruning methods.\n- The authors conducted experiments on various LLM architectures, such as Llama and Llama 2, with different baselines. In addition, they performed pruning of image classifiers, such as ConvNext and ViT.\n- Inference on CPU accelerates by the proposed pruning up to 3 times, depending on the sparsity.\n\nWEAKNESSES:\n- Although they say that \"AlphaPruning is theoretically driven\" in L83, I could not find any theoretical justification of the method in the text.\n- While \"Here, we provide a brief overview of HT-SR theory\" (L100), it is not described, making it challenging to understand the proposed method.\n- Although not an expert of this domain, I think using stronger baselines would make this work better.\nFor example, [Gale+19] demonstrated even simple magnitude pruning retains performance of ResNet-50 and Transformer even the sparsity level is set to 80%.\n\n[Gale+19] Trevor Gale, Erich Elsen, Sara Hooker \"The State of Sparsity in Deep Neural Networks\" arXiv 2019.\n\nQUESTIONS:\n- Why is $\\mu_{\\mathbf{x}_i}$ is introduced in Equation (1)? It is merely used in the text except for L125.\n- What is the actual advantage of unstructured pruning as AlphaPruning over other acceleration techniques, such as quantization?\n\nLIMITATIONS:\n- As written in Weaknesses, the baselines are quite limited.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Alpha Pruning, a novel approach for pruning large language models based on Heavy-Tailed Self-Regularization theory. Instead of applying a uniform pruning ratio across layers, Alpha Pruning utilizes PL_Alpha_Hill, derived from empirical spectral densities (ESDs), to assess how well-trained each layer is. It then assigns a lower pruning ratio to well-trained layers to preserve model performance. Alpha Pruning is evaluated across various LLM architectures and datasets, demonstrating superior performance compared to baseline uniform pruning and SOTA  methods like OWL.\n\nSTRENGTHS:\n- Alpha Pruning leverages HT-SR theory to provide a principled method for guiding layer-wise pruning decisions, contrasting with heuristic-based approaches\n- This pruning method has been evaluated on various large language models and exhibits robust performance, showcasing its effectiveness and generalizability.\n- By evaluating the importance of each layer and assigning non-uniform pruning ratios, Alpha Pruning can complement existing pruning techniques such as magnitude-based pruning and Wanda. It is also compatible with other model acceleration techniques, such as structured pruning and quantization.\n- Despite the challenges of unstructured pruning in achieving significant speedups compared to structured methods, Alpha Pruning still achieves noticeable efficiency gains with high pruning ratios\n\nWEAKNESSES:\n- The paper’s explanation, particularly concerning HT-SR theory and terms used in the method section, may be challenging to grasp for readers unfamiliar with these concepts. Figure 1. a could benefit from clearer explanations.\n- While Alpha Pruning is compared against uniform pruning and OWL across multiple architectures, other layer-wise pruning methods prevalent in the computer vision community [1] could provide additional comparative insights.\n\n\n[1] Lee, Jaeho, et al. \"Layer-adaptive sparsity for the magnitude-based pruning.\" arXiv preprint arXiv:2010.07611 (2020).\n\nQUESTIONS:\n- How does the PL_Alpha_Hill metric evolve during fine-tuning? Does its value typically decrease as a block becomes better trained?\n- What might be the reason why a block is better trained than others? Is it possible to include PL_Alpha_Hill in the training process to train each block equally and accelerate the training?\n- What factors contribute to the non-linear relationship between sparsity and speedups is shown in the Table. 4? Is the maintenance of early layers a significant factor in this observation?\n\nLIMITATIONS:\nThe limitation of this work is well discussed in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
The paper introduces AlphaPruning, a novel method for pruning large language models (LLMs) using Heavy-Tailed Self-Regularization (HT-SR) Theory. AlphaPruning uses ESDs of weight matrices to determine layerwise pruning ratios. The method demonstrates the ability to prune LLaMA-7B to 80% sparsity while maintaining reasonable perplexity.

STRENGTHS:
1. The proposed method, AlphaPruning, achieved SOTA performance across various tasks compared to OWL. 

2. The experiments conducted were comprehensive and diverse.

WEAKNESSES:
1. The motivation is not explicitly explained. It appears to apply a theoretical concept to the pruning area without providing quantitative or qualitative proof.

QUESTIONS:
1. Can you explain why the property of $W^T*W$ can decide the sparsity of the layer and can be combined with different pruning metrics?

2. To my knowledge, large language models (LLMs) are generally well-trained. The article concludes that PL_Alpha_Hill can indicate whether a layer is well-trained. Are there other methods or indicators that demonstrate if these layers are not well-trained?

3. Have you compared the sparsity allocation of different layers with OWL? If so, is the sparsity allocation similar to that of OWL?

LIMITATIONS:
This article includes a limitation.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-13