AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models
Authors: Haiquan Lu
Yefan Zhou
Shiwei Liu
Zhangyang Wang
Michael W. Mahoney
Yaoqing Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: Pruning
large language models
heavy tails
Primary area: deep_learning_architectures
TL;DR: We use methods derived from HT-SR Theory to develop improved methods for pruning LLMs.

Abstract: Recent work on pruning large language models (LLMs) has shown that one can eliminate a large number of parameters without compromising performance, making pruning a promising strategy to reduce LLM model size. Existing LLM pruning strategies typically assign uniform pruning ratios across layers, limiting overall pruning ability; and recent work on layerwise pruning of LLMs is often based on heuristics that can easily lead to suboptimal performance. In this paper, we leverage Heavy-Tailed Self-Regularization (HT-SR) Theory, in particular the shape of empirical spectral densities (ESDs) of weight matrices, to design improved layerwise pruning ratios for LLMs. Our analysis reveals a wide variability in how well-trained, and thus relatedly how prunable, different layers of an LLM are. Based on this, we propose AlphaPruning, which uses shape metrics to allocate layerwise sparsity ratios in a more theoretically-principled manner. AlphaPruning can be used in conjunction with multiple existing LLM pruning methods. Our empirical results show that AlphaPruning prunes LLaMA-7B to 80% sparsity while maintaining reasonable perplexity, marking a first in the literature on LLMs.

AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models
Haiquan Lu"; Yefan Zhou²* Shiwei Liu Zhangyang Wang', Michael W. Mahoney 5.6.7, Yaoqing Yang"
Nankai University. *Dartmouth College. UUniversity of Oxford
Uniiversity of Texas at Austin, "International Computer Science Institute
Lawrennc Berkeley National Laboratory, UUniversity of California Berkeley
Abstract
Recent work on pruning large language models (LLMs) has shown that one can eliminate large number of parameters without compromising performance, mak- ing pruning a promising strategy to reduce LLM model size. Existing LLM pruning strategies typically assign uniform pruning ratios across layers, limiting overall pruning ability: and recent work on layerwise pruning of LLMs often based on heuristics that can easily lead to suboptimal performance. In this paper, we Regulaaauaii (HT-SR) Theory, in particular the shape of empirical spectral densities (ESDs) of weight matrices, to design improved layerwise pruning ratios for LLMs. Our analysis reveals wide variability in how well-trained, and thus relatedly how prunable, different layers of an LLM are. Based this, we propose AlphaPruning which uses shape metrics allocate layerwise sparsity ratios in a more theoretically-priniillee manner. AlphaPruning can be used in conjunction with multiple existing LLM pruning methods. Our empirical results show that AlphaPruning prunes LLaMA-7B 80% sparsity while maintaining reasonable perplexity, marking a first in the literature on LLMs. ? We have open- open-sources sourced our code."
leverage Heavy-Tailed
Introduction
Recent work pruning large language models (LLMs) [Jaiswal et al., 2023, Frantar and Alistarh, 2023a, Sun al., 2023] has shown the ability reduce the number of parameters significantly, without compromising performance, resulting in notable savings in memory footprint, computing time, and energy consumption. Unlike Pre-LLM pruning methods [Sanh al., 2020, Kurtic al., 2022], existing LLM pruning approaches typically allocate the "sparsity budget" (i.e., the number of pruned parameters pruning ratios) uniformly across layers, making it difficult to increase sparsity to very high levels. Relatively little effort has been put into developing theoreticallyyprinciii to compute layerwise pruning ratios. For example, the Outlier Weighed Layerwise sparsity (OWL) method [Yin et al., 2023] uses a nonuniform layerwise sparsity based on the distribution of outlier activations. However, OWL relies on heuristics related to the presence of outliers [Kovaleva et al., 2021, Puccetti et al., 2022, Dettmers et al., 2022]. This can lead to suboptimal performance in the absence of outliers, and this can make it difficult to achieve very aggressive levels of sparsity. For example, Yin al. [2023] shows that pruning LLMs 80% sparsity often significantly degrades the prediction performance of LLMs.
ways
In developing principled approach to allocate sparsity budgets across layers, draw inspiration from Seaay--Tailes Self- Regulaazaaiiiiinn (HT-SR) Theory [Martin et al., 2021, Martin and Mahoney,
First two authors contributed equally. hhtpss//gituubccm/hhaaaaaaa
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

Weight matrices
ESDs
Metric values
Pruning ratios
LLM
More heavy-tailod
PL futree
"1
V1
Less heavy-tailed
V2
Pt. firiing
mapping
Hi
V3
ESD
Figure 1: The pipeline diagram of AlphaPruning. Our post-training layer-wise pruning method involves the following steps: (i) Performing ESD analysis on all weight matrices of a base LLM and (ii) employing PL fitting to derive the layer-wise metric values (that measures the HT exponent). Then, (iii) using the layer-wise metric values, we assign layer-wise pruning ratios to each layer through a linear assignment function.
2021a,b, 2019, 2017, 2020, Yang et al., 2023, Zhou et al., 2023, Qing et al., 2024]. HT-SR theory analyzes the weight matrices of models to derive quantities (related to the shape of the weight matrix eigenspectrum). which help characterize model capacity and quality. Applications of HT-SR to model selection [Martin and Mahoney, 2019, 2020, 2021a, Martin et al., 2021, Yang et al., 2023] and layer- wise adaptive training [Zhou et al., 2023] demonstrate the effectiveness of the theory in estimating model and layer quality. Furthermore, in the context of pruning using this kind of low-cost weight analysis is advantageous because it requires no data or gradient backpropagation.
LLMs.
Our study consists of two main parts. In the first part, we evaluate the effectiveness of various weight matrix-based metrics for allocating layer-wise sparsity. Our primary finding indicates that shape metrics generally outperform scale metrics in determining layer importance for pruning. Shape metrics capture the shape properties of the empirical spectral densities (ESDs) of layer weight matrices, whereas scale metrics, like matrix norms, reflect the size scale of ESDs. This offers novel perspective since shape metrics are less frequently used in the literature than scale metrics, which have been used to create regularizers [Yoshida and Miyato, 2017] and inform pruning [Han et al., 2015].
In the second part of the paper, we introduce a theoretically principled layer-wise sparsity allocation method, AlphaPruning, based on metrics that quantify unique heavy-tailed (HTed) shape of ESDs. According to HT-SR theory, well-trained models display strong correlations among the weight matrix elements, leading HT structures in layer weight matrices' ESDs [Martin and Mahoney, 2019, 2020, 2021a]. Moreover, layers with more pronounced HTT properties are typically better trained than others. We quantify the HT properties by fitting a power law (PL.) distribution [Alstott et al., 2014, Clauset et al., 2009] to ESD and using the PL exponent as the HT metric PL Alpha Hill. The principle of AlphaPruning is to allocate less sparsity to more well-trained (more HTed) layers, indicated by lower PL.Alpha_Hii Hill values, thereby preserving their quality during pruning. Figure I illustrates the pipeline of AlphaPruning.
We conducted comprehensive empirical evaluation to assess the generalizability of AlphaPruning in LLM pruning. This evaluation involved comparisons with various baseline methods, integration with existing techniques, and testing across different architectures. We also conducted sanity checks to ensure that AlphaPrunimg indeed reduces the variation PL.Alpha_Hii Hill values across layers. Our key contributions are summarized below.
This paper is the first study principled layer-wise sparsity allocation from the HT-SR perspective. We systematically evaluate multiple weight matrix-based metrics to compute sparsity based on their effectiveness in estimating layer quality, discovering interesting finding: shape metrics outperform scale metrics in allocating sparsity, despite the latter being more commonly used.
'Following Zhou et al. [2023]. we use a biased estimator, called PL.Alpha AllphaHaill, to replace the commonly used MLE estimator of the PL. exponent [Alstott et al., 2014, Clauset et al., 2009, Martin and Mahoney, 202 10] for allocating layer-wise sparsity ratios. This estimator proves to have a smaller variance and thus is easier to integrate with training models that require repeated estimation of the HT exponent.

---

We introduce novel sparsity allocation method, AlphaPruning, inspired by HT-SR theory, which demonstrates superior performance in LLM pruning. This method assigns sparsities based on the heavy-tailed shape of ESDs in layer weight matrices, previously unexplored concept. Our empirical evaluations span a range of L.LM architectures, including the L.LMMA V1-3 families [Tou- vron et al., 2023a,b], OPT families [Zhang et al., 2023], Vicuna-7B [Zheng et al., 2024], and Mistral-7B [Jiang al., 2023]. The results show that AlphaPruning outperforms OWL [Yin et al., 2023], the current SOTA non-uniform sparsity allocation method for LLMs, reducing perplexity by 304.31 and achieving an average accuracy gain of 4.6% over 7 zero-shot tasks 80% sparsity, while providing a 3.06x end-to-end speedup on CPUs for LLaMA-7B on the DeepSparse [Neural- Magic, 2021] inference engine. AlphaPruning also outperforms six layer- wise allocation methods, including global thresholding [Frankle and Carbin, 2018], ER [Mocanu et al.. 2018], ER-Plus [Liu et al., 2022a], LAMP [Lee et al., 2020], rank selection [Kuzmin et al., 2019, EI Halabi et al., 2022], and layer-wise error thresholding [Ye et al., 2020, Zhuang et al., 2018].
AlphaPruning provides layer-wise budget allocation (e.g.. sparsity), demonstrating remarkable generalizability and can be integrated with multiple LLM compression techniques enhance performance. This includes unstructured pruning (Wanda [Sun et al., 2023]. SparseGPT [Frantar and Alistarh, 2023b]). with or without fine-tuning [Hu et al., 2021], semi-structured (Domi- noSearch [Sun et al., 2021]), structured pruning (LLMPruner [Ma et al., 2023]. OSSCAR [Meng et al., 2024]), mixed- precision quantization [Tang et al., 2022]. Additionally, method to large Computer Vision (CV) architectures, such as Vision Transformers (ViT) [Dosovit- skiy et al., 2020], and ConvNext [Liu et al., 2022b]. In this case, OWL significantly underperforms Al AlphaPruning due to the lack of outlier features. This indicates HT metrics are more general than outlier metrics.
have extended this
AlphaPruning theoretically driven, and its improvements can be interpreted by HT-SR metrics. We demonstrate that model performance correlates with model-wise and layer-wise changes in AL. Hill before and after pruning. Furthermore, compared to baseline pruning methods, AlphaPruning not only improves model performance but also achieves a lower mean of layer-wise AL. Hill. HT-SR theory suggests that AlphaPruning preserves the quality of model layers "on average", minimizing the damage caused by pruning.
We provide an overview of related work in Appendix C.
2 Background and Setup
2.1 Notation
Consider a NN with L layers. W; is one of the weight matrices extracted from the i-th layer with shape m X n (m 2 n). We note that the "layer" used in this work refers to the transformer block (layer), and each block contains multiple weight matrices, such as the attention layer weight matrix, and projection layer weight matrix. The correlation matrix X; W W; is an n X n symmetric matrix, and the ESD of X; is formulated as
1 PX. = n
8x,(X,)
where X1 (X,) S ... S An (X,) are the eigenvalues of X, and 5 is the Dirac delta function. The ESD a probability measure, which can be viewed as a distribution of the eigenvalues of X,.
2.2 HT-SR theory and metries
Here, we provide brief overview of HT-SR theory. HR-SR theory originated as semi semi-empinee semi-empirical theory, with early seminal work [Martin and Mahoney, 2019, Martin et al., 2021] examining the empirical spectral density (ESD) of weight matrices, specifically the eigenspectrum of the correlation matrix X; W W. This research found that the structures of the ESDs strongly correlate with training quality. These findings are rooted in statistical physics and Random Matrix Theory (Couillet and Liao, 2022], detailed in Table of Martin and Mahoney [2019]. It is well-known [Wang et al., 2024, Couillet and Liao, 2022] that spikes in ESD represent "signals," while the bulk represents noise, which follows the Marchenko- Pastur law, In the theoretical setting of Wang et al. [2024], the signal or the spike aligns with ground -ruth features from the teacher model, and that corresponds to increased correlations in weight elements. Furthermore, Kothapalli et al. [2024] show that heavy tails in ESD
3

---

originate from the interaction between spikes and bulk, which can be quantified precisely using recent advances the free-pree preo-abbabllltyy theory [Landau et al., 2023], and the interaction characterizes the "bulk-decay" phase in the tve-plus-one phase model in Martin and Mahoney [2019]. a critical phase between classical "bulk+spike" model and heavy-tail models.
To quantify the structure of ESDs, HT-SR theory provides several metrics, collectively known as HT-- SR metrics. These metrics are typically categorized into two groups: scale metrics and shape metrics.
Scale metrics. Scale metries refer to those obtained from measuring various norms of weight matrices. As demonstrated empirically in Yang et al. [2023], these metrics are often strongly correlated with the generalization gap (which is the gap between training and test performance), instead of the quality of the models. In this paper, we mainly study two scale metrics, Frobeni us .Nrm and Spectral Normm The Frobenius _Nrm metric is calculated by the squared Frobenius norm of the weight matrix [W]][: and the Spectral metric can be calculated by the square of the spectral radius of the weight matrix [W12.
Shape metrics. Drawing analytic methods from Random Matrix Theory, HT-SR work analyzes poorly trained and well-trained models and concludes that the performance of these models usually correlates with shapes emerging in their ESDs, such as "bulk+spike" shape or "heavy-tailed" shape. The metrics used to characterize these ESD shapes are called shape metrics, and we mainly studied four of them: PL Alpha Hill, Alpha Hat, Stable Rank, and Entropy. PL Alpha Hill is the main metric used in our method, and we define it in Section 3.2. The definitions of other shape metries (including Alpha Hat, Stable Rank, Entropy) can be found in Appendix D.
3 Alpha-Pruning
In this section, we first outline the motivation behind AlphaPruning, followed by an introduction to the layer-wise importance metric, PL_Alpha_Hil _iill, and the sparsity allocation function, shown in Figure 1. Our empirical analysis reveals that shape metrics generally outperform scale metrics in guiding sparsity allocation with the same function. Notably, PL Alpha Hill, the shape metric used in AlphaPruning, achieves the best results preliminary evaluations.
3.1 Rationale
HT-SR theory, introduced in Section 2.2, examines the ESD of weight matrices and finds a strong cor- relation between heavy-tailed structures in the ESD and training quality. It suggests that heavy-tailed structures emerge from feature learning, where useful correlations are extracted during optimization. Layers with more heavy-tailed ESDs tend capture more signals, indicating better training quality. Inspired by these findings, we propose to assign sparsity based on the heavy-tailed properties of each layer's ESD. Layers with more heavy-tailed ESDs, which contain more learned signals, assigned lower sparsity, while layers with light- fight-tailed ESDs are assigned higher sparsity. In practice, the heavy-tailed structure is measured by fitting PI. distribution to the ESD, and extracting the PI. exponent a as the indicator. This is why our method named AlphaPruning.
3.2 Estimating layer quality by HT metric
AlphaPruning relies on estimating the layer quality based on the HT characteristic of the layer ESDs, which is quantified by HT metric PL.Alpha Hill. Given an ESD μx, of weight matrix's correlation matrix, we fit a PL density function p(A) on it, taking values within an interval (Amin- Amax), formally defined as:
p(A) x Amin
Amax-
The estimated exponent a is then used as metric to characterize the HT extent of the ESD, with lower value means more HTed. We estimate the PL coefficient using the Hill estimator [Hill, 1975, Xiao et al., 2023b, Zhou et al.. 2023], and we refer to it as the AllllpaaHiil metric. The Hill estimator defined as:
k
Al
(E
where {M.)%1 is sorted in ascending order, and k is a tunable parameter that adjusts the lower eigenvalue threshold Amin for (truncated) PL estimation. We adopt the Fix-finger method [Yang

---

et al., 2023] to select the k, which sets k such that Amin aligns with the peak of the ESD. Note that PL. Hill and other scale/shape metrics are calculated for each weight matrix individually.
3.3 Allocating sparsity based on the layer quality
AlphaPruning allocates sparsity for each layer (transformer block) by using mapping function
R - R to map sequence of layer quality measures q (91, (91.92.... 92. QT. into corresponding sparsities p(q).
9:
Quin
(q);
(32 - 81) +
(4)
Gmax
qmin
Here, Φ(q); represents the i-th element of the resulting vector ((q),, di represents the i-th element of the input vector q, and Qmin: Imax represent the minimum and maximum values of q. The normalization factor v) adjusts the sparsity levels to achieve the target global sparsity S. Each layer's sparsity is normalized within the interval [y81, 782). is calculated using the equation Σ ((q),d; =
S Σ di, in which d, the number of parameters of W,, Both sides of the equation represent the total number of remaining parameters. The (#1, 82) are tunable hyperparameters that adjust the non-uniformity of the sparsity distribution. We note that sparsity allocation is executed a per-block basis, averaging the HT-SR metric across all matrices within a block to determine qi- This design is supported by ablation study, presented in Appendix F, which shows that it yields superior performance over a per-matrix allocation. Hyperparameter settings for all experiments are provided in Appendix G.
3.4 Shape vs. scale metrics for sparsity allocation
In this study, we evaluated various HT-SR metrics, as defined Section 2.2, assess their effective- ness in estimating layer quality q for computing layer-wise sparsity. Our preliminary experiments involved pruning the LLaMA-7B model 70% sparsity and assessing its performance through WikiTeet perplexity and accuracy across seven zero-shot tasks, with results presented in Table 1. We applied each HT-SK metric in conjunction with three intra-layer pruning techniques (which only determine which matrix elements to prunc), such as Magnitude, Wanda, and SparseGPT, to Vision Transformers (ViT) described
thoroughly evaluate their efficacy. Further experiments in Appendix I.1. Across all tests, shape metrics consistently outperformed scale metrics in assigning layer-wise sparsities. This finding suggests that shape metrics are more robust and yield more reliable predictions of layer quality, which extends previous research [Yang et al., 2023, Zhou al., 2023, Martin et al., 2021] on estimating model quality. Notably, the shape metric PL. Alpha Hill that focuses on estimating the HT shape, proved to be the most effective. Consequently, we have adopted PL Alpal Hill as the primary metric in our proposed method, AlphaPruning.
Metrie used for layerwise pruning ratios
Perplexity on WikiTeex (L)
Average accuracy on zero-shot tasks (1) Magnitude
Magnitude
Wanda
SparseGPT
Wanda
SparseGPT
Uniform
48419.13
85.77
26.30
32.30
36.73
41.52
Frobenius Norm
30136.37
59.82
24.95
33.23
37.62
43.67
Spectral.Nort
48073.99
246.84
29.01
32.81
33.15
41.14
Entropy
3716.07
41.15
22.02
33.58
39.39
43.53
Stable_Rand
851.65
41.24
24.91
34.91
39.74
42.97
Alpha Hat 1256.02
27.60
20.02
33.48
43.57
43.83
PL Alpha Hill 231.01
23.86
18.54
35.67
44.42
45.48
Table 1: Evaluating shape metries versus scale metries on allocating layerwise sparsities on LLMs. Shape metrics are obtained from the shapes of the ESDs. Scale metrics are norm-based metrics measuring the scale of weights matrices (which can also be obtained from the ESD). The results are conducted on LLaMA-7B at 70% sparsity. We show WikiText validation perplexity and average accuracy on seven different zero-shot tasks as evaluation metrics. We observe that shape metrics outperform scale metries and PL Alpha Hill performs the best.
5

---

4 Empirical results
In this section, we evaluate the performance, generalizability, and interpretability of AlphaPruning. Section 4.1 outlines our experimental setup. In Section 4.2, we evaluate AlphaPruning' perfor- mance by comparing it to the SOTA method OWL and five other baseline methods, and we analyze the efficiency of LLMs pruned by AlphaPruning using practical metrics such as FLOPs and la- tency. Section 4.3 evaluates the generalizability of AlphaPruning by integrating it with various LLM compression techniques including post-pruning fine-tuning, struttrru pruning, structured pruning, and mixed-
mixed-precistee precision quantization. Additionally, we extend its application CV tasks. Section 4.4 provides an analysis of the layer-wise sparsities and the PL.Alpha_Hil Hill distribution to further elucidate the effectiveness and implications of Al AlphaPruning
4.1 Experimental setup
Models and Evaluation. We evaluate AlphaPruning on the three most widely adopted LLM model families: LLaMA 7B/13B/30B/65B [Touvron et al., 2023a], LLaMA-2 7B/13B/70B [Touvron al., 2023b], OPT /25M/350M/2.7B/6.. 7B, and other advanced LLMs: LLaMA-3-8B, Vicuna-7B, Mistral-7B. Our evaluation protocol aligns with established methodologies for LLM pruning [Xiao et al., 2023a], including assessments of language modeling proficiency and zero-shot capabilities. Specifically, we evaluate the perplexity on the held-out WikiText [Merity et al., 2016] validation set, and use seven tasks, including BoolQ [Clark al., 20191, RTE [Wang et al., 2018). HellaSwag [Zellers et al., 2019], WinoGrande [Sakaguchi et al., 2021], J, ARC Easy and Challenge [Clark et al., 2018] and OpenbookQA [Mihaylov et al., 2018] for downstream zero-shot evaluation [Gao et al., 2023].
Baselines. We apply the layer-wise sparsities determined by AlphaPruning to three LLM pruning methods, including Magnitude [Han et al., 2015], SparseGPT [Frantar and Alistarh, 2023b] and Wanda [Sun et al., 2023]. Magnitude-
pruning is a simple and strong baseline in which weights are discarded based on their magnitudes. Wanda and SparseGPT are two strong LLM pruning baselines due to their capability to sustain reasonable performance even relatively high sparsity levels (around 50%). All these methods originally used uniform layerwise sparsity. We incorporate AlphaPruning directly into these baselines, and demonstrate that this results in improved performance. Besides, we also compare A.phaPruning with OWL [Yin et al., 2023], a recently proposed non-uniform LLM pruning method and six layer- wise pruming methods, including global thresholding [Frankle and Carbin, 2018], ER [Mocanu et al., 2018J, -Pus [Liu et al., 2022a]. LAMP [Lee et al., 2020], rank selection [Kuzmin et al., 2019, El Halabi et al., 2022], and layer- wise error thresholding Yee et al., 2020, Zhuang et al., 2018].
4.2 Main results
Language Modeling. In Table 2, we report the perplexity of the pruned LLaMA and LLaMA-2 models at 70% sparsity. We provide results for more sparsity levels in Figure and Appendix H. AlphaPruning, as a general layerwise sparsity method, consistently demonstrates performance improvements when used in conjunction with various pruning methods. For example, in the case of I.LaMA-7B with a sparsity of 70%, AlphaPruming produces sparse networks with a perplexity of 23.001, significantly outperforming the Magnitude- based pruning bascline of 48419.13. Notably, when applied to Wanda and SparseGPT, two robust LLM pruning methods, AlphaPruning still achieves substantial perplexity reductions, evidenced by a decrease of 61.91 for Wanda and 7.76 for SparseGPT, in the case of LLLMA- with a sparsity of 70%.
Zero-shot tasks. We conducted empirical evaluations to determine the zero-shot ability of pruned LLMs diverse zero-shot downstream tasks with prompting. The results are shown in Table 3, where we show the mean zero-shot accuracy on zero-shot tasks of pruned LLaMA and LLaMA-2 models at sparsity of 70%. AlphaPruning consistently improves accuracy across all settings. For example, AlphaPruning achieves an average accuracy gain of 8.79, 6.05, and 2.61 over 7 tasks and 7 models compared to Magnitude, Wanda, and SparseGPT alone, respectively. These results highlight the promise of AlphaPruning for more challenging zero-shot downstream tasks.
More baseline comparison. For allocating layerwise sparsity ratios, we compare AlphaPruning with other allocation methods. The experiments involve pruming the LLaMA-7B model and LLaMA- 13B to various sparsities. We use Wanda as the basic pruning method, with results presented in Figure 2 and Figure 3. Results indicate that AlphaPruning significantly outperforms all baseline

---

LLaMA
LLaMA-2
Method
Layerwise sparsity
13B
30B
65B
7B
13B
70B
Dense model
5.68
5.09
4.77
3.56
5.12
4.57
3.12
Uniform
48419.13
84527.45
977.76
46.91
49911.45
214.04
1481.95
Magnitude
OWL
19527.58 58
11464.69
242.57
15.16
59176.45 42
57.55
17.18
2029.20
16.01
8900.32
15.27
Uniform
85.77
$4.03
17.35
15.17
74.26
45.36
10.56
Wanda
OWL
24.57
17.17
10.75
8.61
30.38
20.70
8.52
Ours
23.86
14.21
9.68
7.86
28.87
14.16
7.83
Uniform
26.30
18.85
12.95
10.
27.65
19.77
9.28
SparseGPT
OWL
19.49
14.55
10.28
8.28
20.40
15.27
7.65
Ours
18.54
12.81
9.77
7.83
19.34
12,20
7.57
Table 2: WikiText validation perplexity for pruned LLaMA and LLaMA-2 models at 70% sparsity.
Our method (AlhhPruning) is compared uniform layerwise sparsity and OWL, each paired with magnitude- based pruning, Wanda, and SparseGPT. Lower perplexity indicates improved model performance.
LLaMA
LLaMA-2
Method
Layerwise sparsity
7B
13B
30B
65B
7B
13B
70B
Dense model
60.08
62.59
65.47
67.05
59.71
63.05
67.06
Uniform
32.30
34.95
32.39
45.04
33.38
33.56
43.65
Magnitude
OWL
33.57
36.86
33.88
53.42
34.17
36.98
50.49
Ours
35.67
38.23
42.56
55.22
37.86
44.53
50.72
Uniform
36.73
38.90
51.07
54,90
34.66
37.16
56.72
Wanda
OWL
43.41
45.91
52.38
57.34
40.46
45.04
57.19
Ours
44.42
47.48
54.48
58.93
41.98
47.21
$7.63
Uniform
41.52
45.67
52.75
57.81
41.84
45.17
59.68
SparseGPT
OWL
44.65
47.61
53.16
58.25
44.96
47.94
59.20
Ours
45.48
48.41
54.07
59.72
44.68
49.26
60.23
Table 3: Comparison of mean zero-shot accuracies (%) for pruned LLaMA and LLaMA-2 models at 70% sparsity. We evaluate our method (AlphaPruning) against uniform layerwise sparsity and OWI, each integrated with magnitude-based pruning, Wanda, and SparseGPT. Higher accuracy values indicate better zero-shot ability.
methods in relatively high-sparsity regimes. For LLaMA-7B and LLaMA-13B pruned to 80% sparsity, AlphaPruning reduces perplexity by 304.31 and 200.54 compared to OWL, respectively. Achieving high levels of sparsity is crucial for unstructured sparsity to yield significant speedups by leveraging existing sparse kernels. Sparse kernels such as Flash-LLM [Xia et al., 2023] and Sputnik [Gale et al., 2020] have shown that unstructured sparsity outperforms dense computation in terms performance, but only when sparsity levels reach 60% and 71%, respectively. The importance of achieving high sparsity is further substantiated in the following section, which demonstrates that high sparsity levels facilitate significant end-to-end inference speedup. Additional comparisons with rank selection and layer-wise error thresholding, as well as results using SparseGPT as the pruning method and low sparsity results, can be found Appendices H.
GPUs
Uniforn
ER
OWL
AlphaPruming (Ours)
50
900
300
Accuracy (%%
Perplexity
Perplexity
500
200
40
100
100
65
75
65
Sparsity (%)
Sparsily (%)
Spansity (%)
Figure 2: The WikiText validation perplexity for LLaMA-7B
Figure 3: Mean accuracy of
(left) and LLaMA-13B (right) pruned with different sparsities
7 zero-shot tasks for LLaMA- 7B pruned with different spar- sities using Wanda.
using Wanda.

---

Efficiency measure. To verify the sparse LLM pruned by our method can indeed achieve speedups when deployed on the CPU, we provide new results in Table 4. We apply our method Llama2- 7B-Chat-hf, prune it to different sparsities, and then test its end-to-end decode latency using DeepSparse [Kurtic al., 2023] inference engine
Intel Xeon Gold 6126 CPU with 24 cores. The results indicate that when the global sparsity reaches 80%, the speedup reaches 3.06x.
In Appendix 1.2, we evaluate the pruned L.M by efficiency metrics other than sparsity, such as FLOPs. Compared with uniform sparsity ratios, our approach is able to achieve better performance-Fl.n trade-off. In Appendix 1.3, we show that AlphaPruning can control the minimum layer sparsity without losing the performance advantage to meet the hardware requirements of a memory- limited device. In Appendix I.4, we report the runtime of AlphaPruning to show that the computational overhead is reasonable. The computational complexity is not large because the most computation- intensive aspect of our method involves performing SVD decomposition on weight matrices.
Sparsity
Dense
10%
20%
30%
40%
50%
60%
70%
80%
90%
Lateney (ms)
307.46
306.27
304.87
293.26
264.41
177.55
148.16
133.76
100.35
81.40
Throughput (tokens/sec)
3.25
3.26
3.28
3.41
3.78
5.63
6.74
7.47
9.96
12,28
Speodup
1.00x
1.00x
1.01x
1.05x
1.16x
1.73x
2.07x
2.30x
3.06x
3.78x
Table 4: End-to-ent decode latency and speedup of AlphaPruning measured on the DeepSparse inference engine.
4.3
Corroborating results
To demonstrate the generalizability of AlphaPruning, we first evaluate the performance of the model pruned by AlphaPruning can be well recovered by fine-tuning. Then we apply AlphaPruning to other LLM compression techniques (semi-structured, structured pruning, and quantization), and CV models.
Method
Sparsity
Samples
Perplexity (1)
Accuracy (t)
Dense
5.68
60.08
Method
Model
60%
70%
80%
Uniform
50%
20K
6.86
54,85
Uniform
LLaMA V3 7B
23.35
126.57
880.57
Ours
50%
20K
6.79
55.58
Ours
LLaMA- -3-7B
18.91
101.73
560.22
Uniform
Vicuna-FE -Bu
12.64
110.05
6667.68
Uniform
60%
20K
8.32
51.81
Vicuna
35.87
1081.76
Ours
60%
20K
8.23
53.39
Uniform
Mistral-7B 7B
11.47
60.08
353.62
Uniform
70%
30K
11.21
49.00
Ours
Mistral-BB
10.49
39.39
201.62
Ours
70%
30K
10.95
49.51
Uniform
100K
20.11
39.32
Table 5: WikiText validation perplexity (L) of more LLMs pruned by uniform sparsity and method combined with Wanda.
Ours
80%
100K
18.53
40.02
Table 6: Wiki'Tett validation perplexity and zero- shot tasks accuracy of SparseGPT pruned LLaMA- 7B at various sparsities after LoRA fine-tuning on C4 dataset samples.
Fine-tuning. We show the performance of LLMS pruned by AlphaPruning can be well recovered by fine-tuning. We investigate the parameter-efficient strategies for fine-tuning LLMss: LoRA fine- tuning [Hu et al., 2021]. Fine-tuning is conducted on the C4 training set [Raffel et al., 2020] with the pre-training auto-regressive loss. The pruned mask is fixed during fine-tuning. The low-rank (r = 8) adapter is applied to the query and value projection matrices in the attention layers. We fine-tune LLaMA-7B pruned by SparseGPT at various sparsities. Table 6 summarizes the results for perplexity and mean zero-shot accuracy after fine-tuning pruned LLaMA-7B models. We can see the performance of pruned LLMs can be notably improved with very light LoRA fine-tuning. In Appendix I.8, we further compare A AlphaPruning with baselines. The results show that the advantages of AlphaPruning don't diminish after fine-tuning.
More LLM architectures. To demonstrate that the effectiveness of AlphaPruning is robust across various more advanced LLMs, we also apply AlphaPruning to LLaMA-3-7BB Vicuna-7B [Zheng al., 2024], and Mistral-7B [Jiang et al., 2023]. The results in Table 5 show that, a general method, AlphaPruning can consistently achieve performance improvement across different architectures. Results for OPT families can be found in Appendix I.5.

---

Integration with other compression techniques. To demonstrate the generalizability of our non-uniform layerwise sparsity, we integrated AlphaPruning with three prominent L.MM compression methods: N:M spar-
80
Accuracy (%)
60
sity, structured pruning, and quantization. In Ap-
pendix 1.6, we examine a mixed N:8 sparsity setup ing DominoSearch [Sun et al., 2021] as well as combine AlphaPruning with structured pruning methods LLM- Pruner [Ma et al., 2023] and OSSCAR [Meng et al., 2024]. In Appendix 1.7, we merge our method with mixed- precision quantization as [Tang al., 2022]. Across these configurations, AlphaPruning consistently enhances the performance of baseline methods.
40
Uniform
20
OWT
Ours
80
Sparsity (%)
Figure 4: ImageNet-I K accuracy (1) of the sparse ConvNext model pruned to various sparsity levels by AlphaPruning and other baseline methods, without fine- tuning.
Vision models. To illustrate AlphaPruning in broader context, we study how it performs against other methods for determining layerwise sparsity on CV tasks, where non-uniform layerwise sparsity has been widely used. We consider two modern CV architectures:
ViT [Dosovitskiy et al., 2020] and ConvNext [Liu et al., 2022b]. We adopt Wanda as the pruning approach and compare AlphaPruning with uniform layerwise sparsity and OWL. We present the results ConvNext in Figure 4 and provide more results on DeiT and ViT models in Appendix 1,9. These results affirm that AlphaPruning effectively allocates layerwise sparsities in CV tasks as well, where OWL significantly underperforms AlphaPruning due to the lack of outlier features. This indicates HT metrics are more general than outlier metrics.
4.4 Analyzing LLM pruning via HT-SR perspective
We study how the PL PLlAlpha Hill metric as measure of model quality (based on HT-SR theory) changes before and after pruning. In particular, we show that the proposed method, AlphaPruni SirhaPruning, effectively controls the damage of pruning to model quality, resulting in more favorable distribution (lower mean) of PL Alph Hill among the model layers compared to the baseline pruning method.
Analyzing PL. Alpha Hill affected by pruning. We investigate how Alpah Hill values change before and after pruning. According to HT-SR Theory [Yang et al., 2023, Martin et al., 202 I], models or layers of higher quality typically exhibit lower PLLAlpha_Hill values. As observed in Figure 5a, dense LLaMA models (depicted in gray) consistently show a lower mean AL.Alpha_HHil Hill, with larger models demonstrating lead to increments of the metric value, as pruning is often seen as damaging the quality of the model, the changes are well correlated with the perplexity. More interestingly, AlphaPruning not only outperforms the Uniform pruning perplexity, but it also leads to a smaller mean of AL.Alpha Hill. Figure 5b delves deeper into this phenomenon by visualizing the metric value in layer-wise manner. It is noticeable that AlphaPruning leads to lower PL Alpha Hill than the uniform pruning among the first several blocks. This is due to the mechanism (Figure 8) by which our method prunes the model based on the layer-wise PL Alpha Hill, and prunes less on these more heavy-taile layers.
even lower mean. Furthermore, we can see both pruning methods
Based on these results, we can conclude that: (1) pruning a model a larger sparsity generally hurts the model's i task performance (e.g., perplexity), and this coincides with decreased model quality, as measured by PL.AAlhha _ill; and (2) better pruning method such AlphaPruning can obtain and this is achieved by pruning
a sparse model with a smaller mean of layer-wise PL AL_Alpha_Hil less aggressively in the dense layers with lower Alpha Hill layers.
Here we provide more analytical results in Appendix We show analyses on other LLMs provided in Appendix 1. We compare our HT-SR metrics with other layer quality metrics in Appendix E.2. Additionally, in Appendix E.3, we investigate the distribution of layerwise sparsities allocated by the heavy-tailed metric PI_AlphaHHill In Appendix E.4, we also examine the apparent connection between our method, Al AlphaPruning. and Low-Rank Approximation (LRA) [Zhang al., 2015, Wen et al., 2017, Xu et al., 2019, Barsbey et al., 2021] from two perspectives. First, we study the relation- ship between the ESD used in method and the low-rank properties often used in LRA, where use PL PL_Alpha.. Hill to measure HT and Stable_ Rank to mcasure low-rank properties. Second, we study the differing layer-wise assignment strategies adopted by the two methods. Finally, discuss

---

Dense
Uniform
AlphaPruning
3.6
LLaMA-7B
LLaMA-13B
PL. Alpha Hill
3.4
20
Perplexity
Pl. Alpha Hill
Pl. Alpha Hill
3.2
10
3.0
20 Block Index
10
30
20 Block Index
40
LLaMA-7B LLaMA-13B LLaMA-300 -08
Model-wise heavy-tail analysis
(b) Layer-wise heavy-tail analysis
Figure 5: Analyzing the heavy-tail metric PL_Alpha Hill (lower the better by HT-SR theory) and performance metric WikiText validation perplexity (lower the better) before and after pruning by baseline uniform pruning and Al AlphaPruning (a) The metric value is reported by averaging over all layers within each model. The dashed lines represent the perplexity and the histograms represent the Al.alpha Hill value. (b) The metric is reported by averaging all the matrices within each LLM layer.
how our findings relate to those of Barsbey al. [2021], paper closely related ours, showing that the results from both studies complement each other, offering different yet compatible insights.
Conclusion
We have used methods from HT-SR Theory develop improved methods for pruning LLMs. The basic idea is to analyze the ESDs of trained weight matrices and to use shape metrics from these ESDs to measure how much to prune given layer, with less well-trained layers, measured by these shape metrics from HT-SR Theory, being pruned more aggressively. Our extensive empirical evaluation demonstrates that AlphaPruning offers straightforward yet effective way of determining the layer-wise sparsity ratios. Our analysis reveals that different layers of an LLM are not equally trained (typically, the ESDs of early layers are more HT and thus are more well-trained, compared to later layers), and that shape- based ESD metrics work better for layer quality prediction in pruning pipelines than scale-based ESD metrics. AlphaPruning achieves higher sparsity, without severely hurting performance, and also smaller values of PL_ApphaHHa Hill after pruning. AlphaPruning is also compatible with multiple existing LLM pruning methods and is expected to be integrated with future ones, long the methods allow specifying layerwise ratios.
gekmowledgements.
We want to thank Alex Zhao, Elicie Ye, Zhuang Liu, Xiangyu Yue, and Tianyu Pang for their helpful discussions. Michacl W. Mahoney would like to acknowledge the UC Berkeley CLTC, ARO, IARPA (contract W91 INF20C0035), NSF, and ONR for providing partial support of this work. Yaoging Yang would like to acknowledge support from DOE under Award Number DE-SC0025584, DARPA under Agreement number HR001 12490441, and Dartmouth College. Our conclusions do not necessarily reflect the position or the policy of our sponsors, and no official endorsement should be inferred.
10

---

References
Jeff Alstott, Ed Bullmore, and Dietmar Plenz. powerlaw: a python package for analysis of heavy- distributions. PloS one, 9(1 x:85777,, 2014.
Melih Barsbey, Milad Sefidgaran, Murat A Erdogdu, Gael Richard, and Umut Simsekli. Heavy tails in sgd and compressibility of overparametrized neural networks. Advances in neural information processing systems, 01-229444-97788 2021.
Srinadh Bhojanapalli, Ayan Chakrabarti, Andreas Veit, Michal Lukasik, Himanshu Jain, Frederick Liu, Yin- Wen Chang, and Sanjiv Kumar. Leveraging redundancy attention with reuse transformers. Technical Report Preprint: arXiv:2110.06821, 2021.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of Machine Learning and Systems, 2:129-146, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in Neural Information Processing Systems, 33:1877- 2020.
Christopher Clark, Kenton Lee, Ming-Wei Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. Technical Report
Preprint: arXiv:1905. 10044, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. Technical Report Preprint: arXiv:1803.05453 2018.
Aaron Clauset, Cosma Rohilla Shalizi, and Mark EJ Newman. Power-law distributions in empirical data. SIAM review, 51(4):661-700 2009.
Romain Couillet and Zhenyu Liao. Random Matrix Methods for Machine Learning. Cambridge University Press, 2022.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 Q: 8-bit matrix multiplication for transformers scale. Technical Report Preprint: arXiv:2208.00 077339, 2022.
Zhen Dong. Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed- precision. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 293-302, 2019.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. Technical Report Preprint: arXiv:2010. 1929, 2020.
Marwa El Halabi, Suraj Srinivas, and Simon Lacoste-Julien. Data-efficient structured pruning via submodular optimization. Advances in Neural Information Processing Systems, 35:3661--36626, 2022.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning (ICML), pages 2943-2952, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. Technical Report Preprint: arXiv: Stti:18803003355 2018.
Elias Frantar and Dan Alistarh. Massive language models can be accurately pruned one-shot. Technical Report Preprint: arXiv:2301.00771 2023a.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323-10337, 2023b.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. Technical Report Preprint: arXiv:1902.099 1902.0954. 2019.

---

Trevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. In SC20: International Conference for High Performance Computing. Networking. Storage and Analysis, pages 1-14. IEEE, 2020.
Leo Gao. Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoft, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang. and Andy Zou. framework for few-shot language model evaluation, 12 2023.
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403. 17887 2024.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, pages 1135-1143, 2015.
Bruce M Hill. A simple general approach to inference about the tail of a distribution. The annals of statistics, pages 1163-1174, 1975.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. Technical Report Preprint: arXiv:2106.0966 09685, 2021.
Ajay Jaiswal, Shiwei Liu, Tianlong Chen, and Zhangyang Wang. The emergence of essential sparsity in large pre-trained models: The weights that matter. Technical Report Preprint: arXiv:2306.03805, 2023.
Steven A Janowsky. Pruning versus clipping in neural networks. Physical Review A, (91)))6600, 1989.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. (Preprint: arXiv:2310.06825, 2023.
Schoon Kim, Coleman Hooper, Amir Gholami. Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-spaa quantization. Technical Report Preprint: arXiv:2306.07629 2023.
Vignesh Kothapalli, Tianyu Pang, Shenyang Deng, Zongmin Liu, and Yaoqing Yang. Crafting heavy-tails in weight matrix spectrum without gradient noise. arXiv preprint arXiv:2406.04657, 2024.
Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. Bert busters: Outlier dimensions that disrupt transformers. Technical Report Preprint: MxXiv:2105.0699 2021.
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. Technical Report Preprint: arXiv:2203.07259, 2022.
Eldar Kurtic, Denis Kuznedeley, Elias Frantar, Michael Goin, and Dan Alistarh. Sparse fine-tuning for inference acceleration of large language models, 2023. URL https://arxiv. org/ abs/ 3310.069211
Andrey Kuzmin, Markus Nagel, Saurabh Pitre, Sandeep Pendyam, Tijmen Blankevoort, and Max Welling. Taxonomy and evaluation of structured compression of convolutional neural networks. arXiv preprint 19Xv:1912.00800 2019.
Itamar Landau, Gabriel C Mel, and Surya Ganguli. Singular vectors of sums of rectangular random matrices and optimal estimation of high-rank signals: The extensive spike model. Physical Review E, 108(5)0054129 2023.
Yann LeCun. John S Denker, and Sara A Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, pages 598-605, 1990.
12

---

Jacho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. the magnitude-basee pruning. Technical Report Preprint arXiv:2010.07611, 2020.
sparsity for
Mingbao Lin, Rongrong Ji, Yuxin Zhang, Baochang Zhang, Yongjian Wu, and Yonghong Tian. Channel pruning via automatic structure search. Technical Report Preprint: arXiv:2001.088 108565, 2020.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and Mykola Pechenizkiy. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. Technical Report Preprint: 220x:2202..0643, 2022a.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1 11976-11986, 2022b.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Technical Report Preprint: arXiv:2305. 11627, 2023.
Charles H Martin and Michael W Mahoney. Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior. Technical Report Preprint: arXiv:1710.09553, 2017.
Charles H Martin and Michael W Mahoney. Traditional and heavy tailed self regularization in neural network models. In International Conference Machine Learning, 2019.
Charles H Martin and Michael W Mahoney. Heavy-tailed universality predicts trends in test accuracies for very large pre-trained deep neural networks. SIAM International Conference on Data Mining, 2020.
Charles H Martin and Michael W Mahoney. Post-mortem on deep learning contest: a Simp- son's paradox and the complementary roles of scale metrics versus shape metrics. (Preprint: arXiv:2106.007344 2021a.
Charles H Martin and Michael W Mahoney. Implicit self-regularizzttion in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165):1-73, 2021b.
Charles H Martin, Tongsu Serena Peng, and Michael W Mahoney. Predicting trends in the quality of state-of-the-an neural networks without access training or testing data, Nature Communications, 12(1):1-13, 2021.
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprini arXiv:2403.0388. 2024.
Xiang Meng, Shibal Ibrahim, Kayhan Behdin, Hussein Hazimeh, Natalia Ponomareva, and Rahul Mazumder. Osscar: One-shot structured pruning in vision and language models with combinatorial optimization. arXiv preprint arXiv:2403.12983 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. Technical Report Preprint: arXiv:1609.0784. 2016.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor con- duct electricity? a new dataset for open book question answering. Technical Report Preprint: arXiv:1809.02789 2018.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibeseu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1):2383, 2018.
Michael C Mozer and Paul Smolensky. Skeletonization. A technique for trimming the fat from network via relevance assessment. Advances in Neural Information Processing Systems, 1, 1988.
NeuralMagic. Neuralmagic deepsparse inference engine. 2021. URL https://github. com/ neuraliagic/deepppare..
13

---

Giovanni Puccetti, Anna Rogers, Aleksandr Drozd, and Felice Orletta.. Outliers dimensions that disrupt transformers are driven by frequency. Technical Report Preprint: arXiv:2205.1 1380, 2022.
Peijun Qing. Chongyang Gao, Yefan Zhou, Xingjian Diao, Yaoqing Yang, and Vosoughi Soroush. Alphaexpert: Assigning lora experts based on layer training quality, In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing. 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2 (1 (1):5585-555)) 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Victor Sanh, Thomas Wolf, and Alexander M Rush. Movement pruning: Adaptive sparsity by fine-tuning. Technical Report Preprint: 2xiv:2005500663. 2020.
Sheng Shen, Zhen Dong. Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8815-8821, 2020.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. simple and effective pruning approach for large language models. Technical Report Preprint: arXiv:2306. 1695, 2023.
Wei Sun, Aojun Zhou, Sander Stuijk, Rob Wijnhoven, Andrew O Nelson, Henk Corporaal, et al. Dominosearch: Find layer-wise fine-grained n: M sparse schemes from dense neural networks. Advances in Neural Information Processing Systems. 34:20721-20732 2021.
Chen Tang, Kai Ouyang, Zhi Wang, Yifei Zhu, Wen Ji, Yaowei Wang, and Wenwu Zhu. Mixed- precision neural network quantization via learned layer-wise importance. In European Conference on Computer Vision, pages 259-275. Springer, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. Technical Report Preprint: arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Technical Report Preprint: arXiv:2307.0922
2023b.
Alex Wang, Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. Technical Report Preprint: arXiv: 1..iv:1804000000 804.07461, 2018.
Wenxuan Wang and Zhaopeng Tu. Rethinking the value of transformer components. Technical Report Preprint: arXiv:201 1.020011.03303.. 2020.
Zhichao Wang, Andrew Engel, Anand D Sarwate, Ioana Dumitriu, and Tony Chiang. Spectral evolu- tion and invariance in tmeerrwiddte neural networks. Advances in Neural Information Processing Systems, 36, 2024.
Wei Wen, Cong Xu, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Coordinating filters for faster deep neural networks. In Proceedings of the IEEE international conference on computer vision, pages 658-666 2017.
Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang, Zhongzhu Zhou, Xiafei Qiu, Yong Li, Wei Lin, and Shuaiwen Leon Song. Flash-llm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity. arXiv preprint arXiv:2309. 10285, 2023.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099, 2023a.
Xuanzhe Xiao, Zeng Li, Chuanlong Xie, and Fengwei Zhou. Hcavy-tailed regularization of weight matrices in deep neural networks. Technical Report Preprint: arXiv:2304.02911 2023b.
14

---

Yuhui Xu, Yuxi Li, Shuai Zhang. Wei Wen, Botao Wang, Wenrui Dai, Yingyong Qi, Yiran Chen, Weiyao Lin, and Hongkai Xiong. Trained rank pruning for efficient deep neural networks. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing Computing-NeurIPS -NuurIPS Edition (EMC2-NIPS), pages 14-17. IEEE, 2019.
Yaoqing Yang, Ryan Theisen, Liam Hodgkinson, Joseph E Gonzalez, Kannan Ramchandran, Charles H Martin, and Michael W Mahoney. 'est accuracy vs. generalization gap: Model selection in ulp without accessing training or testing data. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3011-3021, 2023.
Mao Ye, Chengyue Gong, Lizhen Nie, Denny Zhou, Adam Klivans, and Qiang Liu. Good subnetworks provably exist: Pruning via greedy forward selection. In International Conference Machine Learning, pages 10820-10830. PMLR, 2020.
Lu Yin, You Wu, Zhenyu Zhang, Cheng- Yu Hsich, Yaqing Wang, Yiling Jia, Mykola Pechenizkiy, Yi Liang, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (owl); A missing secret sauce for pruning Ilms to high sparsity. Technical Report Preprint: 10.00123 2023.
Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning. arXiv preprint arXiv: 1705. 10941, 2017.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? Technical Report Preprint: 1eXiv:1905.078333 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models, 2022. (Preprint: arXiv:2205. 0006600........... 2023.
Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. IEEE transactions on pattern analysis and machine intelligence, 38(10):1433-195,, 2015.
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging Ilm lim-as-at as-ge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Yefan Zhou, Tianyu Pang, Keqin Liu, Charles H. Martin, Michael W. Mahoney, and Yaoqing Yang. Temperature balancing, layer-wise weight analysis, and neural network training. In Advances in Neural Information Processing Systems, 2023.
Michael Zhu and Suyog Gupta. To prune, or not prune: exploring the efficacy of pruning for model compression. Technical Report Preprint: arXiv: 171.01710 2017.
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discrimination-aware channel pruning for deep neural networks. Advances in neural information processing systems, 31, 2018
15

---

Appendix
A Impact Statements
This paper leverages HT-SR Theory to design improved layerwise pruning ratios for LLMs. Although the proposed method could be applied compress models with adverse applications, we do not see any immediate negative societal impacts stemming from the algorithm itself. Indeed, we see lot of societal value proposing our method to the community. Through the implementation of effective layerwise sparsity, we can achieve substantial reductions in the parameters of LLMs while retaining their functionality. Consequently, this advancement facilitates the deployment of LLMs in resource-constraingg devices, accelerates the predictions for resource-limited LLM services, and contributes to the sustainability of LLM technologies.
B Limitation
This paper's 's empirical evaluation focuses post-training pruning methods without fine-tuning, in alignment with recent research in LLM pruning. such as OWL, Wanda, and SparseGPT. This is due to the substantial computational resources required to restore heavily pruned LLMS to their original performance. Nonetheless, our experiments with limited fine-tuning have demonstrated that the proposed pruning method can achieve only a mild performance drop while yielding significant efficiency improvements.
C Related work
Pruning. Removing weights or connections in trained neural network (NN) generate an efficient, compressed model has a long history [LeCun et al., 1990, Mozer and Smolensky, 1988, Janowsky, 1989, Mocanu al., 2018]. Modern NNs are frequently over-pavannn pee-paaraeterizee [Wang and Tu, 2020, Bhojanapalli et al., 2021], and thus removing redundancies improves computation and memory efficiency. A common approach is out connections with weights smaller than a specified threshold. However, when comes to pruning LLMs [Brown et al., 2020, Touvron et al., 2023a], progress has been limited. Conventional pruning typically requires a round of retraining to restore performance [Blalock et al., 2020], which can be challenging for LLMs. To address the difficulty in retraining, researchers have developed specially tailored pruning algorithms for LLMs. For example, Ma et al. [2023] explored sparse structured LLMs, using Taylor pruning to remove entire weight rows, followed by LoRA fine-tuning, More recent research has shifted towards unstructured pruning without the need for fine-tuning. showing substantial advancements. In particular, SparseGPT [Frantar and Alistarh, 2023b] uses the Hessian inverse for pruning and subsequent weight updates to reduce the reconstruction error of dense and sparse weights, while Wanda [Sun et al., 2023] uses a criterion that incorporates weight magnitudes and input activations, in order to preserve outlier features [Kovaleva et al., 2021, Puccetti et al., 2022, Dettmers et al., 2022]. Our work allocates parameters in a more theoretically-prinn principled manner, enabling pruning LLMs to higher sparsity levels.
magnitude- pruning [Han et al., 2015], which zeros
Layerwise sparsity budgets. Although layerwise sparsity has been widely studied in pre-LLM pruning [Mocanu et al., 2018, Evci et al., 2020, Liu al., 2022a, Gale et al., 2019, Lee et al., 2020], relatively little attention has been devoted to determining the pruning ratios for each layer in LLMs. (Interestingly, this layer-wise approach has been applied to model quantization [Kim et al., 2023, Shen et al., 2020, Dong et al., 2019].) Frantar and Alistarh [2023b] and Sun et al. [2023] apply a uniform pruning ratio across all layers, and Yin et al. [2023] computes the sparsity budgets using the outlier ratio observed within each layer's token feature distribution. Existing work on sparsity budgets has generally used heuristics, such as different forms of size or scale metrics (such as norm- based metrics), determine sparsity budgets per layer. For instance, ABCPruner [Lin et al., 2020] reduces the number of combinations of layer sparsities to search over, but it still requires training to determine the empirical validity of its suggested layer sparsities. Lee al. [2020] modifies Magnitude- pruning by rescaling the importance scores in a layer by a factor dependent on the magnitude of surviving connections in that layer. However, these methods are suboptimal in allocating layerwise sparsities for pruming LLMs. Recently, Outlier Weighed Layerwise sparsity (OWL) [Yin et al., 2023] designs a nonuniform layerwise sparsity based on the distribution of outlier activations in LLMs.
16

---

However, OWL heuristically relies on the emergence of outliers, and this can lead to suboptimal performance when outliers are absent from models. Our work uses HT-SR shape metrics such as PL.Alpha Hill to predict layer importance. and it allocates parameters in more theoretically principled manner, allowing pruning L.MMs to higher sparsity levels than has ever been achieved before.
D Definitions of HT-SR metrics
Here, we define the shape metrics, beyond PL Alpha Hill, that we use in our analysis.
(Alpha Hat) The Alpha Hat metric [Martin al., 2021] has been shown be effective predicting generalization. It the variant of Pl. exponent a (PL Alpha) weighted by the log maximum eigenvalue log Amm* A"m"(log spectral norm):
(5) (Stable (ankk) The Stable Rank metric norm-adjusted measure of the scale of the ESD, and previous work [Yang et al., 2023] has shown its strong correlation with PL Alpha. For a weight matrix W, can be calculated as:
Hat A >
Stable Rank
|| |
(6)
(Entropy) For weight matrix W, the Entropy metric is defined
-1
Entropy log R(W)
Σ Ps log Pc.
where p; of/ oz. vs is the i-th singular value of W, and R(W) refers the rank of
W.
E
Further analysis of results
E.1 Post-pruning layer-wise heavy-tail analysis
We investigate layer-wise PL Alpha Hill values after pruning by Uniform pruning and AlphaPruning more advanced LLMs (LLaMA-V3-8B, Vicuna-7B, Mistral-7B), According HT-SR Theory, models layers of higher quality typically exhibit lower AL.Alpha Hill values. As observed in Figure 6, AlphaPruning leads to smaller layer-wise PL PL_Alpha_HHl Hill This is due to the mechanism (1) by which our method prunes the model based on the layer-wise PL Alpha Hill, and prunes less on these more heavy-tailed layers.
Uniform
AlphaPruning
0.6-
0.3
0.4
Density
Density
0.4
Densily
0.2
0.2
0.2
0.1
0.0
0.0
2
2.5
5.0
7.5
PL Alpha Hill
PL PL_Alpha_ Hiill
PL_Alpha _lill
(a) LLaMA- V3-8B
(b) Vicuna-7B
(c) Mistral-7B
Figure 6: Analyzing the layer-wise heavy-tail metric PL_Alpha_Hir Hill (lower the better by HT-SR theory) after pruning by baseline uniform pruning and AlphaPruning.
E.2 Comparison with other LLM layer quality metries
In addition to Al AlphPrunn proposed in our work, Gromov et al. [2024], Men et al. [2024] are other studies that investigated methods that measure whether layer is well-trained or not, demonstrating
17

---

PL Alpha Hill Hill
Outlier
Uniform sparsities
LLaMA-Ve -I--BB
LLaMA-VI-13B
LLaMA-VI -----VI-33BB
0.9
0.9
0.9
0.8
Sparsity
Sparsity
Sparsity
0.8
0.8-
0.7
0.7
0.7-
0.6
0.6
10
20
30
10
20
30
40
10
20
30
40
50
60
Layer (block) Index
Layer (block) Index
Layer (block) Index
Figure 7: Comparing layer-wise sparsity distributions allocated by AlphaPruning (blue, ours) and OWL (orange). While both methods show similar overall trends, AlphaPruning generates more granular distribution with distinct differences between consecutive layers.
LLMs layers are not equally well-trained. Gromov et al. [2024] developed a method that assesses the similarity between the representations different layers, defined as the angular distance between feature vectors. They found that deeper layers are more similar to their neighboring layers than shallow layers, suggesting that LLMs may not fully the parameters in these deeper layers, indicating these layers are not well-trained. Similarly, Men et [2024] introduced a metric called Block Influence, which measures the impact of each transformer block on hidden states to gauge layer significance. Their findings showed varying degrees of ineffectiveneessreddudaggg across layers, suggesting that these layers are not well-trained.
Besides, in Figure 7, compare the sparsity allocation of AlphaPruning with OWL. We show that the general trends of sparsity distribution generated by the two methods are similar, with lower sparsities allocated earlier layers and higher sparsities allocated to deeper layers. However, method produces a more granular distribution with clearer distinctions between consecutive deep layers, resulting in improved pruning performance.
PL Aipha Hill
4.5
0.8
Sparsity
AlphaPruning
3.5
Unifomm
0.4
2.5
10
20
30
Layer Index
Figure 8: Comparing layerwise sparsities AlphaPruning and uniform sparsities, at 80% global sparsity LLaMA-7B, The curves represent the layerwise sparsities, which are determined by PL Alpha Hill values shown by the histograms.
E.3 Analyzing the layerwise sparsity distribution.
We elaborate on the layerwise sparsities assigned by AlphaPruning and heavy-tailed metric Al.hap Hil1. The metric and sparsities results of LLaMA-7B are presented in Figure 8. We can observe that, the blue bar histograms demonstrate that different layers of one model show diverse PL [ipha_Hill values, this indicates that these layers are not equally well-trained. This conclusion is based on prior research on heavy-tails in weight matrices [Martin et al., 2021, Martin and Ma- honey, 2019], this metric measures the heavy-tailed structure within the correlation matrix. This measurement indicates the amount of correlation among the weight matrix elements, with strong correlations leading to more heavy-tailed empirical spectral density. Such structure is often as a result [Wang et al., 2024] of extracting various useful correlations (or features) from data during optimization.
18

---

Consequently, smaller PL Alpha Hill layers (more heavy-tailed) contain greater number of learned correlations. The larger PL.Alpha Hill layers (less heavy-tailed) tend to retain fewer learned correlations, remaining closer to the random initialization state. Figure 8 shows that AlphaPruning suggests that large PL_Alpha_Hte Hi layers with fewer learned correlations should be allocated with larger sparsity, or being pruned more, while those small PL Alpha Hill layers should be allocated with lower sparsity.
E,4 Connections with Low-rank Approximation.
commonly used compression technique that also involves analyzing the eigenspectrum of the weights is Low-Rank Approximation (LRA) [Zhang et al., 2015, Wen et al., 2017, Xu et al., 2019, Barsbey et al., 2021]. We examine the apparent connection between our method, Al AlphaPruning and LRA from two perspectives. First, we study the relationship between the ESD used in our method and the low-rank properties often used in LRA, where HT and Stable Rank to measure low-rank properties. Second, we study the differing layer-wise assignment strategies adopted by the two methods. Finally, discuss how our findings relate to those of Barsbey et al. [2021], a paper closely related to ours, showing that the results from both studies complement each other, offering different yet compatible insights.
PL Alpha Hill to measure
20
300
sndtis_Ai Lix 1680
2000
2600
Alpha 15il
4.0
Stable
300
Compuess Uniikem
Stablerunk bank
200-
4lor-5800
Stalle rank
Perploxity
1.5
3.5
200
100
10
0-
100
10
20
30
10
15
Tail indice
Layer index
Reduction ratio
(a) Stable Rank of ESD sam-(b) Layerwise metric values of (c) Comparing layer wise assign- pled from Pareto distribution LLaMA-7B
ment strategies LRA
Figure 9: Analyzing ESD properties and assignment strategies for LRA. (a) Stable Rank and PL. Al. hil.lpha.Hlll1.........111....11 show similar pattern (the more heavy-tailed, the more low-ranked) across different ESDs sampled from Pareto distribution. (b) The layer-wise AL_Alpha_Hii Hill and Stable Rank of the LLaMA-7B model exhibit a similar trend. (c) Comparing two assignment strategies for LRA, and "Compress more on HTTed layers" is better. This finding is opposite to pruning-based methods, which find that "Compress less HTed layers" is better.
For the first point, find strong positive correlation between the HTed properties and the low-rank properties used in LRA: as the ESD becomes more HTed, also becomes more low low-ranked. To demonstrate this, sampled eigenvalues from IID Pareto distribution with varying tail indices to form ESDs and then measured Stable_Rt R_nnk on these ESDs. lower PL_Alpha_Hill pill indicates a more HTTed distribution, while lower Stable Stanke_Rank indicates more low-ranked structure. Figure 9a shows that Stable Rank and PL Alpha Hill are positively correlated across different matrix sizes, suggesting a relationship between H'Ted and low-ranked properties. In Figure 9b, we further verify this by measuring PL Alpha Hill and Stable Rank across different layers of the pre-trained LLaMA-7B model. The results reveal that both metrics follow similar trend: shallow layers show lower values, while deeper layers exhibit higher values, indicating consistent behavior across layers. Therefore, we can see that the key metrics used in AlphaPruning and LRA are highly correlated.
For the second point, we clarify that the strategies for assigning higher or lower compression to HTed or low-ranked layers differ between AlphaPruning and LRA. AlphaPruning assigns lower compression ratios (less sparsity) to HTed layers. In contrast, LRA assigns higher compression to low-ranked layers [Zhang et al., 2015, Wen et al., 2017, Xu al., 2019], which, established earlier, tend to be HTed. Therefore, the two methods seemingly use different procedures in assigning layerwise compression ratios. In Figure 9c, we empirically verify that, for LRA, assigning higher compression to HTed layers is indeed more beneficial than its opposite for LLMs, as shown by the results of applying both to LLaMA-7B models. This finding is opposite to the assignment method used in AiphaPruning. We hypothesize that these differences arise from the distinct mechanisms and principles of each method. In more detail, AlphaPruning, which is pruning- based,
19

---

removes elements from the weight matrices, affecting the entire eigenspectrum. LRA, however, removes only the smallest eigenvalues, leaving the larger eigenvalues intact. The guiding principle of AlphaPruning is also different from LRA. It aims to make the model more deterministic and less random by preserving weights corresponding well-trained layers. It does this by preserving HTedd layers that contain more signals and removing light-tailed layers that, according to HT-SR theory, resemble randomly initialized weight matrices. In some sense, it is similar how decision trees choose branches to reduce entropy. LRA, on the other hand, focuses on applying more compression on low-rank matrices where the largest eigenvalues dominate. This allows for minimal impact reconstruction loss when removing small eigenvalues.
Lastly, Barsbey et al. [2021] concluded that models with lower HT measures are generally more compressible than other models, focusing examines compressibility within single model and suggests that, within a model, layers with lower HT values are less compressible than other layers. Our insight supports layer-wis, wise pruning strategies, such as AlphaPruning, which applies less compression to more HTeed layers.
cross-model comparisons. In contrast, work
1.00
1.00
Relutive test acc
0.95
Relative test
0.95
0.90
0.90
Alpta-2.00
2bp
Alsta-1.98
Alpta-1.9
0.85
Alpta 1.9
0.85
Abata 1
0.9
0.3
0.6
0.9
Pruning ratio
Pruning ratio
(a) Uniform Pruning
(b) AlphaPruning
Comparison of pruning models with varying model-wise HT measures (Alpha). Models with higher HT measures are more prunable using both Uniform pruning and AlphaPruning. Relative accuracy is calculated as the post-pruning accuracy divided by the pre-pruning accuracy. The experiments used FCNs trained on CIFAR-10.
Figure 10:
Compress lees en HTed layer
Compress more on Hildd layer
Unifform
Tea aecuracy
aiguracy
Text socuracy
accuracy
$5.0
55.0
$5.0
55.0
52.5
52.5
52.5
62.5
50.0
50.0
0.6
Pruning ratio
Pruning ratio
Pruning ratio
Pruning ratio
(a) Alpha=2.00
(b) Alphaa==11.
(c) Alpha=1.96
(d) Allha=11119
Figure 11: Comparison of our proposed layer-wise assignment strategy with other strategies on models with varying model-wise HT measures. "Compress less on HTeed layer" (AlphaPruning' consistently outperforms Uniform pruning across different models, while "Compress more on HTeed layer" leads worse performance compared to Uniform pruning. The experiments used FCNs trained on CIFAR-10.
We adopt the experimental setup of Barsbey et al. [2021], training fully connected networks (FCNs) with six hidden layers on the CIFAR-10 dataset. The HT measure used in their study referred to as Alpha, quantifies the HTed structure of the weight parameters. The model-wise HT measure is influenced by the "temperature" defined as the ratio of learning rate batch size. Higher temperatures yield models with lower HT measures. Our first experiment reproduces the finding of Barsbey et al. [2021], comparing the compressibility of models with varying model-wise HT measures. The results verified that models with lower HT metrics are more compressible. also tries to verify if the conclusions hold when using our proposed layer-wiss strategies. The results shown in Figure 10 support our hypotheses. The second experiment compares the effectiveness these layer-wise strategies and also checks if the findings are consistent across models with different model-wise HT
20

---

measures. The results shown in Figure 11 confirmed that HTed layers are less compressible and that pruning HTed layers less (which AlphaPruning does) is more effective.
F Ablation study of sparsity allocation
In Section 3.3, we introduced the range hyperparameters and to control the non-uniformity of layer-wise sparsities. To simplify, we define such that $1 8,82 + AlphaPruning allocates sparsity on a "per-block" basis, where all matrices within a block receive the same sparsity, determined by averaging the Ala Hill values across matrices within that block. Alternatively, sparsity can be allocated on a "per-matrix" basis, allowing different sparsities for individual matrices based their PL Alpha Hill values. The ablation study comparing per-mate per-matrix and per-block choices is presented in F.1. The ablation study on comparing different mapping functions is shown in F.2.
per-block
per-matrix
Sparsity 10%
S Spansity 20%
Sparsity 30%
Spunsity 40%
peplesses peplesily
5,7000
poplexity
5,815
haaplexity
6.005
porplexity
6,4
5,6975
5,810
6.000
5,6950
5.805
0.5
0.5
0.2
0.2
hyperparameter T
hyperpanmeteer
hyperparameter T
hyperparameter T
Sparsity 50%
Spunsity 60%
Sparsity 70%
Sparsity 80%
7.250
10.5
perpletity
80
rempexiit
6000
7.225
60-
4000
7.200
40
2000
7.175
9.5
0.2
0.2
0.2
0.4
0.2
hyperperameter
hyperparameter
hyperparameter
hyperparaaecerr
Figure 12: Ablation study sparsity allocation function hyperparameter We use Wanda to
prune LLaMA model using AlphaPruning with both per- per-matrix and per-block allocation methods.
F.1 Per-matrix per-block
In Figure 12, we compared per-matrix and per-block sparsity allocation across different values of the hyperparameter т, using Wanda. The results show regime transition pruning effectiveness between the two methods. For sparsity levels below 50%, the per-matrix approach (gray lines) achieves lower perplexity, indicating better performance. However, for sparsity values of 50% and above, the per-block method (blue line) performs better. We further focused on high sparsity 70%, as higher sparsity is more important to provide efficiency improvements. Specifically, evaluated both per-matrix and per-block methods in combination with various intra-layer pruning techniques. Table 7 shows that the per-block method consistently outperforms the per-matrix method when used in conjunction with three intra-layer pruning techniques.
Additionally, we analyzed the differences between MLP and Attention matrices by visualizing average PL.Alpha_Hi Hill values for seven types of weight matrices in LLaMA-7B, shown in Figure 13. The results indicate that query and key matrices have lower PL_Alpha_Hiil values, suggesting they developed new allocation strategy called "Mixed".
less prunable. Based this insight, which combines per-block and per- per-matrix approaches. As defined, Mixed first assigns a block-wird pruning ratio using the per-block method, then refines it within each block using per- per-matrix allocation. Table 7 demonstrates that this Mixed approach provides further marginal improvements over both per-matrix and per-block methods.
21

---

PL. Alpha Thill different type of layers
Meavisul of Alpha Hill
Method
Uniform
Per-matrix
Per-block
Mixed
4-
Magnitude 48419.13
7384.13
231.01
177.86
Wanda
85.77
58.71
23.86
22.46
3
SparseGPT 26.30
25.10
18.54
18.32
2
Table 7: Comparing perplexity of sparse LLaMA- 7B (sparsity=70%) pruned by four types of sparsity allocation method.
query
key
value
output
gate
up
down
Layer type
Figure 13: Averaged PL.Alpha _ill metric values for seven types of weight matrices in LLaMA-7B.
F.2 Different mapping functions
We provide ablation studies on the choices of sparsity assignment function. We implemented the logarithmic method and compared it with the linear mapping function used in our current approach, shown in Figure 14. The results show that both methods perform similarly when combined with Wanda, but linear mapping slightly outperforms the proposed logarithmic method when combined with SparseGPT.
Logacillmic
Logarithmic
700
Lincar
Läncar
Perplexity
500
Perplexity
300
50-
100
70
65
70
80
Sparsity (%)
Spursity (%)
(a) Wanda
(b) SparseGPT
Figure 14: Comparing different mapping fuunction. Linear (ours) refers to the linear mapping function that is used in our current method. Logarithmic refers to computing the logarithmic of metrics before linear mapping. The model LLaMA-VI-7B. (a-b) refers to combining different intra-layer pruning methods.
G Hypterparamter setting
Here, provide the values of used in the experiments, shown in Table 8.
22

---

Method
Model
Sparsity
Magnitude
Wanda
SparseGPT
Model
40%
50%
60%
70%
80%
LLaMA-7B
0.3
0.2
0.2
LLaMA- 13B
0.5
0.3
0.3
VIT-B
0.3
0.3
0.3
ILaMA-30R
0.4
0.3
0.3
VIT-L
0.3
0.3
0.3
LLaMA 65B
0.4
0.3
0.3
DeiT-S
0,4
0.4
LLAMA2-7B
0.5
0.3
0.2
DeiT-L
0.3
0.3
LLaMA2-13B
0.4
0,3
0,3
ConvNext
0.1 0.1
0.2
0.2
0.2
LLaMA2-70B
0.4
0.3
Table 8: Left: Hyperparameters setting for results in Section 4.2. We report the optimal À after
small hyperparameter sweep within the range of E [0.2, 0.3, 0.4, 0.5]. Right: Hyperparameters setting for results in Section 4.3 for Vision Transformers 40%, 50%, 60% sparsity. We report the optimal À after a small hyperparameter sweep within the range E [0.1, 0.2, 0.3, 0.4, 0.5].
H
More baseline comparison
For assigning layerwise sparsity ratios, we compare AlphaPruning with other methods. In this section, we provide definitions and details of these methods:
Uniform. [Zhu and Gupta, 2017] Every layer pruned with the same target sparsity.
Global [Frankle and Carbin, 2018]. A global threshold uniformly applied to all layers to satisfy the overall sparsity requirement. The specific layerwise sparsity is automatically adjusted based on this threshold.
ER [Mocanu al., 2018]. The sparsity of the convolutional layer is scaled proportionally to '44 1 - ix where h refers to the number of neurons/channels in layer l.
ER-Plus [Liu et al., 2022a]. ER-Plus modifies ER by forcing the last layer as dense it is not,
while keeping the overall parameter count the same.
LAMP [Lee et al., 2020]. This method modifies Magnitude- based pruning by rescaling the importance scores in layer by factor dependent on the magnitude of surviving connections in that layer.
OWL [Yin et al., 2023]. A non-uniform layerwise sparsity based on the distribution of outlier activations in LLMs, probing the possibility of pruning LLMs to high sparsity levels.
We adopt Wanda and SparseGPT as the pruning approach. The results are presented in Table and Table 10, which indicate that AlphaPruning significantly outperforms all baseline methods in relatively high- sparssyy regimes. Besides, we have conducted additional experiments using the "layerwise error thresholding" method, where each layer is pruned sequentially specified in [Zhuang al., 2018, Ye al., 2020]. We have also implemented the rank selection method as specified in Section 5.2 of [EI Halabi et al., 2022]. We present the updated experimental results in Table 1.. We observe that our method outperforms all the other baselines.
Method/Perplexity (.)
10%
20%
30%
40%
50%
60%
70%
80%
Global
14.11
3134
10293
10762
14848
17765
5147
39918.56
LAMP
5.69
5.98
6.3912
7.57
12.86
185.52
15647.87
LAMP (per-block)
5.70
5.82
6.00
6.40
7.25
10.95
98.77
7318.01
ER
5.70
5.81
6.03
6.57
7.80
12.41
119.66
6263.79
ER-Plus
5.70
6.05
8.00
14.04
229.17
6013.91
Uniform
5.70
5.82
6.00
6.39
7.26
10.63
84.52
5889.13
OWL
5.70
5,80
6.01
6.39
7.22
9.35
24.56
1002.87
Ours
5,69
5.81
6.00
6.37
7.18
9.47
23.86
698.56
Table WikiText validation perplexity (L) LLaMA-7B pruned by different allocation methods at various global sparsities using Wanda. AlphaPruning outperforms other layerwise sparsity at high sparsity range.
23

---

Method/Perplexity (1)
10%
20%
30%
40%
50%
60%
70%
80%
LAMP
5.69
5.78
5.96
6.34
7.37
11.27
31.96
274.73
LAMP (per-block)
5.70
5.80
5.96
6.33
7.22
10.45
27.05
224.32
HR
5.70
5.81
6.02
6.49
7.54
11.29
30.20
258.63
Unilorm
5.69
5.89
5.96
6.32
7.22
10.56
26.30
188.11
OWL
5.71
5.81
5.97
6.35
7.22
9.51
19.49
84,94
Ours
5.69
5.81
5.99
6.36
7.30
9.73
18.44
81.98
Table 10: Perplexity (.) of pruning LLaMA-7B into various global sparsities using SparseGPT. We compare our method with three other baseline sparsity allocation methods.
Method
Global sparsity
Perplexity ())
Zero-shot Accuracy (f)
Uniform
70%
26.30
41.52
Layerwise error thresholding
70%
32.54
41.24
Rank selection
70%
21.34
42.90
Ours
70%
18.54
45.48
Table 11: Comparing our method to other layerwise sparsity baseline methods in pruning LLaMA-7B into 70% sparsity. The perplexity evaluated on the WikiText validation set. The zero-shot accuracy is averaged over downstream tasks. Each method is combined with SparseGPT.
I Complementary Experimental Results
I.1 Shape metrics versus scale metrics on Vision Transformers
We further evaluate different metrics for computing layerwise sparsity on Vision Transformers. Shape metrics, including Alpha_Ha Hat, Entropy, Aa.Appha.Hill. and Stable Rank, are obtained from the shapes of the ESDs. Scale metrics, including Frobenius _erm Norm and Spectral_Nowe norm-based metrics measuring the scale of weights matrices (which can also be obtained from the ESD). The results shown in Table 12 align with the results in LLMS that shape metrics outperform scale metrics on allocating layerwise sparsity and PL. PL.Alpha Hill performs the best.
Metric used for layerwise pruning ratios
ViT-L 16/224 accuracy (t)
DeiT-S 16/224 accuracy (t)
40%
50%
60%
40%
50%
60%
Uniform
76.05
68.73
39.45
75.62
68.98
50.49
Frobenius Norm
76.15
68.93
37.46
76.62
72.21
58.93
Spectral.Norm
75.99
67.72
32.98
75.97
70.13
53.76
Entropy
76.37
71.17
49.09
76.77
71.63
58.23
Stable Rank
75.91
68.93
41.26
75.63
68.64
46.86
Alpha_Har Hat
77.11
72.01
52.56
76.94
72.10
59.78
PL Alphpp Hill
76.86
72.12
55.62
77.07
72.38
60.92
Table 12: Evaluating shape metries versus scale metrics on allocating layerwise sparsities on Vision Transformers. Shape metrics are obtained from the shapes of the ESDs. Scale metrics are norm-based metrics measuring the scale of weights matrices (which can also be obtained from the ESD). We choose two models, ViT-L and DeiT-S, without fine-tuning We observe that shape metrics outperform scale metrics and PL Alpha Hill performs the best.
results are shown on ImageNet- IK accuracy
I.2 More results on other efficiency metrics
To further demonstrate the benefits of our approach, provide results in other practical efficiency metrics such as FLOPs. Compared with uniform sparsity ratios, our approach able to achieve better performance-FLee FLOPs trade-off. We have provided new results of FLOPs in Figure 15.
Table 13 summarizes the results from Figure 15, We show that, compared to uniform pruning, our method can achieve significant FLOPs reduction when pruned LLMs are compared at similar perplexity.
24

---

LLaMA-7B
LLaMA-7B
150
12
125
Unilorm Pruning
Uniform Pruming
Perplexity
100
*
AlphaPruning (ours)
Perplexity
11-
AlphaPruning: (ours)
OWL
10
75
9-
*
8-
400
500
600
700
650
700
750
800
850
FLOPs
FLOPPs (G)
(a) Broader perplexity range
(b) Lower perplexity range
Figure 15: Additional results of FLOPs measurement the LLaMA-7B pruned by Uniform and our method with SparseGPT.
Uniform
Perplexity (L)
18.58
21.06
25.70
35.57
61.76
95.82
FLOPSs (L)
716.80
583.49
566.82
533.49
500.16
450.17
416.84
Ours
Perplexity (U)
9.78
15.67
18.63
23.39
35.74
94
87.81
FLOPs (U) 700.14
566.82
533.49
500.16
450.17
416.84
366.85
Table 13: Additional results of FLOPs measurement on the LLaMA-7B pruned by Uniform and our method with SparseGPT.
1.3 Controlling the minimum layer sparsity for memory-limited hardware
To enhance the adaptability of our method to hardware, we demonstrate that our method allows for controlling the minimum sparsity, by adjusting the values of 81 and 82. Additionally, Table 14 demonstrates that increasing the minimum sparsity by changing $1 and $2 doesn't diminish the advantages of our method, compared to the uniform pruning baseline.
Recall that the sparsity of the layer of the model can be determined by Eqn. 4. Increasing s1 leads to a higher minimum sparsity 1781, while maintaining the same global sparsity S. Table 14 displays the results of adjusting $1 and 82 to increase the minimum sparsity while maintaining the global sparsity 70%. At a minimum sparsity of 57%, our method achieves the lowest perplexity. Even when the minimum sparsity is raised to 67%, nearing the uniform pruning baseline, our method still attains perplexity of 49.6, which is 36.17 points lower than that of uniform pruning (85.77).
Method Global sparsity 5
Minimum sparsity
Perplexity
Uniform
70%
70%
85.77
Ours
70%
50%
33.46
Ours
70%
55%
26.19
Ours
70%
57%
23.79
Ours
70%
60%
27.18
Ours
70%
65%
40.22
Ours
70%
67%
49.60
Table 14: Increasing the minimum sparsity of our method, while maintaining a global sparsity of 70%, still yields performance improvements compared to a uniform sparsity ratio. We present the WikiText validation perplexity for LLaMA-7B pruned by both the Uniform method and method, in conjunction with Wanda.
L.4 Computational complexity of AlphaPruming
The computational complexity of AlphaPruning' is large because the most computation-intensiii aspect of our method involves performing SVD decomposition on weight matrices, which can be further optimized through parallel processing. Table 15 presents the runtime of Al AlphaPruning and an optimized version that uses parallel processing. The increase runtime is reasonable, at 32.4 22.44% with Wanda and 8.2% with SparseGPT. These experiments were conducted pruming LLaMA-7B to 70% sparsity. The testing platform used A40 GPUs and an AMD EPYC 7713 64-Core CPU.
25

---

Method
Ours
Original method (sccond)
Percentage of runtime increase (%)
Wanda
176
0%
Wanda w. Ours (1 GPU)
462
178
259.55%
Wanda w. Ours GPUs)
178
32.44%
SparseGPT
699
0%
SparseGPT w. Ours GPU)
462
695
66,47%
SparseGPT us. Ours (8 GPUs)
695
8.2%
Table 15: Runtimes of our method combined with Wanda and SparseGPT on LLaMA-7B.
1,5
OPT family
In addition LLaMA and LLaMA-2, conduct experiments with OPT [Zhang et al., 2023). Table 16 shows the results of comparing our method to uniform sparsity when both are combined with magnitude pruning. Table 17 shows the same comparison when both combined with Wanda [Sun et al., 2023] /SparseGPT [Frantar and Alistarh, 2023b]. In most of the cases, our method outperforms the baseline method and achieves lower perplexity.
Method
Sparsity
OPT-125M
OPT-350M
OPT-2.7B
OPT-6.7B
magnitude
40%
54.60
40.19
30.31
31.89
magnitude w. Ours
40%
52.29
38.20
22.31
19.53
magnitude
50%
193.35
97.79
265.20
968.72
magnitude w. Ours
50%
173.01
95.61
159.44
224.91
Table 16: The perplexity of OPT models pruned by uniform sparsity and our method combined with magnitude pruning. The perplexity is evaluated on WikiText validation set.
Method
Sparsity
OPT-125M
OPT-350M
OPT-2.78
OPT-6.7B
Wanda
70%
334.58
758.81
265.20
158.38
Wanda w. Ours
70%
269.80
654.17
159.44
40.81
SparseGPT
70%
226.30
146.45
26.95
20.38
SparseGPT w. Ours
70%
207.83
136.90
27.40
20.31
Table 17: The perplexity of OPT models pruned by uniform sparsity and our method combined with Wanda and SparseGPT. The perplexity is evaluated on WikiText validation set.
I.6 More results on semi-structured and structured pruning
To assess the potential of our non-uniform layerwise sparsity for hardware- feedwar--rriedly applications, we investigate AlphaPruning across two distinet serdwarr--rrindlyy pruning regimes: N:M sparsity and structured pruning. Following DominoSearch [Sun et al., 2021]. we study the mixed N:8 sparsity configuration. Instead of using uniform N value across all layers, we allow individual layers to possess distinct N values while maintaining the same parameter count. We adopt AlphaPruning to determine the optimal value of N for individual layers. The results shown in Table 18 demonstrate that AlphaPruning consistently outperforms the basclines.
Method
Model
Layerwise Sparsity
4:8
3:8
2:8
Wanda LLaMA-7B
Uniform Mixed
8.57
42.56
2962.00
Ours LLaMA-7B
8.55
22.77
585.81
Table 18: WikiText validation perplexity of pruned LLaMA-7B in Mixed N:8 sparsity configuration. The results are shown with Wanda and our non-uniform layerwise sparsity. Ours can lead to performance improvement at various sparsity levels.
Furthermore, instead pruning weights, we follow the recent methodology introduced in LLM Pruner [Ma et al., 2023], wherein entire neurons and attention heads are removed. This action facilitates the direct acceleration of pruned LLMs on GPUs or TPUs. We replace the uniform layerwise sparsity used by the LLM pruner with non-uniform layerwise sparsity using AlphaPruning. The
26

---

results, shown in Table 19, demonstrate that AlphaPruning can improve model performance at various sparsity levels.
Datasct
Pruning Method
Layerwise Sparsity
20%
40%
60%
80%
WikiText
LLM Pruner
Uniform
16.95
30.38
90.02
1228.17
WikiText
LLM Pruner
Ours
16.78
29.11
71.21
952.77
PTB
LLM Pruner
Uniform
29.51
66.90
192.06
1691.87
PTB
LLM Pruner
Ours
29.11
56.99
144.97
1002.40
Table 19:
Applying AlphaPruning to structured pruning method LLM-Pruner. The results are shown in WikiText validation perplexity of pruned LLaMA-7B at various sparsity levels.
Another structured pruning method in LLM is OSSCAR [Meng et al., 2024], which formulates the structured pruning problem quadratic program with combinatorial constraints. We integrated AlphaPruning with OSSCAR, and we provide the results in Figure 16. OSSCAR prunes only the linear sublayer multi-head attention and the second sublayer of the feed-forward network. applying uniform pruning across each transformer block. By incorporating AlphaPruning 's layer- wise sparsity allocation, achieved non-uniform Mlck-wiis pruning ratios while keeping the global pruning ratio the same. The results show that integrating AlphaPruning Piniiinn with OSSCAR can reduce perplexity at different sparsities.
1000
OSSCAR
Porplexity
500
0
0.5
0.6 Pruning ratio
0.7
Using AlphaPruning to determine layerwise sparsity for OSSCAR. The æ-axis pruning ratio represents the fraction of pruned parameters relative the total parameters the linear sublayer of multi-head attention and the second sublayer of the bed--oowwrdd network, any other type of sublayers not included. The model used is OPT-6.7B, and perplexity (L) is evaluated on WikiText.
Figure 16:
I.7 Mixed-Precision Quantization
We provide additional results to show that our method can enhance mixed-precisio quantization by allocating precision layers. We still use the Alpha Hill metric to estimate the heavy-tail extent of each layer, and more heavy-tailed layers are then allocated with higher precision.
Table 20 shows that, compared to three baselines (random assignment, assigned by the norm of weights, and OWL), our AlphaPruning method can always achieve the lowest perplexity under three types of mixed-precissee quantization. Our experimental setup follows Tang et al. [2022].
Method
Mixed 3/4 Bit
Mixed 2/3/4 Bit
Mixed 2/4 Bit
Random
12.04
11455.54
14817.12
L1 norm
14.61
13959.42 42
33679.21
OWL
9.54
311.95
8429.39
Ours
9.01
261.39
7630.14
Table 20: Perplexity (Q) of different methods allocating precision to different layers for mixed- precision guantization with LLaMA-7B on WikiText.
27

---

L.8 More fine-tuning results
Here, we provide results for LoRA fine-tuning pruned LLaMA-7B with sparsity of 70% using SParseGPT. We compare AlphaPruning with Uniform and OWL. The experiment settings align with Section 4.3. Table 21 summarizes the results for perplexity and mean zero-shot accuracies after fine-tuning pruned LLaMA-7B models, which show that the performance improvement achieved by method doesn't 't diminish after fine-tuning.
Method
Sparsity
Fine-tuning
Perplexity (1)
Zero-shot accuracy (t)
Dense model
X
5.68
60.08
Uniform
70%
X 26.30
41.52
Uniform
70%
LoRA
11.21
49.00
OWL 70%
X
19.49
44.65
OWL
70%
LoRA
11.13
49.33
Ours 70%
X LoRA
18.54
45,48
Ours
70%
10,95
49.51
Table 21: WikiText validation perplexity and mean zero-shot tasks accuracy of SparseGPT pruned LLaMA-7B at 70% sparsity after LoRA fine-tuning on 30,000 C4 dataset samples.
1.9 More results vision models
Here, we present more CV task results. We choose three widely used non-uniform, poyer-wise sparsity methods in the CV context including Global [Frankle and Carbin, 2018], ERK [Mocanu et al., 2018] 81, LAMP [Lee et al., 2020]. Here, we use four ImageNet- IK pre-trained models (ViT-L, ViT-B, DeiT-B, DeiT-S), and we prune them to different sparsities.
VIT-B 16/24
VIT-L 16/224 50%
DeiT-S 16/224
DelT-B 16/224
Method
40%
50%
60%
40%
60%
40%
50%
60%
40%
50%
60%
Uniform
70.87
59.46
29.97
76.05
68.73
39.45
75.62
68.98
50.49
80.08
76.37
61.72
Global [Frankle Carbin, 2018]
66.81
45.75
8.09
75.42
14
74.17
65.77
38.87
79.94
75.09
57.01
ERK [Mocanu et al., 2018]
70.89
60.49
33.15
76.26
69.51
40.57
75.65
80.80
52.95
80.05
76 26.22
63.49
[2020]
75.51
Ours
71.58
64.29
44.21
76.86
72.12
55.62
77.07
72.38
60.92
80.21
77.11
64.56
Table 22: ImageNet-IK- 1K Accuracy (f) with various layerwise sparsity using Magnitude-based pruning. without finetuning. The results are shown at 40%, 50%, and 60% sparsity on ViT-B, ViT-L, DeiT-S, and DeiT-B models. Higher accuracy is better.
L.00 Zero-shot tasks performance
For zero-shot results in Section 4.2, the 7 evaluated zero-shot tasks are: BoolQ [Clark et al., 2019], RTE [Wang et al., 2018), HellaSwag [Zellers et al., 2019], WinoGrande [Sakaguchi et al., 2021] I, ARC Easy and Challenge [Clark et al., 2018] and OpenbookQA [Mihaylov et al., 2018). We show the wise performance Table 23 and Table 24.
Experiments Compute Resources
We conducted all our experiments using NVIDIA L40 (40GB) GPUs. Specifically, we used single NVIDIA L40 GPU for pruning the 7B and 13B models, 4 GPUs for the 30B models, and GPUs for the 65B models. For the LoRA fine-tuning, we operated under a constrained computational budget, employing 2x 40GB GPUs for the 7B models. Detailed information the computational complexity of AlphaPruning is provided in Appendix 1.4.
28

---

Model
Method
BoolQ
RTE
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Mean
Dense
75.02
66.79
56.94
70.00
75.29
41.89
34.60
60.08
Magnitude
38,29
52.71
26.65
$1.38
26.68
19.80
11.60
32.30
OWL w. Magnitude
37.86
52.71
27.32
52.57
28.96
22.01
13.60
33.57
Llama-Vi V1
Ours w. Magnitude
40.31
53.69
30.87
$1.22
36.24
21.76
16.20
35.67
7B
Wanda
56.45
55.96
28.94
51.38
33.32
18.26
13.80
36.73
OWL ur. Wanda
62.63
58.84
34.89
58.64
46.97
24.32
17.60
43.41
Ours Wanda
66.12
59.93
36.26
62.35
43,90
25.17
17.20
44.42
SparsoGPT
64.65
53.69
33.73
56.39
42.93
22.10
17.60
41.52
OWL w. SparseGPT
66.15
52.71
07.08
62.35
48.40
26.02
19.80
44.65
Ours SpurseGPT
68.26
55.96
37.81
64.01
46.17
27.39
18.80
45.48
Dense
77.86
70.75
59.91
72.61
77.40
46.42
33.20
62.59
Magnitude
52.97
50.54
26.58
50.75
28.45
20.56
14,80
34.95
OWL at. Magnitude
55.75
49.10
27 80
50.28
31.23
23.89
20.00
36.86
Llama-Vi V1
Ours w. Magnitude
61.28
46.93
30.24
50.43
31.23
26.28
21.20
38.23
13B
Wanda
61.90
32.71
30.50
53.12
40.91
17.58
15.60
38.90
OWL wr Wanda
62.81
52.71
38.57
63.46
$7.07
26.37
20.40
45.91
Ours w. Wanda
64.83
52.71
41.04
64.96
$9.39
28.24
21.20
47.48
SparseGPT
66.76
52.70
37.08
$3.07
21.00
OWL Sparsegpt
66.82
53.07
40.36
66.22
$7.37
28.67
20,80
47.61
Ours w. Sparsegpt
67.58
54.51
42.43
67,80
$5.47
29.10
22.00
48.41
Dense
82.69
67.15
63.35
75.77
80.43
$2,90
36,00
65 65.47
Magnitude
39.30
46.21
25.77
$2.49
25.29
21.50
16.20
32.39
OWL ut. Magnitude
39,93
58.48
25.94
$2.88
27.31
14.00
33.88
Llama-Vi V1
Ours Magnitude
62.02
47.29
32.61
57.14
47.18
27.47
24.20
42.56
30B
Wanda
66.09
56.68
43.96
67.09
65.28
31.83
26.60
$1.07
OWL us.
65.02
49.46
69.77
68.98
36.62
29.20
52.38
Ours Wanda
63.82
58.12
49.40
71.27
69.40
38.31
31,00
54.48
SparseCPT
68.13
61.01
44.50
68.75
65,70
33.53
27.60
52.75
OWL w. SparseGPT
67.95
55.60
46.96
72.22
67.13
35.49
26,80
53.10
Ours ie. SparseGPT
68.96
57.04
47.49
72.30
67.97
34.73
30.00
54.07
Dense
84.60
69.68
65 40
77.51
80.95
52.82
38.40
67.05
Magnitude
$1.85
54.51
38.55
$6.67
$7.35
29.95
26.40
45.04
OWL w. Magnitude
70.30
52.71
50.01
67.88
65.40
35.75
31.80
53.42
Llama-V VI 65B
Ours w. Magnitude
70.65
62.45
51.35
66.61
66.50
37.54
31.40
55.22
Wanda
76.45
56.68
47.50
69.61
70.33
35.92
27.50
54.90
OWL Wanda
78.50
58.48
50.90
74.11
70.70
38.05
30.60
57.34
Ours Wanda
81.85
61.01
52.55
75.06
71.60
40.02
30.40
58.93
SparseGPT
81.00
58.84
30.70
74.66
70.90
40.19
28.40
57.81
OWL w. SparseGPT
81.30
67.15
51.25
74.98
68.35
37.71
58.25
Ours SpurseGPT
83.80
71.48
52.40
74.59
69.30
38.05
38.05
59.72
Table 23: Accuracies (%) of LLaMA for 7 zero-shot tasks with unstructured 70% sparsity. We
compare AlphaPruning with uniform pruning ratios and OWL using Magnitude-basd pruning, Wanda and SParseGPT.
29

---

Model
Method
BoolQ
RTE
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Mcan
Dense
62.81
59.71
77.73
69.14
76.30
43.43
31.40
59.71
Magnitude
37.95
53.07
25.96
49.25
27.34
22.70
17.00
33,38
OWL w. Magnitude
38.75
$2.35
27.38
48.38
32.28
22.27
17.80
34.17
Llama-Va 7B
Ours w. Magnitude
57.43
$1.99
51.14
31.99
23.81
37.86
Wanda
50.64
52.71
27.80
50.04
30.64
18.60
12.20
34,66
OWL ur. Wanda
62.11
52.71
31.83
55.96
43.52
20.31
16,80
40.46
Ours w. Wanda
62.23
$2.71
34.56
60.85
43.27
22.27
18.00
41.98
SparseGPT
65.38
53.43
33.55
57.62
44.52
22.91
16.40
41.84
OWL SparseGPT
66.94
52.71
36.57
63.06
49.33
24.49
21.60
44,96
Ours SpurseGPT
65.93
54.15
36.83
62.19
49.62
24.23
19.80
44.68
Dense
80.61
65.34
60.04
72.22
79.46
48
35.20
63 63.05
Magnitude
38.72
$2.70
27.56
49.33
31.27
20.73
14.60
33.56
OWI. us. Magnitude
38.69
52.71
34.59
$4.30
38.97
23.81
15.80
36.98
Llama- V2
Ours Magnitude
70.55
$2.71
37.13
62.98
42.51
27.22
18.60
44.53
13B
Wanda
62.39
52.71
28.98
$1.14
35.10
18.00
11.80
37.16
OWL at. Wanda
63.46
52.71
36.31
60.46
55.64
24.91
21.80
45.04
Ours w. Wanda
62.57
54.87
67.32
$4.46
21.60
47.21
SparseGPT
66.64
52.71
36.26
59.91
$4.00
25.85
20.80
45.17
OWL w. SparseGPT
68.04
54.15
39.31
65.75
57.70
27.82
22.80
47. 94
Ours w. SparseGPT
68.13
57.04
41.26
68.03
57.15
29.18
24.00
49.26
Dense
83,50
67.87
66.00
77.98
82.60
$4.27
37.20
67.06
Magnitude
39.15
54.51
42.55
$7.22
56.35
31.57
24.20
43.65
OWL u.. Magnitude
64.45
53.51
45.45
68.06
61.00
33.98
27.00
50.49
Llama-Va V2
Ours w. Magnitude
63.10
$3.07
45.65
67.09
62.55
33.96
29.60
50.72
70B
Wanda
74.30
60.65
49.10
74.43
71.85
38.74
28.00
56.72
Wanda
73.65
63.18
$1.40
74.74
72.70
38.13
29.60
57.63
SparseGPT
80.75
63.50
52.30
75.77
73,60
41.64
29.80
59.68
OWL w. SparseGPT
79.25
64.26
51.95
74.98
73.00
40.53
30.40
59.20
Ours w. SparseGPT
80.40
70.04
51.90
75.06
74.10
40.87
29.20
60.23
Table 24: Accuracies (%) LLaMA-2 for zero-shot tasks with unstructured 70% sparsity. We compare AlphaPruning with uniform pruning ratios and OWL using Magnitude-based pruning, Wanda, and SparseGPT.
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims in the abstract and listed contributions in Section reflect the main contributions made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of the paper in Appendix B.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "LLimitttooss" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
31

---

Although the paper have no direct theoretical results, our work is theoretically- principled, which heavily relies on the HT-SR theory.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided the code for reproducing the main experimental results and detailed the experimental settings in the Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
32

---

Answer: [Yes]
Justification:
Our anonymized version of code available here
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
Poolii//uudess/ode@ubbeae. sbmissiooollccy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//nips. wtips.cccpubllc////occccc http:p/iiiiiggueeee/ooeeeeeaaaia /Ct////ooeeeeeeeeeeeeeeeeee issi www.sssiiccyyyyoooocoooiccsiiiii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental setting is presented in section 4.1 and the hyperparameters setting be seen in Appendix G.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We have conducted tests for the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
33

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors Is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer:
[Yes]
Justification: The information on the computer resources for experiments can be seen in Appendix J.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurips httpe///eeurrppccpppiiiieeee htt.ol/ie//hhcssciiiellliieeeeee
Answer: [Yes]
Justification:
We make sure to conform the NeurlPS Code of Ethics in every respect of the
paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., . if there is special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the societal impacts, which can be seen in Appendix A
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation. generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
34

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have cited the original paper that produced the code package in our paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi paperswithcode. thcode. com/ datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
35

---

13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have communicated the details of the code as part of our submissions and anonymized our assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "rVglAPuqRz", "reviewer": "Reviewer_N9yi", "summary": "The paper introduces AlphaPruning, a novel method for pruning large language models (LLMs) using Heavy-Tailed Self-Regularization (HT-SR) Theory. AlphaPruning uses ESDs of weight matrices to determine layerwise pruning ratios. The method demonstrates the ability to prune LLaMA-7B to 80% sparsity while maintaining reasonable perplexity.", "questions": "1. Can you explain why the property of $W^T*W$ can decide the sparsity of the layer and can be combined with different pruning metrics?\n\n2. To my knowledge, large language models (LLMs) are generally well-trained. The article concludes that PL_Alpha_Hill can indicate whether a layer is well-trained. Are there other methods or indicators that demonstrate if these layers are not well-trained?\n\n3. Have you compared the sparsity allocation of different layers with OWL? If so, is the sparsity allocation similar to that of OWL?", "limitations": "This article includes a limitation.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method, AlphaPruning, achieved SOTA performance across various tasks compared to OWL. \n\n2. The experiments conducted were comprehensive and diverse.", "weaknesses": "1. The motivation is not explicitly explained. It appears to apply a theoretical concept to the pruning area without providing quantitative or qualitative proof.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weakness 1 and Question 1\n**Motivation of our method and why property of $W^\\top W$ can decide layer sparsity**. Our method is grounded in heavy-tail self-regularization (HT-SR) theory, which we use to quantify the training quality of each layer and determine layer-wise sparsity. The rationale is as follows:\n- HR-SR theory originated as a semi-empirical theory, with early empirical work [1-2] examining the empirical spectral density (ESD) of weight matrices, specifically the eigenspectrum of the correlation matrix $W^\\top W$. This research found that the heavy-tailed structure of the ESD strongly correlates with training quality. These findings are rooted in heavy-tailed random matrix theory and statistical physics, as detailed in Table 1 of [2].\n- Recent theoretical work studies how heavy-tails in ESD emerge and why it correlates with training quality. It is well-known [3,5] that spikes in ESD represent \"signals\", while the bulk represents noise, which follows the Marchenko-Pastur law. In the theoretical setting of [3], the signal or the spike aligns with ground-truth features from the teacher model, and that corresponds to increased correlations in weight elements [1-2]. Furthermore, [4] shows that heavy tails in ESD originate from the interaction between spikes and bulk, which can be quantified precisely using recent advances in the free-probability theory [6], and that is the \"bulk-decay\" phase in the five-plus-one phase model in [2]. These studies indicate that layers with more heavy-tailed ESDs have extracted more useful signals during training, indicating better training quality.\n- This insight motivates our sparsity assignment method: layers with more heavy-tailed ESD contain more learned signals and are assigned lower sparsity by our method, while layers with less heavy-tailed ESD retain fewer signals and are assigned higher sparsity. In practice, the heavy-tailed structure is measured by fitting a power-law distribution to the ESD, and extracting the power-law exponent $\\alpha$ as the indicator. This is why our method is named \"AlphaPruning\".\n\n**Why our method can be combined with other pruning metrics?** Our layer-wise sparsity assignment is complementary to other pruning metrics, such as Wanda and SparseGPT. Our method determines the training quality of layers, while other pruning metrics identify the importance of components within each layer (or weight matrix). Thus, our approach decides how much to prune for each layer, while other metrics determine which components to prune within each layer. \n\n**References**\n\n[1] Martin et al. Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data.\n\n[2] Martin et al. Traditional and Heavy-Tailed Self Regularization in Neural Network Models\n\n[3] Wang et al. Spectral Evolution and Invariance in Linear-width Neural Networks\n\n[4] Kothapalli et al. Crafting Heavy-Tails in Weight Matrix Spectrum without Gradient Noise\n\n[5] Couillet and Liao et al. Random Matrix Methods for Machine Learning\n\n[6] Landau et al. Singular vectors of sums of rectangular random matrices and optimal estimation of high-rank signals: The extensive spike model\n\n## Question 2\nIn addition to PL_Alpha_Hill proposed in our work, [7-8] are other studies that investigated methods that measure whether a layer is well-trained or not, demonstrating LLMs layers are not equally well-trained. [7] developed a method that assesses the similarity between the representations at different layers, defined as the angular distance between feature vectors. They found that deeper layers are more similar to their neighboring layers than shallow layers, suggesting that LLMs may not fully utilize the parameters in these deeper layers, indicating these layers are not well-trained. Similarly, [8] introduced a metric called Block Influence, which measures the impact of each transformer block on hidden states to gauge layer significance. Their findings showed varying degrees of ineffectiveness/redundancy across layers, suggesting that these layers are not well-trained.\n\n**References**\n\n[7] Gromov et al. The Unreasonable Ineffectiveness of the Deeper Layers\n\n[8] Men et al. ShortGPT: Layers in Large Language Models are More Redundant Than You Expect\n\n\n## Question 3 \nIn Figure 10 of the rebuttal PDF, we compare the sparsity allocation of the two methods. We show that the general trends of sparsity distribution generated by the two methods are similar, with lower sparsities allocated to earlier layers and higher sparsities allocated to deeper layers. However, our method produces a more granular distribution with clearer distinctions between consecutive deep layers, resulting in improved pruning performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ycqY3CTlTU", "reviewer": "Reviewer_nY3j", "summary": "This paper introduces Alpha Pruning, a novel approach for pruning large language models based on Heavy-Tailed Self-Regularization theory. Instead of applying a uniform pruning ratio across layers, Alpha Pruning utilizes PL_Alpha_Hill, derived from empirical spectral densities (ESDs), to assess how well-trained each layer is. It then assigns a lower pruning ratio to well-trained layers to preserve model performance. Alpha Pruning is evaluated across various LLM architectures and datasets, demonstrating superior performance compared to baseline uniform pruning and SOTA  methods like OWL.", "questions": "- How does the PL_Alpha_Hill metric evolve during fine-tuning? Does its value typically decrease as a block becomes better trained?\n- What might be the reason why a block is better trained than others? Is it possible to include PL_Alpha_Hill in the training process to train each block equally and accelerate the training?\n- What factors contribute to the non-linear relationship between sparsity and speedups is shown in the Table. 4? Is the maintenance of early layers a significant factor in this observation?", "limitations": "The limitation of this work is well discussed in the Appendix.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "- Alpha Pruning leverages HT-SR theory to provide a principled method for guiding layer-wise pruning decisions, contrasting with heuristic-based approaches\n- This pruning method has been evaluated on various large language models and exhibits robust performance, showcasing its effectiveness and generalizability.\n- By evaluating the importance of each layer and assigning non-uniform pruning ratios, Alpha Pruning can complement existing pruning techniques such as magnitude-based pruning and Wanda. It is also compatible with other model acceleration techniques, such as structured pruning and quantization.\n- Despite the challenges of unstructured pruning in achieving significant speedups compared to structured methods, Alpha Pruning still achieves noticeable efficiency gains with high pruning ratios", "weaknesses": "- The paper’s explanation, particularly concerning HT-SR theory and terms used in the method section, may be challenging to grasp for readers unfamiliar with these concepts. Figure 1. a could benefit from clearer explanations.\n- While Alpha Pruning is compared against uniform pruning and OWL across multiple architectures, other layer-wise pruning methods prevalent in the computer vision community [1] could provide additional comparative insights.\n\n\n[1] Lee, Jaeho, et al. \"Layer-adaptive sparsity for the magnitude-based pruning.\" arXiv preprint arXiv:2010.07611 (2020).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weakness 1\nWe provide a detailed explanation of the parts the reviewer suggested, including HT-SR theory, terms in the method, and Figure 1a. We will include these in the updated draft. \n\n- **More details of HT-SR theory:** HT-SR theory [1-2] examines the empirical spectral density (ESD) of weight matrices, specifically the eigenspectrum of the correlation matrix $W^\\top W$. This research found that the heavy-tailed structure in ESD is strongly correlated with training quality. Recent theoretical work [3-4] finds that such a structure is a result of feature learning, a process of extracting various useful correlations (or features) from data during optimization. These studies indicate that layers with more heavy-tailed ESDs have extracted more useful signals during training, indicating better training quality. In practice, the heavy-tailed structure is measured by fitting a power-law distribution to the ESD, and extracting the power-law exponent $\\alpha$ as the indicator. These studies motivate our work, and we allocate the layer-wise sparsity based on the $\\alpha$ metric. This is why our method is named \"AlphaPruning\".\n\n- **Terms used in method:** In our method section, $\\lambda$ represents the eigenvalues of the weight matrices' correlation matrix. The interval ($\\lambda_{\\min }$, $\\lambda_{\\max }$) defines the range of eigenvalues considered as the tail part of ESDs. \n\n- **Figure 1a:** The blue histograms depict the empirical spectral density. The $x$-axis represents eigenvalue magnitudes, and the $y$-axis represents the density, both on a logarithmic scale. The solid red curves depict the empirical distribution of the ESD tail, while the dashed red curves represent the fitted PL distribution. The PL_Alpha_Hill metric in the title is the fitted PL exponent.\n\n## Weakness 2\nWe thank the reviewer for providing the new baseline method LAMP, we provide new experiments on comparing our method with LAMP. The results are shown in Table 25 of the rebuttal PDF. We implement the original LAMP method, which allocates different sparsity for each matrix, and a variant LAMP called LAMP (per block), which allocates the same sparsity for all matrices within a transformer block. This adaption is based on our ablation study comparing per-matrix and per-block strategies in Appendix F. It shows that LAMP (per-block) outperforms the original LAMP, and shows better performance than Uniform pruning baseline when both are combined with SparseGPT. However, our method AlphaPruning outperforms this baseline method. \n\nHere are the experimental settings. We pruned the LLaMA-V1-7B model to three sparsity levels (60%, 70%, 80%) using different methods and reported the perplexity on the WikiText validation set. The hyperparameter search and setup are consistent with the original settings in Appendix G.\n\n## Question 1\nIn Table 26 of the rebuttal PDF, we present the PL_Alpha_Hill and performance metrics (perplexity, zero-shot task accuracy) before and after fine-tuning. We can see that the PL_Alpha_Hill metric is reduced as well as the two performance metrics improved. Here is the experimental setup. We fine-tune the pruned LLaMA-V1-7B on 30K tokens from the C4 dataset. The model is pruned to 70\\% sparsity. The perplexity is evaluated on WiKiText, and the accuracy is evaluated on 7 zero-shot datasets as listed in Section 4.1.\n\nWe select the initial stage of fine-tuning to demonstrate how the metric value evolves. Figure 14 of the rebuttal PDF shows that the PL_Alpha_Hill metric indeed continues decreasing during the fine-tuning.\n\n## Question 2\nThe phenomenon that layers/blocks are not equally well-trained is well-documented in [5-7]. For example, [5] shows that layers or blocks of a model can have imbalanced training quality during training, and balancing the training speed of layers/blocks improves performance. As another example, [6-7] shows that some LLM layers are less useful than others, and removing these LLM layers has negligible impact on performance. However, the underlying reason why some layers are less well-trained than others remains an open question, and we will include that in future studies.\n\nIncorporating PL_Alpha_Hill into the training process is possible. Recent work [5] has used this metric to dynamically adjust the learning rate of each layer during the training and demonstrated their method makes the layers more equally well-trained and improves the generalization performance.\n\n## Question 3\n**The non-linear relationship is due to the low-level hardware operations of sparsity structures rather than the maintenance of early layers.** As shown in Figure 4 of [8], for low sparsity, the computational performance (red lines) increases slowly due to overheads in storing sparse structures and controlling sparse computations. As sparsity increases to moderate and high levels, we see sustained growth of performance until it usually levels off at extremely high sparsities where storage and control overheads dominate. This explains the slow speedup in our low-sparsity regime (e.g., less than 70% sparsity).\n\n**A new ablation study shows that early layer maintenance is irrelevant to the non-linear relationship.** In Figure 11 of the rebuttal PDF, we compare the inference speedup of non-uniform pruning (AlphaPruning) and uniform pruning. The negligible speedup differences between the two methods indicate that non-uniform distribution is unrelated to the observed non-linearity. This is because our method allocates sparsity at the transformer block level, not at the individual layer level, and in transformer architectures, all blocks use identical computational resources.\n\n**References**\n\n[1] Ref [10] in submission\n\n[2] Ref [13] in submission\n\n[3] Ref [58] in submission\n\n[4] Kothapalli et al. 2024\n\n[5] Zhou et al. 2023 Temperature Balancing\n\n[6] Gromov et al. 2024 The Unreasonable Ineffectiveness of the Deeper Layers\n\n[7] Men et al. 2024 ShortGPT\n\n[8] Hoefler et al. 2021 Sparsity in Deep Learning", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B1LQjhQbnw", "reviewer": "Reviewer_hyQK", "summary": "This work presents AlphaPruning, which prunes weight matrices of LLM models with different layer-wise sparsity levels based on the Heavy-tailed self-regularization theory.\nCompared to pruning with uniform sparsity among layers, AlphaPruning alleviates performance degeneration when the sparsity level is high and fine-tuning is not applied.\nAlphaPruning is composable with existing pruning methods.\nExperiments with various LLM models, such as Llama and Llama2, demonstrate the effectiveness.", "questions": "- Why is $\\mu_{\\mathbf{x}_i}$ is introduced in Equation (1)? It is merely used in the text except for L125.\n- What is the actual advantage of unstructured pruning as AlphaPruning over other acceleration techniques, such as quantization?", "limitations": "- As written in Weaknesses, the baselines are quite limited.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The presented method can be composed with other pruning methods.\n- The authors conducted experiments on various LLM architectures, such as Llama and Llama 2, with different baselines. In addition, they performed pruning of image classifiers, such as ConvNext and ViT.\n- Inference on CPU accelerates by the proposed pruning up to 3 times, depending on the sparsity.", "weaknesses": "- Although they say that \"AlphaPruning is theoretically driven\" in L83, I could not find any theoretical justification of the method in the text.\n- While \"Here, we provide a brief overview of HT-SR theory\" (L100), it is not described, making it challenging to understand the proposed method.\n- Although not an expert of this domain, I think using stronger baselines would make this work better.\nFor example, [Gale+19] demonstrated even simple magnitude pruning retains performance of ResNet-50 and Transformer even the sparsity level is set to 80%.\n\n[Gale+19] Trevor Gale, Erich Elsen, Sara Hooker \"The State of Sparsity in Deep Neural Networks\" arXiv 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weakness 1 and 2\nAlphaPruning is grounded in heavy-tail self-regularization (HT-SR) theory, which we use to quantify the training quality of each layer and determine layer-wise sparsity. Here we provide a detailed overview of this theory and explain how AlphaPruning is built based on it. We will include these explanations in the updated draft. \n- HR-SR theory originated as a semi-empirical theory, with early empirical work [1-2] examining the empirical spectral density (ESD) of weight matrices, specifically the eigenspectrum of the correlation matrix $W^\\top W$. This research found that the heavy-tailed structure of the ESD strongly correlates with training quality. These findings are rooted in heavy-tailed random matrix theory and statistical physics, as detailed in Table 1 of [2].\n- Recent theoretical work studies how heavy-tails in ESD emerge and why it correlates with training quality. It is well-known [3-4] that spikes in ESD represent \"signals\", while the bulk represents noise, which follows the Marchenko-Pastur law. In the theoretical setting of [3], the signal or the spike aligns with ground-truth features from the teacher model, and that corresponds to increased correlations in weight elements [1-2]. Furthermore, [5] shows that heavy tails in ESD originate from the interaction between spikes and bulk, which can be quantified precisely using recent advances in the free-probability theory [6], and that is the \"bulk-decay\" phase in the five-plus-one phase model in [2]. These studies indicate that layers with more heavy-tailed ESDs have extracted more useful signals during training, indicating better training quality.\n- This insight motivates our sparsity assignment method: layers with more heavy-tailed ESD contain more learned signals and are assigned lower sparsity by our method, while layers with less heavy-tailed ESD retain fewer signals and are assigned higher sparsity. In practice, the heavy-tailed structure is measured by fitting a power-law distribution to the ESD, and extracting the power-law exponent $\\alpha$ as the indicator. This is why our method is named \"AlphaPruning\".\n\n**References**\n\n[1] Martin et al. Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data.\n\n[2] Martin et al. Traditional and Heavy-Tailed Self Regularization in Neural Network Models\n\n[3] Wang et al. Spectral Evolution and Invariance in Linear-width Neural Networks\n\n[4] Couillet and Liao et al. Random Matrix Methods for Machine Learning\n\n[5] Kothapalli et al. Crafting Heavy-Tails in Weight Matrix Spectrum without Gradient Noise\n\n[6] Landau et al. Singular vectors of sums of rectangular random matrices and optimal estimation of high-rank signals: The extensive spike model\n\n## Weakness 3\nWe thank the reviewers for providing the new reference. The method used in [Gale+19] that performs best for Transformer models is **uniform magnitude pruning** with a **gradual pruning schedule**. The **gradual pruning schedule** refers to gradually increasing the sparsity of the network while training the semi-pruned model. This schedule is computationally intensive for LLMs due to the iterative model training and has not been adopted in LLM pruning literature. Our study focuses on pruning the LLM in a one-shot way without training, which aligns with previous studies [7-9] to ensure a fair comparison. We have included one-shot **uniform magnitude pruning** in the submitted paper, as shown in Tables 2 and 3 of the submitted paper. Our results demonstrate that our approach outperforms this baseline.\n\nWe clarify that in the submitted paper, we have compared with the most relevant and competitive baseline, OWL [7], which is the current SOTA non-uniform sparsity allocation method. We have also compared with other recent LLM pruning baselines, such as Wanda [8], and SparseGPT [9], as well as other baselines from CV pruning literature such as global, ER [10], rank-selection [11], and layer-wise error thresholding [12].\n\n**References**\n\n[7] Yin et al. 2024\n\n[8] Sun et al. 2024\n\n[9] Frantar et al. 2023\n\n[10] Mocanu et al 2018\n\n[11] Kuzmin et al. 2019\n\n[12] Ye et al. 2020\n\n## Question 1\nWe thank the reviewer for pointing out this writing issue. We will remove the $\\mu_{\\mathbf{x}_i}$ in the updated draft.\n\n## Question 2\nBoth unstructured pruning and quantization are effective methods for improving inference speed and reducing memory footprints. The comparison between these two approaches in terms of efficiency can be nuanced depending on hardware deployment and the algorithms used. For example, unstructured sparsity has limited acceleration support in GPUs, compared to quantization, but it holds the great value of speedups on other hardware such as CPU, cerebras' chip, IPU, etc.\n\n**Recent studies indicate that unstructured pruning can slightly outperform quantization in inference speedup.** A SOTA approach SqueezeLLM ([13]) demonstrates that quantizing LLMs to 3 bits can achieve a 2.1× speedup. Meanwhile, recent advances in sparse CPU kernels (DeepSparse) have better support in accelerating unstructured pruning, leading to 3.35$\\times$ speedup in CPU runtimes, as shown in Table 3 of [14], as well as Table 4 of our submitted paper. This is partly because pruning can better maintain and recover performance through fine-tuning compared to quantization, as noted in [14].\n\n**Pruning and quantization are compatible and complementary, and combining both approaches further enhances efficiency.** Table 3 of [14] shows that combining unstructured pruning with quantization (INT8) can achieve up to 9.08× speedup, significantly higher than using either method alone.\n\n**References**\n\n[13] Kim et al. 2024\n\n[14] Kurtic et al. Sparse Fine-tuning for Inference Acceleration of Large Language Models, 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3Mb6qr1YtU", "reviewer": "Reviewer_GYDB", "summary": "This paper introduces AlphaPruning, a novel framework for unstructured LLM pruning. The framework leverages HT-SR theory that utilizes the heavy-tailed shape of ESDs in layer-weight matrices to allocate layer-wise sparsity more effectively. By focusing on shape metrics rather than scale metrics, AlphaPruning demonstrates superior performance in maintaining model accuracy and reducing complexity. The method has been empirically validated across various architectures, showing significant improvements in performance and efficiency compared to existing methods, with strong generalizability to other compression techniques and architectures.", "questions": "+ Section 3.2 proposes a linear mapping to get sparsity from layer quality. Would other mappings, such as first compute the logarithmic of the m and then perform linear mapping, yield better outcomes?\n\n+ I would like to see results on using AlphaPruning to determine layerwise sparsity for other structured pruning methods, such as OSSCAR [1].\n\n[1] Meng, X., Ibrahim, S., Behdin, K., Hazimeh, et al. OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimization.", "limitations": "The authors addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ This paper introduces a novel sparsity allocation method that leverages the heavy-tailed shape of ESDs in layer weight matrices, a concept previously unexplored in the literature.\n\n+ The method is extensively validated with a range of LLM architectures, demonstrating significant improvements over state-of-the-art methods in terms of reducing perplexity, increasing accuracy, and achieving computational efficiency.\n\n+ AlphaPruning exhibits remarkable adaptability, integrating well with various other model compression techniques and extending beyond LLMs to include large vision model architectures, proving its versatility and broad applicability.", "weaknesses": "+ The novelty of the method for allocating sparsity based on layer quality (Section 3.2) is incremental, as a similar idea has previously been proposed in [1].\n\n[1] Zhou, Yefan, et al. Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weakness\nWe present the differences between AlphaPruning and [1], as detailed below:\n- **Different research focus.** Our study investigates post-training LLM pruning, whereas [1] studies model training. \n\n- **The underlying principles of the two works are different.** [1] aims to balance layer quality (or make layers equally well-trained) by tuning learning rates, aiming to improve generalization performance. This work focuses on minimizing pruning damage to the high-quality layers by making them less sparse, thereby reducing performance loss. While both studies use heavy-tailed metrics from HT-SR theory to estimate layer quality, this shared aspect does not make our work a trivial extension or incremental idea of [1]. This work is the first to explore how HT-SR theory can be applied to allocate sparsity, a concept previously unexplored in the literature.\n\n- **Important technical difference: transformer-block-wise measurement instead of matrix-wise measurement.** [1] measures the PL_Alpha_Hill metric for each weight matrix of the model individually. However, as we demonstrate in Appendix F, this approach provides suboptimal results for sparsity allocation. We improved upon this by averaging the metric scores across matrices within a transformer block and using the block-wise average score for sparsity allocation, which yielded significantly better results.\n\n## Question 1\nWe thank the reviewer for suggesting a new mapping function method. We implemented the proposed method and compared it with the linear mapping function used in our current approach, as shown in Figure 12 of the rebuttal PDF. The results show that both methods perform similarly when combined with Wanda, but linear mapping slightly outperforms the proposed logarithmic method when combined with SparseGPT.\n\nHere are the experimental setups for Figure 12. We pruned the LLaMA-V1-7B model to different sparsity levels using two allocation methods and reported the perplexity on the WikiText validation set. The hyperparameter search and setup are consistent with the original settings in Appendix G. \n\n## Question 2\nWe thank the reviewer for the new reference. We integrated AlphaPruning with OSSCAR and provided the updated results in Figure 13 of the rebuttal PDF. OSSCAR prunes only the linear sublayer of multi-head attention and the second sublayer of the feed-forward network, applying uniform pruning across each transformer block. By incorporating AlphaPruning's layer-wise sparsity allocation, we achieved non-uniform block-wise pruning ratios while keeping the global pruning ratio the same. The results show that integrating AlphaPruning with OSSCAR can reduce perplexity at different sparsities. We will include this experiment in the updated draft and cite the corresponding work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We want to thank all the reviewers for the constructive feedback, which helps us improve our paper. Please refer to the attached PDF for our new experiments and see below for our responses to each comment.

Meta-review: The paper proposes a new approach to pruning large language models (LLMs) based on the Heavy-Tailed Self-Regularization (HT-SR) theory. The authors introduce a metric called PL_Alpha_Hill, which measures the heavy-tailed shape of the empirical spectral density (ESD) of layer-weight matrices. They use this metric to allocate layer-wise sparsity more effectively, resulting in improved performance and efficiency compared to existing methods.

The reviewers praise the paper's novelty, technical soundness, and extensive validation across various architectures. However, they also raise concerns about the paper's clarity, the lack of theoretical justification, and the comparison with other pruning methods. The authors' rebuttal addresses these concerns by providing additional explanations, clarifying the relationship between HT-SR theory and the proposed method, and presenting new results that demonstrate the effectiveness of AlphaPruning in combination with other pruning methods.

Overall, the paper appears to be technically sound, and the authors have made a significant effort to address the reviewers' concerns. The paper's contribution to the field of LLM pruning is notable, and it has the potential to inspire further research in this area.

After the rebuttal and discussion, all reviewers lean towards acceptance. Overall, I would thus recommend to accept the submission. However, I would further recommend that the authors take the reviewer feedback into account for the camera-ready version.
Decision: Accept (poster)