Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment
Authors: Wei Li
Lujun Li
Mark G. Lee
Shengjie Sun
Venue: NeurIPS 2024 poster (2024)
Keywords: layerwise sparsity allocation
large language models
model compression
network pruning.
Primary area: natural_language_processing
TL;DR: We present a new layerwise sparsity allocation approach for pruning  large language models.

Abstract: Large Language Models (LLMs) have revolutionized the field of natural language processing with their impressive capabilities. However, their enormous size presents challenges for deploying them in real-world applications. Traditional compression techniques, like pruning, often lead to suboptimal performance due to their uniform pruning ratios and lack of consideration for the varying importance of features across different layers. To address these limitations, we present a novel  Adaptive Layer Sparsity (ALS) approach to optimize LLMs. Our approach consists of two key steps. Firstly, we estimate the correlation matrix between intermediate layers by leveraging the concept of information orthogonality. This novel perspective allows for a precise measurement of the importance of each layer across the model. Secondly, we employ a linear optimization algorithm to develop an adaptive sparse allocation strategy based on evaluating the correlation matrix. This strategy enables us to selectively prune features in intermediate layers, achieving fine-grained optimization of the LLM model. Considering the varying importance across different layers, we can significantly reduce the model size without sacrificing performance. We conduct extensive experiments on publicly available language processing datasets, including the LLaMA-V1|V2|V3 family and OPT, covering various benchmarks. Our experimental results validate the effectiveness of our ALS method, showcasing its superiority over previous approaches. The performance gains demonstrate its potential for enhancing LLMs' efficiency and resource utilization. Notably, our approach surpasses the state-of-the-art models Wanda and SparseGPT, showcasing its ability to excel even under high sparsity levels.  Codes at: https://github.com/lliai/ALS.

Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment
Wei Li Lujun Li Mark Lee"", Shengjie Sun Univeesitt of Birmingham
2 Hong Kong University of Science and Technology, 3 AAISpeech Co., Ltd.
wxL.855student. www.. aataammac..k, uk, lilujurai @gmail iilujiiaammmailll com, M.G. Lee. shengjie. sungai wwtfeaasseeecc.ccmm
Abstract
Large Language Models (LLMs) have revolutionized the field of natural lan- guage processing with their impressive capabilities. However, their enormous size presents challenges for deploying them in real-world applications. Traditional compression techniques, like pruning, often lead to suboptimal performance due their uniform pruning ratios and lack of consideration for the varying importance of features across different layers. To address these limitations, we present novel Adaptive Layer Sparsity (ALS) approach to optimize LLMss. Our approach consists of two key steps. Firstly, estimate the correlation matrix between intermediate layers by leveraging the concept of information orthogonality. This novel per- spective allows for precise measurement of the importance of each layer across the model. Secondly, we employ linear optimization algorithm to develop an adaptive sparse allocation strategy based on evaluating the correlation matrix. This strategy enables us to selectively prune features in intermediate layers, achieving fine-grained optimization of the LLM model. Considering the varying importance across different layers, we can significantly reduce the model size without sac- rificing performance. We conduct extensive experiments on publicly available language processing datasets, including the L.LMA-VIIV2IV family and OPT, covering various benchmarks. Our experimental results validate the effectiveness of our ALS method, showcasing its superiority over previous approaches. The performance gains demonstrate its potential for enhancing LLMs' efficiency and resource utilization. Notably, our approach surpasses the state-of-the-au models Wanda and SparseGPT, showcasing its ability to excel even under high sparsity levels. Codes at: Sttp://iiii....ooaaaa...........
Introduction
Large language models (LLMs) [62, 49, 3] have demonstrated remarkable performance in various natural language processing (NLP) [55, 54, 4] tasks. However, their size and computational require- ments pose significant challenges for widespread adoption and deployment. To address these practical constraints, model compression techniques, such weight pruning and quantization, can potentially reduce the size and computational requirements of LLMs.
The emergence of LLMs has revolutionized the field of NLP. However, despite their revolutionary impact, the massive scale and complexity of L.LMs presents significant challenges model com- pression. Conventional pruning methods [27, 19, 38, 19, 15, 59], which often require one more
"Coresponding authors, project lead with equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Objictivo:
Enieeddng
Comtarattt:
5000 Target Model Sire
Fandbble
Rag
2'2
Head
Original LLMS
Redundancy Metric
Linear Optimization
Adaptive Layer Sparsity
Figure 1: Overview of our framework. We first compute the sum of Redundancy Metric between layer i-th and other layers to construct objective function. Then, we solve a linear programming problem to optimize total sparsity ratios S(a)) (g: is pre-layer sparsity) under constraints.
iterations of fine- fie-ttning retraining preserve performance, have become impractical for LLMs due to the substantial computational cost and time required.
Due to the failure to the Magnitude approach to pruning [28] and other previous methods LLMs, recent efforts such as SparseGPT [17]. Wanda [46]. DSOT [65]. Pruming Large Language Models with BESA [57], and OWL [58] aim address this challenge by reconstructing the layerwise outputs of LLMs. Specifically, SparseGPT introduces a technique for pruning less significant weights and reconstructing layerwise outputs based on importance metric derived from the Hessian matrix. To reduce the computational overhead of SparseGPT, Wanda proposes a simplified strategy that relies solely the product of weight and activation magnitudes for pruning. DSOT computes the reconstruction error incrementally for each layer, optimizing the intra-layer sparse configuration through further weight pruning or growth, which forms the basis for subsequent weight recovery and additional pruning operations. These methods adopt a training-free approach. In contrast, BESA [57] proposes learning the optimal pruning ratio within each layer through training, finding that considering the overall sparsity configuration within a layer enhances the performance of sparse models. However, this method primarily focuses on intra-layer sparsity configuration. It requires substantial training time, typically taking at least 5 hours on an A100-80G GPU, which is considerably slower than other training-free techniques [17, 46, 65]. Another notable method is OWL [58], which proposes a non-uniform layerwise sparsity technique that assigns different sparsity ratios based on the outlier ratio within each layer, leveraging the unique characteristic of LLMs where some features exhibit significantly larger magnitudes by tuning hyperparameters such as the outlier threshold and sparsity upper/lower bounds to obtain optimal parameter setting. Nevertheless, unlike the aforementioned methods, OWL relies heavily empirical analysis without providing solid theoretical foundation for its effécctiveness.
However, existing methods have several significant drawbacks. First, for BESA, DSOT, and some traditional techniques, minimizing the layer-by-layer pruning error does not effectively mitigate the impact of pruning on model performance, as the pruning error accumulates across layers due to its inherent greedy nature [24] and may also become trapped in local optima [13, 22]. Second, LLM pruning methods such as Wanda, SparseGPT, and Magnitude apply uniform sparsity ratio to each layer, despite the significant variations each layer's contribution to the final model performance [57, 65]. To achieve better performance for different layers, the sparsity needs to be manually adjusted for all layers. Third, for the newly proposed OWL method, more theoretical analysis needed on why its outlier-based non-uniform sparsity outperforms uniform sparsity. Moreover, the choice of hyperparameters in OWL, such as the outlier threshold and sparsity upper/lower bounds, is sensitive to model performance, but their optimal ranges not theoretically explained, and the effective ranges and thresholds are derived through manual tuning. Furthermore, the transfcrability of these hyperparameters across different datasets has yet to be systematically studied. Therefore, when applying OWL to new models, complex adjustments by hand must be performed to determine the potentially optimal parameter combination.
To address the multiple challenges of getting trapped in local optima, manually setting sparsity for all layers, and relying on empirical manual experiments to derive optimal sparsity ratios, we propose simple, effective, and efficient method called Adaptive Layer Sparsity (ALS) for allocating sparsity ratios. The overall pipeline of our proposed method is illustrated in Fig. 1. This technique optimizes

---

the pruning rate across different layers. To the best of our knowledge, this is the first attempt to reformulate the sparsity allocation problem in LLMs as linear programming problem. We tackle these challenges by constructing an objective function and constraints. The constraint of the linear programming problem is that the total number of parameters should be less than the target model size. We then compute the independence matrix [25] at both the layer level and intra-layer component level based on the output or input features. According to our experiments, the independence between layers is positively correlated with model performance, shown in Fig. 2 (c). Therefore, maximizing the independence between each layer of the model is considered our objective function. Unlike existing black-box optimization methods, we formulate this problem as a linear one that can be solved by any linear problem solver. This approach enables efficient global sparsity ratio allocation for LLMs ranging from 7B to 70B parameters on single A100-80GB GPU. If the model scale is too large, reaching 160B, we can also perform multi-threaded computation on a CPU. For a 70B model, the global sparsity configuration can be obtained in just 20 minutes
A A100-80G GPU.
To rigorously assess the efficacy of ALS, we conducted extensive experiments on diverse LLMs, including LLaMA-VI [49]. LLaMA-V2 [50]. LLaMA-V3 [1]. and OPT [62] model families, with parameter counts ranging from 6.7 billion to 70 billion. In the main experiments, we evaluated the WikiText-2 perplexity and average accuracy on 7 zero-shot datasets at various sparsity ratios (20% to 70%) for LLaMA- V2 7B/13B (Table 1) and at 50% sparsity for all model families (Tables 2 and 3). Detailed results for each zero-shot dataset LLaMA-V2 V2 family models at 50% sparsity are presented in Table 4. The analysis experiments consist of 6 sets, examining the impact of calibration data, sparsity bounds setting, and model redundancy on performance (Fig. 2). well as the influence of feature selection, standardization, and comparisons with Wanda and LoRA fine-tuning. Additional experiments including detailed of main experiments, analyses of granularity, decreasing functions, visualizations of layer redundancy, sparsity ratio allocation and comparison with OWL method are provided in the Appendix C and D. These experimental results unequivocally demonstrate that ALS consistently yields substantial performance improvements for sparse LLMs across various LLMs and downstream tasks.
2
Related Work
Model Compression method try to design efficient models and reduce the memory and computa- tional requirements of LLMs. These methods can be categorized into quantization [42, 12, 35, 33], sparsification [17, 46, 10, 9] and distillation [56, 29, 30, 31, 32, 11, 53). Quantization converts high bit-width weights and activations into compact, low bit-width representations, while sparsification increases the proportion of zero-valued elements in model weights. Our method optimizes LLM sparsification by strategically allocating sparsity across the model's layers to maximize performance compression techniques (e.g.. OMPQ [35]) for CNN models in vision tasks, our approach focuses on different LLM models and NLP tasks and devises various functions and strategies accordingly.
and minimize computational overhead. In contrast to optimization-base
Sparsity in LLMs has garerred significant attention reduce memory consumption by applying sparsity in the model weights or activations. sparsity techniques can be categorized into two main approaches: structured pruning [34, 23] and unstructured pruning [16, 64, 46, 63]. While the primary focus of these works lies in determining the pruning criteria, such as weight importance and pruning ratio, the enormous parameter scale of LLMSs presents an additional challenge in terms of pruning efficiency. Conventional pruning methods [15, 59, 63, 23, 27, 19, 38, 19], dating back to the early work of Hassibi [20] in the 1990s, which successfully reduced model size and improved efficiency in deep learning architectures by removing redundant weights to create sparse and lightweight models, heavily rely on extensive retraining and are often infeasible for LLMs due to prohibitively high computational overhead and prolonged training times. To address this issue, researchers have developed LLM--pecific pruning techniques that prioritize train- and time efficiency. In the context of structured pruning, LLMpruner [34] explores the application of structured pruning to LLMs and employs LoRA to recover the performance of the pruned model. For unstructured pruning, SparseGPT [17] stands out as notable method that draws inspiration from the Optimal Brain Surgeon (OBS) [20] approach, taking into account the impact of removing individual weights on the network reconstruction loss. SparseGPT introduces an efficient technique for estimating the Hessian matrix, enabling the application of the traditional OBS method to large- scale models. Another prominent unstructured pruning method, Wanda [46], employs simple yet
means to accelcrate inference speed and
3

---

effective strategy based on the product of weight and activation values to identify and eliminate less important weights, further enhancing the pruning speed. Despite these advancements, most existing methods adopt a uniform pruning rate across all layers, which may lead to suboptimal performance. In contrast, our approach introduces a novel layer adaptive pruning strategy that dynamically allocates sparsity based on the importance of each layer, effectively minimizing performance degradation while achieving high compression ratios.
Sparsity Allocation in Network Pruning. Conventional methods for achieving adaptive layer- wise sparsity in neural networks [14, 5, 26] often rely on layer-by-layer pruning approach, where the objective is to minimize the sum of errors introduced in each layer. However, this greedy strategy [24] leads to the accumulation of errors across layers, resulting in suboptimal performance when directly adapted to LLMs. The extensive retraining required on vast datasets further amplifies the challenges of applying these techniques to L.Mss. Recent efforts, such as BESA [57] and DSOT [65], have shifted focus to intra-block sparsity allocation, employing various strategies to optimize the sparsity distribution within individual blocks. Despite operating at finer granularity, these methods fundamentally adhere to a layer-wise pruning paradigm, neglecting the importance of global sparsity allocation. Consequently, the resulting allocation may be locally optimal [13, 22] within each layer but globally suboptimal, potentially leading to solutions stuck in local optima. Recently, new approach called OWL [58] attempts to address this issue by introducing a non-uniform layer-wiss wise sparsity technique. This technique primarily relies manually tuning the outlier threshold and sparsity upper/lower bounds (which are very small values and sensitive to performance) through extensive experimentation to obtain potentially optimal parameter configurations. Although OWL demonstrates the potential for improved sparsity allocation, it heavily depends on empirical analysis and fails to provide solid theoretical foundation for its effectiveness, limiting its generalizability and robustness across different LLMs architectures and datasets.
3 Methodology
3.1 Preliminary
Pruning LLMs is method that aims obtain a sparse representation of the model by eliminating a predetermined fraction of the pre-trained weights. The primary objective is to minimize the
divergence between the outputs generated by the sparse and dense models [21]. However, directly tackling this problem can be challenging due to the massive scale of L.Mss. We discover that the mutual information entropy in Eq. can effectively quantify the degree of discrepancy between different layers of the model.
I (maaz) = H (x:) + H (I₂) - H (zi, 2z))
Neural networks can be decomposed into a sequence of layers. In the decomposed form, we represent the neural network as F {f1, fu). For a given random sample zu E Rd let 2; Ji (fi- (... fi (mo))) E Rd. represents the output of the random sample at the i-th layer.
Based on the previous definitions of the marginal entropies and joint entropy, the mutual information between I; and æj can be derived from Eq. I and formally defined as Eq. 2 [8].
(1:zz)) = f (1))
(1),,,))))00 log p ((1,,1,) dr,do; p ((x;) p (xzz)
High mutual information between layers indicates redundancy, while low mutual information suggests that these layers have learned complementary representations [43]. When two variables x; and x, are independent, their mutual information is zero, i.e., [xx,; xry) O. According to information theory [41] and the Information Bottleneck (IB) theory [48], minimizing the mutual information between layers can reduce redundancy, remove irrelevant information, and enhance the overall representational capacity of the network. This "compression" " of the representation enables the network to extract higher-level and more compact features, thereby reducing the reconstruction error. In summary, by sparsifying layers with higher mutual information and minimizing the mutual information throughout the entire network, the reconstruction error can be minimized.

---

3.2 Redundancy Metric
To approximate the mutual information between layers, we propose employing Monte Carlo sampling,
thereby circumventing the need for intractable integrals. Specifically, we randomly select N samples
(1) (2)
(M) from the training dataset, which follow a probability density function (PDF) P(x).
To 00 F0 (n) For each sample To the outputs at the i-th and j-th layers are denoted as (m) and zy respectively.
(n)
We can estimate the integral using the sample average: i (1T) my)) 2 EN log P
i (gg is the estimated mutual information, p p (I)") p (w) (n))
(n) ( (m) is the joint PDF, and p and
are the marginal PDFs. We aim to approximate these probability densities using kernel density estimation.
Computing marginal and joint probability densities: Based on the kernel density estimation [44], we can utilize it to estimate the probability density functions. This allows us to approximate the probability density functions using the features of the samples. Kernel density estimation is non porapeaaettric method for estimating the probability density function of a random variable. For instance, given set of samples Y {yi, 1/2,,/2..... WNN). The kernel density estimate is defined as: ply) 1 E K (* where (--) is kernel function, commonly used kernels include Gaussian, Epanechnikov, etc., and h is the bandwidth parameter.
NE
We apply kernel density estimation to compute the marginal and joint probability density functions of the samples' outputs at the i-th and j-th layers. The choice of the bandwidth parameter h can be determined through cross pross-validation or other methods. However, to simplify our derivation, we can consider that in nigh-dimensional spaces, the influence of the kernel function K is relatively insignificant. We are mainly focused on the ratio of relative densities [40, 60]. Therefore, the bandwidth parameter l can be cancelled out. We can calculate the marginal and joint probability density functions of the samples' outputs at the i-th and j-th layers using kernel density estimation, which can be found in the Appendix.
Monte Carlo Approximation of Mutual Information. Substituting the kernel density estimates into the Monte Carlo approximation formula for mutual information and simplifying the expression using the feature matrix inner product approximation for the kernel function, mentioned by Tschannen [51], we obtain:
(n)T I3
(n)
N
i (1) R
log
[n]T
[n)T
(m)
F
where || denotes the Frobenius norm. In this approximation, we employ the feature matrix inner product approximate the kernel function, K
((15m) _
(1)
(K)
(n)T (n)
Similarly, for the marginal kernel functions is K
(1)
(n)
Since mutual information has general upper bound, its upper limit depends on the entropy of either x; or Zz. To address this, we can use decreasing functions to transform the range of mutual information, ensuring a bounded and more interpretable metric. For instance, using (msa))) or a Gaussian function, we can redefine the measure as follows. Considering that a batch of data is fed into the model simultaneously and each layer output concurrently, we can omit the ΣN and y- Instead, we can use X; and X, represent the calculations for the entire batch input. Applying | x,|| ((X.:x)) as decreasing function to Eq. 3. Therefore, we can derive the Redundancy Metric (RM) formula, where RM)(-) E [0, 1] according to the Cauchy-Schwarz inequality:
Decreasing function:
| X,,
RM X,,,
X X,
The decreasing function transforms the range of the RM formula such that a value of 0 indicates complete independence between layers, while represents complete redundancy. This formulation
5

---

can serve as the objective function for maximization. The complete derivation process for this section, including the details of Eq. 4, is presented in Appendix B.
3.3 Linear Optimization
Our Redundancy Metric reveals the redundancy among layers in a neural network, guiding sparsity ratio allocation. Experiments on LLaMA2-13B with various sparsity configurations show negative correlation between model redundancy and WikiText-2 perplexity (PPL). Model redundancy is defined the sum of each layer's RM for the remaining layers, as depicted in Fig. 2 (c). Consequently, redundancy minimization is adopted as the objective function, incorporating model size constraints to formulate linear programming problem that yields the optimal sparsity configuration.
Intra-layer Sparsity Allocation. For a given neural network, we construct a redundancy matrix V, where thy = RM (2t, mz)) The sum of non-diagonal elements for each row of the matrix is computed as Pj Σラ= Vay - 1. A smaller Pi indicates stronger independence between x; and the outputs of other layers. We model this relationship using the monotonically decreasing function: wi = AP. where μe is dynamic hyperparameter controlling the difference in sparsity ratios across layers, defined as Ei= thy, which smooths the descent speed (Fig. 8). The importance factor for the first layers is represented by wls With these components, formulate the linear programming problem as follows:
Objective: max
Σ p
wj
L-i i
(5)
Constraints:
Σ s(a)) <B
where S(9=) denotes the model size of the i-th layer under sparsity qi and B represents the target model size. The optimal sparsity configuration is given by q. To maximize the model's representative capacity, our method try to assign smaller sparsity configurations maximizing an objective function. For more fine-grained sparsity allocation, we extend our approach to include intra-layer component-level sparsity allocation. After determining the sparsity ratios for each layer, we treat the remaining parameters in this layer as the target size and construct objective functions for its individual components. By applying ALS at this granular level, obtain secondary sparsity allocation, resulting in unique sparsity ratios for every layer and component. This hierarchical approach enables highly customized and adaptable sparsity distribution throughout the entire network architecture, potentially leading to enhanced efficiency and performance gains.
more independent layers by
Experimental Results
Setup. For pruning, we follow the settings of Wanda, SparseGPT, and Magnitude. Regarding the calibration data used in the linear optimization process, we follow the configurations of SparseGPT and Wanda, selecting data from the C4 dataset and ensuring that all test data zero-shot. We use a calibration data size of 16 for linear optimization hyperparameters. The granularity, explained in Appendix. E for linear optimization results is set to 0.5%. For the values of 24, we the input, although output and intermediate gates can also be used. Hyperparameter analysis is primarily conducted in the analysis section. Details about the experimental environment are provided in Appendix E.
Evaluation and Metrics. We measure the performance of pruned models through zero-shot tasks and language modeling. For zero-shot evaluation, we utilize seven tasks from the EleutherAI LM Harness [47]: Winogrande [39], PIQA [2], OpenBookQA [37]. HellaSwag [61], BoolQ [6]. ARC (Easy and Challenge) [7], and RTE (Recognizing Textual Entailment) [52]. We also include WikiTexx2 [36]. For the first seven datasets,
the accuracy metric provided in the EleutherAl
LM Harness. For WikiText2, we use the word_perplexitg (PPL) metric. During evaluation, we ensure using the same database version, GPU model, and random seed.
Models. We evaluate the performance of ALSS on L.MMs, including LLaMA-VI 7B/13B/30B/5BB [49], L2 7B/13B/70B [50], LLaMA- VV3
8B [1]. OPT 6.7B/13B [62].

---

Table 1: WikiText-2 perplexity performance of ALS varying sparsity rates for sparse LLaMA-V2-7B/13B pruned by the Magnitude, SparseGPT, Wanda metric.
Models
LLaMA-V2-7B
LLaMA-V2-
Sparse
20%
30%
40%
50%
60%
70%
20%
30%
40%
50%
60%
70%
Magnitude
9.20
10.21
13.51
32.87
7.6c4
9c5
7.84
8.19
9.21
11.59
23.43
1.4e3
Magnitude w, ALS
8.91
9.60
11,03
15.19
83.23
2.8c5
7.86
8.18
8.95
10.78
18.52
204.17
SparseGPT
8.94
9.19
9.70
17.21
15.58
42.87
7.86
8.07
8,46
11.32
12.29
27.12
SparseGPT W. ALS
8,90
9.11
9.67
10.99
15.35
39.09
7.87
8.09
8.50
9.44
12.29
26.70
Wanda
8.92
9.23
9.85
12.31
19.57
219.46
7.88
8.11
8.56
11.21
14.42
116.99
Wanda w. ALS
8.90
9.15
9.81
11.61
20.91
214.10
7.90
8.14
8.63
9.86
14.81
91.58
Table 2: WikiText-2 perplexity performance of ALS at 50% sparsity rates for sparse LLaMA-VI-7B/133 V1-7B/13B/30B/65H LLLMM--V-7BB13BB-- LLaMA- 8B/70B and OPT-6.7B/13B pruned by the Magnitude, SparseGPT, Wanda metric.
Models
LLaMA-VI
LLaMA-V2 V2
LLaMA-V3
OPT
Method
7B
13B
30B
65B
7B
13B
70B
8B
70B
6.7B
I3B
Dense
9.38
8.20
6.09
4.93
8.71
7.68
4.52
7.26
2.92
11.29
11.33
Magnitude
42.26
43.61
13.68
8.88
32.87
11.59
8.59
1.1e3
19.29
le3
4.1
Magnitude ALS
16,80
12.61
11.35
8.50
15.19
10.78
6.98
30.20
13.21
9,5e2
4e3
SparseGPT
18.35
9.90
10.07
7.82
17.21
11,32
7.84
16.87
8.49
13.35
12.53
SparseGPT w. ALS
11.87
9.97
8.28
7.97
10.99
9.44
7.58
10.23
7.24
12.29
11.49
Wanda
13.30
10.90
8.74
7.37
12.31
11.21
6.51
15.01
7.01
20.97
18.13
Wanda w. ALS
12.47
10.40
8.42
7.15
11.61
9.86
6.36
12.30
6.82
19.16
16.79
Baselines. We run ALS on LLMs with various methods, including Wanda [45], Magnitude-based pruning [18] and SparseGPT [16].
4.1 Language Modeling
Quantitative Evaluation. In Table 2, we compare the wikitext2 (PPL) performance of different
pruning methods under 50% sparsity the LLaMA-VI, LLaMA-V2, LLaMA-VI. and OPT models, including Dense (unpruned), Magnitude pruning [28], SparseGPT pruning [17]. Wanda pruning [46], and the results of these pruning methods enhanced by ALS. The results show that the ALS generally improves the performance of various pruning methods.
For LLaMA-VI models, Magnitude pruning shows high perplexity, e.g., 42.26 for the 7B model, reduced to 16.8 80 with ALS. SparseGPT performs better, with 18.35 for the 13B model, reduced to 1 11.87 with ALS. Wanda achieves the best results, with 13.30 for the 13B model, reduced to 12.47 with ALSS.
For LLaMA-Va V2 and LLaMA-V3 models, ALS also reduces perplexity significantly. For instance, the Magnitude pruning in 13B LLaMA-V2 model drops from 15.19 10.78, and Wanda pruning the 8B LLaMA-V3 model from 15.01 to 12.30 with ALS.
On the OPT model, perplexity significantly increases after pruning. For instance, the 13B model of OPT has a perplexity as high as 4.09e4 after Magnitude pruning, which remarkably reduces to 3.96e3 with ALS, demonstrating the effect of ALS in handling LLM pruning. However, there is an example where performance does not significantly improve with ALS. For instance, the 13B model of LLLMA-VI has 9.90 perplexity after SparseGPT pruning, which slightly increases with ALS.
In summary, ALS significantly enhances model performance across various pruning methods by effectively mitigating performance loss.
Varying Sparsity Rates. Table presents the perplexity scores of sparse LLaMA-V2 7B and 13B models pruned by Magnitude, SparseGPT, and Wanda methods, with and without ALS, at varying sparsity levels (20% to 70%). The results show that as sparsity increases, perplexity scores generally deteriorate, indicating a decline in language modeling performance. However, as the sparsity level increases, the performance gap between ALS and non-ALS methods widens, with ALS exhibiting better performance at most sparsity levels. This suggests that ALS can help mitigate the performance degradation caused by higher sparsity, becoming increasingly effective at maintaining LLMs performance the sparsity level grows.
7

---

Table 3: Averaged accuracies (%) for zero-shot tasks 50% sparsity rate for sparse LLaMA-VI 7B/13B/30B/65B, LLaMA-V2 7B/13B/70B, LLaMA-V3 8B and OPT-6.7B/133.
Models
LLaMA-VI
LLaMA-V2
LL
OPT
Method
7B
13B
30B
65B
7B
13B
70B
8B
70B
6.7B
13B
Dense
66.18
68.50
71.36
72.59
66.21
68.76
72.92
69.81
75.43
58.13
59.71
Magnitude
53.40
53.73
60.40
68.68
57.06
59.85
66.73
43.29
51.28
41.06
38.37
Magnitude w. ALS
56.28
61.21
62.61
69,42
60.09
63.39
70.28
57.43
53.64
43.31
40.92
SparseGPT
56.10
64.26
65,70
68.93
56.44
60.16
69.44
54.18
70.26
55.19
56.56
SParseGPT ALS
60.58
63.99
69.02
69.02
61.36
65.85
70.16
64.00
71.12
57.85
59.01
58.87
64.74
68.54
71.71
61.88
64.48
71.87
58.12
72.25
47.81
50.46
Wanda W. ALS
61.47
64.82
69.35
71.41
62.84
66.58
71.75
62.48
73.12
47.89
50,50
Table 4: Accuracies (%) for zero-shot tasks with 50% sparsity using LLaMA-V2 family.
LLaMA-V2
Meihad
winograndle
plgga
openbookqu
hellaswung
challenge
Mcan
Dense
69.06
19.11
44.20
15.38
17.74
74.49
46.25
62.82
66.21
Magnitude
63,30
73.67
38.80
65.58
62.94
57.70
37.46
57.04
$7.06
Magnitude w. ALS
65.19
75.46
41.40
69.11
71.38
63.47
39.51
55.24
60.09
7B
SparsoGPT
63.14
71.71
35.60
63.91
69.05
58.29
34.98
54.87
56.44
SparseGPT
67.96
76.39
70.52
41.13
55.96
61.36
Wanda
67.32
76.99
41.40
68.76
75.78
69.23
41.72
53.83
61.88
67.80
70.75
Dense
72.38
80.52
45.20
79.39
80.58
77.53
49.15
65,34
68.76
Magnitude
Magnitude w. ALS
68.35
77.48
43.60
74.42
73.15
69.91
44.63
$5.60
63.39
13B
SparsoGPT
67.25
75.84
41.20
63
63.76
38.40
56.68
SparseGPT w. ALS
71.19
78.13
44.40
74.99
81.16
69.82
43.94
63.18
65.83
Wandaa
69.39
78.13
444.00
75.02
80.34
70.37
42.76
55.72
64.48
Wanda ALS
72.06
78.51
45,80
75.67
81.35
70.33
46.08
62.82
66.58
Dense
77.98
82.70
48.80
83,80
83.79
81,06
$7.34
67.87
72.92
Magnitude
73.64
79.54
44.20
79.29
71.07
74.79
50.68
60.65
66.73
Magnitude w. ALS
74.74
80.96
46,40
80.57
79.63
77.99
54.10
67.87
70.28
70B
SparseGPT
75.37
79.38
77.02
48.38
67.87
SparseGPT
ALS
76.72
80.36
45.20
80.09
81.56
77.86
50.17
69.31
70.16
Wanda ALS
77.27
81.61
43.80
81.30
82.54
78.54
$5.80
71.12
71.75
4.2 Zero-shot Tasks
In Table 3, we present the averaged accuracy performance of pruned LLaMA-VI, LLaMA-V2. LLaMA-V33 V3, and OPT models on seven downstream zero-shot tasks at 50% sparsity ratio. For detailed performance on specific tasks, please refer Table 4, which shows improvements in most tasks. The average accuracy across the majority of tasks demonstrates the effectiveness of ALS in enhancing sparse large language models of any scale. Remarkably, for LLaMA3-88 the incorporation of ALS leads to an improvement of 14. 1414 and 9.87% compared to the Magnitude and SparseGPT baselines, respectively. Similarly, for LLaMAI 13B, the addition of ALS results an improvement of 5.95% compared to the baselines.
The significant performance improvement of LLAMA-V3 8B may be attributed to the fact that the new model is not well-suited for uniform pruning methods. we further use the L.aMA-VE V3 8B model as example to intuitively present the improvements brought by ALS from the perspective of the heat map (Fig. 3 in the Appendix D). The heat map reveals that, at 50% sparsity, the redundancy between layers of the LLaMA-V3 8B model exhibits a distinct pattern compared to other models. The green distribution on both sides indicates that there is small redundancy between the shallow layers and other layers. This suggests that the information captured by the middle and deep layers has more overlap and similarity with other layers. The ALS method takes advantage of this inter-layer redundancy pattern and apply high sparsity ratio into the layers with higher redundancy. maximally preserves the key information and reduces the impact of sparsification on model performance. From the heat maps of different models the Appendix D, it can also be observed that each model requires different sparsity ratio because the redundancy between its layers varies.

---

Table WikiText-2 perplexity of Wanda with Table 6: Zero-shot accuracy (%) Wanda with ALS at 50% sparsity on LLaMA-family models. ALS at 50% sparsity on LLaMA- family models.
Models
V1-7B
V1-13B
VI.30B
V2-7B
V2-13B
Mudels
VI-7B
VI-13B
V1.30B
V2-7B
V2-13B
Douse
9.38
8.20
6.09
8.71
7.68
Dense
66.18
68.50
71.36
06.21
68.76
Wanda
13.30
10.90
8.74
12.31
11.21
Wanda
58.87
64.74
68.34
61.88
64.48
ALS
12.47
10.40
8.42
11.61
9.86
ALS
61.47
64.82
69.35
62.84
66.58
LoRA
7.65
9.25
6.99
9,89
8.30
w. LoRt
71.64
66.71
71.44
65.05
67.81
Table Results of feature choice from varying component output of each layer on WikiText2 and zero-shot tasks.
Table 8: Impact of normalizing features and
per-layer weights for distance function on Wiki'Text2 and zero-shot tasks.
Feature Choice PPL
ACC
Distance Function
PPI.
ACC
In
10.070
60.75
Vanilla
10,078
66.40
Out Gate
10.012
60.56
Feature-Norny 10,076
66.40
10.030
60.76
Feature-Norm+ Weightt-ornn
10.070
68.11
4.3 Ablation Study
In this part, we examine the impact of various components within the ALS framework and compare it with LoRA Fine-tuning, specifically focusing on its bound setting, standardization on weight or feature, granularity choice, which can be found in Appendix. E, feature choice and robustness to calibration samples. All experimental setups are based on the LLaMA2-13B model with Wanda pruning and ALS.
Comparison with LoRA Fine-tuning. Our experiments in Table 5 and Table 6 demonstrate the substantial benefits of combining Wanda+ALS with LoRA fine-tuning across the LLaMA model family. The improvements are most striking in the LLaMA-VI 7B model, which showed a dramatic reduction in perplexity by 4.82 alongside 10.17% increase in accuracy. Larger VI models also benefited, with the 30B and 13B variants showing perplexity reductions of 1.43 and 1.15, coupled with accuracy gains of 2.09% and 1.89% respectively. The LLaMA-V2 models exhibited similar positive trends, with both 7B and 13B versions showing perplexity improvements and accuracy increases. These impressive results were achieved using just 2000 C4 samples for LoRA fine-tuning in a zero-shot setting, highlighting the method's efficiency and effectiveness even with limited training data unrelated to the evaluation tasks.
Feature Selection and Normalization. Table 7 (a) compares the performance of input, output, and gate features in capturing layer independence, with output features achieving slightly lower perplexity. Table 8 (b) demonstrates the significant impact of jointly normalizing features and per-layer weights. Applying this normalization strategy yields substantial improvement in accuracy, increasing from 66.40% to 68.11% ,, while also reducing perplexity from 10.078 to 10.070.
Comparison with OWL. We compared the performance of our proposed method with the OWL method on a set of benchmark datasets. The results are summarized in Table 9. We adopted the optimal parameter settings described in the OWL paper. Across all tested configurations, our method consistently achieved lower values compared to OWL, demonstrating its superior performance. Specifically, in the unstructured 50% setting, Wanda with ALS outperformed OWL by a margin of 0.25 units. Furthermore, the structured pruning settings of 2:4 and 4:8, the advantage of Wanda with ALS increased to 0.95 and 0.44, respectively.
We also investigated the performance of our method in the N:M setting, where N features are selected from M available features. The results are shown in Table 9. Similarly, for OWL, we used the optimal parameter combination reported in their paper. Across all N:M configurations, ALS consistently achieved lower values compared to OWL. As the number of selected features N increased, both methods exhibited performance improvements, but the advantage of ALS became more pronounced. For instance, in the 2:4 case, ALS outperformed OWL by margin of 0.95 units, and this gap further widened to 1.82 in the 4:8 case. Overall, OWL is a method that is highly sensitive to parameter settings, and obtaining the optimal parameters may require dozens of experiments to determine the best combination. Moreover, there is no clear theoretical analysis explaining why such combination should be used.
N:M Results.

---

W00000 Papiekky
3.50
Willbos Peppealty
Willest Pepiesity
1.54
8.90
122
"0"
6.3
3.00
3.80
50
150
200
250
300
0.55
8:300.0
9-555.85
0.0.1
1200
1220
1240
3200
1290
1356
3500
Calbration
Boundes
Recundancy
Figure 2: (a) Calibration data experiment: PPL decreases slightly with more data. (b) Pruning bounds: Model performance remains relatively stable between 30% and 70% bounds. (c) Model redundancy: Higher RM metric, lower performance.
Table WikiText-2 perplexity performance on Table 10: Pruning speed of various methods LLaMA-V2-13B at 50% sparsity rates.
with ALS LLaMA-V2-7B.
Ratio
Wanda ALS
Wanda OWL
ALS
50%
9.86
10.11
Base Method
RM (s)
(ms)
Total (min)
Magnitude (1.62s)
88.59
169
2:4
15.52
16.47
SparseGPT (1058s)
91.32
158
19.16
4:8
11.65
12.09
Wanda (199s)
89.47
160
4.81
Calibration Data. In Fig. 2 (a), we present the performance of pruning methods with different numbers of calibration samples. We use the size of 2, 4, 8, 16, 32, 64, 128, 256. Although this experiment reveals that the model's performance improves with an increase in the size of the calibration data, the improvement is quite limited. Even when comparing the scales of 2 and 256 in calibration samples, the perplexity decreases by only 0.11. These results further highlight the robustness of ALS.
Boundes. In Fig. 2 (b) demonstrates the effect of pruning bounds on the performance of the LLaMA- V2 13B model. When the pruning bounds are set too high (c.g.. 0,0-1.0), the model's performance significantly deteriorates from 10 to103 compared with 0.3-0.7, indicating that aggressive pruning may impair the model's representational capacity. However, when the pruning bounds are between 30% and 70%, the model's performance remains nearly unaffected.
Computation efficiency. As shown in Table 10, our ALS involves two computational phases: the Redundancy Metric (RM) calculation, which consistently takes approximately 90 seconds across all methods, and the Linear Programming (LP) solution, requiring roughly 160-170 milliseconds. The total processing time varies notably depending on the base pruning method employed: Magnitude pruning, requiring just 1.62 seconds for its base operation, achieves the fastest total completion time 1.51 minutes when combined with ALS. Wanda, with its base pruning time of 199 seconds, completes the entire process in 4.81 minutes, while SparseGPT, requiring 1058 seconds for its base operation, takes 19.16 minutes in total. Compared to BESA [57] with 4.5 hours for sparsity allocation and pruning, approach is notably faster, completing the process in minutes rather than hours.
5 Conclusion
In this work, present Adaptive Layer Sparsity (ALS), novel approach for optimizing LLMs through the efficient allocation of sparsity across layers. By minimizing inter-layer redundancy, ALS achieves significant model compression while maintaining performance, demonstrated through extensive experiments on diverse LLMs and tasks. We hope ALS offers valuable insights and practical tools for deploying LLMs under limited computational resources, and that our work may shed light on the role of sparsity in LLMs and its potential for model optimization. Future research will explore the relationship between sparsity allocation and individual weight importance, and investigate the integration of dynamic sparsity allocation with pruning metrics. By pushing the boundaries of model compression and efficiency, we aim to enhance the development of more capable and accessible LLMMs for diverse applications.
10

---

References
[1] AI@Meta. Llama 3 model card. 2024. 3,
[2] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference artificial intelligence (AAAI), volume 34, pages 7432-7439, 2020. 6
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural in information processing systems (NeurlPs), 33:1877-1900 2020.
[4] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprini arXiv:2303. /27/2, 2023.
[5] Yanqi Chen, Zhengyu Ma, Wei Fang, Xiawu Zheng, Zhaofei Yu, and Yonghong Tian. A unified framework for soft threshold pruning. In The Eleventh International Conference on Learning Representations, 2023. 4
[6] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprini 1905 2019. 6
[7] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803. 05457, 2018. 6
[8] Thomas M. Cover and Joy Thomas. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). -ilee-Interscience, USA, 2006. 4
[9] Peijie Dong. Lujun Li, Xiang Liu, Zhenheng Tang, Xuebo Liu, Qiang Wang, and Xiaowen Chu. Lpzero: Language model zero-cost proxy search from zero. arXiv preprint arXiv:2410.0444 2448000 2024. 3
[10] Peijie Dong, Lujun Li, Zhenheng Tang, Xiang Liu, Xinglin Pan, Qiang Wang, and Xiaowen Chu. Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. In ICML, 2024.
[11] Peijie Dong, Lujun Li, and Zimian Wei. Diswot: Student architecture search for distillation without training. In CVPR, 2023. 3
[12] Peijie Dong, Lujun Li, Zimian Wei, Xin Niu, Zhiliang Tian, and Hengyue Pan. Emq: Evolving training-free free proxies for automated mixed precision quantization. In ICCV, 2023.
[13] Xin Dong, Shangyu Chen, and Sinno Jialin Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 4860-4874, Red Hook, NY, USA, 2017.
Curran Associates Inc. 2, 4
[14] Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference Machine Learning (ICML), pages 2943-2952, 2020.
[15] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: neural networks. In International Conference Learning Representations, 2019. 1,
Finding sparse, trainable
[16] Elias Frantar and Dan Alistarh. Massive language models can be accurately pruned in one-shot.
In International Conference on Machine Learning (ICML), 2023. 3,
[17] Elias Frantar and Dan Alistarh. Sparsegpt: massive language models can be accurately pruned in one-shot. Proceedings of the 40th International Conference on Machine Learning, ICML.23. JMLR. 2023, 2,3,"
11

---

[18] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems (NeurIPS). pages 1135-1143, 2015.
[19] Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, NIPS 15, page 1135-1144 1143, Cambridge, MA, USA, 2015. MIT Press. 1,3
[20] B. Hassibi, D.G. Stork, and G.J. Wolff. Optimal brain surgeon and general network pruning. In IEEE International Conference on Neural Networks, pages 293-299 vol.1, 1993. 3
[21] Babak Hassibi, David G Stork, and Gregory Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference neural networks, pages 293-299. IEEE, 1993.
[22] Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI 18, page 1234-22400 AAAI Press, 2018. 2, 4
[23] Zhongzhan Huang, Xinjiang Wang, and Ping Luo. Convolationn-ccihh--iitsiiiiitii Rethinking the criteria channel pruning. CoRR, abs/2004. 2020. 3
assumption:
[24] Chunhui Jiang, Guiying Li, Chao Qian, and Ke Tang. Efficient dnn neuron pruning by minimiz- ing layer-wise nonlinear reconstruction error. Proceedings of the 27th International Joint Conference Artificial Intelligence, IJCAI'18 18, page 2298-2304. AAAI Press, 2018. 2, 4
[25] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3519-3529. PMLR, 09-15 Jun 2019.
[26] Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference on Machine Learning (ICML), pages 5544-5555, 2020. 4
[27] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touretzky, editor, Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmang 1989. 1, 3.3
[28] Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer-adaptive sparsity for the magnitude-basee pruning. In International Conference on Learning Representations (ICLR), 2020. 2,
[29] Lujun Li, Yufan Bao, Peijie Dong, Chuanguang Yang, Anggeng Li, Wenhan Luo, Qifeng Liu, Wei Xue, and Yike Guo. Detkds: Knowledge distillation search for object detectors. In ICML, 2024.7 3
[30] Lujun Li, Peijie Dong, Anggeng Li, Zimian Wei, and Ya Yang. Kd-zero: Evolving knowledge distiller for any teacher-student pairs. NeulPS, 2024. 3
[31] Lujun Li, Peijie Dong, Zimian Wei. and Ya Yang. Automated knowledge distillation via monte carlo tree search. In ICCV, 2023. 3
[32] Lujun Li and Zhe Jin. Shadow knowledge distillation: Bridging offline and online knowledge transfer. In NeulPS, 2022. 3
[33] Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.00997. 2023.
[34] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-pruner: On the structural pruning of large language models. Thirty-sevemth Conference on Neural Information Processing Systems, 2023.
12

---

[35] Yuexiao Ma, Taisong Jin, Xiawu Zheng. Yan Wang, Huixia Li, Yongjian Wu, Guannan Jiang, Wei Zhang, and Rongrong Ji. Ompq: Orthogonal mixed precision quantization. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages
2023. 3
[36] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. 6
[37] Todor Mihaylov, Peter Clark. Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
[38] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Representations, 2017. 1,3
[39] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9):99-106, 2021. 6
[40] David W. Scott. Multivariate density estimation and visualization. Papers 2004,16, Berlin, 2004. 5
[41] C. E. Shannon. A mathematical theory of communication. Bell System Technical Journal, 27(3):379-4233 1948, 4
[42] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang. Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quan- tization for large language models. In The Twelfth International Conference Learning Representations, 2024. 3
[43] Ravid Shwartz-Ziv and Naftali Tishby, Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.0083 2017. 4
[44] Bernard W Silverman. Density estimation for statistics and data analysis. Routledge, 2018.
[45] Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 11655, 2023." 7
[46] Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024. 2,3,7
[47] Sutawika, Schoelkopf, Gao, Abbasi, Biderman, Tow, fattori, Lovering, farzanehnakhaee, Phang. Thite, Fazz, Wang, Muennighoff, Aflah, sdtblck, nopperl, gakada, uttuntian,, researcher, Chris, Etxaniz, Lee, Kasner, Khalid, Hsu, Kanekar, Ammanamanchi, Boykis, and AndyZwei. ReeutterAllln-evaatatinn--aaaaan
v0.4.2, March 2024. 6, 24
[48] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pages 1-5. IEEE, 2015. 4
[49] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2300. 139.. 2023. 1, 6
[50] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0927 2023. 3,
[51] Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucie. On mutual information maximization for representation learning. arXiv preprint 107.v:1907.13625,
2019. 5
[52] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupata, and Afra Alishahi, editors, Proceedings of the 2018 EMNLP Workshop BlackboxNI.P. Analyzing and Interpreting Neural Networks for NIP,, pages 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. 6
13

---

[53] Qiufeng Wang, Xu Yang, Shuxia Lin, and Xin Geng. Learngene: Inheriting condensed knowledge from the ancestry model descendant models. ArXiv, abs/2305.02279, 2023.
[54] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud. Dani Yogatama. Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
[55] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems (NeurlPs), 35:24824-24887, 2022.
[56] Liu Xiaolong, Li Lujun, Li Chao, and Anbang Yao. Norm: Knowledge distillation via n-to-one representation matching. In ICLR, 2023.
[57] Peng Xu, Wenqi Shao, Mengzhao Chen, Shitao Tang, Kaipeng Zhang, Peng Gao, Feng- wei An, Yu Qiao, and Ping Luo. BESA: Pruning large language models with blockwise parameter-effcient sparsity allocation. In The Twelfth International Conference on Learning Representations, 2024. 2, 4, 10
[58] Lu Yin, You Wu, Zhenyu Zhang. Cheng- Yu Hsich, Yaqing Wang, Yiling Jia, Mykola Pech- enizkiy, Yi Liang, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning Ilms high sparsity. arXiv preprint arXiv:2310.05175 2023. 2,4
[59] Miao Yin, Burak Uzkent, Yilin Shen, Hongxia Jin, and Bo Yuan. Gohsp: a unified framework of graph and optimization-basee heterogeneous structured pruning for vision transformer. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, ANAT23/AMM233AAA23,, AAAI Press,
2023. 1,3
[60] Adriano Zambom and Ronaldo Dias. review of kernel density estimation with applications to econometrics. International Econometric Review, 5(1):20-42, 2013. 5
[61] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv:1905.07830 2019.
[62] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint 1.XXi::200501066 2022. 1, 3,
[63] Yuxin Zhang, Mingbao Lin, Fei Chao, Yan Wang, Ke Li, Yunhang Shen, Yongjian Wu, and Rongrong Ji. Lottery jackpots exist in pre-trained models. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2023. 3
[64] Yuxin Zhang, Mingbao Lin, Zhihang Lin, Yiting Luo, Ke Li, Fei Chao, Yongjian Wu, and Rongrong Ji. Learning best combination for efficient n: M sparsity. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[65] Yuxin Zhang, Lirui Zhao, Mingbao Lin, Sun Yunyun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. Dynamic sparse no training: Training-fres free fine-tuning for sparse LLMs. In The Twelfth International Conference on Learning Representations, 2024. 4
14

---

Appendix
A Limitation: Computational Complexity of Intra-Layer Component
Independence Calculation
In the ALS method, key step is computing the independence matrix between layers. While calculating the matrix at the layer level for a 70 billion parameter model with 80 layers requires only an 80 x 80 RM matrix, including intra-layer components increases the matrix size to 7 x 80 x 7 x 80,
leading to 49 times more computational time and resources.
High Computational Complexity: The increased matrix size results in exponential growth in computation and resource consumption.
Excessive Memory Usage: High-dimensional matrix computations require substantial memory, potentially exceeding hardware capacities.
Solutions
Hybrid Solution with C++: Store intermediate data locally and use C++ to handle calcula- tions and solve the linear programming problem. This approach can be up to times faster than using Python alone.
A Atternativlyy calculate sparsity ratios for each layer, determine the parameters to retain, and use these as the target size for further linear programming. This approach requires only an 80 x 80 RM matrix and 7 additional x 7 matrices, without significantly increasing computation time.
Hybrid Solution with C++is preferred solution because it will keep most of independent components maintain the model performance after pruning.
B Detailed Formulae Derivation
This section provides a detailed derivation of the key mathematical formulae used in the paper.
B.1 Detailed Derivation of the Mutual Information Approximation
The objective of LLMs is to minimize the reconstruction error between the outputs of the sparse and dense models. We start by defining the mutual information between the outputs of different layers to quantify the redundancy within the model. Mutual information T(X; Y)) measures the amount of information obtained about one random variable X through observing the other random variable Y. It quantifies the reduction in uncertainty of X due to the knowledge of Y. This measure helps identify layers with high redundancy, which can be pruned to achieve a more efficient representation.
The mutual information / Xy) between two layers i and j is given by:
I (X;; X3) = H (x,) + H (X,) - H (X,
(6)
where H (X₂) and H (X,) are the marginal entropies, and H Xy) is the joint entropy. The
entropy H (X) measures the average amount of information or uncertainty in a random variable X. These entropies are defined as:
(X,) = / p (zi) log (xx) dæ;
/ p (zz) log (xz) dr,
H ((2))
(8)
/ p (x, x.,) log (m,, 2x)) da; dæ,
H HX,, X2) - (x),,))-11
The mutual information can also be expressed in terms of the probability density functions as:
1 T(x; X,) / (2,, x.)) log p xx) dr, dr,
(10)
p (xx)) (x,)
15

---

To approximate these values, we use Monte Carlo sampling. Given N i.dd. samples x(4), x()))(2))))))))))))))))))) ,x.,=! x(M) drawn from the joint distribution p(æ,, ,y), the mutual information can be approximated as:
(n) P
(n)
1
i (X;;)))
log
(11)
2
3/
Using kernel density estimation to approximate the probability densities. For instance:
ply)
1 Nh
Σ K
y
Vi
(12)
Here, (-) is the kernel function, & is the bandwidth parameter, and is the dimension of y. Kernel density estimation a non-parametric method for estimating the probability density function p(y) of random vector y, given set of i.idd. samples Y = {y1: . drawn from p(y). Commonly used kernels include the Gaussian and Epanechnikov kernels.
To compute the marginal and joint probability density functions of the samples' outputs at the i-th and j-th layers, we apply kernel density estimation:
(1))
(xx:)
1
K
X
(13)
Nh
h4
Xj
xj
(xx))
(14)
Nh dy
hij
n=1
x (m)
Xj - x
p xx,,
K
K
(15)
N h n=l
h;
where de and dj are the dimensions of X; and Xj, respectively.
To derive the mutual information approximation, let's start with the Monte Carlo approximation of mutual information (Equation 7). Substituting the kernel density estimates (Equations 9-11) into Equation 7, we get:
N
Σ
K
(x))
h4
K
(n)
h5
-x,
(k)
(XxX,)
log
(16)
N
E
N
K
(n)
xxi
Ekmi
K
(n) (N)
loc
by
Now, consider the feature matrix inner product approximation for the kernel function. Let:
(n)
x
x
[n]T x
K
(17)
h;
(n)
(K)
x
xj
(T)T
(2) x,
K
x
(18)
h3
where || ||- ||P denotes the Frobenius norm.
Therefore, the joint kernel function be approximated by:
K ((x)
~ x
(n)T xj
(19)
Substituting these approximations (Equations 13-15) back into the mutual information formula (Equation 12), we get:
(n)T (n)
i(x;X;) ~
log
(20)
TT)
(1)
[T]
(n)
This leads the final expression for the mutual information approximation using Monte Carlo sampling and kernel density estimation with the feature matrix inner product approximation.
16

---

B.2 Redundancy Metric Derivation
Since mutual information has no general upper bound, its upper limit depends on the entropy of either X; or X,, which represent the i-th and j-th layer output of batch input. To address this, we can use functions transform the range of mutual information, ensuring bounded and more interpretable metric. For instance, using e e((;))))) or a Gaussian function, we can redefine the measure as follows. remove the sum and N. then obtain:
Considering |x?xx x,) |1 e((x)))) for batch of inputs,
X X
x X (((o))))
(21)
X
X F
To derive the Redundancy Metric (RM) formula, we consider that batch is a set of samples processed simultaneously by the neural network. Because a batch is input simultaneously, we remove the summation and division by N, obtaining the following formula:
X x, 1F
RM (X,
(22)
X
X
In this formula, RM(.)) E [o, 1], where a value close to indicates high independence between the layers (low redundancy), and value close 1 indicates high redundancy (low independence).
B.3 Constraints and Objective Functions
Our redundancy metric is used to guide the allocation of sparsity ratios across layers. We construct a redundancy matrix & E L where each element vii represents the redundancy between layers i and j. computed using Equation 18. The overall redundancy of each layer is:
(23)
lower P: indicates higher independence. Using monotonically decreasing function, we define the importance factor as:
Wi P
(24)
where μ is hyperparameter controlling the decay rate.
Our objective is maximize the sum of the weighted sparsity ratios across all layers, subject model size constraint B. The sparsity ratio E [0, 1] represents the fraction of weights to be pruned in the i-th layer. The optimization problem can be formulated as:
4
maximizea
Wj
(25)
subject
Σ s(a) <B
(26)
0 05951, Vi E (1,2.....
(27)
where S(a:) represents the model size of the i-th layer under sparsity ratio gi. and q is the vector of sparsity ratios for all I. layers. The objective function (Equa-
(g1, (2,,42,... -.,............................. 4r." [r] tion 22) aims to allocate higher sparsity ratios layers with higher importance factors. The constraint (Equation 23) ensures that the total model size after pruning does not exceed the budget B.
C More Experimental Results
C.1 Zero-Shot results Details
Results in LLaMA-VI, V1, and OPT series at 50% Sparsity
17

---

Table 11: Accuracies (%) for zero-shot tasks with 50% sparsity using LLaMA-VI family.
LJama VI
Method
winogrande
piqu
openbookga
hellawwag
baolg
arc easy
urC challenge
ne
Mean
Dense
70.09
19.16
44,40
76.18
75.
72.5%
44.71
66.79
66.18
Magnitude
59.35
71.38
35.00
60.89
54.56
54.38
37.12
54.51
53.40
Magnitude A7.S
61.25
74.16
36.60
65.62
59,82
59.98
38.31
54.51
56.28
SparseGPT
63.06
73.61
37.40
64.62
64.43
36.50
53.43
56.10
SparseGPT A7.S
66.77
76.33
39,00
68.83
74.28
64.84
39.59
54.87
60.19
Wanda
66.38
74,76
39,00
68,92
70.70
61.74
38.91
50.54
58.87
SparseGPT ALS
66.30
77.26
38.60
69.59
73.70
65.66
40.02
60.65
61.47
Donse
72.77
80,
44.80
79.06
77.89
74.79
47.78
70.76
68.30
Magnirdde
63.38
70.95
39.80
59.69
54.95
54.29
35.84
50.90
53.73
Magnitude w. ALS
68.04
76.39
42.00
71.10
70.70
63.81
40.27
$7.40
61.21
13B
SparseGPT
70.96
76.66
45.20
73.72
74.89
67.47
42.75
62.45
64.26
SparseGPT w. ALS
71.35
77.31
43.60
73.59
74.19
67.64
42.49
61.73
63.99
Wanda
71.51
77.91
43.60
74.13
75.96
69.65
43.77
61.37
64.74
SparseGPT w. ALS
71.35
77.37
43.00
74.34
75.17
69.70
44.45
63.18
64.82
75.85
82,26
48.40
W2.63
K2.72
78.96
52.M0
67.15
7T.36
Magnitude
66.54
75.68
41.20
67.28
64.31
70.75
47.27
50.18
60.40
Magnitude AL.S
69.93
77.04
41.60
68.67
74.19
72.52
46.76
50.18
62.61
30B
SparseGPT
71.03
77.80
42.40
76.06
77.89
73.04 .4
47.44
59.57
65.92
SparxeGPT ALS
74.51
78.78
45.40
78.54
80.43
77.61
52.40
64.62
69.02
Wanda
74.51
79.60
47.40
79.31
80.64
77.57
$1.54
$7.76
68.54
SparseGPT ALS
73.64
80.20
46.80
79,50
81,28
78_37
53.67
61.37
69.35
Donse
77.43
82.26
47.00
84,
84.86
80
55.55
69.00 08
72.39
Magnirude
74.66
79.43
48.00
79.90
79.63
73.65
51.71
62.45
68.68
Magnitude w. ALS
74.98
80.36
48.40
79,91
80.92
74.62
50.85
65.34
69.42
65B
SparseGPT
74.98
81.01
44.40
$0.19
83.46
75.55
48.63
63.
68.93
SparseGPT W. ALS
73.95
80.90
44.80
$0.35
83.33
74.03
49.06
65.70
69.02
Wanda
77.19
80.69
48.60
81.72
84.71
77.56
52.47
70.40
71.71
Wanda w. ALS
76.40
81.39
47.00
81.68
84.68
76.94
52.39
70.76
71.41
Table 12: Accuracies (%) for zero-shot tasks with 50% sparsity using LLaMA-V2 family.
Llama V2
Method
winogrande
piya
openbookqa
hellaswag
boolg
arc are_easy
azc _challennge
ric
Mean
Dense
69.06
79.11
44.20
75.98
77.74
74,49
46.25
62.82
06.21
Magnitude
63.30
73.67
38.80
65.58
62.94
57 70
37.46
57.04
57.06
Magnitude ALS
65.19
75.46
41.40
69.11
71.38
63..47
39.51
55.24
60.09
SparseGPT
63.14
71.71
35.60
63.91
69.05
34.98
54.87
56.44
SparxeGPT ALS
67.96
76.39
40,00
70.52
70.98
67.97
41.13
55.96
61.36
Wanda
67.32
76.99
41.40
68.76
75.78
41.72
53,83
61.88
SparseGPT ALS
67,80
77.10
44.80
70.75
75.47
69.61
42.32
54.87
62.84
Donse
72.38
80.52
45.20
79.39
80.58
77.33
49.15
63.34
68.76
Magnitude
65.27
77.20
73.01
$7.68
67.17
$5.96
59.85
Magnitude w. ATS
68.35
77.48
43.60
74.42
73.15
69.91
44.63
$5.60
63.39
13B
SparseFTT
67.25
75.84
41.20
69.63
68.50
63.76
38.40
56.68
60.16
SparseGPT ATS
71.19
78.13
44.40
74.99
81.16
69.82
43.94
63.18
65.85
Wanda
69.39
78.13
44.10
75.02
80.34
70.37
42.76
55.72
64.48
SparseGPT w. ALS
72.06
78.51
45.80
75.67
81.35
70.33
46.08
62.82
66.58
77.98
82.70
48
83.80
83.79
$1.00
67.87
72.92
Magnitude
73.64
79.54
44.20
79.
71.07
74.79
50.68
60.65
66.73
Magnitude ALS
80.57
79.63
77.99
67.87
70.28
SparseGPT
75.37
79.38
44.80
79.88
82.81
77 02
48.38
67.55 87
69
SparseGPT ALS
76.72
80.36
45.20
$0.09
81.56
77.86
50.17
69.31
70.16
Wanda
77.58
81.18
46.20
80.95
83.94
78.91
$4.69
71.48
71.87
SparseGPT ALS
77.27
81.61
45.80
$1.30)
82.54
78.54
55,80
71.12
71.75
Table 13: Accuracies (%) for zero-shot tasks with 50% sparsity using LLaMA-V3.
Method Dense
winogrande
piqu
openbookga
hellaswwa
boolg
challlengee
Mean
Ilama V3
73.01
80,52
44.80
79.15
81.19
77.61
53.24
68.95
69.81
Magnitude Magnitude w. ALS
52.72
59.90
35.20
29.78
42.34
43.01
29.78
53.07
43.29
65.35
70.95
37.00
65.50
68.90
59.93
37.29
54.51
57.43
SparseFIT
64.17
71.11
35.80
46.52
69.02
58.42
34.64
53.79
54.18
8B SparxeGPT w. ALS
70.72
75.14
41.00
71.14
80.52
70.08
45.31
58.12
64.00
Wanda Wanda w. ALS
66.93
73.39
37.20
47.03
75:90
65.66
39.25
59.57
58.12
70.17
74.97
39.20
67.65
79.14
67.47
42.41
58.84
62.48
18

---

Table 14: Accuracies (%) for zero-shot tasks with 50% sparsity using OPT family.
OPT
Method
winogrande
piqa
openbookqu
hellaswag
boolq
arc caay
are challenge
rie
Mean
Dense
66.19
76.50
38,20)
67.91
Magnitude
50.67
63.17
29.60
32.41
38.01
38,09
23.81
52.71
41,06
Magnitude w. ALS
52.41
64.69
30.00
39.85
40.86
41.46
26.28
50.90
43.31
6.7B
SparseGPT
63.69
74.86
36.60
61.52
62.88
56,73
31.06
54.15
55.19
SparseGPT w. ALS
65.27
76.77
37.40
67.17
65.84
60.23
34.56
55.60
57.85
Wanda
58.17
64.04
29.60
47.17
60.86
47.81
SparseGPT ALS
56.83
65.02
29.00
48.35
60.89
46.13
25.68
51.26
47.89
Dense
67.88
76.77
39.00
69.84
68.37
61.87
35.67
57.76
59.71
Magnitude
48.62
53.26
25.60
26.92
45.02
30.68
24.15
52.71
38.37
Magnitude w. ALS
51.14
54,90
25.60
26.43
61.53
30.98
25.17
51.63
40,92
13B
SpaiseGPT
63.62
74.27
38.40
64.32
62.36
SparseGPT w. ALS
65.27
76.77
39.00
69.85
65.88
62.08
35.84
57.40
59.01
59.67
65.28
30.70
50.63
62.05
28.93
SparseGPT ALS
60.59
65.94
29.80
50.84
61.96
47.83
29.33
$7.74
50.50
Results LLaMA-V2 7B/13B at various sparsity level
Table 15: LLaMA- V2 13B at various sparsity level
winogrande
piga
openhookqa
hellaswag
baolg
are
are challenge
rtc
mean
Magnitude
49.49
53.21
26.00
29.57
38.72
32.20
24.37
52.71
38.38
Magnitude ALS
53.51
62.89
30.40
40.81
60.49
41.58
25.85
52.71
46.03
SpurseGPT
59.27
63,71
35.20
43.81
64.07
48.44
52.71
49.26
SpurseGPT ALS
60.14
62.46
33,60
43.49
65.60
47.10
29.44
53.43
49.41
Wanda
$1.38
57.18
29,80
31.65
52.69
37.21
20.73
53.07
41.71
SparseGPT ALS
$2.96
59.14
28.80
31.85
$9.30
70
22.18
52.71
42.96
Magnitude
$7.22
70.84
33.00
61.11
47.55
51.94
32.08
$2.71
50.81
Magnitude ALS
64.56
40.20
56.32
57.77
SparseGPT
68.43
75.03
40.60
66.70
76.33
62.08
38, 14
58.12
60.68
60%
SparseGPT wr ALS
73.56
40.50
66.50
73.91
38.57
53.43
59.81
Wanda
67.96
76,28
40,20
65.05
78.32
65.45
40.27
56.68
61.27
SpurseGPT w. ALS
69.69
74.37
41.20
65.43
77.83
39.33
$6.68
60.98
Magniuud
65.27
77.20
40.00
73.01
67.17
41,89
35,96
59.85
Magnitude w. ALS
68.35
77.48
43.60
74.42
73.15
69.91
44.63
$5.60
63.39
SparseGPT
67.25
75. 84
41.20
69.63
63.76
38.40
56.68
60.16
SparseGPT w. ALS
71.19
78.13
44.40
74.99
81.16
69.82
43.94
63.18
65.85
69.39
78.13
44.10
80.34
$5.72
SparseGPT w. ALS
72.06
78.51
45.80
75.67
$1.35
70.33
46.08
62.82
66.58
Magnirude
70.58
78.95
45.00
77.33
74.45
47.18
59.21
65.90
Magnitude ALS
70.24
78.67
45.40
77.72
76.94
75.29
48.38
54,87
65.94
SparseGIT
72.38
79.82
46.00
77.37
78.75
73.23
48.46
58.48
66.86
SparseGPT ALS
71.90
77.53
46.40
77.68
79.30
73.27
47.10
59.93
66.64
Wanda
72.22
79,54
45.60
78.55
80.92
73.70
49.06
58, 48
67.26
SparseGPT ALS
71.98
78.29
46.20
78.57
81.22
49.06
59,21
67.27
Magnitude
71.43
80,31
45.00
18.39
79.82
76.14
49.15
60.29
67.63
Magnitude
78.78
45.80
49.15
SparseGPT
73.09
79,82
45.60
78.96
79.91
76.14
501,99
62,09
68.21
30%
SparseGPT w. ALS
72.77
78.78
46.40
78.96
79.97
49.23
61.73
67.97
Wanda
71.82
79.60
46,20
79.20
80.64
76.09
50.26
60.65
68.06
SparseGPT w. ALS
71.59
79.11
46.40
79.07
80.86
75.55
49.91
61.01
67.94
19

---

Table 16: LLaMA-V2 various sparsity level
winogrande
piga
upenbookya
hellaswag
booly
are easy
are challenge
fie
meaan
Magnitude
49.17
31.74
28.00
26.31
37.95
27.74
27.05
53.07
37.63
Magnitude w. ALS
49.57
55.55
27.00
28.39
44.65
34.39
25.34 34
51.26
39.52
SparseGPT
$7.46
60.28
29.00
37.73
62.66
25.34
45.69
70%
SparseGPT ALS
55.64
60.83
31.40
39.03
62.54
44.32
24.74
56.32
46.85
Wanda
51.70
54,52
26.60
30.13
38.93
31.57
20.90
52.71
38.38
SparseGPT w. ALS
50.20
56.37
28.60
30.01
48.10
32.37
21.76
52.71
40.02
Magnitude
53.20
62.46
32.60
42,93
47.71
44.23
29.44
50.90
45.43
Magnitude w, ALS
58.01
66.81
35.20
51.70
63.09
31.74
58.12
52.07
SparseGIT
63.69
71.49
38,40
60.21
67.86
60.23
36,09
52.71
56.33
60%
SparseGPT w. ALS
64,25
70.95
36.00
64.71
64.39
60.98
35.41
53.43
55.79
Wanda
62.51
72.20
37,60
57.45
68.81
34.81
53.79
55.93
SparseGPT w. ALS
63.38
71.93
38.60
57.72
63.64
59.93
34.64
54,51
55.54
Magniuuee
63.30
73.67
65.38
62.94
Magnitude ALS
65.19
75.46
41,40
69.11
71.38
63.47
39.51
55.24
60.09
SparseGPT
63.14
71.71
35.60
63.91
69.05
58.29
34.98
$4.87
56.44
50%
SparseGPT ALS
67.96
76.39
40.00
70.52
70.98
41.13
$5.96
61.36
Wanda
67.32
76.99
41.40
68.76
75.78
69 (9.23
41.72
53.83
61.88
SpurseGPT ALS
70.75
75.47
Magniude
69.14
76,39
42,60
73.12
70.55
69.07
43.17
55.24
62.48
Magnitude w, ALS
68.82
76.82
44.20
74.64
75.87
71.25
45.73
60.29
64.70
SparseGPT
70.64
78.46
43,40
74.28
76.91
54.51
40%
SparseGPT ALS
70.17
77.20
44,00
74.81
76.61
71.00
44.03
34,51
64.04
SparseGPT ALS
68.75
77.53
44,60
74,65
76.12
72.18
45.22
55.23
64,29
Magnitude
70.64
77.48
46.40
76.10
74.25
73.49
45.90
57.40
65.21
Magnitude w. ALS
70.40
78.40
44.40
76.26
77.09
47.35
65.88
SparseGIT
69.22
78.40
45,20
75.81
77.13
72.77
45.48
53.43
64.68
30%
SparseGiPT
Wunda
68.51
78.24
45,40
76.11
77.28
73.11
46.50
55.96
65.14
SparseGPT
68.51
78.89
44.80
76.
77.16
73.57
45.99
58.48
D Visualization of Correlation Matrices
In this section, we present the visualization of correlation matrices obtained by solving the problem under different experimental settings. Additionally, we provide the numerical results of the Sparse Ratio Allocation. The RM heat maps are different with different models.
D.1 Visualization of RM Matrices
50 % sparsity in LLAMA-VI/V//3: family.
LL-MA-VI
LUaMA-VI 138
LLaVA-VY
LLaNA-VI
47
LLOMM-90070 70
LLAMA-D 130
LLSVA-NS
LLaWW-15 C0
01
Figure 3: 50% sparsity in LLAMA-VI/V2/V3 family.
Various sparsity in LLAMA-V2 7B/13B
20

---

11,4M Seacityy
114 ON, faaely
1L 138, Rpendyy
Li Vp Feanitd
Figure 4: various sparsity in LLAMA-V2 V2 7B/13B family.
D.2 Ratio allocation
Figure 5: The sparsity ratio allocation in 50% sparsity in LLAMA-V1/V2/V3 family.
21

---

D.3 Ratio allocation
LL Ind
LL-5AA-40
LL-140-12
LLSM4-10 LL3MA-10
LLOMA-V 150 0.
LLMANO 15
11:00-12 150
LLAM6-12 130
1LaWW-12 180 in
Figure 6: The sparsity ratio allocation in various sparsity in LLAMA- V2 7B/13B family.
E Extra Figures and Explanations
E.1 Experimental Environment and Hyperparameters
Granularity. The granularity for linear optimization results is set to 0.5%, meaning the sparsity percentages can only have decimal places of 0.5% 0.0%. The experiment is based LLaMA- V2 13B, this study in Fig 7 examines the impact of granularity perplexity (PPL) across selected values. Initially, PPL remains relatively constant 10.07 for granularities 0. and 0.5. It then decreases to 9.86 at granularity of 1 and further to 9.67 at granularity of 5. However, beyond this point, the smoothed curve indicates subsequent rise in PPL., suggesting that excessively high granularity may negatively impact model performance. This analysis highlights critical balance in optimizing granularity to minimize PPL and enhance model accuracy and efficiency.
Environment. All pruning experiments are performed on dual NVIDIA A100 GPUs with 80GB memory. However, our ALS method mainly runs on CPU, while the baseline methods Wanda, SparseGPT, and Magnitude require GPU. The CPU used is an AMD EPYCTM 9554 64-core processor.
Hyperparameters We set weight and feature normalization, calibration data= 2, feature selection= in, granularity=0.05, boundes= 0.3-0.7 for 3070% sparsity and 0.1-0.3 for 20% sparsity.
2.75
6.35
Curstaii
Figure 7: The granularity experiment in LLAMA-VES -- 7B.
22

---

E.2
Ratio allocation
Modified Expression
Original R
RMbx X
RMN xX
Figure 8: The comparison of decreasing function
23

---

E.3 Error Bar
In this subsection, present LLaMA-V2 family error bar in 50% sparsity. The standard deviations are runing multiple experiments and get from the EleutherAl LM Hareess [47] package.
Table 17: Standard Deviations for Zero-Shot Tasks with 50% Sparsity using LLaMA-VE V2 Family
(Scaled by 100).
Llama V2
Method
winogrande
piqa
openbookqa
hellaswag
boolg
are_casy
me__hallenge
Magnitude
1.3804
1.0545
1.8664
0.4969
0.8709
1.01000
1.3787
2.9974
Mognitude w. ALS
1,3692
1.0855
0.8575
1.3952
SparseGPT
1,3308
1.0099
2.0272
0.4989
0.7751
0.9426
144125
3.0096
7B
SparseGPT ALS
1.3238
1.771
1 1.9966
0.4990
0.7645
0,9346
14131
2.9953
Wanda
1.3262
1.0047
2.0395
0.4988
0.7765
0,9430
1.4200
2.9882
Wanda w. ALS
1.3285
1.0625
2.0144
0.4989
0.7700
0,9419
14206
2,9406
Magnitude
1,3540
L,0933
0.8702
Magnitude w. ALS
1.3107
1.0283
1 9874
0.49K8
0.8575
0.9350
1.2066
2.9974
SparseGPT
1,2759
0.9869
2 2.0844
0.4974
0.7584
0.9152
14426
2.9148
SparseGPT ALS
1.2707
1.0077
1 1.9966
0.4990
0.7645
0.9346
1.4426
2.9256
Wanda
1,2686
0.9679
2.0951
0.4964
0.7474
0,9116
14491
2,9308
Wanda ALS
1.2707
0.9956
2.0144
0.4969
0.7556
0.8997
1.4455
2,9033
Magnitude
1,2224
0.9494
2.1408
0.4864
0.7044
0.8752
14611
2,9148
Magnitude w.
0.9494
SparseGPT
1.2173
0.9346
2.1236
0.4878
0.6499
0.8407
1,4881
2.9033
70B
SparseGPT
1,2173
0.9346
2.1236
0.4878
0.6499
0.8407
1.4881
2.9033
Wanda
1.1878
0.9298
2.1613
0.4838
0.6216
0.8232
1.4610
2,7073
Wanda w. ALS
1.1878
0.9298
2.1613
0.4838
0.6216
0.8232
1.4610
2.7073
Accuracles for 7 zero-shot tasks with 50% sparsity using LLaMA-V2 (78)
80
Diloct
vinegrnnh
diga
AFC. challenge
30
Accursyy (6)
40
10
Megnotuce
Specselry
SpensedP ALS
Winsla ALS
Ragnitule
Figure 9: The error bar in 50% sparisity experiment in LLAMA-V2 7B.

---

Accuracies for zero-shot with 50% sparsity using LLaMAN2 (1.38)
vinagronde
pigp
operBechi
Accuncy (N6
Megnltune
Magoitude ALS
Spartncory
Spansorrt ALS
Wendaa s ALS
Figure 10: The error bar in 50% sparisity experiment in LLAMA-V2 13B.
Accuracies for zero-shot tasks with 50% sparsity using LLaMA-2 (70B)
Dateset
winggannde
hel sowng
books
60
50
ACCUrSCy
10
Megnofune
Sparsctory
Spertarom ALS
Wersda
Werrta ALS
Hagninute
Figure 11: The error bar in 50% sparisity experiment in LLAMA-V2 70B.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
See abstract and introduction. See abstract and introduction. The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope, providing a clear overview of the research objectives and findings.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Appendix A. Our paper discusses the limitations the work performed by the authors, providing insights into potential weaknesses and areas for improvement.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
26

---

Answer:
[NA]
Justification: Our paper does not include theoretical results, and therefore does not provide assumptions or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: See Experiment. Our paper fully discloses all the information needed to reproduce the main experimental results, including detailed descriptions of the experimental setup, parameters, and methodologies used, which are essential for verifying the main claims and conclusions.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
27

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Our paper provides open access to the data and code, along with sufficient instructions in the supplemental material to enable others to faithfully reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: /tt///ipss.p/.. cc/
publ c / gui des/de///eeee CCodeSubmm i sss onPol cyy for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nniss.. cc/
wttppcccppccc.ccccce puppplllico heollcc/guiiee ggiidee////eeeege CcoeSSumms as onPol inPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:[Yes)
Justification:
See Experiment.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
See Appendix E.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
28

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tt//nnuissps. cc
Answer: [No]
Justification: NO.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification:
Only technical reports.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
29

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [No]
Justification: No.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
license, copyright information, and terms of use package should be provided. For popular datasets, papers rswi thoode. com / da sets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
assets are released,
30

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: No.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
Justification: NO
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NO.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "5g4wQZCQHj", "reviewer": "Reviewer_8iME", "summary": "The growing size of LLMs make deployment increasingly challenging. Traditional pruning methods underperform due to uniform strategies that ignore varying feature importance across layers. The authors introduces a new Adaptive Layer Sparsity (ALS) approach that estimates inter-layer correlations using information orthogonality, accurately evaluating each layer's significance. It then employs linear optimization to selectively prune features in intermediate layers, achieving precise model optimization. Experiments demonstrate superior efficiency of ALS over existing sparse models like Wanda and SparseGPT, especially at high sparsity levels.", "questions": "1. What is the computation efficiency of ALS-pruned model compared with existing baslines?\n2. What is the overhead of solving the linear programming problem?\n3. It seems the performance improvement of ALS decreases when the model becomes large. Is there any reasons for that?", "limitations": "The authors have addressed the limitations and there is no potential negative societal impact of this work.l", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper covers a wide range of related works and discusses the pros and cons of each work, providing a comprehensive background of this work.\n2. Formulating the adaptive pruning with the use of a correlation matrix to measure the importance of each layer and a linear optimization algorithm makes the algorithm itself generalizable and theoretical sound.", "weaknesses": "1. The presentation of this paper should be improved. Long paragraphs with various discussion points are mixed together, which is a major weakness of this paper.\n2. Evaluation could be improved with numbers on the inference efficiency comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 8iME,\n\nThank you so much for the detailed and constructive comments. Please see our responses below to your concerns and questions one by one.\n\n### **Q1**. Improving the paper's presentation.\n\nOur Response:\n\nWe appreciate the reviewer's suggestions about writing. Based on these recommendations, we commit to carefully revising each paragraph. As part of this revision, the paper's clarity will improve through several changes as follows. Long sections will be divided into **concise paragraphs**, each beginning with an **informative topic sentence**. To guide readers, we'll introduce strategic subheadings such as **normalization techniques** and **reweighting strategies**. Furthermore, we'll create **smooth transitions** between paragraphs and sections, ensuring a logical flow of ideas throughout the paper. \n\n### **Q2**.  Evaluation of inference efficiency.\n\nOur Response:\n\nWe've conducted experiments to address concerns. **Table E** in the global rebuttal showcases results for the **LLaMA-V2 7B model** across various sparsity levels:\n\n| Sparsity| Latency (ms) | Speedup|\n| ---------- | ------------ | ------- |\n| 0% (Dense) | 58.20        | 1x      |\n| 50%        | 24.71        | 2.355x  |\n| 70%        | 21.32        | 2.729x  |\n| 80%        | 20.49        | 2.840x  |\n\nResults show that **50% sparsity** achieves a **2.355x speedup**, reducing latency from 58.1968 ms to 24.7099 ms.\n\nThese results highlight our method's ability to maintain model performance while delivering **substantial computational gains**. \n\nNotably, at **50% sparsity**, we observe a **significant reduction in memory usage**, as the **dense model uses 27.6 GB**, while the **50% sparse model** uses only **13.8 GB**, achieving a 50% reduction. For full details, please refer to **Table F** in the global rebuttal.\n\n### **Q3&Q4**.  Computation efficiency compared to existing baselines and overhead of solving the linear programming problem\n\nOur Response:\n\nOur method's computational efficiency is demonstrated through experiments on the **LLaMA V2 7B** model. The total cost of our approach includes two main components: **Redundancy Metric (RM)** calculation (90 seconds) and **Linear Programming (LP)** solution (160 milliseconds). For Wanda/SparseGPT/Magnitude w ALS, these times are added to their respective pruning times.\n\n|               | RM (s) | LP (ms) | Magnitude/Sparsegpt/Wanda (s) | Total (min)|\n| --------------- | :---------------------: | :-----------------------: | :-----------------------------: | :-----------: |\n| Magnitude w ALS | 88.59                 | 169                     | 1.62                          | 1.51        |\n| Sparsegpt w ALS | 91.32                 | 158                     | 1058                          | 19.16       |\n| Wanda w ALS     | 89.47                 | 160                     | 199                           | 4.81        |\n\nIn comparison, **BESA** [50] requires **4.5 hours** for sparsity allocation and pruning, while our approach is **significantly faster**, completing the process in minutes rather than hours.\n\nFor acceleration, we optimize calculations for large models (81 blocks). We use an 80 x 80 RM matrix plus 7 additional 7 x 7 matrices, instead of a large 7 (components within a block) x 80 (blocks) x 7 x 80 matrix.\n\n### **Q5**. Performance improvement decrease for larger models\n\nOur response :\n\nThe apparent decrease in performance improvement for larger models is an interesting trend we’ve observed with our ALS method. This aligns with broader patterns in model pruning and compression, where diminishing returns are common as models grow larger. However, despite this general trend, our ALS method demonstrates consistent performance improvements across various model sizes, with particularly notable effects on larger models and at higher sparsity levels:\n\n**(1). Significance of Relative Gains**\n\nWhile absolute improvements may appear smaller for larger models, the **relative gains** remain substantial. For Llama3 70B at 50% sparsity, our ALS method improves perplexity from 8.49 to **7.24** with SparseGPT, representing a **14.72% relative improvement**. In the context of large language models, such gains can translate to **meaningful performance enhancements** in real-world applications.\n\n**(2). Inherent Capabilities of Larger Models**\n\nLarger models exhibit **a natural resistance** to pruning. As shown in Table 2 of the paper, the LLaMA-V1 65B model at 50% sparsity only increases in perplexity from 4.93 to 7.37 with Wanda, leaving small room for improvement. Nevertheless, our method further reduces this to 7.15. This observation aligns with the findings of Li et al. [1].\n\n**(3). Effectiveness at Higher Sparsity Levels**\n\nOur method demonstrates particular strength at **higher sparsity levels**, especially for the **Llama3 70B** model, as shown in **Tables A and B** in the global rebuttal. For instance:\n\n- At 70% sparsity, Wanda w. ALS **reduces PPL by 32.98** compared to Wanda alone.\n\n- At 60% sparsity, Magnitude w. ALS **reduces PPL by 341.16** compared to Magnitude alone.\n\nThese significant improvements in performance retention at high sparsity levels are crucial for practical deployments where substantial model size reduction is required.\n\n**(4). Alignment with Theoretical Insights**\n\nThe behavior we observe aligns with recent theoretical insights. Liu et al. [2] demonstrated that larger models can maintain performance even under **random pruning**, supporting the idea of their inherent pruning resistance.\n\nIn essence, the ALS method remains **effective across all model sizes**, offering crucial stability and performance benefits, especially at higher sparsity levels.\n\nReference:\n\n[1] Li, Z., et al. (2020). Train big, then compress: Rethinking model size for efficient training and inference of transformers. ICML 2020.\n\n[2] Liu, S., et al. (2022). The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. arXiv:2202.02643.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y7JbaneYpC", "reviewer": "Reviewer_ZqVF", "summary": "The paper presents an approach called Adaptive Layer Sparsity (ALS) for optimizing large language models (LLMs) by selectively pruning features in intermediate layers. The approach consists of two key steps: estimating the correlation matrix between intermediate layers and employing a linear optimization algorithm to develop an adaptive sparse allocation strategy based on the correlation matrix.", "questions": "N/A", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1) The paper introduces a novel approach for optimizing LLMs by considering the varying importance of features across different layers.\n\n2) The approach is based on a well-structured methodology, including estimating the correlation matrix and employing a linear optimization algorithm.\n\n3) Compared with BESA which takes 5 hours on an A100-80G GPU, the proposed method takes less time and is more efficient.", "weaknesses": "1) From Table 1, it seems that when the sparse ratio is low (e.g., 20%), the performance improvement with ALS is marginal. Does that indicate it is useless for not very sparse models?\n\n2) From Table 2, for stronger models (e.g., 65B and 70B), the improvement is far less than smaller models. Therefore, it is questionable about the applicability of the proposed method to strong models.\n\n3) The authors conducted comprehensive experiments on Llama-series models, but omit Llama-3-80B. Are there any justifications? Since Llama-3-80B is one of the strongest models for now, it would enhance the paper if you can include such results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ZqVF,\n\nThank you very much for your detailed and constructive feedback. We will address your concerns and questions as follows.\n\n### **Q1**. Marginal improvement at low sparsity ratios\n\nOur Response:\n\nAt lower sparsity levels (20-30%), improvements from ALS are less pronounced, as even uniform pruning performs well when only a small number of parameters are removed. For example, with **LLaMA-V2-7B** at **20%** sparsity in Table 2 in the paper, perplexity only improves from **8.92 to 8.90** using Wanda w ALS (dense is 8.71), which leave very small room to improve.\n\nHowever, our method maintains **model stability** across various sparsity levels and model sizes, showing **consistent improvements**. It particularly excels at **high sparsities** crucial for real-world deployment, accelerating inference in resource-constrained environments. For instance, with the LLaMA-V2-13B model (Table 1):\n\n**(1).** At **50% sparsity**: Perplexity reduces by **1.88** \n\n**(2).** At **60% sparsity**: Perplexity reduces by **4.91** \n\n**(3).** At **70% sparsity**: Perplexity dramatically reduces from 1.4e3 to **204.17**\n\nThese results demonstrate ALS's effectiveness, particularly in **preventing model collapse**. Specifically:\n\n**(1).** **LLaMA-V2-7B at 60% sparsity**: 83.23 perplexity where Magnitude pruning fails.\n\n**(2).** **LLaMA-V2-13B at 70% sparsity**: 204.17 perplexity where Magnitude pruning fails.\n\n**Llama3 70B** in **Table A** in the global rebuttal, shows that:\n\n**(1).** At **50% sparsity**: Perplexity improves from 8.49 to **7.24** with SparseGPT+ALS\n\n**(2).**  At **60% sparsity**: 13.01 to 12.36\n\nALS **excels** in scenarios requiring substantial model size reduction, offering a **versatile tool** for efficient LLM deployment across various compression levels. Consequently, these results underscore our approach's **effectiveness**, highlighting its potential for efficient deployment in **resource-limited environments**.\n\n### **Q2**. Decreased improvement for stronger models\n\nOur response :\n\nWe have observed that larger models show less performance improvement with our ALS method. This observation aligns with common patterns in model pruning and compression, where returns tend to diminish as models grow in size. However, our ALS method still shows consistent improvements across different model sizes, especially notable in larger models and at higher sparsity levels:\n\n\n\n**(1). Significance of Relative Gains**\n\nWhile absolute improvements may appear smaller for larger models, the **relative gains** remain substantial. For Llama3 70B at 50% sparsity, our ALS method improves perplexity from 8.49 to **7.24** with SparseGPT, representing a **14.72% relative improvement**. In the context of large language models, such gains can translate to **meaningful performance enhancements** in real-world applications.\n\n**(2). Inherent Capabilities of Larger Models**\n\nLarger models exhibit **a natural resistance** to pruning. As shown in Table 2 of the paper, the LLaMA-V1 65B model at 50% sparsity only increases in perplexity from 4.93 to 7.37 with Wanda, leaving small room for improvement. Nevertheless, our method further reduces this to 7.15. This observation aligns with the findings of Li et al. [1].\n\n**(3). Effectiveness at Higher Sparsity Levels**\n\nOur method demonstrates particular strength at **higher sparsity levels**, especially for the **Llama3 70B** model, as shown in **Tables A and B** in the global rebuttal. For instance:\n\n- At 70% sparsity, Wanda w. ALS **reduces PPL by 32.98** compared to Wanda alone.\n\n- At 60% sparsity, Magnitude w. ALS **reduces PPL by 341.16** compared to Magnitude alone.\n\nThese significant improvements in performance retention at high sparsity levels are crucial for practical deployments where substantial model size reduction is required.\n\n**(4). Alignment with Theoretical Insights**\n\nThe behavior we observe aligns with recent theoretical insights. Liu et al. [2] demonstrated that larger models can maintain performance even under **random pruning**, supporting the idea of their inherent pruning resistance.\n\nIn essence, the ALS method remains **effective across all model sizes**, offering crucial stability and performance benefits, especially at higher sparsity levels.\n\n\n### **Q3**. Omission of Llama-3-80B in experiments\n\nOur response\n\nWe have conducted comprehensive experiments on the Llama3 70B model. Our findings, detailed in **Tables A, B and pdf** in the global rebuttal, offer valuable insights into our method's performance on SOTA large language models.\n\n**Overall Trends:**\n\n**(1).** Our ALS method consistently outperforms other pruning methods (Magnitude, SparseGPT, and Wanda) across all sparsity levels (50%, 60%, and 70%).\n\n**(2).** The improvements are particularly pronounced at higher sparsity levels.\n\n**Significant Improvements:**\n\n**(1).** **Extreme Perplexity Reduction at 60% Sparsity:**\n\n- Magnitude pruning: PPL reduces by **341.16** (a remarkable 93.5% reduction)\n\n**(2).** **Consistent Improvements for Top Performers:**\n\n- At 50% sparsity, SparseGPT's PPL reduces by **1.25** (14.7% reduction)\n- At 70% sparsity, Wanda's PPL reduces by **32.98** (21.2% reduction)\n\n**(3).** **Robustness at 70% Sparsity:**\n\n- Magnitude pruning: PPL reduced from 95886.56 to **35844.32** (62.6% improvement). ALS significantly improves performance in challenging scenarios.\n\nIn conclusion, our ALS method demonstrates **consistent improvements** across various sparsity levels and particularly **excels** at **higher sparsities**, maintaining reasonable performance even at 70% sparsity.\n\nReference:\n\n[1] Li, Z., et al. (2020). Train big, then compress: Rethinking model size for efficient training and inference of transformers. ICML 2020.\n\n[2] Liu, S., et al. (2022). The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. arXiv:2202.02643.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bba0kNjk82", "reviewer": "Reviewer_7kYG", "summary": "This paper proposes using mutual information to measure layer redundancy in LLMs and employs a linear optimization algorithm based on this measure to derive adaptive sparsity strategies, enabling dynamic sparsity configuration across different layers. The authors conducted experiments on four models, including three versions of LLaMA and OPT, comparing the proposed approach with existing pruning algorithms such as Wanda and SparseGPT. The results demonstrate that the proposed method outperforms these existing algorithms in most scenarios.", "questions": "- How is the intra-layer pruning calculation performed?\n- How is ALS integrated with the baseline method?\n- What is the meaning of L-i+1 in equation (5)?\n- The analysis in lines 274-276 refers to the 13B model, but the data corresponds to the 7B model in the table, which is confusing. Could you clarify this discrepancy?\n- What would be the effect of combining Wanda+ALS with a small amount of LoRA fine-tuning?\n- What is the relationship between sparsity and efficiency as studied in this paper? Does it reduce computation time or memory usage?", "limitations": "The paper lacks some crucial details in certain areas. For instance, in section 4.3, the process of LoRA fine-tuning is mentioned but not adequately explained. It would be beneficial if the authors could provide more comprehensive information on this aspect.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The perspective of using mutual information is intuitive and helps identify redundant layers in the model effectively.\n- The pruning algorithm is efficient, requiring only a short time to complete the process.\n- The experiments on LLaMA-3 are significant, demonstrating the effectiveness of non-uniform sparsity allocation across layers.", "weaknesses": "While the combination with ALS effectively improves the baseline performance, there are concerns about the practical value of the method. Specifically, when comparing the pruned models with dense models of similar size, the reported performance is relatively close. An extreme example of this is found in Table 2, where the perplexity (PPL) of the 50% sparse LLaMA-v1 30B model is higher than that of the dense LLaMA-v1 13B model, regardless of the pruning method used. This raises questions about the overall effectiveness of the pruning approach in terms of model efficiency and performance trade-offs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 7kYG,\n\nThank you for your valuable and insightful comments. We have provided detailed responses to your concerns and questions below.\n\n### **Q1**. Pruned models' performance compared to dense models\n\nOur response:\n\n**(1). Uncommon and Unfair Comparison in Sparsity Research**\n\n- Comparing sparse models to dense models of similar size is **uncommon in sparsity research**. Typically, sparse large models are compared to sparse smaller models, as comparing them to unpruned models is generally considered unfair.\n- Sparse models underperforming similarly-sized dense models is common with SOTA methods like Wanda[41], SparseGPT[13], and Magnitude[26] pruning. This is **not specific to our approach**, but rather a characteristic of current sparsification techniques. This comparison is not the primary focus of our work.\n- Obtaining and post-training sparse large models is **cost-effective** using pre-trained models. For example, 4 GPU hours in A100 80GB of LoRA training achieve **better performance** and **faster inference** than dense models. Training a dense model like Llama2 7B from scratch requires 184,320 GPU hours.\n\n\n**(2). Performance Improvements and Efficiency Gains**\n\n- **Tables 2 and 3** in the paper show that our ALS method **consistently improves performance** across various model sizes and families at 50% sparsity. Notable examples include:\n\n  **(a)**. LLaMA-V3 8B:\n\n  ​\tMagnitude pruning: Perplexity reduces by1069 (from 1.1e3 to 30.20)\n\n  ​\tSparseGPT: Accuracy increases by 9.82%\n\n  **(b).** LLaMA-V2 13B:\n\n  ​\tSparseGPT: Perplexity reduces by 1.88\n\n  ​\tWanda: Accuracy increases by 2.1%\n\n- **Further improvements with LoRA fine-tuning (Table C and D in the gobal rebuttal):**\n\n  **(a).** LLaMA-V1 30B (50% sparse) compared to 13B dense:\n\n  ​\tPerplexity: Reduced by **1.21** \n\n  ​\tAccuracy: Increased by **2.94%** \n\n  **(b).** LLaMA-V2 13B (50% sparse) compared to 7B dense:\n\n  ​\tPerplexity: Reduced by **0.13** \n\n  ​\tAccuracy: Increased by **0.53%** \n\n- **Inference acceleration:**\n\n  **(a).** LLaMA-V2 7B at 50% sparsity: 2.355x speedup in inference time\n\n  **(b).** LLaMA-V2 13B (50% sparse) vs 7B dense: Better throughput (0.0118 vs 0.0084 items/sec), Lower   latency (41.24 vs 58.20 ms)\n\n- Significant decrease in memory usage, at 50% sparsity, **47.5% memory reduction** as shown in **Table F** in the global rebuttal.\n\n### **Q2&Q3**. Intra-layer pruning calculation method and Integration of ALS with baseline methods\n\nOur response:\n\nThe integration of **ALS with baselines** involves three main steps, ensuring a **fine-grained** and **adaptive sparsity distribution** at both inter-layer and intra-layer levels:\n\n\n**(1). Calculate adaptive sparsity ratios for blocks**\n\n- Compute the **Redundancy Metric (RM)** matrix for layers (Equation 4)\n- Apply **Linear Optimization** (Section 3.3)\n\n**(2). Calculate adaptive sparsity ratios for sub-blocks**\n\n- Compute RM matrix for intra-layer components\n- Use Linear Optimization for optimal intra-layer sparsity\n\n**(3). Apply sparsity ratios to baseline pruning methods**\n\n- Use the calculated ratios as input for existing pruning techniques\n\nThis approach ensures a **hierarchical, adaptive sparsity distribution** throughout the network architecture. The detailed algorithm pipeline is illustrated in **Figure 1** of the paper\n\n*Note: Detailed implementation is in the submitted code, to be released upon paper acceptance*\n\n### **Q4**. Meaning of L-i+1 in equation (5)\n\nOur response:\n\nTo clarify the meaning of **L - i + 1** in equation (5), it's important to note that this term plays a crucial role in calculating each layer's **average independence** with **subsequent layers**:\n\n**Calculation method**: For layer i, we compute its independence with layers i+1 to L, as shown in Appendix B.3, line 595.\n\n**Examples**:\n\n**(1).** Layer 2: average over L-1 layers (divided by L-1). Subsequent layers follow this pattern\n\n**Benefits of this approach**:\n\nThis formulation captures the **hierarchical nature** of neural networks and provides **adaptive sparsity allocation**. **Adaptive weighting** prioritizes earlier layers, ensuring **independence** from previous layers and reflecting each layer's importance.\n\n### **Q5**. Discrepancy in analysis of 13B vs 7B model\n\nOur response:\n\nThank you for identifying this typo, which should refer to the 7B model. We will fix this typo in the revision.\n\n### **Q6 & Limitation**. Effect of combining Wanda+ALS with LoRA fine-tuning\n\nOur response:\n\nIn addition to the LoRA experiment presented in Table 5 of the paper, we have conducted further experiments. The effect of combining **Wanda+ALS with LoRA fine-tuning** copmaring with **Wanda+ALS**  is significantly positive (**Table C and D** in the global rebuttal).\n\n**(1). For LLaMA-V1** at 50% sparsity: **30B** have PPL reduced by **1.43** and ACC rose by **2.09%**; **13B** have PPL reduced by **1.15** and ACC rise by **1.89%**; **7B** have PPL reduced by **4.82** and ACC rose by 10.17%.\n\n**(2).** **For LLaMA-V2**: **13B** have PPL reduced by **1.56** and ACC rose by **1.23%**; **7B** have PPL reduced by **1.72** and ACC rose by 2.21%.\n\nNotably, LoRA fine-tuning uses 2000 C4 samples (same dataset as Llama pretraining) in a zero-shot setting, unrelated to evaluation tasks.\n### **Q7**. Relationship between sparsity and efficiency\n\nOur response:\n\nOur study shows **significant improvements** in computation time and memory usage with sparsity:\n\n**(1).** **Computation Time & Memory Usage**, **Table E,F** in the global rebuttal:\n\n- At 50% sparsity: **2.355x speedup, 47.5% memory reduction**\n- At 70% sparsity: **2.729x speedup**, **63.9% memory reduction**\n\n**(2).** **Practical benefits**: Even at moderate sparsity (50%), significant improvements in both speed and memory are achieved\n\nIn conclusion, our sparsification approach significantly reduces **computation time and memory usage** without substantial performance loss, while also supporting 2:4 and 4:8 structured pruning (see Appendix C).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EVdDgadHZk", "reviewer": "Reviewer_2MGm", "summary": "This paper introduces Adaptive Layer Sparsity (ALS), a novel approach aimed at optimizing large language models (LLMs) through selective pruning. The key contributions of this work include a method that estimates the correlation between intermediate layers using information orthogonality, enabling the precise measurement of each layer's importance. Additionally, the paper formulates the sparsity allocation problem as a linear programming optimization, facilitating efficient global optimization of layer-wise sparsity ratios. Extensive experiments conducted on various LLM families, including LLaMA and OPT, demonstrate consistent performance improvements over existing pruning methods across different model sizes and sparsity levels. The authors also perform analysis experiments to examine the impact of various factors, such as calibration data, sparsity bounds, and feature selection, on the method's performance.", "questions": "See weakness 2. Could you provide some details about the normalization and reweighting strategies mentioned in the ablation study section?", "limitations": "N/A", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper presents a novel approach for optimizing LLMs. ALS addresses significant challenges in LLM compression, particularly the challenge of manual sparsity setting and suboptimal performance due to uniform pruning ratios.\n\n2. The experimental design of this research is solid and comprehensive. The authors conduct extensive experiments on various LLM families (LLaMA-V1/V2/V3, OPT) with different parameter sizes, providing a robust evaluation of their method's effectiveness.\n\n3. The proposed method, ALS, is simple and effective, and has good computational efficiency.\n\n4. The paper is well-structured and clearly written. The authors provide a detailed explanation of their methodology, including the key steps of estimating the correlation matrix and employing linear optimization for sparse allocation strategy.", "weaknesses": "1. There is significant overlap between the introduction and related work sections, which could be consolidated to improve flow and reduce repetition.\n\n2. Some details about ALS, such as normalization and reweighting strategies mentioned in the ablation study section, are not clearly described in the \"method\" section. These should be elaborated with formulas in the method section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 2MGm,\n\nThank you so much for the detailed and constructive comments, and the recognition of the novelty of the proposed method, the writing, and the experimental evaluation on image classification benchmarks. Please see our responses below to your concerns and questions one by one.\n\n### **Q1**. Overlap between introduction and related work sections:\n\nOur response: \n\nWe appreciate the reviewer's keen observation regarding the overlap between our introduction and related work sections.\n\nTo address this issue and enhance the overall coherence of our manuscript, we will do  the following reorganization:\n\n**(1).** **Introduction**: We'll streamline this section to focus on:\n\n- Presenting the **core problem** of LLM compression\n- Outlining our **motivation** for developing a new approach\n- Providing a **high-level overview** of our key contributions and approach\n\n**(2).** **Related Work**: Correspondingly, the Related Work section will be expanded and refined to:\n\n- Offer a more **in-depth analysis** of existing methods\n- **Clearly distinguish** our approach from previous work\n- Provide a **critical evaluation** of the strengths and limitations of current techniques\n\nBy implementing these changes, we aim to improve the **logical flow** of the paper, eliminate redundancy without sacrificing important content, and offer readers a **clearer roadmap** of the field and our contributions to it.\n\n### **Q2**. Lack of details on normalization and reweighting strategies\n\nOur response: \n\nThank you for highlighting the need for more details on our normalization and reweighting strategies.\n\nTo address this concern, we'll add a new subsection titled \"**Normalization and Reweighting Strategies**\" to our methodology section. This addition will include:\n\n**(1).** **Feature Normalization**: We'll detail our approach to standardizing input features across layers, explaining how this helps balance the influence of different feature scales. For example, we'll expand on the process mentioned in Section 3.2, line 185, which introduces feature normalization.\n\n**(2).** **Weight Normalization**: We'll describe our process for normalizing layer weights, which is key to ensuring fair comparisons across the network. We'll elaborate on the statement in Section 3.3, starting from line 220.\n\n**(3).** **Reweighting Mechanism**: We'll introduce our novel reweighting strategy that adjusts the importance of layers based on their position and contribution to the model's output.\n\n**(4).** **Integration with ALS**: Finally, we'll explain how these techniques are seamlessly integrated into the broader ALS framework, demonstrating how they enhance its overall effectiveness.\n\nBy including this information, we aim to provide a clearer, more comprehensive picture of our approach.\n\nWe appreciate your attention to detail, as it allows us to present a more robust and transparent description of our research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers, Area Chairs, and Program Chairs,

We sincerely thank all four reviewers for their feedback and constructive comments. In the initial review, 3 Accept ratings are given. Reviewers have acknowledged the **novelty**, **theoretical soundness**, **impact**, **efficiency**, **comprehensive evaluation**, **performance**, and **clear presentation** of our work.

**[Novelty]:**

-  Reviewer ZqVF: Novel LLM optimization considering layers
-  Reviewer 2MGm: Novel LLM optimization approach

**[Theoretical Soundness]:**

-  Reviewer 8iME: Adaptive pruning generalizable, theoretically sound
-  Reviewer ZqVF: Well-structured methodology

**[Impactful]:**

-  Reviewer 2MGm: Addresses LLM compression challenges effectively
-  Reviewer 7kYG: Intuitive redundant layer identification method
-  Reviewer 8iME: ALS evaluates layer significance accurately

**[Efficiency]:**

-  Reviewer ZqVF: More efficient than BESA
-  Reviewer 7kYG: Efficient, quick pruning algorithm process
-  Reviewer 2MGm: Simple, effective, efficient method
-  Reviewer 8iME: Superior efficiency at high sparsity

**[Performance]:**

-  Reviewer 7kYG: Effective non-uniform sparsity allocation demonstrated
-  Reviewer ZqVF: Outperforms existing algorithms in scenarios
-  Reviewer 2MGm: Outperforms existing algorithms in scenarios

**[Comprehensive Evaluation]:**

-  Reviewer 2MGm: Solid, comprehensive experimental design conducted
-  Reviewer 8iME: Superior efficiency over existing models
-  Reviewer ZqVF: Comprehensive experiments on Llama-series models

**[Clear Presentation]:**

-  Reviewer 8iME: Comprehensive background of related works
-  Reviewer 2MGm: Well-structured, clear methodology explanation provided

We've conducted additional experiments to address comments. We'll revise our manuscript to strengthen our work and address concerns.

Best wishes,

Authors

---
Additional Experiments (details  in the pdf file):

### Llama3 70B ALS experiments series

#### Table A: Wikitext

| Sparsity               | 50%      | 60%       | 70%       |
| ---------------------- | -------- | --------- | --------- |
| Dense                  | 2.92     | 2.92      | 2.92      |
| Magnitude              | 19.29    | 364.92    | 95886.56  |
| ***Magnitude w. ALS*** | 13.21    | 23.76     | 35844.32  |
| SparseGPT              | 8.49     | 13.01     | 50.937    |
| ***SparseGPT w. ALS*** | 7.24     | 12.36     | **49.14** |
| Wanda                  | 7.01     | 10.41     | 155.603   |
| ***Wanda w. ALS***     | **6.82** | **10.05** | 122.62    |

#### Table B: Averaged accuracies(%)

| Sparsity               | 50%       | 60%       | 70%       |
| ---------------------- | --------- | --------- | --------- |
| Dense                  | 75.43     | 75.43     | 75.43     |
| Magnitude              | 51.28     | 44.50     | 39.62     |
| ***Magnitude w. ALS*** | 53.64     | 45.97     | 39.89     |
| SparseGPT              | 70.26     | 58.66     | **43.83** |
| ***SparseGPT w. ALS*** | 71.12     | 58.61     | 40.34     |
| Wanda                  | 72.25     | 66.03     | 40.56     |
| ***Wanda w. ALS***     | **73.12** | **66.34** | 42.4      |

### Llama-V1,V2 ALS w Lora experiments series

#### Table C: Wikitext

| Models                    | V1-7B    | V1-13B   | V1-30B   | V2-7B    | V2-13B  |
| ------------------------- | -------- | -------- | -------- | -------- | ------- |
| Dense                     | 9.38     | 8.2      | 6.09     | 8.71     | 7.68    |
| Wanda                     | 13.3     | 10.9     | 8.74     | 12.31    | 11.21   |
| Wanda w. ALS              | 12.47    | 10.4     | 8.42     | 11.61    | 9.86    |
| ***Wanda w. ALS & Lora*** | **7.65** | **9.25** | **6.99** | **9.89** | **8.3** |

#### Table D: Averaged accuracies(%)

| Models                    | V1 7B     | V1 13B    | V1 30B    | V2 7B | V2 13B    |
| ------------------------- | --------- | --------- | --------- | ----- | --------- |
| Dense                     | 66.18     | 68.5      | 71.36     | 66.21 | 68.76     |
| Wanda                     | 58.87     | 64.74     | 68.54     | 61.88 | 64.48     |
| Wanda w. ALS              | 61.47     | 64.82     | 69.35     | 62.84 | 66.58     |
| ***Wanda w. ALS & Lora*** | **71.64** | **66.71** | **71.44** | **65.05** | **67.81** |

### Latency Experiment

#### Table E: Llama V2 7B

| Sparsity               | Dense   | 30%    | 40%     | 50%     | 60%     | 70%     | 80%     |
| ---------------------- | ------- | ------ | ------- | ------- | ------- | ------- | ------- |
| Latency (ms)           | 58.1968 | 58.136 | 33.0388 | 24.7099 | 23.4007 | 21.3224 | 20.4892 |
| Throughput (items/sec) | 0.0084  | 0.0084 | 0.0148  | 0.0198  | 0.0209  | 0.0229  | 0.0238  |
| Speedup                | 1x      | 1.001x | 1.761x  | 2.355x  | 2.487x  | 2.729x  | 2.840x  |

### Memory Usage with unstructured Sparsity for Llama V2-7B

#### Table F: Memory usage 

| Configuration | Memory Usage (GB) |
| ------------- | ----------------- |
| FP32 Dense    | 26.3              |
| FP32 30%      | 19.1              |
| FP32 40%      | 16.2              |
| FP32 50%      | 13.8              |
| FP32 60%      | 11.9              |
| FP32 70%      | 9.5               |

Meta-review: The paper introduces Adaptive Layer Sparsity, a method to optimize LLMs by selectively pruning less important features in intermediate layers based on a correlation matrix, thereby reducing model size without compromising performance. Through extensive experiments, ALS demonstrates good performance compared to existing methods.

The paper is well-written and easy to follow. The reviewers had some questions and concerns about the experiments, technical details, and the motivation behind the paper. During the rebuttal, the authors addressed these issues, and the reviewers were satisfied with the responses and additional experiments. Therefore, I recommend accepting the paper.
Decision: Accept (poster)