Title: Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment
Venue: NeurIPS 2024
Accepted: yes
Keywords: layerwise sparsity allocation

large language models

model compression

network pruning.
Abstract: Large Language Models (LLMs) have revolutionized the field of natural language processing with their impressive capabilities. However, their enormous size presents challenges for deploying them in real-world applications. Traditional compression techniques, like pruning, often lead to suboptimal performance due to their uniform pruning ratios and lack of consideration for the varying importance of features across different layers. To address these limitations, we present a novel  Adaptive Layer Sparsity (ALS) approach to optimize LLMs. Our approach consists of two key steps. Firstly, we estimate the correlation matrix between intermediate layers by leveraging the concept of information orthogonality. This novel perspective allows for a precise measurement of the importance of each layer across the model. Secondly, we employ a linear optimization algorithm to develop an adaptive sparse allocation strategy based on evaluating the correlation matrix. This strategy enables us to selectively prune features in intermediate layers, achieving fine-grained optimization of the LLM model. Considering the varying importance across different layers, we can significantly reduce the model size without sacrificing performance. We conduct extensive experiments on publicly available language processing datasets, including the LLaMA-V1|V2|V3 family and OPT, covering various benchmarks. Our experimental results validate the effectiveness of our ALS method, showcasing its superiority over previous approaches. The performance gains demonstrate its potential for enhancing LLMs' efficiency and resource utilization. Notably, our approach surpasses the state-of-the-art models Wanda and SparseGPT, showcasing its ability to excel even under high sparsity levels.  Codes at: https://github.com/lliai/ALS.

Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment

Abstract

Large Language Models (LLMs) have revolutionized the field of natural language processing with their impressive capabilities. However, their enormous size presents challenges for deploying them in real-world applications. Traditional compression techniques, like pruning, often lead to suboptimal performance due to their uniform pruning ratios and lack of consideration for the varying importance of features across different layers. To address these limitations, we present a novel Adaptive Layer Sparsity (ALS) approach to optimize LLMs. Our approach consists of two key steps. Firstly, we estimate the correlation matrix between intermediate layers by leveraging the concept of information orthogonality. This novel perspective allows for a precise measurement of the importance of each layer across the model. Secondly, we employ a linear optimization algorithm to develop an adaptive sparse allocation strategy based on evaluating the correlation matrix. This strategy enables us to selectively prune features in intermediate layers, achieving fine-grained optimization of the LLM model. Considering the varying importance across different layers, we can significantly reduce the model size without sacrificing performance. We conduct extensive experiments on publicly available language processing datasets, including the LLaMA-V1|V2|V3 family and OPT, covering various benchmarks. Our experimental results validate the effectiveness of our ALS method, showcasing its superiority over previous approaches. The performance gains demonstrate its potential for enhancing LLMs' efficiency and resource utilization. Notably, our approach surpasses the state-of-the-art models Wanda and SparseGPT, showcasing its ability to excel even under high sparsity levels. Codes at: https://github.com/lliai/ALS.

1 Introduction

Large language models (LLMs) [62, 49, 3] have demonstrated remarkable performance in various natural language processing (NLP) [55, 54, 4] tasks. However, their size and computational requirements pose significant challenges for widespread adoption and deployment. To address these practical constraints, model compression techniques, such as weight pruning and quantization, can potentially reduce the size and computational requirements of LLMs.

The emergence of LLMs has revolutionized the field of NLP. However, despite their revolutionary impact, the massive scale and complexity of LLMs presents significant challenges for model compression. Conventional pruning methods [27, 19, 38, 19, 15, 59], which often require one or more

𝜔௜

= 𝑒

ି

௅

∑ ೕసభ

ಽ

∑ ೕసభ

ಽ

𝜔௜

= 𝑒

ି

௅

∑ ೕసభ

ಽ

∑ ೕసభ

ಽ

𝑚𝑎𝑥௤ ෍

𝑞௜

𝐿 - 𝑖 + 1 ෍𝜔௜

௅

௜

௅

௜

⋯

⋯

Embedding

Head

0

1

L

i-1

i

Objective:

Constraints:

෍𝑆 (௤೔)

௅

௜

⋯

⋯

Embedding

Head

0

1

L

i-1

i

𝜔

𝜔

𝜔

𝜔

𝜔

𝜔

𝜔

𝜔

𝜔

Original LLMs

Redundancy Metric

Linear Optimization

Adaptive Layer Sparsity

Constraint2

Constraint1

Feasible

Region

016

X

iterations of fine-tuning or retraining to preserve performance, have become impractical for LLMs due to the substantial computational cost and time required.

Due to the failure to the Magnitude approach to pruning [28] and other previous methods on LLMs, recent efforts such as SparseGPT [17], Wanda [46], DSOT [65], Pruning Large Language Models with BESA [57], and OWL [58] aim to address this challenge by reconstructing the layerwise outputs of LLMs. Specifically, SparseGPT introduces a technique for pruning less significant weights and reconstructing layerwise outputs based on an importance metric derived from the Hessian matrix. To reduce the computational overhead of SparseGPT, Wanda proposes a simplified strategy that relies solely on the product of weight and activation magnitudes for pruning. DSOT computes the reconstruction error incrementally for each layer, optimizing the intra-layer sparse configuration through further weight pruning or growth, which forms the basis for subsequent weight recovery and additional pruning operations. These methods adopt a training-free approach. In contrast, BESA [57] proposes learning the optimal pruning ratio within each layer through training, finding that considering the overall sparsity configuration within a layer enhances the performance of sparse models. However, this method primarily focuses on intra-layer sparsity configuration. It requires substantial training time, typically taking at least 5 hours on an A100-80G GPU, which is considerably slower than other training-free techniques [17, 46, 65]. Another notable method is OWL [58], which proposes a non-uniform layerwise sparsity technique that assigns different sparsity ratios based on the outlier ratio within each layer, leveraging the unique characteristic of LLMs where some features exhibit significantly larger magnitudes by tuning hyperparameters such as the outlier threshold and sparsity upper/lower bounds to obtain optimal parameter setting. Nevertheless, unlike the aforementioned methods, OWL relies heavily on empirical analysis without providing a solid theoretical foundation for its effectiveness.

However, existing methods have several significant drawbacks. First, for BESA, DSOT, and some traditional techniques, minimizing the layer-by-layer pruning error does not effectively mitigate the impact of pruning on model performance, as the pruning error accumulates across layers due to its inherent greedy nature [24] and may also become trapped in local optima [13, 22]. Second, LLM pruning methods such as Wanda, SparseGPT, and Magnitude apply uniform sparsity ratio to each layer, despite the significant variations in each layer's contribution to the final model performance [57, 65]. To achieve better performance for different layers, the sparsity needs to be manually adjusted for all layers. Third, for the newly proposed OWL method, more theoretical analysis is needed on why its outlier-based non-uniform sparsity outperforms uniform sparsity. Moreover, the choice of hyperparameters in OWL, such as the outlier threshold and sparsity upper/lower bounds, is sensitive to model performance, but their optimal ranges are not theoretically explained, and the effective ranges and thresholds are derived through manual tuning. Furthermore, the transferability of these hyperparameters across different datasets has yet to be systematically studied. Therefore, when applying OWL to new models, complex adjustments by hand must be performed to determine the potentially optimal parameter combination.

To address the multiple challenges of getting trapped in local optima, manually setting sparsity for all layers, and relying on empirical manual experiments to derive optimal sparsity ratios, we propose a simple, effective, and efficient method called Adaptive Layer Sparsity (ALS) for allocating sparsity ratios. The overall pipeline of our proposed method is illustrated in Fig. 1. This technique optimizes

the pruning rate across different layers. To the best of our knowledge, this is the first attempt to reformulate the sparsity allocation problem in LLMs as a linear programming problem. We tackle these challenges by constructing an objective function and constraints. The constraint of the linear programming problem is that the total number of parameters should be less than the target model size. We then compute the independence matrix [25] at both the layer level and intra-layer component level based on the output or input features. According to our experiments, the independence between layers is positively correlated with model performance, as shown in Fig. 2 (c). Therefore, maximizing the independence between each layer of the model is considered our objective function. Unlike existing black-box optimization methods, we formulate this problem as a linear one that can be solved by any linear problem solver. This approach enables efficient global sparsity ratio allocation for LLMs ranging from 7B to 70B parameters on a single A100-80GB GPU. If the model scale is too large, reaching 160B, we can also perform multi-threaded computation on a CPU. For a 70B model, the global sparsity configuration can be obtained in just 20 minutes on an A100-80G GPU.

To rigorously assess the efficacy of ALS, we conducted extensive experiments on diverse LLMs, including LLaMA-V1 [49], LLaMA-V2 [50], LLaMA-V3 [1], and OPT [62] model families, with parameter counts ranging from 6.7 billion to 70 billion. In the main experiments, we evaluated the WikiText-2 perplexity and average accuracy on 7 zero-shot datasets at various sparsity ratios (20% to 70%) for LLaMA-V2 7B/13B (Table 1) and at 50% sparsity for all model families (Tables 2 and 3). Detailed results for each zero-shot dataset on LLaMA-V2 family models at 50% sparsity are presented in Table 4. The analysis experiments consist of 6 sets, examining the impact of calibration data, sparsity bounds setting, and model redundancy on performance (Fig. 2), as well as the influence of feature selection, standardization, and comparisons with Wanda and LoRA fine-tuning. Additional experiments including detailed of main experiments, analyses of granularity, decreasing functions, visualizations of layer redundancy, sparsity ratio allocation and comparison with OWL method are provided in the Appendix C and D. These experimental results unequivocally demonstrate that ALS consistently yields substantial performance improvements for sparse LLMs across various LLMs and downstream tasks.

2 Related Work

Model Compression method try to design efficient models and reduce the memory and computational requirements of LLMs. These methods can be categorized into quantization [42, 12, 35, 33], sparsification [17, 46, 10, 9] and distillation [56, 29, 30, 31, 32, 11, 53]. Quantization converts high bit-width weights and activations into compact, low bit-width representations, while sparsification increases the proportion of zero-valued elements in model weights. Our method optimizes LLM sparsification by strategically allocating sparsity across the model's layers to maximize performance and minimize computational overhead. In contrast to optimization-based compression techniques ( e.g. , OMPQ [35]) for CNN models in vision tasks, our approach focuses on different LLM models and NLP tasks and devises various functions and strategies accordingly.

Sparsity in LLMs has garnered significant attention as a means to accelerate inference speed and reduce memory consumption by applying sparsity in the model weights or activations. sparsity techniques can be categorized into two main approaches: structured pruning [34, 23] and unstructured pruning [16, 64, 46, 63]. While the primary focus of these works lies in determining the pruning criteria, such as weight importance and pruning ratio, the enormous parameter scale of LLMs presents an additional challenge in terms of pruning efficiency. Conventional pruning methods [15, 59, 63, 23, 27, 19, 38, 19], dating back to the early work of Hassibi [20] in the 1990s, which successfully reduced model size and improved efficiency in deep learning architectures by removing redundant weights to create sparse and lightweight models, heavily rely on extensive retraining and are often infeasible for LLMs due to prohibitively high computational overhead and prolonged training times. To address this issue, researchers have developed LLM-specific pruning techniques that prioritize train-free and time efficiency. In the context of structured pruning, LLMpruner [34] explores the application of structured pruning to LLMs and employs LoRA to recover the performance of the pruned model. For unstructured pruning, SparseGPT [17] stands out as a notable method that draws inspiration from the Optimal Brain Surgeon (OBS) [20] approach, taking into account the impact of removing individual weights on the network reconstruction loss. SparseGPT introduces an efficient technique for estimating the Hessian matrix, enabling the application of the traditional OBS method to largescale models. Another prominent unstructured pruning method, Wanda [46], employs a simple yet

effective strategy based on the product of weight and activation values to identify and eliminate less important weights, further enhancing the pruning speed. Despite these advancements, most existing methods adopt a uniform pruning rate across all layers, which may lead to suboptimal performance. In contrast, our approach introduces a novel layer adaptive pruning strategy that dynamically allocates sparsity based on the importance of each layer, effectively minimizing performance degradation while achieving high compression ratios.

Sparsity Allocation in Network Pruning. Conventional methods for achieving adaptive layerwise sparsity in neural networks [14, 5, 26] often rely on a layer-by-layer pruning approach, where the objective is to minimize the sum of errors introduced in each layer. However, this greedy strategy [24] leads to the accumulation of errors across layers, resulting in suboptimal performance when directly adapted to LLMs. The extensive retraining required on vast datasets further amplifies the challenges of applying these techniques to LLMs. Recent efforts, such as BESA [57] and DSOT [65], have shifted focus to intra-block sparsity allocation, employing various strategies to optimize the sparsity distribution within individual blocks. Despite operating at a finer granularity, these methods fundamentally adhere to a layer-wise pruning paradigm, neglecting the importance of global sparsity allocation. Consequently, the resulting allocation may be locally optimal [13, 22] within each layer but globally suboptimal, potentially leading to solutions stuck in local optima. Recently, a new approach called OWL [58] attempts to address this issue by introducing a non-uniform layer-wise sparsity technique. This technique primarily relies on manually tuning the outlier threshold and sparsity upper/lower bounds (which are very small values and sensitive to performance) through extensive experimentation to obtain potentially optimal parameter configurations. Although OWL demonstrates the potential for improved sparsity allocation, it heavily depends on empirical analysis and fails to provide a solid theoretical foundation for its effectiveness, limiting its generalizability and robustness across different LLMs architectures and datasets.

3 Methodology

3.1 Preliminary

Pruning LLMs is a method that aims to obtain a sparse representation of the model by eliminating a predetermined fraction of the pre-trained weights. The primary objective is to minimize the divergence between the outputs generated by the sparse and dense models [21]. However, directly tackling this problem can be challenging due to the massive scale of LLMs. We discover that the mutual information entropy in Eq. 1 can effectively quantify the degree of discrepancy between different layers of the model.

Neural networks can be decomposed into a sequence of layers. In the decomposed form, we represent the neural network as F = { f 1 , f 2 , . . . , f L } , For a given random sample x 0 ∈ R d 0 , let x i = f i ( f i -1 ( . . . f 1 ( x 0 ))) ∈ R d i represents the output of the random sample at the i -th layer.

Based on the previous definitions of the marginal entropies and joint entropy, the mutual information between x i and x j can be derived from Eq. 1 and formally defined as Eq. 2 [8].

High mutual information between layers indicates redundancy, while low mutual information suggests that these layers have learned complementary representations [43]. When two variables x i and x j are independent, their mutual information is zero, i.e., I ( x i ; x j ) = 0 . According to information theory [41] and the Information Bottleneck (IB) theory [48], minimizing the mutual information between layers can reduce redundancy, remove irrelevant information, and enhance the overall representational capacity of the network. This "compression" of the representation enables the network to extract higher-level and more compact features, thereby reducing the reconstruction error. In summary, by sparsifying layers with higher mutual information and minimizing the mutual information throughout the entire network, the reconstruction error can be minimized.

3.2 Redundancy Metric

To approximate the mutual information between layers, we propose employing Monte Carlo sampling, thereby circumventing the need for intractable integrals. Specifically, we randomly select N samples x (1) 0 , x (2) 0 , . . . , x ( N ) 0 from the training dataset, which follow a probability density function (PDF) P ( x ) . For each sample x ( n ) 0 , the outputs at the i -th and j -th layers are denoted as x ( n ) i and x ( n ) j , respectively.

( ( n ) ( n ) ) .

We can estimate the integral using the sample average: ˆ I ( x i ; x j ) ≈ 1 N ∑ N n =1 log p x i ,x j p ( x ( n ) i ) p ( x ( n ) j ) ˆ I ( x i ; x j ) is the estimated mutual information, p ( x ( n ) i , x ( n ) j ) is the joint PDF, and p ( x ( n ) i ) and p ( x ( n ) j ) are the marginal PDFs. We aim to approximate these probability densities using kernel density estimation.

Computing marginal and joint probability densities : Based on the kernel density estimation [44], we can utilize it to estimate the probability density functions. This allows us to approximate the probability density functions using the features of the samples. Kernel density estimation is a non-parametric method for estimating the probability density function of a random variable. For instance, given a set of samples Y = { y 1 , y 2 , . . . , y N } , The kernel density estimate is defined as: ˆ p ( y ) = 1 Nh ∑ N i =1 K ( y -y i h ) , where K ( · ) is a kernel function, commonly used kernels include Gaussian, Epanechnikov, etc., and h is the bandwidth parameter.

We apply kernel density estimation to compute the marginal and joint probability density functions of the samples' outputs at the i -th and j -th layers. The choice of the bandwidth parameter h can be determined through cross-validation or other methods. However, to simplify our derivation, we can consider that in high-dimensional spaces, the influence of the kernel function K is relatively insignificant. We are mainly focused on the ratio of relative densities [40, 60]. Therefore, the bandwidth parameter h can be cancelled out. We can calculate the marginal and joint probability density functions of the samples' outputs at the i -th and j -th layers using kernel density estimation, which can be found in the Appendix.

Monte Carlo Approximation of Mutual Information. Substituting the kernel density estimates into the Monte Carlo approximation formula for mutual information and simplifying the expression using the feature matrix inner product approximation for the kernel function, as mentioned by Tschannen [51], we obtain:

where ∥ · ∥ F denotes the Frobenius norm. In this approximation, we employ the feature matrix inner product to approximate the kernel function, K (( x ( n ) i , x ( n ) j ) , ( x ( k ) i , x ( k ) j )) ≈ ∥ ∥ ∥ x ( n ) T i x ( n ) j ∥ ∥ ∥ F . Similarly, for the marginal kernel functions is K ( x ( n ) i , x ( k ) i ) ≈ ∥ ∥ ∥ x ( n ) T i x ( n ) i ∥ ∥ ∥ F .

Decreasing function : Since mutual information has no general upper bound, its upper limit depends on the entropy of either x i or x j . To address this, we can use decreasing functions to transform the range of mutual information, ensuring a bounded and more interpretable metric. For instance, using e ˆ I ( x i ; x j ) or a Gaussian function, we can redefine the measure as follows. Considering that a batch of data is fed into the model simultaneously and each layer output concurrently, we can omit the ∑ N n =1 and 1 N . Instead, we can use X i and X j to represent the calculations for the entire batch input. Applying ∥ ∥ X T i X j ∥ ∥ e ˆ I ( X i ; X j ) as a decreasing function to Eq. 3. Therefore, we can derive the Redundancy Metric (RM) formula, where RM ( · ) ∈ [0 , 1] according to the Cauchy-Schwarz inequality:

The decreasing function transforms the range of the RM formula such that a value of 0 indicates complete independence between layers, while 1 represents complete redundancy. This formulation

can serve as the objective function for maximization. The complete derivation process for this section, including the details of Eq. 4, is presented in Appendix B.

3.3 Linear Optimization

Our Redundancy Metric reveals the redundancy among layers in a neural network, guiding sparsity ratio allocation. Experiments on LLaMA2-13B with various sparsity configurations show a negative correlation between model redundancy and WikiText-2 perplexity (PPL). Model redundancy is defined as the sum of each layer's RM for the remaining layers, as depicted in Fig. 2 (c). Consequently, redundancy minimization is adopted as the objective function, incorporating model size constraints to formulate a linear programming problem that yields the optimal sparsity configuration.

Intra-layer Sparsity Allocation. For a given neural network, we construct a redundancy matrix Ψ , where ψ ij = RM ( x i , x j ) . The sum of non-diagonal elements for each row of the matrix is computed as ρ i = ∑ L j =1 ψ ij -1 . A smaller ρ i indicates stronger independence between x i and the outputs of other layers. We model this relationship using the monotonically decreasing function: ω i = e -1 µ ρ i , where µ is a dynamic hyperparameter controlling the difference in sparsity ratios across layers, defined as 1 n ∑ n j =1 ψ ij , which smooths the descent speed (Fig. 8). The importance factor for the first i layers is represented by ω i . With these components, we formulate the linear programming problem as follows:

where S ( q i ) denotes the model size of the i -th layer under sparsity q i , and B represents the target model size. The optimal sparsity configuration is given by q . To maximize the model's representative capacity, our method try to assign smaller sparsity configurations to more independent layers by maximizing an objective function. For a more fine-grained sparsity allocation, we extend our approach to include intra-layer component-level sparsity allocation. After determining the sparsity ratios for each layer, we treat the remaining parameters in this layer as the target size and construct objective functions for its individual components. By applying ALS at this granular level, we obtain a secondary sparsity allocation, resulting in unique sparsity ratios for every layer and component. This hierarchical approach enables a highly customized and adaptable sparsity distribution throughout the entire network architecture, potentially leading to enhanced efficiency and performance gains.

4 Experimental Results

Setup. For pruning, we follow the settings of Wanda, SparseGPT, and Magnitude. Regarding the calibration data used in the linear optimization process, we follow the configurations of SparseGPT and Wanda, selecting data from the C4 dataset and ensuring that all test data are zero-shot. We use a calibration data size of 16 for linear optimization hyperparameters. The granularity, explained in Appendix. E.1 for linear optimization results is set to 0.5%. For the values of x i , we use the input, although output and intermediate gates can also be used. Hyperparameter analysis is primarily conducted in the analysis section. Details about the experimental environment are provided in Appendix E.1.

Evaluation and Metrics. We measure the performance of pruned models through zero-shot tasks and language modeling. For zero-shot evaluation, we utilize seven tasks from the EleutherAI LM Harness [47]: Winogrande [39], PIQA [2], OpenBookQA [37], HellaSwag [61], BoolQ [6], ARC (Easy and Challenge) [7], and RTE (Recognizing Textual Entailment) [52]. We also include WikiText2 [36]. For the first seven datasets, we use the accuracy metric provided in the EleutherAI LMHarness. For WikiText2, we use the word_perplexity (PPL) metric. During evaluation, we ensure using the same database version, GPU model, and random seed.

Models. Weevaluate the performance of ALS on LLMs, including LLaMA-V1 7B/13B/30B/65B [49], LLaMA-V2 7B/13B/70B [50], LLaMA-V3 8B [1], OPT 6.7B/13B [62].

Baselines. We run ALS on LLMs with various methods, including Wanda [45], Magnitude-based pruning [18] and SparseGPT [16].

4.1 Language Modeling

Quantitative Evaluation. In Table 2, we compare the wikitext2 (PPL) performance of different pruning methods under 50% sparsity on the LLaMA-V1, LLaMA-V2, LLaMA-V3, and OPT models, including Dense (unpruned), Magnitude pruning [28], SparseGPT pruning [17], Wanda pruning [46], and the results of these pruning methods enhanced by ALS. The results show that the ALS generally improves the performance of various pruning methods.

For LLaMA-V1 models, Magnitude pruning shows high perplexity, e.g., 42.26 for the 7B model, reduced to 16.80 with ALS. SparseGPT performs better, with 18.35 for the 13B model, reduced to 11.87 with ALS. Wanda achieves the best results, with 13.30 for the 13B model, reduced to 12.47 with ALS.

For LLaMA-V2 and LLaMA-V3 models, ALS also reduces perplexity significantly. For instance, the Magnitude pruning in 13B LLaMA-V2 model drops from 15.19 to 10.78, and Wanda pruning in the 8B LLaMA-V3 model from 15.01 to 12.30 with ALS.

On the OPT model, perplexity significantly increases after pruning. For instance, the 13B model of OPT has a perplexity as high as 4.09e4 after Magnitude pruning, which remarkably reduces to 3.96e3 with ALS, demonstrating the effect of ALS in handling LLM pruning. However, there is an example where performance does not significantly improve with ALS. For instance, the 13B model of LLaMA-V1 has 9.90 perplexity after SparseGPT pruning, which slightly increases with ALS.

In summary, ALS significantly enhances model performance across various pruning methods by effectively mitigating performance loss.

Varying Sparsity Rates. Table 1 presents the perplexity scores of sparse LLaMA-V2 7B and 13B models pruned by Magnitude, SparseGPT, and Wanda methods, with and without ALS, at varying sparsity levels (20% to 70%). The results show that as sparsity increases, perplexity scores generally deteriorate, indicating a decline in language modeling performance. However, as the sparsity level increases, the performance gap between ALS and non-ALS methods widens, with ALS exhibiting better performance at most sparsity levels. This suggests that ALS can help mitigate the performance degradation caused by higher sparsity, becoming increasingly effective at maintaining LLMs performance as the sparsity level grows.

4.2 Zero-shot Tasks

In Table 3, we present the averaged accuracy performance of pruned LLaMA-V1, LLaMA-V2, LLaMA-V3, and OPT models on seven downstream zero-shot tasks at a 50% sparsity ratio. For detailed performance on specific tasks, please refer to Table 4, which shows improvements in most tasks. The average accuracy across the majority of tasks demonstrates the effectiveness of ALS in enhancing sparse large language models of any scale. Remarkably, for LLaMA3-8B, the incorporation of ALS leads to an improvement of 14.14% and 9.87% compared to the Magnitude and SparseGPT baselines, respectively. Similarly, for LLaMA1-13B, the addition of ALS results in an improvement of 5.95% compared to the baselines.

The significant performance improvement of LLaMA-V3 8B may be attributed to the fact that the new model is not well-suited for uniform pruning methods. we further use the LLaMA-V3 8B model as an example to intuitively present the improvements brought by ALS from the perspective of the heat map (Fig. 3 in the Appendix D). The heat map reveals that, at 50% sparsity, the redundancy between layers of the LLaMA-V3 8B model exhibits a distinct pattern compared to other models. The green distribution on both sides indicates that there is small redundancy between the shallow layers and other layers. This suggests that the information captured by the middle and deep layers has more overlap and similarity with other layers. The ALS method takes advantage of this inter-layer redundancy pattern and apply high sparsity ratio into the layers with higher redundancy. It maximally preserves the key information and reduces the impact of sparsification on model performance. From the heat maps of different models in the Appendix D, it can also be observed that each model requires a different sparsity ratio because the redundancy between its layers varies.

4.3 Ablation Study

In this part, we examine the impact of various components within the ALS framework and compare it with LoRA Fine-tuning, specifically focusing on its bound setting, standardization on weight or feature, granularity choice, which can be found in Appendix. E.1, feature choice and robustness to calibration samples. All experimental setups are based on the LLaMA2-13B model with Wanda pruning and ALS.

Comparison with LoRA Fine-tuning. Our experiments in Table 5 and Table 6 demonstrate the substantial benefits of combining Wanda+ALS with LoRA fine-tuning across the LLaMA model family. The improvements are most striking in the LLaMA-V1 7B model, which showed a dramatic reduction in perplexity by 4.82 alongside a 10.17% increase in accuracy. Larger V1 models also benefited, with the 30B and 13B variants showing perplexity reductions of 1.43 and 1.15, coupled with accuracy gains of 2.09% and 1.89% respectively. The LLaMA-V2 models exhibited similar positive trends, with both 7B and 13B versions showing perplexity improvements and accuracy increases. These impressive results were achieved using just 2000 C4 samples for LoRA fine-tuning in a zero-shot setting, highlighting the method's efficiency and effectiveness even with limited training data unrelated to the evaluation tasks.

Feature Selection and Normalization. Table 7 (a) compares the performance of input, output, and gate features in capturing layer independence, with output features achieving slightly lower perplexity. Table 8 (b) demonstrates the significant impact of jointly normalizing features and per-layer weights. Applying this normalization strategy yields a substantial improvement in accuracy, increasing from 66.40% to 68.11%, while also reducing perplexity from 10.078 to 10.070.

Comparison with OWL. We compared the performance of our proposed method with the OWL method on a set of benchmark datasets. The results are summarized in Table 9. We adopted the optimal parameter settings described in the OWL paper. Across all tested configurations, our method consistently achieved lower values compared to OWL, demonstrating its superior performance. Specifically, in the unstructured 50% setting, Wanda with ALS outperformed OWL by a margin of 0.25 units. Furthermore, in the structured pruning settings of 2:4 and 4:8, the advantage of Wanda with ALS increased to 0.95 and 0.44, respectively.

N:M Results. We also investigated the performance of our method in the N:M setting, where N features are selected from M available features. The results are shown in Table 9. Similarly, for OWL, we used the optimal parameter combination reported in their paper. Across all N:M configurations, ALS consistently achieved lower values compared to OWL. As the number of selected features N increased, both methods exhibited performance improvements, but the advantage of ALS became more pronounced. For instance, in the 2:4 case, ALS outperformed OWL by a margin of 0.95 units, and this gap further widened to 1.82 in the 4:8 case. Overall, OWL is a method that is highly sensitive to parameter settings, and obtaining the optimal parameters may require dozens of experiments to determine the best combination. Moreover, there is no clear theoretical analysis explaining why such a combination should be used.

0

50

100

150

200

250

300

Calibration Data

9.86

9.88

9.9

9.92

9.94

9.96

Wikitext2 Perplexity

0.45-0.55

0.30-0.70

0.15-0.85

0.0-1.0

Boundes

10 1

10 2

10 3

Wikitext2 Perplexity

1200

1220

1240

1260

1280

1300

1320

10 1

10 2

10 3

Wikitext2 Perplexity

Calibration Data. In Fig. 2 (a), we present the performance of pruning methods with different numbers of calibration samples. We use the size of 2, 4, 8, 16, 32, 64, 128, 256. Although this experiment reveals that the model's performance improves with an increase in the size of the calibration data, the improvement is quite limited. Even when comparing the scales of 2 and 256 in calibration samples, the perplexity decreases by only 0.11. These results further highlight the robustness of ALS.

Boundes. In Fig. 2 (b) demonstrates the effect of pruning bounds on the performance of the LLaMAV2 13B model. When the pruning bounds are set too high (e.g., 0.0-1.0), the model's performance significantly deteriorates from 10 1 to 10 3 compared with 0.3-0.7, indicating that aggressive pruning may impair the model's representational capacity. However, when the pruning bounds are set between 30% and 70%, the model's performance remains nearly unaffected.

Computation efficiency. As shown in Table 10, our ALS involves two computational phases: the Redundancy Metric (RM) calculation, which consistently takes approximately 90 seconds across all methods, and the Linear Programming (LP) solution, requiring roughly 160-170 milliseconds. The total processing time varies notably depending on the base pruning method employed: Magnitude pruning, requiring just 1.62 seconds for its base operation, achieves the fastest total completion time of 1.51 minutes when combined with ALS. Wanda, with its base pruning time of 199 seconds, completes the entire process in 4.81 minutes, while SparseGPT, requiring 1058 seconds for its base operation, takes 19.16 minutes in total. Compared to BESA [57] with 4.5 hours for sparsity allocation and pruning, our approach is notably faster, completing the process in minutes rather than hours.

5 Conclusion

In this work, we present Adaptive Layer Sparsity (ALS), a novel approach for optimizing LLMs through the efficient allocation of sparsity across layers. By minimizing inter-layer redundancy, ALS achieves significant model compression while maintaining performance, as demonstrated through extensive experiments on diverse LLMs and tasks. We hope ALS offers valuable insights and practical tools for deploying LLMs under limited computational resources, and that our work may shed light on the role of sparsity in LLMs and its potential for model optimization. Future research will explore the relationship between sparsity allocation and individual weight importance, and investigate the integration of dynamic sparsity allocation with pruning metrics. By pushing the boundaries of model compression and efficiency, we aim to enhance the development of more capable and accessible LLMs for diverse applications.

References

AI@Meta. Llama 3 model card. 2024. 3, 6

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence (AAAI) , volume 34, pages 7432-7439, 2020. 6

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems (NeurIPs) , 33:1877-1901, 2020. 1

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 , 2023. 1

Yanqi Chen, Zhengyu Ma, Wei Fang, Xiawu Zheng, Zhaofei Yu, and Yonghong Tian. A unified framework for soft threshold pruning. In The Eleventh International Conference on Learning Representations , 2023. 4

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044 , 2019. 6

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv , abs/1803.05457, 2018. 6

Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing) . Wiley-Interscience, USA, 2006. 4

Peijie Dong, Lujun Li, Xiang Liu, Zhenheng Tang, Xuebo Liu, Qiang Wang, and Xiaowen Chu. Lpzero: Language model zero-cost proxy search from zero. arXiv preprint arXiv:2410.04808 , 2024. 3

Peijie Dong, Lujun Li, Zhenheng Tang, Xiang Liu, Xinglin Pan, Qiang Wang, and Xiaowen Chu. Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. In ICML , 2024. 3

Peijie Dong, Lujun Li, and Zimian Wei. Diswot: Student architecture search for distillation without training. In CVPR , 2023. 3

Peijie Dong, Lujun Li, Zimian Wei, Xin Niu, Zhiliang Tian, and Hengyue Pan. Emq: Evolving training-free proxies for automated mixed precision quantization. In ICCV , 2023. 3

Xin Dong, Shangyu Chen, and Sinno Jialin Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 4860-4874, Red Hook, NY, USA, 2017. Curran Associates Inc. 2, 4

Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning (ICML) , pages 2943-2952, 2020. 4

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations , 2019. 1, 3

Elias Frantar and Dan Alistarh. Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning (ICML) , 2023. 3, 7

Elias Frantar and Dan Alistarh. Sparsegpt: massive language models can be accurately pruned in one-shot. In Proceedings of the 40th International Conference on Machine Learning , ICML'23. JMLR.org, 2023. 2, 3, 7

Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems (NeurIPS) , pages 1135-1143, 2015. 7

Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1 , NIPS'15, page 1135-1143, Cambridge, MA, USA, 2015. MIT Press. 1, 3

B. Hassibi, D.G. Stork, and G.J. Wolff. Optimal brain surgeon and general network pruning. In IEEE International Conference on Neural Networks , pages 293-299 vol.1, 1993. 3

Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks , pages 293-299. IEEE, 1993. 4

Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence , IJCAI'18, page 2234-2240. AAAI Press, 2018. 2, 4

Zhongzhan Huang, Xinjiang Wang, and Ping Luo. Convolution-weight-distribution assumption: Rethinking the criteria of channel pruning. CoRR , abs/2004.11627, 2020. 3

Chunhui Jiang, Guiying Li, Chao Qian, and Ke Tang. Efficient dnn neuron pruning by minimizing layer-wise nonlinear reconstruction error. In Proceedings of the 27th International Joint Conference on Artificial Intelligence , IJCAI'18, page 2298-2304. AAAI Press, 2018. 2, 4

Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 3519-3529. PMLR, 09-15 Jun 2019. 3

Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference on Machine Learning (ICML) , pages 5544-5555, 2020. 4

Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touretzky, editor, Advances in Neural Information Processing Systems , volume 2. Morgan-Kaufmann, 1989. 1, 3

Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer-adaptive sparsity for the magnitude-based pruning. In International Conference on Learning Representations (ICLR) , 2020. 2, 7

Lujun Li, Yufan Bao, Peijie Dong, Chuanguang Yang, Anggeng Li, Wenhan Luo, Qifeng Liu, Wei Xue, and Yike Guo. Detkds: Knowledge distillation search for object detectors. In ICML , 2024. 3

Lujun Li, Peijie Dong, Anggeng Li, Zimian Wei, and Ya Yang. Kd-zero: Evolving knowledge distiller for any teacher-student pairs. NeuIPS , 2024. 3

Lujun Li, Peijie Dong, Zimian Wei, and Ya Yang. Automated knowledge distillation via monte carlo tree search. In ICCV , 2023. 3

Lujun Li and Zhe Jin. Shadow knowledge distillation: Bridging offline and online knowledge transfer. In NeuIPS , 2022. 3

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978 , 2023. 3

Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-pruner: On the structural pruning of large language models. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. 3

Yuexiao Ma, Taisong Jin, Xiawu Zheng, Yan Wang, Huixia Li, Yongjian Wu, Guannan Jiang, Wei Zhang, and Rongrong Ji. Ompq: Orthogonal mixed precision quantization. In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 9029-9037, 2023. 3

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations , 2017. 6

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP , 2018. 6

Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Representations , 2017. 1, 3

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM , 64(9):99-106, 2021. 6

David W. Scott. Multivariate density estimation and visualization. Papers 2004,16, Berlin, 2004. 5

C. E. Shannon. A mathematical theory of communication. Bell System Technical Journal , 27(3):379-423, 1948. 4

Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. In The Twelfth International Conference on Learning Representations , 2024. 3

Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810 , 2017. 4

Bernard W Silverman. Density estimation for statistics and data analysis . Routledge, 2018. 5

Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695 , 2023. 7

Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations , 2024. 2, 3, 7

Sutawika, Schoelkopf, Gao, Abbasi, Biderman, Tow, fattori, Lovering, farzanehnakhaee, Phang, Thite, Fazz, Wang, Muennighoff, Aflah, sdtblck, nopperl, gakada, tttyuntian, researcher, Chris, Etxaniz, Lee, Kasner, Khalid, Hsu, Kanekar, Ammanamanchi, Boykis, and AndyZwei. EleutherAI/lm-evaluation-harness: v0.4.2, March 2024. 6, 24

Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw) , pages 1-5. IEEE, 2015. 4

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023. 1, 3, 6

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023. 3, 6

Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv:1907.13625 , 2019. 5

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupała, and Afra Alishahi, editors, Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pages 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. 6

Qiufeng Wang, Xu Yang, Shuxia Lin, and Xin Geng. Learngene: Inheriting condensed knowledge from the ancestry model to descendant models. ArXiv , abs/2305.02279, 2023. 3

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research , 2022. 1

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems (NeurIPs) , 35:24824-24837, 2022. 1

Liu Xiaolong, Li Lujun, Li Chao, and Anbang Yao. Norm: Knowledge distillation via n-to-one representation matching. In ICLR , 2023. 3

Peng Xu, Wenqi Shao, Mengzhao Chen, Shitao Tang, Kaipeng Zhang, Peng Gao, Fengwei An, Yu Qiao, and Ping Luo. BESA: Pruning large language models with blockwise parameter-efficient sparsity allocation. In The Twelfth International Conference on Learning Representations , 2024. 2, 4, 10

Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Mykola Pechenizkiy, Yi Liang, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity. arXiv preprint arXiv:2310.05175 , 2023. 2, 4

Miao Yin, Burak Uzkent, Yilin Shen, Hongxia Jin, and Bo Yuan. Gohsp: a unified framework of graph and optimization-based heterogeneous structured pruning for vision transformer. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence , AAAI'23/IAAI'23/EAAI'23. AAAI Press, 2023. 1, 3

Adriano Z Zambom and Ronaldo Dias. A review of kernel density estimation with applications to econometrics. International Econometric Review , 5(1):20-42, 2013. 5

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830 , 2019. 6

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022. 1, 3, 6

Yuxin Zhang, Mingbao Lin, Fei Chao, Yan Wang, Ke Li, Yunhang Shen, Yongjian Wu, and Rongrong Ji. Lottery jackpots exist in pre-trained models. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 2023. 3

Yuxin Zhang, Mingbao Lin, Zhihang Lin, Yiting Luo, Ke Li, Fei Chao, Yongjian Wu, and Rongrong Ji. Learning best combination for efficient n: M sparsity. In Advances in Neural Information Processing Systems (NeurIPS) , 2022. 3

Yuxin Zhang, Lirui Zhao, Mingbao Lin, Sun Yunyun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. Dynamic sparse no training: Training-free fine-tuning for sparse LLMs. In The Twelfth International Conference on Learning Representations , 2024. 2, 4

Appendix

A Limitation: Computational Complexity of Intra-Layer Component Independence Calculation

In the ALS method, a key step is computing the independence matrix between layers. While calculating the matrix at the layer level for a 70 billion parameter model with 80 layers requires only an 80 × 80 RMmatrix, including intra-layer components increases the matrix size to 7 × 80 × 7 × 80 , leading to 49 times more computational time and resources.

High Computational Complexity : The increased matrix size results in exponential growth in computation and resource consumption.

Excessive Memory Usage : High-dimensional matrix computations require substantial memory, potentially exceeding hardware capacities.

Solutions

Hybrid Solution with C++ : Store intermediate data locally and use C++ to handle calculations and solve the linear programming problem. This approach can be up to 50 times faster than using Python alone.

Alternatively, calculate sparsity ratios for each layer, determine the parameters to retain, and use these as the target size for further linear programming. This approach requires only an 80 × 80 RM matrix and 7 additional 7 × 7 matrices, without significantly increasing computation time.

Hybrid Solution with C++ is a preferred solution because it will keep most of independent components to maintain the model performance after pruning.

B Detailed Formulae Derivation

This section provides a detailed derivation of the key mathematical formulae used in the paper.

B.1 Detailed Derivation of the Mutual Information Approximation

The objective of LLMs is to minimize the reconstruction error between the outputs of the sparse and dense models. We start by defining the mutual information between the outputs of different layers to quantify the redundancy within the model. Mutual information I ( X ; Y ) measures the amount of information obtained about one random variable X through observing the other random variable Y . It quantifies the reduction in uncertainty of X due to the knowledge of Y . This measure helps identify layers with high redundancy, which can be pruned to achieve a more efficient representation.

The mutual information I ( X i ; X j ) between two layers i and j is given by:

where H ( X i ) and H ( X j ) are the marginal entropies, and H ( X i , X j ) is the joint entropy. The entropy H ( X ) measures the average amount of information or uncertainty in a random variable X . These entropies are defined as:

The mutual information can also be expressed in terms of the probability density functions as:

To approximate these values, we use Monte Carlo sampling. Given N i.i.d. samples x (1) , x (2) , . . . , x ( N ) drawn from the joint distribution p ( x i , x j ) , the mutual information can be approximated as:

Using kernel density estimation to approximate the probability densities. For instance:

Here, K ( · ) is the kernel function, h is the bandwidth parameter, and d is the dimension of y . Kernel density estimation is a non-parametric method for estimating the probability density function p ( y ) of a random vector y , given a set of i.i.d. samples Y = { y 1 , y 2 , . . . , y N } drawn from p ( y ) . Commonly used kernels include the Gaussian and Epanechnikov kernels.

To compute the marginal and joint probability density functions of the samples' outputs at the i -th and j -th layers, we apply kernel density estimation:

where d i and d j are the dimensions of x i and x j , respectively.

To derive the mutual information approximation, let's start with the Monte Carlo approximation of mutual information (Equation 7). Substituting the kernel density estimates (Equations 9-11) into Equation 7, we get:

Now, consider the feature matrix inner product approximation for the kernel function. Let:

where ∥ · ∥ F denotes the Frobenius norm.

Therefore, the joint kernel function can be approximated by:

Substituting these approximations (Equations 13-15) back into the mutual information formula (Equation 12), we get:

This leads to the final expression for the mutual information approximation using Monte Carlo sampling and kernel density estimation with the feature matrix inner product approximation.

B.2 Redundancy Metric Derivation

Since mutual information has no general upper bound, its upper limit depends on the entropy of either X i or X j , which represent the i -th and j -th layer output of a batch input. To address this, we can use functions to transform the range of mutual information, ensuring a bounded and more interpretable metric. For instance, using e ˆ I ( X i ; X j ) or a Gaussian function, we can redefine the measure as follows. Considering ∥ ∥ x T i x j ∥ ∥ F e ˆ I ( X i ; X j ) for a batch of inputs, we can remove the sum and 1 N , then obtain:

To derive the Redundancy Metric (RM) formula, we consider that a batch is a set of samples processed simultaneously by the neural network. Because a batch is input simultaneously, we remove the summation and division by N , obtaining the following formula:

In this formula, RM ( · ) ∈ [0 , 1] , where a value close to 0 indicates high independence between the layers (low redundancy), and a value close to 1 indicates high redundancy (low independence).

B.3 Constraints and Objective Functions

Our redundancy metric is used to guide the allocation of sparsity ratios across layers. We construct a redundancy matrix Ψ ∈ R L × L where each element ψ ij represents the redundancy between layers i and j , computed using Equation 18. The overall redundancy of each layer is:

A lower ρ i indicates higher independence. Using a monotonically decreasing function, we define the importance factor ω i as:

where µ is a hyperparameter controlling the decay rate.

Our objective is to maximize the sum of the weighted sparsity ratios across all layers, subject to a model size constraint B . The sparsity ratio q i ∈ [0 , 1] represents the fraction of weights to be pruned in the i -th layer. The optimization problem can be formulated as:

where S ( q i ) represents the model size of the i -th layer under sparsity ratio q i , and q = [ q 1 , q 2 , . . . , q L ] T is the vector of sparsity ratios for all L layers. The objective function (Equation 22) aims to allocate higher sparsity ratios to layers with higher importance factors. The constraint (Equation 23) ensures that the total model size after pruning does not exceed the budget B .

C More Experimental Results

C.1 Zero-Shot results Details

Results in LLaMA-V1,V2,V3 and OPT series at 50% Sparsity

Results in LLaMA-V2 7B/13B at various sparsity level

D Visualization of Correlation Matrices

In this section, we present the visualization of correlation matrices obtained by solving the problem under different experimental settings. Additionally, we provide the numerical results of the Sparse Ratio Allocation. The RM heat maps are different with different models.

D.1 Visualization of RM Matrices

50% sparsity in LLAMA-V1/V2/V3 family.

LLaMA-V1 7B

1

6

11

16

21

26

31

1

6

11

16

21

26

31

LLaMA-V1 13B

1

6

11

16

21

26

31

36

1

6

11

16

21

26

31

36

LLaMA-V1 30B

1

6

11

16

21

26

31

36

41

46

51

56

1

6

11

16

21

26

31

36

41

46

51

56

LLaMA-V1 65B

1

6

11

16

21

26

31

36

41

46

51

56

61

66

71

76

1

6

11

16

21

26

31

36

41

46

51

56

61

66

71

76

LLaMA-V2 7B

1

6

11

16

21

26

31

1

6

11

16

21

26

31

LLaMA-V2 13B

1

6

11

16

21

26

31

36

1

6

11

16

21

26

31

36

LLaMA-V2 70B

1

6

11

16

21

26

31

36

41

46

51

56

61

66

71

76

1

6

11

16

21

26

31

36

41

46

51

56

61

66

71

76

LLaMA-V3 8B

1

6

11

16

21

26

31

1

6

11

16

21

26

31

0.2

0.4

0.6

0.8

1

Various sparsity in LLAMA-V2 7B/13B

LLaMA-V2 7B, Sparsity 0.3

1

6

11

16

21

26

31

1

6

11

16

21

26

31

LLaMA-V2 7B, Sparsity 0.4

1

6

11

16

21

26

31

1

6

11

16

21

26

31

LLaMA-V2 7B, Sparsity 0.5

1

6

11

16

21

26

31

1

6

11

16

21

26

31

LLaMA-V2 7B, Sparsity 0.6

1

6

11

16

21

26

31

1

6

11

16

21

26

31

LLaMA-V2 7B, Sparsity 0.7

1

6

11

16

21

26

31

1

6

11

16

21

26

31

LLaMA-V2 13B, Sparsity 0.3

1

6

11

16

21

26

31

36

1

6

11

16

21

26

31

36

LLaMA-V2 13B, Sparsity 0.4

1

6

11

16

21

26

31

36

1

6

11

16

21

26

31

36

LLaMA-V2 13B, Sparsity 0.5

1

6

11

16

21

26

31

36

1

6

11

16

21

26

31

36

LLaMA-V2 13B, Sparsity 0.6

1

6

11

16

21

26

31

36

LLaMA-V2 13B, Sparsity 0.7

1

6

11

16

21

26

31

36

0.2

0.4

0.6

0.8

1

1

6

11

16

21

26

31

36

1

6

11

16

21

26

31

36

D.2 Ratio allocation

LLaMA-V1 7B.

1

6

11

16

21

26

31

Q

K

V

O

gate

up

down

LLaMA-V1 13B.

1

6

11

16

21

26

31

36

Q

K

V

O

gate

up

down

LLaMA-V1 30B.

1

6

11

16

21

26

31

36

41

46

51

56

Q

K

V

O

gate

up

down

LLaMA-V1 65B.

1

6

11

16

21

26

31

36

41

46

51

56

61

66

71

76

Q

K

V

O

gate

up

down

LLaMA-V2 7B.

1

6

11

16

21

26

31

Q

K

V

O

gate

up

down

LLaMA-V2 13B.

1

6

11

16

21

26

31

36

Q

K

V

O

gate

up

down

LLaMA-V2 70B.

1

6

16

21

26

31

36

41

46

51

56

61

66

71

76

Q

K

V

O

gate

up

down

LLaMA-V3 8b.

1

6

11

16

21

26

31

Q

K

V

O

gate

up

down

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

11

D.3 Ratio allocation

LLaMA-V2 7B in 0.3

5

10

15

20

25

30

Q

K

V

O

gate

up

down

LLaMA-V2 7B in 0.4

5

10

15

20

25

30

Q

K

V

O

gate

up

down

LLaMA-V2 7B in 0.5

5

10

15

20

25

30

Q

K

V

O

gate

up

down

LLaMA-V2 7B in 0.6

5

10

15

20

25

30

Q

K

V

O

gate

up

down

LLaMA-V2 7B in 0.7

5

10

15

20

25

30

Q

K

V

O

gate

up

down

LLaMA-V2 13B in 0.3

5

10

15

20

25

30

35

40

Q

K

V

O

gate

up

down

LLaMA-V2 13B in 0.4

5

10

15

20

25

30

35

40

Q

K

V

O

gate

up

down

LLaMA-V2 13B in 0.5

5

10

15

20

25

30

35

40

Q

K

V

O

gate

up

down

LLaMA-V2 13B in 0.6

5

10

15

20

25

30

35

40

Q

K

V

O

gate

up

down

LLaMA-V2 13B in 0.7

5

10

15

20

25

30

35

40

Q

K

V

O

gate

up

down

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

E Extra Figures and Explanations

E.1 Experimental Environment and Hyperparameters

Granularity. The granularity for linear optimization results is set to 0.5%, meaning the sparsity percentages can only have decimal places of 0.5% or 0.0%. The experiment is based on LLaMA-V2 13B, this study in Fig 7 examines the impact of granularity on perplexity (PPL) across selected values. Initially, PPL remains relatively constant at 10.07 for granularities of 0.1 and 0.5. It then decreases to 9.86 at a granularity of 1 and further to 9.67 at a granularity of 5. However, beyond this point, the smoothed curve indicates a subsequent rise in PPL, suggesting that excessively high granularity may negatively impact model performance. This analysis highlights a critical balance in optimizing granularity to minimize PPL and enhance model accuracy and efficiency.

Environment. All pruning experiments are performed on dual NVIDIA A100 GPUs with 80GB memory. However, our ALS method mainly runs on CPU, while the baseline methods Wanda, SparseGPT, and Magnitude require GPU. The CPU used is an AMD EPYC™ 9554 64-core processor.

Hyperparameters We set weight and feature normalization, calibration data= 2, feature selection= in, granularity=0.05, boundes= 0.3-0.7 for 3070% sparsity and 0.1-0.3 for 20% sparsity.

Granularity vs PPL (Smoothed)

0

1

2

3

4

5

Granularity

8.25

8.50

8.75

9.00

9.25

9.50

9.75

10.00

PPL

E.2 Ratio allocation

E.3 Error Bar

In this subsection, we present LLaMA-V2 family error bar in 50% sparsity. The standard deviations are runing multiple experiments and get from the EleutherAI LM Harness [47] package.

Accuracies for 7 zero-shot tasks with 50% sparsity using LLaMA-V2 (7B)

80

Dataset

winogrande

piqa

70

openbookqa

arc_easy

60

arc_challenge

rte

Accuracy (%)

50

40

30

20

10

0

SparseGPT

Tasks

Accuracies for 7zero-shot tasks with50% sparsity using LLaMA-V2(13B)

Dataset

80

winogrande

piqa

openbookqa

70

arc_easy

arc_challenge

60

rte

Accuracy (%)

50

40

30

20

10

0

epu

Tasks

Accuracies for 7zero-shot tasks with 50% sparsity using LLaMA-V2(70B)

Dataset

winogrande

80

piqa

openbookqa

70

arc_easy

arc_challenge

60

rte

Accuracy (%)

50

40

30

20

10

0

SparseGPT

SparseGPTw.ALS

Tasks

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See abstract and introduction. See abstract and introduction. The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope, providing a clear overview of the research objectives and findings.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: See Appendix A. Our paper discusses the limitations of the work performed by the authors, providing insights into potential weaknesses and areas for improvement.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not include theoretical results, and therefore does not provide assumptions or proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Experiment. Our paper fully discloses all the information needed to reproduce the main experimental results, including detailed descriptions of the experimental setup, parameters, and methodologies used, which are essential for verifying the main claims and conclusions.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our paper provides open access to the data and code, along with sufficient instructions in the supplemental material to enable others to faithfully reproduce the main experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer:[Yes]

Justification: See Experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See Appendix E.3.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [No]

Justification: NO.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: Only technical reports.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [No]

Justification: No.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: No.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [No]

Justification: NO

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: NO.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Adaptive Layer Sparsity (ALS), a novel approach aimed at optimizing large language models (LLMs) through selective pruning. The key contributions of this work include a method that estimates the correlation between intermediate layers using information orthogonality, enabling the precise measurement of each layer's importance. Additionally, the paper formulates the sparsity allocation problem as a linear programming optimization, facilitating efficient global optimization of layer-wise sparsity ratios. Extensive experiments conducted on various LLM families, including LLaMA and OPT, demonstrate consistent performance improvements over existing pruning methods across different model sizes and sparsity levels. The authors also perform analysis experiments to examine the impact of various factors, such as calibration data, sparsity bounds, and feature selection, on the method's performance.\n\nSTRENGTHS:\n1. The paper presents a novel approach for optimizing LLMs. ALS addresses significant challenges in LLM compression, particularly the challenge of manual sparsity setting and suboptimal performance due to uniform pruning ratios.\n\n2. The experimental design of this research is solid and comprehensive. The authors conduct extensive experiments on various LLM families (LLaMA-V1/V2/V3, OPT) with different parameter sizes, providing a robust evaluation of their method's effectiveness.\n\n3. The proposed method, ALS, is simple and effective, and has good computational efficiency.\n\n4. The paper is well-structured and clearly written. The authors provide a detailed explanation of their methodology, including the key steps of estimating the correlation matrix and employing linear optimization for sparse allocation strategy.\n\nWEAKNESSES:\n1. There is significant overlap between the introduction and related work sections, which could be consolidated to improve flow and reduce repetition.\n\n2. Some details about ALS, such as normalization and reweighting strategies mentioned in the ablation study section, are not clearly described in the \"method\" section. These should be elaborated with formulas in the method section.\n\nQUESTIONS:\nSee weakness 2. Could you provide some details about the normalization and reweighting strategies mentioned in the ablation study section?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes using mutual information to measure layer redundancy in LLMs and employs a linear optimization algorithm based on this measure to derive adaptive sparsity strategies, enabling dynamic sparsity configuration across different layers. The authors conducted experiments on four models, including three versions of LLaMA and OPT, comparing the proposed approach with existing pruning algorithms such as Wanda and SparseGPT. The results demonstrate that the proposed method outperforms these existing algorithms in most scenarios.\n\nSTRENGTHS:\n- The perspective of using mutual information is intuitive and helps identify redundant layers in the model effectively.\n- The pruning algorithm is efficient, requiring only a short time to complete the process.\n- The experiments on LLaMA-3 are significant, demonstrating the effectiveness of non-uniform sparsity allocation across layers.\n\nWEAKNESSES:\nWhile the combination with ALS effectively improves the baseline performance, there are concerns about the practical value of the method. Specifically, when comparing the pruned models with dense models of similar size, the reported performance is relatively close. An extreme example of this is found in Table 2, where the perplexity (PPL) of the 50% sparse LLaMA-v1 30B model is higher than that of the dense LLaMA-v1 13B model, regardless of the pruning method used. This raises questions about the overall effectiveness of the pruning approach in terms of model efficiency and performance trade-offs.\n\nQUESTIONS:\n- How is the intra-layer pruning calculation performed?\n- How is ALS integrated with the baseline method?\n- What is the meaning of L-i+1 in equation (5)?\n- The analysis in lines 274-276 refers to the 13B model, but the data corresponds to the 7B model in the table, which is confusing. Could you clarify this discrepancy?\n- What would be the effect of combining Wanda+ALS with a small amount of LoRA fine-tuning?\n- What is the relationship between sparsity and efficiency as studied in this paper? Does it reduce computation time or memory usage?\n\nLIMITATIONS:\nThe paper lacks some crucial details in certain areas. For instance, in section 4.3, the process of LoRA fine-tuning is mentioned but not adequately explained. It would be beneficial if the authors could provide more comprehensive information on this aspect.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

comments: SUMMARY:
The paper presents an approach called Adaptive Layer Sparsity (ALS) for optimizing large language models (LLMs) by selectively pruning features in intermediate layers. The approach consists of two key steps: estimating the correlation matrix between intermediate layers and employing a linear optimization algorithm to develop an adaptive sparse allocation strategy based on the correlation matrix.

STRENGTHS:
1) The paper introduces a novel approach for optimizing LLMs by considering the varying importance of features across different layers.

2) The approach is based on a well-structured methodology, including estimating the correlation matrix and employing a linear optimization algorithm.

3) Compared with BESA which takes 5 hours on an A100-80G GPU, the proposed method takes less time and is more efficient.

WEAKNESSES:
1) From Table 1, it seems that when the sparse ratio is low (e.g., 20%), the performance improvement with ALS is marginal. Does that indicate it is useless for not very sparse models?

2) From Table 2, for stronger models (e.g., 65B and 70B), the improvement is far less than smaller models. Therefore, it is questionable about the applicability of the proposed method to strong models.

3) The authors conducted comprehensive experiments on Llama-series models, but omit Llama-3-80B. Are there any justifications? Since Llama-3-80B is one of the strongest models for now, it would enhance the paper if you can include such results.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-11

comments: SUMMARY:
The growing size of LLMs make deployment increasingly challenging. Traditional pruning methods underperform due to uniform strategies that ignore varying feature importance across layers. The authors introduces a new Adaptive Layer Sparsity (ALS) approach that estimates inter-layer correlations using information orthogonality, accurately evaluating each layer's significance. It then employs linear optimization to selectively prune features in intermediate layers, achieving precise model optimization. Experiments demonstrate superior efficiency of ALS over existing sparse models like Wanda and SparseGPT, especially at high sparsity levels.

STRENGTHS:
1. The paper covers a wide range of related works and discusses the pros and cons of each work, providing a comprehensive background of this work.
2. Formulating the adaptive pruning with the use of a correlation matrix to measure the importance of each layer and a linear optimization algorithm makes the algorithm itself generalizable and theoretical sound.

WEAKNESSES:
1. The presentation of this paper should be improved. Long paragraphs with various discussion points are mixed together, which is a major weakness of this paper.
2. Evaluation could be improved with numbers on the inference efficiency comparison.

QUESTIONS:
1. What is the computation efficiency of ALS-pruned model compared with existing baslines?
2. What is the overhead of solving the linear programming problem?
3. It seems the performance improvement of ALS decreases when the model becomes large. Is there any reasons for that?

LIMITATIONS:
The authors have addressed the limitations and there is no potential negative societal impact of this work.l
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12