Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment

Authors: wei  li
lujun  li
mark  lee
shengjie  sun
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-25 10:28:54

Abstract: Large Language Models (LLMs) have revolutionized the field of natural language processing with their impressive capabilities. However, their enormous size presents challenges for deploying them in real-world applications. Traditional compression techniques, like pruning, often lead to suboptimal performance due to their uniform pruning ratios and lack of consideration for the varying importance of features across different layers. To address these limitations, we present a novel  Adaptive Layer Sparsity (ALS) approach to optimize LLMs. Our approach consists of two key steps. Firstly, we estimate the correlation matrix between intermediate layers by leveraging the concept of information orthogonality. This novel perspective allows for a precise measurement of the importance of each layer across the model. Secondly, we employ a linear optimization algorithm to develop an adaptive sparse allocation strategy based on evaluating the correlation matrix. This strategy enables us to selectively prune features in intermediate layers, achieving fine-grained optimization of the LLM model. Considering the varying importance across different layers, we can significantly reduce the model size without sacrificing performance. We conduct extensive experiments on publicly available language processing datasets, including the LLaMA-V1|V2|V3 family and OPT, covering various benchmarks. Our experimental results validate the effectiveness of our ALS method, showcasing its superiority over previous approaches. The performance gains demonstrate its potential for enhancing LLMs' efficiency and resource utilization. Notably, our approach surpasses the state-of-the-art models Wanda and SparseGPT, showcasing its ability to excel even under high sparsity levels.  Codes at: https://github.com/lliai/ALS.

Main paper: 
 introduction: 
 Large language models (LLMs) [62, 49, 3] have demonstrated remarkable performance in various natural language processing (NLP) [55, 54, 4] tasks. However, their size and computational requirements pose significant challenges for widespread adoption and deployment. To address these practical constraints, model compression techniques, such as weight pruning and quantization, can potentially reduce the size and computational requirements of LLMs.The emergence of LLMs has revolutionized the field of NLP. However, despite their revolutionary impact, the massive scale and complexity of LLMs presents significant challenges for model compression. Conventional pruning methods [27, 19, 38, 19, 15, 59] , which often require one or more Figure 1 : Overview of our framework. We first compute the sum of Redundancy Metric between layer i-th and other layers to construct objective function. Then, we solve a linear programming problem to optimize total sparsity ratios S (qi) (q i is pre-layer sparsity) under constraints. iterations of fine-tuning or retraining to preserve performance, have become impractical for LLMs due to the substantial computational cost and time required.Due to the failure to the Magnitude approach to pruning [28] and other previous methods on LLMs, recent efforts such as SparseGPT [17] , Wanda [46] , DSOT [65] , Pruning Large Language Models with BESA [57] , and OWL [58] aim to address this challenge by reconstructing the layerwise outputs of LLMs. Specifically, SparseGPT introduces a technique for pruning less significant weights and reconstructing layerwise outputs based on an importance metric derived from the Hessian matrix. To reduce the computational overhead of SparseGPT, Wanda proposes a simplified strategy that relies solely on the product of weight and activation magnitudes for pruning. DSOT computes the reconstruction error incrementally for each layer, optimizing the intra-layer sparse configuration through further weight pruning or growth, which forms the basis for subsequent weight recovery and additional pruning operations. These methods adopt a training-free approach. In contrast, BESA [57] proposes learning the optimal pruning ratio within each layer through training, finding that considering the overall sparsity configuration within a layer enhances the performance of sparse models. However, this method primarily focuses on intra-layer sparsity configuration. It requires substantial training time, typically taking at least 5 hours on an A100-80G GPU, which is considerably slower than other training-free techniques [17, 46, 65] . Another notable method is OWL [58] , which proposes a non-uniform layerwise sparsity technique that assigns different sparsity ratios based on the outlier ratio within each layer, leveraging the unique characteristic of LLMs where some features exhibit significantly larger magnitudes by tuning hyperparameters such as the outlier threshold and sparsity upper/lower bounds to obtain optimal parameter setting. Nevertheless, unlike the aforementioned methods, OWL relies heavily on empirical analysis without providing a solid theoretical foundation for its effectiveness.However, existing methods have several significant drawbacks. First, for BESA, DSOT, and some traditional techniques, minimizing the layer-by-layer pruning error does not effectively mitigate the impact of pruning on model performance, as the pruning error accumulates across layers due to its inherent greedy nature [24] and may also become trapped in local optima [13, 22] . Second, LLM pruning methods such as Wanda, SparseGPT, and Magnitude apply uniform sparsity ratio to each layer, despite the significant variations in each layer's contribution to the final model performance [57, 65] . To achieve better performance for different layers, the sparsity needs to be manually adjusted for all layers. Third, for the newly proposed OWL method, more theoretical analysis is needed on why its outlier-based non-uniform sparsity outperforms uniform sparsity. Moreover, the choice of hyperparameters in OWL, such as the outlier threshold and sparsity upper/lower bounds, is sensitive to model performance, but their optimal ranges are not theoretically explained, and the effective ranges and thresholds are derived through manual tuning. Furthermore, the transferability of these hyperparameters across different datasets has yet to be systematically studied. Therefore, when applying OWL to new models, complex adjustments by hand must be performed to determine the potentially optimal parameter combination.To address the multiple challenges of getting trapped in local optima, manually setting sparsity for all layers, and relying on empirical manual experiments to derive optimal sparsity ratios, we propose a simple, effective, and efficient method called Adaptive Layer Sparsity (ALS) for allocating sparsity ratios. The overall pipeline of our proposed method is illustrated in Fig. 1 . This technique optimizes the pruning rate across different layers. To the best of our knowledge, this is the first attempt to reformulate the sparsity allocation problem in LLMs as a linear programming problem. We tackle these challenges by constructing an objective function and constraints. The constraint of the linear programming problem is that the total number of parameters should be less than the target model size. We then compute the independence matrix [25] at both the layer level and intra-layer component level based on the output or input features. According to our experiments, the independence between layers is positively correlated with model performance, as shown in Fig. 2 (c ). Therefore, maximizing the independence between each layer of the model is considered our objective function. Unlike existing black-box optimization methods, we formulate this problem as a linear one that can be solved by any linear problem solver. This approach enables efficient global sparsity ratio allocation for LLMs ranging from 7B to 70B parameters on a single A100-80GB GPU. If the model scale is too large, reaching 160B, we can also perform multi-threaded computation on a CPU. For a 70B model, the global sparsity configuration can be obtained in just 20 minutes on an A100-80G GPU.To rigorously assess the efficacy of ALS, we conducted extensive experiments on diverse LLMs, including LLaMA-V1 [49] , LLaMA-V2 [50] , LLaMA-V3 [1] , and OPT [62] model families, with parameter counts ranging from 6.7 billion to 70 billion. In the main experiments, we evaluated the WikiText-2 perplexity and average accuracy on 7 zero-shot datasets at various sparsity ratios (20% to 70%) for LLaMA-V2 7B/13B (Table 1 ) and at 50% sparsity for all model families (Tables 2 and 3 ). Detailed results for each zero-shot dataset on LLaMA-V2 family models at 50% sparsity are presented in Table 4 . The analysis experiments consist of 6 sets, examining the impact of calibration data, sparsity bounds setting, and model redundancy on performance (Fig. 2 ), as well as the influence of feature selection, standardization, and comparisons with Wanda and LoRA fine-tuning. Additional experiments including detailed of main experiments, analyses of granularity, decreasing functions, visualizations of layer redundancy, sparsity ratio allocation and comparison with OWL method are provided in the Appendix C and D. These experimental results unequivocally demonstrate that ALS consistently yields substantial performance improvements for sparse LLMs across various LLMs and downstream tasks. 
related work: 
 Model Compression method try to design efficient models and reduce the memory and computational requirements of LLMs. These methods can be categorized into quantization [42, 12, 35, 33] , sparsification [17, 46, 10, 9] and distillation [56, 29, 30, 31, 32, 11, 53] . Quantization converts high bit-width weights and activations into compact, low bit-width representations, while sparsification increases the proportion of zero-valued elements in model weights. Our method optimizes LLM sparsification by strategically allocating sparsity across the model's layers to maximize performance and minimize computational overhead. In contrast to optimization-based compression techniques (e.g., OMPQ [35] ) for CNN models in vision tasks, our approach focuses on different LLM models and NLP tasks and devises various functions and strategies accordingly.Sparsity in LLMs has garnered significant attention as a means to accelerate inference speed and reduce memory consumption by applying sparsity in the model weights or activations. sparsity techniques can be categorized into two main approaches: structured pruning [34, 23] and unstructured pruning [16, 64, 46, 63] . While the primary focus of these works lies in determining the pruning criteria, such as weight importance and pruning ratio, the enormous parameter scale of LLMs presents an additional challenge in terms of pruning efficiency. Conventional pruning methods [15, 59, 63, 23, 27, 19, 38, 19] , dating back to the early work of Hassibi [20] in the 1990s, which successfully reduced model size and improved efficiency in deep learning architectures by removing redundant weights to create sparse and lightweight models, heavily rely on extensive retraining and are often infeasible for LLMs due to prohibitively high computational overhead and prolonged training times. To address this issue, researchers have developed LLM-specific pruning techniques that prioritize train-free and time efficiency. In the context of structured pruning, LLMpruner [34] explores the application of structured pruning to LLMs and employs LoRA to recover the performance of the pruned model. For unstructured pruning, SparseGPT [17] stands out as a notable method that draws inspiration from the Optimal Brain Surgeon (OBS) [20] approach, taking into account the impact of removing individual weights on the network reconstruction loss. SparseGPT introduces an efficient technique for estimating the Hessian matrix, enabling the application of the traditional OBS method to largescale models. Another prominent unstructured pruning method, Wanda [46] , employs a simple yet effective strategy based on the product of weight and activation values to identify and eliminate less important weights, further enhancing the pruning speed. Despite these advancements, most existing methods adopt a uniform pruning rate across all layers, which may lead to suboptimal performance. In contrast, our approach introduces a novel layer adaptive pruning strategy that dynamically allocates sparsity based on the importance of each layer, effectively minimizing performance degradation while achieving high compression ratios.Sparsity Allocation in Network Pruning. Conventional methods for achieving adaptive layerwise sparsity in neural networks [14, 5, 26] often rely on a layer-by-layer pruning approach, where the objective is to minimize the sum of errors introduced in each layer. However, this greedy strategy [24] leads to the accumulation of errors across layers, resulting in suboptimal performance when directly adapted to LLMs. The extensive retraining required on vast datasets further amplifies the challenges of applying these techniques to LLMs. Recent efforts, such as BESA [57] and DSOT [65] , have shifted focus to intra-block sparsity allocation, employing various strategies to optimize the sparsity distribution within individual blocks. Despite operating at a finer granularity, these methods fundamentally adhere to a layer-wise pruning paradigm, neglecting the importance of global sparsity allocation. Consequently, the resulting allocation may be locally optimal [13, 22] within each layer but globally suboptimal, potentially leading to solutions stuck in local optima. Recently, a new approach called OWL [58] attempts to address this issue by introducing a non-uniform layer-wise sparsity technique. This technique primarily relies on manually tuning the outlier threshold and sparsity upper/lower bounds (which are very small values and sensitive to performance) through extensive experimentation to obtain potentially optimal parameter configurations. Although OWL demonstrates the potential for improved sparsity allocation, it heavily depends on empirical analysis and fails to provide a solid theoretical foundation for its effectiveness, limiting its generalizability and robustness across different LLMs architectures and datasets. 
preliminary: 
 Pruning LLMs is a method that aims to obtain a sparse representation of the model by eliminating a predetermined fraction of the pre-trained weights. The primary objective is to minimize the divergence between the outputs generated by the sparse and dense models [21] . However, directly tackling this problem can be challenging due to the massive scale of LLMs. We discover that the mutual information entropy in Eq. 1 can effectively quantify the degree of discrepancy between different layers of the model.EQUATIONNeural networks can be decomposed into a sequence of layers. In the decomposed form, we represent the neural network as F = {f 1 , f 2 , . . . , f L }, For a given random sample x 0 ∈ R d0 , letx i = f i (f i-1 (. . . f 1 (x 0 ))) ∈ R direpresents the output of the random sample at the i-th layer.Based on the previous definitions of the marginal entropies and joint entropy, the mutual information between x i and x j can be derived from Eq. 1 and formally defined as Eq. 2 [8] .EQUATIONHigh mutual information between layers indicates redundancy, while low mutual information suggests that these layers have learned complementary representations [43] . When two variables x i and x j are independent, their mutual information is zero, i.e., I(x i ; x j ) = 0. According to information theory [41] and the Information Bottleneck (IB) theory [48] , minimizing the mutual information between layers can reduce redundancy, remove irrelevant information, and enhance the overall representational capacity of the network. This "compression" of the representation enables the network to extract higher-level and more compact features, thereby reducing the reconstruction error. In summary, by sparsifying layers with higher mutual information and minimizing the mutual information throughout the entire network, the reconstruction error can be minimized.To approximate the mutual information between layers, we propose employing Monte Carlo sampling, thereby circumventing the need for intractable integrals. Specifically, we randomly select N samples x(1)EQUATION0 , . . . , x(N ) 0 from the training dataset, which follow a probability density function (PDF) P (x). For each sample x (n) 0 , the outputs at the i-th and j-th layers are denoted as x (n) i and x(n) j , respectively.We can estimate the integral using the sample average: Î (x i ;x j ) ≈ 1 N N n=1 log p x (n) i ,x (n) j p x (n) i p x (n) j . Î (x i ; x j ) is the estimated mutual information, p x (n) i , x (n) jis the joint PDF, and p x (n) i and p x (n) j are the marginal PDFs. We aim to approximate these probability densities using kernel density estimation.Computing marginal and joint probability densities: Based on the kernel density estimation [44] , we can utilize it to estimate the probability density functions. This allows us to approximate the probability density functions using the features of the samples. Kernel density estimation is a non-parametric method for estimating the probability density function of a random variable. For instance, given a set of samples Y = {y 1 , y 2 , . . . , y N }, The kernel density estimate is defined as:p(y) = 1 N h N i=1 K y-yi h, where K(•) is a kernel function, commonly used kernels include Gaussian, Epanechnikov, etc., and h is the bandwidth parameter.We apply kernel density estimation to compute the marginal and joint probability density functions of the samples' outputs at the i-th and j-th layers. The choice of the bandwidth parameter h can be determined through cross-validation or other methods. However, to simplify our derivation, we can consider that in high-dimensional spaces, the influence of the kernel function K is relatively insignificant. We are mainly focused on the ratio of relative densities [40, 60] . Therefore, the bandwidth parameter h can be cancelled out. We can calculate the marginal and joint probability density functions of the samples' outputs at the i-th and j-th layers using kernel density estimation, which can be found in the Appendix.Monte Carlo Approximation of Mutual Information. Substituting the kernel density estimates into the Monte Carlo approximation formula for mutual information and simplifying the expression using the feature matrix inner product approximation for the kernel function, as mentioned by Tschannen [51] , we obtain:EQUATIONwhere ∥ • ∥ F denotes the Frobenius norm. In this approximation, we employ the feature matrix inner product to approximate the kernel function, K xEQUATIONi , x(k) j ≈ x (n)T i x (n) j F .Similarly, for the marginal kernel functions is K x(n) i , x (k) i ≈ x (n)T i x (n) i F.Decreasing function: Since mutual information has no general upper bound, its upper limit depends on the entropy of either x i or x j . To address this, we can use decreasing functions to transform the range of mutual information, ensuring a bounded and more interpretable metric. For instance, using e Î(xi;xj ) or a Gaussian function, we can redefine the measure as follows. Considering that a batch of data is fed into the model simultaneously and each layer output concurrently, we can omit the N n=1 and 1 N . Instead, we can use X i and X j to represent the calculations for the entire batch input. Applying X T i X j e Î(Xi;Xj ) as a decreasing function to Eq. 3. Therefore, we can derive the Redundancy Metric (RM) formula, where RM (•) ∈ [0, 1] according to the Cauchy-Schwarz inequality:EQUATIONThe decreasing function transforms the range of the RM formula such that a value of 0 indicates complete independence between layers, while 1 represents complete redundancy. This formulation can serve as the objective function for maximization. The complete derivation process for this section, including the details of Eq. 4, is presented in Appendix B.Our Redundancy Metric reveals the redundancy among layers in a neural network, guiding sparsity ratio allocation. Experiments on LLaMA2-13B with various sparsity configurations show a negative correlation between model redundancy and WikiText-2 perplexity (PPL). Model redundancy is defined as the sum of each layer's RM for the remaining layers, as depicted in Fig. 2 (c ). Consequently, redundancy minimization is adopted as the objective function, incorporating model size constraints to formulate a linear programming problem that yields the optimal sparsity configuration.Intra-layer Sparsity Allocation. For a given neural network, we construct a redundancy matrix Ψ, where ψ ij = RM (x i , x j ). The sum of non-diagonal elements for each row of the matrix is computed asρ i = L j=1 ψ ij -1.A smaller ρ i indicates stronger independence between x i and the outputs of other layers. We model this relationship using the monotonically decreasing function:ω i = e -1µ ρi , where µ is a dynamic hyperparameter controlling the difference in sparsity ratios across layers, defined as 1 n n j=1 ψ ij , which smooths the descent speed (Fig. 8 ). The importance factor for the first i layers is represented by ω i . With these components, we formulate the linear programming problem as follows:EQUATIONwhere S (qi) denotes the model size of the i-th layer under sparsity q i , and B represents the target model size. The optimal sparsity configuration is given by q. To maximize the model's representative capacity, our method try to assign smaller sparsity configurations to more independent layers by maximizing an objective function. For a more fine-grained sparsity allocation, we extend our approach to include intra-layer component-level sparsity allocation. After determining the sparsity ratios for each layer, we treat the remaining parameters in this layer as the target size and construct objective functions for its individual components. By applying ALS at this granular level, we obtain a secondary sparsity allocation, resulting in unique sparsity ratios for every layer and component. This hierarchical approach enables a highly customized and adaptable sparsity distribution throughout the entire network architecture, potentially leading to enhanced efficiency and performance gains. 
experimental results: 
 Setup. For pruning, we follow the settings of Wanda, SparseGPT, and Magnitude. Regarding the calibration data used in the linear optimization process, we follow the configurations of SparseGPT and Wanda, selecting data from the C4 dataset and ensuring that all test data are zero-shot. We use a calibration data size of 16 for linear optimization hyperparameters. The granularity, explained in Appendix. E.1 for linear optimization results is set to 0.5%. For the values of x i , we use the input, although output and intermediate gates can also be used. Hyperparameter analysis is primarily conducted in the analysis section. Details about the experimental environment are provided in Appendix E.1.Evaluation and Metrics. We measure the performance of pruned models through zero-shot tasks and language modeling. For zero-shot evaluation, we utilize seven tasks from the EleutherAI LM Harness [47] : Winogrande [39] , PIQA [2] , OpenBookQA [37] , HellaSwag [61] , BoolQ [6] , ARC (Easy and Challenge) [7] , and RTE (Recognizing Textual Entailment) [52] . We also include WikiText2 [36] . For the first seven datasets, we use the accuracy metric provided in the EleutherAI LM Harness. For WikiText2, we use the word_perplexity (PPL) metric. During evaluation, we ensure using the same database version, GPU model, and random seed.Models. We evaluate the performance of ALS on LLMs, including LLaMA-V1 7B/13B/30B/65B [49] , LLaMA-V2 7B/13B/70B [50] , LLaMA-V3 8B [1] , OPT 6.7B/13B [62] . Baselines. We run ALS on LLMs with various methods, including Wanda [45] , Magnitude-based pruning [18] and SparseGPT [16] .Quantitative Evaluation. In Table 2 , we compare the wikitext2 (PPL) performance of different pruning methods under 50% sparsity on the LLaMA-V1, LLaMA-V2, LLaMA-V3, and OPT models, including Dense (unpruned), Magnitude pruning [28] , SparseGPT pruning [17] , Wanda pruning [46] , and the results of these pruning methods enhanced by ALS. The results show that the ALS generally improves the performance of various pruning methods.For LLaMA-V1 models, Magnitude pruning shows high perplexity, e.g., 42 On the OPT model, perplexity significantly increases after pruning. For instance, the 13B model of OPT has a perplexity as high as 4.09e4 after Magnitude pruning, which remarkably reduces to 3.96e3 with ALS, demonstrating the effect of ALS in handling LLM pruning. However, there is an example where performance does not significantly improve with ALS. For instance, the 13B model of LLaMA-V1 has 9.90 perplexity after SparseGPT pruning, which slightly increases with ALS.In summary, ALS significantly enhances model performance across various pruning methods by effectively mitigating performance loss.Varying Sparsity Rates. Table 1 presents the perplexity scores of sparse LLaMA-V2 7B and 13B models pruned by Magnitude, SparseGPT, and Wanda methods, with and without ALS, at varying sparsity levels (20% to 70%). The results show that as sparsity increases, perplexity scores generally deteriorate, indicating a decline in language modeling performance. However, as the sparsity level increases, the performance gap between ALS and non-ALS methods widens, with ALS exhibiting better performance at most sparsity levels. This suggests that ALS can help mitigate the performance degradation caused by higher sparsity, becoming increasingly effective at maintaining LLMs performance as the sparsity level grows. In Table 3 , we present the averaged accuracy performance of pruned LLaMA-V1, LLaMA-V2, LLaMA-V3, and OPT models on seven downstream zero-shot tasks at a 50% sparsity ratio. For detailed performance on specific tasks, please refer to Table 4 , which shows improvements in most tasks. The average accuracy across the majority of tasks demonstrates the effectiveness of ALS in enhancing sparse large language models of any scale. Remarkably, for LLaMA3-8B, the incorporation of ALS leads to an improvement of 14.14% and 9.87% compared to the Magnitude and SparseGPT baselines, respectively. Similarly, for LLaMA1-13B, the addition of ALS results in an improvement of 5.95% compared to the baselines.The significant performance improvement of LLaMA-V3 8B may be attributed to the fact that the new model is not well-suited for uniform pruning methods. we further use the LLaMA-V3 8B model as an example to intuitively present the improvements brought by ALS from the perspective of the heat map (Fig. 3 in the Appendix D). The heat map reveals that, at 50% sparsity, the redundancy between layers of the LLaMA-V3 8B model exhibits a distinct pattern compared to other models. The green distribution on both sides indicates that there is small redundancy between the shallow layers and other layers. This suggests that the information captured by the middle and deep layers has more overlap and similarity with other layers. The ALS method takes advantage of this inter-layer redundancy pattern and apply high sparsity ratio into the layers with higher redundancy. It maximally preserves the key information and reduces the impact of sparsification on model performance. From the heat maps of different models in the Appendix D, it can also be observed that each model requires a different sparsity ratio because the redundancy between its layers varies. In this part, we examine the impact of various components within the ALS framework and compare it with LoRA Fine-tuning, specifically focusing on its bound setting, standardization on weight or feature, granularity choice, which can be found in Appendix. E.1, feature choice and robustness to calibration samples. All experimental setups are based on the LLaMA2-13B model with Wanda pruning and ALS.Comparison with LoRA Fine-tuning. Our experiments in Table 5 and Table 6 demonstrate the substantial benefits of combining Wanda+ALS with LoRA fine-tuning across the LLaMA model family. The improvements are most striking in the LLaMA-V1 7B model, which showed a dramatic reduction in perplexity by 4.82 alongside a 10.17% increase in accuracy. Larger V1 models also benefited, with the 30B and 13B variants showing perplexity reductions of 1.43 and 1.15, coupled with accuracy gains of 2.09% and 1.89% respectively. The LLaMA-V2 models exhibited similar positive trends, with both 7B and 13B versions showing perplexity improvements and accuracy increases. These impressive results were achieved using just 2000 C4 samples for LoRA fine-tuning in a zero-shot setting, highlighting the method's efficiency and effectiveness even with limited training data unrelated to the evaluation tasks.Feature Selection and Normalization. Table 7 (a) compares the performance of input, output, and gate features in capturing layer independence, with output features achieving slightly lower perplexity. Table 8 (b) demonstrates the significant impact of jointly normalizing features and per-layer weights. Applying this normalization strategy yields a substantial improvement in accuracy, increasing from 66.40% to 68.11%, while also reducing perplexity from 10.078 to 10.070.Comparison with OWL. We compared the performance of our proposed method with the OWL method on a set of benchmark datasets. The results are summarized in Table 9 . We adopted the optimal parameter settings described in the OWL paper. Across all tested configurations, our method consistently achieved lower values compared to OWL, demonstrating its superior performance. Specifically, in the unstructured 50% setting, Wanda with ALS outperformed OWL by a margin of 0.25 units. Furthermore, in the structured pruning settings of 2:4 and 4:8, the advantage of Wanda with ALS increased to 0.95 and 0.44, respectively.N:M Results. We also investigated the performance of our method in the N:M setting, where N features are selected from M available features. The results are shown in Table 9 . Similarly, for OWL, we used the optimal parameter combination reported in their paper. Across all N:M configurations, ALS consistently achieved lower values compared to OWL. As the number of selected features N increased, both methods exhibited performance improvements, but the advantage of ALS became more pronounced. For instance, in the 2:4 case, ALS outperformed OWL by a margin of 0.95 units, and this gap further widened to 1.82 in the 4:8 case. Overall, OWL is a method that is highly sensitive to parameter settings, and obtaining the optimal parameters may require dozens of experiments to determine the best combination. Moreover, there is no clear theoretical analysis explaining why such a combination should be used. Calibration Data. In Fig. 2 (a), we present the performance of pruning methods with different numbers of calibration samples. We use the size of 2, 4, 8, 16, 32, 64, 128, 256. Although this experiment reveals that the model's performance improves with an increase in the size of the calibration data, the improvement is quite limited. Even when comparing the scales of 2 and 256 in calibration samples, the perplexity decreases by only 0.11. These results further highlight the robustness of ALS.Boundes. In Fig. 2 (b) demonstrates the effect of pruning bounds on the performance of the LLaMA-V2 13B model. When the pruning bounds are set too high (e.g., 0.0-1.0), the model's performance significantly deteriorates from 10 1 to10 3 compared with 0.3-0.7, indicating that aggressive pruning may impair the model's representational capacity. However, when the pruning bounds are set between 30% and 70%, the model's performance remains nearly unaffected.Computation efficiency. As shown in Table 10 , our ALS involves two computational phases: the Redundancy Metric (RM) calculation, which consistently takes approximately 90 seconds across all methods, and the Linear Programming (LP) solution, requiring roughly 160-170 milliseconds. The total processing time varies notably depending on the base pruning method employed: Magnitude pruning, requiring just 1.62 seconds for its base operation, achieves the fastest total completion time of 1.51 minutes when combined with ALS. Wanda, with its base pruning time of 199 seconds, completes the entire process in 4.81 minutes, while SparseGPT, requiring 1058 seconds for its base operation, takes 19.16 minutes in total. Compared to BESA [57] with 4.5 hours for sparsity allocation and pruning, our approach is notably faster, completing the process in minutes rather than hours. 
conclusion: 
 In this work, we present Adaptive Layer Sparsity (ALS), a novel approach for optimizing LLMs through the efficient allocation of sparsity across layers. By minimizing inter-layer redundancy, ALS achieves significant model compression while maintaining performance, as demonstrated through extensive experiments on diverse LLMs and tasks. We hope ALS offers valuable insights and practical tools for deploying LLMs under limited computational resources, and that our work may shed light on the role of sparsity in LLMs and its potential for model optimization. Future research will explore the relationship between sparsity allocation and individual weight importance, and investigate the integration of dynamic sparsity allocation with pruning metrics. By pushing the boundaries of model compression and efficiency, we aim to enhance the development of more capable and accessible LLMs for diverse applications.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Our paper provides open access to the data and code, along with sufficient instructions in the supplemental material to enable others to faithfully reproduce the main experimental results. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer:[Yes]Justification: See Experiment.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: See Appendix E.3.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: No high risk for misuse. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [No] Justification: No. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: A Limitation: Computational Complexity of Intra-Layer Component Independence CalculationIn the ALS method, a key step is computing the independence matrix between layers. While calculating the matrix at the layer level for a 70 billion parameter model with 80 layers requires only an 80 × 80 RM matrix, including intra-layer components increases the matrix size to 7 × 80 × 7 × 80, leading to 49 times more computational time and resources.• High Computational Complexity: The increased matrix size results in exponential growth in computation and resource consumption. • Excessive Memory Usage: High-dimensional matrix computations require substantial memory, potentially exceeding hardware capacities.• Hybrid Solution with C++: Store intermediate data locally and use C++ to handle calculations and solve the linear programming problem. This approach can be up to 50 times faster than using Python alone. • Alternatively, calculate sparsity ratios for each layer, determine the parameters to retain, and use these as the target size for further linear programming. This approach requires only an 80 × 80 RM matrix and 7 additional 7 × 7 matrices, without significantly increasing computation time.Hybrid Solution with C++ is a preferred solution because it will keep most of independent components to maintain the model performance after pruning.This section provides a detailed derivation of the key mathematical formulae used in the paper.The objective of LLMs is to minimize the reconstruction error between the outputs of the sparse and dense models. We start by defining the mutual information between the outputs of different layers to quantify the redundancy within the model. Mutual information I(X; Y ) measures the amount of information obtained about one random variable X through observing the other random variable Y . It quantifies the reduction in uncertainty of X due to the knowledge of Y . This measure helps identify layers with high redundancy, which can be pruned to achieve a more efficient representation.The mutual information I (X i ; X j ) between two layers i and j is given by:EQUATIONwhere H (X i ) and H (X j ) are the marginal entropies, and H (X i , X j ) is the joint entropy. The entropy H(X) measures the average amount of information or uncertainty in a random variable X. These entropies are defined as:EQUATIONThe mutual information can also be expressed in terms of the probability density functions as:EQUATIONTo approximate these values, we use Monte Carlo sampling. Given N i.i.d. samplesx (1) , x (2) , . . . , x (N ) drawn from the joint distribution p(x i , x j ), the mutual information can be approximated as:EQUATIONUsing kernel density estimation to approximate the probability densities. For instance:EQUATIONHere, K(•) is the kernel function, h is the bandwidth parameter, and d is the dimension of y. Kernel density estimation is a non-parametric method for estimating the probability density function p(y) of a random vector y, given a set of i.i.d. samples Y = {y 1 , y 2 , . . . , y N } drawn from p(y). Commonly used kernels include the Gaussian and Epanechnikov kernels.To compute the marginal and joint probability density functions of the samples' outputs at the i-th and j-th layers, we apply kernel density estimation:EQUATIONwhere d i and d j are the dimensions of x i and x j , respectively.To derive the mutual information approximation, let's start with the Monte Carlo approximation of mutual information (Equation 7). Substituting the kernel density estimates (Equations 9-11) into Equation 7, we get:Î (X i ; X j ) ≈ 1 N N n=1 log N k=1 K x (n) i -x (k) i hi K x (n) j -x (k) j hj N k=1 K x (n) i -x (k) i hi N k=1 K x (n) j -x (k) jhj (16) Now, consider the feature matrix inner product approximation for the kernel function. Let:EQUATION)where ∥ • ∥ F denotes the Frobenius norm.Therefore, the joint kernel function can be approximated by:EQUATIONSubstituting these approximations (Equations 13-15) back into the mutual information formula (Equation 12), we get:EQUATIONThis leads to the final expression for the mutual information approximation using Monte Carlo sampling and kernel density estimation with the feature matrix inner product approximation.Since mutual information has no general upper bound, its upper limit depends on the entropy of either X i or X j , which represent the i-th and j-th layer output of a batch input. To address this, we can use functions to transform the range of mutual information, ensuring a bounded and more interpretable metric. For instance, using e Î(Xi;Xj ) or a Gaussian function, we can redefine the measure as follows. Considering x T i x j F e Î(Xi;Xj ) for a batch of inputs, we can remove the sum and 1 N , then obtain:EQUATIONTo derive the Redundancy Metric (RM) formula, we consider that a batch is a set of samples processed simultaneously by the neural network. Because a batch is input simultaneously, we remove the summation and division by N , obtaining the following formula:EQUATIONIn this formula, RM (•) ∈ [0, 1], where a value close to 0 indicates high independence between the layers (low redundancy), and a value close to 1 indicates high redundancy (low independence).Our redundancy metric is used to guide the allocation of sparsity ratios across layers. We construct a redundancy matrix Ψ ∈ R L×L where each element ψ ij represents the redundancy between layers i and j, computed using Equation 18. The overall redundancy of each layer is:EQUATIONA lower ρ i indicates higher independence. Using a monotonically decreasing function, we define the importance factor ω i as:EQUATION)where µ is a hyperparameter controlling the decay rate.Our objective is to maximize the sum of the weighted sparsity ratios across all layers, subject to a model size constraint B. The sparsity ratio q i ∈ [0, 1] represents the fraction of weights to be pruned in the i-th layer. The optimization problem can be formulated as:EQUATIONsubject toEQUATION)EQUATIONwhere S (qi) represents the model size of the i-th layer under sparsity ratio q i , and q = [q 1 , q 2 , . . . , q L ] T is the vector of sparsity ratios for all L layers. The objective function (Equation 22) aims to allocate higher sparsity ratios to layers with higher importance factors. The constraint (Equation 23) ensures that the total model size after pruning does not exceed the budget B.Results in LLaMA-V1,V2,V3 and OPT series at 50% Sparsity In this section, we present the visualization of correlation matrices obtained by solving the problem under different experimental settings. Additionally, we provide the numerical results of the Sparse Ratio Allocation. The RM heat maps are different with different models.50% sparsity in LLAMA-V1/V2/V3 family.LLaMA-V1 7B LLaMA-V1 7B. Initially, PPL remains relatively constant at 10.07 for granularities of 0.1 and 0.5. It then decreases to 9.86 at a granularity of 1 and further to 9.67 at a granularity of 5. However, beyond this point, the smoothed curve indicates a subsequent rise in PPL, suggesting that excessively high granularity may negatively impact model performance. This analysis highlights a critical balance in optimizing granularity to minimize PPL and enhance model accuracy and efficiency.Environment. All pruning experiments are performed on dual NVIDIA A100 GPUs with 80GB memory. However, our ALS method mainly runs on CPU, while the baseline methods Wanda, SparseGPT, and Magnitude require GPU. The CPU used is an AMD EPYC™ 9554 64-core processor.Hyperparameters We set weight and feature normalization, calibration data= 2, feature selection= in, granularity=0.05, boundes= 0.3-0.7 for 3070% sparsity and 0.1-0.3 for 20% sparsity. • The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 


Decision: The paper introduces Adaptive Layer Sparsity, a method to optimize LLMs by selectively pruning less important features in intermediate layers based on a correlation matrix, thereby reducing model size without compromising performance. Through extensive experiments, ALS demonstrates good performance compared to existing methods.

The paper is well-written and easy to follow. The reviewers had some questions and concerns about the experiments, technical details, and the motivation behind the paper. During the rebuttal, the authors addressed these issues, and the reviewers were satisfied with the responses and additional experiments. Therefore, I recommend accepting the paper.

Reviews: review: paper_summary: paper_summary: The growing size of LLMs make deployment increasingly challenging. Traditional pruning methods underperform due to uniform strategies that ignore varying feature importance across layers. The authors introduces a new Adaptive Layer Sparsity (ALS) approach that estimates inter-layer correlations using information orthogonality, accurately evaluating each layer's significance. It then employs linear optimization to selectively prune features in intermediate layers, achieving precise model optimization. Experiments demonstrate superior efficiency of ALS over existing sparse models like Wanda and SparseGPT, especially at high sparsity levels.
strength_weakness: strength_weakness: 1. The presentation of this paper should be improved. Long paragraphs with various discussion points are mixed together, which is a major weakness of this paper.
2. Evaluation could be improved with numbers on the inference efficiency comparison.
questions: questions: 1. What is the computation efficiency of ALS-pruned model compared with existing baslines?
2. What is the overhead of solving the linear programming problem?
3. It seems the performance improvement of ALS decreases when the model becomes large. Is there any reasons for that?
limitations: limitations: The authors have addressed the limitations and there is no potential negative societal impact of this work.l
paper_summary: paper_summary: The paper presents an approach called Adaptive Layer Sparsity (ALS) for optimizing large language models (LLMs) by selectively pruning features in intermediate layers. The approach consists of two key steps: estimating the correlation matrix between intermediate layers and employing a linear optimization algorithm to develop an adaptive sparse allocation strategy based on the correlation matrix.
strength_weakness: strength_weakness: 1) From Table 1, it seems that when the sparse ratio is low (e.g., 20%), the performance improvement with ALS is marginal. Does that indicate it is useless for not very sparse models?

2) From Table 2, for stronger models (e.g., 65B and 70B), the improvement is far less than smaller models. Therefore, it is questionable about the applicability of the proposed method to strong models.

3) The authors conducted comprehensive experiments on Llama-series models, but omit Llama-3-80B. Are there any justifications? Since Llama-3-80B is one of the strongest models for now, it would enhance the paper if you can include such results.
questions: questions: N/A
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes using mutual information to measure layer redundancy in LLMs and employs a linear optimization algorithm based on this measure to derive adaptive sparsity strategies, enabling dynamic sparsity configuration across different layers. The authors conducted experiments on four models, including three versions of LLaMA and OPT, comparing the proposed approach with existing pruning algorithms such as Wanda and SparseGPT. The results demonstrate that the proposed method outperforms these existing algorithms in most scenarios.
strength_weakness: strength_weakness: While the combination with ALS effectively improves the baseline performance, there are concerns about the practical value of the method. Specifically, when comparing the pruned models with dense models of similar size, the reported performance is relatively close. An extreme example of this is found in Table 2, where the perplexity (PPL) of the 50% sparse LLaMA-v1 30B model is higher than that of the dense LLaMA-v1 13B model, regardless of the pruning method used. This raises questions about the overall effectiveness of the pruning approach in terms of model efficiency and performance trade-offs.
questions: questions: - How is the intra-layer pruning calculation performed?
- How is ALS integrated with the baseline method?
- What is the meaning of L-i+1 in equation (5)?
- The analysis in lines 274-276 refers to the 13B model, but the data corresponds to the 7B model in the table, which is confusing. Could you clarify this discrepancy?
- What would be the effect of combining Wanda+ALS with a small amount of LoRA fine-tuning?
- What is the relationship between sparsity and efficiency as studied in this paper? Does it reduce computation time or memory usage?
limitations: limitations: The paper lacks some crucial details in certain areas. For instance, in section 4.3, the process of LoRA fine-tuning is mentioned but not adequately explained. It would be beneficial if the authors could provide more comprehensive information on this aspect.
paper_summary: paper_summary: This paper introduces Adaptive Layer Sparsity (ALS), a novel approach aimed at optimizing large language models (LLMs) through selective pruning. The key contributions of this work include a method that estimates the correlation between intermediate layers using information orthogonality, enabling the precise measurement of each layer's importance. Additionally, the paper formulates the sparsity allocation problem as a linear programming optimization, facilitating efficient global optimization of layer-wise sparsity ratios. Extensive experiments conducted on various LLM families, including LLaMA and OPT, demonstrate consistent performance improvements over existing pruning methods across different model sizes and sparsity levels. The authors also perform analysis experiments to examine the impact of various factors, such as calibration data, sparsity bounds, and feature selection, on the method's performance.
strength_weakness: strength_weakness: 1. There is significant overlap between the introduction and related work sections, which could be consolidated to improve flow and reduce repetition.

2. Some details about ALS, such as normalization and reweighting strategies mentioned in the ablation study section, are not clearly described in the "method" section. These should be elaborated with formulas in the method section.
questions: questions: See weakness 2. Could you provide some details about the normalization and reweighting strategies mentioned in the ablation study section?
limitations: limitations: N/A

Comments: title: Thanks for the Positive Feedback and Recognition of Our Work and Rebuttal
Look Forward to The Post-Rebuttal Feedback.
comment: Thank you for the response and including the numbers on the cost of solving the LP problem. It appears to be a significant improvement over the baseline.
Dear Reviewer 7kYG,

Thank you so much for the recognition of our responses. We are glad to your positive feedback! Thanks!

Following your constructive suggestions, we will make more efforts to improve our paper further.

Many thanks for your constructive comments, time and patience.

Best regards,

Paper 688 Authors
Thank you for your detailed response, which has resolved most of my concerns. Therefore, I am raising my score to 5.
Dear Reviewer 7kYG,

Thanks for your careful and constructive comments again. We have addressed your concerns in our rebuttal point-by-point. Please let us know if there are any further questions. If our response alleviates your concerns and clarifies the value of our paper, we would be truly grateful if you could reconsider your recommendation. We promise to thoroughly reflect all your comments in the final manuscript. We believe we have faithfully incorporated the feedback from all four reviewers and hope this is reflected positively in your evaluation. Thank you for taking the time to read our response.


Best regards,

Paper 688 Authors