Discovering Sparsity Allocation for  Layer-wise Pruning of Large Language Models
Authors: Lujun Li
Peijie Dong
Zhenheng Tang
Xiang Liu
Qiang Wang
Wenhan Luo
Wei Xue
Qifeng Liu
Xiaowen Chu
Yike Guo
Venue: NeurIPS 2024 poster (2024)
Keywords: network pruning
layerwise sparsity allocation
large language models
model compression.
Primary area: natural_language_processing
TL;DR: In this paper, we present a new method for optimizing layerwise sparsity allocation in large language models.

Abstract: In this paper, we present DSA, the first automated framework for discovering sparsity allocation schemes for layer-wise pruning in Large Language Models (LLMs).  LLMs have become increasingly powerful, but their large parameter counts make them computationally expensive.  Existing pruning methods for compressing LLMs primarily focus on evaluating redundancies and removing element-wise weights.  However, these methods fail to allocate adaptive layer-wise sparsities, leading to performance degradation in challenging tasks.  We observe that per-layer importance statistics can serve as allocation indications, but their effectiveness depends on the allocation function between layers.  To address this issue, we develop an expression discovery framework to explore potential allocation strategies.  Our allocation functions involve two steps: reducing element-wise metrics to per-layer importance scores, and modelling layer importance to sparsity ratios.  To search for the most effective allocation function, we construct a search space consisting of pre-process, reduction, transform, and post-process operations.  We leverage an evolutionary algorithm to perform crossover and mutation on superior candidates within the population, guided by performance evaluation.  Finally, we seamlessly integrate our discovered functions into various uniform methods, resulting in significant performance improvements.  We conduct extensive experiments on multiple challenging tasks such as arithmetic, knowledge reasoning, and multimodal benchmarks spanning GSM8K, MMLU, SQA, and VQA, demonstrating that our DSA method achieves significant performance gains on the LLaMA-1|2|3, Mistral, and OPT models.   Notably, the LLaMA-1|2|3 model pruned by our DSA reaches 4.73\%|6.18\%|10.65\% gain over the state-of-the-art techniques (e.g., Wanda and SparseGPT).

Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models
Lujun Li Peijie Dong"t, Zhenheng Tang
Xiang Liu?, Qiang Wang', Wenhan Luo", Wei Xue Qifeng Liu Xiaowen Chu Yike Guo
Hong Kong University of Science and Technology
Hong Kong University of Science and Technology (Guangzhou)
3 'Hong Kong Baptist University "Harbin Institute of Technology (Shenzhen)
lilujunai @aaaeii@@maaal....m com, adong212@conneee hwwst-gz. edu. en, zhtang@ccmp. hkbu. hk, xliu886e connect. hkust-gz edu. en, giang.wangeen edu. cn, (whluo, weixue, liuqi feng, xwcha, yikeguo fust.
Abstract
In this paper, we present DSA, the first automated framework for discovering sparsity allocation schemes for layer-wise pruning in Large Language Models (LLMs). LLMs have become increasingly powerful, but their large parameter counts make them computationally expensive. Existing pruning methods for compressing LLMs primarily focus on evaluating redundancies and removing element-win wise weights. However, these methods fail to allocate adaptive layer- wise sparsities, leading to performance degradation in challenging tasks. We observe that per-layer importance statistics can serve as allocation indications, but their effectiveness depends on the allocation function between layers. To address this issue, we develop an expression discovery framework to explore potential allocation strategies. Our allocation functions involve two steps: reducing element- wise metrics to per-layer importance scores, and modelling layer importance to sparsity ratios. To search for the most effective allocation function, we construct a search space consisting of pre-process, reduction, transform, and post-process operations. We leverage evolutionary algorithm to perform crossover and mutation on superior candidates within the population, guided by performance evaluation. Finally, seamlessly integrate discovered functions into various uniform methods, resulting in significant performance improvements. We conduct extensive experiments multiple challenging tasks such arithmetic, knowledge reasoning, and multimodal benchmarks spanning GSM8K, MMIJU, SQA, and VQA, demonstrating that our DSA method achieves significant performance gains on the LLaMA-11213, Mistral, and OPT models. Notably, the LLaMA- A-1/213 model pruned by DSA reaches 4.73% 116110.55% gain over the state-of-the-art techniques (e.g., Wanda and SparseGPT).
Introduction
Large language models (LLMs) [63, 51, 4] have ushered in new era of natural language processing (NLP) [56], demonstrating remarkable capabilities understanding and generating human- -ike text [55]. However, recent LLMs have an incredibly large number of parameters, which contributes to their high computational resource consumption. For example, OpenAl's GPT-3 model has 175 billion parameters and consumed 284,000 kWh of energy during its training [9]. The exponential growth in model size and complexity presents challenges, especially for deployment on resource-constrained
*Corresponding authors, equal contribution. Codes https///gittu..co/llla//SAA
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Layerwise Sparsity Ratio
Unitor 75% Spariiyy
Spansay flato itlesl
uniforn 22%k Spaniity
ONL
Uiiiom
Peruayer insprtance [0em)
Per Loyer Imporio rporignne (OWL)
USA [Durs]
Layerwise Sparsity
Papplaaiey
Layer Index @ LLaMA-VI-78
Layer Index @ LLaMA-Vi-7B
Sparsity (%) LLaMA-Vi-78
Figure 1: Sparse ratios by our method (left) and OWL (middle), WikiText-2 perplexity results (right).
devices. As a result, there is a pressing need to develop effective compression techniques [3, 67] that can reduce the size of LLMs while preserving their performance. One promising approach is pruning, which involves removing redundant or less important parameters from the model.
Conventional pruning methods [21] propose extensive pruning metrics [61] and sparse training strategies [21]. However, these traditional methods often involve performance drops on small-scale models and require extra fine-tuning, making them difficult to transfer to LLMs due to differences in model structure and the high cost of the extra fine-tuning. To address this, recent approaches like SparseGPT [16] and Wanda [49] have been developed specifically for pruning LLMs. SparseGPT prunes insignificant weights and reconstructs layer-wist outputs based on an importance metric obtained from the Hessian matrix. Wanda proposes a streamlined approach that simplifies the compu- tations by using only the product of weight and activation magnitudes. Despite these advancements, these pruning methods share common limitation: they uniformly set sparsity ratios for different layers in LLMs, failing to account for the varying importance each layer in the model's s overall performance. Intuitively, the front layers of LLMs are considered more important, as they establish the fundamental language understanding upon which the rest of the model relies. The limitations of uniform pruning contradict this intuition and lead to performance degradation of pruned LLMs with high sparsity ratios or on difficult language understanding tasks. However, achieving non-uniform pruning is also difficult, mainly due to two challenges: (1) Additional costly computations: Most non non-uniform methods are trial-and rial-and-errov paradigms requiring many computations and evaluation overheads. For example, BESA [58] performs differentiable iterative optimization for bisck-wise sparsity allocation based on evaluation results. However, this method primarily focuses on intra-layer sparsity configuration and takes least 5 hours, which is considerably slower compared other training-free approaches. Additionally, the overheads of these methods grow with both the number of layers and the sparse granularity of LLMs. (2) Fixed and empirical allocations: Recent methods like OWL [60] assign different sparsity ratios based on the outlier ratio within each layer. This empirical method requires tuning hyperparameters such as the outlier threshold and sparsity upper/lower bounds to obtain optimal values, and it heavily relies on empirical analysis and handcrafted design without providing a solid theoretical foundation for its effectiveness, making it difficult to scale various models and datasets (see Figure (right)). These dilemmas naturally raise the question: How can we efficiently obtain adaptive allocation strategies capable of handling different models and scenarios?
To answer this question, we analyze the distribution of wlis scores of different layers in LLMs based existing sparse methods. As shown in Figure (left), we observe that the mean values of the per-layer element importance scores of the front layers enjoy larger values. This observation aligns with OWL (see Figure (middle)) and the understanding that the initial layers of LLMs are more important [13]. Furthermore, we find that other reduction operations, such as variance, entropy, etc., also yield distributions with similar trends. Motivated by these findings, we explore the possibility of directly utilizing these layer-wise importance scores as a guiding principle for allocating sparsity ratios across layers. Thus, we transform the reduction values of element-wise scores using various functions (e.g., sine, exponential) and employ the transformed values as layer-wiss sparsity ratios. Such non-uniform manners bring promising gains, which are sensitive to specific transform operations. These observations inspire us to combine advanced reduction operations and transform functions to obtain an effective sparsity allocation based on the original importance scores from uniform pruning methods. However, how to obtain the most promising combinations? Fortunately, the recent advancements in AutoMI. [71, 45, 34] provide potential answers by enabling the automatic search for optimal solutions within defined search space. By formulating this problem as an

---

AutoML task, we can leverage search algorithms to efficiently explore the search space and identify the effective combinations for non-unn non-uniform pruning in LLMs.
Based on our observations, we introduce DSA, an innovative framework that leverages expression discovery and evolutionary algorithms tailor sparsity allocation schemes for LLMs. DSA seeks to find the best functions for mapping element- wise scores + per-layer importances + sparsity ratios. To identify the most effective allocation function, we construct vast search space encompassing diverse set of pre-process, reduction, transformation, and post-process operations. The pre-process operators, such as Frobenius norm and softmax, normalize the original importance values, enabling fair cross-layer comparison. Reduction operations, including variance, standard deviation, entropy, and geometric mean, extract the element- wise importance scores of each layer into a single repre- sentative value. Recognizing the potential for complex nonlinear relationships, we incorporate a wide range of mathematical functions as transformation operations, including sine and cosine. These functions provide flexibility to model intricate patterns and amplify or attenuate the importance scores as needed. Additionally, we introduce post-process operations further increase the upper bound of the function fit. Within this rich search space, DSA employs evolutionary algorithm to explore and discover promising allocation function candidates. The evolutionary process begins by initializing a population of diverse allocation functions, which are then iteratively evaluated and evolved through crossover and mutation operations. The crossover operation exchanges beneficial components be- tween high-pes nigh-performing parent candidates, while mutation introduces random perturbations to promote exploration. Once the evolutionary process converges, DSA selects the top- pep-perforning allocation function candidates and seamlessly integrates them into existing pruning methods, such Wanda and SparseGPT. through a plug-and-play mechanism. By leveraging the discovered allocation functions, these pruning methods can achieve significantly improved performance, maximizing compression while minimizing accuracy degradation. By automating the search process, DSA eliminates the need for manual tuning and expert intervention, reducing the time and effort required find allocation strategies. The expressiveness of the search space and the ability to combine diverse operations enable the discovery of intricate, nonlinear allocation functions tailored to the unique characteristics of each LLM.
We conduct extensive experiments on publicly available language processing datasets and benchmarks. The experimental results demonstrate our method achieves significant performance gains on multiple challenging tasks such arithmetic, knowledge reasoning, and multimodal tasks spanning GSM8K, MMLU, VQAv2, SQA, and VQA benchmarks across multiple model architectures including LLaMA- 11213, Mistral, Vicuna and OPT. Notably, our DSA method yields substantial improvements across all evaluated models, with peak gains of 14.58% in LLaMA-3 8B under magnitude pruning and 10.65% when integrated with SparseGPT. Even under high sparsity ratios of 60-70%, our method maintains robust performance, achieving improvements of 7.68% for LLaMA-2- 13B at 60% sparsity, In multimodal tasks, DSA demonstrates exceptional capability by surpassing conventional pruning methods across all benchmarks, achieving superior scores of 76.08% on VQAv2, 65.57% on SQA, and 54.36% on VQA for LL with Vicuna-7B 50% sparsity. The method's effectiveness is particularly evident in challenging scenarios, maintaining strong performance even under aggressive pruning conditions while consistently outperforming existing sparsity allocation approaches across model scales and architectures.
Related Work
Model compression techniques [24, 57], such quantization [15, 40, 33] and sparsifica- tion [16, 65, 38), are practical approaches to reduce size of large language models (LLMMs). Sparsi- fication, or network pruning, increases the number of zero-valued weights and can be categorized into structured [39, 23] and unstructured [15, 66, 64] pruning. While determining pruning criteria and ratios is crucial, the massive scale of LLMs presents challenges for efficient pruning. Conven- tional retraining based methods [65, 23] are often impractical due to high computational demands. Researchers develop LLM-speeific pruning techniques that prioritize training-free free and time-efficient approaches to address these challenges. SparseGPT [16] introduces an efficient Hessian matrix estimation technique to large-scale models. Wanda [49] further simplifies the approach, reducing the computational overhead and memory requirements. LLM- [39] examines model dependencies by incorporating first-order and approximated Hessian information, providing a comprehensive prun- ing approach. LLM Surgeon [53] adapts Kronecker-factored curvature approximations specifically
3

---

Allocation Function Search Space
Exp
Flement-wise Scores
Var
Cos
Log
Sparsity Ratios
Nerm
Exp
No
Abs
Stdd
Abs
Tnnh
Allocation Function Evolution
Sgrittantin Perfonuaet
Abs
Mean
Exp
Layor Fades
Norm
Laver Index
30
Pro-gricos ops
Log
Abs
Reduction ops
266
No
Transform eps
22
500
1000
1500
2000
Search Iterations
Pose precess ope
Overview of our DSA framework. We search for allocation functions to map element- element-wise scores to sparse ratios. We build pre-process, reduction, transform, and search space for the allocation function, and then perform evolutionary search.
Figure 2:
operations as the
for LLMs. Despite these ad advancements, most existing methods apply a uniform pruning rate across all layers, which may result in suboptimal performance. To address this, we present the first allocation function search for layer-adaptive sparsity, effectively minimizing performance degradation while achieving high compression ratios. Our method differs significantly from traditional layer-wise sparsity approaches for neural networks [12, 5, 26]. These methods often lead to the accumulation of errors across layers, as the pruning decisions for each layer are made independently without considering the global impact the model's performance. Recent BESA [58] has shifted focus to intra-block sparsity allocation, employing various techniques to optimize
sparsity distribution within individual blocks or layers. FLAP [2] applies sparsity ratios process updating for performance compensation. contrast, our method layer-wise and training-free, with finer-grained allocation and an efficient process. OWL [15] requires experts' empirical design and tuning of hyperparameters. By automating the allocation process, our approach eliminates the need for manual intervention of OWL [15]. Our method differs from approaches like Pruner-Zero [44] in both the search object and the technique type. Pruner-Zero is a uniformly sparse method that employs a metric-optimized strategy. In contrast,
method explores non-uniform sparsity allocations, searching for optimal allocation strategy. Additionally, our DSA method deviates from layer-dropping techniques [18, 14], which involve directly removing entire layers from the model architecture. Our approach preserves the model's 's overall architecture while strategically distributing sparsity across layers, allowing for high compression ratios without sacrificing significant performance relying extensive fine-tuning. More discussion are Appendix A.
3 Methodology
3.1 Recap of Sparsity Methods for Large Language Models
Sparse methods introduce sparsity into the model weights by identifying and pruning redundant or less important weights for a given pre-trained dense weight matrix W. To determine which weights should be pruned, sparse methods employ pruning metrics or importance scores, denoted as S(W,,X, Gi), where W, represents the weights of layer 1, and X; and G; are any layer-specific activations, gradient statistics, or the respective. The pruning metric ranks the weights based on their importance, and ranking results with the sparsity ratio á serves as a threshold to select the most significant weights. The pruned weights binary tensor the same shape as W,. The mask is obtained by applying a threshold function f to the pruning metric S and the sparsity ratio dy:
typically represented as sparse mask Mr, which is
M f(S(Wi, X1, Gr).
(I)

---

Table 1: Some operations in our search space, Full operations are in Appendix D.
ID
OP Name
Expression
TP
OP Namc
Expression
OPO0
Mean
mean(.c]
OP09
sigmoid
Ite5 A7
OPO1
std
/var(c))
OP10
softmax
OP02
var
var(z)
OP03
sqrt
V
OP11
exp
OP04
geometric
VIT= m
OP12
abslog
In ær|
OP13
cosine
cos
OP05
corref'
p mean(ar) adla))
OP14
sine
(in(
OP06
12 noror
OP15
log
Inx rank(z)
OP07
II norm
OP16
no_op
OP08
entropy
Biy log 25;
OP17
rank
The pruned weights Wrqussedd are then obtained by wlement-wise multiplicaiionnoo of the original weights Wi and the sparse mask Mi;
Wiinsked M @ Wt:
(2)
where @ denotes element-wise multiplication. The choice of the pruning metric S and the sparsity ratio @ significantly impacts the effectiveness of the sparse method. The basic pruning metric the magnitude- based approach [19], where S(W)) [ employs the element-wise absolute value to assess weight significance. The sparsity ratio ø plays a crucial role in determining the level of sparsity introduced into the model. Higher values of & correspond to higher levels of sparsity, resulting in greater reductions in model size and computational requirements. However, excessive pruning removed. In contrast to may lead to significant performance degradation if important weights traditional methods that use a fixed sparsity ratio ø for all layers, our adaptive sparsity allocation scheme allows the sparsity ratio Φi to vary across layers based on their importance. This approach is motivated by the observation that different layers in a deep neural network contribute differently to the overall model performance, and uniform sparsity ratio may not be optimal. The advance of our adaptive sparsity allocation scheme is its ability to identify and selectively prune the less important layers, achieving better compression while preserving the model'ss performance.
4 Allocation Function Search Space
Allocation Function Representation. Our allocation function A aims to map element wise scores to per-layer importance values V, and subsequently map these importance scores sparsity ratios The allocation functions are represented as computation graphs consisting of various pre- pre-process Tpr.. reduction Treduce, transformation Tuans and post-process Tpast operations, as follows:
Treduss (S))).
(T (V)), A= = V(o(S))
Motivation of Allocation Function Design. As discussed in introduction and Figure/ 1 (left), our design is motivated by analyzing element-wise score distributions: (1) We notice that mean, variance, and entropy values of per-layer element-wise scores can serve allocation indicators, inspiring reduction operations. (2) While basic reduction of element-wise scores showed modest improvements, applying transform functions yielded more promising results, prompting the introduction of transform operations. (3) We include pre-proces to normalize scores for fair comparison and post-proceds to further enhance function fit's upper bound. These observations naturally encourage us to employ the four cascading operations for search space.
Primary Operators. Table presents subset of the primary operators considered in our search space, which is organized into four main categories:
Pre-process operations T Tpre are applied the element-wiss scores S to prepare them for the subsequent reduction step. Tp standardizes inputs by normalizing scores across layers, ensuring consistent performance metrics by addressing scale variations. These operations can include clipping. normalization, or applying non-linear transformations.
Reduction operation Treduce aggregates the pre-processed scores into a single per-layer im- portance score V. Treduce condenses element wise information by extracting representative values and reduces computational complexity. They use statistical measures like mean, standard deviation, variance, and entropy to provide insights into the distribution of the input data.
5

---

Transform operation Tuans models the distribution of per-layer scores V and transforms this into sparsity ratios Ø, enabling the representation of intricate patterns in layer importance. This can involve non-linear transformations like sigmoid, softmax, exponential, and logarithmic functions, which capture complex relationships, while trigonometric functions like sine and cosine capture periodic patterns or cyclical behaviors.
Post-process operation Test plays the role of augmenting the fitting power and flexibility on transform operation. Toost ensures that the sparsity ratios & satisfy any required constraints, such as being between 0 and across all layers. By combining these diverse operators, our framework constructs tailored allocation functions that capture the unique characteristics cach LLM.
Allocation Function Evolution
Search Objectives. Our search goal is to find the optimal combination of operations T that makes the sparse model perform optimally on the validation set given the sparsity metric and the overall model size constraints C. This can be formulated as optimization problem
argmax T
O W X.Y) s.t. Size(Mo O W) <C,
where W represents the weights of the LLMs, X and Y are the input and target data of the verification
set, respectively, and P is the performance metric (e.g., perplexity). The mask M is determined by the sparsity ratios & through combination of operations Tpre. Traduce, Tirass and Tpost. as
M = f(S,o) = f(S, Tectuce(Tpost (Tuans (S))))
(5)
To solve this problem, we need to search a combination of 'T in 4 levels with around 10 options in each level, resulting in rather large space (i.e., ((0*)). In contrast to simple random search, we develop an evolutionary search for optimal allocation function.
Evolution Search Procedure. Our search process begins by generating an initial population of allocation function candidates, which can be created randomly or using heuristic techniques. Each candidate in this population corresponds to a unique combination of operations. Next, the perfor- mance of each candidate allocation function is evaluated. This involves computing the sparsity ratios by applying the candidate function to the sparsity metric, evaluating the pruned model on a validation set using a performance metric, and checking if the pruned model's 's size satisfies the given constraint. Based on this performance evaluation, the fittest candidates are selected for the next generation, considering criteria such as the performance metric, model size constraint, combination thereof. These selected candidates then undergo evolutionary operations like mutation and crossover to generate new population of candidates for the subsequent iteration. The search process continues iterating until a stopping criterion is met, such as maximum number of iterations or satisfactory performance level. To accelerate the search, employ various techniques: (1) Program checking uses static analysis to discard invalid candidates early, reducing computational overhead. (2) Memoization and caching store and reuse results from previous evaluations, avoiding redundant computations. (3) Parallel evaluation distributes the performance evaluation of different candidates across multiple computing resources. (4) Surrogate models approximate computationally expensive evaluations using techniques like neural networks trained on a subset of data. After each iteration, the performance of the best candidates is verified on a held-out validation set separate test set. These acceleration settings allow at least 100 times faster searches. In this way. we search our allocation function only 0.5 day a 1x NVIDIA GPU H800 server based Wanda using perplexity results from the validation set of LLaMA-1-7B on WikiText2 [41]. We confirm that no search was performed on the test set, ensuring the comparisons are completely fair. In addition, the discovered allocation functions are transferable to other tasks without massive costs. Thus, the search cost can be spread across multiple pruning runs.
6 Discovered Allocation Function Analysis
One of the top-performing allocation functions discovered through the evolutionary search process is:
Trutnce (S))
In (In())),,) "" Tpun (Trrm (V)) exp (cos (V")),
(6)

---

Table 2: Mean accuracies (%) of our DSA at 50% sparsity rate 7 zero-shot tasks.
Model
LLaMA-1
LLaMA-2
ILaMA-3
OPT
Method
7B
13B
30B
65B
7B
13B
70B
8B
6.7B
Dense
64.32
66.84
69.80
71.21
64.36
67.08
71.52
68.28
55.50
Magnitude
50.83
51.26
58.22
67.14
54.69
57.37
64.90
40.91
37.90
+ DSA (Ours)
53.73
59.05
60.55
67.86
57.90
61.38
68.76
55.50
40.25
Gain
2.901
7.781
2.331
0.721
3.211
4.011
3.851
14.581
2.351
Wanda
56.60
62.86
66.96
69,42
59.72
62.53
70.14
55.93
45.19
+ DSA (Ours)
59.22
63.03
67.80
70.98
60.80
64.87
70.54
60.70
45.45
Gain
2.621
0.171
0.841
1.561
1.091
2.341
0.401
4.761
0.261
SparseGPT
53.60
62.08
63.97
67.20
54,26
57.92
68.02
51.77
52.38
+ DSA (Ours)
58.33
62.49
67.63
67.32
59.22
64.10
68.70
62.41
55.15
Gain
4.731
0.411
3.661
0.121
4.951
6.181
0.681
10.651
2.771
where Tpe consists of two steps: log and abslog. The log step applies the natural logarithm operation In(S) to the imput importance values S, compressing the range of values and potentially highlighting differences in smaller values. The abslog step computes the absolute value of the natural logarithm, [nn,)),, ensuring that negative values potential cancellations or sign changes. T Testuce applies the geometric the result of abslog. This operation further compresses the range of values and introduces a nonlinear transformation. Trum is the cosine function (z) applied to the output of the geometric mean. This periodic function introduces oscillatory behavior, which can capture potential cyclical patterns or dependencies in the importance values. Finally, Tpust applys the exponential function exp(x) the result of the cosine operation. This step reintroduces nonlinearity and expands the range of values, potentially amplifying or attenuating the importance scores as needed.
treated symmetrically with positive values, preventing operator V VIT== 2i
Stability Analyses. To show that the function
VII In (In(S)), is stable under small
perturbations in the input S, we can derive an expression for the difference V(S + AS) - V(S) and analyze its behavior for small AS. The difference is approximately:
V(S + AS) V(S) 1 \V()) In Σ sgm sgn(In((((5))
II [K)(8) 371
AS ((AS)))
(7)
in(S) S
For small AS, the second-order term (((SS) becomes negligible, and the leading term is linear in AS. The coefficient of AS in this leading term is product of bounded functions of S. Therefore, for small perturbations AS around any positive value of S, the difference is also small, and the function V(S) is stable under such perturbations. More analyses
the Appendix B.
7 Experiments
In this section, we conduct detailed evaluation experiments on multiple tasks and models. For pruning and evaluation, we follow the settings of Wanda, SparseGPT and ensure using the same database version, GPU model, and random seed across all experiments to maintain consistent conditions. More experimental results are in Appendix C.
7.1 Experiments on Zero-shot Tasks
Implementation. To verify the effectiveness and generalizability, we perform extensive evaluation of our models on 7 zero-shot tasks. We employ a set of seven tasks sourced from the EleutherAI
LM Harness [50]. These tasks include Winogrande [46], OpenBook [42], HellaSwag [62],
BoolQ [6], ARC [7], and RTE [54]. To assess the performance of our Dynamic Sparse Allocation
(DSA) method, we evaluate its effectiveness on several models. These LLMs include LLaMA-1 (7B/13B/30B/6BBB [51]. LLaMA-2 (7B/13B/700) 3BB/7BB [52], LLaMA-3 (8B) [1], and OPT (6. (B.7BB/))) [63]. Our allocation function applied to different pruning methods, namely Wanda [49], Magnitude- based pruning [20]. and SparseGPT [15]. For fair comparisons, we follow the same configurations of
SparseGPT and Wanda methods. We select data from the C4 dataset and ensure that all test data used in the evaluation from zero-shot settings.
7

---

Table Mean accuracies (%) of our DSA on 7 zero-shot tasks at 60% sparsity rates.
Table 4: Mean accuracies (%) of our DSA on 7 zero-shot tasks at 70% sparsity rates.
Meibod
LLMMA-2-7B
L1aMA-2-138
LLMMA-3-70B
Michoo
LLaMA-2-78
LLaMA-2-138
LLMMA-27708
Magniudee
50.81
51.16
55.86
Magniude
35.61
35.38
38.76
DSS (Ours)
$7.84
54,28
60.24
+ DSA (Ours)
37.95
46,06
42.98
7,037
3.127
Gain
2.347
7.687
4.221
Wanda
60.90
72,00
40.51
Wanda
36.08
41,46
40.44
DSA (Ours)
62.08
73.24
42.74
+ DSA (Ours)
38,00
43,18
42.78
L
1.24
1.72
2.341
SparseGPT
60.68
70.14
65.03
SparseGPTT
43.61
48.76
43.22
DSA (Ours)
61.31
72,12
67.34
DSA (Ours)
45,56
50,04
45.73
0.631
1.987
Gain
2.511
Sparsity Results on Varying Models. The results in Table demonstrate the effectiveness of our allocation function in improving the performance of pruned models across various methods and model architectures. When integrated with magnitude- based pruning, DSA yields substantial improvements across all evaluated models, with particularly impressive gains in LLaMA-3 SB, where accuracy increases by 14.58% For the Wanda pruning method, DSA consistently enhances performance, achieving notable improvements of 4.76% in LLaMA-3 8B and 2.62% in LLaMA-1 7B, while maintaining stable gains across larger models such as LLaMA-2 70B with 0.40% increase. The integration of DSA with SparseGPT produces the most striking results, with substantial improvements of 10.65% for LLaMA-3 3 8B and 18% for LLaMA-2 13B, demonstrating its exceptional capability to optimize sparsity patterns. DSA shows partieular strength in enhancing smaller models, with LLaMA- 1 7B experiencing gains of 4.73% under SparseGPT and 2.90% under magnitude pruning, while also maintaining effectiveness across larger architectures such as LLaMA-2 70B, where it achieves improvements of 3.85% under magnitude pruning. These consistent performance improvements across different model scales, from the 6.7B OPT to the 70B LLaMA-2, highlight the ability of DSA to migrate well and generalize across different pruning techniques and model architectures, enabling improved performance and efficient compression of L.LMs while minimizing the impact on their zero-shot capabilities.
Sparsity Results under High Pruning Ratios. The experimental results in Table 3 and Table demonstrate the robust performance of DSA under high pruning ratios across different model scales and pruning methods. When integrated with magnitude- based pruning. DSA exhibits remarkable improvements, achieving gains of up to 7.68% for LLaMA-2 13B at 60% sparsity and 7.03% for LLaMA-2-7B at 70% sparsity. In combination with Wanda, DSA consistently enhances performance across all models and sparsity ratios, with particularly notable improvements in LLaMAA3-70B, where it achieves gains of 2.23% and 2.34% 60% and 70% sparsity respectively. The integration with SparseGPT yields steady improvements, with the most significant gains observed in I.LaMA- 3-70B (2.51% % at 70% sparsity) and LLaMA-2-7B (1.95% at 60% sparsity). DSA's effectiveness is particularly evident in challenging scenarios, such maintaining LLaMA-2-13BB performance at 73.24% accuracy even under 70% sparsity when combined with Wanda, and achieving 67.34% accuracy with LLaMA-3-70B at 60% sparsity when integrated with SparseGPT, demonstrating its capability to preserve model performance even under aggressive pruning conditions.
Compare Other Sparsity Allocation Methods. Table 5 shows WikiText-2 perplexity results
demonstrate the superior performance of DSA across varying high sparsity rates from 65% to 80% in LLaMA-1-7B, At 65% sparsity, DSA achieves the lowest perplexity of 1 12.62, outperforming OWL's 13.05 and showing substantial improvement over traditional methods like Uniform (20.85) and ER (45.85). The performance advantage of DSA becomes more pronounced as sparsity increases, reaching a perplexity of 736.81 at 80% sparsity, which represents a significant improvement over OWL (1002.87) and BESA (2208.75). Notably, DSA demonstrates remarkable stability under extreme sparsification, maintaining performance far superior to conventional approaches like Global and plup, which deteriorate dramatically with perplexities of 39918.56 and 6013.91 respectively at 80% sparsity. The consistent superiority of DSA across all sparsity levels, particularly its ability to maintain relatively low perplexity even at 80% sparsity, validates its effectiveness in allocating sparsity while preserving model performance. By constructing tailored allocation functions that capture the unique characteristies of each layer, DSS achieves superior performance compared to other commonly used layerwise sparsity methods, especially at higher sparsity rates.
8

---

Table 5: WikiText-2 perplext (4) performance of various allocation methods with the Wanda metric
for sparse LLaMA-1-7B at varying high sparsity rates (65 (60%)...)
Method
Global [60]
ER-plus [36]
[43]
Uniform (69)
BESA [58]
OWL [60]
DSA (Ours)
65%
867.82
97.28
45.83
20.85
18.52
13.05
12.62
70%
5147
229.17
112.03
81.18
42.58
24.54
22.60
75%
25863.75
1482.93
3287.92
927.42
257.89
152.47
103.32
80%
39918.56
6013.91
11151.18
3499.88
2208.75
1002.87
736.81
Table 6: 50% Sparsity results (%) on GSM8K. Table 7: 50% Sparsity results (%) on MMLU.
LLaMA-1
LLaMA-2
Mistral
LLaMA-1
LLaMA-2
Mistral
Method
Method
7B
13B
7B
13B
7B
7B
13B
7B
13B
7B
Dense
11.07
17.82
14.59
19.86
40.11
Dense
35.28
46.98
41.97
51.47
58.92
Magnitude
1.52
5.99
2.05
6.22
15.53
Magnitude
26.24
30.12
26.04
43.83
50.83
15.60
8.11
13.42
25.40
SparseGPT
29.48
38.29
33.06
47.14
Wanda
7.96
11.52
7.43
9.10
22.74
Wanda
29.81
37.84
32.09
48.06
53.05
Ours
8.22
15.64
8.47
14.27
25.78
Ours
31.05
39.76
33,08
48.38
53.87
Table 8: Results (%) on 7B LLaVA-1.5.
Table 9: Results (%) on 13B
LLaVA-1.5 1.5.
Vicuna-7B
VQAv2
SQA
VQA
Vicuna-13B
VQAv2
SQA
VQA
Dense
78.50
66.80
58.20
Dense
80.00
74.94
61.30
Magnitudc (50%)
63.50
31.24
38.39
Magnitude (50%)
75.79
70.95
52.16
SparseGPT (50%)
75.86
63.92
53.69
SparseGPT (50%)
78.62
71.19
58.23
Wanda (50%)
75.72
63.99
53.05
Wanda (50%)
78.58
70.97
58.03
Ours (50%)
76.08
65.57
54.36
Ours (50
79.10
73.17
58.70
Wanda (4:8)
12.70
58.92
50.20
Wanda (4:8)
77.57
69.79
56.13
Ours (4:8)
73.54
59.84
51.74
Ours (4:8)
77.79
70.50
56.33
Wanda (2:4)
68.92
55,06
45.42
Wanda (2:4)
75.39
64,89
52.52
Ours (2:4)
71.18
57.44
48.25
Ours (2:4)
76.75
67.13
54.05
7.2
Experiments on Arithmetric & Knowledge Reasoning Tasks
We apply our allocation function to Wanda and evaluate the performance on arithmetic and knowledge reasoning tasks, specifically the GSM8K [8] and MMLU [22] datasets using ILaMA-1 7B/13B, LLaMA-2 7B/13B, and Mistral 7B models [25].
Implementation.
Comparison Results. On GSM8K (Table 6), our method consistently outperforms baselines like magniuuee pruning, SparseGPT and Wanda across all evaluated LLaMA-I, LLaMA-2, and Mistral models. The gains are most notable for smaller models like LLaMA-1 7B. Similarly, on MMLU (Table 7), our DSA achieves the highest accuracy among all methods, outperforming Wanda by up to 1.24% on LLaMA-1 7B and showing consistent improvements across larger LLaMA and Mistral models. The results highlight the effectiveness of our allocation strategy in optimizing sparse patterns across architectures, even challenging reasoning tasks.
7.3 Experiments on Multimodal Tasks
Implementation. To explore the applicability of our method towards a more diverse task, we evaluate our method pruning language models on various visual question-answering and reasoning benchmarks, including VQAv2 [17]. SQA [37], and VQA [47]. In particular, our method is applied with Wanda to LLaVA-1.5 [35], where the Vicuna-7B and Vicuna- language models are pruned. In addition, also transfer some pruning methods and make comparisons these multimodal
tasks.
Comparison Results. Table 8 and Table 9 showcase the performance of different pruning methods such as Magnitude, SparseGPT, and Wanda on the Vicuna-7B and Vicuna- 13B models. For LLaVA- 1.5 with Vicuna-7B, at 50% sparsity, our method surpasses conventional pruning methods across all benchmarks, achieving top scores of 76.08% on VQAv2, 65.57% on SQA, and 54.36% on VQA. Under the 4:8 structured sparsity pattern, our method consistently outperforms Wanda across all metrics, showing improvements 0.84%, 0.92%, and 1. 1.54% on VQAv2, SQA, and VQA respectively for the 7B model. The performance gap becomes even pronounced with 2:4 sparsity, where

---

40.0
Evelutonary Candidate PPL
40.0
Candidate PPL
37.5
37.5
35.0
Perplexity
35.0
32.5
32.5
30.0
30.0
27.5
25.0
27.5
22.5
25.0
500
1000
1500
2000
500
1000
1500
2000
Generations
Generations
Figure 3: Comparison search curves evolution search and random search our sparse allocation function discovery for LLaMA-1 7B on WikiText-2. Evolutionary search converges faster than random search and can achieve potential results with better perplexity (L) performance.
our method achieves substamtial gains of 2.26%, 2.38%, and 2.83% over Wanda on the same metrics. For the 13B model, while the improvements under 4:8 sparsity are modest (0.22%, 0.71%, and O. 0.18%). the 2:4 pattern shows more significant gains of 1.36%, 2.24%, and 1.53% on VQAv2, SQA, and VQA respectively. The consistent superiority of our method across different model sizes and sparsity patterns demonstrates its robustness and effectiveness in maintaining model performance under aggressive compression settings.
7.4 Analysis
Search Algorithm Analysis. Figure 3 compares random search with evolutionary search in the function search tasks. Our advanced evolutionary search has faster convergence and final results, e.g. search algorithm exceeds in 700 generations over 1500 generations of the random algorithm.
Sparse Allocation Results Analysis. Figure illustrates the per-layer importance values and the final sparsity ratios with our allocation functions. These distributions nicely tailored to the specific and explanatory nature of the LLMs.
Conclusion
In this paper, we introduce the DSA framework, which offers a powerful and automated approach to discovering tailored sparsity allocation schemes for LLMs. By leveraging expression discovery and evolutionary algorithms, DSA can effectively explore vast search space of operations and uncover intricate, nonlinear allocation functions that map importance metrics to optimal layer-wise sparsity ratios. This automated process eliminates manual tuning and expert intervention, reducing the time and effort required for effective sparsity allocation. Our DSA demonstrates promising results on the LLaMA, Mistral, and OPT models. We hope the DSA framework and its underlying principles will provide valuable insights to the research community, inspiring new avenues for efficient and effective compression of LLMSs and enabling their wider deployment in resource-constrained environments.
Limitations. Following the AutoMI. technical route [29, 11, 28], we also need some cost in search process. We will develop more efficient search algorithms and incorporate domain knowledge to guide and constrain the search process in future work.
Acknowledgements
The research was supported by Theme-based Research Scheme (T45-205/21-N) from Hong Kong RGC, Hong Kong CRF grants under Grant No. C7004-22G and Generative Research and Development Centre from InnoHK.
10

---

References
[1] Al@Meta. Llama 3 model card. 2024. 7
[2] Yongqi An. Xu Zhao, Tao Yu, Ming Tang. and Jinqiao Wang. Fluctuation baasad adaptive structured pruning for large language models. In AAAI Conference on Artificial Intelligence. 2023. 4
[3] Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint 4xXiv:2401.15024, 2024. 2
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 13:1877-1901, February 2020. 1
[5] Yanqi Chen, Zhengyu Ma Wei Fang, Xiawu Zheng. Zhaofei Yu, and Yonghong Tian. unified framework for soft threshold pruning. In The Eleventh ICLR, 2023. 4, 15
[6] Christopher Clark, Kenton Lee, Ming-War Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 10044, Ptss/1905.1000 2019. 7
[7] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/l 05457, 2018. 7
[8] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchro Nakano, et al. Training verifiers to solve math word problems, 2021. URL https://arxiv. o0g/abs/2110.4440000 2021. 9
[9] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 O: 8-bit matrix multiplication for transformers at scale. NeurlPS, 35:30318 ---3338, 2022.
[10] Peijie Dong, Lujun Li, and Zimian Wei. Diswot: Student architecture search for distillation without training. In CVPR, 2023. 15
[11] Peijie Dong, Lujun Li, Zimian Wei, Xin Niu. Zhiliang Tian, and Hengyue Pan. Emq: Evolving training-free proxies for automated mixed precision quantization. In ICCV, pages 17076-17086, 2023.
[12] Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In ICML, pages 2943-2955 PMLR, 2020. 4, 15
[13] Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter optimization at scale. arXiv preprint arXiv:1807.011774 2018. 2
[14] Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Peng Han, Shuo Shang, Aixin Sun, Yequan Wang, and Zhongyuan Wang. Not all layers Ilms are necessary during inference. ArXiv, abs/2403.02138 2024. 4
[15] Elias Frantar and Dan Alistarh. Massive language models can be accurately pruned in one-shot. In ICML, 2023. 3, 4, 7, 15
[16] Elias Frantar and Dan Alistarh. Sparsegpt: massive language models can be accurately pruned in one-shot. In Proceedings of the 40th ICML, volume abs/2301.00777 of ICML'23. JMLR.org, 2023. 3.
[17] Yash Goyal, Tejas Khot, Douglas Summers-Stap Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017. 9
[18) Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. The unreasonable ineffectiveness of the deeper layers. ArXiv, 140.2/883.8887777 2024.

---

[19] Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning. trained quantization and huffman coding. ICLR, 2016.
[20] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NeurIPS, volume 28, pages
2015. 7
[21] Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. In NeurIPS, 2015. 2
[22] Dan Hendrycks. Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.088 033000 2020.
[23] Zhongzhan Huang, Xinjiang Wang. and Ping Luo. Convolution weitrrioooiiiighh-tssrrbbtonnnnn Rethinking the criteria of channel pruning. CoRR, abs/2004. 11627, 2020. 3
distribution assumption:
[24] Ajay Jaiswal, Zhe Gan, Xianzhi Du, Bowen Zhang, Zhangyang Wang, and Yinfei Yang. Compressing Ilms: The truth is rarely pure and never simple. arXiv preprint arXiv:2310.01382,
2023. 3
[25] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b, arXiv preprint
2023. 9
[26] Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In ICMI., pages 5544-5555, 2020. 4, 15
[27] Lujun Li, Yufan Bao, Peijie Dong, Chuanguang Yang, Anggeng Li, Wenhan Luo, Qifeng Liu, Wei Xue, and Yike Guo. Detkds: Knowledge distillation search for object detectors. In ICML, 2024. 15
[28] Lujun Li, Peijie Dong, Anggeng Li, Zimian Wei, and Ya Yang. Kd-zero: Evolving knowledge distiller for any teacher-student pairs. NeulPS, 2024. 10
[29] Lujun Li, Peijie Dong, Zimian Wei, and Ya Yang. Automated knowledge distillation via monte carlo tree search. In ICCV, 2023. 10
[30] Lujun Li, Haosen Sun, Shiwen Li, Peijie Dong, Wenhan Luo, Wei Xue, Qifeng Liu, and Yike. Guo. Auto-gas: Automated proxy discovery for training-free generative architecture search. In ECCV, 2024. 15
[31] Lujun Li, Zimian Wei, Peijie Dong, Wenhan Luo, Wei Xue, Qifeng Liu, and Yike. Guo. Attnzero: Efficient attention discovery for vision transformers. In ECCV, 2024. 15
[32] Wei Li, Lujun Li, Mark Lee, and Shengjie Sun. Als: Adaptive layer sparsity for large language models via activation correlation assessment. In NeulPS, 2024. 15
[33] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023. 3
[34] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. In ICLR, volume abs/1 888.000095 2019. 2
[35] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023. 9
[36] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and Mykola Pechenizkiy. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. arXiv preprint arXiv:2202.026633, 2022. 9
[37] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurIPS, 2022. 9
12

---

[38] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-pruner: On the structural pruning of large language models. In Thirty-seventh Conference on Neural Information Processing Systems, volume abs/2305.1 1627, 2023. 3
[39] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. 2023. 3
[40] Yuexiao Ma, Taisong Jin, Xiawu Zheng, Yan Wang, Huixia Li, Yongjian Wu, Guannan Jiang, Wei Zhang, and Rongrong Ji. Ompq: Orthogonal mixed precision quantization. In AAAI, volume 37, pages 9029-9037, 2023. 3
[41] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In ICLR, volume abss/600.077 078407 2017. 6, 17
[42] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018. 7
[43] Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 9(1):1-12, 2018.
[44] Dong Peijie, Li Lujun, Tang Zhenheng, Pan Xiang, Liuand Xinglin, Wang Qiang, and Chu Xiaowen. Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. ICML, 2024. 15
[45] Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In ICML, 2018. 2
[46] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021. 7
[47] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, 2019. 9
[48] Lujun Sun, Haosen Li, Peijie Dong, Zimian Wei, and Shitong. Shao. Auto-das: Automated proxy discovery for training-free distillation-awar architecture search. In ECCV, 2024. 15
[49] Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprini arXiv: 11695, 2023. 2,
[50] Sutawika, Schoelkopf, Gao, Abbasi, Biderman, Tow, fattori, Lovering, farzanehnakhaee, Phang, Thite, Fazz, Wang, Muennighoff, Aflah, sutblck, nopperl, gakada, tttyuntian, researcher, Chris, Etxaniz, Lee, Kasner, Khalid, Hsu, Kanekar, Ammanamanchi, Boykis, and AndyZwei. TlluutherA/llm-evauuuttoo---nnne
v0.4.2, March 2024. 7
[51] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13977, 2023. 1,7
[52] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint
2023. 7
[53] Tycho F. A. van der Ouderaa, Markus Nagel, Mart van Baalen, Yuki Markus Asano, and Tijmen Blankevoort. The llm surgeon. ArXiv, abs/2312. 17444, 2023.
[54] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding, In Tal Linzen, Grzegorz Chrupata, and Afra Alishahi, editors, EMNLP, 2018. 7
[55] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. 1
13

---

[56] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of- thain-of-thought prompting elicits reasoning in large language models. NeurlPS, 35:24824- 2022.
[57] Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared Ilama: Accelerating language model pre-training via structured pruning. JCLR, 2024. 3
[58] Peng Xu, Wenqi Shao, Mengzhao Chen, Shitao Tang, Kaipeng Zhang, Peng Gao, Fengwei An, Yu Qiao, and Ping Luo. BESA: Pruning large language models with blockwise parameter- efficient sparsity allocation. In ICLR, 2024. 2,4, 9, 15
[59] Mohit Bansal Yi-Lin Sung, Jachong Yoon. Ecoflap: Efficient coarse-to-line layer-wise pruning for vision-language models. In International Conference on Learning Representations (ICLR), 2024. 15
[60] Lu Yin, You Wu, Zhenyu Zhang, Cheng- Yu Hsieh, Yaqing Wang. Yiling Jia, Mykola Pech- enizkiy, Yi Liang, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning Ilms to high sparsity. arXiv preprint arXiv:2310.05177 2023. 2,
[61] Miao Yin, Burak Uzkent, Yilin Shen, Hongxia Jin, and Bo Yuan. Gohsp: a unified framework of graph and optimization-based heterogeneous structured pruning for vision transformer. In AAAI, 2023. 2
[62] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:190..07330 2019. 7
[63] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0f1 01068, abs/2205.0100 01068, 2022. 1,7
[64] Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. Plug-and-play: An efficient post-training pruning method for large language models. In ICLR, 2024.7
[65] Yuxin Zhang, Mingbao Lin, Fei Chao, Yan Wang, Ke Li, Yunhang Shen, Yongjian Wu, and Rongrong Ji. Lottery jackpots exist in pre-trained models. TPAMI, 2023.
[66] Yuxin Zhang, Mingbao Lin, Zhihang Lin, Yiting Luo, Ke Li, Fei Chao, Yongjian Wu, and Rongrong Ji. Learning best combination for efficient n: M sparsity. In NeurIPS, 2022. 3
[67] Yuxin Zhang, Lirui Zhao, Mingbao Lin, Sun Yunyun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Dynamic sparse no training: Training-free fine-tuning for sparse L.Mss. In ICLR, 2024.7 2
[68] Chendi Zhu, Lujun Li, Yuli Wu, and Zhengxing Sun. Saswot: Real-time semantic segmentation architecture search without training. In AAAI, 2024. 15
[69] Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. In ICLR Workshop (ICLRW), 2017. 9
[70] Zimian Zimian Wei, Lujun Li Li, Peijie Dong, Zheng Hui, Anggeng Li, Menglong Lu, Hengyue Pan, and Dongsheng Li. Auto-prox Training-free vision transformer architecture search via automatic proxy discovery. In AAAI, 2024. 15
[71] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe Le. Learning transferable architectures for scalable image recognition. In CVPR, 2018. 2
14

---

Appendix
A
More Discussion
A.1 Comparison with Existing Methods
Compare to existing AutoML techniques. In contrast to common AutoML [10, 70, 68] and evo- lutionary algorithms [31, 30, 48, 27], our method introduces several groundbreaking innovations specifically tailored for sparsity allocation discovery in LLMs. We are the first to frame LLM sparsity allocation as an AutoML problem, opening new avenues for optimizing LLM efficiency. Our approach introduces a distinctive search space customized for LLM sparsity allocation, combining pre- weeprroessing, reduction, transformation, and post-processing operations in novel ways, allowing for more nuanced and effective sparsity distributions. Diverging from typical AutoML methods such as NAS and HPO that search for specific modes or hyperparameters, our framework emphasizes gener- alized function discovery, identifying common patterns across LLMs and formulating interpretable sparsity allocation. Furthermore, we develop LLM-specific acceleration techniques to reduce search time, making our DSA practical for large-scale LLM optimization.
Compare to ECoFLaP [59]. Our work represents the first automated search for adaptive sparsity methods, which significantly differs from traditional adaptive pruning methods like ECoFLaP. We employ an automated search method that eliminates the need for expert design and adapts strategies to different models and tasks, whereas ECoFLaP relies on hand-designed, hyperparameter tuning. Our comprehensive search space systematically maps element- wise scores to per-layer importances to sparsity ratios, in contrast to ECoFLaP's simple linear computation of keep ratio during its two-stage pruning. Notably, our method obtains significant performance gains across various large language and multimodal models, demonstrating superior performance compared to ECoFLaP.
Compare to OWL. When compared OWL [15], our method revolutionizes the field with an automated search approach that removes the need for expert design, while OWL remains constrained by mend-deiggned hyperparameter tuning and fixed form constraints. Our comprehensive search space systematically maps element-wise scores to sparsity ratios, surpassing OWL's limited linear computation of keep ratio based on outlier ratio. Comparative experiments definitively demonstrate our method's s significant outperformance over OWL, as detailed in Table 8.
Compare to Pruner-Zero Our DSA approach differs fundamentally from Pruner-Zero [44] across multiple dimensions including method type, search space, task, strategy, and Input-Output char- acteristics. We uniquely frame LLM sparsity allocation an AutoML challenge, opening novel avenues for enhancing LLM efficiency. Our search space is specifically customized for L.M sparsity allocation, integrating various operations in innovative ways. Additionally, we develop LLM-specific acceleration techniques like program checking, making our approach practical for large-scale LLM optimization.
Compare to existing ayer-wise sparsity approaches. Our method differs significantly from tra- ditional layer-wise sparsity approaches for neural networks [12, 5, 26, 32]. These methods often lead to the accumulation of errors across layers, as the pruning decisions for each layer are made independently without considering the global impact on the model's performance. Furthermore, the extensive retraining required on vast datasets further amplifies the challenges associated with applying these techniques to LLMss. In contrast our approach, recent work such BESA [58] has shifted focus to intra-block sparsity allocation, employing various techniques to optimize the sparsity distribution within individual blocks or layers. Despite operating at finer granularity, these methods fundamentally adhere to a layer-wise pruning paradigm, neglecting the importance of global sparsity allocation across the entire model. Consequently, the resulting allocation may be locally optimal within each layer but globally suboptimal, potentially leading to solutions that are stuck in local optima and fail to fully leverage the potential benefits of non-uniform sparsity distribution. Our DSA method addresses these limitations by introducing a holistic approach that considers the global impact of sparsity allocation across all layers. Unlike traditional layer-wise pruning methods that operate independently on each layer, our method employs an efficient search process to discover an allocation function that dynamically determines the appropriate sparsity level for each layer based on its characteristics and contribution to the overall model performance.
15

---

Table 10: Comparison Method Characteristics of our DSA and Pruner Zero.
Method
Types
Task
Search space
Input
Output
Stralegy
Pruner-Zero
unitorm
symbolic Pruning
unary/binary
element-wise
element-wise
Symbolic
Metric
operations
weights/gradient
score
Regression
DSA (ours)
non con-uniform
adaptive allocation
pee preces/eeeddccion
element wise
layer wise
Evolutionary
function
transforn/post-pen process
sparsity ratios
Algorithm
B Theoretical Understanding of Discovered Allocation Function
To show that the function V VII. In (In(S)) is stable under small perturbations in the input S, can derive expression for the difference VIS AS) V(S) and analyze its behavior for small AS. We define the following intermediate functions:
g(S) In(s)
h,(S) = In(g(S)) = In(In(S))
(S)) = |14(S) = [In(In(S))
1(S) = II (S)
V(S) = V((S)= = In (In(S))
Using the chain rule for differentiation, we can derive the derivative of V(S) with respect to as follows:
dv
dl
Vi(S)
dS
dS
dk:
((S) 1-
Σ
(S)
dS
jfi
Σ sgu h/s (S)) dh;
((S)1-n
II (S)
n
dS
jfi
((S)I-n
Σ sgn(In(In(S))
1
II (S)
n
In(S)
where (ln())))n(S)) the sign function, equal for In(In(S)) and for In(In(8))
Now, we can use Taylor's theorem to approximate V(S + AS) around S as:
dv
V(S + +A) v V(S)+ + (8)AS (S)AS +((AS)3))
ds
=V(S+ In
Σ
1 II [5)(5) sgn(In(InSS)) In(S) s j
AS + A8+0((A))))
Therefore, the difference V(S + AS) - V(S) is approximately:
VSS + AS) - V(S) ~ 1 1(S)1-n Σ
(gn(n((n(S))
In(S) S (S)
AS AS+0(AA5))
j
16

---

Algorithm Evolutionary Search for Allocation Function Discovery
Input: Search space S, population size P, max iterations T, sample ratio top-k k. Output: Best allocation function A"".
1: Initialize population PO with P random allocation functions; do
2: for 1, ........
Sample pool R Pi with ratio r:
4: Top candidates & = GetTopk(R. k);
5: Parent G. == RandomSelect(Cis))
6: Mutant G MUTATF(G"):
Crossover G CROSSOVER(G?, RandomSelect(Gu: G,));;
Pi Pi U G Gū;
9:
P. = Sececc-ToppPerfformeee
P);
10:
end for
11: A argmaxA Prm. (Pm.(A(S).)))
For small AS, the second-order term ((AS)) becomes negligible, and the leading term is linear in AS. The coefficient of AS in this leading term is product of bounded functions of S, namely: V(8)1-n, which is bounded for 8 > 0 sgn(In(In(S)) which is either 1 or In())> which is bounded for S > S. which is bounded for S > 0 II (S), which is bounded for finite S
Therefore, for small perturbations AS around any positive value of S, the difference V(S + AS) V(S) is also small, and the function V(S) is stable under such perturbations.
This stability property is desirable for the function V(S), as ensures that small fluctuations or measurement errors in the input importance values S do not significantly impact the computed result V, leading to robust and consistent computations.
C More Experiment Details
Table 11: WikiTexx-2 perplexity (L) performance of our searched allocation function with different
initial seeds with the Wanda metric for 70% sparse LLaMA-I-8B.
Seed
Searched Allocation Functions
perplexity
Seed-1
ABSLOG- DASLOG-VAR-OO VAR-ATAN,ASS
24.52
Seed-2
ABSLOG.NO CO--VAR-TTE AAR-ATAN,AAA ACOS
23.69
Seed-3
LOG.ABSLOG-GEOMETRE WEWEAEGEEEEEEEEEEEEEEEEEEEECEEEE
MEAN-ACOS SIGMOID
22.60
Seed-4
LOG.ABSLOG-GEOMETRII COCAABBLLCGGOMMEETTCCMCAAACOSSSS MEAN-COS.EXP
22.60
Seed-5
MMSLABSLOG VAR-ATAN.ASIN
24.61
C.1 More details about Evolutionary Search
We commence our experiments by searching for an optimal allocation function based on the Wanda pruning method, utilizing perplexity results from the validation set of the LLaMA-1-7B model the WikiText2 dataset [41]. We ensure that no search is performed on the test set, maintaining a fair and unbiased comparison. Subsequently, to different tasks and scenarios without conducting additional searches, aiming to evaluate its generalizability. To establish a robust and reliable experimental setup. we allocate 20% of the original dataset's training set as held-out test set for the search process. We meticulously confirm that these validation datasets do overlap with the set, preventing any potential data leakage bias in our evaluations. During the search phase, we configure the evolutionary algorithm (Algorithm 1) with a population size of 20, a maximum of 1,000 iterations, a sample ratio of 0.9, and top-k value Throughout this process, we evaluate total of 50 allocation function candidates within the validation set, iteratively refining and improving
directly transfer this discovered allocation function
solutions through the evolutionary mechanisms crossover and mutation. By adhering to this rigorous experimental protocol, we ensure the integrity and validity of our results, enabling a comprehensive assessment the discovered allocation function's effectiveness and its ability generalize across diverse tasks and datasets.
17

---

C.2 Analysis of Search Robustness
Our evolutionary search algorithm shows robustness different initialization seeds both theoretically and experimentally. Theoretically, it maintains robustness by: (1) Starting with a diverse initial popu- lation of allocation functions avoid getting trapped in poor solutions. (2) Using genetic operators like mutation and crossover to explore new regions beyond the initial population. Experimentally, we evaluated the algorithm across 5 different random initialization seeds when searching for an allocation function to sparsify LLaMA-1-8B on WikiText-2 to 70% sparsity using the Wanda metric. The results in Table show: (1) Different seeds discovered different allocation function expressions involving various operations. Despite this variation, all seeds achieved competitive perplexity performance in the range of 22.60 24.61. (2) Two seeds (3 and 4) converged to the same best perplexity of 22.60 despite different initial conditions. This consistent performance across diverse initializations demonstrates the robustness of the search aided by the diverse initial population and exploration via genetic operators. While minor performance variations exist, the overall competitiveness of the results validates the algorithm's 's resilience against initialization biases through effective search space exploration.
D Search Space for Allocation Functions
The search space for allocation functions is organized into four main categories: (1) pre-process operations Te (2) reduction operations Trduuss (3) transform operations Tuans and (4) post-process operations Tposs. The following subsections provide a detailed list the operations included each category, along with their corresponding mathematical formulas.
D.1 Pre-process Operations Tpre
The pre-process operations Te are applied to the element-wise scores S to prepare them for the subsequent reduction step. The operations in this category include:
NO_OP: No operation performed, and the input is returned is.
NO_OP(z) =
AB: Element-wise absolute value operation.
ABS(c)) = |c]|
LOG: Element-wise natural logarithm operation.
LOG(x) In(x:)
ABSLOG: Element-wise absolute value of the natural logarithm operation.
ABSLOG = |n(())
POW: Element-wise power operation with constant exponent.
POW(x, c) xf
EXP: Element-wise exponential operation.
EXP(x) =
NORMALIZE: Normalization operation that scales the input to have mean of and standard deviation of 1.
2
NORMALIZE
where μ is the mean of x, and a is the standard deviation x.
SIGMOID: Flement-wise sigmoid function.
SIGMOID(z) 1
TANH:
Element-wise hyperbolic tangent function.
6t - TANH(=) e + -
18

---

D.2 Reduction Operations Treduce
The reduction operations Treduce aggregate the pre-processed scores into single per-layer importance score V. These operations use statistical measures and matrix operations to provide insights into the distribution and characteristics of the input data. The operations in this category include:
NO_OPt No operation is performed, and the input is returned as is.
NO_OP(a) = æ
GRAM: Gram matrix operation, which computes the matrix multiplication of the input with its transpose.
GRAM(X) x
Correlation coefficient operation, which measures the linear relationship between the elements of the input.
CORREF:
x.
CORREF(2)
|
DIAGONAL: Diagonal operation, which extracts the diagonal elements of the input matrix.
DIAGONAL(X)- diag(X)
FROBENIUS NORM:
Frobenius operation, which computes the square root of the sum the squared elements of the input matrix.
FROBENIUS {
x
1.
L1 NORM:
LI norm operation, which computes the sum of the absolute values of the elements the input.
L1 L1_NORM(2)
DETERMINANT: Determinant operation, which computes the determinant of the input matrix.
DETERMINANT(X) = det(X)
RANK: Rank operation, which computes the rank of the input matrix.
RANK(X) = rank(X)
GEOMETRIC MEAN: Geometric mean operation, which computes the nth root of the product of the elements in the input.
T2
CCOMETRIC_MEANT
x2
MEAN: Mean operation, which computes the arithmetic mean of the elements in the input.
1 Σ
MEAN( (x)
VAR: Variance operation, which computes the variance of the elements in the input.
VAR(x) = Σ (x. 14)
where is the mean of z.
19

---

D.3 Transform Operations Trans and Post-process Operations Tpost
The transform operations Tuans map the per-layer importance scores V/ suitable range for sparsity ratios, while the prsteproess operations Tpost ensure that the sparsity ratios ø satisfy any required constraints. The operations in these categories include:
NO OP: No operation performed, and the input is returned is.
NO_OP(z)
SIN: Sine functiin.
SIN(x) sin(x)
COS: Cosine function.
COS(x:) = cos(xr) (x)
TAN: Tangent function.
TAN(2) = tan (z)
ASIN: Inverse sine (aresin) function.
ASIN(r) sin (x)
ACOS: Inverse cosine (arccos) function.
ACOS(x)) cos (1)
ATAN: Inverse tangent (arctan) function.
ATAN(2 = tan (x)
EXP: Exponential function.
EXP(2) =
LOG: Natural logarithm function.
LOG(z) = In(r)
ABS: Absolute value funetion.
ABS(c)) = |x:
SIGMOID: Sigmoid function.
SIGMOID(z) 1
TANH: Hyperbolic tangent function.
TANH(r) 6 -
e + -
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
No Theory Assumptions and Proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification:
See supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https: httpps:///iipp./eeeeeeee cc/
publ / guides
des CodeSubom ss ssionns sssioooll liyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss. cc/ cttppcccppccccccccc pubppcplco.ooo wwwp//uuissseeeeeeeeeeeeeeeeesee Code Submi ssissiionnn ceiionnoliinn oieliyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:[Yes
Justification:
See Experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
23

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /neurips..... hc/ppublccc/tthhcccclleeee.ee
Answer: [No]
Justification: NO.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification:
Only technical reports.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [No]
Justification: No.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paper wwi hhoode com/ dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
25

---

Answer: [No]
Justification: No.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [No]
Justification: NO
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NO.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "Doa3F7Jo5g", "reviewer": "Reviewer_1f5c", "summary": "This paper introduce a novel model pruning algorithm, DSA, aiming to prune unimportant model weights to increase the sparsity of models. Unlike previous model pruning methods which assign the same sparsity ratio for all layers. DSA proposes a method to calculate the sparsity allocation for different layers, which achieves a more adaptive pruning for each layer. The evaluation shows that DSA has strong empirical performance.", "questions": "See weakness.", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The insight of the paper is clear and easy to understand.\n2. The paper proposes a novel method to estimate the allocation budget for different layers.\n3. The proposed method shows better performance than SOTA pruning methods.", "weaknesses": "1. The presentation of the proposed method is quite confusing. Especially the section 4, I am not quite clear about why the allocation function uses this design. Why do we need the pre-process, reduction, transformation, and post-process? What are the insights for these components?\n2. Performance improvement is not as promising as the abstract claims, especially in structured pruning cases. Structured pruning is a more meaningful setting for model pruning, since it can be directly accelerated by hardware to achieve wall-clock time speedup. In Tables 6 and 7, the performance improvement is around 1%. \n3. Wanda paper also reported 2:4 sparsity. How is the performance of DSA in a 2:4 sparsity setting?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer 1f5c,**\n\nThanks for the valuable feedback. We have tried our best to address all concerns in the last few days. If the reviewer finds our response adequate, **we would appreciate it if the reviewer considers raising the score**. Please see our responses below one by one：\n\n------\n\n> **Q1: About motivation and insights of allocation function design.**\n\n**A1: Our responses** are: \n\n**(1)** Our allocation function design is **motivated by analyzing element-wise score distributions**, as discussed in the **Introduction [Lines 62-78] and Figure 1(left).**  \n\n**(a)** We notice that mean, variance, and entropy values of per-layer element-wise scores can serve as allocation indicators, inspiring reduction operations.  \n\n**(b)** While basic reduction of element-wise scores showed modest improvements, applying transform functions yielded more promising results, prompting the introduction of transform operations. \n\n**(c)** We include pre-process to normalize scores for fair comparison and post-process to further enhance function fit's upper bound.\n\n **(2)**  The key insight of our four-component design is its flexibility in exploring diverse allocation functions tailored to each LLM's characteristics, while capturing complex, non-linear relationships between element-wise scores and sparsity ratios. **As detailed in the Primary Operators section [Lines 184-197]**, each component serves a specific purpose: \n\n **(a) Pre-process:** Standardizes inputs by normalizing scores across layers, ensuring consistent performance metrics by addressing scale variations. \n\n **(b) Reduction:** Condenses element-wise information by extracting representative values per layer through operations like variance or entropy, reducing computational complexity. \n\n **(c) Transformation**: Models complex relationships with functions like sine or exponential, enabling the representation of intricate patterns in layer importance. \n\n **(d) Post-process:** Fine-tunes the allocation function for optimization, enhancing flexibility.\n\n **(3)** Ablation study  **(See Table below)**  shows that the Reduction component is the most influential, followed by the Transformation component, while the Pre-processing and Post-processing components have a smaller impact on the overall DSA performance.\n\n*Table : Perplexity  with LLaMA 7B at 0.7sparsity  on WikiText2*\n\n| DSA   | DSA without Pre-process | DSA without Reduction | DSA without Transformation | DSA without Post-process |\n| ----- | ----------------------- | --------------------- | -------------------------- | ------------------------ |\n| 22.60 | 23.45                   | 26.55                 | 25.61                      | 23.22                    |\n\n------\n\n> **Q2: About performance improvements.**\n\n**A1: Our responses** are: \n\n **(1)** We would like to clarify that our method consistently demonstrates substantial enhancements **(1%-14%) across a spectrum of scales (7B~70B), models (LLaMA-1|2|3, Mistral, LLaVA, and OPT models), and complex tasks (reasoning and multimodal benchmarks)**,  which robustly confirms its effectiveness.  The magnitude of gains naturally varies depending on the model and task. For instance, in Tables 6 & 7, considering the **Dense LLaVA-1.5 Models ranging from 7B to 13B, we observe a only 1.5%** performance increase on VQAv2. Notably, our around  1% gains over Wanda, the current state-of-the-art pruning method, is  promising advancements in this area **(Recognized by Reviewer of38, s93u, gYqu, and VeXf)**.  \n\n**(2)** To ensure fair comparisons, we follow standard settings like 50% sparsity, where baseline results are strong, making additional gains challenging.  \n\n**(3)** Our experiments in Table 8 at higher sparsity levels (65% ~ 80%) on LLaMA-1 show our method's ability to achieve substantial improvements.  \n\n**(4)** Additional results in the rebuttal PDF **(See our general response)** further demonstrate that our method can achieve substantial gains, with improvements ranging from **1% ~ 7% at 60% sparsity, 2% ~ 7.6% at 70% sparsity, and 1.4% ~ 4.2% at 2:4 sparsity**.\n\n> **Q3: About performance in 2:4 sparsity setting.**\n\n**A3**: Following the suggestions, we evaluate our DSA under 2:4 sparsity on LLaVA (See Table below) and LLaMA-2  (See our general response). These consistent gains **(2.2% ~ 2.8% on LLaVA and 1.4% ~ 4.2% on LLaMA-2)**  across different model sizes and datasets suggest that our approach is more effective at maintaining performance under tighter sparsity constraints.\n\n| 7B LLaVA-1.5 | VQAv2 | SQA   | VQA   | 13B LLaVA-1.5 | VQAv2 | SQA   | VQA   |\n| ------------ | ----- | ----- | ----- | ------------- | ----- | ----- | ----- |\n| Dense        | 78.50 | 66.80 | 58.20 | Dense         | 80.00 | 74.94 | 61.30 |\n| Wanda (2:4)  | 68.92 | 55.06 | 45.42 | Wanda (2:4)   | 75.39 | 64.89 | 52.52 |\n| **Ours (2:4)**   | **71.18** | **57.44** | **48.25**| **Ours (2:4)**    | **76.75** | **67.13** | **54.05** |\n\n**Finally,** we hope our response could address the concerns, and we thank the reviewer again for the helpful comments. We are glad to discuss further comments and suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rXjpFkLp2v", "reviewer": "Reviewer_of38", "summary": "This paper introduces DSA, an automated framework for discovering optimal sparsity allocation schemes for layer-wise pruning in LLMs. The proposed framework uses per-layer importance statistics and an evolutionary algorithm to explore effective allocation functions, which are then integrated into various pruning methods. Extensive experiments on challenging tasks demonstrate significant performance gains for models like LLaMA-1|2|3, Mistral, and OPT, achieving notable improvements over state-of-the-art models.", "questions": "1. Are the top-performing allocation functions the same across pruning settings, e.g., sparsity ratio and models? More specifically, does the allocation function in equation 6 generalize across LLMs?  \n\n2. How does the proposed sparsity allocation method apply to neural networks beyond Transformers, e.g., convolutional neural networks?  \n\n3. The authors evaluated all the open-source LLMs in LLaMA and LLaMA-2. However, for LLaMA-3, only the 8b model is evaluated. Have the authors experimented with LLaMA-3-70B?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors evaluated on a wide range of models, ranging from representative LLMs like LLaMA-1/2/3, Mistral, OPT to a multi-modal model LLaVA.  \n\n2. This paper tackles an important problem as to how to assign layerwise pruning ratio for sparsity based pruning. The authors have demonstrated strong results compared to state-of-the-art methods.", "weaknesses": "1. A majority of the results in this paper are performed under the setting of 50% unstructured sparsity, which is a relatively low sparsity level. It would be good to demonstrate more results on higher level of sparsity, e.g., 60% and 70%.\n\n2. The authors described the search space of the proposed algorithm as four transform operations in Equation 3. However, I am not sure if this is the best way to find layerwise sparsity. Is there any motivation for such design of the search space? More specifically, why should we convert the element-wise scores to layer-wise scores first? I think some space should be used to discuss the motivation and insights behind Equation 3.\n\n3. It would be good to show some numbers on the practical runtime speedup of dynamic layerwise pruning as compared to layerwise uniform pruning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer of38,**\n\nThanks for constructive comments.  We have tried our best to address all concerns in the last few days. If the reviewer finds our response adequate, **we would appreciate it if the reviewer considers raising the score**. Please see our responses below one by one：\n\n\n\n------\n\n> **Q1: More results on higher sparsity levels.**\n\n**A1: Our responses** are: \n\n**(1)** To ensure fair comparisons, we follow standard settings like 50% sparsity, where baseline results are strong, making additional gains challenging. \n\n **(2)** Our experiments in **Table 8 at higher sparsity levels (65%-80%) on LLaMA-1** show our method's ability to achieve substantial improvements **(Recognized by Reviewer s93u, gYqu, and VeXf)**.  \n\n**(3)** Following the suggestions, we provide 60% and 70% sparse experiments in the rebuttal PDF **(See our general response).**  Our method demonstrates gains ranging from **1.24% to 7.03% at 60% sparsity and 1.91% to 7.68% at 70% sparsity**, which underscore our effectiveness for high sparsity ratios.\n\n-----\n\n> **Q2: About motivation and insights of design of the search space**.\n\n**A2: Our responses** are:  \n\n**(1)** As discussed in **Introduction [Lines 62-78] and Figure 1 (left),** our design is motivated by analyzing element-wise score distributions, and mean values of per-layer scores, inspiring reduction and transformation ops. While basic reduction showed modest gains, applying transform ops yielded more promising results. We also include pre-processing for score normalization and post-processing to enhance function fit.\n\n **(2) Converting element-wise scores to layer-wise scores is important:** layer-wise scores provide a consolidated view of importance, reducing noise and offering more stable ratios. By focusing on layer-wise metrics, critical layer information can be leveraged for better parameter retention. This process enhances computational efficiency by simplifying complexity from the element to the layer level.\n\n **(3)**  **As detailed in Primary Operators section [Lines 184-197]**, key insights of our design are:  \n\n**(a) Pre-process:** Standardizes inputs by normalizing scores across layers, ensuring consistent performance metrics by addressing scale variations. \n\n **(b) Reduction:** Condenses element-wise information by extracting representative values per layer, reducing computational complexity. \n\n **(c) Transformation**: Models complex relationships with functions, enabling the representation of intricate patterns in layer importance.  \n\n**(d) Post-process:** Fine-tunes the allocation function for optimization, enhancing flexibility. \n\n **(4)**  Ablation study  **(See Table below)**  shows that the Reduction ops is the most influential, followed by the Transformation ops.\n\n*Table : Perplexity with LLaMA 7B at 0.7 sparsity on WikiText2*\n\n| DSA   | DSA without Pre-process | DSA without Reduction | DSA without Transformation | DSA without Post-process |\n| ----- | ----------------------- | --------------------- | -------------------------- | ------------------------ |\n| 22.60 | 23.45                   | 26.55                 | 25.61                      | 23.22                    |\n\n\n\n> **Q3: About practical runtime speedup.**\n\n**A3:** Following the suggestions, we evaluate the runtime speedup   **(See Table below)**  of employing DeepSparse inference engine on LLaMA-V2-7B-chat-hf model. The findings indicate that our method exhibits comparable speedup to uniform pruning.\n\n| Method   | Sparsity               | Dense    | 40%      | 50%      | 60%      | 70%      | 80%      |\n| -------- | ---------------------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| uniform  | Latency (ms)            | 213.8    | 112.5    | 89.1     | 85.5     | 82.2     | 61.1     |\n| uniform  | Throughput (tokens/sec) | 4.7      | 8.9      | 11.2     | 11.7     | 12.2     | 16.4     |\n| uniform  | Speedup                | 1.0x     | 1.9x     | 2.4x     | 2.5x     | 2.6x     | 3.5x     |\n| ours     | Latency (ms)            | 213.8    | 121.4    | 90.8     | 86.0     | 78.3     | 58.6     |\n| ours     | Throughput (tokens/sec) | 4.7      | 8.2      | 11.0     | 11.6     | 12.8     | 17.1     |\n| **ours** | **Speedup**            | **1.0x** | **1.8x** | **2.4x** | **2.5x** | **2.7x** | **3.7x** |\n\n> **Q4: About generalization of allocation functions.**\n\n**A4: Yes.**  We directly transfer top-performing allocation function in Equation 6 to different models and takes without additional search. Our experiments show this allocation functions do show good generalization across models and sparsity settings.\n\n> **Q5: About applicability to non-Transformer networks.**\n\n**A5:** We would like to highlight that **our method is architecture-agnostic and can be applied to various model types, including CNNs. The reason is that our search space is comprehensive and includes diverse types of operations that can be utilized across different model architectures.** To confirm this, we apply our allocation functions to ConvNeXt in Table below.  Our DSA surpasses other methods, especially at higher sparsity levels, showing its effectiveness with CNNs.\n\n*Table: Accuracy (%) of Sparse ConvNeXt-Base on ImageNet-1K.*\n\n| Sparsity        | 50%       | 60%       | 70%       |\n| --------------- | --------- | --------- | --------- |\n| Wanda           | 82.72     | 80.55     | 68.18     |\n| OWL + Wanda     | 82.76     | 80.53     | 68.28     |\n| **DSA + Wanda** | **83.12** | **81.68** | **71.55** |\n\n> **Q6: About experiment with on LLaMA-3 70B.**\n\n**A6:** Following the suggestion, we perform experiments on LLaMA-3 70B in the rebuttal PDF **(See our general response)**. Our method shows **2.23% to 4.38% gains** across different pruning approaches, aligning with our effectiveness on multiple LLMs. \n\n\n\n**Finally,** we hope our response could address the concerns, and we thank the reviewer again for the helpful comments. We are glad to discuss further comments and suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3UQtADjEb6", "reviewer": "Reviewer_s93u", "summary": "This paper presents DSA, which models layer importance to sparsity ratios, and integrates the allocation function discovered by evolutionary algorithms into various methods, resulting in significant performance improvements.", "questions": "Please see weaknesses", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This manuscript is a qualified paper, i.e,\nThe method seems technically sound and straightforward in principle.\nEmpirical results demonstrate the strength of this approach.", "weaknesses": "All the experiments in this article are done on unstructured sparsity. It is well known that unstructured sparsity will not bring the effect of computation acceleration. Have the authors ever tried the experimental results on structured sparsity? \n\nIt is well known that search algorithms are often very time-consuming, so the 0.5day mentioned by L227 is the result of experiments conducted on which model, 7B, 13B and 70B?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer s93u,**\n\nThanks for constructive comments.  We have tried our best to address all concerns in the last few days. If the reviewer finds our response adequate, **we would appreciate it if the reviewer considers raising the score**. Please see our responses below one by one：\n\n\n\n------\n\n> **Q1: About results on structured sparsity.**\n\n**A1: Our responses** are: **(1)** Following the suggestion, we apply our non-uniform layer-wise sparsity allocation method to the structured pruning technique LLM Pruner [32], which accelerates pruned LLMs directly. The results in Table below show that **our  method performs well in structured pruning scenarios** and outperforms OWL.\n\n*Table: Perplexity of Structure Pruning with LLaMA-7B on WikiText-2.*\n\n| Pruning Method | Layerwise Sparsity | 20%       | 40%       | 60%       | 80%        |\n| -------------- | ------------------ | --------- | --------- | --------- | ---------- |\n| LLM Pruner     | Uniform            | 19.09     | 30.39     | 90.017    | 1228.17    |\n| LLM Pruner     | OWL                | 18.57     | 28.65     | 76.99     | 321.64     |\n| **LLM Pruner** | **DSA**            | **17.85** | **26.98** | **68.82** | **202.42** |\n\n\n\n**(2)** We would like to highlight that our method **supports and enhances 2:4 and 4:8 structured sparsity**, compatible with Nvidia GPUs for hardware acceleration. The results presented in Tables 6 and 7, as well as the results in the rebuttal PDF **(See our general response)**, demonstrate the efficacy of our approach, showcasing **gains of 1.4% ~ 4.2% at 2:4 sparsity and ~1% gains at 4:8 sparsity.** These gains are promising advancements in this area **(Recognized by Reviewer 1f5c, of38, gYqu, and VeXf)**.\n\n**(3)** Recent advancements in advanced GPU kernels like NVIDIA cuSPARSE [ref1], Sputnik [ref2], and Flash-LLM [ref3] have rapidly supported unstructured sparsity. This practical relevance extends beyond GPUs to non-GPU hardware such as CPUs (e.g., XNNPACK [ref4]) and specialized accelerators like FPGA accelerator. Our method under unstructured sparsity also **achieves 1.8x~3.7x speed up  with  DeepSparse inference engine** for LLaMA-V2-7B-chat-hf, as presented in Table below. \n\n| Method   | Unstructured Sparsity   | Dense    | 40%      | 50%      | 60%      | 70%      | 80%      |\n| -------- | ----------------------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| ours     | Latency (ms)            | 213.8    | 121.4    | 90.8     | 86.0     | 78.3     | 58.6     |\n| ours     | Throughput (tokens/sec) | 4.7      | 8.2      | 11.0     | 11.6     | 12.8     | 17.1     |\n| **ours** | **Speedup**             | **1.0x** | **1.8x** | **2.4x** | **2.5x** | **2.7x** | **3.7x** |\n\n\n\n------\n\n> **Q2: Details of  the time cost .**\n\n**A2:** As **detailed in the experiment [Line 252-258],**  the 0.5-day search time mentioned is for the **LLaMA-1 7B model** on WikiText2 based on Wanda at 50% sparse setting.  **For other different LLM models, we directly transfer this discovered allocation function without additional search.** We apologize for not explicitly stating this and will clarify this detail in the revision.\n\n\n\n> **References:**\n>\n> [ref1] NVIDIA GPUs scalability to solve multiple (batch) tridiagonal systems implementation of cuThomasBatch. In PPAM 2018.\n>\n> [ref2] Sparse gpu kernels for deep learning. In SC conference 2020.\n>\n> [ref3] Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity. ArXiv:2309.10285.\n>\n> [ref4] Fast sparse convnets. In CVPR2020.\n\n\n\n**Finally,** we hope our response could address the concerns, and we thank the reviewer again for the helpful comments. We are glad to discuss further comments and suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rWyKVEWR2u", "reviewer": "Reviewer_gYqu", "summary": "This article introduces DSA (Discovering Sparsity Allocation) which is designed to automate the discovery of sparsity allocation schemes for layer-wise post-training pruning in large language models (LLMs).", "questions": "1. This article only uses DSA to determine the sparsity ratio of each layer, while Pruner-zero searches for a pruning metric to prune the network. I am curious about the potential performance of DSA combined with Pruner-zero.\n\n2. Can DSA be integrated with the 2:4 constraint to achieve actual acceleration?\n\n3. The article mentions in line 227 that DSA can find potential allocation functions in only 0.5 days. What model was searched in this 0.5-day period, and how large was it?", "limitations": "The authors have addressed the limitations, but I believe they should also mention that current techniques do not support the inference of unstructured sparsity, which prevents them from achieving actual acceleration. While some companies are working on this, their devices are not yet publicly available.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents a framework for automatically discovering effective sparsity allocation functions.\n\n2. This paper demonstrates consistent performance improvements across various tasks and datasets.", "weaknesses": "1. This article seems to combine OWL and Pruner-Zero. It uses the evolutionary algorithm to determine each layer's sparsity ratio. It's not quite innovative for me.\n\n2. This article doesn't offer adequate evidence when the sparsity is higher than 50%. Table 12 shows the higher sparsity results on LLaMA1 7b. However, I think more evidence should be provided.  \n\n\n**Minor:**\n\nL632:Typo of Erdos-Renyi.\n\nThe caption of Table 8: There is no 8B model in the LLaMA-1 family.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer gYq,**\n\nThanks for the valuable feedback. We have tried our best to address all concerns in the last few days. If the reviewer finds our response adequate, **we would appreciate it if the reviewer considers raising the score**. Please see our responses below one by one:\n\n------\n\n> **Q1: Compare to OWL and evolution method.**\n\n**A1:** Our method's novelty lies in being the **first automated search for adaptive sparsity allocation**, significantly differing from traditional adaptive pruning methods like OWL [14] and evolution methods like Pruner-Zero [37]:\n\n**(1) Compare to OWL:** \n\n**(a)** Our method revolutionizes the field with an automated search approach that removes the need for expert design.  In contrast, OWL relies on hand-designed, hyperparameter tuning and is limited by fixed form constraints. \n\n**(b)** Our comprehensive search space systematically maps element-wise scores to sparsity ratios, whereas OWL only computes the keep ratio based on outlier ratio linearly, lacking our introduced nonlinear mapping operators. \n\n**(c)** Comparative experiments show our method's significant outperformance over OWL in Table 8.\n\n**(2) Compare to evolution method:  Our DSA differs Pruner-Zero [37] in method type, search space, task, strategy, and Input-Output characteristics (See Table below).** \n\n**(a)** We uniquely frame LLM sparsity allocation as an AutoML challenge, opening novel avenues for enhancing LLM efficiency.\n\n **(b)** Our search space is customized for LLM sparsity allocation, integrating various operations in innovative ways. \n\n**(c)** We develop LLM-specific acceleration techniques like program checking, making our approach practical for large-scale LLM optimization.\n\n| Method         | Types           | Task                             | Search space                                                | Input                          | Output                         | Strategy                   |\n| -------------- | --------------- | -------------------------------- | ----------------------------------------------------------- | ------------------------------ | ------------------------------ | -------------------------- |\n| Pruner-Zero    | uniform         | symbolic Pruning Metric          | unary/binary operations                                     | element-wise weights/gradients | element-wise score             | symbolic Regression        |\n| **DSA (ours)** | **non-uniform** | **adaptive allocation function** | **pre-process/reduction/transform/post-process operations** | **element-wise score**         | **layer-wise sparsity ratios** | **evolutionary Algorithm** |\n\nPlease note that the title and abstract of Pruner-Zero [37] are accessible in May 2024, **but its full paper is released (on arXiv and OpenReview) in June 2024, after the NeurIPS deadline.** We already discuss OWL and Pruner-Zero in the related work [lines 143-148] and will add more comparisons in the revision.\n\n> **Q2: More results for high sparsity ratios.**\n\n**A2: Our responses** are: \n\n**(1)** To ensure fair comparisons, we follow standard settings like 50% sparsity, where baseline results are strong, making additional gains challenging.  \n\n**(2)** Our experiments in Table 8 at higher **sparsity levels (65%-80%)** on LLaMA-1 show our method's ability to achieve substantial improvements. \n\n **(3)** Following the suggestions, we provide 60% and 70% sparse experiments in the rebuttal PDF **(See our general response)**. Our method demonstrates **gains ranging from 1.24% to 7.03% in LLaMA-2-7B&13B, and LLaMA-3-70B at 60% sparsity. At 70% sparsity, our improvements ranging from 1.91% to 7.68%** underscore the effectiveness of our DSA for high sparsity ratios.\n\n> **Q3: About typos.**\n\n**A3:** We appreciate the reviewer's corrections of typos in Erdos-Renyi and LLaMA-1-7B and commit to fixing them in the revision.\n\n> **Q4: About potential combination with Pruner-Zero.**\n\n**A4: Yes.** Following the suggestion, we conduct experiments combining our method with Pruner-Zero, yielding new state-of-the-art results. This successful integration is due to the orthogonal nature of the two methods: Pruner-Zero optimizes element-wise importance scoring, while our DSA specializes in adaptive layer-wise sparsity allocation.\n\n*Table: Mean accuracies (%) at 0.5 sparsity on 7 zero-shot task.*\n\n| Models      | Pruner-zero | **Pruner-zero+ DSA (Ours)** |\n| ----------- | ----------- | ----------------------- |\n| LLaMA-2-7B  | 58.87       | **62.22**                   |\n| LLaMA-2-13B | 64.83       | **67.05**                  |\n\n> **Q4: About integration with 2:4 constraint.** \n\n**A4:**  Following the suggestions, we evaluate our DSA under 2:4 sparsity on **LLaVA (See Table below) and LLaMA-2  (See our general response)**. These consistent gains **(2.2%  ~ 2.8% on LLaVA and 1.4% ~ 4.2% on LLaMA-2)**  across different model sizes and datasets suggest that our approach is more effective at maintaining performance under tighter sparsity constraints.\n\n| 7B LLaVA-1.5   | VQAv2     | SQA       | VQA       | 13B LLaVA-1.5  | VQAv2     | SQA       | VQA       |\n| -------------- | --------- | --------- | --------- | -------------- | --------- | --------- | --------- |\n| Dense          | 78.50     | 66.80     | 58.20     | Dense          | 80.00     | 74.94     | 61.30     |\n| Wanda (2:4)    | 68.92     | 55.06     | 45.42     | Wanda (2:4)    | 75.39     | 64.89     | 52.52     |\n| **Ours (2:4)** | **71.18** | **57.44** | **48.25** | **Ours (2:4)** | **76.75** | **67.13** | **54.05** |\n\n\n\n------\n\n> **Q5: Details of  the time cost .**\n\n**A5:** As **detailed in the experiment [Line 252-258]**, the 0.5-day search time mentioned is for the **LLaMA-1 7B model** on WikiText2 based on Wanda at 50% sparse setting. We will clarify this detail in the revision.\n\n\n------\n\n\n**Finally,** we hope our response could address the concerns, and we thank the reviewer again for the helpful comments. We are glad to discuss further comments and suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wu2oFlKscO", "reviewer": "Reviewer_VeXf", "summary": "This paper introduces DSA, an automated framework for determining layer-wise sparsity in large language models (LLMs). The approach aims to enhance pruning techniques by using an evolutionary algorithm to discover optimal sparsity allocation functions, thereby improving model performance on various tasks. The proposed method can improve the performance of layer-wise pruning baselines on various datasets.", "questions": "**Q1.**\nCan you provide more insights into the computational cost of the DSA framework compared to traditional pruning methods?", "limitations": "The authors discussed some limitations of their method in the conclusion.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**S1.**\nThis paper tackles an important problem in LLM efficiency by proposing a novel automated framework for sparsity allocation.\n\n**S2.**\nExtensive experiments demonstrate that DSA outperforms existing methods like SparseGPT and Wanda across multiple benchmarks and LLMs.\n\n**S3.**\nThe approach is validated on diverse tasks, including arithmetic, knowledge reasoning, and multimodal tasks, showcasing its versatility and effectiveness.", "weaknesses": "**W1.**\nThe novelty of the approach is somewhat limited as it combines existing techniques such as AutoML and evolutionary algorithms, which are already well-explored in other contexts.\n\n**W2.**\nSome existing works, such as \"ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models\" by Yi-Lin Sung et al., already address the same problem setting of adaptive sparsity allocation across layers, potentially diminishing the perceived novelty of this paper.\n\n**W3.**\nThe methodology may require significant computational resources, which could be a limitation for practical applications.\n\n**W4.**\nThe results on some benchmarks, while improved, may not be sufficiently groundbreaking to warrant acceptance in top-tier conferences like NeurIPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer VeXf ,**\n\nThanks for the valuable feedback. We have tried our best to address all concerns in the last few days. If the reviewer finds our response adequate, **we would appreciate it if the reviewer considers raising the score**. Please see our responses below one by one：\n\n\n\n> **Q1: Compare to existing AutoML techniques.**\n\n**A1:** In contrast to common AutoML and evolutionary algorithms, our method is specifically **tailored for sparsity allocation discovery in LLMs** and involves several novel components: \n\n**(1) Novel problem formulation:** We are the first to frame LLM sparsity allocation as an AutoML problem, opening new avenues for optimizing LLM efficiency.\n\n**(2) Tailored search space:** We introduce a distinctive search space customized for LLM sparsity allocation, combining pre-processing, reduction, transformation, and post-processing operations in novel ways, allowing for more nuanced and effective sparsity distributions.\n\n**(3) Function discovery and generalization:** Diverging from typical AutoML methods such as NAS and HPO that search for specific modes or hyperparameters, our framework emphasizes generalized function discovery, identifying common patterns across LLMs and formuling  interpretable sparsity allocation.\n\n**(4) Search acceleration:** We develop LLM-specific acceleration techniques to reduce search time, making our DSA practical for large-scale LLM optimization.\n\n> **Q2: Compare to ECoFLaP [ref1] .**\n\n**A2:** We clarify that our novelty lies in being the **first automated search for adaptive sparsity methods**, which significantly differs from traditional adaptive pruning methods like ECoFLaP. Our work differs significantly from ECoFLaP in several key aspects：\n\n**(1) Automation and adaptability:** We employ an automated search method that eliminates the need for expert design and adapts strategies to different models and tasks, whereas ECoFLaP relies on hand-designed, hyperparameter tuning.\n\n**(2) Comprehensive search space:** Our comprehensive search space systematically maps   ***element-wise scores → per-layer importances → sparsity ratios***. In contrast, ECoFLaP simply  computes the keep ratio linearly during  its two-stage pruning. \n\n**(3)  Superior performance:** Our method obtains significant performance gains across various large language and multimodal models, demonstrating superior performance compared to ECoFLaP (see Table below).\n\nTable : Perplexity of Wanda, ECoFLaP and our DSA with LLaMA 7B at 0.6 sparsity  on WikiText2\n\n| Dense | Wanda | ECoFLaP(fIrst-order) | ECoFLaP(zeroth-order) | DSA (Ours) |\n| ----- | ----- | -------------------- | --------------------- | ---------- |\n| 7.26  | 10.68 | 10.16                | 9.83                  | **9.15**   |\n\n**We will augment this discussion and cite more Efficient AI studies [ref1]-[ref5] in the revision.**\n\n\n\n> **Q3 & Q5: About computational resources and cost compared to traditional methods**.\n\n**A3: Our responses are:**\n\n**(1)** Our method's main computational cost is in the initial search phase, taking about 0.5 days on LLaMA-1-7B. But the discovered allocation functions are transferable to other models **without additional costs. This one-time cost can be spread across multiple pruning runs.  As discussed in the limitations section.** we've developed acceleration techniques to address the search cost challenge common in AutoML methods.\n\n**(2)** Most conventional pruning methods also require time-consuming hyperparameter tuning, which may be less efficient than our well-optimized automated search process.\n\n**(3)** Applying our allocation function to prune the model proves highly efficient **(see following Table for detailed pruning speeds).** This efficiency stems from two key factors:  **(a)** we utilize element-wise scores from the uniform pruning method directly, avoiding extra forward and backward computations.  **(b)** we employ reduction operations to simplify computations linearly.\n\n*Table : Comparison of time overhead used for computing the pruning metric of LLaMA-1-65B to 50% sparsity.*\n\n| SparseGPT    | BESA      | ECoFLaP(Zeroth-order) | Wanda        | OWL(Wanda)  | DSA(Wanda)  |\n| ------------ | --------- | --------------------- | ------------ | ----------- | ----------- |\n| 1353 seconds | 4.5 hours | 6.6 seconds           | 5.6  seconds | 6.3 seconds | 6.5 seconds |\n\n> **Q4: About significance of results.**\n\n**A4: Our responses** are: \n\n**(1)** We would like to clarify that our method consistently achieves groundbreaking gains across multiple challenging tasks (reasoning and multimodal benchmarks). Notably, the LLaMA-1|2|3 model pruned by our DSA reach 7.48%|5.69%|14.14% gains on the state-of-the-art models Wanda and SparseGPT. \n\n **(2)** To ensure fair comparisons, we follow standard settings like 50% sparsity, where baseline results are strong, making additional gains challenging. \n\n **(3)** Our experiments in Table 8 at higher sparsity levels (65%-80%) on LLaMA-1 show our method's ability to achieve substantial gains. \n\n **(4)** Results in the rebuttal PDF **(See our general response)** further demonstrate that our method can achieve substantial gains, with improvements ranging from **1% ~ 7% at 60% sparsity, 2% ~ 7.6% at 70% sparsity, and 1.4% ~ 4.2% at 2:4 sparsity.**\n\n\n\n> **References:**\n>\n> [ref1] ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models. In ICLR, 2024.\n>\n> [ref2] Training Neural Networks with Fixed Sparse Masks. In NeurIPS, 2021.\n>\n> [ref3] Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy. In ICLR 2024.\n>\n> [ref4] LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning. In NeurIPS 2022.\n>\n> [ref5] VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks. In CVPR 2022.\n\n\n\n**Finally,** we hope our response could address the concerns, and we thank the reviewer again for the helpful comments. We are glad to discuss further comments and suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# **General Response**


**Dear Reviewers, Area Chairs, Senior Area Chairs and Program Chairs,**

We sincerely thank all reviewers for their positive feedback and constructive comments. **In the initial review, 3 Positive ratings are given.** Reviewers positively acknowledge **the novelty of the idea, the methodology employed, the extensive experiments conducted, the superior performance, and the good presentation of the paper**. More encouragingly, **Reviewer of38, 1f5c, VeXf, and gYqu** think our **novel automated** **framework tackles an important problem in LLM efficiency** for the community.

**[Important problem]:**

- **Reviewer VeXf:** "tackles an important problem in LLM efficiency"

- **Reviewer of38:** "tackles an important problem"

**[Novelty]:**

- **Reviewer 1f5c**: "proposes a novel method"
- **Reviewer VeXf**: "novel automated framework"
- **Reviewer gYqu**: "framework for automatically discovering effective sparsity allocation functions"

**[Theoretical Soundness]:**
- **Reviewer s93u**: "technically sound and straightforward in principle"

**[Extensive experiments]:**

- **Reviewer of38**: "evaluated on a wide range of models"

- **Reviewer VeXf:** "Extensive experiments"

**[Superior performance]:**

- **Reviewer 1f5c:** "shows better performance than SOTA pruning methods"
- **Reviewer of38:** "demonstrated strong results compared to state-of-the-art methods"
- **Reviewer s93u:** "Empirical results demonstrate the strength of this approach"
- **Reviewer gYqu**: "demonstrates consistent performance improvements"
- **Reviewer VeXf:** "DSA outperforms existing methods"

**[Good presentation]:**

- **Reviewer 1f5c:** "insight of the paper is clear and easy to understand"
- **Reviewer s93u:** "This manuscript is a qualified paper"

In the past days, we carefully improved the experiments (using all computational resources we have), the clarifications, and the discussions of our work to address the concerns, the questions, and the requests of all four reviewers.

**In the attached rebuttal PDF, we provide detailed experiment results of higher sparsity ratios and 2:4  sparsity on LLaMA-2 and LLaMA-3 (Mean accuracies are summarized in Table below).** Our DSA method consistently boosts mean accuracies on seven zero-shot tasks across various LLaMA models and sparsity levels. The application of DSA leads to notable improvements in performance, **with improvements  ranging from 2.34% to 7.03% in Magnitude pruning, 1.95% to 7.68% in SparseGPT, and 1.92% to 3.30% in Wanda**, demonstrating the effectiveness of DSA in enhancing model accuracy in different scenarios and with various pruning methods.  

*Table: Mean accuracies (%) of our DSA on 7 zero-shot task.*

| Method              | LLaMA-2-7B (60%)  | LLaMA-2-7B (70%)  | LLaMA-2-13B (60%) | LLaMA-2-13B (70%) | LLaMA-3-70B (60%) | LLaMA-3-70B (70%) | LLaMA-2-7B (2:4)  | LLaMA-2-7B (2:4)  |
| ------------------- | ----------------- | ----------------- | ----------------- | ----------------- | ----------------- | ----------------- | ----------------- | ----------------- |
| Magnitude           | 35.61             | 50.81             | 38.38             | 51.16             | 55.86             | 38.76             | 45.58             | 49.89             |
| **Magnitude (DSA)** | **37.95 (2.34↑)** | **57.84 (7.03↑)** | **46.06 (7.68↑)** | **54.28 (3.12↑)** | **60.24 (4.38↑)** | **42.98 (4.22↑)** | **49.78 (4.20↑)** | **53.38 (3.49↑)** |
| SparseGPT           | 43.61             | 60.68             | 48.76             | 70.14             | 65.03             | 43.22             | 50.94             | 54.86             |
| **SparseGPT (DSA)** | **45.56 (1.95↑)** | **61.31 (0.63↑)** | **50.04 (1.28↑)** | **72.12 (1.98↑)** | **67.34 (2.31↑)** | **45.73 (2.51↑)** | **52.66 (1.72↑)** | **56.35 (1.49↑)** |
| Wanda               | 36.08             | 60.90             | 41.46             | 72.00             | 40.51             | 40.44             | 48.75             | 55.03             |
| **Wanda (DSA)**     | **38.00 (1.92↑)** | **62.08 (1.18↑)** | **43.18 (1.72↑)** | **73.24 (1.24↑)** | **42.74 (2.23↑)** | **42.78 (2.34↑)** | **52.05 (3.30↑)** | **57.07 (2.04↑)** |

**Finally**, based on the constructive comments by all reviewers and our responses, **we will carefully revise the manuscript of our work**. We hope our detailed responses help address the concerns, the questions, and the requests of all  reviewers.

Meta-review: This paper receives ratings of 5,6,6,6,4. The majority of reviewers lean towards acceptance, and the reviewer who gave 4 seemed reasonably content with the rebuttal. The paper seeks to automate the process of assigning sparsity ratios for each layer in pruning LLMs, which is an important yet sometimes overlooked problem. The reviewers mentioned the method is novel, the experiments are extensive, and the results generally show improvement. Congratulations to authors for the paper acceptance.
Decision: Accept (poster)