SparseLLM: Towards Global Pruning of Pre-trained Language Models
Authors: Guangji Bai
Yijiang Li
Chen Ling
Kibaek Kim
Liang Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Model
Pruning
Primary area: natural_language_processing
TL;DR: This paper proposed a novel framework for global pruning of large language models, via considering the LLMs as a chain of modules and introducing auxiliary variables.

Abstract: The transformative impact of large language models (LLMs) like LLaMA and GPT on natural language processing is countered by their prohibitive computational demands. Pruning has emerged as a pivotal compression strategy, introducing sparsity to enhance both memory and computational efficiency. Yet, traditional global pruning is impractical for LLMs due to scalability issues, while local pruning, despite its efficiency, leads to suboptimal solutions. Addressing these challenges, we propose *SparseLLM*, a novel framework that redefines the global pruning process into manageable, coordinated subproblems, allowing for resource-efficient optimization with global optimality. SparseLLM's approach, which conceptualizes LLMs as a chain of modular functions and leverages auxiliary variables for problem decomposition, not only facilitates a pragmatic application on LLMs but also demonstrates significant performance improvements, particularly in high-sparsity regimes where it surpasses current state-of-the-art methods. Our source code is publicly available at https://github.com/BaiTheBest/SparseLLM.

SparseLLM: Towards Global Pruning of Pre-trained Language Models
Guangji Bai
Yijiang Li
Kibaek Kim? Emory University, Atlanta, GA. USA
Chen Ling'
Liang Zhao !
2 Argonne National Laboratory, Lemont, II, USA *Corresponding Author {guangji i. bai, chen. 1ing, 1 liang. z 0emory. edu fyijiang. 1i, kimk .oov
Abstract
The transformative impact large language models (LLMs) like LLaMA and GPT natural language processing is countered by their prohibitive computational demands. Pruning has emerged as a pivotal compression strategy, introducing sparsity to enhance both memory and computational efficiency. Yet, traditional global pruning is impractical for LLMs due to scalability issues, while local pruning, despite its efficiency, leads to suboptimal solutions. Addressing these challenges, we propose Sparsel.LM, a novel framework that redefines the global pruning process into manageable, coordinated subproblems, allowing for resource-efficient optimization with global optimality. SparseLLM's s approach, which conceptualizes LLMs as chain of modular functions and leverages auxiliary variables for problem decomposition, not only facilitates pragmatic application on LLMs but also demonstrates significant performance improvements, particularly in high-sparsity regimes, surpassing current ofate-of the art methods. Our source code is publicly available /tttts:////ttt //ggtttuu.... com/ Bai Bwasihhheessttssel...lll
Introduction
Large language models (LLMs) 1 have recently transformed the field of natural language process- ing (NLP) by delivering exceptional results across variety of intricate language benchmarks [3]5 (.. Nonetheless, these models, with billions of parameters, generally necessitate significant computa- tional resources. To make LLMs more accessible, extensive efforts have been devoted to model compression of I.LMs 6 77, including pruning. quantization, knowledge distillation, and low-rank factorization. Pruning, by introducing sparsity, jointly enhances memory and computational effi- ciency and offers unparalleled flexibility, seamlessly integrating with any LLMs, thus standing out as highly effective and widely adopted compression strategy.
Model pruning has a long history g and has proven effective in applications related to vision and smaller language models 19.. However, conventional pruning techniques, which rely global pruning and require loading the entire model into the same GPU [T]. become impractical for today's 's LLMs due to their vast size. Recently, several local pruning methods have been proposed for billion-scale L.Mss. These methods compress each layer separately, and the overall compressed model is then obtained by "stitching together" the individually compressed layers. SparseGPT [12]. an efficient unstructured pruning method for L.MMs with hundreds of billions of parameters, achieves parameter reduction of up 60% with minimal performance loss. Another approach, Wanda [13] introduces a novel pruning criterion that evaluates weights by considering both magnitude and related input activations. Despite its efficiency gains, local pruning only aims minimize the local error for each specific layer under sparsity constraints, resulting in suboptimal solution for the overall model.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

This is because local pruning over-aligns the intermediate layers' activations, leading to suboptimal performance, especially in high-sparsity regimes
14.
To address these challenges and achieve global pruning with low memory consumption, we pro- pose SparseLLM that decomposes the global prun- ing objective into multiple subproblems, each of which can be solved with low resources and coordi- nate to achieve the global pruning objective. More specifically, we first formulate LLMs as a compos- ite function where the output of one module is the input of the next. Based on this formulation, we re- formulate the global pruning goal into an equivalent form with auxiliary variables that facilitate its de- composition and coordination of the subproblems. Then we propose an alternating optimization algo- rithm to efficiently solve the subproblems, achiev- ing computational resource efficiency and global optimality, due to the close-form solution of each subproblem. Empirically, we find that SparseLLM can consistently improve the performance of lo- cal pruming methods, particularly in high sparsity regimes (> 60%), where the perplexity can be sig- nificantly decreased by up around 80% as compared to the state-of-the-art methods.
Global Pruning
Local Pruning
SparseLLM (Ours)
Out w/o prune
Out N w/o prume
Out w/o prune
Layer
Layer
Layer
w/o prune
Inp
w/o prune
Aux
Layer
Layer
Layer
Inp 2 w/l pruns
Aux inp 2
t Layer
Out w/o prune
Aux out 1
Layer 1 T Inp w/o
Inp w/o prune
Inp w/o prune
LLM too large fit single machine!
Only consider local error for pruning!
Globul pruning with low memory cost!
Figure 1: SparseLLM decomposes the global pruning of LLMs into manageable subproblems by leveraging the chain of modules and auxil- iary variables while maintaining dependencies.
Furthermore, our LL framework can be readily applicable to enhance the performance of most existing local pruning solvers, such as SparseGPT and Wanda, with marginal additional computational overhead. This adaptability ensures that our framework can be seamlessly integrated into a wide range of LLMs and pruning methods, making it versatile tool and useful baseline for future research exploiting the sparsity of LLMs.
2 Related work
Pruning, pivotal concept in machine learning that introduces sparsity into neural networks, dates back to the 1980s [8]. It gained renewed attention in the late 2010s, especially for deep neural networks, under the banner of reducing inference costs [15] LLM pruning techniques can broadly be categorized into structured and unstructured prunings.
looks at simplifying the complexity of LLMs by removing certain parameters regardless of the model's inherent structure. This approach typically involves setting threshold to nullify parameters below it, leading to model with non-uniform sparse structure. SparseGPT (12), an efficient unstructured pruning method for L.MSs with hundreds of billions of parameters, achieves up 60% parameter reduction with minimal performance loss. A novel pruning criterion is introduced in Wanda 133. which evaluates weights by considering both magnitude and related input activations. This approach is beneficial in linear layers of LLMs, helping to identify and remove less significant weights. Tuli and Jha (18] proposed DynaTrra, a dynamic inference scheme for pruning activations runtime, supported by specially designed ASIC architecture, AccelTran, to enhance transformer inference throughput.
Unstructured pruning [16] 17
On the other hand, structured pruning involves the selective removal of groups of weights, where "group" might mean blocks of weights, filters, attention heads, or other structures conducive to hardware acceleration. Ma et al. [9] introduced the LLM-Pruner, framework designed for structured pruning of LLMs, which utilizes a combination of first-order data and Hessian information for effective importance estimation. This aids in identifying crucial groups for pruning. Li et al. (20 proposed LoSparse, a novel approach combining low-rank and sparse matrix approximations to balance pruning and expressive power. Tao al. [2I] extended this concept to pruning hidden dimensions in LLMs, including embedding layers and attention heads. ZipLM (22). structured pruning method for L.MMs, is proposed optimize for compression and accuracy while considering specific hardware constraints. More recently, Xia et al introduced LLM-shearing (23), structured pruning method that scales down LLaMA models by selectively pruning layers, heads, and dimensions.
2

---

This approach, combined with dynamic data batching, reduces pre-training compute costs while maintaining competitive performance, outperforming similar open-source models on key tasks.
Our work falls in the category of unstructured pruning of LLMs, where existing methods such as SparseGPT and Wanda only consider entirely local pruning algorithm and suffer from suboptimal performance. We discuss the limitations and challenges of entirely local pruning in Sec.
3
Background and notation
3.1
Global pruning
Given pre-trained neural network f with parameter W and inputs X. global pruning aims to find global sparsity mask M and possibly updated weights W to minimize the global loss L between the final outputs of the uncompressed and compressed model:
minM, w M fX W)).
(I)
where o denotes the element-wise multiplication. In addition to NP-hardness 24] however, a critical challenge in solving Eq. is the huge memory cost, needs to store the entire model in single GPU, rendering this method impractical for modern billion-scale LLMs.
3.2 Local pruning
Local pruning circumvents the memory issue mentioned above by dividing the full model compression into subproblems for each layer and constructing local loss measure the l2 -e-erro between the outputs of the uncompressed and compressed layers. Hence, the local pruning can be formulated by
min W. We X (M, Wx) X
Although smaller than the global pruning, the local pruning still needs to optimize both the mask M, and the remaining weights W, and thus remains NP-hard. Therefore, exactly solving it for larger layers is unrealistie, leading all existing methods resort approximations.
Mask selection & weight reconstruction. A particularly popular approach to separate the problem into mask selection and weight reconstruction [25 226. Concretely, this means first choosing a pruning mask M according to some salient criterion, like the weight magnitude (27), and then optimizing the remaining unpruned weights while keeping the mask unchanged. Importantly, once the mask is fixed, Eq. turns into a linear regression problem that can be easily optimized.
Existing solvers. Early work (88 applied iterated linear regression small networks. Recently, the AdaPrune approach [25] has shown good results for this problem on modern models via magnitude- based weight selection, followed by applying SGD steps to reconstruct the remaining weights. Follow- up works demonstrate that pruning accuracy can be further improved by removing the strict separation between mask selection and weight reconstruction. More recently, I2 developed SparseGPT, an efficient unstructured pruning method for LLMs with hundreds of billions of parameters, achieving up 60% parameter reduction with minimal performance loss. [13] introduced a novel pruning criterion in Wanda, which evaluates weights by considering both magnitude and related input activations.
3.3 What is wrong with local pruning?
As shown in Eq. 2 local pruning focuses on minimizing the error for each specific layer subject to sparsity constraints. This results in a suboptimal solution with respect to the global pruning problem. While the primary goal of pruning is ensure that the input and output of the pruned model align closely with those of the original models, the local pruning overly constrains the activations of all the intermediate layers between the models. leading to performance degradation.
SparseLLM: Towards global pruning for LLMs
We present our proposed method SparseLLM that can address the drawbacks of existing pruning methods by achieving a global pruning with low memory consumption. SparseLLM decomposes global pruning objective into many subproblems, each of which can be solved using low resources and can coordinate each other toward the global pruning objective. An overview of Sparsel on the OPT and LlaMA configurations are shown in Figure 2
3

---

4.1 Motivation
The development of Sp is motivated by the observation: LLMs can be formulated as a composite function such that the output of one module is the input of the next. This allows to reformulate the global pruning goal into its equivalent form with auxiliary variables that enable the decomposition into multiple subproblems. as detailed in Sec. [.2) Then develop resource-efficient algorithm that achieves the alternating optimization of the subproblems with global optimality, thanks to the close- form solution of each subproblem, illustrated in Sec. 4.3
4.2 A unified formulation of pruning
In this section, we present the reformulation of the global pruning problem into an equivalent one by introducing auxiliary variables. This reformulation provides a more flexible form and enables the decomposition of the problem into many manageable subproblems.
The key idea behind our formulation is to decouple the densely parametric parts (linear layers) from
non-parametric parts (activation function, self-attention, layer norm, etc) using a splitting technique. Rather than feeding the output of the dense linear layer We directly into the non con-arametric and
potentially nonlinear layer Φe, we store the output of layer in new variable z W Waa--1 [1] We also represent the output of the non-parametric layer as a vector of activations solve the following problem:
Φe(z)). We then
min (M)))a)))
C(ZL.y),
s.t. z = (M, O Welas-1, A [L]
(3)
ag = (z(e), VlER,
ag, Zt = at 24 pre E 1]
pre
where L represents the total number of dense (linear) layers and [L] (1, L). [L- lll 2t
denotes the complement set of Ω. We use a pre pre to denote the corresponding intermediate variables' values of the original dense (i.e., without pruning) pre-trained model. y denotes the ground-trute -tutt final output of the dense pre-trained model.
In our proposed formulation above, its unified nature lies in the interpretation and application of the set Ω. which denotes the indices of layers subject to the pruning process. Intuitively, Q measures how "global" the pruning is. The bigger the of Ω is, the more layers are connected via the second constraint, and the pruning is more towards the global extreme, and vice versa. The generality and versatility of our formulation is illustrated in the following remark:
Remark 4.1 (Generality and flexibility of Eq. 3. Given an LLM formulated as composite function with dense layers 1 E {1,2, 2, ... L 1}, where L the total number of dense layers and denotes the set of layers subject to the pruming process. Our formulation can seamlessly treat both global and local pruning as special cases under certain conditions. Specifically:
When Ω = 21, L 1}, solving our pruming formulation is equivalent to global pruning, accounting for inter-layer dependencies across the entire network.
When Ω = 6. the formulation simplifies to local pruning. considering each layer independently (the last constraint dominates and "cuts" all layer dependencies with pre-trained values.)
The ability to shift between these two extremes, and potentially any intermediate configurations. demonstrates the flexibility and comprehensiveness of our formulation. By adjusting Ω, one can seamlessly transition from global perspective local perspective. This flexibility not only caters to wide range of pruming strategies but also provides unified framework to compare and contrast the effectiveness of different pruning methods under consistent mathematical lens.
4.3 Algorithm design
In this section, we introduce the algorithm design of SparseL.LM, which alternatively optimizes the subproblems associated with the corresponding variables. This approach is resource-efficient and achieves global optimality, attributed to the closed-form solutions that each subproblem yields.
For the sake of simplicity and clearer presentation, the bias term is omitted in the following equations where its exclusion does not lead to confusion.

---

Layers prune
Auxiliery veriables (trainable]
Pre: trained model's input/output (fixed)
pro
Add & Norm
Down Proj
Feed
Down Proj
Feed
Gy
Normalization
ReLU
SIUU
S/
Multi-Head
Up Proj
Attention
Gate Proj
Proj
Normalization
07-1
SparseLLM on OPT Layer
SparseLLM on LiaMA Layer
Figure 2: Illustration of SparseLLM on OPT and LlaMA. The auxiliary variables and soft constraints
(i.c.. =) allow SparseLLM to decompose the global pruning into manageable subproblems while maintaining the dependencies. Subproblems are analytically solvable and enjoy fast convergence.
The key idea of our algorithm lies behind the flexibility of Ω in our Eq. we want to find better trade-off between completely global (memory bottleneck) and completely local (suboptimal performance) pruning. Naively applying SparseLLM to prune all layers globally is impractical. On the other hand, recent work shows that the feed-forward network (FFN) module in each decoder layer accounts for more than two-thirds of the total parameters in LLM [2)]. Therefore, prioritizes the global pruning of the FFN module, while still adhering to local pruning strategy for the multi-head attention (MHA) module (see Figure 2 This strategy strikes balance between the computational feasibility pruning large-scale models and the effectiveness of the pruning process, adhering to the limitations and practices of state-of-the fef--hh-art LLM pruning frameworks.
SparseLLM
Formally speaking, rather than trying to solve Eq. directly, we first relax the constraints by adding an é2-penalty function to the objective and attack the unconstrained problem:
C(ZL,y) Σ
rE(T)] ||ze (M, C W.)as-112
1E |a. -e--de (
where a, 3 are hyperparameters for controlling the weight of each constraint. OEE denotes the set of indexes for the linear layers in the FFN module of each decoder layer, i.e., linear layers from the same FFN module are pruned globally. For simplicity, the superscript "pre" of as and Zj in the third constraint in Eq. omitted here, i.e., for & ПPEN the a, and dı fixed and equal to the pre-trained model's intermediate value in the second term of Eq. 4 In the following subsections, we illustrate how we approach the pruning of FFN and MHA modules, respectively.
4.3.1 Sp on OPT models
For each decoder layer in a pre-trained LLM, our Eq, Linstantly simplifies to globally pruning the corresponding FFN module within that decoder layer
al alIzete (M++1 C W
Blla, de(z)))1 3 all ae (M, o W Wo
(5)
where layers l and l correspond to the up-projection and wwojeproecttion linear layers.
In this work, we consider the alternating method to optimize our Eq. 5 i.c., optimize each variable while keeping the rest fixed. The careful and elaborate design of our Eq. allows derive closed-form solution to every subproblem as shown below.
Pruning weight. First consider optimizing Eq. with respect to M, and W.. For each linear layer e in FFN module, the optimal solution minimizes || (M, O W Wt)at-111. To solve it, the first W,a+-1, where W, 22,, ( denotes the pseudo-inverse.) Plug decomposed back in original loss and we get |W (M, C We)ar- 11. which aligns with the pruning objective of Eq. [ and can be analytically solved by existing pruning solver e.g.. SParseGPT. The superscript of "pre" for ag-1 is omitted in this section for simpler notation.
step is to decompose
5

---

Updating activation. Minimization for a, is simple least-squares problem similar to weight pruning. However, in this case, the matrix a appears two penalty terms in Eq. []] so we must minimize a e||z441 (M++1 @ We+1 ]agll + B (zz))) ll for ag, holding all other variables fixed. By following a very similar idea Ridge regression, the new value of ag is given by:
(a 8+1 W 1+1 + (a 2+1 + ReLU(z))),
(6)
where We denotes the updated weight matrix after pruning, i.e., We = We.
Updating output. The update for ze requires minimizing the following loss:
3 a ReL.U(z))l al|ze (M, O Wo)a
This problem is non-convex and non-quadratic (because of the non-linear function ReLU), Fortunately, because the ReLU function works entry-wise on its argument, the entries in Z, are de-coupled. Solving Eq. [ particularly easy for the case of ReLU, as it can be solved in closed form followed by simple if-then logic. Specifically, one only needs to compute two solutions of quadratic equation:
zz
(M, O
We)a pre
2
= (a + B) (Baz + azz²))
(8)
where the first solution corresponds those entries of zz that are negative (reduced to zero by ReLU), and the second solution corresponds to those entries of ze that are non-negative.
4.3.2 LLM on LJaMA models
In this section, we introduce how SparseLLM decomposes global pruning into subproblems and solves them iteratively on LlaMA model families. The model architecture LlaMA can be found in Figure 2 Overall, SparseLLM operates similarly on both LlaMA and OPT models, with the main difference being that LlaMA includes an additional dense linear layer, known as the gate projection layer, and uses the SiLU activation function instead of ReLU.
Pruning weight. In this part, Sparsel.IM functions almost identically to its operation on OPTs.
Updating activation ag. Similarly, for updating ag, SparseLLM works nearly the same as on OPT. The minimization for at simple least-squares problem. akin to weight pruning. However, in this case, the matrix ag-1 appears in two penalty terms in Eq. necessitating the minimization of:
al 2||z - (M++1 @ W Wet1)ael1 + 3l a. SiLU(s;) @ zz 12
(9)
for ag, with all other variables held fixed. Following concept similar to Ridge regression, the updated value ag is:
(aWI 6+1 W4+1 + 31) - (a pre
2+1 2+1 + + SiLU(s;) O zz),
(10)
where We denotes the updated weight matrix after pruning, i.e., W. M2 W.
Updating output 2 Updating z, is somewhat simpler in LlaMA since the activation function applies over the gate projection layer. The update requires minimizing the loss:
B a (s: O 2el + al|zz (Me @ We)a pre
(11)
This problem is quadratic when solving for zę with other variables fixed. Through mathematical manipulations, the analytical solution for z is found by solving a quadratic equation:
(M, Wo + SiLU(s:)) o a
(12)
22 +
where the division is dlement-wise and 1 denotes the all-one matrix. Updating gate projection output sy. Updating sy involves minimizing:
Bl - (s: @ zell2 + +alls: (M, C W. W.)a
(13)
where M, and W, denote the mask and layer weights for the gate projection layer. This problem is non-convex and non-quadratic due to the non-linear SiLU function. However, since SiLU operates entry-wise, the entries in 8, are decoupled. Despite LlaMA lacking simple closed-form solution as in OPT (which uses ReL.U), the problem can still be solved quickly and analytically using a lookup table of pre-computed solutions, since each element in St depends only three variables.

---

Remark 4.2 (Global convergence of SparseLLM), Consider the objective function given by Eq. 5
under the condition that the activation function & is ReLU. Notice that (1) the objective function is convex with respect to each variable when all others are fixed, and (2) given that closed-form solutions exist for the subproblems in the alternating optimization scheme, the proposed algorithm resembles multiblock ADMM which has been shown to converge to in many applications.
4.3.3 Pruning of MHAs
SparseLLM also prunes other linear layers besides those in FFNs. By following Eq. 4. for each linear layer out of FFN modules, the pruning objective simplifies to a | - (M4+1 O (+ pr which is equivalent (with some simple math) to that of completely local pruning shown in Eq Existing LLM pruning solvers such as SparseGPT and Wanda are applicable here.
4.4 Time complexity analyses
The proposed SparseLLM consists of three main steps, with the overall time complexity being the sum of the complexities of these steps. In the weights pruning step. the complexity is dominated by the pseudo-inverse computation of matrix a₂ (dimensions n x h), which is (nhh)). Using SparseGPT as the solver, the exact pruning step has a complexity of O(h)). The second step, updating activations, involves matrix inversion of the weight matrix W, (size h x h) with a complexity of O(h)). The third step, updating outputs, has lower complexity. Thus, the overall algorithm complexity is bounded by Oh")), therefore making our method's per-epoch time complexity comparable to SparseGPT.
5 Experiments
Experiments setup. We implemented Sparsel.LM in PyTorch 30 and use the HuggingFace Transformers library (31) for handling models and datasets. All pruning experiments conducted on NVIDIA A 100 GPUs For calibration data, we follow [12 and use 128 2048-token segments, randomly chosen from the first shard of the C4 [32] dataset. This represents generic text data crawled from the internet and ensures our experiments are zero-shot as no task-specific data is seen during pruning. We followed existing work 72 73 and praned all linear layers (in FFN and MHHA) to the target sparsity.
Models, datasets & evaluation. We consider the OPT model family (33 and LlaMA-2 model family [I] in our experiments as well as the most recent LlaMA-3 model. We show results on different sizes of models provide broader picture for the performances of Sparsel.LM. In terms of metrics, we mainly focus on perplexity, which is known to be a challenging and stable metric that is well-suited for evaluating the accuracy of compression methods
34 355 We consider the test of www-WikiText2 (36) (WT2) and PTB (37 as well as a subset of the C4 validation data, all popular benchmarks in LLM compression literature [34 38 [2] 13), For additional interpretability, we also
provide zero-shot accuracy results following the same setup of [3]. which is based on the popular
Eleuther harness [39]
Comparison methods. We compare against three baselines, magnitude pruning [27] applied locally, and two other -fate-of-the-art local pruning methods, SparseGPT [12] and Wanda
5.1 Results and analyses
Pruning vs. model sizes. We begin by exploring the pruning capabilities of SparseLLM across various model sizes in comparison to baseline methods. For each model, we consider unstructured sparsity ranging from 70% to 90% with 10% increment, as well as a 3:4 semi-structured sparsity. The 3:4 seritstructured sparsity is inspired by our preliminary results that suggest good performance SparseLLM at high sparsity regimes. However, note that two of our baselines, Magnitude and Wanda, are unable to be configured to this sparsity out-of-box. We conduct sensitivity study on the calibration sample sizes (see Appendix (A.3) and use calibration sample sizes between 32 and 64 for all experiments. Moreover, we prune the first 50% of the Transformer decoder layers in each model to achieve a balance between the computation resources and the performances. Detailed results can be found in Table and Table 2 as well as Table in Appendix [A.5] Note that in T for LlaMA-3 model, we only compare SparseGPT to the proposed SparseLLM. The perplexity results of the dense models reported next the names of the models.
From the tables, it shows a general trend of increasing perplexity with increasing sparsity. Moreover, observe a trend of decreasing perplexity for SparseGPT and SparseLLM at the same sparsity
7

---

Table 1: Perplexity of OPT models for sparsity 2 70%; the lower the perplexity, the better.
OPT-1.36 (WikiTeaz2 (WTZ): 14,62, PIB: 20.29; CA: 16.07)
Spunity
70%
80%
90%
3:4
Dataset
PTB
C4
PIB
C4
C4
C4
Magnitade
6420.80
4828.13
3435.99
9998.71
1.1e4
5347.8 79
8209.
1.0c4
4917.02
Wanda
21.56
34.77
25.78
142.20
146.76
142.24
5692.65
4751.69
4501,73
SparseGPT
18.04
28.19
21
69.67
93.36
60.83
2596.70
2361.86
1363.08
252.81
238.41
146.21
Mp
17.82
20.99
58.36
1192.36
635.76
128.83
106.01
OPT-2.76 (WikiText2 (WT2): 12.47;
PTB: 17.97;
C4: 14.32)
Spanity
70%
80%
90%
3:4
Dataset
WT2
PTB
WT2
PTB
C4
WT2
PTB
C4
WT2
C4
Maguitude
1691.74
1237.08
1415.02
1.0c4
7916.69
6050.07
7.9c5
5.3c5
4.7c5
Wanda
140.09
90.06
4746.96
5678.66
3.5e4
2.4c4
SPareePT
13.79
21.18
16.18
24.32
37.82
25.92
2662.74
2285.01
1776.08
91.02
91.79
64.95
SparseLLM
13.82
21.07
16.14
23.87
37.09
24.00
1200.12
759.11
527.70
77.14
52.77
OPT 3h (WikiText2 (WT2): 10.13; PTR: 14.52; C4: 12.06)
Spansity
70%
80%
85%
3:4
Dataset
WT2
PTB
C4
WT2
PTB
C4
WT2
PTB
C4
WT2
PTR
C4
Magnitude
9037.12
7734.58
5909.47
1.1c4
9140.88
6340.22
1.364
1,304
9087.50
Wanda
30.94
39.26
33.31
4216.04
2894.77
2450.57
1.164
1.1c4
7244.96
SPurseGTT
10.89
16.35
13.39
21.42
33.62
21.01
8408.03
6380.30
3416.23
4715.16
7454.37
2.1
Spursel. M 10.96
16.57
13.38
19.07
28.77
19:29
2052.27
1536.51
538.61
289.17
687.48
677.13
OPT-306 (WikfTex22 (WT2): 9.56; PTB: 14.04; C4: 11.45)
Sparsity
70%
80%
90%
3:4
Dataset
WT2
PTB
C4
WT2
PTB
C4
WT2
PTB
C4
WT2
PTB
C4
Magnitude
8691.40
4769.89
4732.66
8941.81
5292.98
5092.26
3.8e7
3.0c7
1.4c7
Wanda
7766.61
5547.45
$741.74
8770.33
6020.70
7132.20
6354.15
4296.37
4654.27
SparseGPT
9.58
17.67
5169.50
3555.24
441.35
464.73
209.44
Spurse LLM 9.56
14.40
11.94
15.61
19.64
16.61
3050.63
2712.39
1758.63
51.28
73.61
37.99
OPT-66h (WikiTex2 (WT2); 9.34; PTB: 13.36; C4: 10.99)
Spunity
70%
80%
90%
3:4
Dataset
WT2
PTB
C4
WT2
PTB
C4
WT2
PTB
C4
WT2
FIB
C4
Magnitude
0OM
0OM
Wanda
0OM
0OM
00M
DOM
0OM
0OM
00M
0OM
OOM
SparseGPT 9.45
13.64
11.37
28.27
57.41
26.26
7803.10
6594.88
4433.35
6694.37
6829.59
3799.87
Sparse LL
9.37
13.66
11.37
16.45
21.00
17.70
7504.17
5644.65
3683.91
4641.8
5296.93
1618.43
Table 2: Perplexity of LlaMA models for sparsity 2 70%; the lower the perplexity, the better.
L.MA (WikiTes22 (WT2): 5.47; PTR: 37.91; 7.26)
Sparsity
90%
Dataset
WT2
PTR
WT2
PTB
C4
WT2
PTB
C4
WT2
PTR
C4
Magnitade
1058.00
544.43
889.46
6380.27
NaN
4162.92
9498.91
1.02p4
7539.65
Wanda
2644.22
4040.95
1630.09
1814.01
3376.35
1124.26
5206.93
4607.30
2780.45
SparseGPT
15.98
302.15
18.58
53,20
803,02
52.57
344.97
2503.82
279.77
68.28
784.79
60.45
Sparsel. M
16.15
274.35
18.23
49.96
664.39
47.39
225.23
2233.55
181.56
64.17
667.27
54.56
LiaMA (WikiTex12 (WTZ):
50.94;
6.73)
Sparsity
70%
80%
90%
3:4
Dataset
WT2
PTB
C4
WT2
PTB
C4
WT2
PTBB
C4
WT2
PTB
C4
Magnitade
30.34
2317.39
28.48
4133.98
4706.65
12.69
5580.71
5514.22
5090.63
Wanda
23.42
502.53
32.65
295.29
2340.13
261.15
3003.49
3804.69
1738.73
SparseGPT
12.98
267.63
45.59
$50.59
45.20
825.99
673.33
63.48
660.70
$6.29
LM
12.95
277.76
15.77
36.36
$78.35
38.63
646.15
1078.94
466.98
53.71
632.11
50.40
IJaMA-3 8b (WikiText2 (WT2): 6. 14; PTR: 11 18:18: C4: 9.45)
Sparsity
70%
80%
90%
3:4
Dataset
WT2
PTB
C4
WT2
PTB
C4
WT2
PTB
C4
WT2
PTB
C4
SparseGPT 22.37
36.56
72.87
113.95
261.18
96.75
107.52
102.11
1 20.98
33.78
28.94
$7.83
85.98
72.18
197.47
241.68
181.69
76.33
99.54
93.68
8

---

Table Perplexity of 2:4 sparsity; the lower the perplexity, the better.
OPT-1.3h
OPT-2,7b
OPT-6.7b
OPT-13h
Dataset
WT2
PTB
C4
WT2
PTB
C4
WT2
PTB
C4
WT2
PTB
C4
Magnitude
96.68
133.92
48.08
272.34
308.55
267.70
64,11
92.23
82.67
67.07
110.77
52.61
Wanda
15.63
24.04
18.23
13.66
21.67
16.10
11.86
18.54
14.77
10.33
15.35
12:54
SpaneGPT 15.11
17.88
19.28
16.90
13.51
15.14
12.48
SpunseLLM
14,97
23.40
17.67
12.62
19.28
15.12
11.07
16.73
13.42
10.20
15.14
12.41
with increasing model sizes. However, such a trend is not obvious for Magnitude and Wanda. We also observe that SparseGPT and SparseLLM consistently outperform Magnitude and Wanda by significant margin. For smaller sparsity, SparseLLM achieves comparable perplexity to SparseGPT. As we increase the sparsity, SparseLLM starts demonstrate noticeable improvements SparseGPT. In numerous instances for the OPT model family, SparseLLM achieves perplexity reductions of more than 50% compared to SParseGPT. We also that performance improvements from SparseLLM over SparseGPT are more significant for the OPT model family than the LlaMA-2 model family.
We provide additional set of perplexity results for a 2:4 semi-structured sparsity for few OPT models in [ We see that SparseLLM and SparseGPT generally outperform Magnitude and Wanda while SparseLLM has comparable if not slightly better performances compared to SparseGPT with the 2:4 semi-structured sparsity. Note that 2:4 semi-structure sparsity is considered to be low sparsity regime.
Zero-shot experiments. To further conclude the evaluations and discus- sions, we show results for several zero- shot tasks in Table 4 and Table 5 as well as Table appendix[A.5] com- paring SparseGPT and SparseLLM. These evaluations are known to be relatively noisy [40], but more inter- pretable. We also report the results for zero-shot tasks from the dense models in the "Dense" row. We see that the accuracy of both methods decreases with increasing sparsity, which is ex- pected, more parameters are pruned. similar trend of increasing accuracy with increasing model size is observed too. Across all the tasks, OBQA and ARC-c remain the most challenging ones as the accuracy for both methods is 30% or below 30% while both methods perform well for BoolQ, RTE, WinoGrande, and ARC-e. In general, SparseLLM is able to achieve higher accuracy in the majority of tasks across the models of different sizes in both OPT and LlaMA-2 model families.
0.8
0.7
0.8
g066
0.6
Treining
0.4
0.3
0.2
Index
Epoch
Figure 3: Fast convergence of SparseLLM. Training loss per epoch for pruning layer of OPT-125m at 80% sparsity (Left) and layer of LlaMA-2 13b 70% sparsity (Right).
Training loss vs. epochs in SparseLLM. Figure illustrates the change in training loss over epochs for SparseLLM, with the training loss plotted on scale of 10 for clarity. We observe that the training loss decreases rapidly during the initial epochs, highlighting the efficiency of SpMrseLLM in achieving effective global pruning within short period. This rapid convergence is largely due to the closed-for form solutions employed by SparseLLM for various subproblems, which streamline the pruning process and ensure optimal layer-wise pruning without extensive iterative computations. These analytical solutions enable SparseLLM to perform precise pruning operations quickly, making it powerful tool for optimizing large-scale models like LlaMA, significantly reducing model size while maintaining high accuracy.
Conclusion
Our work presents SparseLLM, a cutting-ede edge framework poised redefine the compression LLMss through sparsity. By adeptly circumventing the scalability issues of global pruning and optimizing local suboptimality of existing methods, SparseLLM stands as a significant advancement in the field. Our empirical results affirm its efficacy, particularly in high-sparsity environments. It achieves a notable reduction perplexity, thereby setting a new precedent for model compression. The versatility and minimal computational overhead of SparseLLM complement its integration

---

Table Accuracy (%) of cero tasks for OPT models; the higher the accuracy, the better.
OPT-13b
Sparsity
Method
BoolQ
RTE
HellaSwan
WinoGrandARC-
ARC-c
OBQA
Mean
Dense
65.87
57.76
52.44
66.02
67.82
33.46
28.62
53.14
SparseGPT
54.87
67.47
32.85
70%
LP
63.85
55.23
50.73
65.67
66.46
31.83
27.20
51,57
SparseGPT
59.72 60.89
52.35
62.50
31.23
80%
L.
53.07
46.19
62.12
62.21
30.38
23.00
48.27
SparseGPT
47.49
52.71
33.17
51.54
39.98
21.33
17.80
37.72
90%
S L..
53.43
52.71
38.19
52.96
46.68
25.26
17:40
40.95
SparseGPT
47.55
53.43
31.30
50.20
37.63
22,53
17.60
37.18
3:4
Mp
51.13
52.35
38.51
55.96
49.24
24.83
21.40
41.92
OPT-300
Sparsity
Method
BoolQ
RTE
HellaSwag
WinoGrandsRRC-e
ARC-c
OBQA
Mean
Dense
70.46
61.82
54.27
69.02
70.47
35.49
30.20
55.96
S SparseGPT
68.78
58.48
53.83
67.64
69.15
34.30
29.60
54.54
70%
S Sparsel L.
69.11
61.73
53.97
68.43
69.78
34.73
29.80
55.36
SpurseGPT
64.86
60.65
49.73
61.40
61.91
31.74
24.20
50.64
80%
pursel
59.57
62.71
32.25
S SparseGPT
37.83
53.79
25.96
49.88
26.47
20.22
12:60
32.39
90%
LM
43.55
52.35
27.31
20.56
33.45
SpurseCPT
55.81
51.26
33.64
54.54
42.05
21.33
21.00
39.95
3:4
Sparsel LM
60.83
54.15
39.35
55.41
45.24
24.06
22.20
43.03
Table 5: Accuracy (%) of zero-shot tasks for LlaMA models; the higher the accuracy, the better.
LlaMA-2 7b
Spursity
Method
BoolQ
RTE
HellaSwag
WinoGrandARCCe
ARC
OBQA
Mean
Dense
75.05
66.43
56.92
69.93
75.34
41.89
34.40
59.99
SparseGPT
68.26
$7.04
28.58
70%
Sparsel LM
67.61
57.31
40.12
61.39
59.39
28.76
21.40
48.13
SparseGPT
59.36
52.71
28.83
48.7
34.22
18.34
14.40
36.65
80%
S LM
60.12
53.07
28.62
50.59
34.55
18.69
14,30
37.13
S SparseGPT
39.02
52.34
26.66
47,80
28.32
17.37
12.40
31.99
90%
LP
39.45
52.71
26.79
51.17
28.32
19.52
12.50
32,92
SparseLPTT
$3.94
54.15
28.09
49.17
31.57
17.41
14,80
35.59
3:4
L.M
57.34
53.43
32.45
14.4
LlaMA-2 13b
Sparsity
Method
BoolQ
HellaSwag WinoGrandeARC-e
ARC-c
OBQA
Dense
77.89
70.40
59.94
72.77
77.40
46.50
33.20
62.59
SparseGPT
70.03
53.43
42.20
66.54
64.94
31.66
25.40
50.60
70%
LP
69.87
54.15
64
64.97
31.40
25.80
SparseGPT
62.69
52.71
28.94
50.91
36.24
18.17
14.00
37.67
80%
L.
52.86
18.77
90%
SparseGPT
50.21
$1.35
26.71
49.14
26.68
19.71
13.2
33.86
Sparsel LM
55.35
52.05
26.89
51.34
27.35
19.62
14.20
35.26
SparseGPT
61.28
53.71
47.99
33.21
18.26
14.00
36.69
3:4
SparseLLLM
61.71
55.71
28.56
51.62
32.11
18.49
13.8
37.43
with current pruning technologies, underscoring its potential as universal tool for enhancing the performance and accessibility of LLMs.
Acknowledgments and Disclosure of Funding
This work was supported by the National Science Foundation (NSF) Grant N0.1755850. No.184 1841520. No. 1942594, No.2403312, No.2007716, No.2007976. 1907905 This work was supported by the U.S. Department of Energy, Office of Science, Advanced Scientific Computing Research, under Contract DE-ACC2-D6CHI 1357. This research used resources of the Argonne Leadership Computing Facility Argonne National Laboratory, which supported by the Office of Science of the U.S. Department of Energy under contract DE-AC02-06CH111357
10

---

References
[1] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint Siti:2337.009288, 2023.
[2] R OpenAL. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2:13, 2023.
[3] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. arXiv preprint Stti:2006.07688/ 2022.
[4] Michael Bommarito II and Daniel Martin Katz. Gpt takes the bar exam. arXiv preprint arXiv:2212.14400 14402, 2022.
[5] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. 12712, 2023.
[6] Canwen Xu and Julian McAuley. A survey on model compression and acceleration for pretrained language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 10566-10575 2023.
[7] Guangji Bai, Zheng Chai, Chen Ling, Shiyu Wang, Jiaying Lu, Nan Zhang, Tingwei Shi, Ziyang Yu, Mengdan Zhu, Yifei Zhang, et al. Beyond efficiency: A systematic survey of resource-efficiet large language models. arXiv preprint arXiv:2401 000625, 2024.
[8] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances neural information processing systems, 2, 1989.
[9] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. The Journal of Machine Learning Research, 22(1):10822-1100) 11005, 2021.
[10] Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7765-7773, 2018 2018.
[11] Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. Advances Neural Information Processing Systems, 33:1 1819, 2020.
[12] Elias Frantar and Dan Alistarh. Massive language models can be accurately pruned in one-shot. arXiv preprini urXiv:2301.00774 2023.
[13] Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. /6955, 2023.
[14] Yi-Lin Sung, Jaehong Yoon, and Mohit Bansal. Ecoflap: Efficient coarse-to-fine layer-wise pruning for vision-language models. arXiv preprint arXiv:23 10.02999, 2023.
[15] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. arXiv preprint arXiv: 1510:0100 2015.
[16] Nan Zhang, Yanchi Liu, Xujiang Zhao, Wei Cheng, Runxue Bao, Rui Zhang, Prasenjit Mitra, and Haifeng Chen. Pruning as domain-specific Ilm extractor. arXiv preprint 4r52:22405.06275. 2024.
[17] Guangji Bai, Yijiang Li, Zilinghan Li, Liang Zhao, and Kibaek Kim. Fedspallm: Federated pruning of large language models. arXiv preprint arXiv:2410. 14852, 2024.
[18] Shikhar Tuli and Niraj K Jha. Acceltran: A sparsity-aware accelerator for dynamic inference with transformers. IEFE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2023.
11

---

[19] Xinyin Ma, Gongfan Fang, and Xinchao Wang. Amm-runer: On the structural pruning of large language models. arXiv preprint 1xXi:2305.11627, 2023.
[20] Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao Losparse: Structured compression of large language models based on low-rank and sparse approximation. arXiv preprint arXiv:2306. 1222, 2023.
[21] Chaofan Tao, Lu Hou, Haoli Bai, Jiansheng Wei, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. Structured pruning for efficient generative pre-trained language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10880-10895, 2023.
[22] Eldar Kurtic, Elias Frantar, and Dan Alistarh. Ziplm: Hardware-aware structured pruning of language models. arXiv preprint arXiv:2302.04003: 2023.
[23] Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared Ilama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.0663 2023.
[24] Thomas Blumensath and Mike E Davies. Iterative thresholding for sparse approximations. Journal of Fourier analysis and Applications, 14:629-654 65499-6552222 2008
[25] Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry. Accelerated sparse neural training: A provable and efficient method find m transposable masks. Advances in neural information processing systems, 34:21099 2111111111,111111111111111111111 2021.
[26] Woosuk Kwon, Sehoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. fast post-training pruning framework for transformers. Advances in Neural Information Processing Systems, 35:24101-244 101014114111 16, 2022.
[27] Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint 17Xiv:1710.01878, 2017.
[28] Jason Kingdon and Jason Kingdon. Hypothesising neural nets. Intelligent Systems and Financial Forecasting, pages 81-106, 1997.
[29] Zirui Liu, Qingquan Song. Qiang Charles Xiao, Sathiya Keerthi Selvaraj, Rahul Mazumder, Aman Gupta, and Xia Hu. Físplit: Split foed-forward network for optimizing accuracy-efficieng trade-off in language model inference. arXiv preprint arXiv:2401 1.00444, 2024.
[30] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- peshhrerformecc deep learning library. Advances in neural information processing systems, 32, 2019.
[31] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transform- ers: State-of-the-art natural language processing. arXiv preprint arXiv: 1910.0377,, 2019.
[32] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, (11)::585-5555) 2020.
[33] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0106. 2022.
[34] Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27 167-11838 2022.
[35] Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In International Conference on Machine Learning, pages 7750-7774. PMLR, 2023.
[36] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXi::1699.08843, 2016.
12

---

[37] Mitch Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert Maclntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. The penn treebank: Annotating predicate argument structure. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8- 8-11, 1994, 1994.
[38] Gunho Park, Baeseong Park, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. nuqmm: Quantized matmul for efficient inference of large-scale generative language models. arXiv preprint ar
2022.
[39] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation. 12 2023.
[40] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 O): 8-bit matrix multiplication for transformers at scale, arXiv preprint arXiv:2208.073338 2022.
13

---

A Appendix
This section includes supplemental materials (pseudo-code, additional experiments, and plots).
A.1
Pseudo-code of SparseLLM
Algorithm 1 SparseLLM Pruning of OPT Models.
Input:
An OPT decoder layer containing FFN and MHA modules. FFN's up-scaling linear layer pre-trained weight matrix We, FFN's down-scaling linear layer pre-trained weight matrix We+. input of the up-scaling linear layer a output of the down-scaling linear layer x target sparsity p. constraint weight hyperparameters a, 13.
SparseLLM on (F;
Initialize 2, ","" Ur a.
Initialize slack variables
Pre-compute and cache a!_ pseudo-inverse(a""" for step = 1, K do
W. = - W = zetia
Mr, W. arg min WWa (M. @ Wi)a
Prune layer by SparseGPT solver
Me+1, We+1 E+1 arg mig Wet1a. (Me+1 W aael
Prune layer by SparscGPT
solver
Wi-1 = M O W 1+1, W,=Mi M, O W
+ 3dr(2i)) Z (1) Wia zt (a 8) (Ba: )
(aW7 Wett (1)
(a 141 1,
Update activations
for j = -...... in parallel do
if (ze); < then
(2r), (2) (1)
D Update outputs
else
(ze), (z!22)
b. Update outputs
return We, We+1
SparseLLM (): for each linear layer f in MIA module do 11
Fix 24 x²"" ag ag
Fix intermediate variables > Prune layer by SparseGPT solver
Mr, W, arg min
W.a pre (M; @ W.
12
return (We) for all linear layers in MHA
The SparseLLM algorithm presented in Algorithm demonstrates how SparseLLM works on an OPT decoder layer. The key inputs to the algorithm include the pre-trained weight matrices for both the up-scaling and down-scaling linear layers of the FFN, along with set of hyperparameters and sparsity constraint. The goal of SparseLLM is achieve targeted level of sparsity in the linear layers without significantly compromising the model's 's performance.
Initiating with the pre-trained weights, Sparsel.I. employs series of pruning and activation update steps across iterations. In each iteration, solves optimization problems to prune the current and subsequent layer weights, followed by updating the activation variables. The utilization of SParseGPT solvers for pruning and the strategic update of activations ensures that the pruned network approximates the original network's behavior as closely as possible. The final output of the algorithm a pair of pruned weight matrices for the consecutive layers, which are expected to deliver comparable improved performance with a reduced number of parameters.
A.2 Two-layer Demo on the Details behind our Global Pruning
Figure iilustrates the SparseLLM pruning method compared to conventional global pruning and local pruning, using two-layer neural network as abstraction for simplicity. The figure is divided into three main parts:
On the left, conventional global pruning is depicted. This method applies global mask to the entire network, resulting in significant memory costs due to poor scalability. Both functions f1 and fa are pruned using the same mask across all layers, leading to high memory usage.
14

---

Global Pruning
Local Pruning
Adaptive Global Pruning (Ours)
Y
Y
Global mask
X f2
Dense layer 2
f
K
f2
Locally
f2
X
f2
Prune
Jointly Prune w/o global
Jointly Prune
(Huge (Huge memory cost)
ai
Activation
Pre-trained model
surrogate
Fixed
t
Soft constraints
Initialize
a1
Non-linaar
Auxiiary veriables
function
Output
surrogate
Fixed
2
prn
Initialize
t
X f
Dense tayer 1
x
f
Locally
f
f x
Prune
t
X
X
X
Figure 4: Illustration of LLM pruning method compared to conventional global pruning and local pruning. We consider a two-layer neural network as an abstraction for simplicity. Global pruning (left) is memory prohibitive due to poor scalability. Local pruming (mid) considers pruning each layer independently, while inevitably sacrificing performance due the ignorance of global supervision. Our adaptive global pruning (right) achieves global pruning with low memory cost by leveraging auxiliary variables and soft constraints.
In the middle, local pruning is shown, where each layer is pruned independently. This approach reduces memory costs by applying separate masks to each layer. However, it inevitably sacrifices performance because ignores global supervision, which can lead suboptimal pruning decisions that do not consider the network as a whole.
On the right, the adaptive global pruning method of SparseLLM is presented. This method achieves global pruning with low memory cost by leveraging auxiliary variables and soft constraints. combines the benefits of global pruning- considinnni the entire network structure with efficient memory usage. The introduction of auxiliary variables allows for flexible and adaptive pruning, ensuring that the overall performance of the network maintained while keeping memory costs low.
Thus, the figure highlights the trade-offss between different pruning strategies. Conventional global pruning incurs high memory costs, local pruning reduces memory usage at the expense of performance, and the adaptive global pruning of SparseLLM strikes a balance by maintaining performance with lower memory requirements through the use of auxiliary variables and soft constraints.
A.3 Calibration Samples
Figure 5 and Figure present how perplexity changes with the calibration sample sizes on the datasets PTB and C4 for OPT-2.7b and LlaMA-2 7b, respectively. In both figures, as the number of calibration samples increases, the perplexity decreases for both SparseGPT and Sparsel.IM. This indicates that having more calibration samples can be beneficial in the pruning process. Perplexity decreases more rapidly from samples to 32 samples. Beyond 32 samples, the rate at which perplexity decreases starts to slow down. In addition, increasing the number of calibration samples requires more computational resources, e.g., memory and computation time, in the overall pruning process. This suggests that the calibration sample sizes should be between 32 and 64 to ensure good performance while maintaining computational efficiency. Lastly, the figures show that SparseLLM achieves better perplexity than SparseGPT does with 32 or larger sample sizes for both OPT and LlaMA-2 models.
A.4 Computation Time vs. Model Sizes
We study how the computation time per layer of SparseGPT and SparseLLM varies with different model sizes, as illustrated in Table [ and Table7 OPT models and LlaMA-2 models. The rate at which the time taken increases is comparable for SparseGPT and SparseLLM as the model size increases. Additionally, computation time for Spursel.. are reported for configuration of 4 to 10 epochs. As we have reported in Section Sparsel.LM can reduce the training loss in as few as 2 to epochs. This suggests that the proposed SparseLLM remains computationally efficient.
15

---

OPT-2.7b 70% sparsity with
OPT-2.70 70% sparsity with
18.0
SparseGPT SparseLLM
SperseGPT
23.0
SparseLLM
22.5
Perplesity
17.0
18.3
21.5
18.0
128
32
64
128
Calibration Samples
Calibration Samp
Figure 5: Sensitivity of OPT-2.7b the calibration sample sizes for datasets PTB and C4.
LiaMA-2 7b 80% sparsity with PTB
LlaMA-2 2 76 80% sparsity with C4
1400
SparseGPT SparieLLM
SparseGPT SparseLLM
65
1200
Perplesity
1100
Perplexity
60
900
55
800
700
50
600
16
32
64
Calibration Sampler
#
Calibration
Figure 6: Sensitivity of LlaMA-2 7b models the calibration sample sizes for datasets PTB and C4.
Table 6: Computation time in seconds OPT models.
Method
OPT- 125m
OPT- 1.3b
OPT- 2.7b
OPT- 6.7b
OPT- 13b
OPT- 30b
OPT- 66b
SparseGPT 2.30
10.18
18.35
24.40
28.65
48.91
103.19
SparseLL 16.34
22.79
42.86
174.08
85.62
174.07
284.59
Table 7: Computation time in seconds of LlaMA-2 models.
Method
Llama-2
Llama-2 13b
7b
SparseGPT 11.94
16.58
M 146.80
252.48
A.5 Experiment Results for Additional Models
Detailed results on perplexity and zero-shot task accuracy for additional models are reported in Table 8 and T Similar to other models, we report the perplexity results for the dense model next the name of the model in the table. In particular, we see that SparseGPT and LLM outperform Magnitude and Wanda with a significant margin across different sparsity. SparseL.LM shares similar perplexity with SparseGPT for smaller sparsity but demonstrates much better perplexity for larger sparsity. Similar perplexity trends are observed across all three datasets, although, PTB, having the highest perplexity for each sparsity and method, likely the most challenging dataset among the three. For the zero-shot taks accuracy, we see that SparseL.LM achieves comparable results to SparseGPT for smaller perplexity and the performance improvements are more obvious and significant with higher sparsity.
16

---

Table 8: Perplexity high sparsity regimes 2 70%): the lower the perplexity, the better.
OPT-125m (WikTex22 (WT2): 27.66; PTB: 38.99; C4: 26.56)
Sparsity
70%
80%
90%
3:4
Dataset
WT2
PTB
WT2
PTB
C4
WT2
PTB
C4
WT2
PTR
C4
Magnitude
3806.96
4475.29
Wanda
351.83
412.52
248.94
1912.45
2512.93
1066.86
4940.89
4157.27
3126.02
SparseGPT
239.26
265.83
156.33
2072.12
1952.85
1050.83
6131.57
6963.27
2443.33
148266
2215.44
657.26
Sparsel.TA M 208.46
255.75
137.72
1358.10
1418.09
654.54
5291.64
5067.41
2003,09
914.87
1210.84
450.01
OPT 6.7T-6.7b (WikiTex12 (WT2): 10.86; PTB: 15.77;
C4: 12.71)
Spursity
70%
80%
90%
3:4
WT2
Magnitude
7054.21
5437.44
4850.25
7937.49
5971.86
6031.54
2.404
2.5c4
2.1c4
Wanda
54.95
129.73
116.67
1493.58
1196.93
996,00
2.1e4
2.0g4
L8ee4
SparsoGPT
12.27
18,90
15.28
31.04
51.26
29.42
8871.24
5713.57
3797.20
$70.08
361.81
328.15
L
12.16
128.99
Table 9: Aceuracy (%) of zero-shot tasks; the higher the accuracy, the better.
OPT-6.7b
Sparsity
Method
BoolQ
RTE
HellaSwag
WinoGrandeRRCe
ARC-c
OBQA
Mean
Dense
66.12
56.03
50.49
65.27
65.72
30.63
27.60
$1.69
SparseGPT
54.87
63.85
29.27
25.40
70%
M
60.61
54.51
48.8
62.9
64.14
30.03
26.60
49.66
SparseUPT
42.22
25.85
21.40
80%
Sparsel L.
$8.69
51.26
43.78
59.67
58.38
26.88
22.00
45.81
SparseGPT
38.53
53.07
26.00
48.07
26.81
21.67
14.40
32.65
L
46.48
52.71
26.21
51.70
27.44
19.71
13:40
33.95
SparseGPT
46,70
54.15
28.82
51.07
32.45
18.17
15.40
35.25
SparseLLM 53.49
53,42
36.24
53.51
43.94
22.61
17.40
40.09
A.6 Hyperparameter and /3 Selection
We select a and |3 from the set (0.01, 0.1, 1, 5, 10, 100) and perform study models to understand the impact of the hyperparameters. Results for OPT-1.3b with 70% sparsity is shown Table I0
Hyperparameters a and /3 are used Eq. 5
Table 10: Ablations of the hyperparameters and 9 on OPT-1.3b with 70% sparsity (in perplexity)
0.01
0.1
100
0.01
0.1
18.04
17.82
17.96
18.04
18.40
18.20
18.02
18.11
17.87
17.96
18.22
18.06
18.02
18.03
17.92
17.96
18.04
10
18.03
18.01
17.96
17.96
17.96
18.03
A.7 Limitations and Future Work
While SparseLLM marks significant step forward in the efficient pruning of large language models, is important to acknowledge the inherent rade-offfs associated with any model compression technique. Firstly, while our method reduces the complexity LLMS and enhances computational efficiency, there is inevitable balance between sparsity and performance that requires careful calibration. Additionally, in this work, we still assume homogeneous sparsity, i.e., the pruning sparsity for each layer is the same and equal to the global sparsity. How to achieve heterogencous sparsity under our framework and fully fulfill the potential of global pruning is of great interest. Lastly, the effectiveness of SparseLLM, like any pruning method, may vary across different models and tasks, and generalizability to all scenarios remains an area for further exploration.
17

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect our paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have included section to discuss our paper's 's limitation in Sec. A.7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
We provided all assumptions and proofs for theories in our paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provided comprehensive information as well as the source code of our method.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification:
We provided open-access data and code to reproduce our experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeeee 7/nips.com htc/://bbb///gdddde//eeeeeueisss ssiniilicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provided the necessary details to run our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: results.
We closely following existing literature in reporting the experimental
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provided necessary details on the compute resources we used our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httppp///eeurrpp..ccocopbl//sscs
Answer: [Yes]
Justification: Our paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Our paper has no societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credited and cited licenses of existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provided proper documents for released code and data.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve these.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "KLcH4FcRF7", "reviewer": "Reviewer_FMFD", "summary": "This paper presents SparseLLM, a framework to prune large language models by decomposing the global pruning objective into multiple sub-problems, each of which can be solved with low resources, when combined, solve the global pruning objective. The method reformulates LLMs as a chain of modular functions and uses auxiliary variables to enable problem decomposition. Empirically, SparseLLM shows consistent improvements over the local pruning methods especially in high-sparsity regimens.", "questions": "I've asked most of my questions in the weaknesses section above.", "limitations": "The limitations section is a bit vague and only discusses the requirement for certain structural properties of the network to be satisfied in order to demonstrate the effectiveness of SparseLLM. I would like to add that this does not affect my score.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel method to address the limitations of both global and local pruning for large language models.\n2. The proposed approach is well-grounded in theory with clear mathematical foundations.", "weaknesses": "1. The experiments are a bit weak in model choice. Older models like OPT and Llama-2 are chosen, when a lot of new and better performing models have come out like Mistral, Gemma, and even Llama 3 (which came out in April).\n2. Experiments based on lower sparsity levels are also missing, I would like to see the comparison and time computations at 10/20/50% sparsity as well.\n3. The dense baseline (0% sparsity) is missing from all tables. It's important to gauge the effectiveness of the proposed method.\n4. For small models, it seems that SparseLLM performs on par with SparseGPT, with the added computational complexity.\n5. Ablations on the effectiveness of the hyperparameters $\\alpha$ and $\\beta$ are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer FMFD,\n\nWe sincerely appreciate that you found our paper and method interesting with solid results. Please refer to our response below for details:\n\n> *\"The experiments are a bit weak in model choice. Older models like OPT and Llama-2 are chosen, when many new and better-performing models have come out like Mistral, Gemma, and even Llama 3 (which came out in April).\"*\n\n**A1.**\nWe have performed additional experiments on the LlaMA-3 8b model and presented the perplexity results in Table 1 of our one-page PDF. Our results demonstrate that *SparseLLM* achieves competitive performance on the latest LlaMA-3 model, confirming its applicability to state-of-the-art models.\n\n> *\"Experiments based on lower sparsity levels are also missing, I would like to see the comparison and time computations at 10/20/50% sparsity as well.\"*\n\n**A2.**\nWe provide the perplexity and computation time results for sparsity 10/20/50% on OPT-1.3b and OPT-2.7b in Table 2 and Table 3 in our one-page PDF.  We see similar perplexity results for all four methods for the 10% and 20% sparsity, as naive magnitude pruning can achieve pretty good perplexity. However, we start to see improvements in perplexity from *SparseLLM* starting from 50% sparsity and the improvements are more significant with subsequent even higher sparsity, as shown in the original manuscript. \n\n> *\"The dense baseline (0% sparsity) is missing from all tables. It's important to gauge the effectiveness of the proposed method.\"*\n\n**A3.**\nThe performance of the dense baselines was provided in our original manuscript. In Table 1 (of the original manuscript), it is placed in parentheses after each LLM's name. In Table 2 (of the original manuscript), it is marked as \"Dense\" below the row of dataset names. We will modify our paper to improve its readability in the future.\n\n> *\"For small models, it seems that SparseLLM performs on par with SparseGPT, with the added computational complexity.\"*\n\n**A4.**\n*SparseLLM* consistently achieves competitive results for small models (e.g., OPT-125m, OPT-1.3b, and OPT-2.7b), significantly decreasing perplexity by up to 80% compared to SparseGPT and Wanda. Notable improvements include OPT-2.7b at 90% sparsity: *SparseLLM* achieves a perplexity of 759.11 versus SparseGPT's 2285.01 for PTB, and 527.70 versus 1776.08 for C4, representing over **60%** improvements in both cases. For OPT-125m at 80\\% sparsity for C4, *SparseLLM* achieves a perplexity of 654.54 versus SparseGPT's 1050.83, representing over **40%** improvement.\n\n*SparseLLM* is a generic framework, with mathematical proof that both local pruning and global pruning are special cases. By flexibly switching between these extremes, the computational complexity of *SparseLLM* can be the same as local pruning.\n\n> *\"Ablations on the effectiveness of the hyperparameters $\\alpha$ and $\\beta$ are missing.\"*\n\n**A5.**\nWe present the ablation studies for the hyperparameters $\\alpha$ and $\\beta$ in Table 4 of our one-page PDF. Specifically, we consider the model OPT-1.3b with 70% sparsity on the WikiText2 dataset. We vary the values of $\\alpha$ and $\\beta$ from the set {$0.01, 0.1, 1, 5, 10, 100$} and compare the resulting perplexity. In the table, we use ``-\" to indicate instances of numerical instability. The best combination of $\\alpha$ and $\\beta$ we found is $(0.1, 0.1)$, while the perplexity is in general insensitive to the choice of $\\alpha$ and $\\beta$, which is a good sign, meaning that our method is robust to the choice of hyperparameters.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZFrcuynAit", "reviewer": "Reviewer_snMo", "summary": "This paper introduces SparseLLM, a novel pruning technique targeted at the FFN layers in LLMs. By treating global and local (layer-wise) pruning as special cases in the proposed formulation, SparseLLM can circumvent the limitations of both extremes. The proposed method introduces auxiliary variables and soft constraints within LLM feedforward layers, which helps decompose pruning into subproblems that can be solved analytically. SparseLLM is evaluated on modern LLMs such as OPT and LLaMa2, pruning their weights in both unstructured and N:M patterns; the performance is compared with other state-of-the-art LLM pruning techniques such as SparseGPT and Wanda.", "questions": "* To clarify, does 3:4 mean that 3 out of 4 consecutive elements are zero? N:M sparsity is traditionally defined differently: at most N out of M consecutive elements are non-zero.\n* Can SparseLLM handle alternative activation functions such as GeLU?\n* Can SparseLLM handle networks with residual connections between layers?", "limitations": "Limitations are adequately discussed in Section A.7 (Appendix).", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "* The authors address an important and relevant problem: how to induce relatively high unstructured and semi-structured sparsity (>70%) into LLMs both cheaply and effectively (in terms of accuracy w.r.t. dense model).\n* The paper is very well-written and the core ideas have been presented clearly in an easy-to-understand manner.\n* The proposed formulation permits the systematic exploration of global and local (layer-wise) sparsity and everything in between. This is quite powerful.\n* The benchmarks are fairly strong, using modern medium-scale language models and comparisons to SoTA approaches like SparseGPT/Wanda; however, performance is a bit lacking (see comments below).", "weaknesses": "* The proposed approach appears to obtain accuracy figures on par with SparseGPT at 70% unstructured sparsity. In higher sparsity regimes, SparseLLM seems to outperform SparseGPT and Wanda; however, most of the perplexity values reported for these regimes by all approaches (including SparseLLM) are extremely high; I'd argue that the obtained zero-shot models are pretty much unusable in the real world. I'm not sure I understand why saving pruning+retraining time is important in this regime - wouldn't additional training to close this gap make more sense? If so, SparseLLM could potentially be a good initialization for such an approach.\n* Why are there no experiments performed for 2:4 sparsity, especially since results for 3:4 are reported? 2:4 is particularly relevant since hardware acceleration for this sparsity pattern is possible with today's GPUs. Does SparseLLM outperform SparseGPT/Wanda in this case?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer snMo,\n\nWe sincerely appreciate that you found our paper and method interesting with solid results. Please refer to our response below for details:\n\n>  *\"The proposed approach appears to obtain accuracy figures on par with SparseGPT at 70% unstructured sparsity. In higher sparsity regimes, SparseLLM seems to outperform SparseGPT and Wanda; however, most of the perplexity values reported for these regimes by all approaches (including SparseLLM) are extremely high; I'd argue that the obtained zero-shot models are pretty much unusable in the real world. I'm not sure I understand why saving ``pruning and retraining\" time is important in this regime - wouldn't additional training to close this gap make more sense? If so, SparseLLM could potentially be a good initialization for such an approach.\"*\n\n**A1.**\n*SparseLLM* achieves an accuracy improvement over SparseGPT by an average of 1-3% on zero-shot tasks. For example, at 70% sparsity with LLaMA-2 7b on the WinoGrande dataset, *SparseLLM* outperforms SparseGPT by 2.3 percentage points (61.39 vs. 59.04), and at 70% sparsity with LLaMA-2 13b on the RTE dataset, *SparseLLM* shows a 3.25 percentage point improvement (61.73 vs. 58.48).\n\nIn higher sparsity regimes, for example, 80% sparsity, the perplexity of *SparseLLM* on datasets WT2 and C4 is practically useful. For instance, *SparseLLM* achieves perplexity values of 15.61 and 16.61 on WT2 and C4 for OPT-30b at 80% sparsity. Additionally, *SparseLLM* achieves perplexity values of 16.45 and 17.70 on WT2 and C4 for OPT-66b at 80% sparsity. The perplexity of pruned model by *SparseLLM* is close to that of some smaller dense models, and just needs a little push.\n\n*SparseLLM* could provide a better initialization for the \"prune and re-train\" approach, potentially enhancing performance and closing the gap caused by pruning. Re-training methods such as [1] *Zhang, Yuxin, et al. \"Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs.\" ICLR 2024* could complement our *SparseLLM*. We will explore such approaches and discuss this topic in the limitations and future work section of our paper.\n\n> *\"Why are there no experiments performed for 2:4 sparsity, especially since results for 3:4 are reported? 2:4 is particularly relevant since hardware acceleration for this sparsity pattern is possible with today's GPUs. Does SparseLLM outperform SparseGPT/Wanda in this case?\"*\n\n**A2.**\nWe have added the results for 2:4 sparsity in Table 5 in our one-page PDF, where *SparseLLM* can consistently beat the comparison methods, which demonstrates the potential of *SparseLLM* to achieve practical GPU acceleration.\n\n> *\"To clarify, does 3:4 mean that 3 out of 4 consecutive elements are zero? N:M sparsity is traditionally defined differently: at most N out of M consecutive elements are non-zero.\"*\n\n**A3.**\nIn our paper, $N$:$M$ means every group of consecutive $M$ values contains at least $N$ zeros, which we follow the definition from [2] *Mishra, Asit, et al. \"Accelerating sparse deep neural networks.\" arXiv preprint arXiv:2104.08378 (2021).*\n\n> *\"Can SparseLLM handle alternative activation functions such as GeLU?\"*\n\n**A4.**\nYes, *SparseLLM* can handle alternative activation functions such as GeLU in a similar way to how it handles SiLU. Both activation functions are element-wise operators, meaning each output position is calculated independently. By following the approach used for the SiLU activation function in LLaMA (Lines 233-238), leveraging a pre-computed look-up table, one can obtain analytical solutions for each subproblem.\n\n> *\"Can SparseLLM handle networks with residual connections between layers?\"*\n\n**A5.**\nYes, *SparseLLM* can handle architectures with residual connections. An intuitive way to explain this is by referring to Figure 2 in our original manuscript. Specifically, in the sub-figure of *SparseLLM* on the LLaMA layer, residual connections can be regarded as a special case of the bottom half of the FFN module in the LLaMA layer. In this case, the \"up proj\" layer is replaced by a trivial identity mapping, and the dot-product aggregation (black round dot) is replaced by summation. Since the \"up proj\" layer is replaced by a non-parametric function, the auxiliary variable $z_{\\ell}$ can be discarded.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u52U1lNWEZ", "reviewer": "Reviewer_35o1", "summary": "In this paper, the author proposes a method to globally prune large language models (LLMs) without consuming significant memory. By using auxiliary variables, the LLM can be pruned separately while maintaining dependencies. The evaluation demonstrates that the proposed method outperforms previous approaches in terms of perplexity and accuracy across various sparsity settings.", "questions": "1. What model sizes benefit more from the proposed pruning method?\n2. Can this method be extended to structured pruning, which may be more useful?", "limitations": "Please refer to the weaknesses section.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is promising and innovative.\n2. The paper is well-written and easy to follow.\n3. The experiments are comprehensive, with detailed settings provided.", "weaknesses": "1. The use cases appear tricky regarding model size. For smaller models (<7B), they can fit into GPU memory (A100), allowing global pruning. For larger models (>70B), achieving 90% sparsity does not outperform the smaller versions (7B).\n2. The work only considers unstructured pruning. Unstructured sparsity may not accurately reflect the actual model size and inference time.\n3. There is no discussion on memory consumption across various model sizes and pruning methods, which may undermine the claim that global pruning is infeasible for LLMs due to memory constraints.\n4. Although it outperforms previous methods, the performance at 90% sparsity drops significantly, reducing its practical usefulness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 35o1,\n\nWe are grateful for your recognition of the novelty of our method. Please find our detailed response below:\n\n> *\"The use cases appear tricky regarding model size. For smaller models (<7B), they can fit into GPU memory (A100), allowing global pruning. For larger models (>70B), achieving 90% sparsity does not outperform the smaller versions (7B).\"*\n\n**A1.**\nIn this work, our major goal is to improve local pruning methods for LLMs under **resource-constrained** environments, where high-end GPUs such as A100 are typically unavailable. Representative examples include academic labs and hospitals, edge devices, and mobile computing, which are very prevalent in reality.\n\nGlobally pruning LLMs with sizes greater than or equal to 7B under resource constraints is typically infeasible and might require distributed training. The major contribution of *SparseLLM* is the systematic exploration of global and local (layer-wise) pruning and everything in between using a theoretically sound technique. This allows us to decompose global pruning into smaller and decoupled subproblems that can be seamlessly combined with distributed and resource-efficient training.\n\nMoreover, *SparseLLM* can provide larger pruned models that outperform smaller dense models. For instance, we tested *SparseLLM* with 40% sparsity on Llama-2-13B and achieved a perplexity of **5.09** on WT2 and **7.01** on C4, which are **lower** than dense Llama-2-7B (5.47 on WT2 and 7.26 on C4). This shows that our approach is getting close to Pareto optimality and just needs a little push.\n\n> *\"The work only considers unstructured pruning. Unstructured sparsity may not accurately reflect the actual model size and inference time.\"*\n\n**A2.**\nIn this work, we considered not only unstructured pruning but also $N$:$M$ sparsity, or **semi-structured** pruning. The $N$:$M$ sparsity pruning requires that every $M$ consecutive parameters have at least $N$ zero elements. This can leverage NVIDIA’s sparse tensor cores to accelerate matrix multiplication in practice [1] *Asit Mishra, et al. \"Accelerating sparse deep neural networks.\" arXiv preprint arXiv:2104.08378, 2021*. As shown in Table 1 (of the original manuscript), *SparseLLM* achieves competitive performance with 3:4 sparsity pruning in most cases, indicating its potential for actual GPU acceleration.\n\n> *\"There is no discussion on memory consumption across various model sizes and pruning methods, which may undermine the claim that global pruning is infeasible for LLMs due to memory constraints.\"*\n\n**A3.**\nWe provide the memory cost analysis of global pruning as below:\n\nFor zero-order pruning methods, such as magnitude pruning, the memory cost is relatively low because there is no need for forward or backward propagation. The weights are pruned based solely on their absolute values, resulting in a memory complexity of $O(N)$, where $N$ is the size of the LLM model.\n\nFor first-order methods that use derivatives to estimate the score of each parameter for pruning, an end-to-end forward and backward propagation is required to estimate the pruning mask. Additionally, adjusting the unpruned weights also necessitates another end-to-end propagation. The memory complexity for such methods includes storing the parameters, optimizer states, activations, gradients, and pruning mask, leading to a total memory cost (in GB) of $5$-$10\\times$ model size, which is already prohibitive for most LLMs on a normal GPU.\n\nFor second-order methods that use the Hessian, which are the most effective and commonly-used pruning methods, the memory cost is significantly higher due to the need to store and compute second-order derivatives. This results in a memory complexity of $O(N^2)$.\n\nIn conclusion, global pruning with first or second-order methods is extremely memory expensive, making them impractical for large-scale LLMs. Our proposed method, *SparseLLM*, is very useful as it decomposes the global pruning problem into manageable subproblems, significantly reducing the memory requirements and making it feasible for resource-constrained environments.\n\n> *\"Although it outperforms previous methods, the performance at 90\\% sparsity drops significantly, reducing its practical usefulness.\"*\n\n**A4.**\nThe 90% sparsity level is just one of the sparsities we have shown in our experiments. Despite the significant challenge, the performance improvement of our method over baselines at 90% sparsity is meaningful, demonstrating that *SparseLLM* can achieve better perplexity than SparseGPT and Wanda can even at extremely high sparsity levels, which is a non-trivial achievement.\n\nWe believe the significant performance improvement of *SparseLLM* remains useful, as it can provide a better initialization for the \"prune and re-train\" approach. Re-training methods such as [2] *Zhang, Yuxin, et al. \"Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs.\" ICLR 2024* could complement our *SparseLLM*, potentially close the gap and lead to more practically useful models.\n\n> *\"What model sizes benefit more from the proposed pruning method?\"*\n\n**A5.**\nWe empirically proved that *SparseLLM* can achieve competitive performance over comparison methods over various model sizes, from 125 million up to 66 billion. The performance gap between *SparseLLM* and other methods at a given sparsity will in general decrease as the model size increases, which, however, is due to that larger models are more over-parameterized and easier to prune. This monotonic decreasing pattern is true for all pruning methods including SparseGPT and Wanda.\n\n> *\"Can this method be extended to structured pruning, which may be more useful?\"*\n\n**A6.**\nAlthough in this work we mainly focus on unstructured and $N$:$M$ sparsity pruning, we believe extending our *SparseLLM* further to structured pruning is potentially feasible and interesting to explore. The high-level idea of *SparseLLM* is generic and not restricted to specific model pruning algorithms.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cF700YrU6c", "reviewer": "Reviewer_K3H3", "summary": "This paper tries to improve the pruning technique for LLM to enhance computational and memory efficiency. The proposed SparseLLM, circumvents the scalability problem of global pruning and suboptimal performance due to local pruning. It breaks down global pruning into subproblems", "questions": "1. Regarding the global pruning, what is it necessary to fit the models within one GPU? Why not applying tensor parallel?\n2. What is the comparison of training loss convergence between SparseLLM and other baseline methods?", "limitations": "The authors address the limitations and there is no potential negative societal impact of this work.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Pruning LLM remains to be an interesting problem given the size of LLM keeps growing and serving these models on less powerful devices are worth researching.\n2. The idea of decomposing the global pruning into smaller subproblems and conceptualizing LLM as several modular functions are practical.", "weaknesses": "1. It appears the pruning procedure differs from model to model. If the model architecture changes, the pruning needs to be adjusted. Therefore, I have doubts on the generality of the proposed solution. \n2. The costs of pruning are not specified in the evaluation. For example , in Table 2, it seems for most of the tasks, SparseLLM has a comparable performance as SparseGPT. In this case, what is the advantage of SparseLLM?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer K3H3,\n\nThank you for finding our method interesting and practically useful. Please refer to our response below for details:\n\n> *\"It appears the pruning procedure differs from model to model. If the model architecture changes, the pruning needs to be adjusted. Therefore, I have doubts about the generality of the proposed solution.\"*\n\n**A1.**\nThe pruning procedure such as the closed-form solutions for each subproblem in our *SparseLLM* could be different from model to model, but the generality of our method roots in the mathematical formulation of our Eq. 3 and is guaranteed theoretically. More specifically, as long as the neural network architecture satisfies the Directed Acyclic Graph (DAG), which is, in general, the case for LLMs, our Eq. 3 of *SpasreLLM* can handle that.\n\nThat being said, we have discussed this weakness in the limitation section of our manuscript. We are happy to include more discussion there and explore extending *SparseLLM* onto more diverse model architectures in the future work.\n\n\n> *\"The costs of pruning are not specified in the evaluation. For example, in Table 2, it seems for most of the tasks, SparseLLM has a comparable performance to SparseGPT. In this case, what is the advantage of SparseLLM?\"*\n\n**A2.**\n*SparseLLM* consistently achieves competitive results, significantly decreasing perplexity by up to 80% compared to SparseGPT and Wanda. Notable improvements include OPT-2.7b at 90% sparsity: *SparseLLM* achieves a perplexity of 759.11 versus SparseGPT's 2285.01 for PTB, and 527.70 versus 1776.08 for C4, representing over **60%** improvements in both cases. For OPT-125m at 80% sparsity for C4, *SparseLLM* achieves a perplexity of 654.54 versus SparseGPT's 1050.83, representing over **40%** improvement.\n\n*SparseLLM* is a generic framework, that both local pruning and global pruning are special cases. By flexibly switching between these extremes, the computational complexity of *SparseLLM* can be the same as local pruning.\n\n> *\"Regarding the global pruning, what is it necessary to fit the models within one GPU? Why not apply tensor parallel?\"*\n\n**A3.**\nIn this work, we consider pruning methods for LLMs under resource-constrained environments, where it is likely that only one GPU is available. Representative examples include academic labs and hospitals, edge devices, and mobile computing, which are prevalent in reality. \n\nBy utilizing more computational resources (e.g., bigger and more powerful GPUs, distributed training including tensor parallel, pipeline parallel, etc) one can to some extent achieve the vanilla or extreme global pruning of LLMs, but that goes beyond the main focus of this work. However, we agree how to achieve the extreme global pruning from a high-performance computing perspective is definitely a challenging but interesting future direction on the basis of our manuscript.\n\n> *\"What is the comparison of training loss convergence between SparseLLM and other baseline methods?\"*\n\n**A4.**\nSince baseline methods such as SparseGPT and Wanda both consider one-shot pruning, there is no convergence curve for those methods. However, it is still possible to compare the training loss of *SparseLLM* over SparseGPT and Wanda. For example, under the setting of pruning layer 3 of OPT-125M at 80% sparsity, the training loss of our *SparseLLM* is **0.3** after 2 epochs and is **0.15** when convergence is achieved. On the contrary, the training loss of SparseGPT is **0.8**, after the one-shot pruning. Hence, *SparseLLM* can further reduce the training loss of one-shot pruning methods such as SparseGPT and Wanda via an iterative algorithm.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We sincerely thank all your professional and constructive comments, especially given the time and workload for this year's NeurIPS. We have provided detailed responses to each individual comment and hope we have addressed all your concerns. Below is a brief summary of the new experimental results added during the rebuttal period:

- Results for LlaMA-3 8B
- Low sparsity regime results: 10%, 20%, and 50% sparsity for all pruning methods
- Ablation study on the hyperparameters $\alpha$ and $\beta$
- 2:4 sparsity results

**All newly added results have been compiled into our one-page PDF, which we invite you to review for further details.**

Sincerely,  
The Authors

Meta-review: The paper introduces SparseLLM, a framework designed to globally prune large language models (LLMs) by decomposing the pruning process into manageable subproblems. This approach allows for resource-efficient optimization and demonstrates significant performance improvements, particularly in high-sparsity regimes where it surpasses current state-of-the-art methods. SparseLLM conceptualizes LLMs as a chain of modular functions and leverages auxiliary variables for problem decomposition, facilitating practical application on LLMs. 

Reviewers agree the proposed method is noval, promising and theoretical grounding.  However, they also raised concerns including model choice in experiments, lack of lower sparsity level reults, and missing dense baselines.

The authors address reviewer concerns by providing additional experimental results, including those for the latest LlaMA-3 model, lower sparsity regime results, and ablation studies on hyperparameters. The rebuttals are structured, detailed and addressed most of the concerns.

All the reviewers give the same 5 ratings (weak accept).
Decision: Accept (poster)