ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models
Authors: Xiang Meng
Kayhan Behdin
Haoyue Wang
Rahul Mazumder
Venue: NeurIPS 2024 poster (2024)
Keywords: network pruning
unstructured pruning
large language model
ADMM
pre-conditioned conjugate gradient
optimization
Primary area: optimization

Abstract: The impressive performance of Large Language Models (LLMs) across various natural language processing tasks comes at the cost of vast computational resources and storage requirements. One-shot pruning techniques offer a way to alleviate these burdens by removing redundant weights without the need for retraining. Yet, the massive scale of LLMs often forces current pruning approaches to rely on heuristics instead of optimization-based techniques, potentially resulting in suboptimal compression. In this paper, we introduce ALPS, an optimization-based framework that tackles the pruning problem using the operator splitting technique and a preconditioned conjugate gradient-based post-processing step. Our approach incorporates novel techniques to accelerate and theoretically guarantee convergence while leveraging vectorization and GPU parallelism for efficiency. ALPS substantially outperforms state-of-the-art methods in terms of the pruning objective and perplexity reduction, particularly for highly sparse models. On the LLaMA3-8B model with 70\% sparsity, ALPS achieves a 29\% reduction in test perplexity on the WikiText dataset and a 8\% improvement in zero-shot benchmark performance compared to existing methods. Our code is available at https://github.com/mazumder-lab/ALPS.

ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models
Xiang Meng
Kayhan Behdin
Operations Research Center Massachusetts Institute of Technology mengxOmit.eed
Operations Research Center
Massachusetts Institute of Technology behdinkSmit edu
Haoyue Wang
Rahul Mazumder
Operations Research Center Massachusetts Institute of Technology haoyuew@mit. edu
Operations Research Center Massachusetts Institute of Technology rahulmaz@mit edu
Abstract
The impressive performance of Large Language Models (LLMs) across various natural language processing tasks comes at the cost of vast computational resources and storage requirements. One-shot pruning techniques offer way alleviate these burdens by removing redundant weights without the need for retraining. Yet, the massive scale of LLMs often forces current pruning approaches rely heuristics instead of peiimization-based techniques, potentially resulting in suboptimal compression. In this paper, we introduce ALPS, an optimization- based framework that tackles the pruning problem using the operator splitting technique and a preconditioned conjugate gradient-base post-processing step. Our approach incorporates novel techniques accelerate and theoretically guarantee convergence while leveraging vectorization and GPU parallelism for efficiency. ALPS outperforms saateoo-the-ane methods in terms of the pruning objective and perplexity reduction, particularly for highly sparse models. On the LLaMA3-8B model with 70% sparsity, ALPS achieves 29% reduction in test perplexity the WikiText dataset and a 8% improvement in zero-shot benchmark performance compared existing methods. Our code is available at htttp:://gl httpt/ttttttutttt/////// com/
mazumder- - lab/ALPS
Introduction
Large Language Models (LLMM)) have revolutionized the field natural language processing, demon- strating remarkable performance across wide spectrum of tasks, from question answering and text
generation to sentiment analysis and named entity recognition (Wei et al. 2022 Bubeck et al.] 2023) Achiam al. 2023 The success of LLMs in part be attributed to their massive scale -fateeof- the-art models like OPT-175B [Zhang et al. 2022a] and LLaMA3 (Dubey et al. 2024 have hundreds steep
of billions of parameters. However, this enormous size computational resources. For instance, the OPT-175B model requires at least 320 GB of memory to store its parameters in half- balf-prrcision (FP16) format, necessitating the use of multiple high-end GPUs for inference [Frantar and Alistarh 2023). To make LLMs more accessible and efficient, considerable efforts have been made to compress these models, with particular emphasis on model quantization
in terms of storage and
techniques [Lin et al. 2023 Behdin et al.
2023 Dettmers et al. 2023
Network pruning (LeCun al. 1989 Hassibi and Stork 1992 Han al. 2015 complementary approach to quantization, has received comparatively less attention in the realm of LLMs. Pruning
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

aims to reduce the model size by identifying and removing redundant or less important weights, resulting sparser and more efficient network. Traditional pruning methods rely on iterative retraining to recover accuracy after each pruning stage [Han et al. 2015 Luo et al. 2017 Molchanov al. 2016 Liu et al., 2018], which can be computationally expensive and cime-consuming. To address this, recent research has focused on one-shot pruning methods [He et al. 2017, Singh and Alistarh 2020] that compress a pre-trained model using only a small amount of data (c.g., a few thousand samples)-the key idea here is to perform pruning while retaining model accuracy as much as possible without expensive fine-tunigg/retrainigg on the entire dataset. Many prior works [Frantar and Alistarh 2022 Yu al., 2022 Benbaki et al. 2023] on one-shot pruning address such pruning-accuracy tradeofts using optimization based approaches.
Despite the progress made in one-shot pruning, the massive scale of LLMs poses additional challenges, as many one-shot pruning methods designed for vision models cannot be directly applied due to their large model sizes. To overcome this, existing LLM pruning methods often rely on heuristic approaches to prune instead of solving optimization problems. For instance, SparseGPT [Frantar and Alistarh 2023) approximates the OBS [Hassibi and Stork [192]] algorithm by employing partial weight updates and adaptive mask selection to reduce costly Hessian computation. Similarly, Wanda [Sun al. 2023] prunes weights based on the product of their magnitudes and corresponding input activations. Zhang et al. 2023) propose to iteratively grow and prune the weight mask according to the change reconstruction error achieved by each update. While these heuristics enable pruning at scale, they may lead to suboptimal compression (and hence, suboptimal compression-accuuns aacuuaacy tradeoffs) compared to advanced optimization-base upproaches,
show in this paper.
In this paper, we propose ALPST] an optimization- based framework for one-shot LLM pruning. ALPS consists of two key components. First, it formulates pruning LLMs as an fg-constrained optimization problem and solves it directly using the operator splitting technique (i.e., ADMM) [oydd et al.] 2011 Davis and Yin 2016) without any simplification. The proposed algorithm simultaneously finds the support of the weights and updates them. After the support stabilizes, ALPS fixes the support and employs preconditioned conjugate gradient (PCG) [Nocedal and Wright 1999. Section 5] to compute the optimal weights on the support. Our modified PCG leverages sparse matrix structure (arising from pruning) and GPU computation to solve large systems efficiently, providing significant speed advantage over direct matrix inversion. An outline of our proposed optimization- based framework ALPS is given in Figure Compared to previous heuristics, ALPS offers higher quality supports and weights, as demonstrated in Section 4.1 This improvement translates to a better performance of the pruned model compared to existing methods, particularly in the challenging high-sparsity regime.
Nout
ADMM
PCG
X
X
W
S: support of W (grny cells)
Optimal weight
Figure 1: Overview of the proposed ALPS algorithm. (Left) The pruming problem with a layerwise reconstruc- tion objective and an fu constraint on the weights (Section B.J. (Middle) ADMM with a p-update scheme (Algorithm (1) is employed to determine high- support for the weight matrix W (Section 3.2 (Right) The optimization problem is restricted the obtained support, and modified PCG method (Algorithm) 2 used to solve for the optimal weight values within the support (Section 3.3).
Contributions. Our technical contributions are:
1. We introduce ALPS, a novel one-shot LLM pruning framework that formulates an &g fo-constrained optimization problem with layer- wise reconstruction objective. By extending the operator split- ting technique (i.e., ADMM) to this non-convex, non-continuous problem, ALPS simultaneously
'DMM-based L.M Pruning in one-Shot
The "support" of the weights refers the set of indices corresponding to non-zero weights within layer.

---

finds a high-quality support and updates the weights on the support. This approach leads to improvements over oftte-of-the-rrr art heuristics in terms of the pruning objective. Furthermore, we provide theoretical convergence guarantees for our proposed algorithm, which, to the best of our knowledge, is a novel convergence result for fn-constrained problems.
2. We further enhance the performance of ALPS through two techniques. First, we design novel penalty parameter updating scheme that enables ALPS to find better support and accelerates its convergence. Second, we propose a post-processing technique to further improve the performance of the pruned models-We we fix the support determined by ADMM and optimally solve the resulting quadratic problem using the PCG method. We utilize vectorization to solve the problem in single pass and leverage GPU parallelism to further accelerate PCG. Our proposed method achieves a 20x-200x speedup compared to the vanilla backsolve approach.
3. ALPS substantially improves upon statc-of-the-an art methods for one-shot unstructured pruning of LLMs. For the LLaMA3-8B model with 70% sparsity, ALPS achieves 29% reduction in test perplexity on the WikiTeet dataset and a 4%-13%% improvement in performance on zero-shot benchmark evaluations. We also adapt ALPS to the popular N:M sparsity format (Zhou et al., 2021] and observe 30-10% higher performance compared to existing methods. Our code is publicly available at: https:/ ///ggtttubbb... cow/mazunder- lab/ALPS
2 Related Work
Network pruning is a well-established technique for reducing the complexity of deep neural networks by removing redundant weights (LeCun et al. 1989 Han et al. 2015]. Pruning methods can be classified based on the structure of the resulting sparse network and the training requirements. In terms of structure, pruning can be categorized into unstructured pruning, which removes individual weights (Han et al.) 2015 Guo et al.] 2016], and structured pruning, which removes entire structures such as channels, filters, or attention heads (Lebedev and Lempitsky 2016 Wen et al., 2016 Voita et al., [019 El Halabi et al., 2022]. Unstructured pruning offers better flexibility and higher sparsity levels but requires specialized hardware for acceleration, while structured pruning is more hardware- -wwerrrffiedly but may suffer from larger performance loss. Based on the training requirements, pruning methods can be classified into three categories: (i) one-shot pruning, which directly removes weights from a pre-trained model without further training [Gale ct al. 2019 Frantar and Alistarh 2022 Meng et al. 2024a b). (ii) gradual pruning, which begins with pre-trained model but alternates between pruning and fine-tuning via SGD to recover performance [Molchanvv et al., 2016,] Zhu and Guptal 2017 Blalock et al. 2020 Kurtic al. [022], and (iii) training from scratch, where the model is trained from randomly initialized weights, and the sparse network structure is either determined before training or evolves during the training process, [Mocanu
Network pruning.
et al. 2018 Dettmers and Zettlemoyer| 2019 Evci et al. 2020, Kusupati et al. 2020 Chen et al.
2021
In this paper, we focus on one-shot unstructured pruning.
Post-training unstructured pruning. Based on their pruning objectives, there are three types of methods, which assign a score
post-training unstructured pruning methods: (i) Importance-
to each weight (e.g.. its absolute value) to assess its significance and decide whether it should be eliminated (Han et al. 2015 Lee et al., 2018 Molchanov et al., 2019 Sun et al.) 2023). (ii) Second-order techniques, which consider a local quadratic approximation of the loss function around the pre-trained model and remove weights based on their influence on the loss (Hassibi and Stork 1992, Singh and Alistarh 2020 Yu et al. 2022 Benbaki et al. 2023]. These approaches employ the empirical Fisher information matrix to estimate the Hessian matrix efficiently. (iii) Layer-wise pruning algorithms, which adapt OBS (Hassibi and Stork 1992) framework to the layer-wise reconstruction
objective (Dong et al., 2017 Frantar and Alistarh| 2022 2023]. These methods prune each layer separately to address the computational challenge of calculating the full Hessian required in OBS. This work considers layer-wise reconstruction error as the pruning objective.
Unstructured pruning in LLMs. While pruning algorithms designed for convolutional networks (Singh and Alistarh [2200] Chen et al.) 2020 Frantar and Alistarh| [022] can be readily adapted to moderate-sized language models like BERT Vaswann et | 2017] pruning LLMs with billions of parameters presents distinct challenges. The immense model size and extensive datasets associ- ated with LLMs render traditional pruning methods computationally infeasible [Ma et al. 2023). SparseGPT [Frantar and Alistarh| 2023) utilizes partial weight updates and adaptive mask selection to mitigate the expensive Hessian computation, while Wanda (Sun et al., 2023] directly obtains
3

---

sparse LLM model using a criterion that considers the product of the absolute values of weights and their activations. DSnoT [Zhang et al. 2023] iteratively grow and prune the weight mask according to the change in reconstruction error achieved by each update. [Boža] 2024) introduces an efficient approach to determine the optimal weights on a given support and extends this technique to develop
heuristics for updating the support.
ADMM in network pruning, The operator-slliting technique [Boyd et al. 2011 Davis and Yin 2016 (also known as Alternating Direction Method of Multipliers, ADMM is a well-known
approach for solving composite optimization or optimization problems with coupled variables (or constraints), and has been used earlier in network pruning. Ye et al. [2018) applied ADMM to solve the original loss function under sparsity constraint, and Boža 2024 used ADMM to solve convex pruning problem with fixed support. Moreover, (Zhang et al. 2018 employed ADMM to train deep neural networks under sparsity constraints, while Ye et al. [2019] utilized ADMM to perform concurrent adversarial training and weight pruning. Our proposed method differs significantly from previous methods two key aspects: (i) ALPS solves the pruning problem with an &n constraint at LLM scale, simultancously optimizes over the weights and the sparsity pattern. (ii) We introduce novel penalty parameter update scheme that ensures convergence both practically and theoretically.
3 ALPS: Effective LLM pruning in One-shot
3.1 Problem formulation
common approach in post-training unstructured pruming of LLMs is to decompose the full-model compression problem into layer-wise subproblems. The quality of the solution for each subproblem is assessed by measuring the &g error between the output of the dense layer and that of the pruned one, given set of input activations.
Formally, let W X.o denote the (dense) weight matrix of layer l, where Ni, and Nout denote the input and output dimension of the layer, respectively. Given a set of calibration samples, the input activations can be represented X E Rx.L where L. is the sequence length. The goal of pruning is to find a sparse weight matrix W that minimizes the reconstruction error between the original and pruned layer outputs, while satisfying a target sparsity constraint. addition, we add a ridge term that penalizes the distance between W and W, preventing W from diverging too far from the original weights. This layer-wis, wise pruning problem can be formulated co-osstradeeed optimization problem:
millweR Xooe
|WW - X + Ag
W W
s.t.
|| S k,
(I)
where A2 2 0 and || llo denotes the fo-(pseudo)norm, which counts the number of non-zero elements.
3.2 Operator-plliting for layer-wise pruning
Optimization of Problem (T) is quite challenging: we need to simultaneously find a support of W and a corresponding set of optimal weights (that minimize the objective restricted to the support). Notably, W may contain over 100 million parameters in the LLM setting, making (I) even more computationally demanding. To address this, we employ an operator-splitting technique (Boyd) et al. 201T Davis and Yin 2016] (also known as ADMM), which decomposes the problem into two computationally 'friendlier' subproblems. Specifically, we reformulate problem (T) by introducing a copy D of weight matrix W:
minw DER Nour
XW XWI + A W W + 00 1 s.t.
W- = D, (2)
where the penalty function "oo- IJD]|o>>* imposes the to constraint |Di|o < k by assigning a value of zero when this condition is met and infinity otherwise. This reformulation separates the objective function into two independent parts while coupling the variables W and D through the linear constraint W
D. We consider the augmented Lagrangian function of this problem:
(W,,D XW X-
1 + W WI
11DD + +(,, W W DD+ L -
B-W-DD D k.

---

where > 0 is the quadratic penalty parameter. We minimize the augmented Lagrangian with respect W and D alternatively, followed by a dual update. We get the following update at iteration t:
w(++)) = arg minw p (W, v(0) (H +
(HW -V( + pD(1).
D(++)) arg minp Lp( (W D, v Pse (w(++) + /p).
v(+1)= = v + p(W((+)) (((1)),
where H = x + XzI. Here, the W-update aims to minimize the objective by solving a system of equations, while the D update enforces sparsity by using the projection operator (t((), which projects an input matrix onto the set of matrices with at most k non-zero elements. The dual update on matrix V ensures consistency between W and D. As the iterations 4 progress, our proposed method concurrently identifies the support of the weight matrix and updates the weights on the determined support.
update scheme. In practice, we observe that the sequence of updates 4 and the resulting solution can be sensitive to the choice of the penalty parameter p. small p leads slow convergence due to large changes in the support of D across iterations, while a large may compromise solution quality though the support stabilizes early on. To balance support quality and convergence speed, we introduce a novel penalty parameter update scheme. Starting with small p. we gradually increase it every few iterations, with the increase rate proportional to the change in the support of D. The detailed update scheme is provided in Appendix B.1 This scheme allows our algorithm to explore and find a good support when p is small and to converge rapidly as p grows, as demonstrated experimentally in Appendix 1.2.1
Algorithm D outlines the proposed operator-splitinng technique with the p update scheme. The convergence of Algorithm is guaranteed by the following theorem, with its proof provided in Appendix We note that existing convergence results for operator-spliting type methods (e.g..
ADMM) focus on convex or continuous problems [Hong et al. 2016 Wang et al. 2019]. However, our result guarantees the convergence on non-convex, non- continunuuuus Ey oo-conttraieed problem. which, to the best of our knowledge, is a novel convergence result for such a problem.
} Theorem 1. Let (D()) 1= and {w(") Jt=o be the sequences generated in Algorithm ! Suppose the penalty parameter 1oc chosen in Algorithm 1 [saiisiiss E 1/p, oc. It then holds
max
|D - D | C//p,.
[1] - [1]] s
(5)
where C is a constant depending on X, W, X2 and Ex 1/pt. In particular, there exists a matrix D such that D + D and W!!) + D as t + xc.
Algorithm 1 ADMM for layer-wise pruning with by constraint
Input: Initial penalty Po.
1: Initialize (0) = ON,, Now. and D (o) W = W
2: for L 0, 1.... do
3: Update (++1), ((++1) and (++1) according to with p= Pe.
4: Increase to get P++1 based on the change in the support of Supp (D(1).
5: end for
Computational cost. The primary computational cost of Algorithm I arises from the W- W-update step in display (4. which involves solving system of linear equations. In the update, the inverse of H + pI can be reused across iterations and needs to be updated when changes. To avoid re-computing the inverse, store the eigenvalue decomposition H QMQ For varying p values, the inverse can be efficiently calculated as (H + pl)
Q(M + pl) 1 - requiring only
single matrix-matre multiplication. Additionally, the term HW in W-update remains constant across iterations and can be pre- coepute and stored. Thus, each iteration of update 44) requires at most two matrix-matrix multiplications, leading a time complexity of ONN Nout).
Extension other sparsity patterns. Algorithm can extended to support M sparsity Zhou et al. 202 Hubara et al. 2021). pattern in which neural network has at most N non-zero
5

---

weights in each group of M consecutive weights. This sparsity pattern enables inference time acceleration on specialized hardware like NVIDIA A100 GPUs. To accommodate N M sparsity, we modify the D-update step in (4) by replacing the projection operator Pa(-) with projection onto the set of matrices satisfying N M sparsity. This modification can be easily implemented by
applying magnitude pruning [Zhou et al. 2021] W + v(t) Our approach can be further generalized to handle other structured sparsity patterns, such as block sparsity [Gray et al., 2017] and row sparsity [Meng et al., 2024b). Similar to the N M sparsity adaptation, this is achieved by modifying the D- Dpddatee step to project onto the set of matrices satisfying the desired sparsity pattern. Importantly, our convergence results and the refining procedure introduced in Section 3.3 remain applicable to these various sparsity patterns.
3.3 Efficiently refining weights after support stabilization
Our proposed p-update technique enables Algorithm to search for a high-quality support when is small, and the support stabilizes quickly as p increases. However, once the support stabilizes, the convergence rate of Algorithm becomes slow in practice. To accelerate the optimization process on the support, we employ a Preconditioned Conjugate Gradient (PCG) [Nocedal and Wright 1999 Section 5] method with GPU parallelism and vectorization for efficient computation.
Formally, we introduce a post-processing technique that fixes the support of the current solution W and refines the solution within this support, leading to the following problem:
mill ER Nen x Mostt
X IXW-
X
Xa W
WI
s.t. Supp(W) S.
(6)
(6) decomposes into separate least squares problems across the columns of W. However, as illustrated in Figure (Middle), the supports of the columns of W are different. Using direct matrix inversion (backsolve) to solve these problems would involve solving Nout linear equations, each requiring the inversion of a submatrix of = X X + A2I. Since the submatrices under consideration vary across different columns, parallelization is not straightforward, and must solve Nout different linear equations, each with size ( LLMs, where Nout and Ni, are of the order 10*, this would result in significant computational expense. We present workaround discussed next.
To efficiently solve problem (6). we propose using the Preconditioned Conjugate Gradient (PCG) method, a high-peerfrmance numerical linear algebra technique for approximately solving systems of equations through repeated matrix-matrin multiplications. We further enhance PCG's performance by introducing two novel acceleration strategies. First, instead of solving for each column of W separately, we solve the entire problem in a single pass by directly solving the linear equation
HW = HW using PCG and projecting W onto the given support S in each iteration (Algorithm line 8). We leverage vectorization to significantly enhance the speed. Second, we perform the matrix-man matrix multiplications involved in PCG on the GPU, further utilizing GPU acceleration to expedite the computations. Algorithm 2 provides the detailed steps for the PCG method, and Figurd] offers an overview of our proposed ALPS method.
Algorithm 2 PCG with vectorization for solving problem (6)
Input: Support S, pre-conditioner M = Diag initial solution Wo
1: Set Rg H(W- -Wo)
2: Project Rg onto the support S by setting all elements outside the support to zero.
3: Set Zy M R, and Po Zz
4: for t = 0, 1.... do
5: Q' = Tr(R, Z (P..HPP
6:
W = + aaP
7:
R = R, R&,,H
8:
Project R,+1 onto the support S by setting all elements outside the support to zero
9:
Z++1 -
10:
if RG+1 is sufficiently small then break end if
11:
12:
B. = TT 1
14:
P == Z:+1 + 3,P,
15: end for

---

4 Experimental Results
This section compares our proposed framework, ALPS, with sate-of-theearr unstructured pruning methods for LLMs. Detailed information on the experimental setup and reproducibility is provided in Appendix B.I while additional results are presented in Appendix B.2
Models and datasets. We evaluate the performance of ALPS on the OPT model family Zhang al. 2022b] with sizes ranging from 1.3 billion to 30 billion parameters, the LLaMA2 model family [Touvron et al.) [2023] with 7 billion and 13 billion parameters, and the LLaMA3 model
[Dubey al., 2024] with 8 billion parameters. Following the approach of Frantar and Alistarh
2023 we use 128 segments of 2048 tokens each, randomly selected from the first shard of the C4 dataset (Raffel et al. 2020], as calibration data. We assess the performance using perplexity and
zero-shot evaluation benchmarks, with perplexity calculated according to the procedure described by HuggingFace [Per 2022]. using full stride. The test sets of raw- auw-WikiText2 [Merity et al. 2017]. PTB [Marcus et al. 1994]. and a subset of the C4 validation data, which are popular benchmarks in LLM pruning literature (Yao et al., 2022 Xiao et al. 2023 Meng et al. 2024b], are used for evaluation. Additionally, we consider five zero-shor tasks: MMLU [Hendrycks al. [02T]. PIQA (Bisk et al.
2020 LAMBADA [Paperno et al. 2016). ARC-Easy and ARC- Challenle (Clark et al. 2018).
Competing methods. We compare ALPS with several one-shot pruning methods for LLMs, including (i) Magnitude Pruning (MP. [Han al. 2015)), (ii) SparseGPT [Frantar and Alistarh 2023). (iii
Wanda Sun et al. (023). and (iv) DSno'T Zhang et al. 2023
4.1 Reconstruction error on single layer
We first evaluate the performance of our pro- posed ALPS framework on single layer. Specif- ically, we prune a linear layer in the OPT-13B model with input and output dimensions of 5120 to various sparsity levels and compute the rel- ative reconstruction error of the pruned weight W using X XWI2 [XW] The re- sults are shown in Figure As demonstrated, ALPS achieves significantly lower reconstruc- tion errors compared to other methods, espe- cially at high sparsity levels. For instance, at sparsity level of 0.8, ALPS yields a 7.6% relative reconstruction error, while SparseGPT shows 12% error, and other methods exceed 20%. As demonstrated in Sections [4. and [43] our method's 's superior ability to approximate the dense model's output at each layer translates to much better performance in the pruned model.
single layer
Spo
Varth
DoviT
AJS
6.
0.2
0.2
0.5
2A
3.0 0.2
Figure 2: Performance analysis of pruning the "self attn.k "_oj" layer in the first block of the OPT-13B model at various sparsity levels. The plot shows the relative reconstruction error of pruned weights, comparing different pruning methods.
We attribute the superior performance of ALPS in solving the reconstruction problem at each layer to two key aspects: (i) Algorithm[] obtains high-quality support by directly opti- mizing for an optimal subset of weights that contribute the most to recovering the dense model's output (ii) The PCG method in Algorithm Pefffciently solves the reconstruction problem on fixed support, further reducing the reconstruction error. To verify these claims, we conducted the following two ablation studies.
Firstly, we compare the quality of the support determined by various pruning methods. For each method, we prune the layer to different sparsity levels and fix the support of the weights matrix provided by the method. We then solve the post-processing problem (6) with this support to optimality and compute the relative reconstruction error of the resulting weights. This approach ensures that the reconstruction error depends solely on the quality of the support. Table I (left) presents the performance of each method. As shown, the support determined by ALPS yields 20% ~ 40% lower reconstruction error compared to other methods, demonstrating its effectiveness in finding high-quality supports.
7

---

Sparsity
MP
SparseGPT
Wanda
DSnoT
ALPS
Sparsity
w/o pp.
ALPS
Backsolive
0.5
1.10e22
9.40ee3
1.17e:2
1.14e-2
7,56e-3
Error
Time(s)
Error
Time(s)
Error
3.18o-2
0.77
1.24e-2
131
1.10o-2
0.6
2.25p-2
1888-2
2.39c-2
2.38e-2
1.47e-2
0.6
6,01e-2
0.79
2,43e-2
95.0
2.25c-2
0.7
4.38e-2
3.60e-2
4.65e-2
4.58c-2
2.78e=2
0.7
1.09c-1
0.78
4,56o-2
64.1
4,38o-2
0.8
8.50e-2
6.95e-2
8.99e-2
9.02e-2
5.32e.2
1.98c-1
0.77
8.63e-2
37.8
8.50e-2
0.9
1.78c-1
1.47e-1
1.87e11
2.01e-1
1.13e-1
3.66e-1
0.76
1,78c-1
15.0
1,78c-1
Table 1: Performance analysis of pruning the "self_attn.k __oji layer in the first block of the OPT-13B
model at various sparsity levels. (Left) Relative reconstruction error || - X X [|/|||/W / of the optimal weights W constrained to the support determined by each pruning method. (Right) Comparison of time and reconstruction error for three scenarios, all using magnitude pruning to determine the support and then: (i) no post-processing (w/o pp.). (ii) refining the weights with ALPS, and (iii) refining the weights optimally with backsolve.
We then evaluate the effectiveness of proposed post-processing method in finding optimal weights on a given support. We first apply magnitude pruning (MP) to determine the support of (i) without post-processing (w/o pp.). (ii) using 2 to solve (6) to refine the weights on the support (ALPS), and (iii) using PyTorch's ' torch. www.llin.ll....o sooll. solve function to solve (6) to optimality (Backsolve). We assessed both the relative reconstruction error and the computational time for each scenario, with results presented in Table [ (right). The findings demonstrate that the post- post-processing procedure significantly lowers the reconstruction error. Notably, our PCG method achieves errors comparable to the optimal solution but is 20x-200x faster, underscoring the efficiency and effectiveness of our approach.
the weights and then consider three scenarios:
Algorithm
4.2 Pruning OPT and LLaMA models
This section focuses on pruning OPT models and LLaMA models to various sparsity levels and evaluating the performance of the pruned models using perplexity and zero-shot benchmarks. The
Model
Algorichm
WikiTes22
PTB
LAMBADA
PIQA
ARC-Easy 1
ARC ARC.CChallenge 1
MP
940X10)
6689110)
5652(10)
0.000:0000
52.12(2:.00)
26.18000000)
20.9001:0000
Wandu
100.8+3.8)
128.4(331)
9.77(-063) 78.58(+2.34)
59.38(-0.46)
18.0(+00.6) 37.92-0.322
OPT-1.38
SpureGPT
22.02(+2:22)
30.10(13.11)
27.27(11104) 37.05(11.78)
62 62.38(2077)
19.69(10.78) 0..79(10.58)
DSOOT
197.5
370.004:0.66
205 4:166661)
8.2(20023)
56 96.0020)377
(7.4%(:0.57) 43727
ALPS
29.58.12.38)
10.68:11.13)
28.52/ 122)
1.711
64.43(10.37)
1.000(1100))
20.08
MP
12219(_0)
10993(1))
9060(100
0.00
52.9((00000)
26.50(10.00)
19.71(20.00)
Wanda
363 1(+56.5
379 379.4)))7.6)
224 42++4)
5.02(=040))
58.0 38)
34 38)
17.61(+0.28)
OPT-2.77
SpuseGPT
18.93(11.62)
10.8%(=1.11)
23.11(#0.66)
1.)7
66 66.50(00.32)
09.55(20.50)
21.67(=0.57)
DSoOT
(16.2(1677) 1.558(11.5)
9.39(20063) 51)
59 (3(22))))
47
18443(10.78)
ALPS
1.33
35.76(10990)
42.53(1240) 20.75
67.62(10.36)
22.27(10.433)
MP
9970(100
4779(20)
5055(100)
0.00
52.67(20.00)
26.60(20.00)
21.16(20.00)
Wanda
162.9(8877)
10.69(
2880200100 206.0::13.5)
18.13(30.16)
32(200))))
17.22(20.39)
OFT-6.7B
SpuseGPT
21.14(=0.69)
29.34(=0.44)
47,20(+084) (9.00(#062)
69 (3(23))
54.67=0.00 20)
24.08(+0.28)
DSMIT
7965(+465)
6572(+783)
4764(+478)
0.15(-0.1))
53.19(00.36)
29 13(-777)))
18 46(
ALPS
24.89 0..99
26)
17.00(00600
$6.0411332 32)
71.39=0227
0..52
26.19(=0.52)
MP
52455K(=0)
(4
155 (000=))))
0.00x=0.00)
52.99(00.00)
25 (.00000))
22 25(=00000
Wanda 2.)
66 16)50
50.19(12228)
10.8((1099)
63 63.03(10.72)
03.66(10.52)
23.50010.63)
OPT-138
SpiseCiPT
25 25.50(+0.32) 19.29(+0.42)
47.97(+1.75)) (677)0045)
60 (0(33))))
53.51(-0.49)
26223(00.6)
DSiTT
60. 60.11(20911 18:13(22294)
14443:1041 16.
40.52(00.60 61:57()))66)
62)
20.77(20067)
ALPS 21.49(=040) 16.71((0.62)
$7.08/+1.62) (5.07(00.46
71.73/00.22)
59.50(00500
27.97(
MP
26271(=0)
12693(-0)
13057(-0)
0.00==000)
52.23(=0.00)
-0(0000)
19:97(#0.00)
Wanda
19884(1188)
1467(1100)
1992/1191)
0.01(1001)
52221(2027)
25.61(20.10)
20.20(10.41)
OPT-30B
SpansCPPT
1361(20022)
(894(0025)
60,50(1088) 333)
73.74(44.27)
63.26(:0(44)
28:0(()0440)
DSwTT
11328/-368)
5685(1218)
297)
0.010200000
52.48(10.30)
26.36(10.15)
20. 15.0033)))
ALPS
1266((00.2)
17.47(+0.2)
(3.00)0333)
63.52(+138))
78.05(-0.38)
65.52(-0.75)
30.55 00.55+00.76)
Table 2: Performance analysis for one-shot unstructured pruning of OPT models (1.3B 30B) at
70% sparsity. We run each method five times and report the mean and standard deviation of each performance criterion. Here, denotes lower values corresponding to better performance, and denotes higher values corresponding to better performance.
8

---

Perplesity on Wiki lext2 dataset (ower values are preferred)
Accuracy of PIQA dataset (higher values are preferred)
HF
NP
20a
SparsedPT
Spars/CPT
Wandia
Wardd
-d-
DonoT
Dinot
"d- AIPS
- AIPS
100
60-
101
0.5
0.7
95
03
Starsity
Spaniy
Figure 3: Performance analysis for one-shot unstructured pruning of LLL.MA-88 model at various sparsity levels on two datasets: WikiText2 (Left) and PIQA (Right). We run each method five times and plot the shaded region the area between the mean (solid line) and two standard deviations above and below the mean.
performance the pruned LLaMA3-SB model different sparsity levels on the WikiText2 and PIQA datasets is presented in Figure 3 Table 2 showcases the performance of OPT models with 70% sparsity on various datasets, Additional results on different models, sparsity levels, and datasets are provided in Appendix B.2.5
Figure demonstrates that ALPS outperforms other competitors when sparsity levels exceed 50%, and the performance gap between ALPS and other methods widens as the sparsity level increases. For instance, ALPS achieves 60% perplexity reduction the WikiText2 dataset compared to other methods at 80% sparsity level. This observation aligns with our findings in Section 4.1 confirming that ALPS's highly advanced optimization method in solving layer-wise reconstruction problems enables it to better preserve performance at medium-to-highh sparsity levels compared to other methods. Table 2 further validates this fact, showing that ALPS outperforms other methods by large margin across all models on all criteria. This suggests the superiority of ALPS in pruning models medium-to- to-highh sparsity levels.
4.3
N:M sparsity
We further assess ALPS's performance on M sparsity patterns, with Table Bisting the results for pruning OPT-30B and LLaMA2-13B models at 2:4 and 4:8 sparse patterns (see Appendix B.2.5 for other models). ALPS outperforms other methods on most datasets, achieving larger performance improvements in N M pruning compared unstructured pruning the same sparsity level. This due to the higher complexity of the sparsity pruning problem, which ALPS, as highly advanced optimization algorithm, handle more effectively than competing heuristics.
Conclusion
We present ALPS, an efficient optimization-bant based framework for one-shot unstructured LLM pruning. ALPS employs the operator splitting technique to effectively solve the én-constrained layer-wise pruning problem. To enhance the performance of our algorithm, we introduce novel penalty parameter updating scheme and post-processing procedure using PCG with vectorization/GPP parallelism that takes into account problem-strueteeeeeeeeeeee structure. We also establish novel convergence guarantees for our algorithm. ALPS can efficiently perform high-qualit pruning of LLMs at scale. Our experiments confirm that ALPS outperforms existing pruning methods terms of both the pruning objective and the performance of the pruned model. Future work will consider extending ALPS to incorporate structured pruning constraints and quantization to get better understanding of the strengths and scope of our optimization- based approach.

---

Model
Sparsity
Algonihm
WikiTex22
PTB.
C4J
ARC-Easy
ARC-Challenge
MP
1981(20)
2061(10)
1656(10)
58.22(40.00)
40.61((0000)
18.94(10.00))
Wanda 13.23(10.40)
1695(20022)
14.67020.18)
74.87(10.14)
64.21(=0A0)
29.23(+0037)
2:4
SpaneGPP
10.90020.07)
14.02(:0.10)
12.04(20.44
75.59(10.28
31.23((10.2) 06.75(=0.55)
DSnaTT 12.36(+0.10) 10)
15.73(+0.07)
13.55(+0.08)
74.94(+0022)
64 19)
29.44(+0.50)
ALPS
10.64(20.09)
(3.5((10.06)
11.69
75.93(10.09)
66.82 0.54)
31.45(10.37)
ОPT-30B
MP
564.1(=0.0)
1487(=10)
(00SE
62 (2.44000000)
0.04
22.27(+0000)
Wanda
10.78(#0008)
14.07(
12.13(+0.03))
75.6((00.18)
06)90(004))
30.48(+0.41)
SparsGPT
10.30110.06)
11.52((0000)) 13.35(10.15)
76.10(10.25)
0788
32.17(10.88)
1083(+0008)
1393(+0.04)
12.16(+0.0))
7542(++0.1 18)
66.3((-044)
30.58(+0446)
ALPS
00.14:00.05
13.18(00.12)
11.32(+0000)
76.61 50)
67889-0225)
32.01(+0:49)
8.89(+0.00)
200.0(=0.0)
+1.40(00.00
57.60(00.00)
30.30(00:00)
18.93(11.02)
73.16(10.27)
33.92(10.37)) (3.67(
2:4
SpaseCPT 9.21C+0.05)
61.82(13.16)
73.94(10.60)
0..80
87.54(22.04)
22.30(+0.23)
ALPS
8.14(+0.10)
52.34(=2.08)
9.36(=033)
74.72(+0058)
68.71(00.59)
38.96(+0.50)
MP
7.32(10.00)
137.3(10.0)
9.11(20.00)
74.43(10.00)
63.01(00.00)
35.67(110.0)
Wanda
7.02(+0.01)
55,07(+0079)
8.84(+002))
70.69+-0.31)
67.05(-0.13)
38.26(+0.56)
4.8
SpansGPTT
7.01(40.03)
47.64(=0.81)
8.52(=...14
75.84(10026))
69.36(=0.84)
38.74(+.0.55)
DSadT
7.13610002)
54.07(20088)
8.95(+0.02)
75.61(110.23)
37.58(110.63)
ALPS
6.81(10.07)
42.15(10.8))
8.22(10.17)
76.52(10.57)
(9.69)
39.22(
Table 3: Performance analysis for one-shot pruning of OPT-30B and LLaMA2-13B at 2 and 4
sparsity patterns. We run each method five times and report the mean and standard deviation each performance criterion. Here, + denotes lower values correspond to better performance, and f denotes higher values correspond to better performance.
Acknowledgements
This research is supported in part by grants from ONR (N0001421 12841 and N000142212665), We acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing HPC resources that have contributed to the research results reported within this paper. Additionally, we thank Google for providing us with Google Cloud Credits. We thank Shibal Ibrahim, Mehdi Makni, and Gabriel Afriat for their helpful discussions.
10

---

References
Perplexity of fixed-length models, 2022. perplexity
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 087774, 2023.
Kayhan Behdin, Ayan Acharya, Aman Gupta, Sathiya Keerthi, and Rahul Mazumder. Quantease: Optimization- quantization for language models-an efficient and intuitive algorithm. arXiv preprint arXiv:2309.00 0000000099900,,,, 2023.
Riade Benbaki, Wenyu Chen, Xiang Meng, Hussein Hazimeh, Natalia Ponomareva, Zhe Zhao, and Rahul Mazumder. Fast chita: Neural network pruning with combinatorial optimization. arXiv preprint arXiv:2302. /4623, 2023.
Yonatan Bisk. Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense natural language. Proceedings of the AAAI conference on artificial intelligence. volume 34, pages 7432-7439, 2020.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of machine learning and systems, 2: 2:129-146, 2020.
Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends@ in Machine learning, 3(1):1-122, 2011.
Vladimír Boža. Fast and optimal weight update for pruned large language models. arXiv preprint M.Xiv:2401.02938, 2024.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:23033. /27/2, 2023.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. Advances in neural information processing systems, 33: 13:15834-18846 2020.
Tianyi Chen, Bo Ji, Tianyu Ding, Biyi Fang. Guanyi Wang. Zhihui Zhu, Luming Liang, Yixin Shi, Sheng Yi, and Xiao Tu. Only train once: A one-shot neural network training and pruning framework. Advances in Neural Information Processing Systems, 34: 14399637-196,, 2021.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1803.05435 2018.
Damek Davis and Wotao Yin. Convergence rate analysis of several splitting schemes. Splitting methods communication, imaging, science, and engineering, pages 115-163, 2016.
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint 1rX1v:1907.048400 2019.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedeley, Elias Frantar, Saleh Ashk- boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized represen- tation for near-lossless Ilm weight compression. arXiv preprint arXiv:2306.00007, 2023.
Xin Dong. Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in neural information processing systems, 30, 2017.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llaaa herd of models. arXiv preprint arXiv:2407.21780 2024.

---

Marwa El Halabi, Suraj Srinivas, and Simon Lacoste-Juie Julien. Data-efficient structured pruning via submodular optimization. Advances in Neural Information Processing Systems, 35:36613-3660 2022.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International conference on machine learning, pages 2942.2952. PMLR, 2020.
Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning, Advances in Neural Information Processing Systems, 35:4475-4488, 2022.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323-10337. PMLR, 2023,
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint www.iv1902.0957777 2019.
Scott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights. arXiv preprint arXiv: /71 1.09224, 3(2):2, 2017.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns, Advances in neural information processing systems, 29, 2016.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
Y. He, X. Zhang, and J. Sun. Channel pruning for accelerating very deep neural networks. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 1398-1406, Los Alamitos, CA, USA, oct 2017. IEEE Computer Society. doi: 10. 109/ICCV.001.155. URL https
//doi. wet.eeecmmptteesssitttt... org. 1119/ICCV 2017 155
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Mingyi Hong, Zhi-Quan Luo, and Meisam Razaviyayn. Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems. SIAM Journal on Optimization, 26(1): 337-364, 2016.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0096) 2021.
Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry. Accelerated sparse neural training: A provable and efficient method to find n: m transposable masks. Advances in neural information processing systems, 34:21099 21111 2021.
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. arXiv preprint arXiv:2203.07259, 2022.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnale sparsity. In International Conference on Machine Learning, pages 5544-5555. PMLR, 2020.
Vadim Lebedev and Victor Lempitsky. Fast convnets using group-wise brain damage. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2554 2016.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
12

---

Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340, 2018.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation- aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint arXiv: 1800005000 2018.
Jinn-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In IEEE International Conference on Computer Vision, ICCV 2017, Venice Italy, October 22-29, 2017, pages 5068-5078 IEEE Computer Society, 2017. doi: 10.1109/ICCV. 2017.541.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Lim-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21 120:21702-2120, 2023.
Mitchell Mareus, Grace Kim, Mary Ann Mareinkiewicz, Robert Maclntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. The penn treebank: Annotating predicate argument structure. In Proceedings of the Workshop on Human Language Technology, HLT '94, page 1 11--119, USA. 1994. Association for Computational Linguistics. ISBN 1558603573. 1075835. URL httptttttpss.ooo.. org/ 10. 1111//11///1111111 1075835
doi: 10.3115/1075811
Xiang Meng, Wenyu Chen, Riade Benbaki, and Rahul Mazumder. Falcon: Flop-aware combinatorial optimization for neural network pruning. In International Conference on Artificial Intelligence and Statistics, pages 4384-4392 PMLR, 2024a.
Xiang Meng, Shibal Ibrahim, Kayhan Behdin, Hussein Hazimeh, Natalia Ponomareva, and Rahul Mazumder. Osscar: One-shot structured pruning in vision and language models with combinatorial optimization. arXiv preprint arXiv:2403.1298. 2024b.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mix-
ture models. In International Conference on Learning Representations, 2017. URL https
//oppnnevvew..ee///eee wwt./forummm Bi=iiyy2Bddas
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, (1):2383,, 2018.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv: 1,06444 2016.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 11264-11272, 2019.
Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer, 1999.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Mareo Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.0603: 2016.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito,, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21))), jan 2020. ISSN 1532-4435.
Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint ar arXiv:2402.079.0 2024.
13

---

Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. Advances in Neural Information Processing Systems, 33:18098-18109, 2020.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. / 1695, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv: 1905.099950 2019.
Yu Wang, Wotao Yin, and Jinshan Zeng. Global convergence of admm in nonconvex nonsmooth optimization. Journal of Scientific Computing, 78:29-63. -3, 2019.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.0766 2022.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances in neural information processing systems, 29, 2016.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning. 2023.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27168 27 183. Curran Associates, Inc., 2022.
Shaokai Ye, Tianyun Zhang, Kaiqi Zhang, Jiayu Li, Kaidi Xu, Yunfei Yang, Fuxun Yu, Jian Tang, Makan Fardad, Sijia Liu, et al. Progressive weight pruning of deep neural networks using admm. arXiv preprint arXiv: 1810.0777 2018
Shaokai Ye, Kaidi Xu, Sijia Liu, Hao Cheng, Jan- Henrik Lambrechts, Huan Zhang, Aojun Zhou, Kaisheng Ma, Yanzhi Wang, and Xue Lin. Adversarial robustness vs. model compression, or both? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 111-120, 2019.
Xin Yu, Thiago Serra, Srikumar Ramalingam, and Shandian Zhe. The combinatorial brain surgeon: pruning weights that cancel one another in neural networks. In International Conference Machine Learning, pages 25668-25683 PMLR, 2022.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0000 00066, 2022a.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint Xrxiv:2005. 00068, 2022b.
Tianyun Zhang, Shaokai Ye, Kaiqi Zhang, Jian Tang, Wujie Wen, Makan Fardad, and Yanzhi Wang. A systematic dnn weight pruning framework using alternating direction method of multipliers. In Proceedings of the European conference on computer vision (ECCV), pages 184-199, 2018.
14

---

Yuxin Zhang, Lirui Zhao, Mingbao Lin, Yunyun Sun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. Dynamic sparse no training: Training free fine-tuning for sparse llms. arXiv preprint arXiv:2310.08915 2023.
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning n. m fine-grained structured sparse neural networks from scratch. arXiv preprint arXiv:2102.0.. 04010, 21X1002102.00000, 2021.
Michael Zhu and Suyog Gupta. To prune, not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXi::1710.0188 2017.
15

---

A Proofs of Theorem I
Proof. For the sake of conciseness, throughout the proof, we denote = A2l and [X"X+ X 2 W. To establish the theorem, we first present the following two lemmas. The proofs
of these two lemmas are given Section A. and A.2 respectively.
Lemma 1. Let [ and {V(1) t t 0, it holds
be the sequence generated Algorithm
Then for any
HV
| | S |
HD
lFt
(7)
Pr.
and
( (IG-HD<01
Pt (IG
HD |G-
[F
HV ||
(8)
Pt
t=o be the sequence generated in Algorithm Suppose [P:] is non-decreasing. Thenfor for any 2 0, holds
Lemma 2. Let ] [w 1=0 and {v(")
DD +
II
H
IV
G|G
Pr
Po
Ps
Returning to the proof of the main theorem, combining Lemma 2 with the initialization of Algorithm
D giies
|(
3
H||2
|
3 G
| [F
II
|
Pt
Pr
Pu
Ps:
(10)
DC
exp
3||H||2
| + 3|
Σ
Ps
Let
1
||
3| Eي
C(X, Po, := = 2||G||r + 2 H||
exp
3 H 2
s=0 Ps
Ps
(11)
be the constant depending on X, W and Ex
1/p.. Lemma together with 10 leads to
HH
|V(++| < |G HD 19)
Pt
(12)
< |G|| + ||||
[ |F
|
1 c(x, Wo, $ 2
and
2 ||G-DD'S (D 1| HH
C(X, W. po. tu.
(13)
Pt
Pt
Pt.
It then follows from ((++)
(t++1) (V(++1)- - v(t)) that
| + ||V(
C(X, W. po. Po.tu.
w(+1)) D(++)
(14)
Pt
Pt
Therefore, we prove the desired incquality. Since Σ 1lps < cc, (D] is a Cauchy sequence. and therefore there exists a matrix D such that ( + D. It follows from [W(++) (((+)) + that (()) + D. The proof is completed.
16

---

A.1 Proof of Lemmal 1
Proof. According to the W update rule in 4 it holds
W(++) -D(1) + p()) (H + ++ p:I) (G v + p.D(.)) (()) + V
V(1)
p(1)
(H + p,.t)= Pt - D( (1) + (H + P.I) (G - v(0) + V
Pt.
I I H
H
-1
V
HD
I
(G v +
Pt
Pt
Pt
Pr
I+
H
-1
(G
HD
(0)
1
I
I
H
V
Pt
p1
Pt
Pt
1
H
HV(()
I+
G
HD
(4)
Pt
Pr.
Pt
(15)
Therefore, obtain
V(t)
HV
w(++1) - ( +
I
G HD
(0)
+
p())
Pt
Pt
Pt
F
HV
G
HD
(8)
(16)
Pt
Pt.
F
||
HD
HD
||
Pt
Pt
Denote Ż = ((i,5) E [N x [Nott]
D (1)
0}. It follows from the D-update rule and the
definition of the projection operator that
v(9
2
V(() 2
-
w(++1)
E
w(++1)
min
+
Pr.
F
IC[Nia]>x[N [[Nuud] Non (K.))Z
Pr.
b,
V(t)
V(O)
< Σ w(++1) + (iJ)e
Σ
w(++)--
(17)
Pt
iJ
(1.))
Pt ig
V(
Together with (66), we get
D ((+1) w(+1) V(t)
1
H
|G
HD
10
(18)
Pt
F
Pr
Pr.
It then follows from the V Upddae rule that
|V(++
Vit)
|
D (++1) w(++))
S (DG-HD((1
(19)
Pt
F
Pt
This establishes the inequality 7 Furthermore, by summing up 16 and triangle inequality, we verify the inequality 8
18) and applying the
O
A.2 Proof of Lemma 2
Proof. It follows from Lemma
I that
| || $ ||
HD +
Pt
(20)
H
H
F +
||]|
+
Pr.
17

---

and
2
HH
D
D
|| 5
|
HD
(9)
Pt
Pr.
(21)
2
H
|
v
H
|||||
Pt
This further implies
D
1
1+
2
||
|D
|
(8)
||
G||
2
H
V
F
(22)
Pt
Pt.
Pi
Combining inequalities
20 and
22
yields
|
|V(++
||
(
D
(1+1
|
PL+1
Pt
3||H||2 12
3|||||
2/|H||2||(())
|D
(23)
Pt
Pt
p
3 3||
|
G|G
F
+
Pr
Pt
Pr
Denote a, = |D |[V(*))| (//0l///PP pc. then the above inequality can be rewritten as
(1
3 H
3|||||
ac+1 $
1
at
+
(24)
Pt
Pr
Therefore,
04+1
a2
3
+
oo (1 + + H pk))))
3
H p/Pk)
P: II.
+ H
(25)
ax
3 G||
3 H
2/pp)
Pt
then follows from telescoping that
3 G
S 0g
(26)
+
H
Pt
$=0
Recalling the definition of a, completes the proof.
B Experimental Details
B.1 Experimental setup
We performed all experiments computing cluster using Intel Xeon Gold 6248 machine with 20 CPUs and a single NVIDIA V GPU, which is equipped with 192GB of CPU RAM and 32GB of CUDA memory. The PyTorch library Paszke al. [017] used to implement all language models and pruning methods for our experiments.
Pruning problem setup. For given sparsity s, the lo constraint A in the pruning problem to Non.. Following the pruning framework proposed by Frantar and Alistarh 2023) Meng al. (2024b), we solve the LLM pruning problem sequentially, layer by layer. For layer E, the input activation matrix X in ( is set as the output of the previous & pruned layers on N calibration samples.
Data pre-processing, Let E Diag(X X X + A21) 1/2 To achieve better scaling, we define W' W and reformulate Equation (I into the following equivalent form:
min W'ERN Now Nout
XW - XEW + Ag W EEW s.t. || < k
(27)
Practically, we apply Algorithm solve Equation 277 and recover the solution to the original problem by setting W = EW' It important to note that this pre-processing step does not alter
18

---

the procedure of Algorithm| I or affect the convergence analysis. It only modifies the updates within Algorithm | and can lead to a better convergence rate in practice.
Hyperparamter choice. We choose 2 = 0.01 Tr(X X). In Algorithm we set pa = 0.1. And update p every 3 iteration based on step function that depends on the current value of Pr and St := Supp (D)) A Supp (D(t=3)) |, which represents the number of elements in the symmetric
difference between Supp (D()) and Supp (D() Specifically, we set
1.3pt
if > 0.1k:
Pc+1
1.2ps
if 8: 0.005k,
(28)
1.1ps
if 8t
2 1.
If S; = it indicates that p is sufficiently large and the support has stabilized. In this case, we terminate Algorithm [ set S as the support of W, and apply Algorithm D with 10 iterations to solve problem (6) with support S.
Implementation details. Below are the configuration and implementation details for the competing methods and our proposed framework ALPS.
MP: For each layer in the LLM, we perform magnitude pruning by sorting the absolute values of all entries of the dense weight W in descending order, keeping the top & entries unchanged, and setting the remaining entries to zero.
SparseGPPT: We utilize the authors' implementation (codes available on GitHub) with default hyperparameter settings to perform one-shot unstructured LLM pruning.
Wanda: We utilize the authors' implementation (codes available on GitHub) with default hyperpa- rameter settings to perform one-shot unstructured LLM pruning.
DSnoT:: We utilize the authors' implementation (codes available on GitHub) with default hyperpa- rameter settings to perform one-shot unstructured LLM pruning.
B.2
Additional experimental results
B.2.1
The importance of the م update scheme
Our proposed update scheme, theoretically supported by Theorem 1 ensures that ALPS converges rapidly while finding high quality solutions. In contrast, ADMM with a fixed p may fail to converge when applied to to constrained least squares problems. To provide empirical evidence for this claim, we compare ALPS with ADMM using fixed & values. We examined two key metrics: reconstruction loss (objective) and the rate of change of the support (of weights) between consecutive iterations (this measures the convergence speed of the algorithm). Tables 55 the results for these metrics, respectively. Our findings show that ADMM with a large p(=33 3) converges quickly but yields poor solutions, while a small p(= 0.3) fails to converge. ALPS, utilizing our p update scheme, achieves both rapid convergence and high-quality solutions.
Supp change/ / Iter
5
10
20
30
50
100
ALPS
1.63e-1
1.28e-1
5.95e-2
5.32e-2
5.31e-2
5.31e-2
ADMM(p 0.3)
7.83e-2
7.55e-2
7.50c-2
7.47c-2
7.47e-2
7.45e-2
ADMM(p = 0.3)
9.32e-2
8.18
7.64e-2
7.53e-2
7.45e-2
7.42e-2
Table 4: The relative reconstruction error |XW
XW
XW | over iterations, comparing
ALPS with ADMM using fixed penalty parameter
Supp change/ / Iter
5
10
20
30
50
100
ALPS
20.2%
17.0%
2.8%
0.0%
0.0%
0.0%
ADMM(p p = 0.3)
6.4%
7.0%
7.0%
7.0%
6.9%
6.9%
ADMM(p 0.3)
0.2%
0.1%
0.1%
< 0.1%
< 0.1%
< 0.1%
Table 5: The rate of change of the support (of weights) between consecutive iterations, comparing ALPS with ADMM using a fixed penalty parameter
19

---

Algorithm
OPT-L.3B
OPT-2.7B
OPT-6.7B
OPT-13B
OPT:30B
LLaMA2
LLaMA2
MP
4.7
8.9
23
47
120
25
46
24
Wanda
99
161
280
502
1027
214
407
1118
SparseGPT
728
2980
6662
2319
2392
DSnoT
125
213
417
758
1528
347
651
1176
ALPS
2360
6069
14323
48366
3043
7145
6735
Table Runtime (in seconds) comparison for one-shot unstructured pruning of OPT models and LLaMA models. Here, runtime includes input activation generation and model pruning.
B.2.2 Runtime comparison
We compare the runtime of ALPS with other methods Table 6 ALPS employs an advanced optimization method to solve the layerwise reconstruction problem, which results in longer running times compared to other algorithms. However, it's important to note that ALPS runtime still negligible when compared fine-tuning methods for LLMs, e.g., LoRA Hu al. 2021.
B.2.3 Comparison of ALPS and ADMM-Grad
We discuss the difference between ALPS and ADMM- Boža 2024 another interesting work using ADMM for L.LM unstructured pruning. [Boaa 2024 selects the sparsity mask via iterative magnitude pruning and applies ADMM to solve problem (6) with the selected sparsity mask. ALPS, contrast, an end-to-end approach that directly targets the to constrained least squares problem ① By simultancously optimizing both weights and sparsity patterns, ALPS achieves lower layerwise reconstruction loss compared to ADMM-Grad, as demonstrated Table 7
Additionally, since [Boža 2024]
employs ADMM for solving problem we compare it with our
proposed PCG procedure. We tested both approaches for solving problem (6) with a given support. Results presented in Table sjswow that PCG outperforms ADMM-Gran in both computational time and objective value. The time advantage of PCG stems from its ability to backsolve without explicitly computing matrix inverses.
Sparsity
0.4
0.5
0.6
0.7
0.8
0.9
ALPS
3.55e-3
7.56e-3
1.47e-2
2.77e-2
5.32e-2
1.13e-1
ADMM-Grad
4.47c-3
9.53e-3
1.83e-2
3.36e-2
6.19c-2
1.25c-1
Table 7: Relative reconstruction error XW ADMM-Grad across different sparsity levels.
XW
/
XW|
comparison between ALPS and
Sparsity
0.4
0.5
0.6
0.7
0.8
0.9
time
0.01s
0.01s
0.01s
0.01s
0.01s
0.01s
ALPS
loss
3.55e-3
7.56e-3
1.47e-2
2.77e-2
5.32e-2
1.13e-1
time
0.1
0.1
0.1
0.1
0.1
0.1
ADMM-Grad
loss
5.le-3
1.12e-2
2.32e-2
4.49e-2
8.83e-2
2.00e-1
Table 8: Comparison of PCG (Algorithm 2 and ADMM [Boža
2024
for solving problem (6).
Results show runtime (seconds) and layerwise reconstruction loss supports with different sparsity levels.
XWW
XWI
X across
B.2.4 Pruned model performance on MMLU benchmark
We evaluated the one-shot unstructured pruning performance of MP, SparseGPT, Wanda, DSnoT, and ALPS on ILLMAA-8B using the MMLLU benchmark to give a comprehensive assessment of each method's effectiveness. Table 1 shows the mean accuracy across all MMLU categories. The results demonstrate that ALPS outperforms other methods, further validating its effectiveness in producing high-performance pruned models. We also observe significant performance degradation on MMLU
20

---

at high sparsity levels, suggesting that one-shot pruning should be combined with fine-tuning HHu techniques to maintain model performance practical
al., 202T] or prompting [Sahoo al. 2024
applications.
Sparsity
MP
Wanda
SparseGPT
DSnoTT
ALPS
0.4
47.30
53,48
56.74
54.19
57.42
36.11
42.64
50.87
44.49
51.01
0.6
25.35
29.50
37.54
28.96
40.40
0.7
23.10
24.78
27.99
24.73
28.71
2:4
25.30
28.24
34.27
29.25
33.98
4:8
28.70
32.64
38.37
34.59
41.21
Table 9: Performance analysis for one-shot unstructured pruning of LLaMA-3 8B models at various sparsity levels using MMLU benchmark.
B.2.5
Comprehensive model performance across sparsity levels
We compare the one-shot unstructured pruning performance of MP, SparseGPT, Wanda, DSnoT, and ALPS on OPT-1.3B-30R models and LLaMA2-7B, LLaMA2-131 13-M1BBBBB LLaMA3-8B models in Tables
17 The models are pruned at unstructured sparsity levels of 40%, 50%, 60%, 70%, 80%, and 90%, well at 2:4 and 4:8 sparsity patterns. We evaluate the perplexity of the pruned models on WikiText2, PTB, and C4 datasets. Additionally, we assess the accuracy the pruned models on PIQA, LAMBADA, ARC-Easy, and ARC-Challenge. However, LAMBADA accuracy results for the LLaMA model have been omitted since LLaMA models have unsatisfactory performance on this dataset without further modifications. For each combination of model, sparsity, and pruning approach, we run each method five times and report the mean and standard deviation each performance criterion.
21

---

Model
Algorithm
PEB
C4
LAMIADM
PIGNT
ARC
ARC'Caallnge
MP
1400020000
95. 25.30
0:360300001
45 (5,8000000)
19.71(80100)
Wands
1600--034)
21 311300000 20
16496-006)
46. (6.7740445
68 (6686+0 90)
24.01(10200 (4590(0))6)
13B
SpaneCPPT
16.34:00115
19
26, 1690
3000010.30)
23.0211..24
DSarT 16.32-00.11
206
47 45)
23.05 15 26)
MP
1189
Wands
19 19)
(385
54. 90 40)
72 46
50 57
26 35.06:40.55)
0PT-2270
SpanoCaPT
1261
000
13 000
M. 47 80
72 471440255
5% 42)50455
26, S7 M0)
DSaoT
17.9400499 1283:20.10
499
13 0..
55 32)
72 25)
99. 13:
26. 26.18
ALPS
1262
15.78 -002)
-35770004
58 68.73600 96)
72.8 +0.05
60 +022
26.08 56)
SIP
0000
200000
Wands
000
15
02
64.0913034
22 0P7
24
29.58 20)
OPT-6 7B
SpanoCaPT
1094-006)
1384 15)
-2.0
64 54 36
28
DSso'
009
000
12220000022
64. 64.5110 80)
22 21:23::01 14)
64 65 10
25,09
29.09.10.22 22)
ALPS
10.6
1364
12.00(00.05
66. 40:00 46)
75. :0255
65.49 16
20.20
MP
50.87:0000
22
43. 000
01.20000000
68 00000000000
52. 00
23.72(40.00)
Wanda
10:00-000)
13.210:000
0000)-0000)
15)
(2.13
OPT-138
Sp
12 10.36-00006
64.07 01.43:0003
75.06
3L $1.55 6600000000
1299:000))
6488
25 47
66 24
ALPS
10.32
1268-0003
11.39-007
64 23:+0 56)
75 MEMHHUS 15)
S2,47 (0
50)
MP
20:30000000
25 0000
16. 00000000
33 00
73 00
27.50 50,00000
Wands
9:45530003
12 12.30-000
10 10
66 3110000333
77 000000115
06 05 14)
ST. 31.18030.44
3PT:30B
SpaneCIPT
9.47(30006)
12:1300000
00
07 05
77.31 14)
69 13
53.52(20.33
DSarf
9.40++0.04)
12 0005)
000
67.35 +50 40)
76 96 14)
66 21
+1.79
ALPS
9.47:00005)
12
00:00=0002
67 07:42
1:37030000
13:0000
(00
Wands 6:00:+0000
2600000000
77 (5)5-0007)
66 400 19
40-12 (4)
LLaMM2 TB
SpaneGaPT
27:75(-057) 6.11:2000)
72 1.111.::000 25)
618111975
29.33110.22
DSauT
5.90:00.00)
24594:0055
7.500
76 06.76:20115
67
58
38.40(+0.27)
ALPS
6.07220002)
27. 2T:14:20.49
7.5% 20.04)
(7.31(20
68.74120009)
(0.277
MIP
+000)0000
20 00000000
10:00
MC 10
Wands
5.38:00001
+0.00)
10.
1LMMA2-138
SpaneCart
71 FUL
DSaoT
5.30:00.00
32.48-0.13 (2)
10.000
78.61100110
4184(10.19)
ALPS
3,36/20001
33 0::40:022 20)
6.10.0000)
78 78.38150011 15)
03,70:10.00
XIP
20000:000
73:55150000
Wanda
7.4020000)
12042000)
00.70=000000
77.74 19)
T5. 13.55
45.50(20.34)
ILMMA--R
SpinoCiPT
2.50)+0.03)
11 (109)000)
(0.5(-0000)
76 76.230--0000 19)
76, 76766(+061)
+0.3(+00.55)
DSaoT
7.30200000)
000))
10.60:000000
72.58(2:0255
45.29(10.55)
7.8540000)
Table 10: Performance analysis for one-shot unstructured pruning of OPT models and LLaMA models at 40% sparsity. (1): higher is better; ({): lower is better. We omit LLaMA results on LAMBADA due to its poor performance without modifications.
Model
Alymitten
WikiTowet
LAMBADA
ARC-Fasy
3005
35600=000
131000001
18
Wands
20.00 0.362
18. (W 14)
41, 85)
68 08.901300300
52,99300669 68)
22.6010222
GPT-1338
SpaneCiPT
17.77)_0.009
22.05 25)
17,55 220
45.82121.009 (0)
68.071:0445)
48.54(20.4)
22.59)1...27)
Disuf
19:24:
19.24-011))
26 100 636
19.10 10 15)
41 (1-00)
68. 540+00000 199
St. AR 599
21 1..50++0,0 56)
ALPS
21.60:0244 17.2%-0.1))
24)
16.97 15)
47,53 100
70,36 389
(4.17()0447
(3.30
1198820000
138.7000
55,45
68.61
22.070100000
Wands
2.7B
SpansePPT
03
8318011777 76
7L 71-951-0441)
25.96110330 31
DSauT
04.4
05
71 200
24.32/+0.35)
ALPS
0.13)
17 26
14.19-00000 100
(7
72.511::044)
59 59.25
25.57(400266
SP
552
00
2.2
20.00
6 (0740100000
21.55(10.00)
Wanls
1202(-0.11
1773:
10)
05
60
35
63 (6)
27.55()
OPT-6715
Sp
11600-000)
15 44)
66 65
24 MALLOUS
67 ADL
DSa/T
11997:00121
15.70000100
0.05
(2.74110356
27.6002.227
ALPS
1..4
15.2K-02
35
52)
MP
29332100
275
BOO 30000
SA 100
Wands
01.99
0 23
02
7471E2019) 71
03.33110333 (4
30.07/100118
OPT-138
Sp
11:17-00100 10)
(391
11 11.90-000
64 25
65 5500000000 20
S0 17)
DSaoT
14.00001100 11.3320007)
14.
12
63 000
30.26:10.399
ALPS 10720007
13.8% (006)
-1.70-0000
65.2101949
75
66.9% 100
(5020)
Wanda
10000-0000)
13.37
64955100000 03.4
76.77(10004)
68 18)
32.06/10.34
OPT-308
SpineCPPT
12864=0.00 9.55(5005)
11)
67.83 11.44(2000)
00:
76 77)
54
32.58/+0,55)
DSS/T
10000/0000)
0300
67.00 67.02(10357)
76 76.721-0115
68. 21)
32.83(40021)
APPS
12 - 10)
60. 67.71 56)
60 35 209
SIP
1183-0000
17,000000
20
MC BOEOO
Wands
699330000
09.8
66, 14
1LMMA2-BB TB
SpaneCPPT
7.00/00.02)
2655(2235)
8.66
16.1211:0055
13.07111110
(6.40
DSaoT
6.90120002)
30.25 16)
1.500
75 23.471200222
34,11110.44 300011033
46)
ALPS
30.95=2261)
8.26 10.10)
76.43:00011
37.18 (9)
MP
6.10200.00
79:30(=00000 000
8.23 10.000
73. 73.7214:0000
50.00000000
(4.00)
Wands
5500+0.003
40224 26)
7.69
78,30 (0.0)
7LAM 59
40 00 51)
LLAMA2 13B
SpaneClPT
6.00
4181 36)
7.60:10.06)
77. (7.60(19.25)
09:27(1:0022
01.00
37.92 1.99100000
3000
77.00
ALPS
5.954)0044)
40.330-0.60
7,46 20.06)
76.2811:0115)
70.17110.00)
40.82(1044)
MP
51,00000)
01
00:501:000
(7.0013000
605212000
52.42110.00
Wands
9.57050003
05.35:0003
(3.40:0001)
75 75.57 92)
71
40,48/50.50
LLMMA-S0
SpaneCaPT
9.47330000)
14.30 10)
12.331200177 (5.22-0007)
75. 12.333.00000 599
41. 11.181111177 1.17
DSsur
0.94+00005
13 -005)
75 75.44(1-0200 200
71.
24.44400000 S4)
90 00.52/40.200
ALPS
9.10:2000)
15.38-005)
11.78:0023
76.99/::0.23
75.15110.45)
42.99(2)1.3))
Table 11: Performance analysis for one-shot unstructured pruning of OPT models and LLaMA models at 50% sparsity. (1): higher is better; (4): lower is better. We omit LLaMA results on LAMBADA due to its poor performance without modifications.
22

---

Mode
Algorilus
WikeTed
LAMBADA
ARC-Easy
ARC Challieee
6136:20)
401713001
2233:-00
30.35,20.00
19.20080000)
Wands
27.17-00.30
62.55
25,46640 54)
06
21,00(00.00) 42.05
SponeGrr
22 22.80200300 30)
12
200 30)
19 260
65
46. 10.18:30:221
21 (1
DSmoT
30 30.47:00.34)
50.00
29.39110.41
20
63
44.76.20.39 63
20 308442-0445
41.15
21.51(00.55
28.17
1..1000.50 50)
1962
K..4-1115
67.53 00 500
$1.00 (0.54)
22.53 444
MP
4255,205
STA7
34 20.000
20 30230
Wonda
244 0028
19774100
7FT-277
SporeGIPT
24.00
1624110
70
10.17) 55
34-26620-400
D
21 21,764+0-433
A2P9
1547:-0
1575
S3A4 1
71,54:0000
56 20.100
MP
5648330
55 000
20 10W.11000
Wunda
24.50
16.17/20.12
49
19.10 (99
72.500-0225 25)
25 35.25662..222 22)
OPT6 78
Sponetii
18.5k-00.21)
18. 10.78
50 -000
75 50 90
64 t4 16.38)
27 10
D
05.3
20.30
16.00:00.11)
5155K-120 250
71 01
57 $7.58 11
24 44.5 56)
1348 14)
10.261
299
10
21 196)
117700:20
Wanda
16.
25
S818 100
71:55-0
28-78 20.55
1F1-138
SpuneCPP
13.04:00.00
17.33
13. 1. 16)
6199611 1000
(5114-014)
10.
21 NA 260
58
ALPS
12.05:00155)
15.77
12.55/50.10 16)
11.3
74
6485 20.43)
30007
MP
15234::00
26, 30 20.000
19 19.000
Wanda
24.85:2199 977
15.88
21.13(#3100) 1.0
38.3
28.37
74 07
64.13 20.48)
20 20,04C50.300
OPE-1R
SponsttrT
1069)-007)
14.5 (10.00
18 175
67 76 0.73)
76 (0
67 6% (0.35
52.6 (0.51)
DSmaT
1438820
24)
10.00=0.00)
11500
76.34
(2-66))))67)
7059:20)
28531812-0
20))
63:100
44.9% 20.00
Wanda
05.29/10.84)
(2.0(+
59
LLAMAZ 7B
SpansGPT
10.19:0000
215
13,52:10.20
30.05
30 20.776200000 66)
DSmT
1147::00000
14.8
(3.4((+00005
70,05 199
59 00.00::6.0 20.
25. (5.40(20.59)) 59)
ALPS
9.36120.19)
107.641442
10.36:10.34)
73.68200114 14)
60 60.56 22.13)
31.58(20.96)
MP
T0.00 (0100
401 10.0
4n 51C 10,00
Wanda
8.4411000)
19.70100000
(0.44110000
75,00
63 63.46.00.10
LLMMAC-1B
SporsearT
33
74.99
D
10.92110.04)
74.9910024)
63 63
7,58 10g
66.90 +3.90)
8.9--026)
76.16(=0.34 349
67.54 144
78)
MP
200064;2000)
66661(30000)
39 30,05
Wanda
25 055)
57 07,57)
20 S418000 160
65. M0000-0 199
00 10.092100 10:37)
27 8N (20044)
SpansePPT
(5.45:0011))
22.59
17.921200.50)
71.81(20.35)
63 20.35)
31 31.9662:100 1055
DSme
23.05=0..16
56. 0.0
27.71(
67.85((=0-49)
60. 60.51:-0.300
25.07()0.66)
10.19-0.18
20.86
15.07)9/777
67.10 73.4122966)
34.52:9085)
Table 12: Performance analysis for one-shot unstructured pruning of OPT models and LLaMA models at 60% sparsity. (1): higher is better; (4): lower is better. We omit LLaMA results on LAMBADA due to its poor performance without modifications.
Midel
Alperitm
Wilelent
PIB
CF1
LAMBADA
ARC Eayy
ARC Challege
Wands
1008::338)
128.4/33.13
75.58(22.54)
59.3003946)
37.92
SpanaGPT
52.03:=2.233
30.00=3.330 31)
37.05
23.20
62.88
40.7%:00555
19,60=00.7)
DSnoT
200 A6.666511
ALPS
39.50
28.5241.255
32.11/31.7)
64.36--377
45.01-110033
20.00-0-332)
MP
9WKX 11
UTAK 0100
16.33 110000
Wanda
365 16.55
379
37.66
5.0 40)
34 20.35
17.8120125)
OFT-2.7B
SpIneGPT
28.03(=1.62)
40 40.8%=10 119
23 (1C3666666
49 40.55(=n:500 66.3020.33
21:57(En.55)
114809155
116 11620-677
18 10--6-07765)
ALPS
15:3611344)
35.76=090
75
42.5 40)
67.6213336)
477
00000000
02
100000
Wanda
162.9:88.7)
204.0: 10.6)
13.55
2 10)
1.13363.00 16)
OFT-6.7B
SparoCPPT
21 11:14:9060)
29
44)
0.
69211 76
20
24 (0.55)
DSuoT
798312446)
2773)
4768
0.1
18 20.45)
ALPS
18.99
24 00266 26)
71
00.74
-0.52)
MP
5365992309
120640 40)
155 160K
BANK 100
52 10000
25 25 10000
22 2000
Wanda
63:3::222)
66.50 1.93)
50 1911 28)
43.60 63.07220.72
23.50 (.63))
OPT-33
SPinoGPT
19.20-00423
25 50
16 7450455
4T (.99
69.1611033 383
5%
53.5100044
36 2323% 0.99)
DSnoT
78.13-22411
ALPS
26.7):0062)
1547
$7.08
2627101:0
12503(10)
13057 (3057100)
0.00 0000
25,460 52.2002-000
Winds
10384==199
164)
6602 1911
000-000
52.20000220
1561in00 1
20
OF7-308
SpanaGiPT
23
13 35)
25
6522
20
DSaoT
11525(2368)
2258
2978
000:0002
52
52.4801-.000
26.366
20.15
APPS
22
17.47-0127
7506 1-0 96
65.5 (0255
-0.55-0.766
XIP
2005
Wanda 7608:
3280 -6
6845662 19)
55.9988048
29
18 (.555
LLaMA2-01 20
SpanoGiPT
27:30-0055
104K
DSuoT
322.0
6191112005
56.3001025
19.00200220
693
17.66111.00
5495
10000000
192.0
Wonds 17.30001.531
410 300 500
47.01(=11 10)
LLMMAZ 138
SpunaGPT
1953.20.60
275 275 16.9)
20.04(10084)
40.1411:044)
48.2722111
26.31:20.55)
DSmdT
47:21:20.70
555 4.8)
47.01(50065)
27.19
ACPS
34.3
15L8 10 9)
(80
70220100000
$0.48 20.55)
29.3
20.5)
XIF
$72014
2022 201)
21170300000
2130
Winds
1168:3311
174 8 888)
121 22+-2)
55.25
12.03 0.43)
12.44-0.19)
LLaMA3
SpuneGiPT
43.15:2621
35 38.00222443)
02.4411:1.00
43.88
2147720777
DSauT
10K.:2233)
166.0:4000
114.2:4:221
31.60 55.5013.20
17.78.20.13)
Table 13: Performance analysis for ohe-shot unstructured pruning of OPT models and LLaMA models at 70% sparsity. (1): higher is better; (4): lower is better. We omit LLaMA results on LAMBADA due to its poor performance without modifications.
23

---

Midel
Aggontnn
Wikllet
PEW
LAMBADY
PRUNT
ARC
ARCCChileege
MP
800001-00
24 20
0.
Wanda
SpewsiP
3327(4412)
108564776
521 23 20
178362
311.0:277 4)
130K=145)
0.52646.05
591(1647)
55.7760008
55 555560050)
20 298114034
285004003000
15
18:00:40-244
18.66:0439
DSuT
79937
255
0.00
15.70
(5.59
4/2%
12672961.5
11.3 5055
Winda
(59862700)
4831-763
000000000
53 20
0P1-2778
SporeeiPi
156.2:11.20
734N624
DSwT
18323/
25
0000000000
(0
20-38:00277 2627660322
4/25
1..2
12.0
99 0.000
18-12-00.50
MP
2270
100000000
52 35
25 50000000
21,50
21,56 20000
Wands
4317(42083
2429 190)
2
54 27:50.53
15.25 20.377
OP7-6.78
SperwxiPT
10M 1295.9
92.51 (
56 1M 20)
14 TN + 780
A0 55
18 67C 00:20
DSnT
1099 2902 2902)
8165622179
2375
53. 136
25 754
20.17
4125
(3
22.
25 L35
62.4
44 23 500
50, 14: 00:36
2000
900573300
25.253330000
Wanda
6256
21 10
(0.47)
OPT-138
SpenwCiPT
26.8/16331
ALP5
DSmTT
11077:24500
42.9233005
5MO0L
2455
37.17
3L 1L3RS177 177
725
201
22-54 03
51584200)
25 0
Wanda
8184(2379))
2355
538
0000
$1.9860035
252 20 220
2007 10171
0P7-366
Speew<iPP
50.33:4581)
48. (+10))
10 1355
63 63.
4S1 1772590041
21 1100010.911
D8neT
1611)
5179
31
25 P3
ALPS
25 1.24
38 (664775577
19.99:1.144
44,66
697226061)
56.8 029
34.96 20.39)
MP
1500 (0)
20T0 10.00
Wanda
314012692)
3158
4788=
$2.07600.06
LLaMA2-7 7B
SpeneCIPT
1149:360)
$3.04133.11)
54.86630.90
44.8211885)
38 3022711020 20)
Wanda
11204120
55.0430.000
211192111
20 20665-99-111
12MA2-198
SparsCCPT
1008/340)
605.413364)
64.11222.25)
54.4419.53
25. 40:
26
DSeoT
(390
10634/2842)
13U001-9335)
$2.1120.30
26 26.738002500
22, 26.11
ALPS
3468/+2755) 5)
31.55+1..55
50.39
34. 34:19770777
23 22/00:00.321
MP
201419(1608)
26364((1000)
624815(:0000
52.7763000
24 24.77
20,6
Wands
1385 4408)
696 10-1000--1
52.804-0.49
270
20 172 10.400.35
11aMAS-BR
SperwciPP
19.9
342.3 (42.((1
(08.5
54 54.44+0.65
09867(00.3) 87)
18. 10.581
DSneT
106108-50
454.0(::7.44
52 $2.74630.300
28.01:20.20
19.09 20.40)
120.6
1589
61.8(13.358)
57.19150.48)
33.35(20.40)
16.86:20.58)
Table 14: Performance analysis for one-shot unstructured pruning of OPT models and LLaMA models at 80% sparsity. (1): higher is better; (4): lower is better. We omit LLaMA results on LAMBADA due to its poor performance without modifications.
Model
Ageriian
Wilfferr2
PTB
LAMBADA
PIQA
ARC-Bay
ARC Challere
XIP
Winsda
13657)
212022942)
780861-2527)
000010000
$3,99200491
2541:2054)
28.15:11.16)
OPT-11 30
SpaceGiPT
777111589)
5799/2585)
2761(1218)
000010.0))
24)
15.72
(4488122478)
1506R(22438)
0000100000
24.721-0.59
19:97100777)
4801)234)
2021
1485
0.10:00022
52.7310055
19.55-0091)
MP
112051200
0000000
000
2338 1500
Wanda
11282641736
0000-00000 00)
21.0101042)
OPTE2 70
SpesGGr
593(-550)
4516)-410)
2884 (13)
0076-0005
26.00-100
0.16
1800144 84215)
11289 10085)
HA52 11355)
0,0 10.09
S3
28 40g 0.56)
10.55
ALPS
1686 (265)
1599 (21)
MMKJI 42.00
6,2%
S2 (0-40)
26 55 0.56
MF
3130
0000
20122300
Wanda
13354
13676611459
0000-0000
52
19.433-074)
OPT66 70
Spesorr
(087741190)
1575)
0000-000
30.1511-.54
11739(14770)
28100-2000 2061)
WiTh (171--455)
0,00 000010.99
S312 -082
26 TM (.49)
10446110010
ALPS
6424 194)
4112 184)
2AMN (199)
601110.55
S2.35 10.8%
M. (034)
16.77)
MP
1000
Wanda
199474499633
00005000
20,0K
137-138
Spesorr
0000-000
4008 19557)
18442
000(10.09
1h (045)
21,8K 10777
ALPS
19020 (4999)
43429L (33117)
10001C12600 (617)
0/22 10:0)
52,8
0.56
MP
2000
Wanda
11122641990)
6063 1005)
0000500000
16
20.2001-0299
OPT:30B
SpesoGr
M095-10677
4966--311
4580
0400-0009
28,8
1000144 2001)
415TL- MA)
(566))
OOD (0.00
ALPS
1888 1213)
1992 19
6903 89.9)
9AR (0.1%)
SLAN (0.54)
20.28
(.46)
10
MP
1598 L10
1800 (0))
S0 10 (100)
14000
Winda
1082X 12400)
2850011499)
SL 11955
1005
21.57
LLaMAR-7B
SpaneGP
150061111
429.31:44.99
21.55
DSw/
15944(18055)
2123 (038)
15297(=13005 1305)
21.59 26336:00411
ALPS
11.7
1800 3373)
7
$3.01-0044 481
26
20.211
MP
JITONK 1:0
34
(100)
31 L0000
Winda
15922)
19851115184)
(05802110944
S2 22,00--0.25
21.15 1:0.22
13.MA2-13
SpaneGP
1443(113))
2651)2-482)
678.31 101.7)
$3.310-0.20 261
26:41::0360
21.35 1:0440
DSa/T
46515(131563
269151296160
2767363142623
$1,8550011 19)
21.91:20255 (0:5)
ALPS
25L 1966
7959624677
11.72
53.42:-0.33
20.47
19:00 20.300
MPP
1TESSIN 0.0
TEM969 (835)
13556 (100)
36.30 (L.))
21.77 1000)
Winda
1033110039
22425
5687L11693
S2. $2.31(-0-455
28 -0.33)
20.73 1.069)
LL-MA3-B
SpaneGP
110263637
(21)
464.5::22.44
13776110.33 331
27.52/-0449)
19:20 L:0655
35033490000
21943(1-5911)
14825/23597)
$3.00 03.05--1.577
20.8% 200365
ALPS
$15. 2)
5246:2309)
212.21:10222
54.47:20.54
28.98 -0.62
17.69
Table 15: Performance analysis for ohe-shot unstructured pruning OPT models and LLaMA models at 90% sparsity. (1): higher is better; (4): lower is better. We omit LLaMA results on LAMBADA due to its poor performance without modifications.
24

---

Model
Algorithm
Wibteed
PEB.
CK
LAMBADA
PIGNT
ARC
ARCCCallegge
MP
42:,000000
12000000
2380300000
Wands
28 28.44-00377
25. 25774-025)
65 65:944054)
47550(+044)
21 (L,0(4044))
OPT-138
SpaneCPPT
24.36-0020
32.0011.5
45
20,00110340 (4
DSarT
32994-0.71
3004:-035
45 55.00
30.75:10.54
22,6
20
56
MP
1155
975.3 000
184 000
U 1000
Wands
21. 21.55(00.50 50
02.510044 43)
20 20770018
33 (11
66 S1 54)
S1 60
21.71 50)
OPT-22705
SpanoCaPT
17445 16)
25 20 200
17
66 (0
S 4447780355 333
23, (3,04000.02 5E)
DSaoT
37.97 29.
03.04::025
56 30
67 (7)5744
49 (00
21,00:20.38
ALPS
21,17 1684--0299
(6)
49.9
(4)
70.99 10.9 24)
58 (4-57
23.40
SIP
205
(00000
Wands
1605/=0.14)
24
17
7L 15
56 43)
24.8)
OPT-667 7B
SpanoCarT
14.25-00100 16)
90 24)
14.01-1
0 22
22 55 84)
90 06-00
DSso'
16.204-0.10
25.321-0 34)
16
51
61
2057 10
00 17)
24.88110.52
ALPS
13.54:0.100 100
17.51: 19)
25
73 20255
61 6L83110550 56
26, (60.64)
MP
405.0_20.0)
379
00
60 00000
36 0000000
21 (1000
Wanda
05.69=0.111
71 14
27.32120.33
GPT-138
Sp
12.96=00.10
15885c0
13
75
73 (3
DSart
S454
2618810 63
ALPS
-2.25-0006)
15.2% 0008)
12.8--0 19)
62.88
78 +022
6 5656 +0446
29.78-10000 74)
MP
200
(0)0
3000
St 22000000
Wands
(32)(20.40)
1699
28)
14 14.6 18)
S2 55
24 84870 14)
31 409
25 29.25 57)
3PT:30B
SpaneCiPT
10.00-00007
14 01:02:00000 10)
12 12.04:0014 000
66
64.7000 499
75. 23
66.75
66755100555
51 3..332
DSarf
12:94-0 10)
13 0.5
96
(1)
74 +0000000
64 19
20 20.4
ALPS
1064:00000
(3755=0055)
15
67.
75.00
ML45
1480413:00
26 100
Wands
12 0000)
8602
-4005-0005)
70 1.10--111 15)
53 15
50.00+0.00 50)
LLaMM2 TB
SpaneGarT
0096
7597
12.04::025)
71 21.
39-991397722
29,88110.75)
DSauT
1266(=00.09
102.0e551)
1485(=0011
69. 60.31
55 55:00:50-42
26.01(10.277
ALPS
88
49 201
10.77:00422
SIP
8.89
00
0080-000
71 +000)
53 10 R00
Wands
9.00:0004)
8893
(1:22:0005)
73. 23.16
(4)
33.02++0337)
1LMMA2-138
SpaneCarr
8.66390 100
(8)
10000
DSaoT
9.11:30005
27.54
(1.40:0007))
72.301:::233
33.3110.63)
ALPS
8.14:20.10)
20.34
74.721-00555)
68.71199599
35 15.96:00.00)
XIP
2000
(00001200000
Wanda
24
44 14.9==10 12)
20
67. 67.5662:0.3 32)
26.11(20,70) 56.30130433
70)
ILMMAS-R
SpinoCiPT
16 35 18)
25 050 066))
18 71)
70, 70.54/+0000 709
61 (1)0)+077)
S1 (1(4+(0.90)
DSaoT
23.00 0.40
28.78120014)
(7.700200055)
56.40130.39 38)
25.00/10.41
2355
13.09
Table 16: Performance analysis for one-shot unstructured pruning of OPT models and LLaMA models at 2 : 4 sparsity. (t): higher is better; (4): lower is better. We omit LLaMA results on LAMBADA due to its poor performance without modifications.
Model
Alyyniten
WikiTowet
LAMBADA
ARC-Fasy
MP
20010=00
00:
000
7310:00001
Wands
22:0
32.41000.0
21,
21.54::000
190
67 1.8330000 600
5008130.39)
22.1((04455
GPT-1338
SpaneGiPT
20.25-00.3)
2 200
19.40
353
41.54(1099)
65 55)
43.01(20775)
21.85(10.44)
Disuf
35.00 232)
54)
24 16)
55.
67 -0.00
40. 00.00+0.00 54)
21
1.766+0033 56)
ALPS
19.49-0337)
18 24)
43.9CC11 10)
68.42
(2.07 72)
166.0-0000
195 00000
2000000000
05.290000000
20.73.10.00)
Wands
17040-048)
2.7B
SpansePPT
20.36 24)
15
20000
24.29110.38) (911055
DSauT
17.5
72
15
22.59 25
ALPS
14.5
0115
15
52.6 $2.6611108 05)
71.46:01199
56 56,91100.00
25.3:10.34)
SP
100
163.72000
100.002000
256 20,000
GC (3.0112900))
43 O0
20500
Wamls
13.6 12)
20.90-0 14)
14 0.0
57.75t OZ)
72 199900000000 39)
(50
D6. +0.4))
OPT-6715
Sp
126000008)
16650
32)
25)
26-5510141
DSaoT
19.0000 13.5%-0.11
13
58 33)
(6.25
ALPS
0231
24)
1M
38
MP
009
300 000
20
13000000
41 10
Wands
13.37
18.5640
200
(3
S7. 77)
73 200
27 54(10.02
OPT-138
Sp
11.77 (1.79-000)
14 70 11)
13
12.54-0100 10)
23 1- 3T)
20 56/10. 59)
DSaoT
16410201000 13.77200100
10)
(3
60
72 (0L3:004)
20244
ALPS
01337-000
14337
12.22
6446 54
30.51630.00
1487C10)
1000)
1000
Wanda
10.7-0008)
04077
03
01.4110050 599
66 66001000400 46)
30.40110.41)
OPT:308
SpineCiPT
10:000000)
13.35:0115)
11.33:0009)
40.0
76 25
67 (8)
32.17 58)
0000
0009(-004))
12 0.166)000)
64.50110355
55)
75 15.401:0115)
30.58/140.6)
APPS
1034-005)
1818 11)
11.30:0000
67 RSN +0 40)
16.6110000 500
67.8 +025
SIP
13000-0000
26 000
Wands 8A3350000
23.9405001
62,00
1LMMA2-B TB
SpaneCPP
3.46120.03
45.0x=1.13)
14.44110035)
13.00
33.52/10.66)
DSaoT
9.50150.011
4800=1100 (2)
10.00(200001
73. 23.34
43)
3056/11..5 34)
ALPS
8.12:50.00
51.21(=2788
9.20
74.78
33.20/+0.500 (6)
MP
7.32/50.00
137.3
0.100
63. 03.00000000 74430100000
15.00)
Wands
7000+00003
55.07-0 79)
90.00)
75 6R 51)
67 17,0000011 13)
S8 260 56)
LLAMA2 13B
SpaneCGPT
7.01:2000)
47.64:2031)
75 (5.841)).20)
08.74
56,0 7.3330000
8000)
75 (4.4)
ALPS
6.81:0007)
42.55:0
H,23
(6.577
69.1110.99)
MP
00
000
174.20:000
65. 1111000
53.1110:0001
20.
Wands
04.5220008)
34.26-0.00 10)
71. 71,33==0.30
6491(50039)
(4.03/90.42)
LLMMA-S0
SpaneCaPT
17 131
19)
14 14
23. 13.2088070 (4)
09.
$4.60300000
DSar
14 14964-009)
2
20)
BK -004
71 29)
65 65,65+00220 23)
01.52(+0681)
ALPS
11.5%:20.12)
1377600-47) 1657:-035)
74.57
(965110061
37.300:1..99
Table 17: Performance analysis for one-shot unstructured pruning of OPT models and LLaMA models at 4 8 sparsity. (t): higher is better; (4): lower is better. We omit LLaMA results on LAMBADA due to its poor performance without modifications.
25

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: At the end of the introduction, we include paragraph that clearly states the contributions and scope of our paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We briefly discuss the limitation of work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
26

---

We clearly state all assumptions in Theorem and provide a formal proof in Appendix A
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We thoroughly describe our proposed pruning approach Algorithms and and we include all other details necessary for reproducing the results in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
27

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Our code is available httptpps//gthhhb..///aaaaeeeeeea
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.co
hhbllic/gulless/oddeebbmm.ssss wtt.SSubmmmsiioooooooooooo. for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ccppp..cc/pulli///gg.. guwdes/ddeeeee CooeeSumeeieeeeeeeeeee ssi onyolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide detailed settings of our proposed pruning framework and its competitors used in our experiments in Appendix B.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: our experiments, for each combination of model, pruning method, and sparsity level, we ran 5 different seeds and reported the mean and standard deviation of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
28

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details of the computational resources used for our experiments in Appendix B.I
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics / http://wwwrp....coo.u.....aaccaa
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics and ensured that the research conducted in this paper fully complies with it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
To the best of our knowledge, our work has no societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
To the best of our knowledge, our work poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: At the beginning of Section 4 cite all the datasets and models used in our experiments.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
31

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "4LJShjG82h", "reviewer": "Reviewer_ZgXD", "summary": "This paper introduces ALPS, a novel optimization-based framework for one-shot unstructured pruning of LLMs. The key contributions are:\n\n- Formulating LLM pruning as an l0-constrained optimization problem solved using operator splitting (ADMM).\n- A penalty parameter update scheme to accelerate convergence.\n- A post-processing step using preconditioned conjugate gradient (PCG) to refine weights.\n- Theoretical convergence guarantees for the proposed algorithm.\n\nThe authors demonstrate that ALPS outperforms state-of-the-art pruning methods, especially at high sparsity levels, on various LLMs including OPT and LLaMA models.", "questions": "1. How does the runtime of ALPS compare to other methods, especially for very large models?\n2. Can (how can) ALPS be extended to handle other types (structured sparsity) of structured sparsity patterns beyond N:M?", "limitations": "The authors discussed some limitations of their work e.g. extending ALPS to incorporate structured pruning constraints and quantization.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method directly addresses the pruning problem without relying on heuristics.\n2. The ADMM algorithm introduced in the paper comes with theoretical convergence guarantees, adding reliability to the approach.\n3. The paper includes comprehensive experiments on large-scale models (up to 30B parameters), demonstrating consistent improvements over existing methods.", "weaknesses": "1. It would be beneficial to include evaluations on more recent models such as LLaMA-3, instruction-tuned models, and extremely large models (70B+ parameters) to demonstrate the method's applicability to the latest advancements in the field.\n2. The current evaluation tasks are relatively limited. Expanding the evaluation to include knowledge-intensive tasks (e.g., MMLU) and reasoning-based tasks (e.g., GSM8K and HumanEval) would provide a more comprehensive assessment of the method's effectiveness.\n3. The current focus is primarily on N:M sparsity patterns. It would be great to explore structured pruning, which could lead to more significant improvements in inference speed, making the approach more versatile and practical for real-world applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their thoughtful feedback. Below, we provide some answers/clarifications.\n\n**It would be beneficial to include evaluations on more recent and extremely large models to demonstrate the method's applicability. The current evaluation tasks are relatively limited. Expanding the evaluation to include knowledge-intensive tasks and reasoning-based tasks would provide a more comprehensive assessment.**\n\n**Reply:** We appreciate the reviewer's valuable suggestion. We will consider comparing ALPS with other methods on the recent model LLaMA-3 and broaden our assessment by incorporating knowledge-intensive and reasoning-based tasks, specifically MMLU and GSM8K. As a side remark, given our academic resource constraints (we conduct our experiments on a V100 GPU with 32GB memory), the largest model we’ve tried in the paper is OPT-30B. \n\n**The current focus is primarily on N:M sparsity patterns. It would be great to explore structured pruning. Can ALPS be extended to handle other types (structured sparsity) of structured sparsity patterns beyond N:M?**\n\n**Reply:**  While we focus on unstructured sparsity and N:M sparsity, our approach can be generalized to handle other structured sparsity patterns, as discussed below. \n\nLet $S$ denote the set of weight matrices with the desired structured sparsity pattern (in the case of unstructured sparsity, $S$ contains all matrices with no more than $k$ non-zero elements). Our method can be extended to find the optimal matrix in $S$ by modifying the $\\mathbf{D}$-update step in Eq.(4) in our paper. We can set $\\mathbf{D}^{(t+1)}$ to be the projection of $\\mathbf{W}^{(t+1)}+\\mathbf{V}^{(t+1)}/\\rho$ onto $S$. Our approach remains efficient and can find high-quality solutions for sparsity patterns with low-cost projection operations, including:\n+ Block sparsity [1]\n+ Hierarchical sparsity [2] \n+ Row sparsity [3]\n\nImportantly, our convergence results (Theorem 1) and proposed PCG step directly apply to these structured sparsity patterns as well.\n\nWe thank the reviewer for bringing up this point, and we will be happy to discuss these extensions in our revision.\n\n[1] Gray, S., Radford, A., & Kingma, D. P. GPU kernels for block-sparse weights.\n\n[2] Wu, Y. N., Tsai, P. A., Muralidharan, S., Parashar, A., Sze, V., & Emer, J. Highlight: Efficient and flexible DNN acceleration with hierarchical structured sparsity.\n\n[3] Meng, X., Ibrahim, S., Behdin, K., Hazimeh, H., Ponomareva, N., & Mazumder, R. OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimization.\n\n\n**How does the runtime of ALPS compare to other methods, especially for very large models?**\n\n**Reply:**  We compare the runtime of ALPS with other methods in the following table. Here, the runtime includes the time for generating input activations $\\mathbf{X}$.\n\n|Method| OPT-1.3B | OPT-2.7B | OPT-6.7B | OPT-13B | OPT-30B | LLaMA-7B|LLaMA-13B|\n|-|-|-|-|-|-|-|-|\n| MP       |4.7|8.9|23|47|120|25|46|\n| Wanda    |99|161|280|502|1027|214|407|\n| DSnoT    |125|213|417|758|1528|347|651|\n| SparseGPT|363|728|1621|2980|6662|1263|2319|\n| ALPS     |963|2360|6069|14323|48366|3043|7145|\n| ALPS-simple|297|599|1375|2470|5021|1013|2595|\n\n\n|Method |MP | Wanda | DSnoT |SparseGPT | ALPS  | ALPS-simple|\n|-|-|-|-|-|-|-|\n|**Loss**|1.98e-1|2.26e-1|2.06e-1|8.54e-2|5.32e-2 |7.52e-2 |\n\nALPS employs an advanced optimization method to solve the layerwise reconstruction problem, which results in longer running times compared to other algorithms. However, it's important to note that ALPS's runtime is still negligible when compared to fine-tuning methods for LLMs. For context, Wanda [1] reports that fine-tuning LLaMa-7B with LoRA takes about 24 hours on a V100 GPU, while full parameter fine-tuning takes 24 days. In contrast, ALPS prunes the model in less than 1 hour.\nFurthermore, in our paper, ALPS is run with a tight convergence criterion. We can improve its runtime further by using a variant, ALPS-simple, which uses a looser convergence criterion. As shown in the above tables, ALPS-simple achieves shorter running times compared to SparseGPT while still maintaining better utility, as measured by single-layer reconstruction loss. We are happy to include the performance results of ALPS-simple in the revised paper if it is of interest to the reviewers.\n\n[1] Sun, M., Liu, Z., Bair, A., & Kolter, J. Z. A simple and effective pruning approach for large language models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YE2bqJlz8d", "reviewer": "Reviewer_xarH", "summary": "This work presents an LLM pruning framework that formulates the problem as finding a sparse weight matrix to reconstruct the layer-wise activations. This work incorporates the operator splitting technique and preconditioned conjugate gradient methods to solve the pruning problem. Experiments demonstrate that the proposed method achieves better single-layer reconstruction error and improved performance on downstream tasks.", "questions": "It would be better to compare with other representative pruning methods beyond the one-shot unstructured pruning, and to explore how this method could be integrated with others.\n\nThis work discusses engineering optimizations, such as incorporating PCG with GPU. Will you open-source the code or provide more efficient results from these engineering efforts?", "limitations": "There is no section for limitation, but the author discusses some in the conclusion. One limitation is the setting of layer-wise activation reconstruction. Recent works, especially the activation-aware pruning methods, reported that activations are not equally important.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This work is motivated by a clear theoretical rationale.", "weaknesses": "Experiments could be more solid since many experiments are run on OPT, which is somewhat out-of-date. It would also be better to consider more challenging benchmarks, such as GSM8K or other questions that require generating a long answer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their thoughtful feedback. Below, we provide some answers/clarifications.\n\n**Experiments could be more solid since many experiments are run on OPT, which is somewhat out-of-date. It would also be better to consider more challenging benchmarks, such as GSM8K or other questions that require generating a long answer.**\n\n**Reply:** We appreciate the reviewer's valuable suggestion. Based on your comment and that of reviewer ZgXD, we will consider comparing ALPS with other methods on the recent model LLaMA-3 and broaden our assessment by incorporating knowledge-intensive and reasoning-based tasks, specifically MMLU and GSM8K.\n\n**It would be better to compare with other representative pruning methods beyond the one-shot unstructured pruning, and to explore how this method could be integrated with others.**\n\n**Reply:** Thank you for this insightful suggestion! Beyond unstructured pruning, numerous representative pruning methods focus on structured sparsity patterns, such as block sparsity [1] and row sparsity [2]. Though direct numerical comparisons with these methods are potentially challenging due to their different settings, the idea of integrating ALPS with structured pruning methods is intriguing. Having a pruned model with *both* unstructured and structured sparsity could optimize both storage costs and inference time. It would be certainly interesting to explore such an integration as future work.\n\n[1] Gray, S., Radford, A., & Kingma, D. P. GPU kernels for block-sparse weights.\n\n[2] Meng, X., Ibrahim, S., Behdin, K., Hazimeh, H., Ponomareva, N., & Mazumder, R. OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimization.\n\n\n**This work discusses engineering optimizations, such as incorporating PCG with GPU. Will you open-source the code or provide more efficient results from these engineering efforts?**\n\n**Reply:** If the paper is accepted, we will release the codes. Additionally, we will provide explanations of such implementations and a tutorial on their usage.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NqMgODClOA", "reviewer": "Reviewer_1PwE", "summary": "The paper introduces ALPS, an optimization-based framework for one-shot pruning of large language models (LLMs). ALPS leverages an ADMM-based algorithm with operator splitting and preconditioned conjugate gradient methods to achieve improvements in sparsity and perplexity over state-of-the-art methods, particularly in high-sparsity regimes.", "questions": "Please refer to my points listed in the weakness section. Besides those,\n\n1. Could you provide a clearer explanation of the practical implications of your theoretical contributions?\n\n2. I encourage the authors to report the overall running time per epoch for ALPS in comparison with Wanda and DSnoT.\n\n3. Could you provide the perplexity of the dense models (without any pruning) in all the tables in the paper?\n\n4. How does ALPS's speed of convergence compare with the referenced methods, and what practical benefits does the novel penalty parameter update scheme offer?", "limitations": "The authors have not thoroughly discussed the limitations and potential negative societal impacts. They should address the limited novelty compared to existing ADMM-based methods and discuss the risks and mitigation strategies for the misuse of more accessible powerful models.", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**S1.**\nALPS achieves substantial reductions in test perplexity and improved zero-shot benchmark performance for highly sparse models.\n\n**S2.**\nProvides theoretical convergence guarantees for $\\ell_0$-constrained optimization problems with ADMM solver.\n\n**S3.**\nImplements efficient post-processing techniques via conjugate projected gradient, enhancing computational performance.", "weaknesses": "**W1.**\nImportant Reference Missing: The paper does not cite \"Fast and optimal weight update for pruned large language models\" by Boža, which addresses a similar problem using an ADMM-based optimization algorithm. This omission is significant as both papers share highly similar problem definitions and solutions.\n\n**W2.**\nLimited Novelty: ALPS closely resembles methods discussed in both \"Fast and optimal pruning\" and \"Progressive weight pruning of deep neural networks using ADMM.\" The primary difference is ALPS's specific application to LLMs. However, this differentiation might not be substantial enough to establish ALPS as a novel contribution. Btw, the reference of \"Progressive weight pruning of deep neural networks using ADMM.\" seems also missing in the paper.\n\n**W3.**\nPerformance at High Sparsity: At very high sparsity levels, ALPS’s perplexity remains significantly higher than the dense model. This indicates that the pruned LLMs by ALPS may still perform poorly and being practically useless, although they are better than those pruned by comparison methods. \n\n**W4.**\nUnfair Comparison: The comparison with methods like Wanda and DSnoT, which do not involve retraining after pruning, is solely based on perplexity. This is unfair because it overlooks the overall running time per iteration/epoch. ALPS’s performance should be compared with these methods by considering both perplexity and computational efficiency to provide a more balanced evaluation. In addition, I believe if Wanda or DSnoT is combined with some re-training techniques, they can achieve a much lower perplexity as well.\n\nReference:\n\nBoža, Vladimír. \"Fast and optimal weight update for pruned large language models.\" arXiv preprint arXiv:2401.02938 (2024).\n\nYe, Shaokai, et al. \"Progressive weight pruning of deep neural networks using ADMM.\" arXiv preprint arXiv:1810.07378 (2018).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reply to W1 and W2:** Thanks for these references—we will include them in our revised paper. ALPS differs significantly from the mentioned works as follows:\n\n+ Comparison with \"Progressive weight pruning of deep neural networks using ADMM\":\nThis paper applies ADMM to the original loss function, requiring expensive full network training via SGD. In contrast, ALPS performs post-training pruning, applying ADMM to layerwise reconstruction problems (least squares objective) with cardinality constraints. This approach enables us to scale ALPS to 30B parameters on a single V100 GPU.\n\n+ Comparison with \"Fast and optimal pruning\" (Boža):  In this interesting work, Boža performs pruning post-training using a layerwise reconstruction loss. They apply ADMM to solve a least squares problem for a *given* sparsity mask.  Boža selects the sparsity mask via iterative magnitude pruning. Using ADMM speeds up the back-solve procedure. \n\n     + ALPS, in contrast, is an end-to-end approach that directly targets the $\\ell_0$ constrained least squares problem with theoretical guarantees. **ALPS simultaneously optimizes over the weights and the sparsity pattern, unlike the work by Boža where ADMM is used to perform the backsolve operation.**  Therefore, ALPS achieves lower layerwise reconstruction loss than Boža, as shown in Table 1 in the general rebuttal.\n\n     + Since Boža employs ADMM for backsolving [finding the weights of the least squares problem given a mask], we also compare it with our PCG procedure, which serves as a backsolve method. We tested both approaches for computing weights on a given support. Results show that PCG outperforms Boža’s ADMM procedure in both computational time and objective value (see Table 2 in general rebuttal). The time advantage of PCG stems from its ability to backsolve without explicitly computing matrix inverses.\n\n+ Novelty of ALPS: \nPlease note that ALPS has several differences/innovations compared to standard ADMM:\n     - A novel penalty parameter updating scheme that enables finding high-quality support and accelerates convergence. As we show in our response to Q1 and Q4, in practice, our proposed ADMM has better convergence properties compared to the usual version of ADMM for $\\ell_0$-constrained least squares problems.\n    - As far as we know, our theory is the **first** theoretical convergence result for ADMM applied to $\\ell_0$-constrained problems.\n    - A PCG-based post-processing technique optimized with vectorization and GPU parallelism, achieving up to 200x speedup compared to direct backsolve for updating weights on a given support. \nThese contributions, taken together, make our work significantly different from existing ADMM methods especially in the area of LLM pruning.\n\n**Reply to W3:** Please note that our study demonstrates ALPS's effectiveness as a pruning method across all sparsity levels, including 60% sparsity and N:M sparsity pattern. In these cases, the pruned models maintain competitive performance.\n\nYou are right that model utility can deteriorate with one-shot pruning at very high sparsity levels — in such cases, we can recover model performance via retraining. Since ALPS-pruned models outperform models pruned by competing methods, they would require fewer retraining epochs to recover lost performance [1]. Consequently, ALPS remains valuable at high sparsity levels by reducing computational costs during the expensive retraining phase. \n\n[1] Meng, X., Ibrahim, S., Behdin, K., Hazimeh, H., Ponomareva, N., & Mazumder, R. OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimization.\n\n**Reply to W4:** Thanks for your thoughtful comment. In our response to reviewer ZgXD, we report the running time of ALPS and other one-shot pruning methods. We emphasize that with retraining, the total runtimes of Wanda and DSnoT become much higher than ALPS. As reported in Wanda, fine-tuning LLaMa-7B with LoRA takes about 24 hours on a V100 GPU, and full parameter fine-tuning takes 24 days. In contrast, ALPS prunes the model in less than 1 hour. As a layer-wise pruning method, ALPS loads weights and activations onto the GPU layer by layer, enabling it to prune models as large as OPT-30B on a single V100 GPU, while retraining often requires more careful memory management. Given these considerations, we think that retraining methods would be much more computationally expensive than ALPS. Please recall that our focus here is on demonstrating the effectiveness of ALPS as a one-shot pruning method.\nAs a side note, as we explained in our reply to ZgXD, the runtime of ALPS can be further decreased by loosening the convergence criteria of ALPS.\n\n**Reply to Q1 and Q4:** Our proposed $\\rho$ update scheme, theoretically supported by Theorem 1, ensures that ALPS converges rapidly while finding high-quality solutions. In contrast, ADMM with a fixed $\\rho$ may fail to converge when applied to $\\ell_0$ constrained least squares problems. To demonstrate this, we compared ALPS with ADMM using fixed $\\rho$ values. We examined two key metrics: reconstruction loss (objective) and the rate of change of the support (of weights) between consecutive iterations (this measures the convergence speed of the algorithm). The results are presented in the general rebuttal (Tabl 3 and 4). Results reveal that ADMM with a large $\\rho(=3)$ converges quickly but yields poor solutions, while a small $\\rho(=0.3)$ fails to converge. ALPS, utilizing our $\\rho$ update scheme, achieves both rapid convergence and high-quality solutions.\n\n**Reply to Q2:** Thanks for your comment — please refer to our response to reviewer ZgXD.\n\n**Reply to Q3:** Please refer to Table 5 in general rebuttal. In the table, we omit LLaMA results on LAMBADA due to its poor performance without modifications.\n\n---\n\nIn light of the above discussions and results, we kindly ask the reviewer to consider increasing their evaluation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their thoughtful comments. In response to their comments, we have conducted additional numerical comparisons and included additional discussions in relation to the existing work. We present a summary below:

+ Comparison with existing methods & Novelty [see Ref 1PwE]: We've clarified key differences between ALPS and prior works on ADMM. In particular, we explain how our approach differs from a direct application of ADMM, and results in better performance. We’ve provided supporting numerical evidence. 

+ New experimental results. Based on referee feedback,  we perform additional numerical experiments in the rebuttal:
   
   - [see Ref 1PwE] As a backsolve method, our proposed PCG outperforms Boža’s ADMM method in both computational time and objective value. ALPS also achieves lower layerwise loss compared to Boža. 
   - [see Ref 1PwE] Our proposed parameter updating scheme enables ALPS to converge rapidly while finding high-quality solutions. In contrast, standard ADMM may fail to converge.
   - [see Ref ZgXD, 1PwE] While ALPS is already quite efficient, we can further accelerate ALPS by using a loose convergence criterion. This results in shorter running times compared to SparseGPT while still maintaining better utility.

+ Potential extensions [see Ref ZgXD]: We discuss how ALPS can be generalized to handle other structured sparsity patterns, including block, hierarchical, and row sparsity. 

 ---

We provide some tables here for space reasons. We refer to these tables in our response to the reviewer 1PwE.  

**Table 1:** The layerwise reconstruction loss on a single layer.

| Sparsity  | 0.4|  0.5 | 0.6  | 0.7 | 0.8 | 0.9|
|-|-|-|-|-|-|-|
|ALPS|3.55e-3|7.56e-3|1.47e-2|2.77e-2|5.32e-2|1.13e-1|
|Boža|4.47e-3|9.53e-3|1.83e-2|3.36e-2|6.19e-2|1.25e-1|

**Table 2:** The layerwise reconstruction loss and runtime for applying PCG and Boža to find the optimal weights on a given support.

| Sparsity  | 0.4|  0.5 | 0.6  | 0.7 | 0.8 | 0.9|
|-|-|-|-|-|-|-|
|ALPS-PCG(Loss)|5.0e-3|1.12e-2|2.27e-2|4.39e-2|8.51e-2|1.78e-1|
|Boža(Loss)|5.1e-3|1.12e-2|2.32e-2|4.49e-2|8.83e-2|2.00.e-1|
|ALPS-PCG(Time)|0.01s|0.01s|0.01s|0.01s|0.01s|0.01s|
|Boža(Time)|0.11s|0.11s|0.11s|0.11s|0.11s|0.11s|

**Table 3:** The reconstruction loss (objective) over iterations, comparing ALPS with ADMM using a fixed penalty parameter $\rho$.

| Loss/Iter   |5 | 10  | 20  | 30| 50| 100|
|---|---|---|---|---|---|---|
| ALPS |1.63e-1 | 1.28e-1  | 5.95e-2  | 5.32e-2  |5.31e-2  |5.31e-2  |
| ADMM($\rho=0.3$)  | 7.83e-2  | 7.55e-2  | 7.50e-2 |  7.47e-2 | 7.47e-2 | 7.45e-2  |
| ADMM($\rho=3$)  |  9.32e-2 | 8.18e-2  |7.64e-2 |  7.53e-2 | 7.45e-2   |7.42e-2 | 

**Table 4:** The rate of change of the support (of weights) between consecutive iterations, comparing ALPS with ADMM using a fixed penalty parameter $\rho$.

| Supp change / Iter   |5 | 10  | 20  | 30| 50 | 100|
|---|---|---|---|---|---|---|
| ALPS |20.2% | 17.0% | 2.8% | 0.0% | 0.0% | 0.0% |
| ADMM($\rho=0.3$)  | 6.4%  | 7.0%  | 7.0%  |7.0%  | 6.9%| 6.9%|
| ADMM($\rho=3$)  | 0.2%  |  <0.1% | <0.1%  |<0.1%   | <0.1% | <0.1% |

**Table 5:** Perplexity and zero-shot evaluation results for dense models.

| Model |  WikiText2↓| PTB ↓ | C4 ↓ | LAMBADA↑ |PIQA↑ |ARC-Easy↑| ARC-Challenge↑|
|---|---|---|---|---|---|---|---|
| OPT-1.3B  |14.63|20.29|16.07|58.80|72.36|50.93|29.44|
| OPT-2.7B  |12.47|17.97|14.34|64.82|74.81|54.34|31.31|
| OPT-6.7B  |10.86|15.77|12.71|68.72|76.39|60.14|34.56|
| OPT-13B   |10.13|14.52|12.06|70.23|76.88|61.83|35.75|
| OPT-30B   |9.56 |14.04|11.44|72.39|78.18|65.40|38.14|
| LLaMA-7B  |5.47|22.51|6.97|-|78.29|69.23|39.93|
| LLaMA-13B |4.88|28.87|6.47|-|78.78|73.27|45.56|

Meta-review: This paper proposed a large language model (LLM) pruning framework, which turns the problem into  layer-wise activation reconstruction problem via sparse weight matrix. The operator splitting technique and preconditioned conjugate gradient methods are deployed and the empirical studies show the better single-layer reconstruction error and correspondingly the improved performance on downstream tasks. 

Overall the reviewers like the theoretical convergence guarantees and the practical applicability of the proposed method. There has been some major concerns from reviewer 1PwE, but after the rebuttal some of them have been addressed. We highly encourage the authors to incorporate the additional artifacts into the revisions to make the paper more solid.
Decision: Accept (poster)