ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models

Authors: xiang  meng
kayhan  behdin
haoyue  wang
rahul  mazumder
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 17:14:40

Abstract: The impressive performance of Large Language Models (LLMs) across various natural language processing tasks comes at the cost of vast computational resources and storage requirements. One-shot pruning techniques offer a way to alleviate these burdens by removing redundant weights without the need for retraining. Yet, the massive scale of LLMs often forces current pruning approaches to rely on heuristics instead of optimization-based techniques, potentially resulting in suboptimal compression. In this paper, we introduce ALPS, an optimization-based framework that tackles the pruning problem using the operator splitting technique and a preconditioned conjugate gradient-based post-processing step. Our approach incorporates novel techniques to accelerate and theoretically guarantee convergence while leveraging vectorization and GPU parallelism for efficiency. ALPS substantially outperforms state-of-the-art methods in terms of the pruning objective and perplexity reduction, particularly for highly sparse models. On the LLaMA3-8B model with 70\% sparsity, ALPS achieves a 29\% reduction in test perplexity on the WikiText dataset and a 8\% improvement in zero-shot benchmark performance compared to existing methods. Our code is available at https://github.com/mazumder-lab/ALPS.

Main paper: 
 introduction: 
 Large Language Models (LLMs) have revolutionized the field of natural language processing, demonstrating remarkable performance across a wide spectrum of tasks, from question answering and text generation to sentiment analysis and named entity recognition [Wei et al., 2022 , Bubeck et al., 2023 , Achiam et al., 2023] . The success of LLMs can in part be attributed to their massive scale-state-ofthe-art models like OPT-175B [Zhang et al., 2022a] and LLaMA3 [Dubey et al., 2024] have hundreds of billions of parameters. However, this enormous size comes at a steep cost in terms of storage and computational resources. For instance, the OPT-175B model requires at least 320 GB of memory to store its parameters in half-precision (FP16) format, necessitating the use of multiple high-end GPUs for inference [Frantar and Alistarh, 2023] . To make LLMs more accessible and efficient, considerable efforts have been made to compress these models, with a particular emphasis on model quantization techniques [Lin et al., 2023 , Behdin et al., 2023 , Dettmers et al., 2023] .Network pruning [LeCun et al., 1989 , Hassibi and Stork, 1992 , Han et al., 2015] , a complementary approach to quantization, has received comparatively less attention in the realm of LLMs. Pruning aims to reduce the model size by identifying and removing redundant or less important weights, resulting in a sparser and more efficient network. Traditional pruning methods rely on iterative retraining to recover accuracy after each pruning stage [Han et al., 2015 , Luo et al., 2017 , Molchanov et al., 2016 , Liu et al., 2018] , which can be computationally expensive and time-consuming. To address this, recent research has focused on one-shot pruning methods [He et al., 2017, Singh and Alistarh, 2020 ] that compress a pre-trained model using only a small amount of data (e.g., a few thousand samples)-the key idea here is to perform pruning while retaining model accuracy as much as possible without expensive fine-tuning/retraining on the entire dataset. Many prior works [Frantar and Alistarh, 2022 , Yu et al., 2022 , Benbaki et al., 2023] on one-shot pruning address such pruning-accuracy tradeoffs using optimization based approaches.Despite the progress made in one-shot pruning, the massive scale of LLMs poses additional challenges, as many one-shot pruning methods designed for vision models cannot be directly applied due to their large model sizes. To overcome this, existing LLM pruning methods often rely on heuristic approaches to prune instead of solving optimization problems. For instance, SparseGPT [Frantar and Alistarh, 2023] approximates the OBS [Hassibi and Stork, 1992] algorithm by employing partial weight updates and adaptive mask selection to reduce costly Hessian computation. Similarly, Wanda [Sun et al., 2023] prunes weights based on the product of their magnitudes and corresponding input activations. Zhang et al. [2023] propose to iteratively grow and prune the weight mask according to the change in reconstruction error achieved by each update. While these heuristics enable pruning at scale, they may lead to suboptimal compression (and hence, suboptimal compression-accuracy tradeoffs) compared to advanced optimization-based approaches, as we show in this paper.In this paper, we propose ALPS1 , an optimization-based framework for one-shot LLM pruning. ALPS consists of two key components. First, it formulates pruning LLMs as an ℓ 0 -constrained optimization problem and solves it directly using the operator splitting technique (i.e., ADMM) [Boyd et al., 2011, Davis and Yin, 2016] without any simplification. The proposed algorithm simultaneously finds the support2 of the weights and updates them. After the support stabilizes, ALPS fixes the support and employs preconditioned conjugate gradient (PCG) [Nocedal and Wright, 1999, Section 5] to compute the optimal weights on the support. Our modified PCG leverages sparse matrix structure (arising from pruning) and GPU computation to solve large systems efficiently, providing a significant speed advantage over direct matrix inversion. An outline of our proposed optimizationbased framework ALPS is given in Figure 1 . Compared to previous heuristics, ALPS offers higher quality supports and weights, as demonstrated in Section 4.1. This improvement translates to a better performance of the pruned model compared to existing methods, particularly in the challenging high-sparsity regime. The pruning problem with a layerwise reconstruction objective and an ℓ0 constraint on the weights (Section 3.1). (Middle) ADMM with a ρ-update scheme (Algorithm 1) is employed to determine high-quality support for the weight matrix W (Section 3.2). (Right)The optimization problem is restricted to the obtained support, and a modified PCG method (Algorithm 2) is used to solve for the optimal weight values within the support (Section 3.3).Contributions. Our technical contributions are:1. We introduce ALPS, a novel one-shot LLM pruning framework that formulates an ℓ 0 -constrained optimization problem with a layer-wise reconstruction objective. By extending the operator splitting technique (i.e., ADMM) to this non-convex, non-continuous problem, ALPS simultaneously finds a high-quality support and updates the weights on the support. This approach leads to improvements over state-of-the-art heuristics in terms of the pruning objective. Furthermore, we provide theoretical convergence guarantees for our proposed algorithm, which, to the best of our knowledge, is a novel convergence result for ℓ 0 -constrained problems. 2. We further enhance the performance of ALPS through two techniques. First, we design a novel penalty parameter updating scheme that enables ALPS to find better support and accelerates its convergence. Second, we propose a post-processing technique to further improve the performance of the pruned models-we fix the support determined by ADMM and optimally solve the resulting quadratic problem using the PCG method. We utilize vectorization to solve the problem in a single pass and leverage GPU parallelism to further accelerate PCG. Our proposed method achieves a 20x-200x speedup compared to the vanilla backsolve approach. 3. ALPS substantially improves upon state-of-the-art methods for one-shot unstructured pruning of LLMs. For the LLaMA3-8B model with 70% sparsity, ALPS achieves a 29% reduction in test perplexity on the WikiText dataset and a 4%-13% improvement in performance on zero-shot benchmark evaluations. We also adapt ALPS to the popular N:M sparsity format [Zhou et al., 2021] and observe a 3%-10% higher performance compared to existing methods. Our code is publicly available at: https://github.com/mazumder-lab/ALPS. 
related work: 
 Network pruning. Network pruning is a well-established technique for reducing the complexity of deep neural networks by removing redundant weights [LeCun et al., 1989 , Han et al., 2015] .Pruning methods can be classified based on the structure of the resulting sparse network and the training requirements. In terms of structure, pruning can be categorized into unstructured pruning, which removes individual weights [Han et al., 2015 , Guo et al., 2016] , and structured pruning, which removes entire structures such as channels, filters, or attention heads [Lebedev and Lempitsky, 2016 , Wen et al., 2016 , Voita et al., 2019 , El Halabi et al., 2022] . Unstructured pruning offers better flexibility and higher sparsity levels but requires specialized hardware for acceleration, while structured pruning is more hardware-friendly but may suffer from larger performance loss. Based on the training requirements, pruning methods can be classified into three categories: (i) one-shot pruning, which directly removes weights from a pre-trained model without further training [Gale et al., 2019 , Frantar and Alistarh, 2022 , Meng et al., 2024a,b] , (ii) gradual pruning, which begins with a pre-trained model but alternates between pruning and fine-tuning via SGD to recover performance [Molchanov et al., 2016 , Zhu and Gupta, 2017 , Blalock et al., 2020 , Kurtic et al., 2022] , and (iii) training from scratch, where the model is trained from randomly initialized weights, and the sparse network structure is either determined before training or evolves during the training process, [Mocanu et al., 2018 , Dettmers and Zettlemoyer, 2019 , Evci et al., 2020 , Kusupati et al., 2020 , Chen et al., 2021] . In this paper, we focus on one-shot unstructured pruning.Post-training unstructured pruning. Based on their pruning objectives, there are three types of post-training unstructured pruning methods: (i) Importance-based methods, which assign a score to each weight (e.g., its absolute value) to assess its significance and decide whether it should be eliminated [Han et al., 2015 , Lee et al., 2018 , Molchanov et al., 2019 , Sun et al., 2023] . (ii) Second-order techniques, which consider a local quadratic approximation of the loss function around the pre-trained model and remove weights based on their influence on the loss [Hassibi and Stork, 1992 , Singh and Alistarh, 2020 , Yu et al., 2022 , Benbaki et al., 2023] . These approaches employ the empirical Fisher information matrix to estimate the Hessian matrix efficiently. (iii) Layer-wise pruning algorithms, which adapt OBS [Hassibi and Stork, 1992] framework to the layer-wise reconstruction objective [Dong et al., 2017 , Frantar and Alistarh, 2022 , 2023] . These methods prune each layer separately to address the computational challenge of calculating the full Hessian required in OBS. This work considers layer-wise reconstruction error as the pruning objective.Unstructured pruning in LLMs. While pruning algorithms designed for convolutional networks [Singh and Alistarh, 2020 , Chen et al., 2020 , Frantar and Alistarh, 2022] can be readily adapted to moderate-sized language models like BERT [Vaswani et al., 2017] , pruning LLMs with billions of parameters presents distinct challenges. The immense model size and extensive datasets associated with LLMs render traditional pruning methods computationally infeasible [Ma et al., 2023] .SparseGPT [Frantar and Alistarh, 2023] utilizes partial weight updates and adaptive mask selection to mitigate the expensive Hessian computation, while Wanda [Sun et al., 2023] directly obtains a sparse LLM model using a criterion that considers the product of the absolute values of weights and their activations. DSnoT [Zhang et al., 2023] iteratively grow and prune the weight mask according to the change in reconstruction error achieved by each update. [Boža, 2024] introduces an efficient approach to determine the optimal weights on a given support and extends this technique to develop heuristics for updating the support.ADMM in network pruning. The operator-splitting technique [Boyd et al., 2011, Davis and Yin, 2016] (also known as Alternating Direction Method of Multipliers, ADMM) is a well-known approach for solving composite optimization or optimization problems with coupled variables (or constraints), and has been used earlier in network pruning. Ye et al. [2018] applied ADMM to solve the original loss function under sparsity constraint, and Boža [2024] used ADMM to solve a convex pruning problem with fixed support. Moreover, Zhang et al. [2018] employed ADMM to train deep neural networks under sparsity constraints, while Ye et al. [2019] utilized ADMM to perform concurrent adversarial training and weight pruning. Our proposed method differs significantly from previous methods in two key aspects: (i) ALPS solves the pruning problem with an ℓ 0 constraint at LLM scale, simultaneously optimizes over the weights and the sparsity pattern. (ii) We introduce a novel penalty parameter update scheme that ensures convergence both practically and theoretically.3 ALPS: Effective LLM pruning in One-shot 
problem formulation: 
 A common approach in post-training unstructured pruning of LLMs is to decompose the full-model compression problem into layer-wise subproblems. The quality of the solution for each subproblem is assessed by measuring the ℓ 2 error between the output of the dense layer and that of the pruned one, given a set of input activations.Formally, let W ∈ R Nin×Nout denote the (dense) weight matrix of layer ℓ, where N in and N out denote the input and output dimension of the layer, respectively. Given a set of N calibration samples, the input activations can be represented as X ∈ R N L×Nin , where L is the sequence length. The goal of pruning is to find a sparse weight matrix W that minimizes the reconstruction error between the original and pruned layer outputs, while satisfying a target sparsity constraint. In addition, we add a ridge term that penalizes the distance between W and W, preventing W from diverging too far from the original weights. This layer-wise pruning problem can be formulated as an ℓ 0 -constrained optimization problem:EQUATIONwhere λ 2 ≥ 0 and ∥ • ∥ 0 denotes the ℓ 0 -(pseudo)norm, which counts the number of non-zero elements.Optimization of Problem (1) is quite challenging: we need to simultaneously find a support of W and a corresponding set of optimal weights (that minimize the objective restricted to the support). Notably, W may contain over 100 million parameters in the LLM setting, making (1) even more computationally demanding. To address this, we employ an operator-splitting technique [Boyd et al., 2011, Davis and Yin, 2016] (also known as ADMM), which decomposes the problem into two computationally 'friendlier' subproblems. Specifically, we reformulate problem (1) by introducing a copy D of weight matrix W:EQUATIONwhere the penalty function "∞ • 1 ∥D∥0>k " imposes the ℓ 0 constraint ∥D∥ 0 ≤ k by assigning a value of zero when this condition is met and infinity otherwise. This reformulation separates the objective function into two independent parts while coupling the variables W and D through the linear constraint W = D. We consider the augmented Lagrangian function of this problem:EQUATIONwhere ρ > 0 is the quadratic penalty parameter. We minimize the augmented Lagrangian with respect to W and D alternatively, followed by a dual update. We get the following update at iteration t:EQUATIONwhere H = X ⊤ X + λ 2 I. Here, the W-update aims to minimize the objective by solving a system of equations, while the D update enforces sparsity by using the projection operator P k (•), which projects an input matrix onto the set of matrices with at most k non-zero elements. The dual update on matrix V ensures consistency between W and D. As the iterations (4) progress, our proposed method concurrently identifies the support of the weight matrix and updates the weights on the determined support.ρ update scheme. In practice, we observe that the sequence of updates (4) and the resulting solution can be sensitive to the choice of the penalty parameter ρ. A small ρ leads to slow convergence due to large changes in the support of D across iterations, while a large ρ may compromise solution quality though the support stabilizes early on. To balance support quality and convergence speed, we introduce a novel penalty parameter update scheme. Starting with a small ρ, we gradually increase it every few iterations, with the increase rate proportional to the change in the support of D. The detailed ρ update scheme is provided in Appendix B.1. This scheme allows our algorithm to explore and find a good support when ρ is small and to converge rapidly as ρ grows, as demonstrated experimentally in Appendix B.2.1.Algorithm 1 outlines the proposed operator-splitting technique with the ρ update scheme. The convergence of Algorithm 1 is guaranteed by the following theorem, with its proof provided in Appendix A. We note that existing convergence results for operator-splitting type methods (e.g., ADMM) focus on convex or continuous problems [Hong et al., 2016 , Wang et al., 2019] . However, our result guarantees the convergence on a non-convex, non-continuous ℓ 0 -constrained problem, which, to the best of our knowledge, is a novel convergence result for such a problem.Theorem 1. Let D (t) ∞ t=0 and W (t) ∞ t=0 be the sequences generated in Algorithm 1. Suppose the penalty parameter {ρ t } ∞ t=1 chosen in Algorithm 1 satisfiesEQUATION)where C is a constant depending on X, W, λ 2 , and ∞ t=1 1/ρ t . In particular, there exists a matrix D such that D (t) → D and W (t) → D as t → ∞.Algorithm 1 ADMM for layer-wise pruning with ℓ 0 constraint Input: Initial penalty ρ 0 .1: Initialize V (0) = 0 Nin×Nout andD (0) = W (0) = W 2: for t = 0, 1, • • • do 3:Update W (t+1) , D (t+1) and V (t+1) according to (4) with ρ = ρ t .Increase ρ t to get ρ t+1 based on the change in the support of Supp D (t) . 5: end for Computational cost. The primary computational cost of Algorithm 1 arises from the W-update step in display (4), which involves solving a system of linear equations. In the update, the inverse of H + ρI can be reused across iterations and needs to be updated when ρ changes. To avoid re-computing the inverse, we store the eigenvalue decomposition H = QMQ ⊤ . For varying ρ values, the inverse can be efficiently calculated as (H + ρI) -1 = Q(M + ρI) -1 Q ⊤ , requiring only a single matrix-matrix multiplication. Additionally, the term H W in W-update remains constant across iterations and can be pre-computed and stored. Thus, each iteration of update (4) requires at most two matrix-matrix multiplications, leading to a time complexity of O(N 2 in N out ). Extension to other sparsity patterns. Algorithm 1 can be extended to support N : M sparsity [Zhou et al., 2021 , Hubara et al., 2021] , a pattern in which a neural network has at most N non-zero weights in each group of M consecutive weights. This sparsity pattern enables inference time acceleration on specialized hardware like NVIDIA A100 GPUs. To accommodate N : M sparsity, we modify the D-update step in (4) by replacing the projection operator P k (•) with a projection onto the set of matrices satisfying N : M sparsity. This modification can be easily implemented by applying magnitude pruning [Zhou et al., 2021] to W (t+1) + V (t) /ρ. Our approach can be further generalized to handle other structured sparsity patterns, such as block sparsity [Gray et al., 2017] and row sparsity [Meng et al., 2024b] . Similar to the N : M sparsity adaptation, this is achieved by modifying the D-update step to project onto the set of matrices satisfying the desired sparsity pattern. Importantly, our convergence results and the refining procedure introduced in Section 3.3 remain applicable to these various sparsity patterns.Our proposed ρ-update technique enables Algorithm 1 to search for a high-quality support when ρ is small, and the support stabilizes quickly as ρ increases. However, once the support stabilizes, the convergence rate of Algorithm 1 becomes slow in practice. To accelerate the optimization process on the support, we employ a Preconditioned Conjugate Gradient (PCG) [Nocedal and Wright, 1999, Section 5 ] method with GPU parallelism and vectorization for efficient computation.Formally, we introduce a post-processing technique that fixes the support S of the current solution W and refines the solution within this support, leading to the following problem:EQUATION(6) decomposes into separate least squares problems across the columns of W. However, as illustrated in Figure 1 (Middle), the supports of the columns of W are different. Using direct matrix inversion (backsolve) to solve these problems would involve solving N out linear equations, each requiring the inversion of a submatrix of H = X ⊤ X + λ 2 I. Since the submatrices under consideration vary across different columns, parallelization is not straightforward, and we must solve N out different linear equations, each with size O(N in ). In LLMs, where N out and N in are of the order 10 4 , this would result in a significant computational expense. We present a workaround as discussed next.To efficiently solve problem (6), we propose using the Preconditioned Conjugate Gradient (PCG) method, a high-performance numerical linear algebra technique for approximately solving systems of equations through repeated matrix-matrix multiplications. We further enhance PCG's performance by introducing two novel acceleration strategies. First, instead of solving for each column of W separately, we solve the entire problem in a single pass by directly solving the linear equation HW = H W using PCG and projecting W onto the given support S in each iteration (Algorithm 2, line 8). We leverage vectorization to significantly enhance the speed. Second, we perform the matrix-matrix multiplications involved in PCG on the GPU, further utilizing GPU acceleration to expedite the computations. Algorithm 2 provides the detailed steps for the PCG method, and Figure1 offers an overview of our proposed ALPS method.Algorithm 2 PCG with vectorization for solving problem ( 6)Input: Support S, pre-conditioner M = Diag(H), initial solution W 0 1: Set R 0 := H( W -W 0 ) 2:Project R 0 onto the support S by setting all elements outside the support to zero.3: Set Z 0 = M -1 R 0 and P 0 = Z 0 4: for t = 0, 1, . . . do 5:α t = Tr(R ⊤ t Z t )/ Tr(P ⊤ t HP t ) 6: W t+1 = W t + α t P t 7: R t+1 = R t -α t HP t 8:Project R t+1 onto the support S by setting all elements outside the support to zero. 9:Z t+1 = M -1 R t+1 10: if R t+1 is sufficiently small then 11: break 12:end if 13:β t = Tr(R ⊤ t+1 Z t+1 )/ Tr(R ⊤ t Z t )14:P t+1 := Z t+1 + β t P t 15: end for 
experimental results: 
 This section compares our proposed framework, ALPS, with state-of-the-art unstructured pruning methods for LLMs. Detailed information on the experimental setup and reproducibility is provided in Appendix B.1, while additional results are presented in Appendix B.2. Models and datasets. We evaluate the performance of ALPS on the OPT model family [Zhang et al., 2022b] with sizes ranging from 1.3 billion to 30 billion parameters, the LLaMA2 model family [Touvron et al., 2023] with 7 billion and 13 billion parameters, and the LLaMA3 model [Dubey et al., 2024] with 8 billion parameters. Following the approach of Frantar and Alistarh, 2023, we use 128 segments of 2048 tokens each, randomly selected from the first shard of the C4 dataset [Raffel et al., 2020] , as calibration data. We assess the performance using perplexity and zero-shot evaluation benchmarks, with perplexity calculated according to the procedure described by HuggingFace [Per, 2022] , using full stride. The test sets of raw-WikiText2 [Merity et al., 2017] , PTB [Marcus et al., 1994] , and a subset of the C4 validation data, which are popular benchmarks in LLM pruning literature [Yao et al., 2022 , Xiao et al., 2023 , Meng et al., 2024b] , are used for evaluation. Additionally, we consider five zero-shot tasks: MMLU [Hendrycks et al., 2021] , PIQA [Bisk et al., 2020] , LAMBADA [Paperno et al., 2016] , ARC-Easy and ARC-Challenge [Clark et al., 2018] .Competing methods. We compare ALPS with several one-shot pruning methods for LLMs, including (i) Magnitude Pruning (MP, [Han et al., 2015] ), (ii) SparseGPT [Frantar and Alistarh, 2023] , (iii) Wanda Sun et al. [2023] , and (iv) DSnoT [Zhang et al., 2023] . We first evaluate the performance of our proposed ALPS framework on a single layer. Specifically, we prune a linear layer in the OPT-13B model with input and output dimensions of 5120 to various sparsity levels and compute the relative reconstruction error of the pruned weight W using ∥X W -XW∥ 2 F /∥X W∥ 2 F . The results are shown in Figure 2 . As demonstrated, ALPS achieves significantly lower reconstruction errors compared to other methods, especially at high sparsity levels. For instance, at a sparsity level of 0.8, ALPS yields a 7.6% relative reconstruction error, while SparseGPT shows a 12% error, and other methods exceed 20%. As demonstrated in Sections 4.2 and 4.3, our method's superior ability to approximate the dense model's output at each layer translates to much better performance in the pruned model.We attribute the superior performance of ALPS in solving the reconstruction problem at each layer to two key aspects: (i) Algorithm 1 obtains a high-quality support by directly optimizing for an optimal subset of weights that contribute the most to recovering the dense model's output (ii) The PCG method in Algorithm 2 efficiently solves the reconstruction problem on a fixed support, further reducing the reconstruction error. To verify these claims, we conducted the following two ablation studies.Firstly, we compare the quality of the support determined by various pruning methods. For each method, we prune the layer to different sparsity levels and fix the support of the weights matrix provided by the method. We then solve the post-processing problem (6) with this support to optimality and compute the relative reconstruction error of the resulting weights. This approach ensures that the reconstruction error depends solely on the quality of the support. of the optimal weights W constrained to the support determined by each pruning method. (Right)Comparison of time and reconstruction error for three scenarios, all using magnitude pruning to determine the support and then: (i) no post-processing (w/o pp.), (ii) refining the weights with ALPS, and (iii) refining the weights optimally with backsolve.We then evaluate the effectiveness of our proposed post-processing method in finding optimal weights on a given support. We first apply magnitude pruning (MP) to determine the support of the weights and then consider three scenarios: (i) without post-processing (w/o pp.), (ii) using Algorithm 2 to solve (6) to refine the weights on the support (ALPS), and (iii) using PyTorch's torch.linalg.solve function to solve (6) to optimality (Backsolve). We assessed both the relative reconstruction error and the computational time for each scenario, with results presented in Table 1 (right). The findings demonstrate that the post-processing procedure significantly lowers the reconstruction error. Notably, our PCG method achieves errors comparable to the optimal solution but is 20x-200x faster, underscoring the efficiency and effectiveness of our approach.This section focuses on pruning OPT models and LLaMA models to various sparsity levels and evaluating the performance of the pruned models using perplexity and zero-shot benchmarks. The Figure 3 : Performance analysis for one-shot unstructured pruning of LLaMA3-8B model at various sparsity levels on two datasets: WikiText2 (Left) and PIQA (Right). We run each method five times and plot the shaded region as the area between the mean (solid line) and two standard deviations above and below the mean. performance of the pruned LLaMA3-8B model at different sparsity levels on the WikiText2 and PIQA datasets is presented in Figure 3 . Table 2 showcases the performance of OPT models with 70% sparsity on various datasets. Additional results on different models, sparsity levels, and datasets are provided in Appendix B.2.5.Figure 3 demonstrates that ALPS outperforms other competitors when sparsity levels exceed 50%, and the performance gap between ALPS and other methods widens as the sparsity level increases. For instance, ALPS achieves a 60% perplexity reduction on the WikiText2 dataset compared to other methods at 80% sparsity level. This observation aligns with our findings in Section 4.1, confirming that ALPS's highly advanced optimization method in solving layer-wise reconstruction problems enables it to better preserve performance at medium-to-high sparsity levels compared to other methods. Table 2 further validates this fact, showing that ALPS outperforms other methods by a large margin across all models on all criteria. This suggests the superiority of ALPS in pruning models at medium-to-high sparsity levels.We further assess ALPS's performance on N : M sparsity patterns, with Table 3 listing the results for pruning OPT-30B and LLaMA2-13B models at 2:4 and 4:8 sparse patterns (see Appendix B.2.5 for other models). ALPS outperforms other methods on most datasets, achieving larger performance improvements in N : M pruning compared to unstructured pruning at the same sparsity level. This is due to the higher complexity of the N : M sparsity pruning problem, which ALPS, as a highly advanced optimization algorithm, can handle more effectively than competing heuristics. 
conclusion: 
 We present ALPS, an efficient optimization-based framework for one-shot unstructured LLM pruning. ALPS employs the operator splitting technique to effectively solve the ℓ 0 -constrained layer-wise pruning problem. To enhance the performance of our algorithm, we introduce a novel penalty parameter updating scheme and a post-processing procedure using PCG with vectorization/GPU parallelism that takes into account problem-structure. We also establish novel convergence guarantees for our algorithm. ALPS can efficiently perform high-quality pruning of LLMs at scale. Our experiments confirm that ALPS outperforms existing pruning methods in terms of both the pruning objective and the performance of the pruned model. Future work will consider extending ALPS to incorporate structured pruning constraints and quantization to get a better understanding of the strengths and scope of our optimization-based approach.  
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We provide detailed settings of our proposed pruning framework and its competitors used in our experiments in Appendix B.1.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: In our experiments, for each combination of model, pruning method, and sparsity level, we ran 5 different seeds and reported the mean and standard deviation of the results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: To the best of our knowledge, our work poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: At the beginning of Section 4, we cite all the datasets and models used in our experiments.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: Our paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our paper does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our paper does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Proof. For the sake of conciseness, throughout the proof, we denote H = X ⊤ X + λ 2 I and G = X ⊤ X + λ 2 I W. To establish the theorem, we first present the following two lemmas. The proofs of these two lemmas are given in Section A.1 and A.2, respectively.Lemma 1. Let D (t) ∞t=0 and V (t) ∞ t=0 be the sequence generated in Algorithm 1. Then for any t ≥ 0, it holdsEQUATIONandEQUATION) Lemma 2. Let D (t) ∞ t=0 , W (t) ∞ t=0 and V (t) ∞ t=0 be the sequence generated in Algorithm 1. Suppose {ρ t } ∞t=0 is non-decreasing. Then for any t ≥ 0, it holds∥D (t) ∥ F + ∥V (t) ∥ F ρ t ≤ t-1 s=0 1 + 3∥H∥ 2 ρ s • ∥D (0) ∥ F + ∥V (0) ∥ F ρ 0 + t-1 s=0 3∥G∥ F ρ s (9)Returning to the proof of the main theorem, combining Lemma 2 with the initialization of Algorithm 1 gives∥D (t) ∥ F + ∥V (t) ∥ F ρ t ≤ t-1 s=0 1 + 3∥H∥ 2 ρ s • ∥D (0) ∥ F + ∥V (0) ∥ F ρ 0 + t-1 s=0 3∥G∥ F ρ s ≤ exp 3∥H∥ 2 ∞ s=0 1 ρ s • ∥G∥ F + 3∥G∥ F ∞ s=0 1 ρ s (10) Let C(X, W, ρ 0 , t u , τ ) := 2∥G∥ F + 2∥H∥ 2 exp 3∥H∥ 2 ∞ s=0 1 ρ s • ∥G∥ F + 3∥G∥ F ∞ s=01 ρ s (11) be the constant depending on X, W and ∞ s=0 1/ρ s . Lemma 1 together with (10) leads toEQUATIONandEQUATION)It then follows fromEQUATIONTherefore, we prove the desired inequality. Since ∞ s=0 1/ρ s < ∞, {D} ∞ t=0 is a Cauchy sequence, and therefore there exists a matrix D such that D (t) → D. It follows from ∥W (t+1) -D (t+1) ∥ F → 0 that W (t) → D. The proof is completed.Proof. According to the W-update rule in (4), it holdsW (t+1) -D (t) + V (t) ρ (t) = (H + ρ t I) -1 (G -V (t) + ρ t D (t) ) -D (t) + V (t) ρ (t) = (H + ρ t I) -1 ρ t -I D (t) + (H + ρ t I) -1 (G -V (t) ) + V (t) ρ t = - 1 ρ t I + H ρ t -1 HD (t) + 1 ρ t I + H ρ t -1 (G -V (t) ) + V (t) ρ t = 1 ρ t I + H ρ t -1 (G -HD (t) ) + 1 ρ t I -I + H ρ t -1 V (t) = 1 ρ t I + H ρ t -1G -HD (t) + HV (t) ρ t (15) Therefore, we obtainEQUATION) Denote I := {(i, j) ∈ [N in ] × [N out ] | D (t) ij = 0}.It follows from the D-update rule and the definition of the projection operator thatEQUATION)Together with ( 16), we getEQUATION)It then follows from the V-update rule thatEQUATIONThis establishes the inequality (7). Furthermore, by summing up ( 16) and ( 18) and applying the triangle inequality, we verify the inequality (8).Proof. It follows from Lemma 1 thatEQUATIONandEQUATIONThis further impliesEQUATION)Combining inequalities ( 20) and ( 22) yieldsEQUATIONDenote a t := ∥D (t) ∥ F + ∥V (t) ∥ F /ρ t , then the above inequality can be rewritten asa t+1 ≤ 1 + 3∥H∥ 2 ρ t a t + 3∥G∥ F ρ t (24)Therefore,EQUATIONIt then follows from telescoping thata t t-1 s=0 (1 + 3∥H∥ 2 /ρ k ) ≤ a 0 + t-1 s=0 3∥G∥ F ρ t (26)Recalling the definition of a t completes the proof.We performed all experiments on a computing cluster using an Intel Xeon Gold 6248 machine with 20 CPUs and a single NVIDIA V100 GPU, which is equipped with 192GB of CPU RAM and 32GB of CUDA memory. The PyTorch library Paszke et al. [2017] was used to implement all language models and pruning methods for our experiments.Pruning problem setup. For a given sparsity s, we set the ℓ 0 constraint k in the pruning problem (1) to ⌊N in N out s⌋. Following the pruning framework proposed by Frantar and Alistarh [2023] , Meng et al. [2024b] , we solve the LLM pruning problem sequentially, layer by layer. For layer ℓ, the input activation matrix X in (1) is set as the output of the previous ℓ -1 pruned layers on N calibration samples.Data pre-processing. Let E = Diag(X ⊤ X + λ 2 I) -1/2 . To achieve better scaling, we define W ′ = E -1 W and reformulate Equation (1) into the following equivalent form:min W ′ ∈R N in ×N out ∥X W -XEW ′ ∥ 2 F + λ 2 ∥ W -EW ′ ∥ 2 F s.t. ∥W ′ ∥ 0 ≤ k (27)Practically, we apply Algorithm 1 to solve Equation ( 27) and recover the solution to the original problem by setting W = EW ′ . It is important to note that this pre-processing step does not alter the procedure of Algorithm 1 or affect the convergence analysis. It only modifies the updates within Algorithm 1 and can lead to a better convergence rate in practice.Hyperparamter choice. We choose λ 2 = 0.01 Tr(X ⊤ X). In Algorithm 1, we set ρ 0 = 0.1. And we update ρ every 3 iteration based on a step function that depends on the current value of ρ t and s t := | Supp D (t) ∆ Supp D (t-3) |, which represents the number of elements in the symmetric difference between Supp D (t) and Supp D (t-3) . Specifically, we setEQUATIONIf s t = 0, it indicates that ρ is sufficiently large and the support has stabilized. In this case, we terminate Algorithm 1, set S as the support of W, and apply Algorithm 2 with 10 iterations to solve problem (6) with support S.Implementation details. Below are the configuration and implementation details for the competing methods and our proposed framework ALPS.• MP: For each layer in the LLM, we perform magnitude pruning by sorting the absolute values of all entries of the dense weight W in descending order, keeping the top k entries unchanged, and setting the remaining entries to zero. • SparseGPT: We utilize the authors' implementation (codes available on GitHub) with default hyperparameter settings to perform one-shot unstructured LLM pruning. • Wanda: We utilize the authors' implementation (codes available on GitHub) with default hyperparameter settings to perform one-shot unstructured LLM pruning. • DSnoT: We utilize the authors' implementation (codes available on GitHub) with default hyperparameter settings to perform one-shot unstructured LLM pruning.Our proposed ρ update scheme, theoretically supported by Theorem 1, ensures that ALPS converges rapidly while finding high quality solutions. In contrast, ADMM with a fixed ρ may fail to converge when applied to ℓ 0 constrained least squares problems. To provide empirical evidence for this claim, we compare ALPS with ADMM using fixed ρ values. We examined two key metrics: reconstruction loss (objective) and the rate of change of the support (of weights) between consecutive iterations (this measures the convergence speed of the algorithm).Tables 4 and 5 present the results for these metrics, respectively. Our findings show that ADMM with a large ρ(= 3) converges quickly but yields poor solutions, while a small ρ(= 0.3) fails to converge. ALPS, utilizing our ρ update scheme, achieves both rapid convergence and high-quality solutions.Supp change / Iter 5 10 20 30 50 100 ALPS 1.63e-1 1.28e-1 5.95e-2 5.32e-2 5.31e-2 5.31e-2 ADMM(ρ = 0.3) 7.83e-2 7.55e-2 7.50e-2 7.47e-2 7.47e-2 7.45e-2 ADMM(ρ = 0.3) 9.32e-2 8.18e-2 7.64e-2 7.53e-2 7.45e-2 7.42e-2 We compare the runtime of ALPS with other methods in Table 6 . ALPS employs an advanced optimization method to solve the layerwise reconstruction problem, which results in longer running times compared to other algorithms. However, it's important to note that ALPS 's runtime is still negligible when compared to fine-tuning methods for LLMs, e.g., LoRA [Hu et al., 2021] .We discuss the difference between ALPS and ADMM-Grad [Boža, 2024] , another interesting work using ADMM for LLM unstructured pruning. [Boža, 2024] selects the sparsity mask via iterative magnitude pruning and applies ADMM to solve problem (6) with the selected sparsity mask. ALPS, in contrast, is an end-to-end approach that directly targets the ℓ 0 constrained least squares problem (1). By simultaneously optimizing both weights and sparsity patterns, ALPS achieves lower layerwise reconstruction loss compared to ADMM-Grad, as demonstrated in Table 7 .Additionally, since [Boža, 2024] employs ADMM for solving problem ( 6), we compare it with our proposed PCG procedure. We tested both approaches for solving problem ( 6) with a given support.Results presented in Table 8 show that PCG outperforms ADMM-Grad in both computational time and objective value. The time advantage of PCG stems from its ability to backsolve without explicitly computing matrix inverses. Sparsity 0.4 0.5 0.6 0.7 0.8 0.9 ALPS 3.55e-3 7.56e-3 1.47e-2 2.77e-2 5.32e-2 1.13e-1 ADMM-Grad 4.47e-3 9.53e-3 1.83e-2 3.36e-2 6.19e-2 1.25e-1 [Boža, 2024] for solving problem (6).Results show runtime (seconds) and layerwise reconstruction loss ∥X W -XW∥ 2 F /∥X W∥ 2 F across supports with different sparsity levels.We evaluated the one-shot unstructured pruning performance of MP, SparseGPT, Wanda, DSnoT, and ALPS on LLaMA3-8B using the MMLU benchmark to give a comprehensive assessment of each method's effectiveness. Table 1 shows the mean accuracy across all MMLU categories. The results demonstrate that ALPS outperforms other methods, further validating its effectiveness in producing high-performance pruned models. We also observe significant performance degradation on MMLU at high sparsity levels, suggesting that one-shot pruning should be combined with fine-tuning [Hu et al., 2021] or prompting [Sahoo et al., 2024] Table 9 : Performance analysis for one-shot unstructured pruning of LLaMA-3 8B models at various sparsity levels using MMLU benchmark.We compare the one-shot unstructured pruning performance of MP, SparseGPT, Wanda, DSnoT, and ALPS on OPT-1.3B-30B models and LLaMA2-7B, LLaMA2-13B, LLaMA3-8B models in Tables 10 11 12 13 14 15 16 17 . The models are pruned at unstructured sparsity levels of 40%, 50%, 60%, 70%, 80%, and 90%, as well as at 2:4 and 4:8 sparsity patterns. We evaluate the perplexity of the pruned models on WikiText2, PTB, and C4 datasets. Additionally, we assess the accuracy of the pruned models on PIQA, LAMBADA, ARC-Easy, and ARC-Challenge. However, LAMBADA accuracy results for the LLaMA model have been omitted since LLaMA models have unsatisfactory performance on this dataset without further modifications. For each combination of model, sparsity, and pruning approach, we run each method five times and report the mean and standard deviation of each performance criterion. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.38th Conference on Neural Information Processing Systems (NeurIPS 2024).ADMM-based LLM Pruning in one-ShotThe "support" of the weights refers to the set of indices corresponding to non-zero weights within a layer. 


Decision: This paper proposed a large language model (LLM) pruning framework, which turns the problem into  layer-wise activation reconstruction problem via sparse weight matrix. The operator splitting technique and preconditioned conjugate gradient methods are deployed and the empirical studies show the better single-layer reconstruction error and correspondingly the improved performance on downstream tasks. 

Overall the reviewers like the theoretical convergence guarantees and the practical applicability of the proposed method. There has been some major concerns from reviewer 1PwE, but after the rebuttal some of them have been addressed. We highly encourage the authors to incorporate the additional artifacts into the revisions to make the paper more solid.

Reviews: review: paper_summary: paper_summary: This paper introduces ALPS, a novel optimization-based framework for one-shot unstructured pruning of LLMs. The key contributions are:

- Formulating LLM pruning as an l0-constrained optimization problem solved using operator splitting (ADMM).
- A penalty parameter update scheme to accelerate convergence.
- A post-processing step using preconditioned conjugate gradient (PCG) to refine weights.
- Theoretical convergence guarantees for the proposed algorithm.

The authors demonstrate that ALPS outperforms state-of-the-art pruning methods, especially at high sparsity levels, on various LLMs including OPT and LLaMA models.
strength_weakness: strength_weakness: 1. It would be beneficial to include evaluations on more recent models such as LLaMA-3, instruction-tuned models, and extremely large models (70B+ parameters) to demonstrate the method's applicability to the latest advancements in the field.
2. The current evaluation tasks are relatively limited. Expanding the evaluation to include knowledge-intensive tasks (e.g., MMLU) and reasoning-based tasks (e.g., GSM8K and HumanEval) would provide a more comprehensive assessment of the method's effectiveness.
3. The current focus is primarily on N:M sparsity patterns. It would be great to explore structured pruning, which could lead to more significant improvements in inference speed, making the approach more versatile and practical for real-world applications.
questions: questions: 1. How does the runtime of ALPS compare to other methods, especially for very large models?
2. Can (how can) ALPS be extended to handle other types (structured sparsity) of structured sparsity patterns beyond N:M?
limitations: limitations: The authors discussed some limitations of their work e.g. extending ALPS to incorporate structured pruning constraints and quantization.
paper_summary: paper_summary: This work presents an LLM pruning framework that formulates the problem as finding a sparse weight matrix to reconstruct the layer-wise activations. This work incorporates the operator splitting technique and preconditioned conjugate gradient methods to solve the pruning problem. Experiments demonstrate that the proposed method achieves better single-layer reconstruction error and improved performance on downstream tasks.
strength_weakness: strength_weakness: Experiments could be more solid since many experiments are run on OPT, which is somewhat out-of-date. It would also be better to consider more challenging benchmarks, such as GSM8K or other questions that require generating a long answer.
questions: questions: It would be better to compare with other representative pruning methods beyond the one-shot unstructured pruning, and to explore how this method could be integrated with others.

This work discusses engineering optimizations, such as incorporating PCG with GPU. Will you open-source the code or provide more efficient results from these engineering efforts?
limitations: limitations: There is no section for limitation, but the author discusses some in the conclusion. One limitation is the setting of layer-wise activation reconstruction. Recent works, especially the activation-aware pruning methods, reported that activations are not equally important.
paper_summary: paper_summary: The paper introduces ALPS, an optimization-based framework for one-shot pruning of large language models (LLMs). ALPS leverages an ADMM-based algorithm with operator splitting and preconditioned conjugate gradient methods to achieve improvements in sparsity and perplexity over state-of-the-art methods, particularly in high-sparsity regimes.
strength_weakness: strength_weakness: **W1.**
Important Reference Missing: The paper does not cite "Fast and optimal weight update for pruned large language models" by Boža, which addresses a similar problem using an ADMM-based optimization algorithm. This omission is significant as both papers share highly similar problem definitions and solutions.

**W2.**
Limited Novelty: ALPS closely resembles methods discussed in both "Fast and optimal pruning" and "Progressive weight pruning of deep neural networks using ADMM." The primary difference is ALPS's specific application to LLMs. However, this differentiation might not be substantial enough to establish ALPS as a novel contribution. Btw, the reference of "Progressive weight pruning of deep neural networks using ADMM." seems also missing in the paper.

**W3.**
Performance at High Sparsity: At very high sparsity levels, ALPS’s perplexity remains significantly higher than the dense model. This indicates that the pruned LLMs by ALPS may still perform poorly and being practically useless, although they are better than those pruned by comparison methods. 

**W4.**
Unfair Comparison: The comparison with methods like Wanda and DSnoT, which do not involve retraining after pruning, is solely based on perplexity. This is unfair because it overlooks the overall running time per iteration/epoch. ALPS’s performance should be compared with these methods by considering both perplexity and computational efficiency to provide a more balanced evaluation. In addition, I believe if Wanda or DSnoT is combined with some re-training techniques, they can achieve a much lower perplexity as well.

Reference:

Boža, Vladimír. "Fast and optimal weight update for pruned large language models." arXiv preprint arXiv:2401.02938 (2024).

Ye, Shaokai, et al. "Progressive weight pruning of deep neural networks using ADMM." arXiv preprint arXiv:1810.07378 (2018).
questions: questions: Please refer to my points listed in the weakness section. Besides those,

1. Could you provide a clearer explanation of the practical implications of your theoretical contributions?

2. I encourage the authors to report the overall running time per epoch for ALPS in comparison with Wanda and DSnoT.

3. Could you provide the perplexity of the dense models (without any pruning) in all the tables in the paper?

4. How does ALPS's speed of convergence compare with the referenced methods, and what practical benefits does the novel penalty parameter update scheme offer?
limitations: limitations: The authors have not thoroughly discussed the limitations and potential negative societal impacts. They should address the limited novelty compared to existing ADMM-based methods and discuss the risks and mitigation strategies for the misuse of more accessible powerful models.

Comments: title: Response to rebuttals
comment: Thank you for your positive feedback. Based on your valuable comment, we have conducted further experiments to compare ALPS with SparseGPT, currently our strongest competitor, on LLaMA3-7B at 50%, 70%, and 2:4 sparsity levels. The results, as shown in the table below, demonstrate that ALPS outperforms SparseGPT on recent models LLaMA3. 

In response to your suggestions, we are currently running extended experiments to compare ALPS on pruning LLaMA3 with other competitors [in the current paper] across a range of sparsity levels. We will include our current results and new results in a revision. 


| 2:4 |Lambada|Piqa|Arc-easy|Arc-challenge| 
|---|---|---|---|---|
|SparseGPT|36.93|70.40|59.01|28.84|
|ALPS|42.79|70.89|61.83|28.92|

| 50% |Lambada|Piqa|Arc-easy|Arc-challenge| 
|---|---|---|---|---|
|SparseGPT|61.87|75.57 |72.39|39.25|
|ALPS|64.04|76.93|73.40|40.61|

| 70% |Lambada|Piqa|Arc-easy|Arc-challenge| 
|---|---|---|---|---|
|SparseGPT|8.13|61.21|41.54|20.05|
|ALPS|16.42|64.58|46.89|21.84|

Additionally, we plan to include evaluation results for MMLU and GSM8K to further enhance the paper. Thanks again for your thoughtful feedback, which has strengthened our work.
Thanks for your feedback and thoughtful comments. We appreciate your willingness and consideration to be open to revising your score. Please let us know if you have any additional questions or need clarification.
Thank you for your detailed responses. The paper is in good shape, but still have some concerns remain unsolved:

1. I understand the resource constraints, but some results on models like LLaMA-3 or a discussion on scaling to 70B+ parameters would be valuable. OPT-30B is relatively old, and to my knowledge, it's sometimes easier to prune them than more recent 'overtrained' models, e.g. Llama-3 and Qwen 1.5 / 2.

2. Including knowledge-intensive (e.g., MMLU) and reasoning-based tasks (e.g., GSM8K) would strengthen the paper.

3. Theoretical analysis of extension is promising, and empirical results or case studies on these structured sparsity patterns would provide practical insights.
I sincerely appreciate the authors' response, which addressed some of my previous concerns. However, given the current dispersion of ratings, I prefer to proceed cautiously and will consider the rebuttal more thoroughly in the next phase of discussion. I am inclined to maintain my current score for now but will remain open to adjusting it during the next phase.