Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision
Authors: Li Shen
Anke Tang
Yong Luo
Tao Sun
Han Hu
Xiaochun Cao
Venue: ICML 2025 poster (2025)
Keywords: Pruning
Large Language Model
Model Compression
Primary area: deep_learning->large_language_models
TL;DR: We propose an iterative method for refining pruned neural network weights, aiming to improve model performance while maintaining sparsity

Abstract: Pruning is a widely used technique for compressing large neural networks that eliminates weights that have minimal impact on the model's performance. Current pruning methods, exemplified by magnitude pruning, assign an importance score to each weight based on its magnitude and remove weights with scores below a certain threshold. Nonetheless, these methods often create a gap between the original dense and the pruned sparse model, potentially impairing performance. Especially when the sparsity ratio is high, the gap becomes more pronounced. To mitigate this issue, we introduce a method to bridge the gap left by pruning by utilizing a low-rank approximation of the difference between the dense and sparse matrices. Our method entails the iterative refinement of the sparse weight matrix augmented by a low-rank adjustment. This technique captures and retains the essential information often lost during pruning, thereby improving the performance of the pruned model. Furthermore, we offer a comprehensive theoretical analysis of our approach, emphasizing its convergence properties and establishing a solid basis for its efficacy. Experimental results on LLaMa models validate its effectiveness on large language models across various pruning techniques and sparsity levels. Our method shows significant improvements: at 50\% sparsity, it reduces perplexity by 53.9\% compared to conventional magnitude pruning on LLaMa-7B. Furthermore, to achieve a specific performance target, our approach enables an 8.6\% reduction in model parameters while maintaining a sparsity ratio of about 50\%.

Targeted Low-rank Refinement:
Enhancing Sparse Language Models with Precision
Li Shen Anke Tang?
Yong Luo
Tao Sun
Han Hu Xiaochun Cao 15
Abstract
1. Introduction
Pruning a crucial technique in the field of model com- pression, partieularly for large language models (LLMs), which have become the cornerstone of natural language pro- cessing tasks (Devlin, 2018; Brown, 2020; Hoffmann et al., 2022). Pruning involves the removal of specific weights or parameters from the neural network that are considered to have minimal impact on the overall performance of the model (LeCun et al., 1989; Hassibi al., 1993; Han et al., 2015b; Frankle Carbin, 2018; Frankle et al., 2020). This technique is especially valuable for LLMs, which often contain billions of parameters and require substantial com- putational resources for both training and inference (Han, 2017; Touvron et al., 2023; Minaee et al., 2024).
Pruning is widely used technique for compress- ing large neural networks that eliminates weights with minimal impact on performance. Current pruning methods, exemplified by magnitude prun- ing, assign importance scores to weights based on their magnitude and remove those below certain threshold. However, these methods introduce a gap between the original dense and pruned sparse models, potentially impairing performance, es- pecially high sparsity ratios. To address this issue, we introduce method that bridges this gap through low-rank approximation of the difference between dense and sparse matrices. Our approach iteratively refines the sparse weight matrix with a low-rank adjustment, capturing essential informa- tion typically lost during pruning. We provide a comprehensive theoretical analysis of our method, establishing its convergence properties and effi- cacy. Experimental results LLaMA models validate method's effectiveness across various pruning techniques and sparsity levels. At 50% sparsity, it reduces perplexity by 53.9% compared to conventional magnitude pruning on LLaMA- 7B. Furthermore, our approach enables an 8.6% reduction in model parameters while maintaining sparsity ratio of about 50%.
One of the most representative pruning techniques is magni- tude pruning, which removes weights that have the smallest absolute values. This method is based on the assumption that smaller weights have less effect on the network's 's overall per- formance (Han et al., 2015a). Furthermore, as model sizes continue to grow, the number of redundant parameters also increases. For LLMs with billions of parameters, even half of the layers can be dropped without significantly affecting performance (Men et al., 2024; Fan al., 2024). However, recent research shows that pruning can cause irreparable loss of knowledge and performance drops, especially for difficult tasks, phenomenon known as the Junk DNA Ily- pothesis (Yin et al., 2024). This consistent degradation in performance is observed across a spectrum of pruning meth- ods, including magnitude pruning, SparseGPT (Frantar Alistarh, 2023), and Wanda (Sun al., 2023), and applies to both unstructured pruning and structured N:M pruning.
'School Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen 518107, China Natiioaa Engineering Research Center for Multimedia Software, School of Computer Seience and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, Wuhan 430072, Hubei, China 'National University of Defense Technology. Hunan, China "School of Information and Electronics, Beijing Institute of Technology. Beijing, China Key Laboratory of Cyberspace Security, Ministry of Education, China. Corre- spondence to: Anke Tang <anketang wwh..edu.en>, Yong Luo <luoyong@ @ whu.edu.en>>
Conventional approaches to post-pruning performance re- covery face three-fold challenge: (1) Computational bur- den: Re-training (Frankle & Carbin, 2018; Xia et al., 2023; Kim et al., 2024a) and knowledge distillation (Hinton, 2015; Wan et al., 2024a; Muralidharan et al., 2024) methods are computationally expensive and time-consuming, (2) Data and model dependency: These techniques typically require either extensive datasets or access teacher model, which may not always be feasible. (3) Spar- sity inconsistency: Recent low-rank approximation meth- ods (Li et al., 2023b; Mozaffari et al., 2024; Zhang & Pa-
high- sigh-perffrming
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision
pyan, 2024) often fail maintain consistent sparsity pat- tern, making them unsuitable for structured pruning. which is crucial for hardware efficiency.
derstanding of the effectiveness, convergence, and sta- bility of our approach.
We evaluate our method on LLMs and show significant perplexity improvements over baselines. This is partic- ularly noteworthy at high sparsity levels, maintaining robust improvements even as high as 70% sparsity.
In this study, we address these challenges of post- post-pruning performance recovery by approximating the dense matrix as the sum of an updated sparse matrix with a maintained spar- sity pattern and a low-rank matrix. We propose an iterative refinement process that concurrently updates the sparse ma- trix and the low-rank component in a data-free manner. This approach effectively recovers crucial information typically lost during pruning. Our method features an adaptive low- rank approximation that dynamically adjusts to complement the sparse matrix, enabling efficient information recovery. Unlike traditional techniques that rely on large datasets or nigh-performing teacher models, our approach operates directly on model weights, offering a computationally effi- cient and broadly applicable solution. This approach aims to improve the performance of the pruned model without significantly increasing the parameter count. Our method combines the advantages of both sparse and low-rank ap- proximation, ensuring the model maintains its efficiency while enhancing its accuracy. In addition our empirical findings, we provide a comprehensive theoretical analysis of the iterative refinement process, which rigorously demon- strates the favorable convergence properties of our method.
2. Related Work
LLM compression. LLMs have become essential in nat- ural language processing tasks, but their substantial size poses challenges in terms of computational resources and efficiency. Various techniques have been proposed to com- press LLMs while maintaining their performance, including pruning, low-rank compression, quantization, and knowl- edge distillation (Cheng et al., 2017; Choudhary et al., 2020; Haroush et al., 2020). In this study, we concentrate on pruning methods and use low-rank approximations to re- store the lost performance. Pruning LLMs with billions of parameters differs significantly from pruning smaller models (Gale et al., 2019; Frankle et al., 2020; Kurtic & Al- istarh, 2022), as current pruning techniques often necessitate extensive re- re-training after the pruning process, which is pro- hibitively costly for L.MMs (Komatsuzaki et al., 2022; Chung et al., 2024: Snell et al., 2024). Pruning methods are often categorized into unstructured and structured pruning (Liu et al., 2017: Fan al., 2019; He & Xiao, 2023). Structured pruning techniques, including layer pruning (Chen & Zhao, 2018; Kim et al., 2024b), channel pruning (He et al., 2017; Zhuang et al., 2018), and N:M pruning (Sun et al., 202 I), focus on climinating entire neurons, layers, or N out of M elements in a regular pattern. Unstructured pruning removes individual weights without considering their structure, often improving performance but making it less efficient for hard- ware. Low-rank compression reduces the dimensions of the weight matrix by focusing on larger singular values in both the column and row spaces (Cheng et al., 2005; Idelbayev & Carreira- Perhirar,errinnn,, 2020). While quantization (Xiao et al., 2023; Lin et al., 2024) and knowledge distillation (Wan et al., 2024b) have been used to compress LLMs, they are orthog- onal to our approach. Low-rank refinement is also used to regain performance during the quantization process (Guo et al., 2023; Loeschcke et al., 2024; Li et al., 2023a).
Experimental results on the LLaMA models demonstrate the effectiveness of our low-rank refinement approach. When applying 50% sparsity, our method achieves 53.9% re- duction in perplexity compared to conventional magnitude pruning. The benefits of our approach become increasingly evident as sparsity increases: at 60% sparsity, we observe a 92.0% decrease in perplexity compared to vanilla sparse models, while at 70% sparsity, an impressive 99.6% reduc- tion is achieved. This highlights the effectiveness of our method, especially in scenarios of high sparsity where tradi- tional approaches typically face considerable challenges.
To summarize, the main contributions of this paper are:
We bridge the gap between the original dense and pruned sparse model by leveraging low-rank com- ponent. This approach effectively fills the gap left by pruning, enhancing the model's performance with minimal parameter increase.
Post-pruning performance recovery. To recover the per- formance after pruning step, several post-pruning recovery techniques have been explored such as re-training the pruned model and using knowledge distillation (Muralidharan et al., 2024). However, these approaches can be computationally expensive and time-consuning. Recent research has fo- cused on incorporating low-rank approximations to recover the lost performance. By adding a low-rank component to the pruned model, it is possible to approximate the origi- nal dense model more closely with minimal computational
We develop an iterative algorithm that incrementally refines the sparse weight matrix and incorporates the low-rank approximation. By prioritizing the preser- vation of weight components associated with larger singular values, our method allows for a more aggres- sive reduction of less important components, leading to a more precise approximation.
We provide a thorough theoretical analysis of our pro- posed method, which offers a rigorous foundation un-
2

---

Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision
W
M
P
S
L
3
3
0
0
1
>
>
3
0
4
?
?
+
1
5
2
1 5
2
3
5
0
?
0
0
(a) The general framework of neural network pruning.
(b) Refine with a low-rank patch.
Figure 1: (a) An overview of the general framework of (local) layer-wise pruning, where W, M, P. and S are the original dense weight matrix, the importance score matrix, the binary pruning mask, and the pruned sparse weight matrix, respectively. Here we show the case of pruning using the magnitude of the weights as the importance score. (b) Our proposed method, where S is updated and low-rank matrix Lg is introduced to minimize the approximation error.
overhead (Chandrasekaran et al., 2009; Li et al., 2023b; Mozaffari et al., 2024; Zhang & Papyan, 2024).
recover some lost performance. During this process, the pruning structure is maintained by enforcing the constraint S(()) = (tt) @ P at each iteration t. This ensures that the pruned weights remain zero throughout the re-training process, preserving the sparsity achieved through pruning.
Low-rank refinement. Low-rank refinement is technique that has been used to recover the performance in many scenarios. While this study concentrates on post-pruning performance recovery, low-rank refinement has also been utilized in other scenarios, including quantization (Guo et al., 2023; Loeseheke et al., 2024; Li et al., 2023a; Zhang al., 2023). However, key distinction of our approach is that it is data-free and can be applied to individual layers, making the refinement process more memory-efficient.
3.2. Singular Value Decomposition
Here we briefly review the Singular Value Decomposition (SVD) of a matrix. Given matrix W E R=xn, its reduced SVD is given by (Olver & Shakiban, 2018):
W = UEVT
(1)
3. Preliminary
3.1. The General Framework of Layer-Wise Pruning
where U E R and V E R orthogonal matri- ces, and Σ R is diagonal matrix containing the singular values of W. The i-th columns of matrices U and V are represented by u, and v, respectively. Ad- ditionally, we use G; to denote the i-th diagonal element of the matrix E. For convenience, we extend this nota- tion to represent functions that map W to its SVD com- ponents. This allows to express the decomposition as W U(W) (W)) (W), alternatively as sum of outer products: W = Σ a,(W),, (W)), (W). While this slightly abuses the original notation, it provides a concise way to refer to the SVD components of a matrix.
Neural network pruning crucial technique for model compression, aiming to reduce parameters while preserv- ing performance. In this section, we first present the gen- eral framework for (local) layer-wise pruning, outlining the key steps and components involved as illustrated in Fig. I(a). Let W E 1mxx" represent a weight matrix, and M = MMW, D) Rmxx denote its corresponding im- portance score matrix. Here, M : Rmxa D + mm is metric function that computes the importance scores based on the weight matrix W and an optional dataset D. The most straightforward approach is to use the magnitude of the weights as the importance score, i.e., Múj = W More sophisticated methods can be employed to capture the importance of each weight more accurately, such as the SparseGPT (Frantar & Alistarh, 2023) and Wanda (Sun al., 2023). Given pruning ratio p E [0, 1). we set thresh- old h the p-th percentile of M the decision boundary for pruning. Then the binary pruning mask P is obtained by P. T(M > h). The pruned sparse weight matrix S is obtained by S = W o P. where O denotes element-wise multiplication. This zeroes out less important weights while retaining significant ones. An optional weight update pro- cedure can be applied to refine the pruned weights further. For example, after the pruning process, re-training phase may be implemented to fine-tune the remaining weights and
4. Method
Low-rank approximation is common technique for model compression, which has been widely used in various fields as in (Zhou & Tao, 2011; Yu et al., 2017). In this study, we focus on the local pruning method and aim to fill the gap between the original dense weight matrix W and the pruned weight S by introducing a low-rank matrix Lę with target rank & approximate AW W.S. We illustrate the main idea in Fig. 1(b), where the dense matrix W is factorized into the superposition of low-rank matrix L and updated sparse matrix S':
W S' + L + L
(2)
3

---

Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision
Algorithm 1 The Proposed Iterative Weight Update Method
Where || ||. denotes the nuclear norm and || I-112 denotes the f--norm, À is a hyperparameter that controls the trade-off between the rank of L and the sparsity of S'. The nuclear norm serves as convex approximation for the rank of a matrix, while the -nomm acts convex proxy for the to- norm (which represents the count of non-zero elements in a matrix). In this convex optimization problem, the ||]|| term promotes low-rank solution for L, whereas the X S|| term promotes sparsity in S' Nevertheless, it's important to note that the solution to Eq.(3) does not necessarily preserve the fixed sparsity pattern of S' that is required to match S in (2). To address this issue, we propose to incorporate the binary mask P into the optimization process to ensure that the sparsity pattern of S' is fixed as S. This is achieved by rewriting (2) as follows:
1: Inputs: Dense weight matrix W, binary mask P, target rank k, number of iterations? T
2: Initialize ( < W @ P
3: for (= = 0 to T - 1 do
4: ( t W- - (tt
5: Compute SVD: [(0] U1I
6:
p(0) + 1+ +1
7: S((11) t (() + P
( (()) V-00T]
p()): 2ptt) r(5);
8: end for
9: ( + W ((T)
10: Returns: S(T), I LTT)
In this formulation, S' should maintain the same sparsity pattern as S, ensuring that S S" @ P and S = S @ P are satisfied. It's important to note that both S' and can have elements of any magnitude without restrictions. Lx is the best rank-k approximation of L, which can be obtained using the SVD of L as L;- = U. (L) (L). Be- ginning with fixed S, we have the bascline:
W = W O P (WG1- (4)
sparse part S!
low-mank part
Where Q is learnable matrix in the same shape as W. Baseline 2 (PCP with mask). By directly substituting the
decomposition from (4) into the optimization problem presented in E3)(3) yields:
Baseline 1 (Zero-shot SVD). The most straightforward method to obtain the low-rank matrix Lg is to directly per- form the SVD difference matrix W-S without updating
+ m Pl
min (1- Q Q
O P P Pll,,
S. This can be expressed as La- = (. - S)E.,, VV (W- - 5). Here, (-). (.), and V+x(+) represent the first & columns of (-). the top-left & x I submatrix of E()), and the first & columns of V(-), respectively.
and L = W ( P)
+ Q + P. (5)
This equation represents a constrained optimization problem where we seek to find the optimal Q. The binary mask P plays crucial role in maintaining the desired sparsity pat- tern. To solve this optimization problem, we can employ it- erative methods such as gradient descent or its variants. Fol- lowing Candès et al. (2011), we set À = 1//mmaae max(m, n). However, the nuclear norm minimization approach has a limitation: it applies equal shrinkage to all rank compo- nents (Zha et al., 2019). This uniform treatment may not be optimal for our objective in Eq.(2), where we aim to approximate the residual matrix W - S using low-rank matrix L2 with a rank lower than that of L.
Parameter efficiency analysis. We analyze the computa- tional efficiency of low-rank refinement by examining its parameter count and FLOPPs (floating-poin operations). We compare these metrics with the dense model and a pruned model, considering the impact of sparsity. For a weight matrix W e Rmxx the dense model has mn parameters and requires 2mn FLOPPs for forward pass. With pruning at sparsity ratio p. these reduce to (1 - p)me parameters and 2(1 - (p11p)mm FI.OPs. Our method introduces low-rank matrix L = B A. which introduces k(m + n)) parameters and 2m m + л)) FLOPs, slightly reducing the overall spar- sity ratio by k( ( + H). The choice of l and p presents trade-off between model size, computational complexity, and performance. In practice, using a rank & of 128 results in only 4.9% increase in parameters, while possibly re- ducing perplexity by half on a ILLMA-7B model with both unstructured 50% sparsity and structured 4:8 sparsity.
To address this limitation, we propose a forward-only method that prioritizes the preservation of low-rank compo- nents associated with larger singular values, while allowing for a more aggressive reduction of components with smaller singular values. This approach offers greater flexibility and precision in low-rank approximation compared to the PCP baseline, aligning more closely with our goal of achieving an efficient low-rank approximation. We begin by setting ((O) W @ P. and then iteratively refine S((+1) using the following update rule:
This optimization problem is generally NP-hard and similar challenges have been addressed in the fields of matrix com- pletion (Chandrasekaran et al., 2011) and robust Principal Component Analysis (PCA) (Candès et al., 10 Peng et al., 2020) by solving a Principal Component Pursuit (PCP) as:
((++1)
=g po
Σ as (1))) us (110) of (100) (6) (1,(0)
min
D||. + (1| L.S'
s.t. L + S' W.
(3)
= po {U,,0, (z E,10, (I. V (r,0)) }. (
4

---

Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision
Specirum of Singular Values
putational bottleneck in our algorithm is the SVD compu- tation performed in each iteration. For a weight matrix 2 nt), the time complexity of SVD is O(mn²). With T iterations, the overall time complexity is O(Tmn²). However, our method remains significantly more efficient than comparable approaches, which typically require thousands of iterations. While the zero-shot SVD baseline is the fastest with only one SVD computation, it generally yields lower-quality results as shown in our experiments. Our method achieves substan- tially better performance than zero-shot SVD with relatively few iterations (I' = 50). For detailed analysis of the infer- ence acceleration and memory usage of our method, please refer to Appendix E.
Singolar Value Magnitde
== W P
1.0
PCP with mask
W E Rmmx (assuming
Ours (k-64)
Ours (k-128)
0.5
Ours (k-512)
100
200
300
400
500
Singular Value Index
(a) Singular value distribution of L
Normaiized Energy A.
1.00
Normalized Energy
0.75
Incrementally increasing the rank n(() from 2 k over T iterations enables more nuanced exploration of the weight space. Furthermore, the consistent application of the binary mask P throughout this optimization process ensures the preservation of the desired sparsity pattern.
S W P
0.50
PCP with mask
Ours (k-64)
0.25
Ours (k-128)
Ours (k-512)
0.00
100
200
300
400
500
Singular Value Index
In Figure 2, we show the singular value spectrum of L and the cumulative energy retention in L2 for different methods using 512 x 512 weight block extracted from a fine-tuned LLaMA-7B model. Subfigure 2a demonstrates that the proposed method with different target rank values h (64, 128, and 512), consistently produces a more pro- nounced decay in singular values compared to both the zero-shot SVD and the PCP baseline. Subligure 2b shows that the cumulative energy retention in L, calculated as E(i = Σj= of(L) E/ (L) or || | /., in- creases more rapidly with truncated rank i for the proposed method compared to the PCP baseline 2.
(b) Cumulative energy retention in L
Figure 2: Analysis of the residual matrix L = W - S' and its low-rank approximation La of zero-shot SVD, PCP baseline (T = 5000), and our method with varying k (64, 128, 512) and T = 50. (a) Singular value spectrum of L. (b) Proportion of total energy captured by the top k singular values, calculated as Σj- (Lx))/ Ej= of (L).
In this equation, (t) represents the difference W S((), and p(() is defined as [1 + [=]+], where T denotes the total number of iterations, k is the target rank of Li. and ranges from 0 to T - 1. The linear schedule of p()) serves sev- eral important purposes: (1) starting with low rank forces the algorithm to first capture the most important singular components, which typically contain the majority of the energy, (2) each iteration then gradually incorporates more subtle details from higher singular components, and (3) this progressive approach helps maintain numerical stability and accelerates convergence compared directly optimizing for the target rank k from the beginning. We summarize the above algorithm in Algorithm 1. In each iteration, our algorithm focuses on eliminating the least significant com- ponents from the residual matrix (*), which are associated with smaller singular values, while preserving those with larger singular values. This approach allows for a progres- sive refinement of the sparse matrix S. As advance through the iterations, we gradually incorporate more subtle details from the original dense weight matrix W into the up- dated sparse term S. For detailed guidelines on selecting the hyperparameters and T, we refer readers to Appendix D.
From Algorithm 1 and observations from Figure 2, we can summarize the technical contributions of our proposed method as follows: (1) Iterative refinement with adaptive rank increase. Our method progressively refines the sparse matrix S while gradually increasing the rank ,'(t) from 2 to k over T iterations, prioritizing the most significant singu- lar components of the residual matrix L first and gradually incorporating more subtle details. (2) Sparsity preservation. Throughout the optimization process, our method strictly maintains the binary mask P. ensuring complete preserva- tion of the desired sparsity pattern. This enables compatibil- ity with both unstructured and structured pruning techniques for hardware-efficiee implementations. This flexibility ex- tends beyond the N:M structured sparsity patterns discussed in our experiments to other hardware-efficien patterns such as block sparsity (e.g., 4x4 blocks) and channel pruning since Theorem 4.1 guarantees that our method preserves any arbitrary sparsity pattern defined by the mask. (3) Efficient information capture. By focusing on preserving compo- nents with larger singular values first, our approach creates low-rank approximations that more efficiently capture the es-
Computational complexity analysis. The primary com-
5

---

Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision
sential information of the original weight matrix compared to alternative methods.
Theorem 4.5 (Error Bound), At each iteration I, the Frobe- nius norm of the difference between the original weight
matrix W and its approximation S(t) + L is bounded by:
4.1. Theoretical Analysis
In this section, we delve into the theoretical underpinnings of our proposed method, offering a more rigorous analysis of its properties and performance. We present key theorems that elucidate the behavior of our algorithm, focusing on two critical aspects: sparsity preservation and convergence.
| - - (s 1
Σ of (L
(10)
1=++1
S k)ou=1 (b"1), (11)
where 0i (I( are the singular values of L() W S(0,, and r rank (L()).
First, we demonstrate that our method maintains the de- sired sparsity pattern throughout the iterative process, en- suring that the final solution adheres to the specified binary mask. This property is crucial for applications where spe- cific weight connections must remain zero.
For more comprehensive treatment, including detailed proofs of these theorems and additional supporting lemmas, we direct the reader to Appendix A. This appendix contains the full mathematical derivations and supplementary results that underpin our theoretical analysis.
Theorem 4.1 (Sparsity Preservation, Proof. A.2.). For all iterations l, the sparsity pattern of ( is preserved and matches the binary mask P, i.e., ( = S @ P for all t.
5. Experiment
Second, we prove the convergence of our algorithm, show- ing that it approaches a well-defined solution as the number of iterations increases. This convergence guarantee provides theoretical justification for the stability and reliability of our method.
5.1. Experimental Setup
We conducted our experiments using LLaMA models, eval- uating their performance on the WikiText-2 (Merity et al., 2016) and standard benchmarks including TruthfulQA (Lin et al., 2021), GSM8K (Cobbe et al., 2021), ARC-C (Clark et al., 2018) and MMLU (Hendrycks et al., 2020). We report perplexity on WikiText-2, where lower values indicate better performance. We evaluate various sparsity levels, including unstructured sparsity and structured N:M sparsity.
Theorem 4.2 (Convergence). For any weight matrix W E RMXn and binary mask P E {0, 16,11 jmxx, the iterative weight update algorithm converges to a solution (S L*) as T + oc, such that W= L'' 5 = 5" 3 P. More- over, 1om ([S(T) l F = O and +ec [I
[F O.
Theorem 4.3 (Asymptotic Convergence). There exists a time step To such that for all t > To, the Frobenius norm of the error decreases monotonically, i.e., for some
It is important to note that our proposed iterative refinement method is entirely data-free and does not require calibration data, as shown in Algorithm 1. We consistently use T = 50 across all experiments, which is sufficient for achieving most of the potential error reduction while maintaining computa- tional efficiency. When implementing Wanda pruning (Sun et al., 2023) and our method combined with Wanda (Wanda Ours), we use 128 sequences from the 'allenai/c4" dataset as calibration data. For evaluation, we use 128 sequences from WikiTexxt22 dataset for perplexity evaluation.
l - (g
L
" S w - (s ((0))
A corollary of Theorem. 4.3 is that if we fix (()) = k for all i, the Frobenius norm of the error decreases monotoni-
cally. In this case, the term P @ L (0.) vanishes, and (44) becomes || | Formally, we have the following corollary:
(t)
E
|| E
5.2. Effectiveness of Low-Rank Refinement
Corollary 4.4 (Monotonic Improvement). If we choose to fix r(t) = k for all t, the Frobenius norm of the error decreases monotonically, i.e.,
Low-rank refinement advantage. First, we aim to examine the benefits of low-rank refinement on 2ikiText-2. In Fig- ure 3, we show the perplexity evaluation for various sparsity levels using magnitude pruning and low-rank refinement methods, with the target rank set to 128. In subfigure 3a, we compare the perplexity of sparsity- pruning and zero-shot SVD refinement to highlight the low-rank refine- ment ad adanttaee. By incorporating a low-rank structure into the pruned weights, the parameter count increases. The x- axis represents the parameter reduction relative to the dense model instead of the sparsity level, and we use dashed
E
E
-
PO
E
50, (9)
where the equality holds if and only if E(t) O.
However, our empirical observations suggest that fixing (tt) = throughout the process is not optimal in practice, and gradually increasing (t) as the iterations progress leads to faster convergence.
6

---

Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision
Table I: Comparison of WikiText validation perplexity (L is better) across various sparsity levels on LLaMA-7B. All methods and sparsity levels use target rank of & 128 (4.9% more parameters).
SPARSITY LEVEL (LLaMA-7B)
METHOD
50%
60% %
70%
4:8
2:4
Dense
5.68
Magnitude
17.29 (0%)
152.36 (0%)
48427.85 (0%)
16.83 (0%)
42.53 (0%)
w/ Zero-shot SVD
8.06 (-53,4%)
13.59 (-91.1%)
283.74 (-99.4%)
9.29 (-44.8%)
12.72 (-70.0%)
w/ PCP with mask
8.70 (-49.7%%
16.67 (-89.1%)
727.54 (-98.5%)
10,60 (-37.0%)
16.62 (-60.9%)
w/ Ours
7.97 (-53.9%)
12.14 (-92.0%)
200.09 (-99.6%)
8.86 (-47.4%)
10.74 (-74.7%)
Wanda
7.26 (0%)
10.69 (0%)
84.69 (0%)
8.57 (0%)
1 11.53 (0%)
w/ Zero-shot SVD
7.09 (-2.3%)
9.60 (-10.2%)
35.65 (-57.9%)
8.14 (-5.0%)
10.48 (-9.1%)
w/ PCP with mask
7.28 (+0.3%)
10.19 (-4.7%)
47.11 (-44.4%)
8.63 (-0.7%)
11.22 (-2.7%)
w/ Ours
6.92 (-4.7%)
8.97 (-16.1%)
32.90 (-61.2%)
7.74 (-9.7%)
9.18 (-20.4%)
Table 2: Comparison of Wiki'Tett validation perplexity (J is better) across various sparsity levels on LLaMA-13B. All methods and sparsity levels use target rank of k 128 (3.8% more parameters).
SPARSITY LEVEL (LLaMA-131)
METHOD
50% %
60%
70%
4:8
2:4
Dense
4.57
Magnitude
5.98 (0%)
9.91 (0%)
408.75 (0%)
6.76 (0%)
8.32 (0%)
w/ Zero-shot SVD
5.73 (-4.2%)
8.83 (-10.9%)
163.96 (5).9%)
6.58 (-2.7%)
8,86 (-6.5%)
w/ Ours
5.65 (-5.5%)
8.83 (-10.9%)
99.27 (-75.7%)
6.40 (-5.3%)
7.76 (-6.7%) %)
black arrow to indicate the correspondence between the pruned model and the model with refinement of the same sparsity level. The results indicate that gain 8.6% param- eter reduction at approximately 50% sparsity, highlighting the superiority of low-rank refinement compared to sparsity- only pruning techniques. In subfigure 3b, we compare our method with other baseline methods. Our method consis- tently outperforms the baseline methods, with the benefits becoming more pronounced higher sparsity levels. It is evident that PCP with mask under-performe compared to zero-shot SVD and our iterative weight update method, al- though it still surpasses sparsity-only pruning. This can be attributed to PCP's uniform shrinkage of all singular values, including smaller ones, and our choice of a very low target rank = 128. To better understand the performance of PCP, we visualize the singular value spectrum of PCP in Figure 4a, Appendix B. PCP yields larger singular values for indices ranging from 102 to nearly 103, demonstrating its less discriminative approach to value reduction.
results for different sparsity levels, ranging from 50% to 70%, as well as structured sparsity patterns like 4:8 and 2:4. For each sparsity level, we compare the performance of low-rank refinement incorporated with magnitude pruning and Wanda pruning (Sun et al., 2023). At 50% sparsity, our method reduces perplexity by 53.9% compared to sparse- only magnitude pruning. This improvement becomes more significant at higher sparsity levels, reaching 92.0% reduc- tion at 60% sparsity and 99.6% reduction at 70% sparsity, Our proposed method consistently achieves lower perplexity values across all sparsity levels. This detailed comparison highlights the effectiveness of our method in maintaining low perplexity even at higher sparsity levels, demonstrating its robustness and superiority over other baseline methods.
Benchmark results. In Table 3, we evaluate the perfor- mance of our method several benchmark datasets. We compare the performance of our method with that of dense models, sparsity-only pruned models, and low-rank refined using zero-shot SVD. Our method consistently outperforms both magnitude pruning and zero-shot SVD across most tasks and model sizes. For the 7B model, achieves a 13.2% reduction in average performance compared to 22.1 % for magnitude pruning and 18.5% for zero-shot SVD.
Comparison across different sparsity types and levels. To delve deeper into the effectiveness of low-rank refinement across varying sparsity types and levels, we offer a com- prehensive comparison of WikiText-2 validation perplexity using LLaMA-7B and 13B. The results presented in Tables I and 2 respectively. Where we show the experimen-
We further validate our approach on the more recent
7

---

Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision
Table 3: Performance comparison on LLaMA models across several benchmark datasets (k: = 128).
METHOD
TruthfulQA
GSM8K
ARC-c
MMLU
AVG.
Dense
34.1
10.3
44.7
32.1
30.3 (0%)
Magnitude 50%
35.3
1.0
33.5
24.6
23.6 (-22.1%)
7B
w/ Zero-shot SVD
34.3
1.5
36.9
26.0
24.7 (-18.5%)
w/ Ours
34.2
3.4
41.5
26.0
26.3 (-13.2%)
SparseGPT2:: 2:4
36.5
2.0
33.1
25.4
24.3 (-19.8%)
w/ Ours
33.8
2.7
36.1
29.1
25.4 (-16.2%)
Dense
36.9
23.4
49.1
52.1
40.4 (0%)
13B Magnitude 2:4
38.4
1.7
34.9
27.8
25.7 (-36.4%)
w/ Zero-shot SVD
37.6
1.8
32.2
27.0
24.7 (-38.9%)
w/ Ours
36.9
9.4
36.7
41.9
31.2 (-22.8%)
LLaMA-3.1 model, which has enhanced reasoning and mathematical capabilities compared to LLaMA-2. Detailed results are presented in Table Appendix C. On this fron- tier model, our method significantly improves performance over magnitude pruning across most benchmarks. Notably, on GSM8K, which tests mathematical reasoning, the dense model achieves 49.8% accuracy while magnitude unstruc- tured pruning at 50% sparsity reduces this just 1.3%. Our low-rank refinement method improves this to 6.5%, showing substantial recovery of the mathematical reason- ing capabilities. Similar improvements are observed across other benchmarks, with our method recovering more than 10 percentage points on HellaSwag, WinoGrande, ARC-e, ARC-c, and MMLU compared to magnitude pruning alone.
that our low-rank approximations preserve more information from the original matrix, thereby enhancing performance. (c) Error Analysis. This subfigure presents the Frobenius norm of the error E, W - -(((t + _.")), noting that E, = L - L; well. It is observed that at the desired target rank, our method achieves lower Frobenius norm error than both zero-shot SVD and the PCP baseline. (d) Convergence Analysis. We illustrate how the approxima- tion error between the dense matrix W and the combined approximation ((t) _ diminishes over iterations, as predicted by Theorem 4.2 and Theorem 4.3. Specifically, it
shows the Frobenius norm || various iterations for k 512 and
||
[ L || at
50.
Investigating the lower bound of rank(L) Following pre- vious analysis, intriguing question arises: Is it possible to fully compress the information contained in W O (1 P) into S In other words, does the rank of L have a lower bound? To address this, we conduct an empirical investiga- tion by applying our method to various target ranks using both synthetic matrices and real-world weight matrices ex- tracted from the LLaMA-7B model. We first construct series of synthetic matrices W of known rank by mul- tiplying two Gaussian random matrices: U E mm and V E RRxr such that W = UVT Here, m = n = 512. We subsequently apply our method along with zero-shot SVD to W, varying the rank r and sparsity level p. and visualize the spectrum of L W S' in Figure 5
5.3. Iterative Weight Update Analysis
In this section, examine the performance of our proposed Algorithm 1 empirically confirm the theoretical properties outlined in Section 4, specifically regarding convergence properties, error reduction, and the lower bound of rank L).
Convergence and error analysis. In Figure 4, we visualize W
several key properties of the residual matrix and its low-rank approximation Lg, we use a log scale for the æ-axis better visualize the decay of singular val- mag- nitude pruning with sparsity level of 50%. (a) Singu- lar Value Spectrum. We first visualize the singular value spectrum of for different methods and hyperparameter configurations for our method. It is clear that the itera- tive weight update method exhibits more pronounced de- cay singular values compared both zero-shot SVD and the PCP baseline. (b) Energy Retention. Here show the cumulative energy retention of L, calculated as Σj=1 of (Lx)) E;= o3
at low ranks. Throughout the analysis,
In both subfigures in Figure 5, Appendix B, the solid lines represent the zero-shot SVD baseline method (L = W + (1 ()). while the dashed lines represent the proposed method. The proposed method consistently shows a steeper decay in singular values compared to the zero-shot SVD method, indicating better compression of information. In subfigure 5a, we observe that for diverse ranks of W and constant sparsity level p 0.5, the rank of L remains invariant. Conversely, subfigure 5b shows that fixing the
|L,ll/// [|. Across var- ious target rank values, our method captures a higher propor- tion of energy within the top singular values, suggesting
8

---

Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision
LLaMA-7B Magnitude Pruning
its sparsity pattern and incorporating low-rank component. This approach effectively recovers crucial information lost during pruning, leading to performance recovery, especially at high sparsity ratios.
Sparse Only
Peplexitt (PPL)
10*
sero-shot SVD
10
These experimental results highlight the potential of com- bining low-rank and sparsity in LLMs. Future work could explore methods for automatically determining the optimal rank for the low-rank component based on the specific char- acteristics of each layer the overall model architecture.
10
8 6%
10'
0.4
0.5
0.6
Relative Parameter Reduction
(a) Low-rank refinement advantage.
Acknowledgements
LLaMA-7B Magnitude Pruning
10
Sparse Only
This work is supported by STI 2030 Major Projects (No. 20212ID2201400 405), Shenzhen Basic Research Project (Natural Science Foundation) Basic Research Key Project (NO. 01112024122212443222 ence Foundation of China (No. 62441619, 62276195 and U23A20318), Open Foundation of Key Laboratory of Cyberspace Security, Ministry of Education of China (..K0...222200003
Perplexity (PPL.)
Zero-shot SVD
w/ PCP with mask
10"
w/ Ours
x12.6
National Natural Sci-
14.5%
10
0.2
0.3
0.4
0.5
0.6
0.7
Sporsity Lovel
(b) Method comparison.
Impact Statement
Figure 3: Perplexity evaluation for different sparsity levels and methods, k = 128 for low-rank refinement. (a) Low- rank refinement advantage over sparsity-only pruning. To ensure fair comparison between methods, this plot, we report results based on total parameter count rather than just sparsity level. (b) Comparison of our proposed iterative weight update method with other baseline methods.
Our study focuses enhancing model performance through auxiliary-task learning, the broader impacts could be consid- ered from various aspects as follows: The proposed method could lead to more efficient machine learning models that require less data to achieve high performance. This could benefit areas where data collection expensive or difficult. Better-pertie setter-performin models could also be particularly useful in high-stakes domains such as healthcare (e.g., improved diagnostics through better image analysis) autonomous systems (e.g., safer navigation through enhanced scene un- derstanding). However, if the auxiliary tasks introduce bi- ased data, the merged model might amplify these biases, leading to unfair or discriminatory outcomes in applications like recruitment systems criminal justice. Future work could investigate this further.
rank of W at 512 and varying the sparsity level p from 0.3 to 0.7, reveals positive correlation between the rank of L and increasing sparsity. More specifically, the rank of L is around conclusion that the rank of L is independent of rank(W) but dependent on the sparsity level This is consistent with the intuitive explanation that as the sparsity level increases, the pruned matrix contains more information, and thus the rank of higher. Similar observations can be made in the L.L.MA7BB model, shown in Figure 2.
min(m,,) n). Empirically, we come to the
References
Brown, T. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
6. Conclusion
In this work, we present an iterative weight update algo- rithm for low-rank refinement of sparse-pruned models. Our approach aims bridge the performance gap between dense and pruned sparse models in large language models. The proposed method offers computationally efficient solution that does not rely on extensive datasets or high-performing teacher models, making it a practical choice for improv- ing sparse model performance. A notable advantage of our method is its sparsity- preserving property, which allows for the concurrent update of the sparse matrix while maintaining
Candès, E. J., Li, X., Ma, Y., and Wright, J. Robust principal component analysis? Journal of the ACM (JACM), 58(3): 1-37, 2011.
Chandrasekaran, V., Sanghavi, S., Parrilo, P. A., and Willsky, A. S. Sparse and low-rank matrix decompositions. IFAC Proceedings Volumes, 42(10)):493-1499 2009.
Chandrasekaran, V., Sanghavi. S., Parrilo, P. A., and Willsky, S. Rank-sparsity incoherence for matrix decomposi-

---

Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision
tion. SIAM Journal on Optimization, 21(2):572-59, 2011.
Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In Inter- national Conference Machine Learning, pp. 10323- 10337. PMLR, 2023.
Chen, S. and Zhao, Q. Shallowing deep networks: Layer- wise pruning based on feature representations. IEEE Iransactions on pattern analysis and machine intelligence, 41(12):3048-3053 2018.
Gale, T., Elsen, E., and Hooker, S. The state of sparsity in deep neural networks. arXiv preprint arXiv: arXiv:1902.00 11.9902.99774 2019.
Cheng, H., Gimbutas, Z., Martinsson, P.-G,, and Rokhlin, V. On the compression of low rank matrices. SIAM Journal on Scientific Computing,
Guo, H., Greengard, P., Xing, E. P., and Kim, Y. L4- lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. arXiv preprint arXiv:2311.12033,1 2023.
2005.
Cheng, Y.. Wang, D., Zhou, P., and Zhang, T. A survey of model compression and acceleration for deep neural networks. arXiv preprint
2017.
Han, S. Efficient methods and hardware for deep learning. PhD thesis, Stanford University, 2017.
Choudhary, T., Mishra, V., Goswami, A., and Sarangapani, A comprehensive survey on model compression and ac- celeration. Artificial Intelligence Review, 53:5113-5155, 2020.
Han, S., Mao, H., and Dally, W. J. Deep compres- sion: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint wrXiv:1510000149 2015a.
Chung. H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fe- dus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-fin finetondd language models. Journal of Machine Learning Research, 25(70): 2024.
Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015b.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint orXiv:/803.088 180300557, 2018.
Haroush, M., Hubara, I., Hoffer, E., and Soudry, D. The knowledge within: Methods for data-free model com- pression. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8494- 8502, 2020.
Cobbe, K., Kosaraju, V. Bavarian, M., Chen, M., Jun, H., Kaiser, L.,, Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 10. /4168, 2021.
Hassibi, B., Stork, D. G., and Wolff, G. J. Optimal brain surgeon and general network pruning. In IEEE interna- tional conference on neural networks, pp. 293-299. IEEE, 1993.
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint 1rXiv:1810.0480) 2018
He, Y. and Xiao, L.. Structured pruning for deep convolu- tional neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 2023.
Fan, A., Grave, E., and Joulin, A. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv: 1556, 2019.
He, Y., Zhang, X., and Sun, J. Channel pruning for acceler- ating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pp. 1389-1397, 2017.
Fan, S. Jiang, X., Li, X., Meng, X., Han, P., Shang, S., Sun, A., Wang. Y., and Wang. Z. Not all layers of lims are necessary during inference. arXiv preprint arXiv:2403.02181 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardr, J. Measuring mas- sive multitask language understanding. arXiv preprint 0XXi:22009003000 2020.
Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 118833006335 2018
Hinton, G. Distilling the knowledge in neural network. arXiv preprint 1503002330 2015.
Frankle, J., Dziugaite, G. K., Roy, D., and Carbin, M. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3259- 3269. PMLR, 2020.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal
10

---

Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision
large language models. arXiv preprint arX1v:2003./555, 2022.
Loeschcke, S., Toftrup, M., Kastoryano, M. J., Belongie, S., and Snæbjarnarson, V. Loqt: Low rank adapters for quantized training. arXiv preprint arXiv:2405.16528. 2024.
Idelbayev, Y. and Carreira-Perpinán, M. A. Low-rank com- pression of neural nets: Learning the rank of each layer. In Proceedings of the IEEE/CVE Conference on Com- puter Vision and Pattern Recognition, pp. 6049-8059, 2020.
Men, X., Xu, M., Zhang. Q., Wang, B., Lin, H., Lu, Y., Han, X., and Chen, W. Shortgpt: Layers in large language mod- cls are more redundant than you expect. arXiv preprint arXiv:2403.0388 2024.
Kim, B.-K., Kim, G., Kim, T.-H., Castells, T., Choi, S., Shin, J., and Song, H.-K. Shortened Ilama: A simple depth pruning for large language models. arXiv preprint 24xiv22402.02834 2024a.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint wrXiv:160.0084333 2016.
Kim, J., Halabi, M. E., Ji, M., and Song, H. O. Layermerge: Neural network depth compression through layer pruning and merging. arXiv preprint arXiv:2406.1 12837, 2024b.
Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R., Amatriain, X., and Gao, J. Large language models: survey. arXiv preprint arXiv:2402.06119 2024.
Mozaffari, M., Yazdanbakhsh, A., Zhang, Z., and Dehnavi, M. M. Slope: Double-pruned sparse plus lazy low- rank adapter pretraining of llms. arXiv preprint arXiv:2405.16322 2024.
Komatsuzaki, A., Puigcerver, J., Lee-Thorg J., Ruiz, C. R., Mustafa, B., Ainslie, J., Tay, Y., Dehghani, M., and Houlsby, N. Sparse upcycling: Training mixture- of-experts from dense checkpoints. arXiv preprint arXiv:2212.05005,, 2022.
Muralidharan, S., Sreenivas, S. T,, Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. Compact language models via pruning and knowledge distillation. arXiv preprint a4X 14679, 2024.
Kurtic, E. and Alistarh, D. Gmp*: Well-tuned gradual magnitude pruning can outperform most bert-pruning methods. arXiv preprint arXiv:2210.066.
2022.
LeCun, Y., Denker, J., and Solla, S. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Olver, J. and Shakiban, C. Applied Linear Algebra. Undergraduate Texts Mathematics. Springer Interna- tional Publishing, Cham, 2018. ISBN 978-3-319-91040- 978-3-319-91041-3 doi: 10.1007/978-3-31--91444
Li, Y., Yu, Y., Liang, C., He, P., Karampatziakis, N., Chen, W., and Zhao, T. Loftq: Nora-fine-tuning-aww quantization for large language models. arXiv preprint arXiv:2310008659 2023a.
URL http: 978-3-319 91041-3.
com/ 10 1007/
Peng, C., Chen, Y., Kang, Z., Chen, C., and Cheng, Q. Ro- bust principal component analysis: A factorization-based approach with linear complexity. Information Sciences, 513:581-599 2020.
Li, Y., Yu, Y., Zhang, Q.. Liang, C., He, P., Chen, W., and Zhao, T. Losparse: Structured compression of large lan- guage models based on low-rank and sparse approxima- tion. In International Conference on Machine Learning, pp. 2036-- PMLR, 2023b.
Snell, C., Lee, J., Xu, K., and Kumar, Scaling Ilm test- time compute optimally can be more effective than scal- ing model parameters. arXiv preprint 2433::2408.03314. 2024.
Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang. W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for on-device 11m compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
Sun, M., Liu, Z., Bair, A., and Kolter, Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1695, 2023.
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.079 079509999....... 2021.
Sun, W., Zhou, A., Stuijk. S., Wijnhoven, R., Nelson, A. O., Corporaal, H., et Dominosearch: Find layer-wise fine- grained M sparse schemes from dense neural networks. Advances neural information processing systems, 34: 20721-20733 2021.
Liu, Z., Li. Shen, Z., Huang, G., Yan. S., and Zhang, C. Learning efficient convolutional networks through net- work slimming. In Proceedings of the IEEE international conference on computer vision, pp. 2736 2736-2744, 2017.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P.,

---

Wan, F., Yang. Z., Zhong, L., Quan, X., Huang, X., and Bi, Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision
Bhosale, S., al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Wan, F., Huang, X., Cai, D., Quan, X., Bi, W., and Shi, S. Knowledge fusion of large language models. arXiv preprint ar arxiv:2401. 10491, 2024a.
W. Fusechat: Knowledge fusion of chat models. arXiv preprint ar X1v:2242..1117,, 2024b.
Xia, M., Gao, T., Zeng. Z., and Chen, D. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.066 10.0660, 2023.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR. 2023.
Yin, L., Jaiswal, A. K., Liu, S., Kundu, S., and Wang, Z. Junk DNA hypothesis: Pruning small pre-trained weights irreversibly and monotonically impairs "difficult" down- stream tasks in llms. In Forty-first International Confer- ence on Machine Learning. 2024.
Yu, X., Liu, T., Wang, X., and Tao, D. On compressing deep models by low rank and sparse decomposition. Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 1370-7374 2017.
Zha, Z., Wen, B., Zhang, J., Zhou, J., and Zhu, C. A comparative study for the nuclear norms minimization methods. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 2050-2054. IEEE, 2019.
Zhang, M., Chen, H., Shen, C., Yang. Z., Ou, L., Yu, X., and Zhuang, B. Loraprune: Pruning meets low- rank parameter-effiieet fine-tuning. arXiv preprint arXiv:2305.1844 2023.
Zhang, S. and Papyan, V. OATS: Owtre Pruning Through Sparse and Low Rank Decomposition, Septem- ber 2024. URL http:// http:://axxiii /twx/aavvvoo../ /ttp///ii..oo/////////////////// 13652. 1332 [es].
Zhou, T. and Tao, D. Godec: Randomized low-rank & sparse matrix decomposition noisy case. Proceed- ings of the 28th International Conference on Machine Learning, ICML 2011, 2011.
Zhuang. Z., Tan, M., Zhuang, B., Liu, J., Guo, Y., Wu, Q., Huang, J., and Zhu, J. Discrimination-aware channel pruning for deep neural networks. Advances in neural information processing systems, 31, 2018.
12

---

Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision
Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision
The appendix is divided into several sections, each giving extra information and details.
Theoretical Analysis
13
A.I Additional Lemmas
13
A.2 Proofs of Theorems
14
B Additional Experimental Results of Iterative Weight Update Analysis
16
Experiments with Llama-3.1-8B
18
Guidelines for Hyperparameter Selection
18
End-to-End Inference Acceleration and Memory Usage
18
E.1 Evaluation without Sparse Matrix Formats
18
E.2 Evaluation with Sparse Matrix Formats and Hardware Acceleration
18
A. Theoretical Analysis
In this section, we provide the proofs for the theorems presented in the main text and additional lemmas.
A.1. Additional Lemmas
In this subsection, present additional lemmas that useful for the proofs of the theorems in the main text.
Lemma A.1 (Mask Norm Inequality I). For any matrix M E Rmxn and binary mask P E {0, 10 1|mx,, the following inequality holds:
P F S |M
(12)
Proof of Lemma A. Let may and Pij the elements of and respectively. By definition of the Frobenius norm and element-wise multiplication:
P
ΣΣ
(pommsy) =
Piy m
(13)
=1
1=1 1=1
Since P is a binary mask, Pty {0, 1], which means plz = Pij. Therefore:
m a
m
1
| M
ΣΣ
Pirm
m = |
(14)
j=1
Taking the square root both sides preserves the incquality:
[ M S |
(15)
Lemma A.2 (Mask Norm Inequality II). For any matrix E Rmxx and binary mask P. E (0,1)mxn, mxn, the the following inequality holds:
(M,P O M)F S M)
(16)
Proof of Lemma A.2. Let mi and Pij be the elements of M and P respectively. By definition of the Frobenius inner product and element-win multiplication:
m
m D
< m
(M, MMPMMM)F
ΣΣ myy miy (P.y
Pimm
m
M)
(17)
i=
i= 1=1
13

---

Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision
Lemma A.3 (Mask Norm Equality I), For any matrix M E R and binary mask E {0, [0,1)mxn, the following equality holds:
(M,PO |
(18)
Proof of Lemma A.3. Because P is binary mask, we have Pij = Pij. Therefore, we can rewrite the Frobenius inner product as:
(M, M) ΣΣ
may (P)gmm)
Σ
píy mi
P 3
(19)
=1
/=1==1
Lemma A.4 (Mask Norm Equality II). For any matrix A, R maxn and binary mask P {0, 10.11mxx xxn, the following equality holds:
(P P
(20)
Proof of Lemma A.4. By definitions of Frobenius inner product and element-wise multiplication, we have:
P Deoaey)(issa
Piy asybij
(21)
(A,
a4 (pisbb))
Ptj aij Day
(22)
m
m
P (
Σ
(pizasy)bus
Σ
Pijazz bij
(23)
j=
For binary mask, pi = Pij. so the three expressions are equivalent.
A.2. Proofs of Theorems
Proof of Theorem 4.1. prove this by induction. For / = 0, ( W ④ P, so the property holds. Assume the property holds for iteration t, i.e., (tt) 500 @ P. We need to prove that it holds for iteration + 1, i.e., S(t+1) s(++1) P.
Therefore, have:
-(t-1)=s()) PO UpO: (L()) E, ((1))
(L() Vitos
(z(t}}
= (Stt) 3 P) + P (SVD terms)
(applying induction hypothesis)
= P (s )- SVD terms)
Po ((++))
Thus, we have shown that if the property holds for I, it also holds for + 1. Combined with the base case, this completes the induction proof, showing that ((t) (()) @ P for all t.
Proof of Theorem 4.2. Let S* and L' be the limits of S(T) and (T) as T + oc, respectively. We will show that these limits exist and satisfy the stated properties. First, note that for any t, S(t) = ((t) @ P by construction of the algorithm. Let
4 = (t++1) s(t [.. p. We need to show that limy cr-1 = 0. At each iteration t, we have:
(1+1)= + +PO
U UnM (I())
Epces: (I())
Vilo: (I())
(24)
= S(() + P W - ((1)
(25)
(0):
14

---

Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision
where ()) denotes the truncated SVD reconstruction from r(0) onwards. From the properties of SVD, can express the difference between the truncated SVDs
w (w- s(0) r(); -W-- k: 1=p0+
(W-5(()
( (1) i
(
(26)
( where of U; and v; are the i-th singular value and corresponding left and right singular vectors of W - SS()) The Frobenius norm of this difference is:
Σ
(0)
(27)
imp(144
1=p0)41
While we cannot directly guarantee that this sum approaches zero as t increases, we can bound it as:
(0) $ (k- (1- (o!0)-1)
(28)
As + 7 - r(1) + ki so k - r(0) + 0. Thus, for any 6 > O, there exists a To such that for all >> To, we have
(1-)(0) (o )² ² Now, considering the effect of the binary mask P. according to Lemma. A.1, we have:
p (w- - s()) ,(o; - (w (w-St)) 1l $ (w--st) )t))
p(u: - (W-S <c
(29)
Setting 5 = 6, we have shown that for any 5 > 0, there exists To such that for all To:
P w 5(() ,0: o W (w-5(t) -(())
k: 1 <
(30)
This implies that for / > To:
4 = |S - 5000 <
(31)
Since 8 can be arbitrarily small, we conclude that limm 6T-1 = 0. This shows that the sequence (S(T) is Cauchy and
therefore converges to some limit S*. Since W = (t + (t) for all t. and ((T) converges to S", it follows that (T))
must converge to L = W - S Finally, since ((1) = (t) @ P for all I, we have S = S @ P. Thus, the algorithm converges to the solution (S", -a) as T + oo, satisfying all the stated properties.
Proof of Theorem 4.3. Let E = W- - (sto) + - = (L))) k: be the error at iteration t. From the update rule in E7),7), we have:
E(++1) W -
W- (s(++1) + L 1121)
(32)
W (5()) +P (I())
+ +L ((+1))
(33)
r(0:
[ p (I())
L (++1)
(34)
r(t):
Let's denote A (L()) ,()) Then E(++)) (t) A)
-L (++1)
By construction, L
(++1) is the best rank-k
b
approximation [((1) ((t) A. Therefore:
+ = [Z( - -A) I ((+1)
[I <1(10) A) -1(0
<
(35)
F
Now, (()) (1) -E(), have:
B(++1) 2
< (t))) A) (0)
(I ( A 2
(36)
F
F
E(()) A |
||-11 (E10, ,,A)
(37)
15

---

Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision
According to Lemma A.2 and A = P @ ( - -E(t). we have:
E A (E"), P
L (1) +E(D))
(38)
EEW,
+ (E P E
(39)
F
EE P
L
10 1l
+
+|| 1
(40)
The last equality follows from Lemma A.3. Substituting this back into our earlier inequality (37):
E
(++1)
E
+ A| 2
E(()) AA4
(41)
(x
+ A 2 F 2
(E
PO
(8)
- 2
P E
o
(42)
F
+ P@
L
(t)
2
+
P
E
2
+
P
E
-00sk
F
F
0
E
2
P
E
(by Lemma A.4)
(43)
(()
2
+
P
(1)
P
E
(44)
10kkk
2
As + 11, - k so
P
+ 0. Therefore, there exists a time step To such that for all t > Tu.
F
E E(++1)
2 |
S P O E
2
I O.
F
Proof of Theorem 4.5, At iteration f, have W (t4)__ + (4) by construction. Let (()) UVVT be the SVD of (
Then E = -U, Ex V the best rank-k approximation of ((), The error can be expressed as:
W (sth + (00
(5( + E(0) - (5 + =b/"
(45)
| L
(0)
UR: kx. Vil 1|
(46)
F
r
Σ of (LW) S ( (E())
(47)
i=k+1
B. Additional Experimental Results of Iterative Weight Update Analysis
In this section, we provide additional experimental results of iterative weight update analysis.
In Figure 4, we present comprehensive analysis of the residual matrix L W S' and compare various methods for computing its low-rank approximation Lg. The figure illustrates four key aspects: (a) the distribution of singular values of L, (b) the cumulative energy retention across different singular values, calculated as Σx= of (Lx)/ Σi (L) (c) the
approximation error in terms of Frobenius norm, L L, and (d) the convergence behavior across iterations.
In Figure 5, we analyze how the singular value spectrum of L varies under different experimental conditions using synthetic weight matrices W. We compare proposed method against the zero-shot SVD baseline (L W o (1 - P)) in two scenarios: (a) varying the rank of the original weight matrix W (64, 128, 256, 512). and (b) different sparsity levels (0.3, 0.5, 0.7) achieved through magnitude pruning. This analysis helps us understand how the structural properties of the residual matrix are affected by these key parameters.
16

---

Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision
Spectrum of Singular Values
Normalized Energy of L
Singulas lar Value Magnitode
S W
1.00
10-
Ours (k=64)
PCP with mask
Normalized Energy
0.75
Ours (k=128)
0.50
Ours (k=512)
0.25
0.00
10
10
10
10'
10
10'
10"
10'
Singular Value Index
Singular Value Index
(a) Singular values distribution of L.
(b) Cumulative energy retention of L.
Frobenius norm of [L. L,Ily
Frohenius norm of ||Z. T.]l|=
60
60
Frobentus Norn
Frobenius Norn
40
Ours (1-10)
Ours-
Ours (1=30)
20
20
Ours (t=40)
Ours (1-50)
10a
10
10"
10²
10
10
10
10²
Singular Value Index
Singular Value Index
(c) Frohenius norm of | I Lellr.
(d)
E (
|r different time steps t (k 512).
Figure 4: Analysis of the residual matrix L = W - S' and its low-rank approximation L, using different methods. Results are shown for zero-shot SVD, PCP baseline (T = 100), and our proposed method with varying & (64, 128, 512) and T = 50. We show the r-axis in log scale and vertical dashed lines at = 64, 128, 512 for better visualization. Subfigures (a), (b), and (c) have shared legend. (a) Singular value spectrum of L. (b) Proportion of total energy captured by the top k: singular values, calculated as Σi=1 of( Σ (L). (c) Frobenius norm of the dropped matrix [L -- Lglle- (d) || - L,llæ at different number of iterations I. Here T 50 and = 512 and the vertical dashed line at 512 indicates the target rank,
Spectrum of Singular Values
Spectrum of Singular Values
700
3r
0.7.
600
800-
Singalar Vahie Magnitude
rank (W)
Magnitude
Sparsity Ratio
500
64
600
0.3
128
400
0.5
256
0.7
300
512
400-
Method
Method
Singulur
W
200
8 W
Ours
200
Ourss
100
0-
0
100
200
300
400
500
100
200
300
400
500
Singular Value Index
Singalar Value Index
(a) Spectrum of L across various ranks of W. (b) Spectrum of L across various sparsity levels.
Figure 5: Singular value spectrum analysis of L across different conditions. We compare our method with the zero-shot SVD method L W o (1 11-2 -P). (a) This subfigure shows the spectrum of L across different ranks of the original weight matrix W (64. 128, 256, 512). (b) This subfigure shows the spectrum of L across different sparsity levels (0.3, 0.5, 0.7) of magnitude pruning.
17

---

Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision
C. Experiments with Llama-3.1-8B
We also conducted additional experiments with more recent frontier models to verify our method's effectiveness on state-of- the-art art architectures. Specifically, we used Llama-3.1 3.1-B,, which represents a significant improvement over the Llama-2 series used our main experiments, particularly in reasoning and mathematical abilities.
Table 4: Evaluation results on Llama-3.1-88 across various benchmarks.
Model
HellaSwag
WinoGrande
ARC-e
TruthfulQA
GSM8K
ARC-c
MMLU
Dense baseline
78.9
73.6
81.1
45.2
49.8
53.4
63.5
Pruning Method
Magnitude 50%
56.4
57.6
56.7
42.9
1.3
35.8
35.3
Magnitude 50% with Ours
66.8
67.8
68.7
38.9
6.5
42.6
45.7
As shown in Table 4, our method significantly improves the performance of the magnitude-prunee model across most benchmarks. Notably, on GSM8K, which tests mathematical reasoning abilities, the dense model achieves 49.8% accuracy, while magnitude pruning at 50% sparsity drastically reduces this to 1.3%. Our low-rank refinement method recovers the performance to 6.5%, representing a substantial improvement over basic magnitude pruning. These results demonstrate that our approach is effective even on frontier models with enhanced reasoning capabilities, While there remains performance gap compared to the dense model, especially on complex reasoning tasks like GSM8K, our method consistently provides substantial recovery of capabilities lost through pruning.
D. Guidelines for Hyperparameter Selection
The proposed algorithm introduces two key hyperparameters: the target rank k and the number of iterations T. For the target rank k, we consistently use &o = 128 across our experiments, which offers a good balance between performance improvement and computational overhead. As shown in Figure 2, larger values of & (e.g.. 512) allow the method to retain more information but with diminishing returns relative to the increased computational cost. The optimal choice of should consider both the model size and the desired performance-efficeee primary computational bottleneck the SVD computation in each iteration, with a time complexity of (TTm²²) )) for weight matrix W R=xn (assuming m Z n). We find that T 50 provides sufficient convergence while keeping computational costs reasonable. As demonstrated in Figure 4(d) in Appendix B, further iterations yield diminishing returns in terms of error reduction, behavior also supported by Theorem 4.5.
trade-off. Regarding the number of iterations T, the
E. End-to-End Inference Acceleration and Memory Usage
In this section, we present a comprehensive analysis of the end-to-end inference acceleration achieved by our method compared to both dense and sparse models.
E.1. Evaluation without Sparse Matrix Formats
For fair comparison with the dense baseline model, we first took conservative approach our evaluation. We stored all matrices in their full dense format and kept all zero elements (the weight matrices as torch. Tensor objects), without leveraging any sparse matrix formats hardware-speciffe optimizations for acceleration. Table 5 shows the parameter count and evaluation time for the complete ARC-Challenge benchmark across different model configurations.
As shown in the table, without specialized formats, our method maintains computational efficiency comparable to the dense baseline, with only marginal increase in relative time (~~106xx) for the largest model configuration.
E.2. Evaluation with Sparse Matrix Formats and Hardware Acceleration
To fully exploit the potential efficiency gains from our method, we also evaluated performance when using sparse matrix formats and hardware acceleration. We converted the weight matrices to their sparse format using torch. sparse to.- sparse_send _st ructured to create torch. sparse SparseSeeee pares the memory usage of our proposed method with and without sparse matrix formats.
objects. Table 6 com-
18

---

Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision
Table 5: Parameter count and evaluation time for the complete ARC-Challenge benchmark (without sparse matrix formats).
Model Configuration
Non-Zero Parameter Count
Parameter Count
ARC-C
Relative Time
Llama-2-7B
Dense baseline
50.5s
1.0x
Magnitude 50% sparse
3.5B
6.7B
50.7s
~1.0x
Proposed Method (k=128, 50%)
3.8B
7.1B
50.5s
~1.0
Llama-2-I4e 2-13B
Dense baseline
13.0B
13.0B
73.2s
1.0x
Magnitude 2:4 sparse
6.7B
13.0B
73.9s
~1,01
Proposed Method (k=128, 2:4)
7.2B
13.5B
77.9s
~1.6x
Table 6: comparison of memory usage with and without end-to-end inference acceleration.
Model Configuration
Memory Usage
Relative Memory Usage
Llama-2-14 1-mna-2-133 23B
Proposed Method (k=128, 2:4), dense format
2x 14.6GB 2x9.1GB
1.0x ~0.6x
Proposed Method (k=128, 2:4), sparse format
When using sparse matrix formats, we observed a significant reduction in memory usage (approximately 38% reduction). The actual inference speedup varies based on factors like batch size and input sequence length. With aanewarr-accelerated N:M structured pruning (on NVIDIA Ampere GPUs and newer), we observed approximately ~-1 1x faster inference in wall-clock time compared dense models. It's important note that for unstructured pruning patterns or on GPUs without dedicated sparse acceleration support, pruning actually result in slower inference times compared to dense computation.
19


Reviews:
[{"review_id": "JLgYcfw9kw", "reviewer": "Reviewer_8WNE", "summary": "Magnitude pruning removes weights that have the smallest absolute values. However, traditional pruning methods require re-training the model to recover performance, which is computationally expensive and requires extensive data or teacher model. To address this, the authors propose to address this by approximating the dense matrix as the sum of a sparse matrix with maintained sparsity and a low-rank matrix. The main contributions are as follows:\n\n- Weight $W$ is dismantled into sparse part $S$ and low rank part $L$. The authors transform searching $S$ and $L$ into an optimization problem as Eq. (3). \n- The authors propose to incorporate the binary mask $P$ into the optimization process to ensure that the sparsity pattern of $S^′$ is fixed as $S$.\n- Iterative refine sparse weight with adaptive rank increase.\n\nIn the experiments, the proposed method is validated on WikiText-2, TruthfulQA, GSM8K, ARC-C and MMLU with Llama models. Experimental results demonstrate that low-rank refinement significantly enhances model performance, particularly at high sparsity levels.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful feedback and constructive suggestions.\n\n**W1: End-to-end inference acceleration is missing. It's better to report speedup for completeness.**\n\n1. Without sparse matrix formats or specialized hardware acceleration, the inference time is as follows:\n\nFirstly, for a fair comparison with the dense baseline model, we took a conservative approach in our evaluation. \nSpecifically, we stored all matrices in their full dense format and kept all zero elements (the weight matrices are `torch.Tensor` objects), without leveraging any sparse matrix formats or hardware-specific optimizations for acceleration.\n\nTable: Parameter Count and Evaluation Time for Complete ARC-Challenge Benchmark.\n\n| Model Configuration          | Non-Zero Parameter Count | Parameter Count | ARC-C | Relative Time |\n| ---------------------------- | ------------------------ | --------------- | ----- | ------------- |\n| *Llama-2-7B*                 |                          |                 |       |               |\n| Dense baseline               |                          |                 | 50.5s | 1.0x          |\n| Magnitude 50% sparse         | 3.5B                     | 6.7B            | 50.7s | ~1.0x         |\n| Proposed Method (k=128, 50%) | 3.8B                     | 7.1B            | 50.5s | ~1.0x         |\n| *Llama-2-13B*                |                          |                 |       |               |\n| Dense baseline               | 13.0B                    | 13.0B           | 73.2s | 1.0×          |\n| Magnitude 2:4 sparse         | 6.7B                     | 13.0B           | 73.9s | ~1.01×        |\n| Proposed Method (k=128, 2:4) | 7.2B                     | 13.5B           | 77.9s | ~1.06×        |\n\n2. With sparse matrix formats and hardware acceleration, we convert the weight matrices to their sparse format. Specifically, we use `torch.sparse.to_sparse_semi_structured` to convert the weight matrices to `torch.sparse.SparseSemiStructuredTensor` objects. The following table shows the memory usage of the proposed method with and without sparse matrix formats.\nThe actual inference speedup varies based on factors like batch size and input sequence length. With hardware-accelerated N:M structured pruning (on NVIDIA Ampere GPUs and newer), we can observe approximately ~1.1x faster inference in wall-clock time compared to dense models. However, it's important to note that for unstructured pruning patterns or on GPUs without dedicated sparse acceleration support, using sparse operations can actually result in slower inference times compared to dense computation.\n\nTable: A comparison of memory usage with and without end-to-end inference acceleration.\n\n| Model Configuration                                                                         | Memory Usage | Relative Memory Usage |\n| ------------------------------------------------------------------------------------------- | ------------ | --------------------- |\n| *Llama-2-13B*                                                                               |              |                       |\n| Proposed Method (k=128, 2:4), weight matrices are `torch.Tensor`                            | 2x14.6GB     | 1.0x                  |\n| Proposed Method (k=128, 2:4), weight matrices are `torch.sparse.SparseSemiStructuredTensor` | 2x9.1GB      | ~0.62x                |\n\n\n**W2: I suggest that the authors add add some frontier models, such as Llama 3.1 8B. Llama is quite old and may lack reasoning and mathematics abilities. This may affect the GSM8K results.**\n\nThank you for the suggestion. We have added Llama-3.1-8B to the experiments. The results are shown in the following table.\n\nTable: Llama-3.1-8B results.\n\n| Model                | HellaSwag | WinoGrande | ARC-e    | TruthfulQA | GSM8K   | ARC-c    | MMLU     |\n| -------------------- | --------- | ---------- | -------- | ---------- | ------- | -------- | -------- |\n| Dense baseline       | 78.9      | 73.6       | 81.1     | 45.2       | 49.8    | 53.4     | 63.5     |\n| *Pruning Method*     |           |            |          |            |         |          |          |\n| Magnitude 50%        | 56.4      | 57.6       | 56.7     | **42.9**   | 1.3     | 35.8     | 35.3     |\n| Magnitude 50% + Ours | **66.8**  | **67.8**   | **68.7** | 38.9       | **6.5** | **42.6** | **45.7** |", "claims_and_evidence": "Most of the claims are clear and convincing.", "methods_and_evaluation_criteria": "Most of the methods and evaluation criteria make sense.", "theoretical_claims": "Most of the proofs make sense.", "experimental_designs_or_analyses": "Most of the experimental designs are valid.", "relation_to_broader_scientific_literature": "This study is closely related to previous post-training pruning works, such as SparseGPT and Wanda.", "essential_references_not_discussed": "No missing related works.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper is well organized and well written. The technical content is explained in sufficient details. The equations are very clear and easy to understand.\n2. Comprehensive experiments performed in this paper. The reviewer appreciates the authors' effort in validating generation tasks such as GSM8K, rather than simply providing perplexity results. Since generation tasks is usually harder, the improvement is significant.\n\nWeaknesses:\n\n1. End-to-end inference acceleration is missing. It's better to report speedup for completeness.", "other_comments_or_suggestions": "Comments:\n\n1. I suggest that the authors add add some frontier models, such as Llama 3.1 8B. Llama is quite old and may lack reasoning and mathematics abilities. This may affect the GSM8K results.", "questions_for_authors": "In general, this paper performs well in its clarity, structure and completeness. The idea is innovative and the improvement is significant. However, end-to-end inference acceleration is missing. I will recommend a weak accept, and I suggest the authors pay attention to the inference part, as well as results on latest language models. Should those weaknesses and questions be addressed I will raise my scores accordingly.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HMcganMxcK", "reviewer": "Reviewer_5GnU", "summary": "This paper introduces a novel approach to improve the performance of sparse language models through low-rank refinement. The main contribution of the paper is a method that refines sparse models using a low-rank refinement, which leads to improved precision. This approach is theoretically grounded, with proofs and additional lemmas provided in the appendix to support the claims.\n\n## update after rebuttal\nI will keep my ratings since most of my concerns are solved.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: While the paper mentions magnitude pruning and N:M structured pruning, it does not discuss structured sparsity techniques, such as block sparsity or channel pruning, which have been shown to improve hardware efficiency and model performance.**\n\nWe thank the reviewer for pointing out the importance of structured sparsity techniques.\nIt is straightforward to apply the proposed method to structured sparsity techniques such as block sparsity or channel pruning by by simply changing the binary mask matrix P.\nThe core algorithm is compatible with any arbitrary binary mask pattern, including structured ones. For structured patterns like block sparsity (e.g., 4×4 blocks) or channel pruning, P would have the corresponding pattern of 0s and 1s.\n\nTheoretically, Theorem 4.1 (Sparsity Preservation) guarantees that the method preserves whatever sparsity pattern is defined by $P$.\n\n**W2: This paper does not include the full details of lemma A4.2 in the Appendix A.**\n\nThe proof of Lemma A4.2 currently appears before the lemma itself. In the revised version, we will relocate the proof to its proper position following Lemma A4.2.\n\n*proof of Lemma A4.2*:  Let $m_{ij}$ and $p_{ij}$ be the elements of $M$ and $P$ respectively. By definition of the Frobenius inner product and element-wise multiplication:\n  \n$$\n\\langle M, P \\odot M \\rangle_F = \\sum_{i=1}^m \\sum_{j=1}^n m_{ij} (p_{ij} m_{ij}) = \\sum_{i=1}^m \\sum_{j=1}^n p_{ij} m_{ij}^2\\leq \\sum_{i=1}^m \\sum_{j=1}^n m_{ij}^2 = \\langle M, M \\rangle_F\n$$\n\n**W3: The iterative refinement process introduces new hyperparameters (e.g., rank k, number of iterations T), but the paper does not provide a clear guideline on how these should be selected across different models and sparsity levels.**  \n\n- **Rank parameter k**:\nWe consistently set $k=128$ across the experiments to demonstrate the effectiveness of low-rank refinement.\nIn Figure 4(a), we visualize the singular value spectrum of the residual matrix $L=W-S'$ for different values of $k$ (from 64 to 512) and $T=50$.\nAs $k$ increases from 64 to 512, we see higher magnitudes for the top singular values. This indicates that larger $k$ values allow the method to retain more information.\nBut when $k$ is too large, the performance gain diminishes and the computational cost increases.\nTherefore, the choice of $k$ should strike a balance between the performance and the computational cost.\n\n- **Number of iterations T**:\nThe primary computational bottleneck in the proposed algorithm is the SVD computation performed in each iteration (line 171, page 4). For a weight matrix $\\mathbf{W} \\in \\mathbb{R}^{m \\times n}$, the time complexity of SVD is $O(mn^2)$ (assuming $m \\geq n$). With $T$ iterations, the overall time complexity is $O(Tmn^2)$.\nOn the other hand, as shown in Figure 4(d) and the diminishing convergence speed as stated by Theorem 4.5 (Error Bound), $T=50$ appears to be a reasonable choice for the number of iterations and further iterations yield diminishing returns in terms of error reduction.\n\n**W4: The computational cost of iterative updates may be high as the size of the weight matrix increases, which may limit the applicability of the method to very large models.**\n\nAs the size of the weight matrix increases, the computational cost of the proposed method increases accordingly.\nThe primary computational bottleneck in the proposed algorithm is the SVD computation performed in each iteration (line 171, page 4). For a weight matrix $\\mathbf{W} \\in \\mathbb{R}^{m \\times n}$, the time complexity of SVD is $O(mn^2)$ (assuming $m \\geq n$). With $T$ iterations, the overall time complexity is $O(Tmn^2)$.\n\nIn contrast, the PCP baseline would typically use an iterative optimization method (like Adam) that also requires SVD computations in each iteration to compute the nuclear norm. Furthermore, the PCP baseline requires substantially more iterations ($T=5000$) to achieve comparable results to the proposed method ($T=50$).\n\nTable: Computational Efficiency Comparison\n\n| Method             | Time Complexity | Typical Iterations | Relative Computational Cost |\n| ------------------ | --------------- | ------------------ | --------------------------- |\n| Proposed Algorithm | $O(Tmn^2)$      | 50 | 1× |\n| PCP Baseline       | $O(Tmn^2)$      | 5000 | ~100× |\n| Zero-shot SVD      | $O(mn^2)$       | 1 | ~0.02× |\n\n**Q1: In line 6 of Algorithm 1 $r(t) = 1 + (k-1) / (T-1)$, what is the intuition behind this equation?**\n\nThis equation in Algorithm 1 defines a linear schedule for increasing the rank from 2 to k across T iterations.\nStarting with low rank forces the algorithm to capture the most important singular components first, and each iteration gradually incorporates more subtle details from higher singular components.\nThe gradual increase in rank helps maintain numerical stability and helps the algorithm to converge faster.\n\n**Q2: How to select the hyperparameters? See weaknesses 3.**\n\nPlease refer to the response to W3.", "claims_and_evidence": "The claims made in the submission appear to be supported by theoretical analysis and experimental results.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria appear to be well-suited for the problem of enhancing sparse language models.", "theoretical_claims": "Yes. The paper includes theoretical analysis with proofs and additional lemmas to support its claims, particularly regarding the iterative weight update algorithm. The theoretical analysis demonstrates the favorable convergence properties of the proposed method and provides a rigorous foundation for its effectiveness.\n\nHowever, this paper does not include the full details of lemma A4.2 in the Appendix A.", "experimental_designs_or_analyses": "Yes, I checked the soundness and validity of the experimental designs. The paper presents experimental results to validate the proposed method, particularly focusing on the Llama models.", "relation_to_broader_scientific_literature": "This paper addresses the challenge of improving the performance of sparse language models, a well-studied problem in machine learning. Prior work has explored techniques like unstructured pruning and N:M structured pruning to reduce parameter count while maintain performance. The paper’s experimental results on the Llama model, align with prior findings that higher sparsity levels can lead to more severe performance degradation. This observation is consistent with the literature on sparse model performance, where sparsity is often traded off against computational cost and precision.", "essential_references_not_discussed": "While the paper mentions magnitude pruning and N:M structured pruning, it does not discuss structured sparsity techniques, such as block sparsity or channel pruning, which have been shown to improve hardware efficiency and model performance.", "other_strengths_and_weaknesses": "Strengths:\n\n1. This paper is well motivated and this approach effectively bridges the gap between dense and sparse models.\n2. The paper is well-structured, with a clear presentation.\n3. The theoretical analysis is solid.\n4. Code is provided for reproduction.\n\nWeaknesses:\n\n1. While the paper mentions magnitude pruning and N:M structured pruning, it does not discuss structured sparsity techniques, such as block sparsity or channel pruning, which have been shown to improve hardware efficiency and model performance.\n2. This paper does not include the full details of lemma A4.2 in the Appendix A.\n3. The iterative refinement process introduces new hyperparameters (e.g., rank k, number of iterations T), but the paper does not provide a clear guideline on how these should be selected across different models and sparsity levels.\n4. The computational cost of iterative updates may be high as the size of the weight matrix increases, which may limit the applicability of the method to very large models.", "other_comments_or_suggestions": "The proposed method is effective at high sparsity levels, but at low sparsity levels, the performance gain is not as significant.", "questions_for_authors": "1. In line 6 of Algorithm 1 $r(t) = 1 + \\frac{k-1}{T-1}$, what is the intuition behind this equation?\n2. How to select the hyperparameters? See weaknesses 3.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sYdqVQDkN2", "reviewer": "Reviewer_oW7q", "summary": "In this work, the authors proposes a low-Rank refinement method to factorize a dense full matrix into a sparse matrix and a low-rank matrix, bridging the performance gap between dense and sparse models. This approach iteratively improves the sparse weight matrix through a low-rank adjustment, thereby increasing model accuracy, particularly at higher levels of sparsity.\n\n## update after rebuttal\n\nI will keep my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort in reviewing our paper. We appreciate your constructive feedback and suggestions.\n\n**W1 (Computational Complexity Analysis): While the paper discusses parameter efficiency of low-rank refinement, it does not thoroughly analyze the computational cost of the iterative update algorithm compared to alternative approaches such as the optimization-based PCP method.**\n\nThe primary computational bottleneck in the proposed algorithm is the SVD computation performed in each iteration (line 171, page 4). For a weight matrix $\\mathbf{W} \\in \\mathbb{R}^{m \\times n}$, the time complexity of SVD is $O(mn^2)$ (assuming $m \\geq n$). With $T$ iterations, the overall time complexity is $O(Tmn^2)$.\n\nIn contrast, the PCP baseline would typically use an iterative optimization method (like Adam) that also requires SVD computations in each iteration to compute the nuclear norm. Furthermore, the PCP baseline requires substantially more iterations ($T=5000$) to achieve comparable results to the proposed method ($T=50$).\n\nTable: Computational Efficiency Comparison\n\n| Method | Time Complexity | Typical Iterations | Relative Computational Cost |\n| ------------------ | --------------- | ------------------ | --------------------------- |\n| Proposed Algorithm | $O(Tmn^2)$ | 50 | 1× |\n| PCP Baseline | $O(Tmn^2)$ | 5000 | ~100× |\n| Zero-shot SVD | $O(mn^2)$ | 1 | ~0.02× |\n\n\n**W2 (Inference Latency and Memory Consumption): The proposed method is theoretically hardware-efficient as it enables structured N:M pruning. However, the inference latency and memory consumption of low-rank refinement is not measured.**\n\nBelow, we provide some missing measurements and analyses. Note that we deliberately employed a conservative evaluation approach to ensure fair comparison with the dense baseline model. We maintained all matrices in their dense representation format, preserving zero elements rather than utilizing sparse matrix formats or specialized hardware acceleration.\n\nTable: Parameter Count and Evaluation Time for Complete ARC-Challenge Benchmark.\n\n| Model Configuration          | Non-Zero Parameter Count | Parameter Count | ARC-C | Relative Time |\n| ---------------------------- | ------------------------ | --------------- | ----- | ------------- |\n| *Llama-2-7B*                 | | | | |\n| Dense baseline               | | | 50.5s | 1.0x          |\n| Magnitude 50% sparse         | 3.5B                     | 6.7B            | 50.7s | ~1.0x         |\n| Proposed Method (k=128, 50%) | 3.8B                     | 7.1B            | 50.5s | ~1.0x         |\n| *Llama-2-13B*                |                          |                 |       |               |\n| Dense baseline               | 13.0B                    | 13.0B           | 73.2s | 1.0×          |\n| Magnitude 2:4 sparse         | 6.7B                     | 13.0B           | 73.9s | ~1.01×        |\n| Proposed Method (k=128, 2:4) | 7.2B                     | 13.5B           | 77.9s | ~1.06×        |\n\n**Q1: The rank k is manually set in the paper, will a poor choice of k lead to unnecessary computational overhead or insufficient recovery of pruned weights?**\n\nYes. The choice of k leads to a trade-off between the performance recovery and the computational cost. The low-rank component adds $k(m+n)$ parameters and $2k(m+n)$ FLOPs. Therefore, an unnecessarily high k value directly increases these costs with diminishing returns. Figure 2(b) in the paper shows the cumulative energy retention for different k values, and the curve begins to flatten as k increases, indicating diminishing returns.\n\n**Q2: If given fixed inputs, does the iterative refinement method always converge to a stable solution?**\n\nYes. As shown in Theorem 4.2, Theorem 4.3 and Corollary 4.4 in the manuscript, the iterative refinement method always converges to a stable solution as $T \\to \\infty$ and the approximation error is bounded by:\n\n$$\n\\left\\\\|W  - \\left(S^{(t)} + L_k^{(t)}\\right)\\right\\\\|_F \\leq \\sqrt{(r-k)} \\sigma\\_{k+1}\\left(L^{(t)}\\right),\n$$\n\nwhere $r$ is the rank of the weight matrix $W$, $\\sigma_{k+1}\\left(L^{(t)}\\right)$ is the $(k+1)$-th largest singular value of $L^{(t)}$.", "claims_and_evidence": "Yes, the claims are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, in this work, the proposed methods and evaluation criteria make sense.", "theoretical_claims": "Yes. I checked the correctness of them and found no issues among them. The iterative refinement process looks right. The theoretical analysis about the convergence property and the error bound is well-established.", "experimental_designs_or_analyses": "Yes, I have reviewed the soundness and validity of the experimental designs and analyses related to large language model pruning. The experiments are carried out on the well-known Llama 7B and 13B models at various levels of sparsity. In addition to evaluating PPL, several standard benchmarks are also assessed. The experiments are comprehensive.", "relation_to_broader_scientific_literature": "The key contributions of the paper on Targeted Low-rank Refinement are closely related to prior findings in pruning, low-rank approximation, and post-pruning performance recovery. \n\nThe study builds on the extensive body of research on pruning techniques, particularly magnitude pruning, which removes low-magnitude weights to reduce model size [1]. It addresses a known limitation of pruning: performance degradation due to the loss of important information, especially at high sparsity levels, as discussed in works such as the Junk DNA Hypothesis[2].\n\nThe idea of using low-rank approximations to restore lost model capacity has been explored in previous research [3], but prior methods often struggle with maintaining the structured sparsity patterns needed for hardware efficiency, which this paper explicitly addresses.\n\n[1] Han, S., et al. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding.\n\n[2] Yin, L., et al. Junk DNA hypothesis: Pruning small pre-trained weights irreversibly and monotonically impairs ”difficult” downstream tasks in llms. ICML 2024.\n\n[3] Zhou, T. and Tao, D. Godec: Randomized low-rank & sparse matrix decomposition in noisy case. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011.", "essential_references_not_discussed": "No. The authors has discussed almost all essential references in this work.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The proposed approach is a data-free and plug-in-and-play method, orthogonal to existing pruning methods.\n\n2. The proposed iterative refinement method addressing a key limitation of existing low-rank refinement techniques, i.e. prior methods often struggle with maintaining the structured sparsity patterns needed for hardware efficiency, which this paper explicitly addresses.\n\n3. Experiments on LLaMa models demonstrate improvements over conventional magnitude pruning and Wanda pruning.\n\nWeaknesses:\n\n1. While the paper discusses parameter efficiency of low-rank refinement, it does not thoroughly analyze the computational cost of the iterative update algorithm compared to alternative approaches such as the optimization-based PCP method.\n\n2. The proposed method is theoretically hardware-efficient as it enables structured N:M pruning. However, the inference latency and memory consumption of low-rank refinement is not measured.", "other_comments_or_suggestions": "In abstract “Nonetheless, these methods often create a gap between the original dense and the pruned sparse model, …”, “create a gap” is slightly awkward, “introduce” is a smoother expression.", "questions_for_authors": "1. The rank k is manually set in the paper, will a poor choice of k lead to unnecessary computational overhead or insufficient recovery of pruned weights?\n\n2. If given fixed inputs, does the iterative refinement method always converge to a stable solution?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aHO7fmhCsw", "reviewer": "Reviewer_XcmX", "summary": "The paper introduces a novel method to improve the performance of pruned large language models (LLMs) by combining sparsity with a low-rank approximation. The authors propose an iterative refinement algorithm that updates the sparse weight matrix while incorporating a low-rank component to approximate the difference between the original dense matrix and the pruned sparse matrix. This approach aims to recover information lost during pruning without requiring extensive retraining or large datasets, maintaining the sparsity pattern for hardware efficiency. Key contributions include:\n\n1. An iterative weight update method (Algorithm 1) that refines the sparse matrix and adds a low-rank patch, progressively increasing the rank from 2 to a target \\( k \\) over \\( T \\) iterations, preserving the sparsity pattern using a binary mask.\n2. The method reduces perplexity compared to baseline pruning techniques (e.g., magnitude pruning and Wanda) across various sparsity levels.\n3. It also achieves competitive performance on benchmark datasets like TruthfulQA, GSM8K, ARC-c, and MMLU w.r.t. Magnitude and Zero-shot SVD\n4. The paper provides a theoretical analysis proving sparsity preservation (Theorem 4.1), convergence to a solution (Theorem 4.2), and monotonic error reduction after a certain iteration (Theorem 4.3).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We authors greatly thank the reviewer for constructive comments on this work. We would like to clarify the following points:\n\n**W1: Evaluation Scope: Lacks zero-shot and few-shot task evaluations (cf. Wanda’s Tables 2, 21), limiting practical relevance.**  \n**Q3: Zero-shot Task Evaluation: Why were zero-shot accuracies (e.g., as in Wanda’s Table 2, also Tables in the appendix e.g., Table 23) not included? Adding these could align your evaluation with Wanda’s, enhancing practical relevance. A justification or additional results would influence my view on the method’s applicability.**\n\nBelow, we provide additional results on more benchmarks.\n\nTable: Performance of Llama-7B on three additional benchmarks, along with the results from Table 3.\n\n| Model                         | *HellaSwag* | *WinoGrande* | *ARC-e*  | TruthfulQA | GSM8K   | ARC-c    | MMLU     |\n| ----------------------------- | ----------- | ------------ | -------- | ---------- | ------- | -------- | -------- |\n| Dense baseline                | 76.2        | 70.0         | 72.8     | 34.1       | 10.3    | 44.7     | 32.1     |\n| | | | | | | | |\n| Magnitude 50%                 | 60.9        | 59.3         | 54.3     | **35.3**   | 1.0     | 33.5     | 24.6     |\n| Magnitude 50% + Zero-shot SVD | 69.2        | **65.5**     | 63.6     | 34.3       | 1.5     | 36.9     | **26.0** |\n| **Magnitude 50% + Ours**      | **69.8**    | 65.3         | **64.3** | 34.2       | **3.4** | **41.5** | **26.0** |\n\n\n**W2: Missing details on calibration data and iteration specifics (e.g., ( T ) selection).**\n\n1. The proposed iterative refinement method is entirely data-free and does not require calibration data. \nAs shown in Algorithm 1, the only inputs are:\n\n- Dense weight matrix $W$\n- Binary mask $P$ (from pruning)\n- Target rank $k$\n- Number of iterations $T$\n\nWe use WikiText-2 for perplexity evaluation and 'allenai/c4' as the calibration data for Wanda pruning as well as Wanda + Ours.\n\n2. We consistently uses T=50 across experiments, which is sufficient for achieving most of the potential error reduction while maintaining computational efficiency. Since the overall time complexity is $O(T \\cdot min(mn^2, m^2n))$, where $m$, $n$ are the number of rows and columns of $W$.\n\n**W3: Omits SparseGPT and fine-tuning baselines, reducing comparative strength against Wanda.**.  \n**Q4: SparseGPT Comparison: Why was SparseGPT excluded from experimental comparisons despite its relevance (cf. Wanda’s Table 3)? Including it could better position your method; its absence weakens the competitive analysis.**\n\nThank you for the suggestion.\nWe provide the results of SparseGPT 2:4 and SparseGPT 2:4 + Ours in the following table.\n\nTable: Performance of SparseGPT 2:4 and SparseGPT 2:4 + Ours using Llama-7B on three additional benchmarks, along with the results from Table 3.\n\n| Method               | *HellaSwag* | *WinoGrande* | *ARC-e*  | TruthfulQA | GSM8K   | ARC-c    | MMLU     |\n| -------------------- | ----------- | ------------ | -------- | ---------- | ------- | -------- | -------- |\n| SparseGPT 2:4        | 58.6        | 63.9         | 56.6     | **36.5**   | 2.0     | 33.1     | 25.4     |\n| SparseGPT 2:4 + Ours | **65.1**    | **66.9**     | **59.9** | 33.8       | **2.7** | **36.1** | **29.1** |\n\n\n**Q1: Calibration Data Details: What calibration data (size, source) was used for perplexity experiments? Wanda specifies 128 C4 sequences; this omission affects reproducibility. A response detailing this would strengthen confidence in the results.**\n\nThank you for your suggestion, we will add these details in revised version.\nWe also use 128 C4 sequences as the calibration data for Wanda pruning as well as Wanda + Ours. For perplexity evaluation, we use 128 sequences from WikiText-2 dataset.\n\n**Q2: Performance Target for Parameter Reduction: The abstract claims an 8.6% parameter reduction for a specific target at 50% sparsity. What is this target, and can you provide supporting data? Without this, the claim feels unsubstantiated.**\n\nThe performance target is the perplexity metric on WikiText-2 dataset, we will revise the abstract to clarify this. We provide the detailed results in Figure 3(a).", "claims_and_evidence": "- **Claim 1**: The method bridges the gap between dense and sparse models using a low-rank component.  \n  - **Assessment**: Well-supported\n\n- **Claim 2**: The iterative algorithm enhances precision by prioritizing larger singular values.  \n  - **Assessment**: Well-supported, though the empirical comparison with PCP could be expanded to quantify precision gains more explicitly.\n\n- **Claim 3**: Significant perplexity improvements, especially at high sparsity levels (e.g., 99.6% reduction at 70% sparsity).  \n  - **Assessment**: There is evidence though the lack of zero-shot task performance (e.g., accuracy on downstream tasks) limits the scope of evaluation compared to Wanda’s broader benchmarks (e.g., Table 2 in Wanda).\n\n- **Claim 4**: The method enables a reduction in model parameters while maintaining 50% sparsity and meeting a specific performance target.\n  - **Issue**: Lacks details and remains vague. The paper does not specify the performance target or provide a direct comparison showing parameter reduction versus performance trade-offs.", "methods_and_evaluation_criteria": "- Methodology yes.\n- Evaluation criteria is appropriate for language modeling (perplexity) and generalizability (benchmarks). However, unlike Wanda, which includes zero-shot task accuracies (e.g., Table 23) and few-shot results (e.g., MMLU in Table 21), this paper lacks zero-shot accuracy metrics (and few-shots on MMLU), limiting its comparability on downstream tasks. Adding such evaluations would strengthen the assessment of practical utility.", "theoretical_claims": "The proofs generally seem to be correct but rely on assumptions about singular value decay that could be sensitive to matrix properties.", "experimental_designs_or_analyses": "- **Perplexity Comparison (Tables 1, 2)**: Tests LLaMa-7B and LLaMa-13B across sparsity levels with \\( k=128 \\).\n  - **Soundness**: The design is valid, using WikiText-2 perplexity as a standard metric, consistent with Wanda and SparseGPT. Results are reproducible with a fixed \\( k \\) and \\( T=50 \\).\n  - **Issues**: The paper lacks details on calibration data (e.g., size, source), unlike Wanda (128 sequences from C4). This affects reproducibility. Additionally, only magnitude pruning and Wanda are baselines, omitting SparseGPT (a key competitor in Wanda’s Table 3).\n\n- **Benchmark Evaluation (Table 3)**: Assesses performance on four datasets.\n  - **Soundness**: The choice of datasets is reasonable for LLM evaluation, and comparisons with dense and sparse models are fair.\n  - **Issues**: Sample sizes and statistical significance (e.g., variance across runs) are not reported, unlike Wanda’s robustness analysis (Table 18). This reduces confidence in the results’ stability.\n\n- **Iterative Analysis (Figures 4, 5)**: Visualizes singular value spectra, energy retention, and convergence.\n  - **Soundness**: The synthetic and real-world (LLaMa-7B) analyses are well-designed to support theoretical claims.\n  - **Issues**: Limited to one sparsity level (50%) in Figure 4, missing higher sparsity cases (e.g., 70%) where Wanda shows larger gaps \n\nThe experiments are sound but incomplete compared to Wanda, which includes zero-shot accuracies, few-shot tasks, and robustness analysis, highlighting gaps in broader evaluation.", "relation_to_broader_scientific_literature": "- **LLM Pruning**: Extends magnitude pruning (Han et al., 2015), SparseGPT (Frantar & Alistarh, 2023), and Wanda (Sun et al., 2023) by adding low-rank refinement, addressing the performance gap noted in the Junk DNA Hypothesis (Yin et al., 2024).\n- **Low-rank Approximation**: Leverages SVD and iterative refinement, drawing from matrix completion (Chandrasekaran et al., 2011) and robust PCA (Candès et al., 2011), adapting them for sparsity preservation.", "essential_references_not_discussed": "- **SparseGPT (Frantar & Alistarh, 2023)**: Cited but not experimentally compared in Tables 1–3, despite being a key baseline in Wanda (Tables 2, 3). Its inclusion would contextualize the method’s superiority over second-order pruning approaches.\n- **LoRA (Hu et al., 2021)**: Wanda uses LoRA for fine-tuning (Section 5), showing performance recovery. The absence of fine-tuning comparisons here misses a practical recovery baseline.\n- **Yin et al. (2024) - Junk DNA Hypothesis**: Cited, but its implications (irreversible loss at high sparsity) could be explored more deeply with zero-shot task evaluations, as Wanda does.\n\nThese omissions limit the paper’s positioning against state-of-the-art recovery and pruning methods.", "other_strengths_and_weaknesses": "- **Strengths**:\n  - **Originality**: Creative integration of sparsity and low-rank refinement, distinct from Wanda’s metric-based pruning.\n  - **Significance**: Addresses high-sparsity performance degradation, a critical issue for LLM deployment.\n  - **Clarity**: Well-written with clear figures (e.g., Figure 1) and algorithmic exposition.\n\n- **Weaknesses**:\n  - **Evaluation Scope**: Lacks zero-shot and few-shot task evaluations (cf. Wanda’s Tables 2, 21), limiting practical relevance.\n  - **Reproducibility**: Missing details on calibration data and iteration specifics (e.g., \\( T \\) selection).\n  - **Comparison Depth**: Omits SparseGPT and fine-tuning baselines, reducing comparative strength against Wanda.", "other_comments_or_suggestions": "- **Typos**: \n  - Page 7, Table 2: \"Dense\" perplexity should be 5.09 (per Wanda), not 4.57.\n- **Suggestion**: Include a runtime comparison (e.g., vs. Wanda’s Table 4) to quantify “computationally efficient.”", "questions_for_authors": "1. **Calibration Data Details**: What calibration data (size, source) was used for perplexity experiments? Wanda specifies 128 C4 sequences; this omission affects reproducibility. A response detailing this would strengthen confidence in the results.\n2. **Performance Target for Parameter Reduction**: The abstract claims an 8.6% parameter reduction for a specific target at 50% sparsity. What is this target, and can you provide supporting data? Without this, the claim feels unsubstantiated.\n3. **Zero-shot Task Evaluation**: Why were zero-shot accuracies (e.g., as in Wanda’s Table 2, also Tables in the appendix e.g., Table 23) not included? Adding these could align your evaluation with Wanda’s, enhancing practical relevance. A justification or additional results would influence my view on the method’s applicability.\n4. **SparseGPT Comparison**: Why was SparseGPT excluded from experimental comparisons despite its relevance (cf. Wanda’s Table 3)? Including it could better position your method; its absence weakens the competitive analysis.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper gives a new way to improve/finetune a pruned model, using low rank matrices during the re-training. The paper is well written and the paper has fairly extensive experiments, along with some theoretical convergence analyses. Reviewers are all fairly positive about the paper, and the authors are encouraged to incorporate comments from the review and rebuttal into the final version.
Decision: Accept (poster)