Compress Large Language Models via  Collaboration Between Learning and Matrix Approximation
Authors: Yuesen Liao
Zhiwei Li
Binrui Wu
Zihao Cheng
Su Zhao
Shuai Chen
WEIZHONG ZHANG
Venue: NeurIPS 2025 poster ()
Keywords: Model Compression
Large Language Models
Primary area: deep_learning
TL;DR: We propose a bilevel framework that jointly learns layer-wise sparsity and low-rank structure via truncated Gaussian sampling and efficient matrix approximation, achieving better performance for compressing LLMs.

Abstract: Sparse and low-rank matrix composite approximation has emerged as a promising paradigm for compressing large language models (LLMs), offering a more flexible pruning structure than conventional methods based solely on sparse matrices. The significant variation in weight redundancy across layers, along with the differing rank and sparsity structures of weight matrices, makes identifying the globally optimal pruning structure extremely challenging. Existing methods often depend on uniform or manually designed heuristic rules to allocate weight sparsity across layers, subsequently compressing each matrix using matrix approximation techniques. Given the above theoretical difficulty in global compression of LLMs and the limited computational and data resources available compared to the training phase, we argue that a collaboration between learning and matrix approximation is essential for effective compression. In this paper, we propose a novel LLM compression framework based on generalized bilevel optimization that naturally formulates an effective collaborative mechanism. Specifically, the outer loop frames the weight allocation task as a probabilistic optimization problem, enabling the automatic learning of both layer-wise sparsities and matrix-wise retained ranks, while the inner loop solves the corresponding sparsity and rank-constrained model compression problem via matrix approximation. Our main technical contributions include two key innovations for efficiently solving this bilevel optimization problem. First,  we  introduce a truncated Gaussian prior-based probabilistic parameterization integrated with a policy gradient estimator, which avoids expensive backpropagation and stabilizes the optimization process. Second, we design an adapted QR-based matrix approximation algorithm that significantly accelerates inner loop computations. Extensive experiments on Phi-3 and the LLama-2/3 family demonstrate the effectiveness of our method. Notably, it maintains over 95\% zero-shot accuracy under 50\% sparsity and achieves up to 2× inference speedup.

Compress Large Language Models via Collaboration Between Learning and Matrix Approximation
Yuesen Liao'; Zhiwei Li Binrui Zihao Zihoo Cheng' Su Zhao?, Shuai Chen?, Weizhong Zhang!
Fuudan University, "Meituan Inc.
Shanghai Key Laboratory of Intelligent Information Processing {ysliao24, zw1i23, brwu23, zhcheng25] fudan. edu, (zhaosu04, chenshuai31) tuan com weizhongzhang@fudan edu.
Abstract
Sparse and low-rank matrix composite approximation has emerged promising paradigm for compressing large language models (LLMs), offering more flexible pruning structure than conventional methods based solely sparse matrices. The significant variation in weight redundancy across layers, along with the differing rank and sparsity structures of weight matrices, makes identifying the globally op- timal pruning structure extremely challenging. Existing methods often depend on uniform manually designed heuristic rules to allocate weight sparsity across lay- ers, subsequently compressing each matrix using matrix approximation techniques. Given the above theoretical difficulty in global compression of LLMs and the lim- ited computational and data resources available compared to the training phase, we argue that collaboration between learning and matrix approximation is essential for effective compression. In this paper, we propose novel LLM compression framework based on generalized bilevel optimization that naturally formulates an effective collaborative mechanism. Specifically, the outer loop frames the weight allocation task a probabilistic optimization problem, enabling the automatic learning of both layer-wise sparsities and matrix-wise retained ranks, while the inner loop solves the corresponding sparsity and constrannd model compres- sion problem via matrix approximation. Our main technical contributions include two key innovations for efficiently solving this bilevel optimization problem. First, we introduce a truncated Gaussian prior-based probabilistic parameterization inte- grated with a policy gradient estimator, which avoids expensive backpropagation and stabilizes the optimization process. Second, we design adapted QR-based matrix approximation algorithm that significantly accelerates inner loop computa- tions. Extensive experiments Phi-3 and the LLama-2/3 family demonstrate the effectiveness of our method. Notably, maintains over 95% zero-shot accuracy under 50% sparsity and achieves up to 2x inference speedup.
Introduction
Model compression g
29 31 is widely adopted paradigm for improving the inference effi-
ciency of large language models (LLMs). Its core principle is to reduce model size by removing redundant parameters or approximating the model weights with low-rank matrices [37] while pre- serving the performance. Although promising results have been repeatedly reported in the literature,
"Equal Contribution.
t Corresponning Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

outer loop
Sampling
c (H))))e))
Policy Gradient
a-
W
K..M
Wsa x)
Waree
War
Le
X6
01
Wid
W.,
O
L
Qei based
W1a
W1a
W120
inner loop
8: L-yoyr Sparsity
K: Matrix Rank Ratio
Layer Sequence
Trainable
(X,Y):
Training Data
O
:Ouput
C Loss
Figure 1: Process diagram of our bilevel framework. sample and K from No, (Mt. o² assign sparsity allocation for each layer; @: compress matrix Wi, E W via adapted QR- RPCA under the sparsity allocation s; and Ki,j @: forwardpass the compressed model W(s, k) and get the loss; O update the distribution Nr, (Ja, a² I) based policy gradient estimator.
challenges remain due to the complex model architecture, vast optimization space, and limited data and computational resources compared to those available during the training stage.
In this paper, we focus on the emerging compression paradigm based on sparse and low-rank referred as robust principal component analysis (RPCA) [4] in the field of classical matrix analysis, which adopts a more flexible pruning structure than conventional methods based solely sparse matrices. Existing methods [17 44 typically adopt a uniform sparsity allocation over layers, i.c., setting an equal pruning proportion for each layer and subsequently compressing each matrix using matrix approximation techniques. Recognizing the heterogeneous redundancy across layers, recent works 18 19 40 41] have introduced manually designed heuristic rules to allocate varying sparsity levels different layers.
matrix composite approximation 17
20 44
However, the performance of these methods is often less effective than expected. The main reason is the significant variation in weight redundancy across layers, along with differing rank and sparsity structures of weight matrices. These factors make finding the globally optimal pruning structure ex- tremely challenging. This highlights the need for layer-wise sparsity and matrix-wise rank allocation in EPCA-baaed compression methods. Given the theoretical difficulty of global compression for LLMs and the limited computational resources and data compared the training phase, we argue that collaboration between learning and matrix approximation is essential for effective compression.
In this paper, we propose novel bilevel optimization framework [15] 30 that naturally formulates an effective collaborative mechanism. In line with recent perspectives 19 40 41,, we adopt the view matrix approximation problem. Instead of metric-based heuristics [9 41, we model the weight allocation task of outer loop probabilistic optimization problem, enabling the automatic learning of both layer-wise sparsities and matrix-wise retained ranks, while the inner loop solves the corresponding RPCA subproblem obtain the sparse and low-rank decomposition under the current allocation scheme. The bilevel framework poses difficulties due to the implicit differentiation through the inner loop solutions and the substantial computational overhead of the inner RPCA problem. To address these challenges, we introduce the following two key technical innovations. First, for the outer loop, we use truncated Gaussian prior enable continuous probabilistic modeling within bounded support. The truncation helps stabilize training by preventing gradient explosion in low- density regions. Through this reparameterization, we apply policy gradient [32] to update the prior parameters without backpropagating through the compressed model, reducing memory overhead.
that once a global sparsity allocation is provided, the compression task can be reduced
Second, instead of costly SVD-based solvers [1 49, we use an adapted QR-based matrix fitting scheme [22. which significantly accelerates inner loop computations. Our method is intuitively visualized in Figure Empirical results on the Phi-3 and Llama family model show that our method consistently learns better compression configurations and achieves superior performance under various sparsities. For the LLama2-13B model, our method preserves over 95% MMLU accuracy under 50% sparsity setting, with practical speedup of 2x.
Our main contributions are summarized as follows:
We propose a bilevel optimization framework that enables effective collaboration between learning and matrix approximation for LLM compression.

---

We introduce two main technical contributions: policy gradient method based on truncated Gaussian modeling, and QR-based RPCA algorithm for efficient matrix upproximation.
Extensive experiments demonstrate that our method consistently outperforms existing model compression methods, even under high prune rates.
Related Works
LLM Pruning.
LLM pruning methods can be broadly categorized into structured [12
13 22 39
34 38 45 46] approaches. Structured pruning removes entire components (e.g., layers, neurons, channels), with methods like LLM-Pruner [22] using gradient-based importance scores. Unstructured pruning. such as Wanda [31] and SparseGPT [8) prunes individual weights and can remove up 30% with little accuracy drop. Wanda uses activation
and unstructured 8
scoring, while SparseGPT estimates Hessians for efficient weight reconstruction. However, both approaches face trade-offs between speedup and performance. Hybrid methods that combine sparsity and low-rank decomposition can better balance these aspects by integrating the strengths of both.
Sparsity and Low Rank. Compression methods combining sparsity and low-rank decomposition are increasingly used for LLM compression. LoRAP [17] applies low-rank approximation to Attention matrices and enforces sparsity on MLP blocks, reflecting their distinct structures. LoSparse [20] decomposes weight matrices into low-rank factors U. and sparse component S, updating all parts while pruning S to meet sparsity budget. OATS [4 and HASSLE-free (23) alternate between
low-rank approximation and sparsification using fixed sparsity allocations. We focus on this class of RPCA- compression methods
base framework to achieve stronger performance.
Sparsity Allocation. Many LLM pruning methods minimize layer-wise reconstruction loss ||W;X; - Willl and assume uniform sparsity across layers, often yielding suboptimal re- Several recent methods explore sparsity allocation strategies 26 40). FLAP [2 allocates sparsity based on fluctuation scores, OWL [41 leverages activation outliers, DSA [88] searches for optimal allocation functions, and ALS [19 formulates the problem linear programming. However, these methods depend on fixed validation sets and lack joint optimization with training. In contrast, formulate sparsity allocation a learnable optimization problem driven by training data, while treating pruning as a matrix approximation task solvable by existing frameworks.
Preliminary
RPCA Framework for Matrix Approximation. As adopt RPCA 44 base solver in proposed compression method, we present its basics as follows. Given a weight matrix W E Rm**, xn the target sparsity K and rank RPCA approximates W the sum of low-rank matrix L and sparse matrix by solving the following optimization problem:
min L
W - S
s.t. rank(L) <, S S K.
Problem I is usually solved via alternating optimization, with the following update rules:
L = "IuncatedSVD(W - S r),
(2)
S -L+1),,
where TruncatedSVD(W r) denotes the rank-r approximation matrix W obtained by retaining only the top-r singular values and their corresponding singular vectors. The operator Pa, (.)
denotes the projection into the feasible set of the sparse matrix S, i.e., (S || < K, E Rmx*). Typically, this projection enforces a sparsity constraint by retaining only the top- largest- magnitude entries and setting the rest to zero. Following Wanda and OATS, apply diagonal scaling matrix D v diag( X X) I "xx the weight matrix and perform approximation WD, where deentes the input activation. This is used by default unless otherwise specified.
Discussion. It is important to note that the above solver is not efficient enough due to the expensive SVD process and cannot directly adapted develop our compression method. Therefore, this paper, introduce efficient RPCA algorithm based QR decomposition, detailed in Section 4.3
3

---

4 Method
In this section, we present the details of our method. Section 4.1
introduces the overall design of the bilevel optimization framework. Section 4.2 describes our first technical contribution: we propose truncated Gaussian prior and integrate it with policy gradient estimator to stabilize the training process and avoid expensive implict differentiation. Section 4.3 presents our second technical contribution: an efficient RPCA algorithm adapted from QR decomposition.
4.1 Bilevel Framework
We introduce our bilevel optimization framework for LLM compression, which formulates effective collaborative mechanism between learning and matrix approximation. The inner loop performs model compression by solving RPCA problem under specific sparsity allocation scheme given by the outer loop. The outer loop formulates the learning problem of sparsity allocation into a probabilistic optimization task, enabling the automatic learning of both layer-wise sparsities and matrix- retained ranks based on the model compressed by the inner loop. The workflow is shown in Figure
Inner Loop. We first describe how given allocation scheme determines the sparsity structure of each matrix the model. To capture differences parameter redundancy across layers, we allocate a sparsity ratio s; to each layer, indicating the proportion of parameters to be pruned. For all matrices (W.,)) M in layer i that lie in R xn the total number of retained parameters after compression is mn(11 st). We let Ki, be the proportion of parameters allocated W., for the low rank matrix Lij, i.e., we assign mn(1 - 8; 8i),i,j parameters to Li.j- This yields the target rank and sparsity:
f==1 1=1
mn st)
TJJ
KiJ = mn((1 s;) - ks.)).
m + n
We group all s; and K, into two vectors (s, k)) and compress each matrix accordingly by solving a series of RPCA problems described in Section 3 That is, we can obtain a set of RPCA problems
presented in the definition below.
Definition 1. The RPCA decomposition of the full parameter set W & (Wiylé E [1,4,, [1, under sparsity allocation scheme (s, k) is denoted as
Wis semem
RPCA(W, s.
W arg min I ku.
[ Lijj S
where each matrix Wi is decomposed into low-rank component Ligi and a sparse component Sij with target rank Vs.j and sparsity budget Kij computed from from Eqn. (3).
Outer Loop. In the outer loop, we begin by modeling the allocation scheme (s, k) using a suitable probabilistic distribution p(-10) parameterized by 0. A sparsity allocation is sampled from this distribution and passed into the inner loop to generate a compressed model W(s, k). The performance of the resulting model is then evaluated using a loss function. The overall objective is to minimize the expected loss over sampled allocation schemes. To this end, we optimize the probability parameters via gradient-based methods, enabling the framework to adaptively explore and refine sparsity patterns that lead to improved model performance.
Therefore, the overall bilevel optimization framework can be formulated as follows:
N
((s,))p 0) ((W(s))) = Σ effe, W(s, v:),
min dec
N
s.t. W(s, K) RPCA(W,, s,
(4)
where C is the feasible region for 0 to control the sparsity, which will be specified in Section 4.2 ((me, y: ) represents the training dataset, f(- W(s, x)) the compressed model under allocation scheme (s, ro), and f(.,-) denotes the loss function.

---

Challenges. This bilevel optimization framework presents two main challenges. First, the outer objective hard optimize due the implicit differentiation through the inner loop, requiring appropriate gradient estimators and a well-designed probabilisstcc model p(. 0). Second, repeatedly solving RPCA problems in the inner loop computationally expensive. In the following sections, we introduce our proposed techniques to address these challenges.
4.2 Outer Optimization
Policy Gradient. To address the difficulty of computing gradients with respect to the parameter 0. we adopt a policy gradient estimator. This approach avoids implicit differentiation by directly computing gradients based on the loss function. The derivation is as follows:
Ve (p() [C(V(s, x))] Ep(x.<(0) [C(W(s, K)) Ve logop(s,x<00)
(5)
In practice, we sample a mini-batch B, evaluate the loss of the compressed model under each sampled allocation (s, k),, and compute the policy gradient of the parameters & as: Ge = (M(, k)) Ve p(s, , k 0). This yields an unbiased estimator; the proof is provided in Appendix D.1
Remark 1. Policy gradient methods are known to suffer from high variance due to the stochastic nature of the sampling process, which can lead to instability during training. To mitigate this issue, we subtract a control variate Lg(W(s', K²)) Vo log p(ss, k:(0), which has zero mean but is highly correlated with the original gradient. Here, (s', ,,!') is independent sample drawn from the same distribution as (s, k). This auriancerreductinn technique leads to the final gradient estimator:
or 9o (Es(WW(s, - Lsssssss' Vo log p(s.p(ss,) k/0))
Truncated Gaussian Distribution. Computing policy gradients requires Ve log p(s, k(0). Gaus- sian distributions are often used for convenience, but their support unbounded, conflicting with the constraints (e.g., sparsity ratio in 1). Moreover, shrinking variance for convergence can cause gradient explosion. To address these issues, we employ a truncated Gaussian distribution (1, o²) for probabilistic modeling, which restricts the Gaussian distribution ( o²) to the interval [/u 14 7]. This truncation limits the sampling range and provides bounded support, thereby stabilizing training and facilitating policy gradient computation. For a random variable x Nr, (u, o²), its probability density function (PDF) is given by:
{ 0,
for x E [M [1-09 y, 7].
+ (3)
+ (=2)²
otherwise,
where á and & denote the PDF and CDF of the standard Gaussian distribution NO. 1), respectively. The detailed sampling method for the truncated Gaussian is described in Appendix C2
Remark 2. For the p(æ; p., o², 7). its parameter vector is [u, o², 7 To avoid the gradient explosion during training, we fix the variance o². In addtion, to control the range of x and ensure convergence, manually reduce according to the annealing schedule [50). More details are provided in (..2 Therefore, the only trainable parameter of p(-) is μ. i.e., O = μ. For simplicity, we do not distinguish between μ and 0 in the remainder of this section.
Appendix A.2.
Each element in gradient V log p(s, K kl o2,7) can be computed using the lemma below.
Lemma 1. Let a random variable x follow the truncated Gaussian distribution æ ~ (N, The gradient of the log- density with respect to the mean parameter is given by:
Vp logplo (n) V, log {
+ (2) - (=)))))) Vollo (00=))(((0))))
t ( (=
p
(7)
0²
Combining Eqn. (6 and Eqn. we compute the gradient g with μ = (1ua, Mx)). Recall that μ is the mean of (8, k), its feasible region can be defined as C (#: || l1 2 pL. [0. 1]L+LM] Since C can be rewriten as C {M 1 M4 2 pL, μ [0, 1/ which is convex, we can project a onto C using projection (-) after gradient descent. See Appendix C.1 for details.
5

---

Algorithm Bilevel Optimization Framework
Input: Model weights W, over all prune rate p, rank ratio Ko, parameter o² and 7, learning rate "/
1: Initialize parameters ja (1s, M4x 13
2: for each iteration do
3: Sample a mini batch B
4: Reduce 7 according annealing schedule
5: Sample (s(9) re()) from p(s, μr, 0², ,7), 1.2
6: Apply RPCA, i.e., Algorithm 2 to obtain the compressed weights W(s(0, ,x((), 1,2
7: Compute
)), i= 1,2, and the gradient:
Las (M x2 x((1))] V logp(s()) 1, pu, p1,
8: Update: μa proje "1947)
9: end for Compressed weights W(ts (r)
Output:
4.3 Inner Optimization
QR-based RPCA algorithm. Conventional RPCA algorithms repeatedly perform costly SVD computations, resulting in high computational overhead. In the inner loop, follow and adapt the method proposed [42), replacing SVD with a more efficient QR-based algorithm. Specifically, noting that the low-rank matrix L in Problem [ can be factorized the product of two matrices U E mgmxx x and V E Rx where is the target rank, we obtain the following reformulation:
min U.V.S
W UV S|? s.l. rank(UV) Sr, ||||| S K
This problem can be solved by alternating minimization over U, V, and S, yielding the update rules:
U = -(W - VV.VV (V.V)),
Ve+1 (U,_UU++) U U ( - St),
= P.. -, V4+1).
The optimization objective only depends on the product UV, rather than the specific factorization. Therefore, we aim find any pair (U', V') such that U'V' = UV. This insight allows us to reinterpret the optimization as a projection problem.
U
c- Ve-i
U.-i (UL, U.+1) UT4 S.) Ilcr....) (W- St),
(10)
where (T(U))2)) denotes the orthogonal projection onto the column space of U++1-
Since (V.VT) is full-rank, the column space of U.+1 is equivalent to that of (W - -S Thus, perform QR decomposition this matrix:
(W (W-S))V; V. Q.R,,
(11)
where Q₁ E R is orthonormal and spans the column space of U,+1, leading to the expression:
U44 = Q.Q. Q.Q. (W)))).
(12)
and we accordingly set the update rules as:
U Qu, V Q7 (W - -S.).
(13)
We present the RPCA algorithm based on QR decomposition in Appendix C Algorithm
2
Remark 3. The QR-based method reduces the per-iteration complexity to ( much lower than the O(mn min(m, n)) cost of SVD when & min(m, n)), while maintaining approximation quality. Remark 4. Rather than reinitializing the inner RPCA subroutine from scratch at each iteration, we warm-start the optimization using the previous solution and adjust the low-rank factors U, V incrementally according to the rank update Ar, thereby improving efficiency and stability.

---

p- 0.50, 5e
:p 14.3
Probability Deneity
8-
p y=
ta:PPL - 16.3
μ= 0.57, v=
6-
:PPPL 17.2
4-
Taget Sionatce
(b) Complexity of QR and SVD.
2-
Decomposttion Type Time
8.30
0.35
0.40
0.45
0.50
0.55
0.60
0.65
0.70
p(x; μ. o, y)
(a) Trajectory of the 25-th layer's sparsity during optimization.
(c) Time cost QR and SVD,
Figure 2: (a) the variation of the truncated Gaussian distribution as training progresses reflects how the structure gradually learns to approach the optimum. (b) due to r < min (m, n),, QR has lower complexity. (c) QR decomposition provides significant acceleration practical implementation.
By integrating the truncated Gaussion prior with policy gradient estimator and the QR-based RPCA solver, we obtain the complete bilevel optimization framework presented in Algorithm [ For the last step, since the range of (s, k)) is vanished, we compress the model under (Ma, (xx) without sampling. present the convergence analysis the bilevel optimization framework in Appendix [D.4
Experiments
In Section 5.1
we introduce the overall experimental setup and the baselines used for comparison.
Section 5.2 presents the pruning performance across multiple LLMs. In Section 5.3 we explores the effectiveness of method under high sparsity settings. Finally, Section 5.4 conducts ablation studies to validate the effectiveness of each component in our framework.
5.1 Experimental Setups
Models. For our main experiments, we select representative models from two prominent open-source architectures: the Phi family (specifically Phi-3-mini ( and the Llama family (including Llama2-78
Llama2-13B 33 and Llama3-8B (7).
Baseline. We select SOTA compression methods baselines, including unstructured pruning methods such as SparseGPT [8] and Wanda [31] as well as RPCA based approaches like OATS and QR, where QR refers using only the inner QR-based algorithm in Section 4.3 without the bilevel optimization framework. [44]. Our experiments compare these compression methods at low prune rates (5 50%) and validate the effectiveness of sparsity allocation at higher prune rates (2 60%).
Configurations. We use C4 [27 as the training dataset, with batch size set to 32, and length set to 256. In addition, the inner-level optimization employs the fixed 32 samples as the calibration dataset. Gamma is set from 0.05 to 0.005. In training, we use the Adam optimizer [16] and set the learning rate 1e-2. The experiments are all completed with one single 80GB NVIDIA A100.
Evaluation. We use L.Mevaluation--arit main benchmarks include: 1) WikiText2 [24 perplexity, 2) zero-shot tasks (including PIQA (3). HellaSwag [43), Winogrande (28). OoonBookQA (25), RTE [35], BoolQ [5], ARC-e and ARC-c (6), and 3) few shot tasks, like MMLU [14. In addition, we test the CPU inference speedup of the pruned model on Intel(R) Xeon(R) Platinum 8369B CPU 2. 90GHz with 32 cpu cores.
[10] evaluate the performance after pruning. The
5.2 Comparison of Compression Methods
Table presents the main results, comparing the performance of different models using various compression methods across multiple prune rates. Our approach achieves top performance across all three task types. Notably, on the WikiText2 benchmark, it reduces Phi-3-mini's perplexity by about 5% over the SOTA OATS 50% sparsity. Results on MMLU and zero-shot accuracy further show that RPCA-based methods (OATS, QR. and ours) outperform methods relying solely on sparse matrix
The initialization details of ua and Me are provided in Appendix A.2 A.2
7

---

Table Performance comparison across various methods and models with different prune rates, The best performance for each prune rate is in bold.
Prune
Method
Phi-3-mini
Ilans2-78
Liams2-33
Lland--B
WikTTest2
MMLLU
sero-hat
WikFTes22
MMLUU
versedut
Wikifeet2
MMLLU
zeroedot
WikiTeet2
MMLUU
suroodhu
0%
Dene
9.50
70.34
71.99
8.79
50.12
66.27
7.91
$6.41
56.72
10.18
64.97
60.71
SprnsCiP
10.27
71.45
9.05
8.13
53.97
68.76
69.34
10:54
68.35
7106
0.10
10002
05.09
8.13
53.59
08.
8,00
61:36
71U1
Ouss
9.98
09.60
71.91
8.83
50.10
06.19
8.03
56.13
08:74
3.05
65.37
00.82
SpareCPP
13.00
63.47
69.15
0.94
45.53
66.13
8.83
$4.48
58.35
10.01
10.91
67.58
Wonda
12
04.15
08.30
0.85
8.73
53.05
08 15
0.76
67
11:53
47.21
65.63
58.16
SpanseCPP
16.00
53.32
06:36
11.06
41.95
02.09
10.21
68.91
06
11:95
53.00
66
Wiasda
17:23
$4.57
65:03
11.43
37.16
62.53
10.05
49.59
66 25
12.36
49.53
63.27
30%
CXTS
15.18
59,99
68.41
10.57
44.7
63.49
9.49
52.44
67.77
10.87
56.46
66.71
QRi
58-16
10.96
65.09
51-11
14.87
00.57
10.49
46.10
53.79
$6.97
compression. Our method further excels by adaptively allocating sparsity. Interestingly, Llama3-88 even surpasses its unpruned version at low prune rates, suggesting that pruning redundant weights may be beneficial. Moreover, the -R-bd method with uniform allocation achieves performance close to OATS while requiring only 1/20 of its runtime, highlighting both efficiency and effectiveness, thereby offering promising direction for scalable LLM compression.
5.3 Comparison with OWL Sparsity Allocation
We further investigate the effectiveness of different sparsity allocation strategies under high prune rates. To ensure a fair comparison, we adopt the QR decomposition introduced in Section 4.3 as the sole compression algorithm. Table[[compares the results of various sparsity allocation strategies, where uniform denotes applying the same prune rates to all layers, and OWL leverages outlier information allocate sparsity. It can be observed that our method leads better performance. providing valuable insights for future research on high sparsity compression.
Table 2: Comparison of sparsity allocation methods under high sparsity.
Phi-3-mini
Lamma2-7B
Llama3-BB
Prune rate
Method
MMMLU
izero-shot
MMLUU
zero-shot
MMLU
zero-shot
Uniform
48.8
38.22
56.01
16.92
32.99
58.61
20.03
34,06
56.62
60%
OWI.
35.37
51.43
58.18
15.38
39.33
59.79
17.39
44.07
59.27
Ours
32.46
52.27
89.24
15.07
39.78
60.52
17.03
43.91
59.34
Uniform
1375.75
25.5
40.88
122.9
24.53
42,49
111.37
26.8
41.14
70%
OWL
462.67
28.5
46.33
56.38
26.66
48.05
67.72
26.9
45.81
Ours
208.7
30.27
49.53
47.21
29.38
$1.62
58.34
27.6
47.04
5.4 Ablation Study
We conduct ablation studies the components of the proposed bilevel optimization framework using Phi-3-mini with 50% sparsity. We first compare the effects of layer-wise sparsity and matrix rank ratio allocation the performance of the compressed model, where w/o Rank denotes only sparsity allocation without rank ratio allocation, w/o Sparsity denotes only rank ratio allocation without sparsity allocation, and w/o Rank & Sparsity denotes neither allocation being applied. The results, shown in Table 3, demonstrate that rank ratio allocation has more significant impact on performance, factor that has been overlooked in other sparsity allocation methods. We also compare the runtime of different RPCA solvers in Table 2c where QR QR-based method is significantly faster than SVD. Additionally, we further compare the performance and runtime of the two decomposition methods under varying iteration counts, as shown in Figure 3 At 80 iterations, both methods achieve similar zero-shot shot accuracy, while the QR-based method completes in just a few minutes.
Further Analysis
In this section, we discuss the allocation results and actual acceleration effects of various types of model compression, as well as the performance our bilevel optimization framework when combined with other metric-based compression methods.
8

---

Table Ablation study on layer- wise sparsity and matrix rank ratio allocation.
Figure 3: Impact of the number of itera- tions on accuracy and time cost.
Alloc
Perplexity
MMMLU
zero-shot
14
Sparsity & Rank
14.87
60.57
69.37
w/o Rank
14,94
$9.94
68.48
w/o Spursity
14.89
60.04
68.93
w/o Rank & Sparsity
15.30
58.28
67.48
Narberof Pristiin
150
120
150
Q2.42
6.1 Allocation Visualization
Figure 4 Dillustrates the allocation results of Phi-3-mini obtained by our bilevel framework. The prune rate gradually increases across layers. This reflects the variation in parameter redundancy across layers. Regarding rank ratios across matrices, our method accurately captures the heterogeneous low-rankness. For matrices in MIP, fewer budgets are allocated to the low-rank component, allowing more flexibility for the sparse component to preserve model fitting capacity. Conversely, for attention blocks, our method allocates more rank budget to fully exploit the underlying structure.
e_proj
Prune Eate
k
"_proj
o_proj
gate_up ppoproj
down_pro proj
AB
Layer Index
Layer Index
Layer-wise Sparsity Allocation
Matrix-wise Rank Ratio Allocation
Figure 4: Left: Layer-wise sparsity allocation Right: Matrix-wise rank ratio allocation A.
6.2 CPU SpeedUp
Following the settings in OATS and OWL, use the DeepSparse engine evaluate the actual CPU acceleration achieved by various compression methods on the Llama2-13B model. As shown in Table 4 RPCA-based methods yield greater speedups (up to 1.99 at 50% sparsity) thanks to structured low-rank components, and also outperform purely sparse methods in accuracy.
Table 4: Throughput and speedup comparison among different prune types.
Prune rate
Prune type
Method Throughput (B/s)
Speedup (x)
Perplexity
Dense
2.38
1.00
Unstructured
Wanda
2.93
1.23
8.77
40%
Low rank & Sparsity
Ours
3.92
1.64
8.31
Unstructured
Wanda
3,99
1.68
10.05
50%
Semi-unstructured (2:4)
Wanda
4.38
1.84
16.53
Low rank Sparsity
Ours
4.75
1.99
9.23
6.3
Integration with Other Compression Methods
To extend our framework beyond low-rank and sparse decomposition, we apply it to peetricbbsse pruning methods like Wanda and SparseGPT. shows results on Phi-3-mini 50% sparsity. Compared to uniform and OWL-based allocations, our method consistently outperforms both. This demonstrates our approach's potential as a general sparsity allocation mechanism across compression methods, providing new insights for model compression. The performance improvement brought by our method is less pronounced compared to the RPCA-based approach, suggesting that the latter presents more challenging problem that requires accurate sparsity allocation optimization.

---

Table 5: Integration with other compression methods.
Base method
Alloc method
Perplexity
t MMLU
1 zero-shot
Uniform
17.23
54.57
65.03
Wanda
OWL
16.22
55.27
65.97
Ours
16.01
55.78
66.14
Uniform
16.80
53.22
66.36
SparseGPT
OWL
17.39
56,35
65.95
Ours
16.18
56.86
66.63
Conclusion
In this work, we propose bilevel optimization framework that unifies learning and matrix approxima- tion for LLM compression. By formulating sparsity and rank allocation as probabilistic optimization problem and solving the matrix approximation subtask via RPCA. method effectively captures weight redundancy structures. We introduce truncated Gaussian prior for probabilistic parameteri- zation, combined with policy gradient estimator, which avoids impilicit differentiation through the inner loop and stabilizes training. Additionally, we design a QR-based RPCA solver that significantly accelerates the inner loop computation. Our collaborative mechanism offers new perspective and practical methodology for advancing efficient and effective model compression.
Acknowledgements
This work was supported by the National Nature Science Foundation of China (62472097), Shanghai Municipal Science and Technology Commission (Grant 1o.24511106111 Fudan Kunpeng&Ascent Center of Cultivation and AI for Science Foundation Fudan University (FudanX24A1029), The computations in this research were performed on the CFFF platform of Fudan University.
10

---

References
[1] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary. Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmaan, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lec, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin. Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy. Olatunji Ruwase Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang. Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang. Guanhua Wang, Lijuan Wang, Shuohang Wang, Xin Wang, Yu Wang. Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Jilong Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang. Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: highly capable language model locally on your phone, 2024.
[2] Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation- based adaptive structured pruning for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10865-10873, 2024.
[3] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019.
[4] Emmanuel Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM), 58(3):1-37, 2011.
[5] Christopher Clark, Kenton Lee, Ming- Klin-Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. 2019.
[6] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018.
[7] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
[8] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323-10337. PMIR, 2023.
[9] Elias Frantar, Carlos Riquelme Ruiz, Neil Houlsby. Dan Alistarh, and Utku Evei. Scaling laws for pparsely-connectee foundation models. In The Twelfth International Conference on Learning Representations.
[10]
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for shot language model evaluation, 07 2024.

---

[11] Saeed Ghadimi, Guanghui Lan, and Hongchao Zhang. Mini-batch stochastic approxima- tion methods for nonconvex stochastic composite optimization. Mathematical Programming. 155(1))267-3055 2016.
[12] Song Guo, Jiahang Xu, Li Lyna Zhang, and Mao Yang. Compresso: Structured pruning with col- laborative prompting learns compact large language models. arXiv preprint arXiv:2310.0501,, 2023.
[13] Shwai He, Guoheng Sun, Zheyu Shen, and Ang Li. What matters in transformers? not all attention is needed. arXiv preprint arXiv:2406. 15786, 2024.
[14] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021.
[15] Kaiyi Ji, Junjie Yang, and Yingbin Liang. Bilevel optimization: Convergence analysis and enhanced design. In International conference machine learning, pages 4882-4892. PMLR, 2021.
[16] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- fional Conference on Learning Representations (ICLR 2015), 2015.
[17] Guangyan Li, Yongqiang Tang, and Wensheng Zhang. Lorap: transformer sub-layers deserve differentiated structured compression for large language models. In Proceedings of the 41st International Conference on Machine Learning. pages 28657-28672. 2024.
[18] Lujun Li, Peijie Dong, Zhenheng Tang, Xiang Liu, Qiang Wang, Wenhan Luo, Wei Xue, Qifeng Liu, Xiaowen Chu, and Yike Guo. Discovering sparsity allocation for layer-wise pruning of large language models. Advances Neurul Information Processing Systems, 37:141292-141317, 2024.
[19] Wei Li, Lujun Li, Mark Lee, and Shengjie Sun. Adaptive layer sparsity for large language models via activation correlation assessment. Advances in Neural Information Processing Systems, 37:109350-109380 2024.
[20] Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Losparse: Structured compression of large language models based on low-rank and sparse approximation. In International Conference on Machine Learning, pages 10336-20350. PMLR, 2023.
[21] Zhouchen Lin, Minming Chen, and Yi Ma. The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices. arXiv preprint arXiv:1009.500 5055, 2010.
[22] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-21722 2023.
[23] Mehdi Makni, Kayhan Behdin, Zheng Xu, Natalia Ponomareva, and Rahul Mazumder. A unified framework for sparse plus low-rank matrix decomposition for 1lms. In The Second Conference on Parsimony and Learning (Proceedings Track).
[24] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2022.
[25] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering, 2018.
[26] J. Pablo Muñoz, Jinjie Yuan, and Nilesh Jain. Multipruner: Balanced structure removal in foundation models, 2025.
[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-67 2020.
[28] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: adversarial winograd schema challenge at scale, 2019.
12

---

[29] Victor Sanh, Lysandre Debut, Julien Chaumond. and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
[30] Ankur Sinha, Pekka Malo, and Kalyanmoy Deb. A review on bilevel optimization: From classical to evolutionary approaches and applications. JEEE transactions on evolutionary computation, 22(2):276-295, 2017.
[31] Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations.
[32] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[33] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini vrXii:2307.09288 2023.
[34] Tycho FA van der Ouderaa, Markus Nagel, Mart Van Baalen, and Tijmen Blankevoort. The Ilm surgeon. In The Twelfth International Conference on Learning Representutions.
[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019.
[36] Weiran Wang and Miguel A Serreira-eerpiánn Projection onto the probability simplex: An efficient algorithm with a simple proof, and an application. arXiv preprint 1XXiv:130.11541, 2013.
[37] Xin Wang, Yu Zheng. Zhongwei Wan, and Mi Zhang. Svd-lim: Truncation-aware singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07377, 2024.
[38] Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang, Zhongzhu Zhou, Xiafei Qiu, Yong Li, Wei Lin, and Shuaiwen Leon Song. Flash-lim: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity. Proceedings of the VLDB Endowment, 17(2):211-224, 2023.
[39] Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared Ilama: Accelerating language model pre-training via structured pruning, In The Twelfth International Conference Learning Representations.
[40] Peng Xu, Wenqi Shao, Mengzhao Chen, Shitao Tang, Kaipeng Zhang. Peng Gao, Fengwei An, Yu Qiao, and Ping Luo. Besa: Pruning large language models with blockwise parameter-effiiient sparsity allocation. arXiv preprint arXiv:2402.1 16880, 2024.
[41] Lu Yin, You WI, Zhenyu Zhang, Cheng-Ye Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang, et al. Outlier weighed layerwise sparsity (owl): missing secret sauce for pruning Ilms high sparsity. In The Forty-First International Conference on Machine Learning, 2024.
[42] Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7370-7379, 2017.
[43] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence?, 2019.
[44] Stephen Zhang and Vardan Papyan. OATS: Outlier-aware pruning through sparse and low rank decomposition. In The Thirteenth International Conference on Learning Representations, 2025.
[45] Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. Plug-and-play: An efficient post-training pruning method for large language models. In The Twelfth International Conference Learning Representations, 2024.
13

---

[46] Yuxin Zhang, Lirui Zhao, Mingbao Lin, Sun Yunyun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Dynamic sparse no training: Training-free fine-tuning for sparse Ilms. In The Twelfth International Conference on Learning Representations.
[47] Xiao Zhou, Renjie Pi, Weizhong Zhang, Yong Lin, Zonghao Chen, and Tong Zhang. Prob- abilistic bilevel coreset selection. In International conference on machine learning, pages 27287-27302. PMLR, 2022.
[48] Xiao Zhou, Weizhong Zhang, Hang Xu, and Tong Zhang. Effective sparsification of neural networks with global sparsity constraint. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Puttern Recognition, pages 3599-3608, 2021.
[49] Zihan Zhou, Xiaodong Li, John Wright, Emmanuel Candes, and Yi Ma. Stable principal component pursuit. In 2010 IEEE international symposium on information theory, pages 1518 1522. IEEE, 2010.
[50] Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.0187, 2017.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main contributions of this paper lie in the proposed bilevel optimization framework that jointly integrates learning and matrix approximation, along with series of technical innovations, which are elaborated in detail in the method section. Furthermore, comprehensive experiments are conducted to validate the effectiveness of our approach and demonstrate its advantages.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitations of our work in Appendix Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide a full derivation of our theoretical results, including assumptions and proofs, in the appendix Section D
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed descriptions of all experimental configurations, and we believe the results can be reproduced regardless of whether the code and data are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Yes, we will make experimental code available in the supplementary materials. The data used our experiments is open source.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
public//giide//Ddelebbiiiss oow.ssoboisiiooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips //iiss.c//pule cttp:/scccpbblccgcuiceeeeeesc guides http:///lwlllliiieeeooooetaaaaaa
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The experimental configuration has been described throughout the manuscript and supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: Standard metrics and statistical summaries are reported consistently.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details of the computing platform and training resources are provided in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httt::::/eerrrpp..cpuuul/////ai
Answer: [Yes]
Justification: This work was carried out in strict accordance with the ethical guidelines forth by NeurIPS.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., - if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the broader impacts in Appendix Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our study does not involve any such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We used publicly available datasets and models in accordance with their licenses and terms, and provided proper citations.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.paperssit
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
We did not release any new assets in this work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No part of our experiments involved crowdsourcing or the use of human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (TRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No part of our experiments involved crowdsourcing or the use of human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: Large language models (LLMs) serve as the primary subjects of our study. As detailed in Section [ provide comprehensive documentation on: (1) the pruning methodology applied to LLMs; (2) evaluation protocols used to assess the pruned models.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
cc / http://eurrrps.cccoeeeeeeeeeeeee
21


Reviews:
[{"review_id": "6I1qHEuB9p", "reviewer": "Reviewer_AQTX", "summary": "Authors propose a novel optimization framework based on Robust PCA for the compression of LLMs. They combine learning-based sparsity allocation and QR-based matrix approximation algorithms. There are two technical contributions in the paper: (1) a truncated Gaussian prior-based probabilistic parameterization with policy gradient estimator for the outer loop optimization, and (2) a QR-based matrix approximation algorithm for efficient inner loop computation. The approach automatically learns layer-wise sparsities and matrix-wise retained ranks, demonstrating strong performance on Phi-3 and Llama model families.", "questions": "1. All speedup evaluations are on CPU using DeepSparse. Given that most LLM inference happens on GPUs, can you provide GPU inference benchmarks? How does the structured sparsity from RPCA translate to GPU acceleration?\n2. Can you provide the convergence analysis of the proposed method? How does different hp affect the convergence rate?\n3. There are multiple hyperparameters, how do they affect the results? I feel like it will be really difficult to handle these many hyperparameters.\n4. Time taken for bilevel optimization compared to one-shot methods like Wanda/SparseGPT?", "limitations": "The authors appropriately discuss some of the limitations in the appendix, including computational overhead and hyperparameter sensitivity. Additional limitations include scalability to very large models and the need for careful tuning of different hyperparameters.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper is built on a strong technical and theoretical foundation. Even though RPCA has been used for model pruning; problem formulation, methodology and optimization framework proposed in the paper is novel. \n2. Authors have done proper analysis of all the theorems and provided the proofs in appendix.\n3. Authors have done comprehensive evaluation and ablation across different model families (Phi-3, Llama-2/3).\n\nWeaknesses:\n1. Technical novelty: While the combined optimization framework is novel, the individual components (RPCA for compression, policy gradient methods) are well-established. \n2. Scalability concerns: Despite improvements, the bilevel optimization requires multiple sampling iterations and may face challenges scaling to very large models (>100B parameters).\n3. Limited Accuracy and CPU SpeedUp compared to baseline: Zero shot accuracy gain not much significant and also inference speedup looks minimal compared to other methods.\n4. Hyperparameter sensitivity: The method introduces several hyperparameters which might be very difficult to tune. Also ablation for some of the important hp (for example for all the hp of truncated Gaussian distribution) gamma should be provided.\n5. Comparisons with some recent structured/unstructured pruning like GBLM-Pruner[1], ATP[2].\n6. Inference speedups on GPUs are not shown. \n7. Missing Convergence Analysis for the overall bilevel optimization.\n\nReferences:\n\n1. Das, Rocktim Jyoti, et al. \"Beyond size: How gradients shape pruning decisions in large language models.\" arXiv preprint arXiv:2311.04902 (2023).\n\n2. Huang, Weizhong, et al. \"Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective.\" arXiv preprint arXiv:2502.14770 (2025).", "was_revised": true, "final_justification": "Based on author response and remarks from other reviewers, I have decided to retain the score.", "rebuttal": "## W1: Novel framework and well-established components.\nWe thank the reviewer for acknowledging the novelty of our framework. \n\nWhile the individual components are well-established, our contribution lies in designing a collaborative mechanism and solving strategy that coordinate them under a bilevel framework, which, to the best of our knowledge, is the first of its kind in model pruning. We believe such collaboration framework would be effective and helpful to developing high-precision pruning methods for LLMs, especially in the scenario with limited computational and data resources.\n\nMoreover, beyond the general framework, we incorporate several specific design choices to enhance training efficiency. For instance, we adopt a truncated Gaussian prior, which effectively mitigates the risk of gradient explosion (see Section 4.2). This issue can arise when using a standard Gaussian distribution, as its gradient contains a $1/\\sigma^2$ term. To ensure convergence, one typically needs to reduce $\\sigma$ to concentrate the sampling distribution, but this can lead to unstable gradients. The truncated version addresses this problem by allowing us to concentrate samples via truncation while keeping $\\sigma$ fixed, thereby maintaining gradient stability.\n\n## W2: Scalability concerns.\nThank you for pointing out this concern. Due to the space limit, please refer to our response to reviewer ZdES's W2 & Q1.\n\n## W3: Limited improvement on zero shot and speedup.\n\nRegarding zero-shot accuracy, we would like to clarify two key points. \n - Zero-shot task is relatively easy. At low sparsity levels (e.g., below 50%), the performance degradation caused by pruning is generally minor—even for baseline methods, the post-pruning performance remains close to that of the original model. This suggests that the room for further improvement is quite limited, making it nearly impossible to achieve significant performance gains under such settings.\n\n- Our method demonstrates clear and significant improvements under high sparsity levels (60–70%), as shown in Table 2 of the main paper and **Table R1** below.\n\n#### Table R1. Performance comparison under high prune rates.\n|Prune rate | Method | phi-3 wiki | phi-3 MMLU |phi-3 zero-shot | llama2-7b wiki | llama2-7b MMLU | llama2-7b zero-shot | llama3-8b wiki | llama3-8b MMLU | llama3-8b zero-shot |\n|---|---|---|---|---|---|---|---|---|---| ---|\n| 60% |SparseGPT | 51.44 | 37.37 | 54.61 | 18.38 | 29.77 | 57.92 | 23.21 | 34.23 | 55.39 |\n| 60% | Wanda | 52.28 | 36.96 | 55.48 | 18.12 | 30.41 | 57.64 | 23.63 | 31.82 | 54.86 |\n| 60% | OATS | 42.37 | 43.96 | 57.82 |16.78  | 34.64 | 59.43 | 21.30 | 37.26 | 57.78 |\n| 60% | QR |48.8  | 38.22 | 56.01 | 16.92 | 32.99 | 58.61 | 20.03 | 34.06 | 56.62 |\n| 60% | Ours | 32.46 | 52.27 | 59.24 | 15.07 | 39.78 | 60.52 | 17.03 | 43.91 | 59.34 |\n| 70% |SparseGPT | 1175 | 25.83 | 42.34 | 112.28 | 22.88 | 43.37 | 98.28 | 22.88 | 41.37 |\n| 70% | Wanda | 2124.9 | 26.01 | 38.94 | 235.7 | 25.3 | 39.71 | 183.0 | 25.45 | 38.15 |\n| 70% | OATS | 762.4 | 26.73 | 45.69 | 88.56 | 25.51 | 46.31 | 89.27 | 26.85 | 43.28 |\n| 70% | QR | 1375 | 25.5 | 40.88 | 122.9 | 24.53 | 42.49 | 111.37 | 26.8 | 41.14 |\n| 70% | Ours | 208.7 | 30.27 | 49.53 | 47.21 | 29.38 | 51.62 | 58.34 | 27.6 | 47.04 |\n\nRegarding inference speedup, we argue that\n- We present the results in Table 4 to illustrate the additional speedup gained from the low rank and sparse structure; specifically, low rank matrices are generally more hardware-friendly for acceleration, rather than highlighting their overall superiority in speedup. \n- At the same compression rates, the compressed models obtained from different pruning strategies usually yield comparable inference speedups, as the additional speedup derived from the unique matrix structure cannot be excessively high.\n- Consequently, our objective is to achieve higher accuracy than the baselines while maintaining the same compression ratio, as demonstrated by the results presented in the main text.\n\n## W4 & Q3: Hyperparameter sensitivity.\nWe acknowledge that our bilevel framework introduces several hyperparameters. However, we would like to clarify following facts to show that our algorithm is not sensitive to these hyperparameters. \n- In fact, we tuned these hyperparameters on Phi-3 mini and applied the same configuration across all other model experiments. Besides, parameters such as the initialization of the truncated Gaussian mean $\\mu$ and the number of inner-loop iterations are consistent with those used in OATS. \n- We also conduct ablation studies on these key hps, and the results are presented in **Table R2** below. \n- Moreover, theoretical analysis (see our response to reviewer tY2u's Question 2) shows the dependency of convergence on the parameters such as $\\gamma$.\n#### Table R2. Ablation study of hyperparameters Phi3-mini with 50% prune rate, wikitext2 PPL\n|$\\sigma^2$ | 1e-3 | 5e-3 | 1e-2 | 5e-2 | 1e-1 |\n|---|---|---|---|---|---|\n| PPL | 15.16 | 14.94 | 14.87 | 14.92 | 15.07 |\n|$\\gamma_0$ | **1e-3** | **5e-3** | **1e-2** | **5e-2** | **1e-1** |\n| PPL | 15.22 | 15.01 | 14.93 | 14.87 | 14.99 |\n|$\\gamma_{target}$ | **1e-3** | **5e-3** | **1e-2** | **5e-2** | **1e-1** | \n| PPL | 14.88 | 14.87 | 14.90 | 15.07 | 15.18 |\n\n## W5: Comparisons with GBLMpruner & ATP.\nThank you for pointing out these recent works in model compression. We present the comparison results in **Table R3**. \n\nIt is worth clarifying that ATP originally uses a customed and simplified version of the `lm_eval` package (based on version 0.3.0). Since we use an updated version (0.4.2) consistent with OATS in our main experiments, there may be slight discrepancies in evaluation results, which is also mentioned in OATS's Appendix A.12. To ensure fairness, we re-evaluated the perplexity on Wikitext2 of our methods using the version adopted in ATP's implementation.\n\nDue to our learned sparsity allocation and the integration of low-rank matrices, our method outperforms both GBLM-Pruner and ATP across all settings.\n#### Table R3. Comparison with GBLM and ATP on LLaMA2-7B. \"-\" means missing results in the original paper.\n|Prune rate | Method | PPL$\\downarrow$ | MMLU$\\uparrow$ | Zero-shot$\\uparrow$ |\n|---|---|---|---|---|\n| 0%  | Dense | 5.47 | 50.12 | 66.27 |\n| 50% | GBLMpruner | 6.86 | - | - |\n| 50% | ATP+wanda | 6.82 | - | 64.49 |\n| 50% | Ours | 6.47 | 46.10 | 65.34 |\n| 60% | ATP+wanda | 9.15 | 32.80 |58.63|\n| 60% | Ours | 8.79 | 39.78 | 60.52 |\n\n## W6 & Q1: GPU Inference speedups.\nWe appreciate the reviewer’s comment and agree that presenting GPU inference speedup is important. In our initial experiments, we followed the evaluation protocols of OATS and OWL, which focus on CPU speed up, and therefore we did not include GPU results.\nTo address this, we followed the ATP protocol and conducted GPU inference tests using `nm-vllm`. The results are presented in **Table R4**.\nCompared with unstructured methods, our approach, Low rank & Sparse, leads to more practically meaningful GPU inference acceleration.\n#### Table R4 GPU inference speedup LLaMA2-13B, on a single 80 GB A100 GPU\n|Prune rate|Prune Type|Method|Throughput (tokens/s)|Speedup|Perplexity |\n|---|---|---|---|---|---|\n| 0% | Dense | - | 47.31 | 1.0 | 7.91 |\n| 40% | Unstructured | Wanda | 68.73 | 1.45 | 8.77 |\n| 40% | Low rank & Sparse | Ours | 79.87 | 1.69 | 8.31 |\n| 50% | Unstructured | Wanda | 83.15 | 1.76 | 10.05 |\n| 50% | Semi-structured | Wanda | 92.36 | 1.95 | 16.53 |\n| 50% | Low rank & Sparse | Ours | 102.45 | 2.17 | 9.23 |\n\n## W7 & Q2: Convergence Analysis.\nDue to space limitations in the responses, we provide a detailed convergence analysis of the bilevel framework in our response to **Reviewer tY2u**'s Question 2. We will include this analysis in the appendix of the revised version if accepted.\n\nTheoretical analysis demonstrate that a smaller value of $\\gamma$ leads to reduced gradient variance, which in turn facilitates faster convergence of the optimization process.\n\n\n## Q4: Time taken compared to one-shot methods:\nWe acknowledge that, unlike one‑shot methods such as Wanda or SparseGPT, our bilevel framework incurs additional runtime. However, our entire process still completes within a few hours on standard hardware, which is always affordable in downstearm applications.\n\nImportantly, model compression is a one‑time offline cost, where **peak GPU memory usage often outweighs total runtime in practical deployments. Our method uses no more memory than one‑shot approaches but delivers superior accuracy and inference speedups**, making the extra time investment worthwhile.\nThe results of compression time are summarized in **Table R5**. Even **QR**, a simplified version of our method, still outperforms SparseGPT and Wanda under comparable time cost.\n\n#### Table R5. Compression time, Phi-3 mini 60% prune rate\n|Method | Prune Time | Wiki | MMLU | Zero-shot |\n|---|---|---|---|---|\n|SparseGPT | 27min | 51.44 | 37.37 | 54.61 |\n|Wanda | 12min | 52.28 | 36.96 | 55.48 |\n|OATS | 5h | 42.37 | 43.96 | 57.82 |\n|QR | 15min |48.8  | 38.22 | 56.01 |\n|Ours | 4.5h | 32.46 | 52.27 | 59.24 |"}, {"review_id": "ifYGAHLIfq", "reviewer": "Reviewer_tY2u", "summary": "This paper proposes a compression method for LLMs using a bilevel optimization framework that integrates learning with sparse and low-rank matrix approximations. The outer loop optimizes sparsity allocation using truncated Gaussian priors and policy gradients, while the inner loop efficiently solves matrix approximation problems via QR decomposition instead of conventional SVD. Experiments demonstrate competitive performance in accuracy preservation under significant sparsity.", "questions": "How sensitive is the method to variations in critical hyperparameters like variance and truncation?\n\nAre there theoretical insights or proofs available regarding the convergence of your bilevel optimization?\n\nIs the any indication that the proposed method can scale efficiently to models significantly larger than those tested?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[Strengths]\nThe integration of bilevel optimization and truncated Gaussian-based policy gradients seems novel.\n\nThey demonstrate improvements over several existing methods in terms of accuracy and speed on multiple popular LLM architectures such as Phi-3, Llama2-7B, Llama2-13B, and Llama3-8B.\n\n[Weaknesses]\nTheoretical convergence guarantees, optimality conditions, and stability analyses of the proposed method are not explored sufficiently.\n\nSuccess heavily dependent on several hyperparameters like variance and truncation parameters, which may require extensive tuning and limit practical utility.\n\nAlthough QR decomposition is more efficient than SVD, the method remains computationally demanding for large-scale applications.\n\nPractical deployment may be hindered due to extensive hyperparameter tuning requirements.", "was_revised": true, "final_justification": "The authors have sufficiently addressed my initial concerns in their rebuttal. I have updated my score accordingly.", "rebuttal": "## W1: Theoretical guarantees, optimality conditions and so on.\nThank you for pointing out this limitation. We provide a theoretical convergence analysis with optimality conditions in our response to your Question 2. We will incorporate these discussions into the revised version if accepted. \n\n## W2, W4 & Q1: Hyperparameters' sensitivity.\nWe acknowledge that our bilevel framework introduces several hyperparameters. However, we would like to clarify following facts to show that our algorithm is not sensitive to these hyperparameters. \n- In fact, we tuned these hyperparameters on Phi-3 mini and applied the same configuration across all other model experiments. Besides, parameters such as the initialization of the truncated Gaussian mean $\\mu$ and the number of inner-loop iterations are consistent with those used in OATS. \n- We also conduct ablation studies on these key hps, and the results are presented in **Table R1** below. \n- Moreover, theoretical analysis (see our response to your Question 2) shows the dependency of convergence on the parameters such as $\\gamma$.\n#### Table R1. Ablation study of hyperparameters Phi3-mini with 50% prune rate, wikitext2 PPL\n|$\\sigma^2$ | 1e-3 | 5e-3 | 1e-2 | 5e-2 | 1e-1 |\n|---|---|---|---|---|---|\n| PPL | 15.16 | 14.94 | 14.87 | 14.92 | 15.07 |\n|$\\gamma_0$ | **1e-3** | **5e-3** | **1e-2** | **5e-2** | **1e-1** |\n| PPL | 15.22 | 15.01 | 14.93 | 14.87 | 14.99 |\n|$\\gamma_{target}$ | **1e-3** | **5e-3** | **1e-2** | **5e-2** | **1e-1** | \n| PPL | 14.88 | 14.87 | 14.90 | 15.07 | 15.18 |\n## W3 & Q3: Large-scale applications.\nThank you for pointing out this concern. Due to the space limit, please refer to our response to reviewer ZdES's W2 & Q1.\n\n## Q2  Theoretical analysis:\nWe would like to deliver a formal form of our theorem below by adding some assumptions following [r1].\n\n**Theorem** For bilevel problem with the form as:\n$$\n\\min_ {\\theta \\in \\mathcal{C}} \\Phi(\\theta) = \\mathbb{E}_ {(s, \\kappa) \\sim p(\\cdot \\mid \\theta)} \\mathcal{L}(\\mathcal{W}(s, \\kappa)) \\quad \\text{with} \\quad \\mathcal{W}(s, \\kappa) = \\text{RPCA}(\\mathcal{W}, s, \\kappa).\n$$\nWe assume $\\Phi(\\theta)$ is $L$-smooth, and the policy gradient variance satisfies:\n\n$$\n\\mathbb{E}\\left\\lVert\\mathcal{L}(\\mathcal{W}(s, \\kappa))\\nabla_{\\theta} \\log p(s, \\kappa \\mid \\theta)-\\nabla_{\\theta} \\Phi(\\theta) \\right\\rVert^2 \\leq \\tilde{\\sigma}^2.\n$$\n\nLet $\\eta < 1/L$ and define the gradient mapping $\\mathcal{G}^t$ at iteration $t$ as:\n\n$$\n\\mathcal{G}^t = \\frac{1}{\\eta}\\left(\\theta^t - \\mathcal{P}_{\\mathcal{C}}\\left(\\theta^t - \\eta \\nabla _{\\theta} \\Phi(\\theta^t)\\right)\\right),\n$$\n\nthen when $T \\rightarrow \\infty$, we have:\n\n$$\n\\frac{1}{T} \\sum_{t=1}^T \\mathbb{E} \\lVert \\mathcal{G}^t\\rVert^2 \\leq \\frac{8 - 2L\\eta}{2 - L\\eta} \\tilde{\\sigma}^2.\n$$\n\n**Remark** We would like to point out the following:\n\n- We give this theorem just to show that our algorithm behaves similarly to general projected/proximal stochastic gradient descent algorithms for one-level optimization problems (e.g., [r2]), rather than to characterize how fast our algorithm converges. Therefore, we do not consider variance reduction techniques, which are out of scope for this work.\n\n- $\\tilde{\\sigma}^2$ is the variance of our gradient estimator PGE with a single sampling of (s,$\\kappa$). It can be reduced using multiple sampling, larger batch sizes and other variance reduction techniques, hence the LHS of Equation (1) can converge to a small value.\n\n-  As shown in line 912 of Appendix D.3, $\\tilde{\\sigma}^2$ becomes small when $\\gamma$ is small. In our experiments, $\\gamma$ is gradually reduced to a smaller value to facilitate convergence.\n---\nWe need the following lemmas about the properties of the projection operator, which can be found in [r2].\n\n**Lemma 1 (Firmly Nonexpansive Operators)**  \nLet $\\mathcal{C} \\subset \\mathbb{R}^d$ be compact and convex. For any $\\mathbf{u}, \\mathbf{v} \\in \\mathbb{R}^d$:\n\n$$\n\\lVert \\mathcal{P}_ {\\mathcal{C}}(\\mathbf{u}) - \\mathcal{P}_ {\\mathcal{C}}(\\mathbf{v}) \\rVert^2 \\leq (\\mathbf{u} - \\mathbf{v})^\\top \\left( \\mathcal{P}_ {\\mathcal{C}}(\\mathbf{u}) - \\mathcal{P}_ {\\mathcal{C}}(\\mathbf{v}) \\right).\n$$\n\n**Lemma 2**  \nLet $\\mathcal{C} \\subset \\mathbb{R}^d$ be compact and convex. Then for any $\\mathbf{c} \\in \\mathcal{C}$ and $\\mathbf{u}, \\mathbf{v} \\in \\mathbb{R}^d$:\n\n$$\n\\lVert\\mathcal{P}_ {\\mathcal{C}}(\\mathbf{c} + \\mathbf{u}) - \\mathcal{P}_ {\\mathcal{C}}(\\mathbf{c} + \\mathbf{v}) \\rVert \\leq \\lVert \\mathbf{u} - \\mathbf{v} \\rVert.\n$$\n\n---\n**Proof of Theorem:**\nWe define:\n\n$$\n\\mathbf{g}^t = \\mathcal{L}_ {\\mathcal{B}}(\\mathcal{W}(s, \\kappa)) \\nabla_ {\\theta} \\log p(s, \\kappa| \\theta^t).\n$$\n\nOur method updates $\\theta$ as:\n\n$$\n\\theta^{t+1} = \\mathcal{P}_{\\mathcal{C}}\\left( \\theta^t - \\eta \\mathbf{g}^t \\right).\n$$\n\nLet the stochastic and deterministic gradient mappings be\n\n$$\n\\begin{aligned}\n    \\hat{\\mathcal{G}}^t &= \\frac{1}{\\eta}\\left(\\theta^{t}-\\mathcal{P}_ {\\mathcal{C}}\\left(\\theta^t- \\eta \\mathbf{g}^t\\right)\\right) = \\frac{1}{\\eta}\\left(\\theta^{t}-\\theta^{t+1}\\right), \\\\\\\\\n    \\mathcal{G}^t &= \\frac{1}{\\eta}\\left(\\theta^{t}-\\mathcal{P}_ {\\mathcal{C}}\\left(\\theta^t- \\eta \\nabla\\Phi(\\theta^t)\\right)\\right),\n\\end{aligned}\n$$\n\nwe can have\n\n$$\n\\begin{aligned}\n    \\Phi(\\theta^{t+1}) &\\leq \\Phi(\\theta^{t}) + \\langle \\nabla\\Phi(\\theta^{t}), \\theta^{t+1}-\\theta^t \\rangle + \\frac{L}{2} \\lVert \\theta^{t+1}-\\theta^t \\rVert^2 \\\\\\\\\n    &= \\Phi(\\theta^{t}) - \\eta \\langle \\nabla\\Phi(\\theta^{t}), \\hat{\\mathcal{G}}^t \\rangle + \\frac{L\\eta^2}{2} \\lVert \\hat{\\mathcal{G}}^t \\rVert^2 \\\\\\\\\n    &= \\Phi(\\theta^{t}) - \\eta \\langle \\nabla\\Phi(\\theta^{t}) - \\mathbf{g}^t + \\mathbf{g}^t, \\hat{\\mathcal{G}}^t \\rangle + \\frac{L\\eta^2}{2} \\lVert \\hat{\\mathcal{G}}^t \\rVert^2 \\\\\\\\\n    &= \\Phi(\\theta^{t}) - \\eta \\langle \\mathbf{g}^t, \\hat{\\mathcal{G}}^t \\rangle + \\frac{L\\eta^2}{2} \\lVert \\hat{\\mathcal{G}}^t \\rVert^2 + \\eta \\langle \\delta^t, \\hat{\\mathcal{G}}^t \\rangle \\quad \\text{(here } \\delta^t = \\mathbf{g}^t - \\nabla\\Phi(\\theta^{t})) \\\\\\\\\n    &\\leq \\Phi(\\theta^{t}) - \\eta \\lVert \\hat{\\mathcal{G}}^t \\rVert^2 + \\frac{L\\eta^2}{2} \\lVert \\hat{\\mathcal{G}}^t \\rVert^2 + \\eta \\langle \\delta^t, \\hat{\\mathcal{G}}^t \\rangle  \\quad \\quad \\text{(Lemma 1)}\\\\\\\\\n    &\\leq \\Phi(\\theta^{t}) - (\\eta - \\frac{L\\eta^2}{2}) \\lVert \\hat{\\mathcal{G}}^t \\rVert^2 + \\eta \\langle \\delta^t, \\hat{\\mathcal{G}}^t \\rangle \\\\\\\\\n    &= \\Phi(\\theta^{t}) - (\\eta - \\frac{L\\eta^2}{2}) \\lVert \\hat{\\mathcal{G}}^t \\rVert^2 + \\eta \\langle \\delta^t, \\mathcal{G}^t \\rangle + \\eta \\langle \\delta^t, \\hat{\\mathcal{G}}^t - \\mathcal{G}^t \\rangle \\\\\\\\\n    &\\leq \\Phi(\\theta^{t}) - (\\eta - \\frac{L\\eta^2}{2}) \\lVert \\hat{\\mathcal{G}}^t \\rVert^2 + \\eta \\langle \\delta^t, \\mathcal{G}^t \\rangle + \\eta \\lVert \\delta^t \\rVert \\lVert \\hat{\\mathcal{G}}^t - \\mathcal{G}^t \\rVert \\\\\\\\\n    &\\leq \\Phi(\\theta^{t}) - (\\eta - \\frac{L\\eta^2}{2}) \\lVert \\hat{\\mathcal{G}}^t \\rVert^2 + \\eta \\langle \\delta^t, \\mathcal{G}^t \\rangle + \\eta \\lVert \\delta^t \\rVert^2. \\quad \\quad (\\text{Lemma 2})\n\\end{aligned}\n$$\n\nTherefore, we can get\n\n$$\n(\\eta - \\frac{L\\eta^2}{2}) \\lVert \\hat{\\mathcal{G}}^t \\rVert^2 \\leq \\Phi(\\theta^{t}) - \\Phi(\\theta^{t+1}) + \\eta \\langle \\delta^t, \\mathcal{G}^t \\rangle + \\eta \\lVert \\delta^t \\rVert^2.\n$$\n\nThus, we obtain\n\n$$\n\\sum_{t=1}^T (\\eta - \\frac{L\\eta^2}{2}) \\lVert \\hat{\\mathcal{G}}^t \\rVert^2 \\leq \\Phi(\\theta^{1}) - \\Phi(\\theta^{T+1}) + \\eta \\sum_{t=1}^T \\left( \\langle \\delta^t, \\mathcal{G}^t \\rangle + \\lVert \\delta^t \\rVert^2 \\right). \\tag{1}\n$$\n\nNow, we turn to analyze the term $\\langle \\delta^t, \\mathcal{G}^t \\rangle$ as follows:\n\n$$\n\\mathbb{E} \\langle \\delta^t, \\mathcal{G}^t \\rangle = \\mathbb{E}_ {\\theta^t} \\mathbb{E}_ {\\cdot | \\theta^t} \\left( \\langle \\mathbf{g}^t - \\nabla\\Phi(\\theta^t), \\mathcal{G}^t \\rangle \\mid \\theta^t \\right) = 0. \\tag{2}\n$$\n\nFor $\\lVert \\delta^t \\rVert^2$, we have\n\n$$\n\\mathbb{E} \\lVert \\delta^t \\rVert^2 = \\mathbb{E} \\lVert \\mathbf{g}^t - \\nabla\\Phi(\\theta^{t}) \\rVert^2 \\leq \\tilde{\\sigma}^2. \\tag{3}\n$$\n\nCombining inequalities (2), (1), and (3), we have\n\n$$\n\\frac{1}{T} \\sum_{t=1}^T \\mathbb{E} \\lVert \\hat{\\mathcal{G}}^t \\rVert^2 \\leq \\frac{\\Phi(\\theta^1) - \\Phi^*}{(1 - L\\eta/2)T} + \\frac{\\tilde{\\sigma}^2}{1 - L\\eta/2}. \\tag{4}\n$$\n\nFinally, we bound $\\mathbb{E} \\lVert \\mathcal{G}^t \\rVert^2$ as follows:\n\n$$\n\\begin{align*}\n    \\mathbb{E} \\lVert \\mathcal{G}^t \\rVert^2 &\\leq 2 \\mathbb{E} \\lVert \\hat{\\mathcal{G}}^t \\rVert^2 + 2 \\mathbb{E} \\lVert \\mathbf{g}^t - \\nabla \\Phi(\\theta^t) \\rVert^2 \\\\\\\\\n    &\\leq 2 \\mathbb{E} \\lVert \\hat{\\mathcal{G}}^t \\rVert^2 + 2 \\tilde{\\sigma}^2. \\tag{5}\n\\end{align*}\n$$\n\nCombining inequalities (5) and (4), as $T \\rightarrow \\infty$, we obtain\n\n$$\n\\frac{1}{T} \\sum_ {t=1}^T \\mathbb{E} \\lVert \\mathcal{G}^t \\rVert^2 \\leq \\frac{2}{1 - L\\eta/2} \\left( \\frac{\\Phi(\\theta^1) - \\Phi^*}{T} + (2 - L\\eta/2)\\tilde{\\sigma}^2 \\right) \\rightarrow \\frac{8 - 2L\\eta}{2 - L\\eta} \\tilde{\\sigma}^2.\n$$\n\n$\\blacksquare$\n\n### References\n[r1] Pedregosa et al., \"Hyperparameter optimization with approximate gradient.\"\n\n[r2] Ghadimi and Lan, \"Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization.\""}, {"review_id": "2btJ8FoNh7", "reviewer": "Reviewer_ZdES", "summary": "This paper proposes a bilevel optimization framework that jointly integrates learning and matrix approximation for compressing large language models. The work focuses on developing a pruning methodology for LLMs with comprehensive evaluation protocols to assess the performance of pruned models. While the technical details are not fully available in the provided context, the paper appears to present a systematic approach to model compression that aims to reduce the computational requirements of LLMs while maintaining their performance through an integrated optimization strategy.", "questions": "1. Scalability concerns: How does the proposed bilevel optimization framework scale with model size? Has it been tested on models with 10B+ parameters, and what are the memory and computational requirements during the compression process?\n\n2. Comparison with one-shot methods: How does the proposed method compare with recent one-shot pruning methods like SparseGPT in terms of both compression quality and time required for compression?\n\n3. Generalization across architectures: Has the method been evaluated across different LLM architectures (e.g., GPT, LLaMA, BLOOM), and does it require architecture-specific tuning?\n\n4. Fine-tuning requirements: Does the compressed model require task-specific fine-tuning to maintain performance, or can it maintain zero-shot capabilities after compression?\n\n5. Hardware efficiency: Have the authors evaluated the actual inference speedup on different hardware platforms (GPUs, CPUs, edge devices), considering that theoretical compression doesn't always translate to practical speedup?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. Novel optimization framework: The proposed bilevel optimization approach that jointly considers learning and matrix approximation represents a theoretically principled method for model compression, potentially offering advantages over traditional sequential compression approaches.\n\n2. Comprehensive evaluation: The paper mentions providing \"comprehensive documentation\" on evaluation protocols, suggesting thorough experimental validation of the proposed method across multiple metrics and benchmarks.\n\n3. Technical innovation: The authors claim \"a series of technical innovations\" in their method section, indicating potential contributions beyond the basic framework that could advance the state-of-the-art in LLM compression.\n\n**Weaknesses**\n\n1. Limited recent citations: The paper appears to lack references to several recent important works in LLM compression and pruning. Recent papers that should be cited include:\n\n    a) \"GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers\" (Frantar et al., ICLR 2023)\n\n    b) \"Outlier Suppression: Pushing the Limits of Low-bit Transformer Language Models\" (Wei et al., NeurIPS 2022)\n\n    c) AffineQuant: Affine Transformation Quantization for Large Language Models (Ma et al., ICLR 2024)\n\n2. Unclear computational overhead: Without access to the main content, it's unclear whether the bilevel optimization framework introduces significant computational overhead during the compression process, which could limit its practical applicability to very large models.\n\n3. Limited discussion of trade-offs: The available context doesn't indicate whether the paper thoroughly discusses the trade-offs between compression ratio, model performance, and computational efficiency during inference.", "was_revised": true, "final_justification": "The author resolved my confusion and I will maintain my score.", "rebuttal": "## W1: Limited citations on quantization.\nThank you for this valuable suggestion. We fully agree that incorporating these recent works in the area of LLM compression will strengthen the contextualization and relevance of our work. We will include a paragraph reviewing these papers in the revised version if accepted.\n\n## W2 & Q1: Computational overhead and Scability.\n Actually, our method has good performance on scalability due to the following reasons:\n\n1. **On memory cost.** Since our pruning process does not require backpropagation, the computations can be performed in a layer-wise manner. Specifically, we load and process the model one layer at a time, which effectively prevents memory usage from scaling with the overall model size. \n\n2. **On computational cost.** Thanks to the efficiency of our QR decomposition—over 75× faster than SVD—the computational cost for the inner loop is of the same order as a single forward pass. In all our experiments, we set the number of epochs to 1, making the overall cost affordable for most downstream practitioners, even when working with large-scale LLMs.\n\n**Results on Time Cost.** **Table R2** reports the time costs for QR decomposition, and other time (model (layer) loading, and GPU forward computation, etc.) per layer. The results show that these costs scale approximately linearly with the model size. This verifies our analysis above. Notice that loading and forward occupies most of the time cost, which can be improved by  engineering optimization techniques such as DeepSpeed. \n#### Table R1. Time taken of QR-based RPCA and other time\n|Model Size | RPCA time | Other time | GPU cost|\n|---|---|---|---|\n|Phi-3 mini(3.8B) | 3.39 | 22.12 | 14.28G |\n|LLaMA2-7B | 5.86 | 43.71 | 18.58G |\n|LLaMA2-13B | 11.29 | 63.89 | 22.86G |\n|LLaMA3-70B | 21.46 | 117.31 | 55G |\n\n**Results on LLaMA3-70B.** Table R2 shows that our method can outperform  the baseline OATS. More results will be included  in the revised version if accepted.\n\n#### Table R2. Results on  LLaMA3-70B.\n|Method|Time|PPL|MMLU|zero-shot|\n|:-:|:-:|:-:|:-:|:-:|\n|Dense|-|4.03|76.67|75.28|\n|OATS|120h|4.95 |74.02 |73.41|\n|Ours|80h|4.67|75.13|73.79|\n\n## W3: Limited discussion of trade-offs.\nWe thank the reviewer for pointing out the lack of discussion on the trade-offs. In the revised version, we will include an additional section that explicitly analyzes the relationship among these factors. \n\nSpecifically, based on our findings and prior work, we observe that under sparsity levels below 50%, the compressed model retains most of its original performance while achieving meaningful acceleration on both CPU and GPU platforms. When the sparsity increases beyond this point, although the inference speed continues to improve, the performance degradation becomes significant. We suggest that a sparsity level around 50% strikes a favorable balance between model efficiency and performance in practical scenarios.\n\n\n## Q2: Comparison with one-shot methods.\nWe acknowledge that, unlike one‑shot methods such as Wanda or SparseGPT, our bilevel framework incurs additional runtime. However, our entire process still completes within a few hours on standard hardware, which is always affordable in downstearm applications.\n\nImportantly, model compression is a one‑time offline cost, where **peak GPU memory usage often outweighs total runtime in practical deployments. Our method uses no more memory than one‑shot approaches but delivers superior accuracy and inference speedups**, making the extra time investment worthwhile.\nThe results of compression time are summarized in **Table R3**. Even **QR**, a simplified version of our method, still outperforms SparseGPT and Wanda under comparable time cost.\n\n#### Table R3. Compression time, Phi-3 mini 60% prune rate\n|Method | Prune Time | Wiki | MMLU | Zero-shot |\n|---|---|---|---|---|\n|SparseGPT | 27min | 51.44 | 37.37 | 54.61 |\n|Wanda | 12min | 52.28 | 36.96 | 55.48 |\n|OATS | 5h | 42.37 | 43.96 | 57.82 |\n|QR | 15min |48.8  | 38.22 | 56.01 |\n|Ours | 4.5h | 32.46 | 52.27 | 59.24 |\n\n## Q3: Generalization across architectures.\nOur main experiments focus on Phi and LLaMA2/3 models. Since our method performs optimization at the matrix level without relying on specific model architectures, it is generally applicable to other network structures as well. To further address the reviewer’s concern, we have conducted additional experiments on Qwen2.5-3B, with results shown in the table below.\n\n#### Table R4. Additional experiments on Qwen2.5 3B\n|Model | Prune rate | Wiki | MMLU | Zero-shot |\n|---|---|---|---|---|\n|Qwen2.5-3B | 0% | 11.02 | 65.99 | 68.49 |\n|Qwen2.5-3B | 50%(Ours) | 14.41 | 59.13 | 66.05 |\n\n## Q4: Fine-tuning requirements.\nTo ensure a fair comparison aligned with current research paradigms, we did not perform any task-specific fine-tuning for any of the compression methods and instead evaluated their zero-shot performance. We would like to emphasize two points: \n- First, at a sparsity level of 50%, the compressed models are able to retain over 95% of their zero-shot performance, making fine-tuning unnecessary. \n- Second, fair comparisons under fine-tuning settings are non-trivial. While our method can efficiently fine-tuning compressed model using low-rank component, methods like Wanda and SparseGPT require careful structural modifications, which fall beyond the scope of this work.  We leave this direction for future exploration.\n## Q5: Hardware efficiency.\nWe appreciate the reviewer’s comment and agree that presenting inference speedup over different platforms is important. In our initial experiments, we followed the evaluation protocols of OATS and OWL, which focus on CPU speed up, and therefore we did not include GPU results.\nTo address this, we followed the ATP[r1] protocol and conducted GPU inference tests using `nm-vllm`. The results are presented in **Table R5**.\nCompared with unstructured methods, our approach, Low rank & Sparse, leads to more practically meaningful GPU inference acceleration.\n#### Table R5. GPU inference speedup LLaMA2-13B, on a single 80 GB A100 GPU\n|Prune rate|Prune Type|Method|Throughput (tokens/s)|Speedup|Perplexity |\n|---|---|---|---|---|---|\n| 0% | Dense | - | 47.31 | 1.0 | 7.91 |\n| 40% | Unstructured | Wanda | 68.73 | 1.45 | 8.77 |\n| 40% | Low rank & Sparse | Ours | 79.87 | 1.69 | 8.31 |\n| 50% | Unstructured | Wanda | 83.15 | 1.76 | 10.05 |\n| 50% | Semi-structured | Wanda | 92.36 | 1.95 | 16.53 |\n| 50% | Low rank & Sparse | Ours | 102.45 | 2.17 | 9.23 |\n\n### References\n[r1] Lujun Li et al. \"Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models.\""}, {"review_id": "ARh2dbfOEB", "reviewer": "Reviewer_sRwV", "summary": "The authors introduce a novel algorithm to compress LLMs via a layer-wise sparse plus low-rank decomposition, using policy gradient to allocate bits across layers (and, within each layer, between the sparse and low rank components).\nThe algorithm consists of two levels of optimization: an outer level to determine the sparsity percentage and rank for each layer, and an inner level to produce a sparse plus low-rank approximation with the given sparsity and rank.\n\nThe inner level of optimization is robust PCA, with an innovation that replaces the computationally-expensive SVD step with a lightweight QR-based approximation.\nThis is essential for the computational efficiency of the compression algorithm.\nThe outer level uses policy gradient under a truncated Gaussian model and overall bit budget constraint.\nThe means of the truncated Gaussians are trainable, with a fixed variance and domain that is gradually reduced over the training process for stability reasons.\n\nPerplexity, task accuracy, and CPU speedup results are presented over several models, with the proposed method overall outperforming baselines.\nSome ablation studies and theoretical results provide justification for components of the optimization algorithm.", "questions": "(1) How were the values of hyperparameters chosen (e.g., the annealing schedule for the truncated Gaussian support, and the number of inner iterations), and how sensitive is the accuracy of the algorithm to such hyperparameters?\nIs there good set of hyperparameters that works across many different LLMs, or are the optimal values expected to be somewhat data-dependent?\nEven if they were chosen heuristically, a short statement or discussion would be interesting.\n\nHow does the accuracy of RPCA with the QR approximation differ with number of iterations?\nA convergence plot for RPCA with and without the QR approximation of SVD could be helpful.\n\n(2) How does the compression time of the algorithm compare to baseline approaches?\n\n(3) How does the empirical sparsity allocation of this method (Figure 4) compare to other sparsity allocation methods like OWL?\nAre the differences mainly subtle, or are there clear trends in how this method allocates sparsity between layers?\n\n(4) Instead of diagonally scaling W based on the input activations, is it possible to have a fully calibration-aware loss function, i.e. $||(W - L - S)X||_F^2$ (i.e., is such an algorithm easy to derive)?\nIs it known whether this would help downstream accuracy much, compared to the currently-adopted diagonal scaling method?\n\nI don't reasonably see my score decreasing (except if the questions are ignored entirely), and it could increase with very compelling experimental results (a regime or metric where the proposed method does much better than the baselines, such as accuracy under high sparsity levels, or compression time, which a clear corresponding discussion).", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Quality**\n\n_Strengths_. The compression algorithm proposed is technically sound and well-polished, and all major components appear to be thoughtfully chosen.\nThe algorithm is elegant and natural for the problem at hand.\nClaims are supported by a broad range of of results, both empirical and theoretical.\nAll proofs presented are detailed and correct.\n\nAcross LLMs and average sparsity levels, the method beats baselines in terms of perplexity, accuracy, and CPU inference speed.\nAblation studies highlight the importance of sparsity allocation, and justify the use of the QR approximation and the inclusion of both sparsity and rank allocation in the outer-level optimization.\n\nAlgorithmic details are supported by theoretical derivations, and design choices of the policy gradient level are supported by bias and variance analysis.\n\n_Weaknesses_.\nThe results section could be improved to better highlight the empirical impact of the paper.\nFor instance, the benefit in compression time over, e.g., OATS, is briefly touched upon. \nThat contribution could be emphasized via a small table comparing compression times over the different algorithms compared in Table 1, and should also appear in the introduction.\nAlso, based on Table 2 (sparsity allocation study), accuracy improvements are more pronounced at higher sparsity levels.\nCould Table 1 also include results at 60 and 70\\% sparsity?\n\nSpeedup on CPU is presented, but no inference latency/bandwidth results on GPU are shown.\nThough, since GPU results may require custom CUDA kernels, this is only a minor point.\n\nIn addition, the algorithm relies on some hyperparameters (e.g., those of the truncated Gaussian distribution, as well as the number of inner iterations).\nHyperparameter values used for experiments are listed, but it is unclear how sensitive the accuracy of the algorithm is to those parameters (across different models being compressed).\nMore discussion thereof could be helpful to evaluate ease of practical adoption.\n\n**Clarity**\n\n_Strengths_. The writing is easy to understand, as are the proofs in the appendix.\nThe structure of the paper makes sense.\nThe contributions and benefits of the new algorithm are clearly stated, as well as the technical challenges that it solves.\n\n_Weaknesses_.\nSome experimental details are missing.\nFor instance, for Table 3, it is not clear how the allocations are determined for the \"w/o Rank,\" etc., rows of the table.\nFor the \"Integration with other compression methods\" section, more background on Wanda and SparseGPT, as well as details on how the proposed algorithm is integrated with those methods, are required for readers not familiar with the literature.\n\n**Significance**\n\n_Strengths_.\nThe paper provides a novel method for allocating sparsity in sparse + low-rank LLM compression.\nPast works have approached this problem heuristically, whereas this one uses a principled optimization framework.\nAs such, this work has good potential for adoption in future works in the LLM compression community (including outside sparse + low-rank compression; e.g., it would also be useful in the LLM quantization community).\nAs touched upon. e.g., in the \"allocation visualization\" section, this work provides insights on the inner workings of LLMs (e.g., the low-rank structure of different layers, as well as which layers store more \"dense\" data).\n\n_Weaknesses_.\nN/A.\n\n**Originality**\n\n_Strengths_.\nTo my knowledge, the bilevel optimization framework with a policy gradient outer level for LLM compression is novel.\nAlthough all of the components already exist (e.g., the QR approximation of SVD, and use of RPCA for sparse + low-rank decompositions), they are put together in a novel and principled way.\nThe authors make clear the primary novel contributions of the work.\n\n_Weaknesses_.\nTo those unfamiliar with some of the literature, it is unclear whether some heuristics, e.g., subtracting a control variate and the annealing schedule of the truncated Gaussian support, are novel or standard practice.", "was_revised": true, "final_justification": "The author's rebuttal clarified key points of confusion, namely:\n- Which components were original (particularly, the truncated Gaussian distribution) and which were common in the literature,\n- Sensitivity of the method to hyperparameters and how the hyperparameters were chosen,\n- GPU speedup of the algorithm,\n- Ability of the algorithm to be fully calibration-aware.\n\nI have no outstanding concerns. In the context of the rebuttal and the other reviews, I will maintain my score (5). Overall, I think the bilevel optimization framework is principled and an interesting innovation.", "rebuttal": "Thank you very much for your positive evaluation and thoughtful suggestions. We sincerely appreciate your recognition of our work.\n## Quality W1: Results presentation.\nWe sincerely appreciate your suggestions. Due to space limitation, we present the experimental results under higher sparsity level in our response to reviewer  AQTX's **Table R1**. All other suggestions will be taken in-detail in the revised verision if accepted.\n\n## Quality W2: GPU results.\nThanks for your suggestion. In our initial experiments, we followed the evaluation protocols of OATS and OWL, which focus on CPU speed up, and therefore we did not include GPU results.\nTo address this, we followed the ATP[r1] protocol and conducted GPU inference tests using `nm-vllm`. \nDue to the space limit, please refer to our response to reviewer AQTX's W6&Q1.\n\n## Quality W3: Hyperparameter sensitivity.\nThanks. We would like to clarify following facts to show that our algorithm is not sensitive to these hyperparameters. \n- In fact, we tuned these hyperparameters on Phi-3 mini and applied the same configuration across all other model experiments. Besides, parameters such as the initialization of the truncated Gaussian mean $\\mu$ and the number of inner-loop iterations are consistent with those used in OATS. \n- We also conduct ablation studies on these key hyperparameters, and the results are presented in **Table R1** below. \n#### Table R1. Ablation study of hyperparameters Phi3-mini with 50% prune rate, wikitext2 PPL\n|$\\sigma^2$ | 1e-3 | 5e-3 | 1e-2 | 5e-2 | 1e-1 |\n|---|---|---|---|---|---|\n| PPL | 15.16 | 14.94 | 14.87 | 14.92 | 15.07 |\n|$\\gamma_0$ | **1e-3** | **5e-3** | **1e-2** | **5e-2** | **1e-1** |\n| PPL | 15.22 | 15.01 | 14.93 | 14.87 | 14.99 |\n|$\\gamma_{target}$ | **1e-3** | **5e-3** | **1e-2** | **5e-2** | **1e-1** | \n| PPL | 14.88 | 14.87 | 14.90 | 15.07 | 15.18 |\n\n## Clarity W1: Experimental details.\nWe sincerely apologize for the omissions in the original submission and appreciate your insightful comments. We will ensure that these aspects are addressed thoroughly in the final version.\n\nFirst, regarding Table 3(in submission), our goal is to isolate the effect of different components in our method—namely, sparsity allocation across layers and rank ratio across matrices. Specifically:\n\n* **“w/o Rank”** refers to **not performing learnable rank allocation**; each matrix is assigned a fixed rank ratio, while the layer-wise sparsity is still learned.\n* **“w/o Sparsity”** denotes the setting where **compression rates are fixed uniformly across layers**, and only the rank ratios of matrices are learned.\n* **“w/o Sparsity & Rank”** disables both learnable sparsity and rank allocation. All layers and matrices are compressed with uniform configurations.\n\nSecond, we agree that more background on Wanda and SparseGPT would be beneficial, especially for readers less familiar with the literature.  Specifically, we replaced the inner optimization in our bilevel framework—from an RPCA-based method using QR decomposition—to Wanda and SparseGPT, enabling the outer loop to learn the sparsity allocation for each layer. We will add a paragraph to introduce these methods in the revised version if accepted.\n\n## Originality W1: Heuristics' originality.\nWe apologize for not clarifying the originality and standardness of some of our design choices. We will include following clarifications in the revised version if accepted.\n\n- First, subtracting a zero-mean and highly correlated control variate is a core idea in variance reduction techniques used in statistics and Monte Carlo methods. However, the specific design of variance reduction strategies can vary significantly across different applications. Our contribution lies in tailoring a variance reduction approach specifically for our bilevel optimization framework, which enables more stable policy gradient estimation.\n- Second, the truncated Gaussian prior is our novel design, which effectively mitigates the risk of gradient explosion (see Section 4.2). This issue can arise when using a standard Gaussian distribution, as its gradient contains a $1/\\sigma^2$ term. To ensure convergence, one typically needs to reduce $\\sigma$ to concentrate the sampling distribution, but this can lead to unstable gradients. The truncated version addresses this problem by allowing us to concentrate samples via truncation while keeping $\\sigma$ fixed, thereby maintaining gradient stability.\n- Third, the cubic annealing schedule has been used in other applications, such as control the prune rate during training [r2]. We introduce it to our truncated Gaussian prior and find it works well.\n\n\n## Q1: Hyperparameters and RPCA convergence.\nRegarding hyperparameters' sensitivity, please refer to our response to Quality W3.\n\nUnlike conventional RPCA solvers that rely on accurate decomposition, our bilevel pruning framework only requires approximate gradients. The QR-based estimator strikes a balance between efficiency and performance. Empirical results confirm that this choice does not hinder optimization quality.\n#### Table R2. Relative error(%) of q_proj and o_proj in Phi-3 mini's 1st Layer. The matrix o_proj has relative higher rank, leading to larger approximation error for both SVD and QR.\n|Matrix|Method | 1 |5 |20 |40 |80 | 120|160 |Total Time(s)|\n|---|---|---|---|---|---|---|---|---|---|\n|q_proj|SVD | 3.62| 2.94|2.63 |2.54 |2.49 |2.47 | 2.46 |  126.97  |\n|q_proj|QR |14.7 |7.17| 4.33| 3.50| 3.02|2.83 |2.69 |  0.71s |\n|o_proj|SVD | 14.37 | 11.96 | 10.83 | 10.53 | 10.36 | 10.31 | 10.30 |119.88s  |\n|o_proj|QR  | 15.69 | 13.36 | 11.78 | 11.13 | 10.91 | 10.75 | 10.68 | 0.69s |\n## Q2: Compression time.\nWe acknowledge that, unlike one‑shot methods such as Wanda or SparseGPT, our bilevel framework incurs additional runtime. However, our entire process still completes within a few hours on standard hardware, which is always affordable in downstearm applications.\n\nImportantly, model compression is a one‑time offline cost, where **peak GPU memory usage often outweighs total runtime in practical deployments. Our method uses no more memory than one‑shot approaches but delivers superior accuracy and inference speedups**, making the extra time investment worthwhile.\nThe results of compression time are summarized in **Table R3**. Even **QR**, a simplified version of our method, still outperforms SparseGPT and Wanda under comparable time cost.\n\n#### Table R3. Compression time, Phi-3 mini 60% prune rate\n|Method | Prune Time | Wiki | MMLU | Zero-shot |\n|---|---|---|---|---|\n|SparseGPT | 27min | 51.44 | 37.37 | 54.61 |\n|Wanda | 12min | 52.28 | 36.96 | 55.48 |\n|OATS | 5h | 42.37 | 43.96 | 57.82 |\n|QR | 15min |48.8  | 38.22 | 56.01 |\n|Ours | 4.5h | 32.46 | 52.27 | 59.24 |\n\n## Q3: Sparsity allocation compared with OWL.\nDue to the rebuttal constraint, we are unable to present the full comparison figure. Instead, we report the sparsity allocations of several selected layers in the table below. Compared with OWL, our method exhibits a similar overall trend—allocating more parameters to the early layers, applying stronger compression to the middle-to-late layers, and preserving more parameters in the final layer. This aligns with the general understanding of layer-wise redundancy in large language models. Notably, our method shows larger differences than OWL, more accurately reflecting the redundancy across layers and contributes to better overall performance.\n#### Table R4. Sparsity allocation OWL & Ours Phi-3 mini 60% prune rate\n|Method\\Layer | 0 | 6 | 12 | 18 | 24 | 31| PPL |\n|---|---|---|---|---|---|---| ---|\n|OWL | 0.519 | 0.537 | 0.560 | 0.630 | 0.669 | 0.595 | 35.37|\n|Ours | 0.544 | 0.518 | 0.553 | 0.618 | 0.690 | 0.563 | 32.46 |\n## Q4: Fully calibration-aware Optimization.\nThank you very much for your insightful suggestion regarding the improvement of our algorithm. After careful derivation, we confirm that the calibration-aware loss function you proposed can indeed be formulated and optimized. We present the derivation in detail below.\n\nHere is a brief derivation of the refined optimization problem:\n$$\n\\begin{aligned}\n    & min_ {U,V,S}  \\lVert(W- UV-S)X\\rVert_ F^2,  \\\\\\\\\n    &\\text{s.t.} ~\\text{rank}(UV) \\leq r, ~\\lVert S\\rVert_ 0 \\leq K. \\\\\\\\\n\\end{aligned}\n$$\nWe still use alternating minimization strategy. In each iteration, we update $U$, $V$, and  $S$ as follows:\n$$\n\\begin{cases}\n    U_{t+1} = B_{t+1} V_t^\\top (V_t A V_t^\\top)^{-1},\\\\\\\\\n    V_{t+1} = (U_{t+1}^\\top U_{t+1})^{-1} U_{t+1}^\\top (B_{t+1}A^{-1}),\\\\\\\\\n    S = \\mathcal{P_C}(W-U_{t+1}V_{t+1}),\\\\\n\\end{cases}\n$$\nwhere $A=XX^\\top, B_{t+1}=(W-S_t)XX^\\top$.\nBased on the derivation in the main text, we obtain the following update rules using QR decomposition:\n$$\n\\begin{cases}\n    Q_t R_t = B_{t+1}V_i^\\top,\\\\\\\\\n    U_{t+1} = Q,\\\\\\\\\n    V_{t+1} = Q^\\top(B_{t+1}A^{-1}).\n\\end{cases}\n$$\n\nWe conduct experiment to evaluate this approach and report the results in **Table R5**. It shows that performance of these two methods are comparable. If the paper is accepted, we will include additional experiments in the revised version to evaluate the superiority of calibration-aware method.\n#### Table R5. Experiment on Phi-3 mini 50% prune rate\n| Method  | PPL |MMLU | zero-shot |\n| --- | ---  | --- | --- |\n| Ours | 14.87 | 60.57 | 69.37|\n| Calibration-aware | 14.79 | 60.76  | 69.53 |\n\n### References\n[r1] Lujun Li et al. \"Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models.\"\n\n[r2] Zhu, M. H. et al. \"To Prune, or Not to Prune: Exploring the Efficacy of Pruning for Model Compression.\""}, {"review_id": "5rFR0OdCbI", "reviewer": "Reviewer_nSyU", "summary": "This paper studies the problem of assigning rank and sparsity values to different layers under the sparse + low rank compression setup. They present a two step optimization problem, where the outer step decides the allocation and the inner step does the compression. A policy gradient approximation is used for the outer step, while a QR-based solver is used for the inner loop.", "questions": "See my comments above.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The problem that is studied is interesting from a methodological perspective, though I have doubts how useful it'll be in practice. The authors provide CPU inference results but in reality it is rare to use CPUs for LLM inference. Unless some sort of GPU acceleration can be shown, this work's application will be limited.\n\nAlso, the policy gradient part is not properly motivated, and it looks out of place (in addition to the truncated normal prior). I think the authors should provide more context and justification as to why the stochastic approximation is good in practice.\n\nThe experiments have a basic flaw. This paper, following Wanda and OATS, uses a diagonal approximation to $X^TX$, which is known to be a bad approximation, even though it simplifies the algebra. Therefore, even though the non-uniform allocation of this paper shows improvements over OATS, I'm not convinced the models from the proposed method are actually good. I would like to ask authors to provide comparisons with other sparse + low rank methods, such as https://openreview.net/pdf?id=hyN75SAJTI.", "was_revised": true, "final_justification": "Though I appreciate the authors' efforts to address my concerns, as can be seen from my discussions with them, there are a lot of details that need to be hashed out, new baselines need to be added, etc. Therefore, I think the paper needs to go through a major revision and these issues cannot be addressed via a short rebuttal. Hence, I recommend the paper to be rejected.", "rebuttal": "## W1: GPU acceleration.\nThanks for your suggestion. In our initial experiments, we followed the evaluation protocols of OATS and OWL, which focus on CPU speed up, and therefore we did not include GPU results.\nTo address this, we followed the ATP[r1] protocol and conducted GPU inference tests using `nm-vllm`. The results are presented in **Table R1**.\nCompared with unstructured methods, our approach, Low rank & Sparse, leads to more practically meaningful GPU inference acceleration.\n#### Table R1 GPU inference speedup LLaMA2-13B, on a single 80 GB A100 GPU\n|Prune rate|Prune Type|Method|Throughput (tokens/s)|Speedup|Perplexity |\n|---|---|---|---|---|---|\n| 0% | Dense | - | 47.31 | 1.0 | 7.91 |\n| 40% | Unstructured | Wanda | 68.73 | 1.45 | 8.77 |\n| 40% | Low rank & Sparse | Ours | 79.87 | 1.69 | 8.31 |\n| 50% | Unstructured | Wanda | 83.15 | 1.76 | 10.05 |\n| 50% | Semi-structured | Wanda | 92.36 | 1.95 | 16.53 |\n| 50% | Low rank & Sparse | Ours | 102.45 | 2.17 | 9.23 |\n\n## W2: More justification about Motivation.\nThanks for your suggestion. We provide a detailed explanation of their rationale and implementation below. \n\n**Motivation of policy gradient.** Our bilevel optimization problem in Eqn. (4) of the paper takes the form of \n$$\n\\min_{\\theta \\in \\mathcal{C}} \\Phi(\\theta) = \\mathbb{E}_{(s, \\kappa) \\sim p(\\cdot \\mid \\theta)} \\mathcal{L}(\\mathcal{W}(s, \\kappa)), \\quad \\text{with} \\quad \\mathcal{W}(s, \\kappa) = \\text{RPCA}(\\mathcal{W}, s, \\kappa).\n$$\nIt involves a nested structure where the outer objective depends on the output of a non-differentiable inner procedure (specifically, RPCA decomposition of weight matrices). \n\n\n\nExisting approaches to bilevel optimization (e.g., [r2, r3]) typically rely on **implicit differentiation**. As discussed in the submission's main text, if we were to follow the conventional approach using implicit differentiation, the resulting gradient estimate would involve second-order derivatives and take the following form: \n$$\n\\nabla_{\\theta} \\Phi(\\theta) \\approx \\nabla_{\\theta} \\mathcal{W}(s, \\kappa) \\nabla_{\\mathcal{W}} \\mathcal{L}(\\mathcal{W}(s, \\kappa))\n$$\nHowever, in our case, the RPCA solution $\\mathcal{W}(s, \\kappa)$ does not admit a tractable or closed-form gradient, making $\\nabla_{\\theta} \\mathcal{W}(s, \\kappa)$ cannot be computed directly.  \n\nTo overcome this challenge, we resort to policy gradient estimator, which allow us to estimate $\\nabla_{\\theta} \\Phi(\\theta)$  without calculating $\\nabla_{\\theta} \\mathcal{W}(s, \\kappa)$. The PGE is given by such a simpler form:\n$$\n\\nabla_ {\\theta} \\Phi(\\theta) = \\mathbb{E}_ {(s, \\kappa) \\sim p(\\cdot \\mid \\theta)} \\left[ \\mathcal{L}(\\mathcal{W}(s, \\kappa))\\nabla_ {\\theta} \\log p(s, \\kappa \\mid \\theta) \\right].\n$$\nNote that explicitly computing the expectation above is infeasible, we use the policy gradient $\\mathcal{L}_ {\\mathcal{B}}(\\mathcal{W}(s, \\kappa))\\nabla_{\\theta} \\log p(s, \\kappa \\mid \\theta)$, which is an unbiased estimation of $\\nabla_{\\theta} \\Phi(\\theta)$. **This is why we need and can use this stochastic approximation.**\n\n**Motivation of truncated Gaussion.** The truncated Gaussian prior is our novel design, which effectively mitigates the risk of gradient explosion (see Section 4.2). This issue can arise when using a standard Gaussian distribution, as its gradient contains a $1/\\sigma^2$ term. To ensure convergence, one typically needs to reduce $\\sigma$ to concentrate the sampling distribution, but this can lead to unstable gradients. The truncated version addresses this problem by allowing us to concentrate samples via truncation while keeping $\\sigma$ fixed, thereby maintaining gradient stability.\n\n## W3: The use of diagonal approximation of $X^\\top X$ and More Comparison with HASLLE-free.\nWe thank the reviewer for raising this important concern. \n\n**Reasons of using diagonal approximation.** We would like to discuss the reasons why we use diagonal approximation: \n- We adopt this approximation primarily to ensure a fair comparison. Since OATS, the strong baseline in our study, employs this diagonal approximation, we also follow this choice to isolate the superiority of our proposed collaborative allocation framework.\n\n- The core contribution of our work is the collaborative allocation framework, instead of  the specific approximation to $X^\\top X$. \n\n\n**Effects of $X^\\top X$.** To address the concern, we conduct additional experiments by removing the diagonal approximation used in OATS and directly applying our bilevel framework on the unmodified low-rank sparse structure. The results in **Table R2** below show that our method continues to deliver improvements, indicating that the proposed framework is robust and effective even without relying on this approximation.\n\n#### Table R2 Comparison with OATS on Phi-3 mini 40% prune rate, with and without diag $X^\\top X$. \n|Method | PPL | MMLU | zero-shot | \n|---|---|---|---|\n| OATS (w. diag $X^\\top X$) | 11.50 | 65.84 | 70.04 |\n| Ours (w. diag $X^\\top X$) | 11.03 | 65.92 | 70.56 |\n| OATS (w/o diag $X^\\top X$) | 18.34 | 65.31 | 68.22 |\n| Ours (w/o diag $X^\\top X$) | 16.48 | 65.73 | 69.14 |\n| Ours(w. full Hassian) | 10.83 | 66.07 | 70.61 |\n\n**Comparisons with HASSLE-free.** According to the reviewer’s suggestion, we compare our method with the HASSLE-free approach. Since HASSLE-free adopts a fixed semi-structured sparsity pattern and a predefined rank, it differs from our adaptive sparsity allocation strategy. As a first step, we reproduced their configuration using our QR-based compression under the same fixed sparsity setting. While this led to slightly lower performance, we argue that such a comparison is not entirely fair, as fixed sparsity allocation is not the main contribution of our method. For a more meaningful evaluation, we further compared both methods under the same overall sparsity ratio (50%), using results directly reported in their original paper. In this setting, our method outperforms HASSLE-free, demonstrating the effectiveness of our adaptive allocation strategy.\n\n\n#### Table R3 Comparison with HASSLE-free on LLaMA3-8B 50% prune rate\n|Method | PPL(wiki) |PIQA|ARC-E|ARC-C|\n|---|---|---|---|---|\n|OATS(3:8+LR) | 11.43 | 75.24 | 65.91 | 39.85 |\n|QR(3:8+LR) | 11.57 | 75.06 | 65.32 | 40.87 |\n|HASSLE-free(3:8+LR) |11.36 |75.79 |67.55|41.04|\n|Ours | 10.18 |78.12 |72.88 |45.42 |\n\n### References\n[r1] Lujun Li et al. \"Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models.\"\n\n[r2] Pedregosa. \"Hyperparameter optimization with approximate gradient. \" \n\n[r3] Grazzi et al. \"On the iteration complexity of hypergradient computation.\""}]

Author rebuttal:

Meta-review: This paper presents a novel and well-motivated bilevel optimization framework for Large Language Model (LLM) compression, integrating learning-based sparsity allocation with matrix approximation. The work makes significant technical contributions, and its initial strengths were substantially reinforced by the author's response during the discussion period that addressed most of reviewers concerns with new experiments and analysis.

There are two key reasons for acceptance: (1) Originality and technical innovation: Reviewers consistently mentioned the novelty of the bilevel framework and its core technical contributions. This includes a truncated Gaussian prior-based probabilistic parameterization with a policy gradient estimator for the outer loop, and an efficient QR-based RPCA algorithm for inner loop computations. The theoretical foundation, including convergence analysis, has been well-addressed. (2) Strong empirical performance: The method demonstrates superior performance in perplexity, MMLU, and zero-shot accuracy across various LLMs (Phi-3, Llama-2/3), particularly excelling under high sparsity levels.

There is still one important concern regarding the actual speed-up achievable with a GPU. AC believes this concern remains valid; however, it does not diminish the paper's empirical success in outperforming other unstructured and semi-structured pruning methods. AC requests that the authors accurately and thoroughly document the limitations of speed-up when using GPUs, ensuring that the community is fully informed about potential directions for future improvements.
Decision: Accept (poster)