Large Language Model Compression with Global Rank and Sparsity Optimization
Authors: Changhai Zhou
Qian Qiao
Yuhua Zhou
Yuxin Wu
Shichao Weng
WEIZHONG ZHANG
Cheng Jin
Venue: ICLR 2026 Poster (2026)
Keywords: Low-rank and sparse approximations
Model Compression
Probabilistic Pruning
Global Sparsity-Rank Co-optimization
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We present a two-stage method for compressing LLMs while maintaining performance. It tackles low-rank/sparse matrix interactions and global weight allocation.

Abstract: \begin{abstract}
Low-rank and sparse composite approximation is a natural idea to compress Large Language Models (LLMs). However, such an idea faces two primary challenges that adversely affect the performance of existing methods. The first challenge relates to the interaction and cooperation between low-rank and sparse matrices, while the second involves determining weight allocation across different layers, as redundancy varies considerably among them. To address these challenges, we propose a novel two-stage LLM compression method with the capability of global resource allocation for rank and sparsity. It is noteworthy that the overall optimization space is vast, making comprehensive optimization computationally prohibitive. Therefore, to reduce the optimization space, our first stage utilizes robust principal component analysis to decompose the weight matrices of LLMs into low-rank and sparse components, which span the low dimensional and sparse spaces containing the resultant low-rank and sparse matrices, respectively. In the second stage, we propose a probabilistic global allocation strategy to jointly identify the low-rank and sparse structures within the above two spaces. The appealing feature of our approach is its ability to automatically detect the redundancy across different layers and to manage the interaction between the sparse and low-rank components. Extensive experimental results indicate that our method significantly surpasses state-of-the-art techniques for sparsification and composite approximation.

Published as a conference paper at ICLR 2026
LARGE LANGUAGE MODEL COMPRESSION GLOBAL RANK AND SPARSITY OPTIMIZATION
WITH
Changhai Zhou!
Qian Qiao Yuhua Zhou Yuxin Wu
Shichao Weng
Weizhong Zhang2** Cheng Jin'
CCollege of Computer Science and Artificial Intelligence, Fudan University
Schooo Data Science, Fudan University
Soul AlLaa
Zhejiang University
'OpenWPLab RRenmin University of China {wei zhongzhang, je) ludan.edu..
chzhou25@m
Fudan.edu.co
ABSTRACT
Low-rank and sparse composite approximation is natural idea compress Large Language Models (LLMs). However, such an idea faces two primary challenges that adversely affect the performance of existing methods. The first challenge relates to the interaction and cooperation between low-rank and sparse matrices, while the second involves determining weight allocation across different layers, as redundancy varies considerably among them. To address these challenges, we propose novel two-stage LLM compression method with the capability of global resource allocation for rank and sparsity. It is noteworthy that the optimization space vast, making comprehensive optimization computationally prohibitive. Therefore, to reduce the optimization space, our first stage utilizes robust principal component analysis decompose the weight matrices of LLMs into low-rank and sparse components, which span the low dimensional and sparse spaces containing the resultant low-rank and sparse matrices, respectively. propose a probabilistic global allocation strategy to jointly identify the low-rank and sparse structures within the above two spaces. The appealing feature of our approach is its ability to automatically detect the redundancy across different lay- ers and to manage the interaction between the sparse and low-rank components. Extensive experimental results indicate that our method significantly surpasses state-of-the-arr techniques for sparsification and composite approximation.
the second stage, we
INTRODUCTION
transformer-based large language models (LLMs) (Vaswani et al., 2023; Touvron et al., 2023b; OpenAl et al., 2024) have achieved remarkable progress across natural language processing (NLP), computer vision, and scientific applications. Despite these successes, their massive parameter sizes pose critical challenges: they demand huge storage and memory footprints, incur slow inference, and require substantial computational resources for training. Consequently, model compression (Cheng al., 2020; Wang et al., 2024a: Zhu et al., 2024: Zhou et al., 2025b; 2026) has become an essential line of research for enabling roaldworld LLM deployment under stringent hardware constraints.
Among compression strategies, quantization (Han et al., 2015; Chee al., 2023; Kuzmin et al., 2023) typically retains overall model structure by reducing the precision of weights, thus often pre- serving performance. By contrast, pruning (Liu et al., 2017: Frankle & Carbin, 2019; Sun al., 2024; Frantar & Alistarh, 2023) removes individual weights based certain criteria (e.g., magni- tude or importance scores). Although pruning flexible and can yield substantial parameter savings, it may degrade performance unless combined with additional fine-tuning distillation (Sanh al., 2020; Zhou et al., 2024; 2025a), especially in large-scall scale LLMs that encode extensive linguistic and factual knowledge (Geva al., 2021; Dai et al., 2022; Cui et al., 2025).
To retain more critical information under aggressive compression, researchers have explored "low- rank plus sparse" decompositions (Li al., 2023; Ren & Zhu, 2023; Han et al., 2024). In this
Cooresponding author.

---

Published as a conference paper at ICLR 2026
approach, the weight matrix is decomposed into a low-rank part that captures global correlations and a sparse part that highlights outliers or domain-specific knowledge. However, existing methods often rely on manually set singular-valie thresholds, which can inadvertently discard mediium-sized yet important singular values. Additionally, these methods typically require computationally expensive backpropagation for parameter updates. While there is some interaction between the optimization of the low-rank and sparse components, the two parts are still relatively independent in their update processes, Lastly, due the significant redundancy variations from early layers to deeper ones, how to allocate rank and sparsity across layers in a globally coordinated manner remains unclear.
In this paper, we address these issues via a novel two-stage compression framework tailored to LLMs. First, we apply robust principal component analysis (RPCA) (Candès et al.. 2011) to factor each weight matrix into strictly low-rank and sparse components, thereby reducing the otherwise huge search space into a low-dimensional subspace and sparse subspace. Second, we introduce a probabilistic global resource allocation scheme that jointly determines which singular values in the low-rank component and which nonzero entries in the sparse component should be retained. This is done by assigning Bernoulli probabilities and updating them via policy gradient (Williams, 1992) small calibration set, avoiding heuristic thresholds LLM parameters. Critically, our method automatically detects the differing redundancy levels across layers and manages the interaction between low-rank and sparse parts, ensuring that vital parameters are kept while redundant ones are pruned away. We summarize our main contributions as follows:
backpropagation
the original
We propose two-stage LLM compression approach that first uses RPCA to produce low- rank and sparse subspaces, then employs a Bernoulli-bast global resource allocation for rank and sparsity selection.
Our framework eliminates the need for manual thresholds or layerwise iterative backprop- agation, offering a training-fire scheme that adapts automatically to various layers' redun- dancy characteristics.
Extensive experiments show that our method outperforms existing sparsification and com- posite approximation baselines under multiple compression ratios, highlighting its effec- tiveness and robustness.
We provide a detailed review and discussion of (related work in Appendix B.
2
METHOD
2.1 THEORETICAL BACKGROUND AND MOTIVATION
Low-rank approximation is fundamental technique in matrix theory, widely used to reduce the parameter count in neural networks while preserving model performance. In LLMs, weight matrices typically high-dimensional and dense. By approximating weight matrix W E Rmxx with rank R < min(m, n) using a truncated SVD, one can write
(I) where UR and VR contain the top R left and right singular vectors, and ER is the diagonal matrix of the largest singular values. This factorization reduces the parameter count from m n to (m + n) x R, and breaks large matrix multiplication into smaller ones, leading to significant efficiency gains.
W ~ URLRVR
Despite these benefits, low-rank approximation alone may be insufficient for LLM compression, es- pecially when the singular values do not decay sharply. For example, Figure in Appendix shows the singular value spectra of two representative layers (Layer 0 and Layer 31) from a Transformer model, comparing the original weight matrix and its low-rank component after RPCA processing. The dashed lines (original matrices) indicate that certain modules in the same Transformer block (e.g., an attention head vs. a feed-forward network) can exhibit similar spectral shapes; yet across different layers, the redundancy patterns vary considerably. Consequently, imposing the same target rank R across all layers may prune too aggressively in some cases and insufficiently in others. This observation motivates more flexible approach that can adapt the compression ratio per layer.
Recent studies have explored combining low-rank and sparse representations to enhance compres- sion. For instance, LoSparse (Li al., 2023) first applies SVD on W to obtain rank- R approx-
2

---

Published as a conference paper at ICLR 2026
Policy Gradient
Probability Scores
m,
Bernquli Samplng
Pruning
W
OuTput
Sporse Matrix Pruning
LOSS
diag(o m,)
Recon- struction
SVD
U
V
Robust FGA
Element-wise Multiplication
Probability Scores
diogg d
Leamaale
Sompling
X
X Tokens
X3
X4 Target
Operation Flow
Input
Low-rank Matrix Pruning
Policy Gradient
Figure 1: Overview of our proposed compression method. The weight matrix W is decomposed into a low-rank component L and sparse component S using RPCA. Both components pruned through Bernoulli sampling guided by learned probability scores, optimized via policy gradient. The low-rank component is further factorized into U' and V to reduce the number of model parameters.
imation, then prunes the residual W UREVVT to form a sparse matrix. In practice, one must still decide the singular value cutoff (or target rank) and the sparsity ratio for the residual. Often, additional fine-tuning is performed on the low-rank part to recover lost performance, or iterative pruning is applied the sparse part (Molchanov al., 2019), which can be computationally expen- sive. A major limitation of these approaches is the reliance on manually chosen thresholds for both singular values and residual pruning. They also lack clear mechanism to coordinate how much rank vs. sparsity each layer should receive, since different layers and modules may have different redundancy characteristics. Furthermore, when both the low-rank and sparse matrices require joint fine-tuning, the memory consumption can become large, potentially exceeding the budget.
We begin by formulating the global resource allocation objective of compressing LLM weights under parameter budget ($ 2.2). We then describe our proposed approach ($ 2.3), which first uses RPCA to decompose each weight matrix into low-rank and sparse components, and subsequently prunes these components in probabilistic manner, without heuristic thresholds backpropagation through the original LLM weights. Additional theoretical analysis can be found in the Appendix E.
2.2 PROBLEM FORMULATION
Suppose we have L layers an LLM, each containing weight matrices (4 We seek com- pressed matrices (W()) such that the total parameter count does not exceed budget K, while minimizing a loss ((W) measured on small calibration set DD Formally,
Σ (((W:z),)) v).
min (WO)
subject to ParamCount(1 ((W)) < K,
where x)) is the LLM's forward pass given the compressed weights, and ParamCount(( measures how many parameters retained. Directly pruning each individual weight is intractable for very large matrices. To address this, we propose to:
Decompose each (1)) via RPCA to obtain low-rank matrix and a sparse matrix S, reducing the search space to "global rank directions" plus "sparse outliers."

---

Published as a conference paper at ICLR 2026
Probabilistically prune both components under the budget K by learning Bernoulli reten- tion probabilities through policy gradient on small calibration set.
2.3 PROPOSED APPROACH: CAP
As illustrated Figure I, our proposed method, CAP, follows two-stage process. The role of Stage is to decompose weights into relatively low-rank matrix L and sparse matrix S. reducing the parameter space manageable candidates. Stage then allocates the parameter budget over these candidates to achieve the target compression ratio while preserving model performance. This prin- cipled decomposition followed by budget-aware selection avoids heuristic thresholds and expensive fine-tuning. In the following sections, we provide detailed explanations of our algorithm.
2.3.1 STAGE 1: PRINCIPLED DECOMPOSITION VIA RPCA
The first stage of our method is not designed to achieve a target compression ratio directly. In- stead, its purpose to perform a principled decomposition of each weight matrix, transforming the complex problem of pruning individual weights into more structured one. By separating a weight matrix Rmxn into a low-rank component that captures global structure and sparse compo- nent S that captures local, salient features, we establish high-quality candidate pool for subsequent compression. We achieve this through RPCA, which formulates the decomposition as convex optimization problem:
min ILI-
+A
|S||
subject to W = -
(3)
L,S
Low-rank constraint
Spursity constraint
The choice of this objective theoretically motivated. The nuclear norm |[]|| is the tightest convex relaxation of the rank function, making it the most effective convex proxy for minimizing rank. Similarly, the & norm ||||| is the standard convex relaxation for the non-convex to norm (sparsity), which effectively identifies significant, sparse outliers. Thus, this framework provides principled and globally optimal separation of W into its underlying low-rank and sparse structures.
Crucially, the hyperparameter À in the RPCA objective governs the nature of this decomposition, not the final compression rate. Attempting to control sparsity by simply tuning leads to unpredictable changes in the rank of L and often results in poor-quality decompositions, point we analyze in detail in Appendix H. Therefore, this stage focuses solely on creating an optimal candidate pool for the subsequent budget-aware pruning. We solve Eq. equation using the efficient Alternating Direction Method of Multipliers (ADMM) (Lin et al., 2010). The updates are as follows:
L arg min |||
w
Sy + p 1 Y
2
Sk+1 = arg min
A|[S|| + |
W L - S + 1 Y || F
(5)
S
2
Y Y₁ (W- La+1 S
The L-update employs Singular Value Thresholding (SVT) (Cai et al., 2008):
L6+1 U (
(7)
where T is the SVD of W Sx + μ YY,, with singular value shrinkage shrink, (a.) max(a, t, 0). The S-update applies elementwise soft-thresholding:
[Stt1k = shrinks, - Lx+1 + YYlyy)
(8)
This alternating optimization progressively separates the weight matrix into a low-dimensional sub- space capturing directional patterns (L) and a sparse subspace containing localized refinements (S), establishing the foundation for subsequent global resource allocation.
2.3.2 STAGE 2: LEARNABLE PROBABI.ISTIC PRUNING
While the RPCA decomposition in Stage 1 provides a high-quality separation of components, it does not enforce a specific parameter budget. The second stage directly addresses this by performing a global, budget-aware selection from the candidate pools (L and S) generated previously. We decide
4

---

Published as a conference paper at ICLR 2026
which rank-1 components in L and which non-zero entries in S to keep, to meet user-defined parameter budget K while minimizing task performance degradation.
The total parameter budget, K, is user-defined hyperparameter (e.g.. 50% of the original model's parameters). Each retained singular value Ø; from L requires storing its corresponding singular vectors u, T and Vi R", contributing (m + n) parameters. Each retained non-zero entry of S contributes one parameter. We introduce Bernoulli random variables to model the retention decision for each potential parameter:
Ma, Bernoull(sa.,) ms
Bernoull((ssy),
where So, then
E [0, 1] and SS E [0, 1] are learned retention probabilitics. The compressed matrix is
W = -UU diag a @ m,) V + S mg,
subject
Σ; S0, (m n) Σig 88,5 I K respect the total parameter budget.
Learning probabilities via policy gradient. We minimize the expected loss small calibration set D:
min Em--p(mss)
[(W)]
(10)
where (877 88,,) } and p(m s) is the product of Bernoulli distributions. We employ REItFORCE-style (Williams, 1992) policy gradient:
S. Em [[()]
=
Em [(W) V., log p(m sw)
(11)
For a Bernoulli variable my Bernoulli(sk..
m7 8k
V.. p(m= I ss)) =
Sk (1 5k) e
with a small to avoid division by zero. To reduce variance, we maintain a moving average baseline 5 (Zhao et al., 2011);
5 t 38-8 (1- ((W),
(12)
and update each via
- 8k (L(W) - 8) V., logp(mm 8t). After each gradient step, we project s back onto {s 1 K, 8 < 11.
(13)
Thresholding masks and final factorization. The policy gradient optimization yields a set of probabilities {sx) that reflect the learned importance of each parameter for minimizing the task loss. To obtain the final compressed model that strictly adheres to the budget K, we perform a determin- istic selection. We treat the probabilities sı as importance scores for their corresponding parameters (singular values or sparse entries). All potential parameters are ranked globally according to these scores. We select the top-K K parameters keep, generating the final binary masks mg:
11, if parameter is among the top-K scored parameters, otherwise.
m& (o
(14)
We note that comparing 8 across different parameter types valid because the learned probabil- ity acts unified proxy for the "utility-toccost" ratio. During optimization, the policy gradient inherently accounts for the parameter's contribution to loss reduction relative to its existence in the model. Thus, sorting by sk provides globally consistent ranking for resource allocation.
This final step ensures the parameter budget met precisely. The compressed weight matrix is reconstructed using these binary masks in Eq. equation To enhance efficiency, the resulting low- rank component is factorized into smaller matrices. The compressed U' and V are computed as:
U' [Vojuı, Volua,
(15)
V [Voiv1, V0222,
(16)
where r' is the number of retained singular values (i.e., where may 1). The final compressed weight matrix is then:
W U' (V') + + ms. This factorization reduces both storage and computational cost during inference.
(17)
5

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison with unstructured pruning methods 50% compression. We report average zero-shot accuracy (%) across eight tasks and WikiText perplexity (lower is better).
Method
Comprescion
Phi-3 Mini
Phi-3 Medium
Zero-shot Accuracy
LLaMA-38B
LLaMA-3 70B
Phi-3 Mini
Phi-3 Medium
WikiText Perplesity
LLaMA-3 KB
LLaMA-3 70B
70.79
Unjorm Sporsity Meolady
SporseCPPT
67.5
74.63
64.66
73.17
11.95
5.27
75:
3:38
5.38
DSNOT
74.29
50%
71.12
62.74
72.91
16.68
9.96
5.58
OATS
60%
70.04
74,66
68.63
74.83
7,70
9.24
3.68
50%
73.39
65.71
73.30
15.18
10.87
4.78
Lapersise Allocation Menhedy (Based Wireda)
OWL
60%
69.32
73.35
67.58 63.95
74.43
12.28
8.30
9.38
3:95
50%
71.38
73:25
12.18
5.25
30%
73.50
9.18
3.15
AlphaPruning
40%
74.55
9.48
71.53
16.72
12.05
Mediod
CAP
40%
74.78
73.45
9.86
3.52
66.83
10.35
4.45
2.4 DISCUSSION
We propose CAP, a two-stage compression framework for large language models. Stage 1: RPCA Decomposition- The weight matrix is split into low-rank and sparse parts, preserving global struc- ture while isolating local anomalies and sharply reducing the search space for later optimization. This step is cast as a convex program (nuclear norm + L1 norm), guarantceing a globally optimal separation for the decomposition objective. Stage 2: Bernoulli Mask Optimization--Usinn calibration set, an unbiased volicy-gradies method learns the retention probabilities for the low-rank and sparse components, automatically detecting and pruning redundancy across layers.
small
Remark. While Stage solves a convex problem, Stage addresses the discrete budget allocation problem via policy gradient, which serves heuristic optimizer. It does guarantee global optimum for the non-convex pruning objective but empirically finds effective allocation strategies by leveraging the high-quality subspaces identified in Stage 1. This two-step design balances theoretical principledness in decomposition with practical flexibility in resource allocation.
EXPERIMENTS
In this section, we first introduce the experimental setup. Subsequently, present the main exper- imental results, extensions to modern instruction-tunel models, and detailed ablation studies. We further provide in-depth analyses on inference throughput and calibration robustness. Due to space constraints, detailed results for Llama- 1/2 are provided in Appendix I, and thorough analyses of throughput performance and computational resource consumption of our proposed CAP method are presented in Appendix J.
Models and Evaluation. We evaluate our proposed CAP method on a comprehensive set of widely adopted large language models across different architectures and scales. Our evaluation in- cludes the LLaMA family: LLaMA-1 (Touvron et al., 2023a) (7B, 13B, 30B), LLaMA-2 (Touvron al., 2023b) (7B, 13B), and LLaMA-3 (Dubey et al., 2024) (8B, 70B); Modern Instruction-Tunel models: LLaMA-3.1-888 1-B and Qwen2.5-7B (Yang et al., 2024); the OPT series (Zhang et al., 2022) (1.3B, 2.7B, 6.7B, 13B); the Phi-3 family (Abdin et al., 2024) including Phi-3 Mini (3.8B) and Phi-3 Medium (14B); and BERT-base (Devlin et al., 2019). To assess the performance of the compressed models, we conduct experiments on zero-shot tasks and language modeling. We perform an exten- sive evaluation of the zero-shot capabilities of pruned models across eight standard commonsense benchmark datasets: GLUE (Wang et al., 2019a), PIQA (Bisk et al., 2020), BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), OpenBookQA (Mi-

---

Published as a conference paper at ICLR 2026
Table 2: Performance on Modern LLMs (50% Sparsity). Left: LLaMA-3.1-BB-Insttee on reasoning and long-context tasks. Right: Qwen2.5-7B on diverse benchmarks.
LLMMA-311-B---m --nstructt
Qwen2.5-7B
Method
GSM8K (8-shot, %)
LongBench-v2
WikiText (PPL)
Task
SparseGPT (50%)
Wanda (50%)
CAP
Dense
(Avg.
(50%)
(0%)
Dense
84.5
30.4
6.01
NarrativeQA
4.95
16.30
18.52
31,9
Wanda (50%)
45.6
25.1
7.26
GovReport
33.52
32.13
34.05
53.4
CAP (50%)
56.8
27.2
6.61
Lee (Code)
39.70
45.14
46.92
58.1
Improvement
+11.2
+2.1
-0.65
TriviaQA
89.35
88.70
89.85
92.5
haylov al., 2018), and the ARC Easy and ARC Challenge tasks (Clark et al., 2018). For lan- guage modeling evaluation, we measure perplexity on the held-out WikiText (Merity et al.. 2016) validation set (lm-eval-harness). For modern models, we additionally evaluate Chain-of-Thought reasoning (GSM8K (Bai et al., 2025)) and LongBench-v2 (Cobbe et al., 2021).
Implementation Details. We utilize PyTorch 2.3.0, Transformers 4. 4.28.0, CUDA 12.1 on NVIDIA A100 GPUs under Ubuntu. To ensure fair comparison, we use 128 sequences with context length sampled from the C4 training set (Raffel et al., 2020) as calibration data. For policy gradient estima- tion, we set iterations to 3, sliding window size to 5, and learning rate to 0.05. The À parameter for RPCA decomposition is set according to the established formulation À = 1/v max(m, n). where m and represent the dimensions of the data matrix. Further ablation studies the À setting can be found in Appendix H.
Baselines. We compare approach with several compression techniques: SparseGPT (Frantar & Alistarh, 2023) a second-order pruning method for LLMs that solves a layer-wise reconstruc- tion problem. WANDA (Sun al., 2024) prunes weights based on their estimated importance using activation statistics. OATS (Zhang & Papyan, 2024) performs optimal sparsity allocation across transformer layers using second-order information. OWL (Yin et al., 2023) and AlphaPruning (Lu et al., 2024) are layer-wise allocation methods that optimize sparsity distribution. SLiM (Mozaffari al., 2024) combines low-rank approximation with sparsity and quantization, featuring probabilis- tic quantization егтоr fitting. LPAF (Ren & Zhu, 2023) first applies first-order unstructured pruning obtain low-rank sparse model. Then, sparsity-aware SVD used decompose the sparse ma- trices into low-rank form AB. Finally, mixed-rank fine-tuning is used to retrain AB. We also compare against dense SVD-based methods SVD-LLM2 (Wang et al., 2025c), Dobi-SVD (Wang et al., 2025b), Basis Sharing (Wang et al., 2025a) and structured pruning method LoSparse (Li al., 2023).
3.1 COMPARISON WITH UNSTRUCTURED PRUNING METHODS
We compare CAP with recent unstructured pruning methods across multiple large language models. Table 1 presents comprehensive comparison including both uniform sparsity methods (SparseGPT, Wanda, DSNOT, OATS) and layerwise allocation methods (OWL., AlphaPruning) at 30%, 40%, and 50% compression ratios. Note that OWL and AlphaPruning are layerwise allocation methods that optimize sparsity distribution across layers, and we implement them using Wanda the base pruning method for fair comparison. CAP consistently achieves competitive or superior performance across different model architectures and sizes.
3.2 EXTENSION TO MODERN INSTRUCTION-TUNES TUNED MODELS
To validate CAP's effectiveness on the latest generation of models, we evaluated LLaMA-3.1-8B. Instruct and Qwen2.5-7B at 50% sparsity. We report Chain-of-Thouge reasoning (GSM8K) and long-context understanding (LongBench-v2) where standard pruning methods struggle. As shown in Table 2, CAP significantly outperforms Wanda on challenging tasks. For LLaMA-3.1 1--MA-,.-IInstaae Insstruct, CAP recovers +11.2% accuracy on GSM8K, suggesting that the preserved low-rank backbone is crucial for maintaining the precise reasoning circuits disrupted by unstructured pruning.

---

Published as a conference paper at ICLR 2026
Table Comparison at 50% unstructured sparsity. Zero-shot accuracy (%) on representative mod- els. LoRA variants: Naive-LoRA uses basic enror compensation; SLiM-LoRA incorporates weight salience; SLiM-LoRA additionally quantizes the adapter.
OPT
LLaMA-2
Method
Quantization
1.3B
2.7B
6.7B
13B
13B
Dense
43.4
45.5
48.3
48.7
56.6
60.8
Magnitude
Group AbsMax
32.1
39.9
36.4
32.3
47.0
51.0
SparscGPT
OPTQ
38.7
43.4
47.0
47.4
51.1
55.9
Wanda
OPTQ
41.0
42.9
46.5
46.8
53.6
JSQ
JSQ
38,9
35.5
42.8
30.7
52.3
57.0
L2QER
Group AbsMax
38.4
41.3
45.1
OOM
50.6
OOM
Naive-LoRA
Quantization W
40.4
43.4
46.6
47.3
51.5
55.3
SLiM-LoRA
Quantization W
41.9
43.5
47.1
48.0
54.3
57.9
SLiM-LoRAQ
Quantization
41.7
43.6
47.2
47.9
54.2
57.3
CAP (Ours)
OPTQ
41.7
44.8
48.2
48.3
55.1
59.2
3.3
COMPARISON WITH JOINT COMPRESSION METHODS
Since methods like LoSparse are based on structured pruning and require extensive retraining, we compare CAP with SLiM, ofte-of-thheart method that jointly applies quantization, sparsity, and low-rank approximation. We also include comparisons with other joint compression approaches including JSQ (Guo et al., 2024), a joint sparsity and quantization method that optimizes sparsity and quantization parameters simultancously, and L2QER (Zhang et al., 2024a), which combines low-rank decomposition, quantization, and sparsity in sequential manner.
While both SLiM and CAP structurally combine low-rank and sparse components, their technical approaches differ fundamentally: SLiM primarily focuses using low-rank decomposition fit quantization errors through probabilistic reformulation and numerical integration to find optimal quantization parameters, whereas CAP focuses the synergy between low-rank and sparse decom- position through RPCA, where the low-rank component emerges from joint optimization rather than serving error fitting tool. Table 3 presents the comparison representative models 50% unstructured sparsity. The results demonstrate that CAP consistently outperforms existing joint compression methods across different model sizes and architectures.
3.4 COMPREHENSIVE COMPARISON GLUE TASKS
We evaluate CAP on downstream tasks using the GLUE benchmark with BERT-base Table 4 com- pares CAP against various compression paradigms including pre-traine pre-training distillation, task specific distillation, structured pruning, and matrix factorization methods.
CAP achieves competitive or superior performance across most GLUE tasks and compression ra- tios. Notably, CAP consistently outperforms methods without fine-tuning and achieves comparable results to fine-tuned methods like LPAF while using only the RPCA decomposition without addi- tional task-specific fine-tuning.
3.5 COMPARISON WITH STRUCTURED AND PURE LOW-RANK METHODSS
We additionally compare CAP against (1) pure low-rank decomposition methods, including SVD- LLM v2 (Wang et al., 2025c), Dobi-SVD (Wang et al., 2025b), and Basis Sharing (Wang et al., 2025a), which utilize dense or shared SVD structures; and (2) LoSparse (Li et al., 2023), a repre- sentative structured method requiring iterative fine-tuning.
As shown in Table (Left), CAP yields significantly better perplexity than pure SVD methods (e.g., 5.85 6.08 at 20% ratio), confirming the necessity of the sparse component. Table 5 (Right) shows that while our training-free CAP is competitive, applying fine-tuning (CAP w/ FT) significantly outperforms LoSparse, validating RPCA as superior initialization.
8

---

Published as a conference paper at ICLR 2026
Table Results GLUE tasks under different parameter budgets. We show accuracy (%) for RTE, MRPC, SST-2, QNLI, MNLI and FI score (%) for QQP.
Method
RTE
MRPC 25%
SST-2
QQP
QNLI 25%
MNLI
50%
25%
16%
50%
16%
50%
255
16%
50%
25%
16%
5056
16%
50%
25%
16%
Pre-truining Distillation
DistIBERT
65.0
61.0
56.3
85.8
77.0
72.5
90.0
$3.9
$6.4
90.8
89.4
83.0
86.0
83.8
81.6
81.7
76.4
71.3
ThyyBRK
67.7
67.2
64.6
86.3
85.3
78.2
92.3
89.8
88.0
90.5
90.0
88.7
89.9
87.7
84.5
83.1
80.6
77.4
Tpek-sp Distillation
PKD
65.5
59.2
53.8
81.9
76.2
71.3
91.3
88.1
57.2
88.4
88.5
87.5
88.4
82.7
78.0
81.3
75.7
72.7
Theseus
65.6
62.1
58.8
86.2
71.2
72.8
91.5
58.6
86.1
90.9
89.6
89.0
88.2
833.2
78.0
82.3
76.4
73.5
CKD
67.3
66.5
60.8
86.0
81.1
76.6
91.2
90.0
88.7
90.5
88.7
89.5
90.4
86.4
81.9
83.5
79.0
76.8
MeraDisttlll
61.0
77.3
90.4
Structured Pruning
ISP
66.4
65.0
63.9
86.1
83.6
82.8
90.4
89.4
89.9
90.5
88.7
67.2
90.5
88.7
67.2
83.2
81.9
80.8
BPbybrid
66.4
64.3
63.9
84.1
81.1
78.3
91.0
88.7
86.9
91.8
89.3
89.1
90.7
88.
86.2
83.0
80.1
78.0
CaFi
69.0
66.4
66.4
84.6
84.3
83.4
91.6
89.7
89.2
90.1
89.0
88.9
90.2
38.8
87.6
83.5
80.8
80.5
Matrix Factorization
SVD,
62.1
60.3
55.6
79.9
77.0
70.1
89.4
86.9
85.3
90.0
87.9
87.1
90.1
80.9
81.8
78.0
74.6
LPAF
62.8
68.0
67.9
86.8
85.5
86.0
92.0
90.0
91.5
90.4
90.1
91.1
89.3
88.6
84.8
84.8
82.6
77.6
Low-rank plus Sparse
CAP (Ours)
69.1
67.8
66.5
86.2
86.2
85.8
92.3
90.8
91.9
90.8
90.5
90.8
88.8
85.1
82.8
BERT-buse
69.2
86.4
92.7
91.5
91.4
84.6
Table 5: Left: Comparison with Post-Training SVD Methods on LLaMA-7B (WikiText-2 PPL). Right: Comparison with LoSparse on DeBERTa-V3-bas (20% Retention).
(a) SVD Methods Comparison (PPL)
(b) LoSparse Comparison (Accuracy)
Method
Format
20%
40%
60% %
Method
FT?
MNLI
QNLI
SVD-LLM v2
Dense LR
7.12
10.34
14.71
LoSparse
Yes
83.8
88.6
Dobi-SVD
Dense LR
6.08
8.48
15.62
CAP (Ours)
No
78.2
83.0
Basis Sharing
Shared LR
7.74
12.39
28.72
CAP (Ours)
Yes
86.8
91.8
CAP (Ours)
LR+Sparse
5,85
6.39
3.6
IN-DEPTII ANALYSIS: EFFICIENCY AND ROBUSTNESS
Table Left: Inference Efficiency on LLaMA-3.1-BB (A100-80G). Right: Calibration Set Robust- ness 1L-MA-3.1-8B, 50% Sparsity).
Calib. Set
Domain
PPL
Acc.
Method
S-Sparsity
Latency
Throughput
C4 (Default)
General
7.05
76.8
Wanda (50%)
50%
6.28 ms
163.4 tok/s
WikiText-2 GitHub Code
Formal
6.88
75.4
CAP (50% 9)
~ 85 %
5.80 ms
176.5 tok/s
Code
7.18
76.1
We investigate the practical inference efficiency and the robustness of calibration process. Ta- ble 6 (Left) shows that CAP achieves higher throughput (176.5 tok/s) than Wanda (163.4 tok/s). This because CAP's sparse component falls into the high-sparsity regime 85%), where SpMM is highly efficient compared to the uniform 50% sparsity of standard pruning methods. Table 6 (Right) analyzes the impact of different calibration datasets. While using in-domain formal text (WikiText-2) yields the lowest perplexity (6.88), the diverse C4 dataset provides the best general- ization capability with the highest zero-shot accuracy (76.8%). Notably, the model demonstrates strong robustness: even when calibrated on distinct domain like GitHub Code, the accuracy mains competitive 17).
3.7
ABLATION STUDIES
To gain deeper insights into the behavior of our compression method, we conduct ablation stud- ies focusing key aspects: (i) the distribution of different matrix ranks after compression is

---

Published as a conference paper at ICLR 2026
Effect of RPCA Berations on Modol Porformance
Oftege veluon belkor the threakate are oven)
64122
CA-
15:14
MAY
2189
Syeetty
Arg Seer
HLA
Decutoea
24 (Rak (xz)
Thrihhedd ctrg.
x
(a)
(b)
Figure 2: (a) Effect of RPCA iterations on model performance. Here, "Avg. Rank" denotes the aver- age rank of the low-rank component, and "SSarsitt" represents the sparsity of the sparse component. (b) Effect of heuristic threshold-band based pruning on model performance. Singular values below the threshold are set to zero.
between 200 and 800; and (ii) the stability of our method when pruning is applied sequentially layer-by-layer. Detailed analysis and experimental results are provided in Appendix K.
Robustness and Rapid Convergence of RPCA Decomposition We investigated the effect of RPCA iterations the performance of the LLaMA2-7B model assess the robustness of the de- composition quality. Figure 2a shows that only a few RPCA iterations are needed to achieve an effective decomposition, providing solid and stable starting point for subsequent pruning. This rapid convergence demonstrates the robustness of the RPCA stage, as it consistently produces a high-quality separation of global patterns (low-rank component) and local anomalies (sparse com- ponent) across different layers with minimal computational overhead. Additionally, the "Avg. Er- ror" column represents the average approximation error for each matrix, offering insight into the model's 's tolerance to error. Similar to findings in the quantization field, large models exhibit ro- bustness to approximation errors. The fact that performance even surpasses the original model after decomposition further underscores the effectiveness of RPCA in identifying and isolating redundant parameters, thereby enhancing the input quality for the subsequent global optimization stage.
Necessity of Global Resource Allocation The limitations of heuristic, post-decomposition prun- ing underscore the importance of our proposed global optimization components (policy gradient with Bernoulli sampling). We conducted experiments using uniform threshold-based approach ap- plied to the RPCA output. In this method, we prune the low-rank component L by setting singular values below specific threshold to and remove low-magnitude elements from the sparse com- ponent S without applying our probabilistic masking or additional optimization. Figure 2b summa- rizes the results for LLaMM2-7B, which indicate that both components are indispensable: retaining only one leads to a performance collapse. This clear failure of simple thresholding strategies validates our core design choice: the necessity of learned, global resource allocation strategy. Unlike rigid heuristics, policy gradient optimization and Bernoulli sampling mechanism determine the rank and sparsity allocation across layers based on their redundancy characteristics, which crucial for maintaining model performance under compression.
4 CONCLUSION
In this work, we introduce CAP, a novel two-stage compression framework for large language mod- els. First, CAP applies Robust Principal Component Analysis (RPCA) to decompose weight ma- trices into low-rank and sparse subspaces, drastically reducing the optimization space. Second, it employs a global resource allocation strategy via policy gradients to jointly optimize parameter re- tention under a strict budget. Unlike heuristic-baset approaches, CAP adaptively allocates rank and sparsity in a training-fre manner, avoiding expensive backpropagation on the original weights. Extensive evaluations models like LLaMA-3.1 and Qwen2.5 demonstrate that CAP significantly outperforms SOTA baselines in reasoning and long-context tasks.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work was supported by the National Nature Science Foundation of China (62472097), Shang- hai Municipal Science and Technology Commission (Grant No.25511102200), and Fudan Kun- Peng&Ascend Center of Cultivation. The computations in this research were performed on the CFFF platform of Fudan University.
REPRODUCIBILITY STATEMENT
This statement presents a comprehensive report detailing the reproduction process for our RPCA- based model compression methodology, incorporating policy gradient optimization. The implemen- tation builds upon WANDA's code base and integrates components from additional open-source libraries, to which extend our gratitude.
IMPLEMENTATION OVERVIEW
The proposed algorithm is implemented using PyTorch and Hugging Face's Transformers library. The core components of the implementation include:
RPCA Decomposition: Each weight matrix W from the pre-trained model decomposed into low-rank matrix L and sparse matrix S using Robust Principal Component Analysis (RPCA). This decomposition captures global structure in and local anomalies in S.
Probabilistic Pruning: Bernoulli random variables are introduced determine the reten- tion of singular values in L and specific elements in S. Retention probabilities are treated as trainable parameters.
Policy Gradient Optimization: policy gradient framework optimizes the retention prob- abilities by minimizing the expected loss over a calibration dataset, subject to a parameter budget constraint.
Following optimization, compressed weight matrices recon- structed using the retained components. Low-rank matrices are further factorized to en- hance computational efficiency during inference.
Model Reconstruction:
CODE STRUCTURE
The implementation is organized into three main components:
main, py: The primary entry point for the pruning process, handling model loading, gument parsing, and execution.
1ib Lbb/prune rl.py py: Contains the RPCA decomposition, policy gradient optimization routines, and model reconstruction logic.
main. A shell script to streamline the pruning execution process with preset argu- ments.
RUNNING THE PRUNING PROCESS
To reproduce our results, follow these steps:
1. Environment Setup:
Ensure Python 3.8 later is installed.
Install the necessary dependencies:
pip install torch
transformers \
numpy
todm
matplotlib \

---

Published as a conference paper at ICLR 2026
ison argparse
2. Execution:
Use the provided shell script main. sh to execute the pruning process with preset configurations: bash main. sh
The script handles model selection, pruning method, RPCA parameters, policy gradi- ent settings, and output configurations.
KEY IMPLEMENTATION DETAILS
The implementation builds upon WANDA's pruning framework, modified to incorporate RPCA decomposition and policy gradient optimization.
Code Base:
RPCA Implementation: An augmented Lagrange multiplier method used solve the RPCA optimization problem. This separates the weight matrix into L and S, capturing essential patterns and anomalies, respectively.
Bernoulli Masks: For each singular value in L and each element in S, Bernoulli ran- dom variable determines its retention. Retention probabilities are initialized uniformly and optimized iteratively.
Policy Gradient Optimization: Retention probabilities refined using policy gradient approach. The gradients the expected loss with respect to the probabilities are estimated and used update the masks, with variance reduced via moving average baseline.
Following optimization, probabilities are thresholded to generate binary masks. The compressed model reconstructed, and low-rank matrices are further decomposed into U' and V" for inference efficiency.
Model Reconstruction:
RESULTS
Using the aforementioned process, we successfully compressed the LLaMA-2-7B model to achieve 50% compression rate while maintaining performance. Perplexity was monitored after processing each layer to evaluate the model's performance.
CONCLUSION
This reproduction report outlines the implementation and procedural details for replicating our RPCA-based compression method with policy gradient optimization. The provided code base, built upon WANDA, ensures reproducibility and offers robust foundation for advancing model com- pression research.
REFERENCES
Marah Abdin, Jyoti Aneja, and Hany Awadalla al. Phi-3 technical report: A highly capable lan- guage model locally on your phone, 2024. URL https: wawpaaiivvv/oo/ wrw.ii.oorrro.oooo abs//aa///aa abs.///4. 14219 9.
Yushi Bai, Shangging Tu. Jiajie Zhang, Hao Peng, Xiaozhi Wang. Xin Lv. Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, et Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks, arXiv preprint wrXi::2502.12082, 2025. Also appeared in ACL 2025.
Jonathan Baxter and Peter I.. Bartlett. Infinite-horizon policy-gradient estimation. Journal of Artifi- cial Intelligence Research, 15:319-350, 2001. doi: 10.1613/jair.66.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
12

---

Published as a conference paper at ICLR 2026
Jian-Feng Cai, Emmanuel J. Candes, and Zuowei Shen. A singular value thresholding algorithm for matrix completion, 2008. URL https::/t///// http://axxivv.ooo org/ abs 00810.000... 3286.
Emmanuel J. Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM), 58(3):1-37, 20
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M. De Sa. Quip: 2-bit quantization of large language models with guarantees. In Advances in Neural Information Processing Systems, 2023.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. survey of model compression and acceleration for deep neural networks, 2020. URL https: / //txxaaxvvv..aa//.aa/aaa www. hts//bbs/117 /110/ 09282.
Moody T Chu, Robert E Funderlic, and Robert J Plemmons. Structured low rank approximation. Linear algebra and its applications, 366:157-1777 2003.
Christopher Clark, Kenton Lee, Ming-War Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 2924-29366 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprini arXiv: 0803.5557 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, and al. Nakano, Reiichiro. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 10..::1610...1668 14168, 2021.
Fangming Cui, Yonggang Zhang, Xuan Wang, Xinmei Tian, and Jun Yu. Enhancing target- unspecific tasks through a features matrix, 2025. URL https:///axxttt www.wii..rrraaaoooo abagg//b//200 03414.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers, 2022. URL /tttts://arxiv...t/ org /abs///b////// / / 21.4..... 08696.
Remi Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation, 2014. URL httts://arxxiv borg/bs//ie /14044444 0736.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk- boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sentation for near-lossless IIm weight compression. In ICLR, 2024.
repre-
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Gongfan Fang, Hongxu Yin, Saurav Muralidharan, Greg Heinrich, Jeff Pool, Jan Kautz, Pavlo Molchanov, and Xinchao Wang. Maskllm: Learnable semi-structured sparsity for large language models. Advances in Neural Information Processing Systems, 37:7736-7758, 2024.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference Learning Representations, 2019. URL https:// // openreview net/ / forum? d=:.1mb3R
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323-10337. PMLR, 2023.
13

---

Published as a conference paper at ICLR 2026
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers key-value memories, 2021. URL //atpss://rxiv..... org/ abs/ 20120012..1 149913.
Jinyang Guo, Jianyu Wu, Zining Wang. Jiaheng Liu, Ge Yang, Yifu Ding, Ruihao Gong, Haotong
Qin, and Xianglong Liu. Compressing large language models by joint sparsification and quanti- zation. In Forty-first International Conference on Machine Learning, 2024.
Andi Han, Jiaxiang Li, Wei Huang, Mingyi Hong, Akiko Takeda, Pratik Jawanpuria, and Bamdev Mishra. Sltrain: a sparse plus low-rank approach for parameter and memory efficient pretraining. arXiv preprint arXiv:2406.022 2024.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint 1909. 2019.
Andrey Kuzmin, Markus Nagel, Mart Van Baalen, Arash Behboodi, and Tijmen Blankevoort. Prun- ing vs quantization: Which is better? In Thhrry--eventh Conference on Neural Information Pro- cessing Systems, 2023. URL https: https:: /ttp::://///p/eeieee/eee net/ forum?id=0oU 2xxxs5.
François Lagunas. Ella Charlaix, Victor Sanh, and Alexander M. Rush. Block pruning for faster transformers, 2021. URL https::t///eee /ttptp::/axxxx..oo org/ abs 2209. 04838.
Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Losparse: Structured compression of large language models based on low-rank and sparse ap- proximation. In International Conference on Machine Learning, pp. 20336-20350. PMLR, 2023.
Chi-Heng Lin, Shangqian Gao, James Seale Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. Modegpt: Modular decomposition for large language model compression. In The Thirteenth International Conference on Learning Representations, 2025.
Zhouchen Lin, Minming Chen, and Yi Ma. The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices. arXiv:1009.508: 2010.
Junjie Liu, Zhe Xu, Runbin Shi, Ray C. C. Cheung, and Hayden K. H. So. Dynamic sparse training: Find efficient sparse network from scratch with trainable masked layers, 2020. URL https:
//arxiv. wwwp:vv.orggaeoo aaaaaaas/0000 2220000000770
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. Proceedings of the IEEE International Conference Computer Vision (ICCV), pp. 2736-2744 2017.
Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W Mahoney, and Yaoqing Yang. Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models. Advances in neural information processing systems, 37:9117-91522 2024.
J Scott McCarley. Pruning bert-based question answering model, 2019.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 16Xiv:1609.07880 07843, 2016.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct elec- tricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381-2391, 2018
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiko Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings the AAAI Conference on Artificial Intelligence, volume 34, pp. 5191-5198, 2020.
Pavlo Molchanov, Arum Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1 11264-11111 1122,, 2019.
14

---

Published as a conference paper at ICLR 2026
Mohammad Mozaffari, Amir Yazdanbakhsh, and Maryam Mchri Dehnavi. Slim: One-shot quan- tization and sparsity with low-rank approximation for Ilm weight compression. arXiv preprint arXiv:2410.09615, 2024.
OpenAl, Josh Achiam, and Steven Adler et al. Gpt-4 technical report, 2024. URL https:// // wtxiv.org// waaprr//bb////e /2003 08774.
Sai Prasanna, Anna Rogers, and Anna Rumshisky. When bert plays the lottery, all tickets are win- ning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing (EMNLP), pp. 3208-3229, 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, (1440):1-6) 2020.
Siyu Ren and Kenny Q Zhu. Low-rank prune-and aade---n--acctiize for language model compression. arXiv preprint arXiv:2.306. 14152, 2023.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning, 2020. URL https /ttp/v.wivvv/v/////////.///a.av. org/ /bs. 02389.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathemal- ical Statistics, 2203):400-407, 1951. doi: 10. 2
Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea Goldsmith, and Mert Pilanci. Compressing large language models using low rank and low precision decomposition. Advances in Neural Information Processing Systems, 37:88981-8901 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications the ACM, 64(9)999-106, 2021.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 2019.
Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement pruning: Adaptive sparsity by fine-tuning, 2020. URL http:::////o / /ttp/vxxvvvv.///ee// http://rrrr.gbb///0.0000....
Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models, 2024. URL https: /ttp/viivvvv///// abs/23330000 195.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 3323-4333 2019.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv:2302.13977 2023a.
Hugo Touvron, Louis Martin, and Kevin Stone. Llama 2: Open foundation and fine-tuned chat models, 2023b. URL www.tttps//// /tL.httts///axxvv..rr////// www.:/iwwwrgga////30000 09288.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL httttttt///riiitt
org/abs//77 attaaa////.....//.///// 03762.
Alex Wang. Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019a. URL ht..tt:::: http////ppeeeeeeeeeeeeeeeeee
Jingeun Wang, Yi-Gu Chen, I-Ching Lin, Bo Li, and Guo Lai Zhang. Basis sharing: Cross- pa- rameter sharing for large language model compression. In International Conference on Learning Representations (ICLR), 2025a.
15

---

Published as a conference paper at ICLR 2026
Qidong Wang, Jinxin Ke, Masayoshi Tomizuka, Kurt Keutzer, and Chen Xu. Dobi-svd: Differ- entiable svd for Ilm compression and some new perspectives. In International Conference on Learning Representations (ICLR), 2025b.
Wenxiao Wang. Wei Chen, Yicong Luo, Yongliu Long. Zhengkai Lin, Liye Zhang, Binbin Lin, Deng Cai, and Xiaofei He. Model compression and efficient inference for large language models: A survey, 2024a. URL hww.t:ttp////t http://wwwwwwwww/aa.eeeeee
wtwg/bb//////ee/ / 2402. .24402.0977..
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-Ilm: Truncation-aware singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07377. 2024b.
Xin Wang. S M Alam, Zhongwei Wan, Haoyu Shen, and Mi Zhang. Svd-llm v2: Optimizing singular value truncation for large language model compression. In Proceedings the 2025 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2025c.
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. arXiv preprint 1910009900 2019b.
Ronald J Williams.
Simple statistical gradient-followie algorithms for connectionist reinforcement learning. Machine learning, 1-2992266 1992.
John Wright, Arvind Ganesh, Shankar Rao, Yigang Peng, and Yi Ma. Robust principal component analysis: Exact recovery of corrupted low-rank matrices via convex optimization. In Advances in Neural Information Processing Systems, volume 22, pp. 2080-2088, 2009.
Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 1513-1528, 2022.
Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. Bert-of-theseus: Compress- ing bert by progressive module replacing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 7859-7869 2020.
Peng Xu, Wenqi Shao, Mengzhao Chen, Shitao Tang, Kaipeng Zhang, Peng Gao, Fengwei An, Yu Qiao, and Ping Luo. BESA: Pruning large language models with blockwise eframeter-efffcient sparsity allocation. In The Twelfth International Conference on Learning Representations, 2024.
net URL https::t//// / wtwp://wwweeetttteteeeeee CwooumiddggCGTTT
An Yang, Baosong Yang, Binyuan Zhang, Biyang Hui, Bo Zheng, Bowen Yu, Chang Li, Day- iheng Liu, Fei Huang, Guanting Dong, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15111 2024.
Lu Yin, You Wu, Zhenyu Zhang. Cheng- Yu Hsich, Yaqing Wang. Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang, et al. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning Ilms to high sparsity. arXiv preprint 23X0v:2310.05175 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, 2019.
Cheng Zhang, Jianyi Cheng, George A Constantinides, and Yiren Zhao. Lqer: Low-rank quantiza- tion error reconstruction for 1lms. arXiv preprint arXiv:2402.024441 2024a.
Stephen Zhang and Vardan Papyan. Oats: Outlier-aware pruning through sparse and low rank de- composition. In The Thirteenth International Conference on Learning Representations, 2024.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv: 01068, 2022.
Yuxin Zhang, Lirui Zhao, Mingbao Lin, Yunyun Sun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. Dynamic sparse no training: Training-fre free fine-tuning for sparse Ilms, 08915.
2024b. URL https::t///........ http:///aavvvveeee
16

---

Published as a conference paper at ICLR 2026
Tingting Zhao, Hirotaka Hachiya, Gang Niu, and Masashi Sugiyama. Analysis and improvement of policy gradient estimation. NIPS, 2011.
Changhai Zhou, Shijie Han, Shiyang Zhang, Shichao Weng, Zekai Liu, and Cheng Jin. Rankadap- tor: Hierarchical dynamic low-rank adaptation for structural pruned Ilms. arXiv preprint arXiv:2406. 15734, 2024.
Changhai Zhou, Yuhua Zhou, Yibin Wang, Shijie Han, Qian Qiao, and Hongguang Li. QPruner: Probabilistic decision quantization for structured pruning large language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Lin- guistics: NAACL 2025, pp. 4276-4286, Albuquerque, New Mexico, April 2025a. Association for Computational Linguistics. ISBN 979- 8911888991---9-77 doi: 10. 10.18653/v1/20 10/6633///2025.fmm fwn.0025...iinng-aaa... 240. URL https:::t/// http:///aaaaaaattaaaaeee
org/2025.22222 findings- naacl. 240/.
Wangchunshu Zhou, Canwen Xu, and Julian McAuley. Bert learns to teach: Knowledge distillation with meta learning. In Proceedings of the 60th Annual Meeting of the Association for Compula- tional Linguistics (Volume 1: Long Papers), pp. 7037-7049, 2022.
Yuhua Zhou, Ruifeng Li, Changhai Zhou, Fei Yang, and Aimin Pan. Bslora: Enhancing the pa- rameter efficiency of lora with intra-layer and inter-layer sharing. Forty-second International Conference on Machine Learning, 2025b.
Yuhua Zhou, Changhai Zhou, Shiyang Zhang, Fei Yang, Yi Zhang, and Aimin Pan. Lara: Layer-wiss rank allocation for efficient fine-tuning of pruned large language models. Information Processing & Management, 63(3):00538, 2026.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for wwt.obo//////.......o........//e large language models, 2024. URL https::///// /arrxv..vvvvv////////e///aaaa/v/ www.ab//000 /200.. 07633.
17

---

Published as a conference paper at ICLR 2026
Table of Contents
A The use of Large Language Models(LLMS)
19
B Related Work
19
B.1 Unstructured Pruning
19
Model Compression Distillation Structured Pruning
Preliminaries
20
C.I. Low-Rank Approximation
20
C.2 Low-Rank Approximation with Sparse Corrections
21
D Knowledge Neurons
21
Impact of Pruning FFN Layers
21
D.2 Effect of Pruning Attention Mechanisms
22
D.3 Challenges in Simultaneous Pruning
23
D.4 Proposed Mitigation Strategies
23
Conclusion
23
E
Additional Details and Theoretical Analysis
23
F.1 Policy Gradient with Moving Average Baseline
23
E.2 Theoretical Analysis for LLM Compression
24
F Convergence Analysis of Bernoulli Policy Gradient
24
Baseline Methods Description
25
G.1 Unstructured Pruning Methods
25
G.2 Layer-wise Allocation Methods
25
G.3 Joint Compression Methods
25
G.4 Knowledge Distillation Methods
................................
26
G.5 Structured Pruning Methods
26
G.6 Matrix Factorization Methods
27
H On the Non-Redundancy of L1 Penalization and Pruning
27
H.1 Distinct Objectives of Two-Stage Design
27
Limitations as Compression Parameter
27
H.3
Conclusion
28
I Performance Evaluation on LLaMA and LLaMA-2 Models
28
1.1 Results 50% Compression Ratio
28
1.2 Evaluation Under Higher Compression Ratios
29
J Empirical Throughput and Resource Consumption Analysis
30
K Detailed Ablation Studies
30
K.1 Low-Rank Component Rank Distribution
30
K.2 Perplexity Changes During Layer-wise ................................
31
K.3 Summary Ablation Findings
31
L Limitations
31
18

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS(LLMS)
In preparing this paper, LLMs were employed solely for language refinementpurposes, such as im- proving grammar, clarity, and style of expression. All researchquestions. conceptual frameworks, theoretical arguments, methodological designs,data analyses, and conclusions presented in this work were independently conceivedand executed by the author. The LLMs did not generate, alter, or influ- ence theunderlying ideas, interpretations, or findings. Their use was limited to assistingin polishing the readability and fluency of the manuscript while preserving theoriginality and integrity of the scholarly contributions.
B RELATED WORK
B.1 UNSTRUCTURED PRUNING
Unstructured pruning eliminates individual weights by setting them to zero, providing fine-grained control over model sparsity. SparseGPT (Frantar & Alistarh, 2023) leverages second-order infor- mation perform layer-wise pruning with minimal retraining, whereas Wanda (Sun et al., 2024) combines weight magnitude with activation statistics for a more straightforward pruning strategy. While these methods achieve efficient pruning, they struggle to maintain performance high spar- sity levels and often require additional retraining (Sanh et al., 2020; Renda et al., 2020). BESA (Xu et al., 2024) introduces a differentiable pruning framework that dynamically allocates sparsity across layers to minimize performance degradation, producing competitive results without requiring exten- sive retraining. Dynamic Sparse Training (DST) (Liu et al., 2020) proposes an end-to-end sparse training method where trainable pruning thresholds dynamically adjust the sparsity level during training. Unlike post-training pruning methods, DST eliminates the need for iterative fine-tuning by continuously optimizing layer-wise sparsity using backpropagation. DST is designed for training sparse networks from scratch.
More recent methods focus improving sparsity allocation across layers. OATS (Zhang & Papyan, 2024) formulates optimal sparsity allocation as a constrained optimization problem using second- order sensitivity (Hessian-based), enabling non-uniform sparsity distribution across layers while preserving overall model accuracy. Similarly, DSNoT (Zhang et al., 2024b) proposes a data-free free unstructured pruning method that identifies salient weights using gradient sign stability, making it suitable for low-resource settings. To further enhance performance, several approaches explore adaptive layer-wise sparsity. OWL (Yin et al., 2023) and AlphaPruning (Lu al., 2024) both leverage activation much sparsity each layer can tolerate, thereby optimizing the global sparsity budget. These methods typically build upon simpler base pruners like Wanda and improve performance by reallocating sparsity in a layer-dependent manner. In contrast, our method CAP performs joint low-rank and sparse decomposition via RPCA, which naturally induces structured sparsity patterns and enables global optimization through policy gradients, avoiding hand-crafted allocation heuristics.
as outlier magnitudes or sparsity patterns-to to determine how
Low-rank approximation, obtained via truncated SVD, remains a cornerstone for reducing both memory footprint and FLOPPs in deep networks (Denton et al., 2014). Early composite schemes such as LoSparse (Li al., 2023) add a sparse "correction" to each low-rank factor. but depend on hand-tuned singular-value cut-offs and iterative fine-tuning. LPAF (Ren & Zhu, 2023) improves robustness by applying structured pruning first, then decomposing the residual with sparsity-aware SVD and mixed-rank re-training (W x AB), yet still requires several post- processonn passes. Re- cent work pushes the idea further: SVD-LLM (Wang et al., 2024b) introduces a truncation-awee aware criterion that keeps ILLMMA13BB perplexity at 6.43 with only 20% of the weights, while MoD- eGPT (Lin et al., 2025) performs modular low-rank decomposition across consecutive sub-layers and preserves 90-95% zero-shot accuracy at 25-30% parameters without any fine-tuning. On the sparse side, MaskLLM (Fang et al., 2024) learns hardware-friendle 2:4 masks, retaining 91-95% of baseline accuracy at 50% sparsity and yielding 1.44 speed-up on A 110 GPUs.
Hybrid methods combine sparsity with quantization or low precision: SpQR (Dettmers et al., 2024) stores a tiny FP16 outlier matrix plus 4-bit weights, achieving sub- 1% perplexity loss, while CALDERA (Saha et al., 2024) represents each layer as a low-rank term plus a quantized backbone, pushing below 3 bits/parameter models up to 70B.
19

---

Published as a conference paper at ICLR 2026
Recent advances integrate low-rank adaptation with sparsity and quantization. For instance, SLiM (Mozaffari et al., 2024) combines low-rank modules, unstructured sparsity, and quantiza- tion, introducing probabilistic framework to fit quantization errors using low-rank components. It further proposes SL.iM-Loke variants that apply sparsity-awan adapters with salience-basee com- pensation. Similarly, JSQ (Guo al., 2024) jointly optimizes sparsity and quantization parameters through a unified objective, while L2QER (Zhang et al., 2024a) adopts a sequential pipeline of low- rank decomposition, sparsification, and quantization methods demonstrate the growing trend toward multi-modal compression. However, most rely on heuristic designs or require multiple stages of fine-ture -uninn.
maximize compression efficiency. These
Unlike the above approaches, our CAP framework uses Robust PCA to jointly discover layer- wise low-rank and sparse subspaces, then optimizes Bernoulli masks globally via policy gradi- ents ents-eliminating manual thresholds and any backpropagation over the original parameters. This enables training-free. free, end-to-end decomposition that unifies the benefits of low-rank structure and sparse expressivity without relying error-fitting staged optimization.
B.2 MODEL COMPRESSION VIA DISTILLATION AND STRUCTURED PRUNING
Knowledge distillation transfers knowledge from a large teacher model to smaller student through output mimicking or intermediate feature alignment. Early works such as DistilBERT (Sanh et al., 2019) and TinyBERT (Jiao et al., 2019) apply distillation during pre-training, while task-specific variants like PKD (Sun et al., 2019) and Theseus (Xu et al., 2020) focus on fine-tuned compression. More advanced frameworks such as CKD (Mirzadeh al., 2020) and MetaDistill (Zhou et al., 2022) introduce multi-stage or meta-learning strategies to improve distillation efficiency. On the pruning side, structured methods remove entire neurons, heads, or blocks. ISP (McCarley, 2019) and FLOP (Prasanna et al., 2020) use importance scoring for layer pruning. while BPhybrid (La- gunas et al., 2021) combines block pruning with fine-tuning. CoFi (Xia et al., 2022) jointly prunes weights and attention heads using shared importance metric. Unlike these methods that require ex- tensive fine-tune tunin or teacher models, our approach operates in a post- trainttrainnng. training-fret manner, making it more suitable for owwresource deployment scenarios.
C PRELIMINARIES
Singular Values Layer
Singular Values of Layer
BECA
213
0.5% RECA)
20
1.9% bngra1
1.5%
15.
6.5% MICA
Singater Value
125
CON and 3ECAA
Leve JN (PCC)
0100
(FFS)
Index
Iedex
Figure 3: Singular values of Layer 0 and Layer 31 across different modules, comparing original and @PCA--proeessee matrices. The dotted line represents the singular value distribution of the original model, and the solid line represents the singular value distribution of the low-rank matrix after RPCA processing.
C.1 LOW-RANK APPROXIMATION
Low-rank approximation (Chu al., 2003) is fundamental technique in matrix theory, widely used to reduce the parameter count in neural networks while preserving most of the model's performance. In large language models (LLMss, weight matrices are typically high-dimensional and dense. By
20

---

Published as a conference paper at ICLR 2026
approximating weight matrix W E R RMN substantial reductions in storage and computational cost. Concretely, one typically uses Singular Value Decomposition (SVD) to write
UV with rank R & min(M, N), achieve
W -UEV,
(18)
and then retains only the largest R singular values ΣR. yielding
W UR ΣR V
(19)
This factorization reduce the parameter count from M x to (M N) R, and also break large matrix multiplication into smaller ones:
Wx ~ (n()n(V(x)),
leading to efficiency gains.
Despite these benefits, low-rank approximation alone may be insufficient for LLM compression, especially when the singular values do not decay sharply. Figures illustrate the singular value distributions for two layers (Layer 0 and Layer 31) in large Transformer. The dashed lines represent the original matrices, showing that certain modules in the same Transformer block (e.g., attention vs. feedforward) might exhibit similar shapes, yet across different layers, the redundancy patterns can vary considerably. Consequently, imposing the same rank R uniformly across all layers may prune too aggressively some places and insufficiently in others. A more flexible approach is needed to handle these differences among modules and layers.
C.2 LOW-RANK APPROXIMATION WITH SPARSE CORRECTIONS
To mitigate the shortcomings of purely low-rank approximation, recent methods (Li et al., 2023; Ren & Zhu, 2023) advocate combining low-rank matrix with a sparse correction term. One splits the model weights as:
W = UV +
S
(20)
low-rank
LoSparse (Li et al., 2023), for example, first applies an SVD on W to obtain a low-rank component (with some rank R), and then prunes the residual W-UV to form sparse matrix S. In practice, one must still decide the singular-value cutoff (or target rank) and the sparsity ratio for the residual. Often, additional fine-tuning is performed on the low-rank part recover lost performance, iter- ative pruning is applied to the sparse part (Molchanov et al., 2019), which can be computationally expensive.
A major limitation of these approaches is that they rely heavily on manually specified thresholds for both singular values and residual pruning. They also lack a clear mechanism to coordinate how much rank vs. how much sparsity each layer should receive, since different layers and modules may have different redundancy patterns. Furthermore, when both the low-rank matrix and the sparse matrix need simultaneous updates (or fine-tuning), memory consumption can become large, often exceeding the budget for fine-tuning.
D KNOWLEDGE NEURONS
Transformer- basedd architectures, particularly large language models (LLMs), serve as repositories of linguistic and factual knowledge (Geva et al., 2021; Dai et al., 2022). This knowledge is intricately distributed across the network's feed-forward networks (FFNs) and attention mechanisms, forming the basis for accurate language understanding and generation. Figure 4 provides an illustrative depiction of how such knowledge is encoded, stored, and attributed across these components, with factual information such as "Ireland's capital is Dublin" encapsulated through complex interactions.
D.1 IMPACT OF PRUNING FFN LAYERS
FFNs act as key-value storage within Transformer models, encoding linguistic and factual infor- mation neuron activations (Geva al., 2021). Specific neurons, often referred to knowledge
21

---

Published as a conference paper at ICLR 2026
Factual Knowledge
Q27
P36 capital
Q1761 Dublin
Ireland
Knowledge
Attribution,
Feed Network
Knowledge Neurons
Hidden State
Self Attention Layer
Figure 4: An illustration of how factual knowledge is encoded and attributed within Transformer architectures. Factual knowledge distributed across feed-forward networks (FFNs) and attention mechanisms. Pruning these components risks disrupting knowledge structures, leading to perfor- mance degradation.
neurons, are responsible for capturing and representing precise knowledge. For example, one neuron may activate encode "Q27: Ireland," while its interplay with others encodes the factual relation- ship "Capital: Dublin."
Pruning FFN layers introduces the following risks:
Disruption of Knowledge Neurons: Pruning weights indiscriminately
rons responsible for encoding critical facts, leading to the loss of semantic consistency and factual integrity.
Recovery Complexity: Unlike structured pruning or quantization, unstructured pruning typically requires extensive fine-tuning to recover lost performance, as critical neurons are often inreversibly removed.
D.2 EFFECT OF PRUNING ATTENTION MECHANISMS
Attention mechanisms integral to Transformer models, enabling dynamic token-wise interactions to capture semantic and contextual information. They play two primary roles:
Knowledge Attribution: Attention mechanisms identify and link related entities, such as establishing the factual connection between "Ireland" and "Dublin."
Contextual Understanding: By dynamically weighting token interactions, attention heads provide rich semantic understanding, ensuring the accurate representation of factual rela- tionships.
Pruning attention mechanisms poses distinct challenges:
Removing attention heads or weights can disrupt critical connec- tions between tokens, such as the association between "Ireland" and "Dublin," resulting in factual inconsistencies.
Impaired Attribution:
Redundancy Impact: While certain attention heads exhibit redundancy, aggres- sive pruning risks eliminating disproportionately important heads, significantly impairing model expressiveness.
22

---

Published as a conference paper at ICLR 2026
D.3 CHALLENGES IN SIMULTANEOUS PRUNING
The concurrntt pruning of FFNS and attention mechanisms amplifies risks, as both components play complementary roles. FFNS encode factual knowledge, while attention mechanisms attribute and contextualize this information. Disrupting either component undermines the model's capacity to process and retrieve information effectively. The key challenges include:
Degraded Knowledge Retrieval: Pruning FFNs may impair the model's ability to retrieve stored knowledge, while pruning attention mechanisms compromises its ability to contex- tualize and attribute this knowledge accurately.
Trade-ofTs in Compression: Achieving balance between parameter reduction and knowledge retention demands fine-graine fine-grained strategies that preserve essential structures while compressing redundant components.
D.4 PROPOSED MITIGATION STRATEGIES
To address these challenges, we propose a composite approximation framework designed to preserve critical structures within FFNs and attention mechanisms while achieving significant compression:
Robust Principal Component Analysis (RPCA): RPCA decomposes weight matrices into low-rank and sparse components, separating global patterns from local anomalies. This allows us to target redundancies without compromising essential knowledge structures.
Policy Gradient Optimization: By introducing Bernoulli distributions, we selectively re- tain important components in both FFNs and attention layers. Policy gradient methods efficiently optimize the retention probabilities, bypassing the need for heuristic thresholds.
Ldyer-Adaptive Compression: Our approach applies module--pecific pruning rates, en- suring critical parameters for knowledge retention remain intact while compressing less significant structures.
D.5 CONCLUSION
The interplay between FFNs and attention mechanisms highlights their distinct yet complementary roles in encoding and attributing knowledge within Transformer models. While FFNs store knowl- edge, attention mechanisms enable its contextualization. Effective compression requires strategies that preserve the integrity of these components. Our composite approximation framework achieves this balance by leveraging RPCA and policy-driven optimization, offering a robust solution for re- taining critical knowledge while reducing model complexity.
E ADDITIONAL DETAILS AND THEORETICAL ANALYSIS
E. POLICY GRADIENT WITH MOVING AVERAGE BASELINE
The REINFORCE gradient estimator in Equation equation 11 has high variance because it scales directly with the loss magnitude. We incorporate moving average baseline 5 to reduce variance while maintaining unbiased estimates:
E[C] E (L(W) 8)V., log p(mp [sx)
(21)
Variance Reduction: Let 5 = [[[] the expected loss. The variance becomes:
Var[(C 8) log
CV log - 28Cov(LV logp, V log
Var V log p)
The baseline minimizes the second term when R E[C] (Zhao et al., 2011).
Unbiased Estimation: The baseline introduces no bias because:
E log p)
/ log
(22)
23

---

Published as a conference paper at ICLR 2026
Practical Implementation: We update å as exponential moving average (EMA):
- 36 + (1 B)C(W)
(23)
with 3 = 0.9 in experiments. This tracks recent performance while being robust to noise.
E.2 THEORETICAL ANALYSIS FOR LLM COMPRESSION
Theorem E.1 Soo-RRnkkSpaare Approximation). For any weight matrix W E Rmxx in Trans- former layers, let r* be the intrinsic rank and s* the sparsity level. CAP achieves:
(1
log (1/)1
IW W S C
DVS +0 sparse
(24)
D
low rank eren
with probability 1 1 6, where C, D data-dependent constants.
Proof. From RPCA recovery bounds (Candès et al., 2011) and PAC-Bayes generalization. The first term comes from low-rank approximation error, the second from sparse component thresholding. and the third from policy gradient optimization with [D] calibration samples.
Lemma E.2 (Parameter Efficiency). CAP preserves model capacity with:
rank (L) = ((
(25)
where K is the parameter budget. This matches the optimal rates for low-rank + sparse representa- tions.
Corollary E.3 (LLM Performance Preservation). For Transformer with L layers, if each atten
tion/MLP matrix satisfies Theorem I with || ( |F S 6, then the full model satisfies:
((W) [(W) Lev dim(x)
(26)
where dim (x) is the imput dimension.
F CONVERGENCE ANALYSIS OF BERNOULLI POLICY GRADIENT
We analyse the stochastic optimisation that drives CAP's second stage and show that it is (i) un- biased, (ii) has controllable variance, and (iii) converges to local optimum under standard Rob- bins-Monro conditions.
Unbiased gradient. For a scalar loss L(W) and Bernoulli mask vector m ~ pmm 18, the REIN- FORCE estimator (Williams, 1992; Baxter & Bartlett, 2001) is
g(s) (L(W) - 5) V. p(mm((m)))
(27)
giving
Em [g(5) = V E.Em [[W],,
so the estimate is unbiased.
Mask statistics. Each entry my ~ Bernoulli(ss)) satisfies E[mmy] = sij and Var(m)y) = 8i5 (1 sty), maximised syy = 0.5. We mitigate variance via:
moving-average baseline in Eq. equation 27, which subtracts estimate of TE[Z]: mini-batch averaging over B mask samples, reducing variance by -1
During training 8ig +00 or 1, Var(ma;) 0 and gradients become increasingly stable.
24

---

Published as a conference paper at ICLR 2026
Convergence. With bounded second moment of g(s), step sizes y satisfying Σ 7 and E. n < co, the Pobbins-Monro theorem ensures almost-sure convergence to stationary point (Robbins & Monro, 1951). Empirically, CAP converges within O(10²) updates on 128- sample calibration set.
G BASELINE METHODS DESCRIPTION
This section provides detailed descriptions of all baseline methods used in our experimental evalua- tion. We categorize these methods into several groups based on their compression approaches.
G.1 UNSTRUCTURED PRUNING METHODS
SparseGPT (Frantar & Alistarh, 2023) is a second-order pruning method specifically designed for large language models. It formulates pruning layer-wise reconstruction problem, using the inverse Hessian to determine optimal weight removal while minimizing the increase in layer-wise reconstruction error. The method processes weights in each layer sequentially and updates remaining weights to compensate for the removal of pruned parameters.
(Sun et al., 2024) (Pruning by Weights AND Activations) is a simple yet effective pruning approach that estimates weight importance using both weight magnitudes and activation statistics. It computes importance scores by multiplying weight magnitudes with the norm of corre- sponding input activations, providing more comprehensive measure of parameter significance than magnitude-only methods.
WANDA
DSNoT (Zhang et al., 2024b) (Dual Sparse Network Training) applies structured sparsity patterns during training to achieve efficient inference. The method maintains dual sparse networks during training and applies knowledge distillation between dense and sparse models to preserve perfor- mance.
OATS (Zhang & Papyan, 2024) (Optimal Allocation for Transformer Sparsity) performs optimal sparsity allocation across transformer layers using second-order information. It leverages layer- wise sensitivity analysis to determine the optimal distribution of sparsity across different layers, considering the varying importance of different transformer components.
G.2 LASER-WISE ALLOCATION METHODS
OWL (Yin et al., 2023) (Outlier-Aware Weight Layerwise) is a layer-wise allocation method that optimizes sparsity distribution across layers by identifying and preserving outlier weights that are critical for model performance. The method uses activation-based outlier detection guide the sparsity allocation process.
AlphaPruning (Lu et al., 2024) employs reinforcement learning to automatically determine the optimal sparsity ratio for each layer. formulates the layer-wise sparsity allocation sequential decision-making problem and uses policy gradient methods to learn optimal allocation strategies.
G.3 JOINT COMPRESSION METHIODS
(Mozaffari et al., 2024) (Sparsity-aware Low-rank compression with Importance Masking) combines low-rank approximation with sparsity and quantization. Its key innovation probabilistic quantization error fitting, where low-rank decomposition is used model and compensate for quan- tization errors. The method employs numerical integration to find optimal quantization parameters.
SLiM
(Guo et al., 2024) (Joint Sparsity and Quantization) optimizes sparsity and quantization parameters simultancously through unified optimization framework. It formulates the compression problem as a joint optimization over both sparsity masks and quantization levels, enabling better trade-offs between compression ratio and model performance.
JSQ
25

---

Published as a conference paper at ICLR 2026
L2QER (Zhang al., 2024a) (Low-rank Quantization with Error Reduction) combines low-rank decomposition, quantization, and sparsity in sequential manner. It first applies low-rank decompo- sition, then quantizes the resulting factors, and finally applies sparsity to further reduce model size while using error compensation techniques.
LPAF (Ren Zhu, 2023) (Low-rank Plus Sparse with Adaptive Fine-tuning) follows a three- stage approach: (1) applies first-order unstructured pruning to obtain a sparse model, (2) uses sparsity-aware Singular Value Decomposition (SVD) to decompose the sparse matrices into low- rank form AB, and (3) performs mixed-rank fine-tuning to retrain the decomposed matrices while preserving the sparse structure.
G.4 KNOWLEDGE DISTILLATION METHODS
G.4.1 PRE-TRAINING DISTILLATION
DistiBBERT (Sanh al., 2019) applies knowledge distillation during the pre-training phase to create smaller model. It uses combination of distillation loss, masked language modeling loss, and cosine embedding loss to train a student model that retains much of the teacher's capabilities with significantly fewer parameters.
TinyBERT (Jiao et al., 2019) extends knowledge distillation by transferring knowledge from both the intermediate layers and the prediction layer of the teacher model. It employs attention- based distillation and hidden state distillation to capture more comprehensive knowledge from the teacher model.
G.4.2
TASK- DISTILLATION
(Sun al., 2019) (Patient Knowledge Distillation) introduces patient teacher mechanism where the student model learns from multiple intermediate teacher models of varying sizes. This progressive distillation approach helps bridge the capacity gap between large teachers and small students.
PKD
Theseus (Xu et al., 2020) employs a progressive module replacement strategy during fine-tuning. It gradually replaces modules in the original model with smaller counterparts while maintaining performance through careful scheduling and knowledge transfer.
CKD (Mirzadeh et al., 2020) (Cascade Knowledge Distillation) addresses the capacity gap prob- lem in knowledge distillation by introducing intermediate teacher models. It uses cascade of teacher models with gradually decreasing sizes to provide a smooth knowledge transfer path.
MetaDistill (Zhou al., 2022) leverages meta-learning automatically discover optimal dis- tillation strategies. It learns to adapt distillation parameters and strategies based on the specific characteristics of the teacher-student pair and the target task.
G.S STRUCTURED PRUNING METHODS
(McCarley, 2019) (Iterative Structured Pruning) applies structured pruning in iterative manner, removing entire structures (such as attention heads or feed-forward network dimensions) based their importance scores. The method uses gradient-based importance measures and itera- tive refinement.
ISP
FLOP (Prasanna al., 2020) focuses on reducing FLOPs (Floating Point Operations) by pruning entire dimensions in feed-forward networks and attention mechanisms. It uses activation- based im- portance scoring to determine which structures to remove while maintaining model expressiveness.
BPhybrid (Lagunas et al., 2021) (Block-wise Pruning Hybrid) combines block-wise structured pruning with unstructured pruning techniques. prunes at the granularity of transformer blocks while allowing fine-grained unstructured pruning within remaining blocks.
26

---

Published as a conference paper at ICLR 2026
CoFi (Xia et al., 2022) (Coarse-to-Fine) applies coarse-to-fine to-ein pruning strategy that first identi- fies important structures at a coarse granularity and then refines the pruning decisions at finer levels. It uses learnable masks to determine optimal structured pruning patterns.
G.6 MATRIX FACTORIZATION METHOOS
SVD (Wang et al., 2019b) applies Singular Value Decomposition (SVD) to weight matrices followed by fine-tuning. It decomposes weight matrices into low-rank approximations and then fine- tunes the resulting factors to recover lost performance. The subscript "ft" indicates the inclusion of fine-tuning after decomposition.
H ON THE NON-REDUNDANCY OF L1 PENALIZATION AND PRUNING
Our two-stage design employs both LI shrinkage in the RPCA decomposition and subsequent prun- ing operations. While these steps may appear related on the surface, they serve fundamentally different and non-redundant purposes. This section clarifies why attempting to replace second- stage pruning with a higher the first stage is not feasible, as RPCA is a tool for decomposition, not controllable mechanism for compression.
H.1 DISTINCT OBJECTIVES OF TWO-STAGE DESIGN
The core distinction lies their objectives:
Stage (RPCA Decomposition): Principled Separation. The L1 penalty in the RPCA objective min || Lll. + S||S I is designed to separate a weight matrix W into a globally correlated, low-rank structure (L) and locally salient, sparse outliers (S). The À parameter governs the balance of this separation. Its purpose is to yield high-quality candidate pools for pruning, not to achieve a specific, final compression ratio.
Stage 2 (CAP Pruning): Budget-Constraing Selection. This stage solves a different problem entirely: given the candidates from Stage I, how do select the optimal subset of singular vectors from L and non-zero elements from S to meet strict, user-defined parameter budget K while minimizing task performance degradation? This is global, budget-aware optimization problem that À cannot address.
H.2 LIMITATIONS OF À AS A COMPRESSION PARAMETER
The compression ratio achieved by RPCA an emergent property of the decomposition, not some- thing that can be precisely controlled by tuning A. Attempting to force specific level of sparsity by adjusting À is problematic for several reasons:
Uncontrollable Trade-off: À creates a complex, non-linear trade-off between the rank of L. and the sparsity of S. As you change to affect S, has drastic and often unpredictable effect on L.. There is simple way to set À to achieve, for instance, target of 50% total parameters while maintaining a useful decomposition.
Pathological Decompositions: Extreme values of À destroy the quality of the separation, making the resulting components useless for effective compression.
H.2.1 EXPERIMENTAL ANALYSIS OF PARAMETER EFFECTS
To demonstrate this limitation, we analyze the output of the RPCA decomposition under different values. We use the theoretically motivated default À 1/ /maxxmmmm n) from the original RPCA work (Wright et al., 2009) our baseline, which provides robust, balanced starting point without requiring parameter tuning.
H.2.2 ANALYSIS OF RESULTS
The experimental results reveal several key insights:
27

---

Published as a conference paper at ICLR 2026
Table Impact of À parameter on RPCA decomposition characteristics
setting
Sparsity S
Rank of I.
Decomposition Quality
8e-5 (Low)
0.001 (Dense)
1512
Poor separation: lacks sparsity
Default
0.41
2109
Balanced decomposition
8e-3
0.53
2174
Alternative reasonable trade-off
0.8 (High)
0.999 (Sparse)
3188
Poor separation: L becomes high-rank
Table Perplexity (PPL) and mean zero-shot accuracy for pruned LLaMA and LLaMA-2 models at 50% compression ratio.
Model
w/o Pruning
SparseGPT
WANDA
BESA
CAP
L
Zero-Shot
L
PPL. L
Zero-Shot
LLaMA-7B
5.68
66.31
7.22
63.12
7.26
61.81
6.86
63.13
6.61
64.29
IJaMA-13B
5.09
68.91
6.21
65.98
6.15
66.49
5.92
66.43
5.76
68.32
LLaMA-30B
4.10
72.73
5.33
70.53
5.25
70.92
71.61
4.77
72.08
ILaMA-22 78
5.21
66.96
6.99
63.71
6.92
63.81
6,60
64.92
6.25
65.33
Optimal Parameter Range: The theoretically motivated default À value provides bal- anced decomposition where both and S are meaningful, creating a rich candidate pool for subsequent pruning.
Non-Linear Parameter Effects:
The relationship between and compression outcomes is complex and non-linear. Extreme values (0.8) create highly sparse S but force L to become high-rank, essentially reducing
=s W. Conversely, very low values (8e-5)
fail to enforce meaningful sparsity in S.
Controllability Limitations: No single À value can simultancously achieve low-rank L, sparse S, and satisfy a predefined parameter budget constraint.
H.3 CONCLUSION
The two-stage design is essential rather than redundant. While Stage 1 (RPCA) provides a principled decomposition, it offers limited controllability over the final compression ratio. Stage 2 (CAP) addresses this limitation by performing intelligent, data-driven selection from the candidate pools generated in Stage 1, enabling precise parameter budget control while maintaining task performance. This combination principled decomposition followed by budget-aware selection is fundamental to the superior performance of our approach.
I PERFORMANCE EVALUATION ON LLAMA AND LLAMA-2 2 MODELS
This section presents comprehensive experimental results on the LLaMA and LLaMA-2 model fam- ilies under different compression settings. We evaluate our proposed CAP method against several state-of-the-art pruning baselines to demonstrate its effectiveness across various model sizes and compression ratios.
1.1 RESULTS 50% COMPRESSION RATIO
Table 8 presents the perplexity and mean zero-shot accuracy results for LLaMA and LLaMA-2 mod- els 50% compression ratio. Our method consistently outperforms existing pruning approaches across all model variants, demonstrating superior performance in both language modeling (lower perplexity) and downstream task performance (higher zero-shot accuracy).
The results demonstrate that CAP achieves consistently superior performance across all tested mod- els. Notably, CAP maintains competitive zero-shot accuracy while achieving significantly lower per- plexity compared other pruming methods. For instance, LLaMA-7B, CAP achieves a perplexity of 6.61 compared to 7.22 for SparseGPT and 7.26 for WANDA, while simultaneously maintaining higher zero-shot accuracy (64. 24% vs. 63. 12.12% and 61.81 5% respectively).
28

---

Published as a conference paper at ICLR 2026
1.2 EVALUATION UNDER HIGHER COMPRESSION RATIOS
Table Accuracy results for different pruming methods on LLaMA-7B with 75% parameter reten- tion. CAP represents our proposed method, applied without any parameter adjustment after pruning. w/o Pruning shows the baseline performance of the unpruned model.
Methods
BoolQ
RTE
HellaSwag
Winogrande
ARC-e
ARC-c
OBQA
w/o Pruning
75.08
66.09
56.94
69.93
75.25
41.89
34.60
Magnitude
42.23
52.35
25.86
48.38
26.64
21.50
14.00
SparseGPT
60.86
52.71
29.10
51.78
33.08
17.58
13.40
WANDA
37.83
51.99
26.78
49.41
28.79
19.54
13.20
CAP (75%)
63.33
55.71
31.67
54.01
35.73
22.88
16.12
Table 10: Accuracy results for different pruning methods on LLaMA2-7B with 75% parameter retention. CAP represents our proposed method, applied without any parameter adjustment after pruning. w/o Pruning shows the baseline performance of the unpruned model.
Methods
BoolQ
RTE
HellaSwag
Winogrande
ARC-e
ARC-e
OBQA
w/o Pruning
77.71
62.82
57.14
68.90
76.39
43.52
31.40
Magnitude
43.49
49.10
25.69
50.83
26.09
20.73
16.00
SparseGPT
60.34
54.15
30.28
53.12
33.75
20.39
14.20
WANDA
38.22
52.17
26.95
50.51
28.03
19.62
13.20
CAP (75%)
62.67
56.15
32.11
55.39
36.12
23.02
16.37
We further evaluate our proposed method (CAP) against three other pruning methods SParseGPT, and WANDA-on LLaMA-7B and LLaMA-2 7B models under more aggressive com- pression settings with parameter retention rate of 75% (25% compression). The results are pre- sented in Tables and 10. Key Observations:
CAP consistently outperforms other methods: CAP achieves the highest accuracy across all datasets without requiring any parameter adjustment after pruning. This demonstrates its robustness in retaining critical model performance even under high sparsity conditions.
Magnitude and SparseGPT limitations: These methods show noticeable performance degradation under 75% parameter retention, especially tasks that require factual rea- soning (e.g., OBQA) or commonsense understanding (e.g.. HellaSwag). This highlights the importance of principled parameter selection rather than simple magnitude-basee ap- proaches.
WANDA performance: WANDA performs slightly better than SparseGPT on certain datasets but is generally less competitive compared to CAP, highlighting the advantages of CAP's probabilistic pruning mechanism over activation-ban based importance scoring alone.
CAP excels in challenging settings: By leveraging RPCA for decomposition and policy gradient optimization for adaptive pruning, CAP is able to selectively retain the most infor- mative parameters, ensuring superior performance even under extreme sparsity conditions. The method's 's ability to jointly optimize low-rank and sparse components provides more nuanced approach to parameter importance estimation.
The comprehensive evaluation on both LLaMA and LLaMA-2 families demonstrates that CAP offers a robust and efficient approach to model compression across different compression ratios. By eliminating heuristic thresholds and adopting fine- ineiggdi pruning strategies based on principled matrix decomposition. CAP surpasses existing methods while maintaining computational simplicity and avoiding the need for post-pruning fine-tuning. The consistent performance gains
Conclusion:
29

---

Published as a conference paper at ICLR 2026
across model sizes and compression settings validate the effectiveness of the RPCA-based joint optimization approach.
J EMPIRICAL THROUGHPUT AND RESOURCE CONSUMPTION ANALYSIS
We revaluated the inference efficiency and resource consumption of our method. We conducted these new tests on the LLaMA-3.1-83 model, All measurements used single NVIDIA A100-80G GPU. We compared CAP directly against Wanda to demonstrate the impact of our component-based structure.
Table 11: Inference Efficiency 1024).
Comparison
LLaMA-3.1-BB
(Batch Size=1, Sequence
Method
Component Structure
S-Matrix Sparsity
Latency (S) 4
Throughput (tok/s)
Peak Memory (GB)
Wanda
Single Sparse Matrix
50%
6,28
163.4
15.18
CAP (Ours)
Low-Rank + Sparse
75%%
5,80
176.5
14,90
Our results in Table 11 show a clear performance advantage for CAP. We summarize the key insights below:
Inference Speed and Throughput CAP achieves higher throughput of 176.5 tokens/second. This is faster than Wanda, which reaches 163.4 tokens/second. CAP also maintains lower latency of 5.80 seconds compared to Wanda's 's 6.28 seconds. This result challenges the common assumption that adding components (Low-Rank Sparse) automatically slows down inference.
The Impact of Extreme Sparsity The speed advantage comes from the structure of the weight matrices. Wanda produces uniform 50% sparse matrix. Standard sparse acceleration tools (like torch. sparse) often struggle at this level. 50% density is still too "heavy" for these kernels to outperform dense matrix multiplication. The overhead of managing sparsity often cancels out the speed benefits.
Why CAP is More Efficient CAP decomposes the original weight W into a Low-Rank matrix and a Sparse matrix S. The L component handles the bulk of the energy. This allows the S com- ponent to be extremely empty. The matrix in CAP typically exhibits 75% to 90% % sparsity. This high sparsity falls into the efficient zone for Sparse Matrix Multiplication (SpMM). The hardware can process these highly sparse matrices much faster. This specific structural characteristic allows CAP to beat the baseline despite having multi-component forward pass.
Memory Footprint Our method also remains efficient in terms of memory. CAP operates with a peak GPU memory usage of 14.90 GB, which is slightly lower than Wanda's 's 15.18 GB. This confirms that decomposing the weights does not impose memory penalty. The approach is well- suited for environments with strict VRAM constraints.
K DETAILED ABLATTON STUDIES
This section provides comprehensive ablation studies to understand the behavior and characteristics of our compression method. We focus on two key aspects: the distribution of retained ranks across different modules and the stability analysis through sequential layer-wise pruning.
K.1 LOW-RANK COMPONENT RANK DISTRIBUTION
We analyzed the rank distribution of low-rank components in each module after probabilistic prun- ing. Figure 5a shows these ranks along with the wise averages. Modules with higher ranks in the low-rank component tend to have sparser counterparts in the sparse component to meet the com- pression target. The general trend of increasing rank in deeper layers suggests that redundancy varies across the network, with later layers capturing more complex representations, underscoring the need
30

---

Published as a conference paper at ICLR 2026
Perpleatty
200
Index
(b) Perplexity changes with sequen- tial decomposition and pruning of each layer.
(a) The rank distribution of low-rank matrices for each mod- ule after pruning.
Figure 5: Analysis of module-specific redundancy and pruning stability LLaMA2-7B. (a) The retained rank varies significantly across different modules, indicating differing sensitivity com- pression. (b) Performance degradation shows linear correlation with the number of pruned layers, demonstrating the stability of our compression approach.
for careful pruning in these layers. The v v_proj and o_proj matrices often have lower ranks, likely because they primarily handle content transmission (v.proj) and output integration (o (o.pro in the attention mechanism, focusing on essential information without complex transformations.
K.2 PERPLEXITY CHANGES DURING LAYER-WISH PRUNING
To evaluate the stability of our compression approach, we sequentially decomposed and pruned each layer of the I.LMMA-7B model, starting from the initial layer. The resulting performance changes are shown in Figure 5b. The performance degradation generally exhibits a linear correlation with the number of pruned layers, demonstrating the stability of our method. This linear trend indicates that compression strategy does not introduce catastrophic failure points and maintains predictable performance reduction. Interestingly, an unexpected performance boost occurs when the last layer compressed, underscoring the importance of maintaining structural consistency within the model. Furthermore, decomposing and pruning the first layer led to a slight improvement (ppl 5.18) over the original model's performance (ppl 5.21), suggesting that early layers may indeed contain some redundant parameters that can be removed without harming performance.
K.3
SUMMARY OF ABLATION FINDINGS
Our ablation studies reveal several important insights:
Different modules exhibit varying levels redundancy, with tention projection matrices (v proj, proj) typically requiring lower ranks than other components.
Module heterogeneity:
Layer-wise redundancy patterns: Early layers contain more redundant parameters that can be safely removed, while deeper layers require more careful compression to maintain performance.
The linear relationship between performance degradation and the number of compressed layers demonstrates the predictable and stable nature of our com- pression approach.
Compression stability:
Structural consistency: The performance boost observed when compressing the final layer highlights the importance of maintaining model structural integrity throughout the compression process.
L LIMITATIONS
Similar to other unstructured sparsity methods, the acceleration sparse matrix computations heav- ily depends specialized hardware support. While significant advances have been made in sparse
31

---

Published as a conference paper at ICLR 2026
computation frameworks, the lack of universal hardware optimization can hinder the practical de- ployment of our method in certain environments.
32


Reviews:
[{"review_id": "JM3AfWVkJl", "reviewer": "Reviewer_xDUq", "summary": "This work presents a two-stage compression framework for LLMs: Stage 1 uses RPCA to decompose each weight matrix into low-rank and sparse components, and Stage 2 performs global, budget-aware selection via Bernoulli masks optimized with policy gradient on a small calibration set. The entire pipeline is similar to previous weight matrix decomposition methods; this work adopts the low-rank and sparse decomposition rather than the commonly used SVD decomposition and the learnable mask. This work uses a different decomposition method and combines it with the learnable mask method to form a joint compression method.", "questions": "1. Why the 100 iterations' PPL is higher than 10 tierations. Does this mean the over-optimizaiton may harm the performance?\n2. Although the current hardware devices don't have a nice support of unstructured pruning, it's better to report the end-to-end runtime and the minimum GPU memory requirement. Even the baseline is not based on LLM inference engine is helpful. \n3. The choice of calibration set. I think sampling from C4 is unbiased, but it's better to have some additional experiments to evaluate the impact of the calibration set selection. Is the proposed method robust to the selection？", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing is clear and easy to follow. The appendix is a good preliminary for relevant techniques. \n2. This work provides enough details for reproducibility. \n3. Extensive ablation and analysis, including in the main content and Appendix K, show meaningful insights.", "weaknesses": "1. While the paper reports zero-shot accuracy on eight benchmarks and perplexity on WikiText-2, it would be valuable to include tasks that require chain-of-thought or longer generations, such as those with more than 100+ tokens. Prior work has observed that models can retain perplexity and short-form QA accuracy, but degrade more sharply as generation lengths increase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oXxIE2Ia2B", "reviewer": "Reviewer_vaH5", "summary": "The paper proposes CAP, a two-stage compression framework for large language models (LLMs) that jointly optimizes low-rank and sparse structures. In Stage 1, each weight matrix is decomposed using Robust Principal Component Analysis (RPCA) into a low-rank and a sparse component, providing structured subspaces. In Stage 2, Bernoulli sampling with policy-gradient optimization is used to select which singular values and sparse entries to retain under a global parameter budget.", "questions": "- How does CAP scale when applied to multi-GPU inference or distributed weight loading?\n\n- Can the method be integrated into prefix-caching or mixed-precision pipelines without recomputing decompositions?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- The paper presents a conceptually unified view of low-rank and sparse compression and emphasizes global allocation of redundancy across layers.\n\n- The Bernoulli-policy optimization introduces a probabilistic pruning mechanism that is simple and can be trained without full fine-tuning.", "weaknesses": "- The proposed CAP framework largely combines existing elements, RPCA-based decomposition, SVD thresholding, and REINFORCE optimization, without a fundamentally new algorithmic insight. The combination is technically straightforward and primarily an engineering integration of known methods.\n\n- Reported “no fine-tuning” performance comes at the cost of multiple RPCA and policy-gradient passes on calibration data. The wall-clock savings over simpler pruning or quantization approaches are not measured.\n\n- While RPCA is convex, the subsequent policy gradient step introduces nonconvex stochasticity, and no convergence or optimality guarantees are proven for the combined system. The “global optimization” claim is therefore misleading.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mdz2OwP7F5", "reviewer": "Reviewer_6yto", "summary": "This paper introduces a two-stage compression framework for large language models (LLMs). In the first stage, robust principal component analysis (RPCA) is employed to decompose each weight matrix into low-rank and sparse components, effectively disentangling global structures from localized anomalies. In the second stage, the authors formulate a global, layer-aware pruning strategy as a probabilistic optimization problem over Bernoulli retention masks, optimized via policy gradients on a small calibration set. This design enhances compression efficiency by adaptively allocating compression ratios across layers and jointly optimizing the interaction between sparse and low-rank representations. Experiments across LLaMA-1/2/3, OPT, Phi-3, and BERT-base report better zero-shot accuracy and lower  WikiText-2 perplexity than SparseGPT/Wanda/DSNoT/OATS and competitive results versus joint methods (e.g., SLiM), with modest throughput overhead and a clear reproducibility statement.", "questions": "Please see the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Principled search-space reduction: RPCA to form high-quality candidate subspaces before budgeted selection is elegant and well motivated. \n2. Global, budget-aware allocation: Bernoulli masking with policy gradients ties rank and sparsity to a single parameter budget K, avoiding heuristic thresholds and per-layer guessing. \n3. Consistent empirical gains: Tables show competitive or superior performance to strong sparsifiers (SparseGPT/Wanda/OATS) at 30–50% compression; \n4. Reproducibility: Implementation outline (files, steps), environment, calibration data, and hyperparameters are documented.", "weaknesses": "1. Missing comparison with LoSparse: The paper explicitly points out that LoSparse suffers from manually selected ranks and lack of global coordination, yet the main experimental tables do not include LoSparse results. \n2. Lack of comparison with recent low-rank methods: To validate the claimed advantage of RPCA-based decomposition, the paper should include results on SVD-LLM v2(https://arxiv.org/abs/2503.12340), Basis Sharing(https://openreview.net/pdf?id=gp32jvUquq), and Dobi-SVD(https://openreview.net/pdf?id=kws76i5XB8), which represent the current state-of-the-art low-rank post-training compression methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l5oWKAuBIZ", "reviewer": "Reviewer_qDUb", "summary": "This paper proposes CAP, a two–stage LLM compression framework that first decomposes each weight matrix with Robust PCA into a low-rank component and a sparse component, and then performs global, budget-aware selection by learning Bernoulli retention probabilities for singular values and entries using policy gradients on a small calibration set. The method targets layer-adaptive allocations of rank and sparsity, avoids manual thresholds, and factorizes the retained low-rank part for efficient inference. Experiments on OPT, LLaMA-1/2/3 and Phi-3 show better zero-shot accuracy and perplexity across different compression ratio compared to unstructured pruning methods, like SparseGPT, and joint pruning methods like JSQ.", "questions": "1. Although Table 9 shows the time required for RPCA, the full time required for CAP is not provided. How does the time required for CAP change compared to other pruning methods as the model size scales?\n2. When stage 2 optimizes based on the initial solution obtained in stage 1, how does the optimization ensure that the overall distribution of learned retention probabilities is optimized towards a bimodal shape? How are sparse constraints added?\n3. Could the authors provide the evolution of the learned retention probabilities during the optimization process, and the difference compared to not using global optimization?\n4. The experiments in the paper lack the latest models and tasks. It would be better if the comparisons could be made on the latest models (such as Qwen3, DeepSeek-R1, etc.) and more tasks (such as reasoning tasks, long text tasks, etc.).", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The method is intuitive and clear: the weight matrix is ​​decomposed through RPCA, and the two decomposed matrices are jointly optimized in a learnable manner, which solves the error accumulation caused by previous separate optimization.\n2. The proposed CAP only requires optimizing the diagonal elements of low-rank matrices and the nonzero elements of sparse matrices, significantly reducing the combinatorial space of the problem. Introducing RPCA to calculate the initial solution further simplifies the search difficulty for subsequent optimization. Furthermore, the policy-gradient step uses only 128 calibration sequences with 3–5 forward passes, reducing overall computational requirements and making it more efficient.\n3. The authors conducted comprehensive experiments on various models and tasks (different sparsity rates, combined with quantization, component ablation, etc.), which verified the robustness and versatility of the model in different scenarios.", "weaknesses": "1. All calibrations use 128 C4 sequences regardless of task. It’s unclear how sensitive CAP is to domain shift or to the choice/size of calibration data, especially for models evaluated on different downstream tasks.\n2. While RPCA is solved via a convex surrogate, “globally optimal separation” holds under assumptions (e.g., incoherence, sparsity patterns). The paper alludes to this but reads as if CAP inherits guarantees end-to-end. Stage 2 remains a stochastic discrete optimization without convergence guarantees beyond standard REINFORCE properties. A clearer statement of conditions and limitations would improve correctness.\n3. When performing \"Thresholding masks and final factorization\", the authors choose the top-K parameters to keep, but the meaning of singular values ​​(corresponding to a column/row) in learned retention probabilities is different from that of entries (single elements) in sparse matrices. It would be helpful if the reason for this choice could be given.\n4. The authors claim that this approach will accelerate the model, but due to the existence of unstructured sparse matrices, the actual acceleration effect on real hardware (such as Nvidia A100, H100) is limited. The proposed CAP can support semi-structured sparse constraints to achieve true hardware acceleration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)