SAES-SVD: Self-Adaptive Suppression of Accumulated and Local Errors for SVD-based LLM Compression
Authors: Xing Hu
Dawei Yang
Yuan Cheng
Zhixuan Chen
Zukang Xu
Venue: ICLR 2026 Poster (2026)
Keywords: SVD Compression
Large Language Models
Primary area: foundation or frontier models, including LLMs

Abstract: The rapid growth in the parameter scale of large language models (LLMs) has created a high demand for efficient compression techniques. 
As a hardware-agnostic and highly compatible technique, low-rank compression has been widely adopted. However, existing methods typically compress each layer independently by minimizing per-layer reconstruction error, overlooking a critical limitation: the reconstruction error propagates and accumulates through the network, which leads to amplified global deviations from the full-precision baseline.
To address this, we propose **Self-Adaptive Error Suppression SVD (SAES-SVD)**, a LLMs compression framework that jointly optimizes intra-layer reconstruction and inter-layer error compensation.
SAES-SVD is composed of two novel components:
**Cumulative Error-Aware Layer Compression (CEALC),**  which formulates the compression objective as a combination of local reconstruction and weighted cumulative error compensation. Based on it, we derive a closed-form low-rank solution relied on second-order activation statistics, which explicitly aligns each layer's output with its full-precision counterpart to compensate for accumulated errors.
\ding{183} Adaptive Collaborative Error Suppression (ACES), which automatically adjusts the weighting coefficient to enhance the low-rank structure of the compression objective in CELAC. Specifically, the coefficient is optimized to maximize the ratio between the Frobenius norm of the compressed layer's output and that of the compression objective under a fixed rank, thus ensuring that the rank budget is utilized effectively.
Extensive experiments across multiple LLM architectures and tasks show that, without fine-tuning or additional tricks, SAES-SVD consistently improves post-compression performance. For example, at a 0.2 compression ratio on LLaMA-7B, existing methods exhibit an average accuracy drop exceeding 0.05, whereas SAES-SVD restricts the drop to only 0.02. These improvements underscore the potential of SAES-SVD to effectively narrow the gap between compressed models and their full-precision counterparts, paving the way for more reliable compression of LLMs.

Published as a conference paper at ICLR 2026
SAES-SVD: SELF-ADAPTIVE SUPPRESSION OF AC- CUMULATED AND LOCAL ERRORS FOR SVD-BASED LLM COMPRESSION
Xing Hu
1*
Dawei Yang!*0
Yuan Cheng1.21
Zhixuan Chen
Zukang Xu
Houmo Al
Nanjing University
ABSTRACT
The rapid growth in the parameter scale of large language models (LLMs) has cre- ated a high demand for efficient compression techniques. As hardware-agnostic and highly compatible technique, low-rank compression has been widely adopted. However, existing methods typically compress each layer independently by mini- mizing per-layer reconstruction error, overlooking critical limitation: the recon- struction error propagates and accumulates through the network, which leads to amplified global deviations from the full-precision baseline. To address this, we propose Self-Adaptive Error Suppression SVD (SAES-SVD), a LLMs compres- sion framework that jointly optimizes intra-layer reconstruction and inter-layer error compensation. SAES-SVD is composed of two novel components: O Cu- mulative Error-Aware Layer Compression (CEALC), which formulates the com- pression objective as combination of local reconstruction and weighted cumula- tive етrоr compensation. Based on it, we derive a closed-form low-rank solution relied on second-order activation statistics, which explicitly aligns each layer's s output with its full-precision counterpart to compensate for accumulated errors. @ Adaptive Collaborative Error Suppression (ACES), which automatically ad- justs the weighting coefficient to enhance the low-rank structure of the compres- sion objective in CEALC. Specifically, the coefficient is optimized to maximize the ratio between the Frobenius norm of the compressed layer's output and that of the compression objective under fixed rank, thus ensuring that the rank budget is utilized effectively. Extensive experiments across multiple LLM architectures and tasks show that, without fine-tuning mixed-rank strategies, SAES-SVD consistently improves post-compression performance. For example, at a 0.2 com- pression ratio on LLaMA-7B, existing methods exhibit an average accuracy drop exceeding 0.05, whereas SAES-SVD restricts the drop to only 0.02. These im- provements underscore the potential of SAES-SVD to effectively narrow the gap between compressed models and their full-precision counterparts, paving the way for more reliable compression of LLMs.
INTRODUCTION
Large language models (LLMs), including GPT Family (Achiam al. 2023 Dettmers al. 2022 LLaMA Family (Touvron et al. 2023a b Dubey et al. 2024). and Qwen Family (aii et al. 2023
Team 2024; Yang al.] 2025). have achieved state-of-the-rt performance tasks such com- monsense reasoning, document summarization, and few-shot learning. However, these advances come substantial cost: LLMs with billions of parameters demand enormous computational and memory resources, rendering them impractical for deployment on edge devices. To address this challenge, model compression has become a key research direction, with approaches such as quan- tization (Frantar et al. 2022 Hu et al. 2025). network pruning (Ashkboos et al. 2024), knowledge
c Cooresponding author
"Equal contribution
1 This work conducted during his internship Houmo

---

Published as a conference paper at ICLR 2026
distillation (Hinton al. 2014). and low-rank decomposition Yuan al. 2023 Li et al. 2025
Among these, low-rank compression, typically implemented through Truncated Singular Value De- composition (Eckart & Young 1936), directly reduces both parameter count and computational cost
pponents. This efficiency makes it particularly ap-
Cosine Similarit with FP
Adaptive Err Suppression
Wiki Perplexity
ASVD
0.95
10
SVD-LLM
Severe Accumulation
SAES-SVD (Ours)
0.90
Baseline (SVD-LLM)
10a
SAES-SVV SVD (Ours)
14
Layer Index
Compression Ratio
(a)
(b)
Ratained Energy Ratio
Quis: 0.405
max=0.9530
0.40
Accuracy
0.95
0.35
Fixed (Compression Ratio 0.2)
0.30
Adaptive Method (Compression Rabio 0.2)
0.93
mirm
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.1
0.2
0.3
0.4
0.5
0.7
0.8
0.9
Value
Value
(c)
(d)
Figure 1: (a) Cumulative emrrr analysis LLaMA-3-8B, showing the cosine similarity of com- pressed outputs with the original reference across layers. SAES-SVD SVD effectively mitigates error accumulation by adaptively suppressing upstream deviations. (b) Perplexity WikiText2 under varying compression ratios. SAES-SVD consistently achieves lower perplexity than ASVD and SVD-LLM. (c) Retained Energy Ratio achieved by using different weighting coefficients for the cumulative eгrоr compensation term in with rank=128 compression for k-proj in the 14th Layer of LLaMA-3-8B. Here, for the sake of convenience, we use 3 a/(1 + a) as the horizontal axis. (d) Effect of fixed versus adaptive weighting coefficients for the cumulative error compensation on average accuracy across 7 zero-shot benchmarks.
Existing low-rank compression methods typically focus on layer- optimization. For example, methods such as ASVD (Yuan et al.) 2202).. SVD-LLM (Wang et al. 2024). and AdaSVD (Li et al.) 2025) aim to reduce local reconstruction error within individual layers by introducing activation- aware scaling, applying whitening transformations, and performing adaptive iterative refinement, respectively. However, these approaches share a fundamental limitation: they tend minimize the reconstruction error for each layer independently, without considering how errors prop- agate and accumulate through the network. In practice, reconstruction errors from early layers alter the input distribution of subsequent layers, causing these errors to compound and leading an increasingly larger gap from the full- full-precision baseline. Applying SVD-LLM (Wang et al.) 20224) to ILLMMA-7B further supports this observation. Although SVD-LLMM achieves the theoretical min- imum truncation error at each layer, the similarity to full-precision outputs, as shown in Figure Ta decreases steadily from 0.97 to 0.79 across layers. This result indicates that minimizing per-layer reconstruction error alone does not guarantee end-to-end fidelity.
Building on these observations, we propose Self-Adaptive Error Suppression SVD (SAES-SVD). a low-rank compression framework that incorporates cumulative error compensation into the opti- mization objective. The core idea of SAES-SVD is let each layer perceive accumulated com- pression eπтоrs from previous layers and adapt its compression strategy based on its sensitivity to those errors. In this way, conventional independent layer-wise optimization is transformed into a globally coordinated error-suppression mechanism. Specifically, SAES-SVD consists of two main components: Cumulative Error-Aware Layer Compression (CEALC): For each layer, we minimize not only the local reconstruction error but also enforce alignment with the correspond- ing full- pell-precision reference. This alignment helps compensate for errors propagated from upstream compressed layers. The dual objective is formulated as weighted Frobenius norm of reconstruc- tion and alignment errors. From this objective, we derive a closed-fome solution that effectively suppresses reconstruction error and compensates for cumulative error. Moreover, it achieves this
2

---

Published as a conference paper at ICLR 2026
with only minimal additional cost, as the method relies solely second-order statistics of acti- vation deviations and a few extra matrix operations compared with vanilla SVD compression. @ Adaptive Collaborative Error Suppression (ACES): Since each layer has different sensitivity to
compression ratio and cumulative error, using a fixed weighting coefficient (i.e., 0g in Equation 3 for cumulative error compensation often results in suboptimal solutions. As shown in Figure Ic varying the coefficient across layers leads to different values of the Retained Energy Ratio (RER), defined as the proportion of the squared sum of the top-k singular values to the total squared sum after SVD. RER serves as measure of the low-rank structure of the compression objective, where higher RER indicates stronger energy concentration in the leading singular values and thus greater compression efficiency. To address this issue, we introduce an adaptive weighting coefficient for each layer (i.e.. 3 in Equation 14). The coefficient automatically tuned to maximize RER. By concentrating energy on the dominant singular values, this optimization yields compact and infor- mative low-rank representations and enables energy-aware error compensation that improves com- pression efficiency under fixed rank budgets.
We conducted a comprehensive evaluation of SAES-SVD across diverse datasets, model architec- tures, and representative SVD-based baselines. The evaluation included language modeling, clas- sification, and generation tasks. Results from multiple benchmarks, such as Figure [T] show that SAES-SVD effectively suppresses cumulative errors introduced during compression. Figure Ta in- dicates that cosine similarity with full-precision outputs remains consistently higher across layers. Figure [b] and Figure [d further shows that SAES-SVD achieves much lower perplexity and higher accuracy under diverse compression ratios. The main contributions of this work are threefold:
We identify that independent layer-wise compression leads to the amplification of cumulative errors. To address this, we propose CEALC, which moves beyond independent optimization and explicitly incorporates cross-layer error accumulation into the low-rank objective. Building on this formulation, we derive an efficient closed-form low-rank solution that depends only on second-order activation statistics.
We introduce ACES, which use an adaptive coefficient a to control the weighting of cumulative error compensation. The value of a is adjusted to maximize the proportion of retained energy. This optimization enhances the low-rank structure the compression objective and ensures that, under fixed rank budget, the retained components preserve as much information as possible.
CEALC and ACES together form the framework SAES-SVD. Experiments demonstrate that SAES-SVD is both effective and broadly applicable. At the 20% compression ratio, without requiring complex rank allocation additional fine-tuning, it reduces the drop in zero-shot accu- racy by 58% and lowers the perplexity gap by 52% compared with Dip-SVD. These improvements substantially narrow the gap between the compressed model and the full-precision baseline.
2 RELATED WORKS
Large Language Model Compression Techniques. The rapid growth of LLMs has spurred ex- tensive research on compression methods that reduce model size and inference cost while preserving performance. Existing approaches can be broadly categorized into four classes: pruning, quantiza-
tion (Nagel al. 2021; Frantar al.] 20022] Hu al. 2025 Lin al., 2023 Hu al. 2024).
knowlen ledge distillation (Hinton et al.] 2014), and low-rank approximation Yuan et al. [2023] Wang
al.) 2024p. Unstructured pruning (Ashkboos et al., 20244 Zhang al. 2023) removes individ-
ual weights, but its irregular sparsity patterns often yield limited hardware benefits. Structured
pruning (a et al.,,220. 2023 [Dettmers al. 2023) eliminates entire channels or blocks, improving ef-
ficiency but risking significant accuracy loss. Quantization (Frantar et al. 2022 Lin et 2023 Hu
cct al. (2025) reduces numerical precision, offering strong memory savings but typically restricted to 2-8 bits, which limits flexibility and may degrade performance under aggressive settings. Knowl- edge distillation transfers knowledge from a large teacher model to a smaller student (Liu et al.) 2024]. but requires costly retraining. In contrast, low-rank approximation, especially singular value decomposition (SVD). provides a hardware-agnosti post-training alternative that can be combined with other methods, making it particularly attractive solution for LLM compression.
SVD-based Compression for LLMs. SVD (Golub et al. 1987) reduces matrix sizes by truncat- ing the smallest singular values. Low-rank approximation using SVD has been widely studied as a hardware-agnostic and efficient approach for compressing LLMs.. Early works mainly focused on
3

---

Published as a conference paper at ICLR 2026
minimizing the per-layer truncation loss. ASVD Yuan et al. 2023) introduced a learnable scaling matrix to mitigate the loss of discarded singular values, but the improvement remained local and could not guarantee the theoretical minimum truncation loss across all layers. SVD-LLM (Wang et al. 2024 2025b) addressed this limitation by incorporating whitening matrix that normalizes input activations, thereby achieving theoretically minimal truncation error. AdaSVD (Li et al. 2025) introduces an alternating optimization scheme that iteratively updates low-rank matrices and com- pensation terms, thereby progressively reducing compression error. Several subsequent methods have incorporated importance weighting. For example, FW-SVD (Hsu et al.) (022) and GFW- SVD (Chekalina et al. 2025) leverage global importance indicators to guide single-layer compres- sion, whereas DipSVD (Ding et al., 2025) jointly exploits both local and global importance to refine weighting process. Although these techniques improve layer-level stability, they still rely on the assumption that layer errors are independent, leaving the problem of cumulative error unresolved.
3 PRELIMINARIES
For a weight matrix W E R=x" its SVD given by = UEVT, where U R is the left singular vector matrix, V R=x" is the right singular vector matrix, and Σ R=x" is nonnegative diagonal matrix with entries arranged in descending order (with min(m, n) effective diagonal elements). The classical low-rank approximation problem can be formulated as
min A BER
||BB
W
r min(mmn.))
(1)
According to the Eckart-Young-irsky (Eckart & Young 1936) theorem, the rank-r optimal ap- proximation of W is [ V, where U, IR mxx and V, E _aax contain the top-r left and right singular vectors of W. By choosing A = U, and B > we obtain AB [W]r, thus achieving rank-r decomposition for compressing W. However, direct vanilla SVD compression ignores the statistical variations of layer input activations and often leads to sig- nificant accuracy degradation LLM compression (Yuan al. [023] Wang et al. (024). To address this issue, recent methods formulate LLM compression as an activation-aws aware,, layer-wise indepen- dent optimization problem with the following objective:
min Ac,Bc
||A W.X
here, X2 denotes the input activations to the ε-th layer, generated by the compressed upstream layers. In practice, compression in upstream layers introduces distributional shifts in these inputs. As depth increases, the gap between X₃ and its full-precision counterpart X accumulates, eventually causing a substantial global deviation of the model outputs from the full-precision baseline.
4 METHOD
Original LLMs
CEALC
Compressed LLMs
Alignment
Reconstruction
Layer 8-1
a
X
|| ||ABX;
x
A| W xx 1|
Layer e-1
x!
X
ACES
Original Weight
Energy retention
Layer
W
Layer e-1
Whiting
A
x
+ Compensation
Xx, N
Truc SVD
Layer
B
Layer N
Figure 2: Overview of the proposed SAES-SVD framework.
We propose AVES a unified low-rank compression framework that suppresses both per-layer's reconstruction emror and cumulative deviations accumulated from compressed upstream layers. As
4

---

Published as a conference paper at ICLR 2026
illustrated in Figure the framework comprises two key components. The first, Cumulative Error- Aware Layer Compression (CEALC), explicitly incorporates cumulative errors compensation into the compression objective, ensuring that each layer aligns closely with its full-precision counterpart. The second, Adaptive Collaborative Error Suppression (ACES), introduces an adaptive coefficient a to balance local reconstruction fidelity with alignment the full-precision counterpart. The optimal is chosen to maximize the proportion of retained energy within the truncated subspace.
4.1 CUMULATIVE ERROR- AWARE LAYER COMPRESSION
Layer compression objective with cumulative error awareness Traditional SVD-based com- pression methods (Li et al. 2025 Wang et al. (024) typically approximates the output of the current layer's weight matrix on its input X by minimizing Equation 2 However, this approach neglects cumulative errors propagated from upstream compressed layers, which accumulate with increasing depth and amplify the global deviation from the full-precision (FP) baseline. address this issue, we introduce an additional alignment term with respect to the FP reference output, leading to the
following weighted objective:
arg min Ae
( + ae ||A,B,X, - alignment
(3)
inira-layer reconstruction
Here, oe 2 0 controls the alignment strength. Let T, W,X, & be the original output under current inputs and R2 W under FP inputs. Then Equation is equivalent to
Re)l = (1 + ae) Ac - Zell? + Cc. where the constant Ce == |/Tel a [e - (1 + (x))r) 2 is independent Ae, B₁, and
[AA - Te) + aell(
(4)
+ agRe We X, tox!
Ze
(5)
1
1
Therefore, Equation 3 equivalent to
min Ar.
AA 1e
(6)
Theorem 4.1. As illustrated by the detailed derivation in the Appendix A.2 let He
XxX,
and L 1/2 Then, for any rank constraint TE,
arg min Sank(Ue)Sre
|UxXe Zell
arg
min SV)
| MeLe
with V2 = U2H1
(7)
Combined with Theorem 4.1 Equation i equivalent to
arg min | A,B
ZX T L
Eliminating explicit inputs via second-order statisties. Storing raw activations X and their FP counterparts X requires prohibitive memory overhead. For instance, calibration set of 128 samples and sequence length 2048, the activation storage cost exceeds the parameter size by more than orders of magnitude, making it infeasible for practical large-scale compression. To address this, we reformulate the objective in terms of compact second-order statis- tics. Specifically, we define the input covariance He X,X, and the differential covariance A (x/- -X which capture the distributional characteristics and upstream-induced shifts.
with moderate
By substituting X/X, H, + 4,, obtain
We XX,X + xxx/xx)
ZeX,
We (HHH+ B,Di). 3 1 az
We
[0, 1).
(9)
1+a
Substituting Equation 8 into Equation 6 then the optimization objective becomes
arg min
|A - We(He 3, Ar
(10)
Hence, the target in ( only on H; and A., which avoids explicit storage of X; and X/. In practice, these two second-orde -oddr statistics are collected in batches and layer by layer, similar to the strategy used in GPTQ, which greatly reduces space complexity.
5

---

Published as a conference paper at ICLR 2026
Closed-form via truncated SVD. Letting Ty A, and defining the target matrix G
We(He + H 1/2 the Bekart-Youngy Mirsky (Eckart & Young 1936 theorem guarantees that the best rank-r approximation of Ge is obtained by truncated SVD:
T = -Gerr = Üre Er, ViT
(11)
where Ere contains the top-ry singular values, and Ur., Vr, are the conrespondigg left and right singular vectors. Consequently, closed-form solution for Ag and Bg given by
A = U, x1/2
= V. Le
(12)
with L, = H 1/2 precomputed from input statistics. At this point, we complete the derivation of the closed-form solution of CEALC. As shown in Equation [O] the approximation target consists of two weighting terms: W Hy and 3&c. The coefficient 3 controls the strength of compensation for upstream cumulative errors when compressing the current layer l. After applying the whitening matrix H-1/2, a truncated SVD yields the optimal theoretical solution.
4.2 ADAPTIVE COLLABORATIVE ERROR SUPPRESSION
Motivation. The formulation in Section assumes that the alignment coefficient 3r Or fixed advance. practice, however, different layers have different sensitivities to accumu- lated errors from upstream layers. A single alignment strength can disrupt the low-rank structure of the original weights. As shown in Figure Ic] varying 3 alters the low-rank characteristics of the compression objective in Equation 10 An inappropriate choice of 3r scatters spectral energy into non-dominant components, which in turn reduces compression efficiency. To address this, we reinterpret Equation 10 as an approximation problem for the original weight matrix We. The key motivation is to permit moderate weight deviations, but only when such deviations help
I+a
① steers the compressed output toward the FP reference, thereby mitigating cumulative error; and @ maximizes the spectral energy preserved the top-r, singular subspace, avoiding wasteful energy leakage into discarded components.
CEALC naturally satisfies condition @1 where the coefficient 3e controls the strength of the align- ment. Thus, our goal is to determine the optimal Br such that, under a rank budget the retained subspace preserves as much information as possible. Formally, we define
S₁ = WrHeLe.
G.(B,) = Sz + +.DD, with
D; = W.
(13)
a = [0.
Be = 1 az
The optimization target can then be expressed as maximizing the proportion of spectral energy captured by the top-r; singular values of Ge(ße):
E of(G)(3e)
3 = arg max SrE(0,1]
(14)
Σ mmin(me.me) (B)))(3))
where o.) denotes the i-th singular value in decending order after SVD. However, singular values change with 3, and each trial value would require new SVD computation, making direct optimization intractable.
Theorem 4.2 (First-Order Approximation of Fixed Subspace(FS-FOAA). Lei St E Rmxn with singular value decomposition St = T,,,V,T, and let Te E {I...., min(m, n) - 1] be the target rank. Denote by Ur, E gmxx, and Vr, E IRax the matrices formed by the top-Te left and right singular vectors of St, respectively. Define the orthogonal projection matrices
Pf =1=1-U,,ū;
P = Vr V.V.
(15)
For Gc(B)= (D& the relative error ratio can be approximated by
(e(3) = |p Pfll) ||t + 3D. ||
(16)
[G
S + 3D
where S = P and D = P D.Pp are the projections of S, and D, onto the orthogonal complements of the top-re singular subspaces of St. Detailed proof can be referred Appendix A.3

---

Published as a conference paper at ICLR 2026
For computational tractability, freeze the principal subspace at 3 0, leveraging Theorem 4.2 The resulting approximation ratio can be written as
|| + BD ((3)
2 - 3
(17)
I + 3Dellp
+ 4 8 + + C
where
a = a= 1 b= (Se1, DeL), c= c=1Daille = IS B= (St, DE), C= D (19)
(18)
Setting the derivative of ((3) to zero yields a quadratic equation
P(8) = (cB - bC) 3² + (cA - aC) 8 + (bA - aB)= =0
We evaluate all real roots within feasible interval [3min, 3max] (e.g., [0, 1)), along with the end- points and the minimizer 3 of ((3) is then selected. This adaptive selection minimizes the fraction of truncated energy, ensuring that the retained singular spectrum captures the maximum proportion of information. As illustrated in Figure [d] the adaptive strategy consistently outperforms fixed-o settings across different compression ratios, highlighting both its robustness and effectiveness.
Through CEALC and ACES, tively selects oę suppress newly introduced truncation error, while residual error is accumulated into second-order statistics and propagated forward. This allows subsequent layers to anticipate and compensate for upstream cumulative errors. Consequently, the model achieves output fidelity close to the baseline using only single SVD approximation per layer. The detailed implementation can be found in Appendix A.
we establish closed-loop compression mechanism: each layer adap-
EXPERIMENTS.
Models and Datasets. To comprehensively evaluate the performance of SAES-SVD across di- verse models and tasks, we conducted systematic experiments on mainstream open-source L.MSs and standard language understanding and generation benchmarks, covering a wide range of param- eter budgets and hardware settings. Unless otherwise specified, all results are obtained using single-pass SVD per layer without task-specific fine-tuning after compression. For model selection, we considered representative architectures including LLaMA-1 (Touvron et al. 2023a). LLama- 2 (Touvron et al. 2023b) and LLaMA-3 (LLaMA Team, Al @ Metal 2024). Regarding evalua- focused natural language reasoning and understanding benchmarks. Specifically, we measured perplexity on the WikiText2 (Merity et al., 2016) and C4 (Raffel et al.] 2020) dataset with sequence length of 2048 assess the language coherence and modeling pacity of the compressed models. In addition, we evaluated zero-shot accuracy on multiple datasets, including ARC- ShC-Chalee (Clark al. 2018). ARC-Easy (Clark et al., 2018), HellaSwag (Zellers) MathQA (Amini et al.) 2019). PIQA (Bisk et al.) 2020). and WinoGrande (Sakaguchi to comprehensively examine the generalizability of the compressed models on various
tion tasks and datasets,
ct al. 2019) al.
2021
tasks.
Baselines. In a unified calibration dataset and evaluation protocol, we compared our method with a range of well-established SVD-based low-rank compression approaches, including ASVD (Li al. 2025), SVD-LLM (Wang et al. 2024), FW-SVD (Hsu et al. 2022). Dobi-SVD (Wang et al., 2025a), and AdaSVD (Li et al. 2025). Notably, SVD-LLM and DVD rely on fine-tuning, while A- SVD, AdaSVD and Dobi-SVD employ mixed-rank strategies. In contrast, our proposed SAES-SVD does not require additional training nor auxiliary techniques, yet it consistently outperforms these baselines, highlighting its effectiveness and simplicity.
Main Results. We conducted a comprehensive evaluation of SAES-SVI for low-rank compres- sion of LLMS and compared it with several SVD-based baselines across different compression ratios. Overall, SAES-SVD consistently achieved superior performance across tasks and model scales. Ta- summarizze results on the representative LLaMA-7B. SAES-SVD reduced the perplexity gap by more than 35% compared with the strongest baseline and lowered the accuracy drop by 30-40%. At the more challenging 0.6 ratio, it halved the perplexity gap relative to Dobi-SVD and SVD-LLM, while keeping the accuracy drop below 0.03, compared with more than 0.05 for other methods. These results lead to two key observations: (i) SAES-SV SVD consistently reduces perplexity by a large uargin-up 2x improvement aggressive compression levels (ii) it preserves or

---

Published as conference paper at ICLR 2026
Table 1: Perplexity and zeot evaluation of LLaMA-77 across seven benchmark datasets under varying compression ratios. The table compares SAES-SVD with competing SVD-based methods (ASVD* Yuaan et al. 2023), FWSVD (Hsu et al. 2022), SVD-LLMM (Wang et al., 2024). Dobi- SVD* (Wang et al. 2025a), Dip-SVD"). Methods with fine-tuning are marked by ; mixed-rank strategies by Due to the closed-source implementation of Dip-SVD, we were unable to obtain its
performance at the 0.4 compression ratio.
Ratio Method
Wikiz
PTB.
C4|
Openb.1 ARC.et ARC.et WinoG.t HellaS,1
PIQAT
MathQAt
Avg. t Drop)
0.0
Baseline
5.68
8.35
7.34
0.28
0.67
0.38
0.67
0.56
0.78
0.27
0.52
0.00
FWSVD
2e5
3e4
2c3
0.09
0.11
0.06
0.05
0.08
0.10
0.05
0.02
96%
ASVD
11.14
16.55
15.93
0.25
0.53
0.27
0.64
0.41
0.68
0.24
0.43
16.7%
SVD-LLM'
7.94
16.22
15.84
0.22
0,58
0.29
0.63
0.43
0.69
0.24
0.44
14.7%
0.2
Dobi-SVD
8,54
14.83
10.01
0.26
0,59
0,31
0.66
0.44
0,70
0.23
0.46
10.8%
Dip-SVD
7.95
15.60
14.07
0.27
0.63
0.33
0.64
0.45
0.71
0.24
0.47
9.2%
Ours
7,17
15.16
13.77
0.29
0.68
0.36
0.65
0.45
0.75
0.25
0,50
3.9%
FWSVD
2c4
1e4
le4
0.06
0.05
0.02
0.02
0.00
0.05
0.03
0.02
96.6%
ASVD
1c3
3e3
le3
0.13
0.28
0.22
0.48
0.26
0.55
0.19
0.30
41.8%
SVD-LLM'
13.11
63.75
49.83
0.19
0,42
0.25
0.58
0.33
0.60
0.21
0.37
28.3%
0.4
Dobi-SVD-t
13,54
46.38
23.54
0.22
0.41
0.27
0.58
0.34
0.61
0.23
0,38
26.3%
DVD-SVD'
12.76
46.95
34.35
0.22
0.50
0.30
0.61
0.36
0.64
0.22
0.40
22.8%
Ours
10.42
45.13
32.79
0.23
0.50
0.29
0.62
0.36
0.65
0.23
0.41
21.1%
FWSVD
304
2e4
2c4
0.06
0.01
0.00
0.00
0.01
0.01
0.00
0.01
97.8%
ASVD
6c4
4e4
4c5
0.12
0.26
0.21
0.49
0.26
0.53
0.18
0.29
43.8%
0.6
SVD-LLM'
53.74
4c2
3e2
0.14
0.28
0.22
0.50
0.27
0.55
0.21
0.31
39.9%
Dobi-SVD"
46.18
2e2
0.15
0.20
0.52
0.28
0.54
0.22
38.0%
Ours
116.83
93.97
0.16
0.33
0.25
0.30
0.54
0.23
34.1%
Table 2: Zero-shot accuracy of L...MA-33B under different datasets. Results are reported for SVD-LLM', Dip-SVD', and our proposed method. denotes methods that rely on fine-tuning, while indicates methods using mixed-rank strategies.
different compression ratios on seven benchmark
Ratio
Method
Openb.
ARC e
WinoG.
HellaS.
ARC c
PIQA
MathQA
Average
SVD-LLM*
0.302
0.683
0.684
0.470
0.356
0.725
0.265
0.498
0.2
Dip-SVD'
0.306
0.681
0.692
0.490
0.369
0.734
0.258
0.504
Ours
0.308
0.712
0.684
0.477
0.388
0.734
0.265
0.510
SVD-LLM*
0.222
0.521
0.639
0.355
0.248
0.637
0.228
0.407
0.4
DVD-
0.230
0.548
0.644
0.402
0.283
0.661
0.233
0.429
Ours
0.248
0.543
0.654
0.470
0.284
0.631
0.239
0.438
even improves accuracy, while other methods suffer notable degradation. Crucially, SAES-SVD outperforms not only methods requiring fine-tuning (e.g.. SVD-LLM, Dobi-SVD) but also those These findings, together
relying on mixed-rank allocations (c.g., AdaSVD, Dobi-SVD.DDipSVDD with additional experiments shown in Table 5 and Table 6 in appendix, confirm that the gains of SAES-SVD generalize across architectures and model scales, underscoring both its robustness and broad applicability.
Comparison on larger-scale models. We further evaluated our method on larger architectures, including LLaMA-13B and LLaMA-30B, against current SOTA approaches. As shown in Table 2 and Figure 3] we conducted experiments across compression ratios ranging from 0.2 0.4. Our method consistently outperforms all baselines in terms of both average accuracy and perplexity. No- tably, on LLaMA-30B, SAES-SVD reduces perplexity to 5.49, compared with 22.71 for ASVD and 5.63 for SVD-LLM( Note that Dip-SVD results are only available for 7B and 13B, since the original work does not report LLaMA-30B and its implementation is not publicly released), while achieving 10% accuracy improvement over Dip-SVD. These results demonstrate that SAES-SVD remains highly effective at larger scales and delivers substantial gains in compressed model performance. Inference speed. Low-rank approximation simultaneously decreases both computational com- plexity and parameter storage. This dual reduction substantially lowers memory footprint and com- pute requirements. We evaluated the inference speed of LLaMA3-8D under different compres- sion ratios on a single NVIDIA A6000 GPU. As illustrated in Figure 4 our proposed SAES-SVD achieves consistent acceleration over the FP16 baseline, with speedups ranging from 1.29x to 3.79x as the compression ratio increases. These results demonstrate the practicality of SAES-SVD in accelerating inference while significantly reducing memory demands.
8

---

Published as a conference paper at ICLR 2026
LLaMA-14 13B LLaMA-30B
3.89GB
3.79x
20
Compression Ratio
60%
Perplexity
15
42%
9.86GB
1.71x
10
Compression Levels
20%
12.84GB
1.29x
Baseline
6,74
6.61
6.64
6.34
20% Compression
5
5.63
5.49
40% Compression
15.03GB
1.00x
60% Compression
80% Compression
0
ASVD
SVD-LLM
Dip-SVD
Ours
SpeedUp
Figure 3: Wiki-PPL for two large-scale models, across different methods at 20% compression.
Figure 4: Memory usage and inference speedup of LLaMA-3-8B on varying compression ratios.
Ablation study the compo- nents of our method. Avg Acc denotes the average accuracy over PIQA, ARC-e, Hel- laSwag and WinoGrande.
Table 3:
Table 4: Sensitivity analysis of the upper and lower bounds of the adaptive coefficient a in ACES, conducted LLaMA2-7B a 20% compression ratio.
Model
CEALC
ACES
Wiki PPL.
Avg Acct
Omin
Omax
Wiki PPI.
Avg Acct
L.Lama-7BB
9.34
58,66
1
8.96
58.47
x
7.66
62.02
0.1
0.9
7.88
62.81
7.37
63.03
0.2
0.8
7.38
63.02
0.25
0.75
7.37
63.03
LLama3-8B
X
16.59
55.76
0.3
0.7
7.36
63.01
X
12.25
58.82
0.4
0.6
7.63
62.87
11.48
60.18
The proposed SAES-SVD framework consists of two complementary Cumulative Error-Aware Layer Compression (CEALC) and Adaptive Collaborative Error Suppression (ACES). CEALC addresses the limitation of prior methods that treat layer com- pression independent processes; as shown Figure Ial it effectively mitigates cumulative com- pression errors. Building on this, ACES further enhances the low-rank properties of the compression objective shaped by CEALC, leading to superior performance under limited rank budgets, illus- these contributions: CEALC significantly improves both generative and zero-shot tasks, while ACES provides additional perfor- mance gains on top of CEALC.
Ablation Experiments.
components:
trated in Figure ld The ablation results presented in Table 3able
confirm @ablfiionffrm
We conducted a sensitivity analysis of adaptive coefficient a by varying its lower and upper bounds. Experiments were performed on LLaMA-2 7B at 20% compression ratio, with results summarized Table [ The analysis shows that performance relatively stable within moderate ranges of Specifically, when amin 0.25 and amax 0.75, the model achieves the best trade-off, with an average zero-shot accuracy of 63.03%. In contrast, setting the lower bound too high (e.g., Ounin 0.8) slightly degrades perplexity while reducing average accuracy significantly, indicating potential overfitting calibration dataset. These results validate our choice of setting the lower bound 0.25 and the upper bound 0.75 in practice.
CONCLUSION
In this work, we proposed SAES-SYD, principled framework for low-rank compression of large language models that explicitly addresses both local reconstruction mulated error. By introducing Cumulative Error- Aware Layer Compression (CEALC) and Adap- tive Collaborative Error Suppression (ACES). our method effectively mitigates error propagation while preserving critical low-rank structure. Extensive experiments across diverse models and tasks demonstrate that SAES-SVI consistently outperforms prior SVD-based methods in terms of per- plexity and shot accuracy, even under aggressive compression ratios, and does so without re- quiring fine-tuning or mixed-rank heuristics. These findings highlight the robustness, scalability, and practicality of SAES-SVD, providing promising direction for deploying large language mod- els efficiently in real-world scenarios.
and cross-layer aceu-

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv: 08774, 2023.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannanch Ha- jishirzi Kohlmeier. Mathqa: Towards interpretable math word problem solving with operation- based formalisms. In Proceedings of the 57th Anmual Meeting of the Association for Com- putational Linguisties (ACL), pp. 2357-2367, Florence, Italy, 2019. Association for Compu- tational Linguistics. doi: 10. (0.18653/11/119-122)
httpps://aclantthol..oo loy.oolgyoo.o.oo...oo........ org/
P19-12277
Saleh Ashkboos, Maximilian I. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15024 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 100:29999,66, 2023.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Viktoriia Chekalina, Daniil Moskovskiy, Daria Cherniuk, Maxim Kurkin, Andrey Kuznetsov, and Evgeny Frolov. Generalized weightee svd: Scalable Kronecker-factore fisher approxima- tion for compressing large language models. arXiv preprint 17X94 May 2025. URL
bittps://arxie www.:orgggasoee... abs////////// 2505. 2597.17974
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick. and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1003:v/800.05444 2018
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 O: 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35: 30318-30332, 2022.
Tim Dettmers, Ruslan Svirschevski. Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk- boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized repre- sentation for near-lossless Ilm weight compression. arXiv preprint arXiv:2306.03007 2023.
Xuan Ding, Rui Sun, Yunjian Zhang, Xiu Yan, Yueqi Zhou, Kaihao Huang, Suzhong Fu, Chuanlong Xie, and Yao Zhu. Dipsvd: Dual-importance protected svd for efficient Ilm compression. arXiv preprint arXiv:2506.20353, June 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- trika, 1(3):211 11--28, 1936. doi: 10.1007/BPO2288867.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint 20X1v:2210.17323, 2022.
G. H. Golub, Alan Hoffman, and G. W. Stewart. A generalization of the eckart- young mirsky matrix approximation theorem. Linear Algebra and its Applications, 88/89:317-322 1987. doi: 10. 00.10066002243999339991911
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In Advances in Neural Information Processing Systems, 2014.
10

---

Published as a conference paper at ICLR 2026
Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. In International Conference on Learning Representations (ICLR) 2022 Workshop, 2022. URL
org/ abs/2207.
00112 arXiv preprint ar arXiv:2207.001
Xing Hu, Yuan Cheng, Dawei Yang. Zhihang Yuan, Jiangyong Yu, Chen Xu, and Sifan Zhou. I-llm: Efficient integer-only inference for fully-quantized low-bit large language models. arXiv preprint arXiv:2405. 17849, May 2024. URL httpt:://////.. inttps://arxiv: .rgg abs /2055. 17849
Xing Hu, Yuan Cheng, Dawei Yang, Zukang Xu, Zhihang Yuan, Jiangyong Yu, Chen Xu, Zhe Jiang, and Sifan Zhou. Ostquant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting. arXiv preprint arXiv:2501.33987 2025.
Zhiteng Li, Mingyuan Xia, Jingyuan Zhang, Zheng Hui, Linghe Kong, Yulun Zhang, and Xiaokang Yang. Adasvd: Adaptive singular value decomposition for large language models. arXiv preprint orxii:2502.01403, 2025.
Lin, Jiaming Tang, Haotian Tang. Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. LLM-QAT: Data-free quantization aware training for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 467-484, Bangkok, Thailand, aug 2024. Association for Computational Linguis- tics. doi: 10.1 10.118655/v1/224 10n./////...00...nnnnng-dad.... URL Findings-acl.22 26/
LLaMA Team, AI @ Meta. The llama-3 herd of models, 2024. URL https::/arxiv..om
abs/2407. 21783
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems, 2023.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture 2016.
models. arXiv preprint
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Tijmen Blankevoort. A white paper on neural network quantization. arXiv preprint 20X.0:2/06.08295, 2021. abs/2106.08255
doi: 10:48500anxiv22106..833 URL https:://rxiv..
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research. 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 2021.
Qwen Team. Qwen2 technical report. arXiv preprint 2rXiv:2407.10677 2, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini 1397,,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint Axiv:2307.002888 2023b.
Qinsi Wang. Jinghan Ke, Masayoshi Tomizuka, Yiran Chen, Kurt Keutzer, and Chenfeng Xu. Dobi-svd: Differentiable svd for IIm compression and some new perspectives. arXiv preprint arXiv:2502.0277: 02723, 2025a.
11

---

Published as a conference paper at ICLR 2026
Xin Wang. Yu Zheng. Zhongwei Wan, and Mi Zhang. Svd-Ilm: Truncation-aware singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07377, March 2024. StRt:ttps://aat
org abs 0t0oo//203.07777
Xin Wang, Samiul Alam, Zhongwei Wan, Hui Shen, and Mi Zhang. Svd-Ilm v2: Optimizing sin- gular value truncation for large language model compression. arXiv preprini arXiv:2503. March 2025b. /tttps::////xi http:////..bb//...33333333
Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09360 093888 2025.
Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.0582) 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint OrXiv:1905.07833 2019.
Mingyang Zhang, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, Bohan Zhuang, al. Pruning meets low-rank parameter-eeffiiien fine-tuning, arXiv preprint arXiv:2305. 18403, 2023.
12

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 LLM DISCLAIMER
The authors hereby declare the role of large language model (LLM) tools in the preparation of this manuscript: LLMs were solely utilized to assist with text polishing (including refining sentence structure, optimizing lexical expression, and enhancing language fluency) and **writing optimiza- tion** of the paper's narrative content.
is explicitly emphasized that all core components of this research, which determine the originality, scientific validity, and academic value of the work, were independently completed by the research team through manual efforts. These components include, but are not limited to:
The formulation and development of the overall research framework, core ideas, and logical struc- ture of the study;
The design, coding, debugging, and validation of all algorithms and program codes involved in the research;
The design of experimental protocols, collection and preprocessing of experimental data, ecution of experiments, analysis and interpretation of experimental results, and verification of conclusions.
The use of LLM tools did not involve any participation in the conception of research content, gen- eration of technical solutions, implementation experimental processes, or derivation of research conclusions. All content of this paper adheres to academic integrity standards, and the research team assumes full responsibility the scientificity, authenticity, and originality of the work.
13

---

Published as a conference paper at ICLR 2026
A.2 THEORETICAL PROOF OF THEOREM 4.1
AND SUBSEQUENT TRUNCATED SVD
Problem Formulation.
Consider the optimization problem
min J(A, B) = ||AXX - - A.
(20)
where the goal to simplify the objective function [A, B) so that optimization with respect to A and B becomes more tractable. The key idea is to exploit the structure of the matrix X, particularly its row space.
Assumption. To ensure that the following projection operations are well-defined, assume that X has full row rank, because in practice a small ridge parameter À will be introduced and Le (He + AT) 1/2 This assumption is reasonable since the activation matrix X typically has far more columns than rows. If X is p matrix with p < n, then rank(X) p, which guarantees that XX is invertible.
Orthogonal Projection Operator.
We define the projection matrix
P (XxXxxx---
(21)
which is orthogonal projector mapping any row vector in R" onto the space of X. For any matrix with n columns, the product Y projects each row of onto the row space of X.
The matrix P satisfies several important properties:
Symmetry: PT since XXT symmetric and is its inverse.
Idempotence: p2 P. which follows by direct caleulation.
Projection property: For any matrix of the form CX, right multiplication by P leaves it un- changed, i.e., (CX)P CX. In particular, ABX belongs this class.
Decomposition of Z. We decompose Z into components lying in the row space of X and its orthogonal complement:
Zp= ZP,
Z1 Z(I P).
(22)
Clearly,
7= + 7(1 P) = Zp Zı,
(23)
where Zp is the projection of onto the row space of X, and Zj the projection onto its orthogonal complement. Moreover, P
Reformulated Objective.
Using this decomposition, we expand the objective:
J(A, B) = ||ABX - 21
=|AABX - (Zp + 21
(24)
IIAAX Zp) 2
Let U = ABX - 2p and = Since U lies in the row space of X and V lies in its orthog- onal complement, they orthogonal, i.e., Tr(U V) 0. By the Pythagorean theorem for the Frobenius norm, we obtain
= Zpl 1 12.
(25)
By the premise, start from
J(A, B) || ABX ZX -XX |UX- 7P 1P
where U AB. (26)
14

---

Published as a conference paper at ICLR 2026
Step 1: Projection reduction is tight up constant. Decompose Z into its components on the row space X and its orthogonal complement:
Z ZP - P)
Zp + Zi,
Z1 =
(27)
Since UX lies in the space of X, have (UX)P- U.. Therefore,
IUX- 2 | (Zp Z1) |IUX -Zp) -111
= | 11 + 1 (orthogonality in the row/orthogonal subspaces).
(28)
Thus minimizing |UX ZPI is equivalent minimizing | - since they differ by the constant ||_112 independent of U. Hence equation 26 is valid surrogate objective.
Step 2: Whitening reformulation. trace identities:
Let M
ZXT mmxp, Expand the Frobenius norm using
|UUX ZP ||UX - - -)
= - 21 +
= =tr(UHUT)- ( ) +r(ZZZ')
(29)
Because H > 0, its symmetric square roots H= exist and are invertible. Using
(r(UHUT) = /3 tr(UMT) -U MH MH-1/2)
(30)
can complete the square:
|UX - ZPI = |/ - 22 MH 1/2 4
=
|1H -/H-2/1 +
(tr(ZPZT) - ||
(31)
The parenthetical term independent of Therefore the minimizers of equation 26 are exactly the minimizers of
min UU)
(32)
At this point, we have completed the proof of Theorem 4.1 Subsequently, we will perform low-rank approximation based on this.
Step 3: Invertible change of variables and rank preservation.
Define the change of variables
V V/2
L = --
(33)
Since H is invertible, rank(V) = rank(U), so the rank constraint is preserved. The problem is thus equivalent to
min rank(V))r
| M
(34)
This proves the desired whitening equivalence: minimizing |UXX ZPI over rank-r U is equiv- alent to minimizing | ML over rank-r V, with the bijection between minimizers given by
U = V~
15

---

Published as a conference paper at ICLR 2026
A.3 THEORETICAL PROOF OF THEOREM 4.2
FIRST-ORDER
APPROXIMATION FOR TRUNCATION RATIOS
Setup. Let S E R=x" with singular value decomposition = UEVT, and fix a target rank E (1...., min(m, n) 1}. Denote
U.: V, Vidan
PE := P=:=I-U,U; U,U, PR:=I-VVV
Given direction D Rmxn and scalar [0, 1], define the affine path
G(B) = + BD.
(35)
Let ||- || the Frobenius norm, || the spectral norm, and let (1 denote singular values.
True tail energy and ratio. By the Eckart-Young theorem,
E(9)
= min (an
[[G(8) = Ea(G(3))
(36)
We call VE(B) the tail (Frobenius) norm of G(3) at rank The corresponding tail ratio is
P(3) = VE(3)
(37)
|G(B]
Fixed-subspace first-order approximation (FS-FOA). Freeze the leading rank-r subspaces at 3 = (those of 5), and approximate the tail by the projection onto the fixed orthogonal complement:
E(3) | PL G( PR
|| - BD1 Pe-
PLSRR D1 = PLDPR.
(38)
The FS-FOA tail ratio is then
|S| + +BD1
(())
(39)
|| + 3DD1F
Assumption (spectral gap and small perturbation).
Let the spectral gap at r be
(,) xx+1(8) 0,
(40)
and assume the relative perturbation is small:
8 D11 6
< TO. for some sufficiently small To E (0, 1).
(41)
Theorem (FS-FOA is first-order accurate for the tail energy). Under equation 40 equation 41 there exist absolute constants C1, C22 > 0 such that for all B E [0, 1].
E(3) = || + +3D 12: + RE(3),
[Rx(3)
s Ci Ci 8
(42)
In particular, the discrepancy between the true tail energy and its FS-FOA surrogate is second order in T.
Corollary (FS-FOA is first-order accurate the tail ratio).
With the same assumptions, there
exists C" > 0 such that
p(3) s C' 2 of SPDDD (
(43)
Hence ((3) is first-order accurate approximation the true ratio p()).
16

---

Published as a conference paper at ICLR 2026
Proof sketch of equation 42
Let (.)) V.(3) be the leading r-dimensional singular subspaces
of G(5) and (3) = (3),, Px(3)= IVV,( their orthogonal complements.
=
Standard Davis Satis-ahhn/Wedin perturbation bounds imply
[|2(3)) Pall2 & [|Pr(3) - Pnll2 2 T.
(44)
Because E(3) = [ write Pr((3) = Pr. + Et. PR(3) = PR + ER with |L = (T) and expand:
(3) = (PL + EL)(S 3D) ER) PL(S + 3D) + E,
where the Frobenius norm the remainder satisfies || = 0(T [|G(])]))) by the block structure induced by (U+, V.). Squaring norms yields
EBB) = PEEPPR| + (+) = = + 351+BD1122 11? + ( 1DD)))
which is equation 42
The corollary equation
43 follows by dividing by ||GB)]| and using
v (z)) for small x.
Closed form for the FS-FOA ratio and its stationary condition. Both numerator and denomi-
nator of equation 39 are quadratic polynomials in 3:
|| + 3D_11 = + 2b || + 3D
(45)
A C+
(46)
where
a = |S b: (11,
A: |
B: (S, C:
: D
(47)
Let
a 263 cB2
(BB = V A+22; + CB82
(48)
Since the square root is monotone, arg min & coincides with arg min of the squared ratio. Differen- tiating and simplifying gives the quadratic stationary equation
(cB - bC) B² + (cA- (oA--C)3 3 + (bA aB) 0.
(49)
Selection rule. Solve equation 49
keep real roots in [0, 1]. add the boundaries {0, 1}. evaluate
p(3) on this candidate set, and choose the minimizer 3 E [0, 1). In applications with trade-off parameter a 2 0, one uses the bijection
a
3 1,
(50)
1+a
so that a" = 3-
When is FS-FOA accurate. Accuracy is controlled by D11//22/8 When & (e.g.,
r S 0.5), the error in equation 42 equation 43 second order and the stationary points of p reliably approximate those of In near-degenerate cases (small spectral gap large D), one may constrain < 8max < 1 or apply a mild shrinkage to the selected 39*; a one-step refinement that recomputes the leading subspaces at G(B*) further reduces the residual error.
Interpretation. The numerator in equation 39 measures the energy that would be discarded by rank-r truncation (tail) when measured in the fixed orthogonal complement of the leading subspaces of S; the denominator measures the total energy. The quadratic form equation 49 encodes the balance between tail alignment (a, and total alignment (A, B, C),, yielding a closed-fore form, one- dimensional selection of 3 (hence a) that minimizes the truncation ratio to first order.
17

---

Published as a conference paper at ICLR 2026
A.4 QUANTITATIVE RESULTS
Table 5: Performance of Compressed Llama-2-7B Across Benchmarks and Compression Ratios.
Ratio
Method
WikiTexi-2
PTB
C4
ARC_e
WinoG,
HellaS.
PIQA
Average
0
Original
5.68
8.35
7.34
74.62
69.22
76.00
79.11
68.85
SVD
39661.03
69493.00
56954.00
26.39
48.62
25.64
52.99
38.41
FWSVD
8060.35
9684.10
7955.21
26.05
50.20
25.70
52.39
38.59
ASVD
1609.32
7319.49
1271.85
26.81
49.49
25.83
53.81
38.99
0.4
SVD-LLM
161.11
719.44
61.95
36.99
56.04
30.49
56.96
45.12
AdaSVD
14.76
304.62
56.98
41.12
58.17
31.75
58.49
47.38
Ours
11.35
217.20
40.57
43.27
$7.77
32.14
58.92
48.03
SVD
53999,48
39207.00
58558.00
25.80
47.36
25.55
52.67
37.85
FWSVD
8173.21
8615.71
8024.67
25.84
48.70
25.64
52.83
38.25
ASVD
6977.57
15539.44
4785.15
25.13
49.17
25.48
52.94
38.18
0.5
SVD-LLM
272.19
1772.91
129.66
31.65
51.14
28.38
54.57
41.44
AdaSVD
25.58
593.14
113.84
34.18
54.06
28.88
55.50
43.16
Ours
14.02
95.48
48.94
35.56
55.80
29.58
56.58
44.38
SVD
65186.67
79164.00
70381.00
24.49
51.85
25.40
53.16
38.73
FWSVD
27213.30
24962.80
47284.87
25.38
48.46
25.61
51.96
37.85
ASVD
10003.57
15530.19
9983.83
26.68
48.86
25.76
51.80
38.28
0.6
SVD-LLM
89.90
2052.89
561.00
26.73
47.43
26.89
53.48
38.63
AdaSVD
50.33
1216.95
239.18
28.20
51.22
27.36
52.83
39.90
23.89
334.67
100.42
31.02
52.01
30.38
54.62
42.01
Table 6: PPL and accuracy of LLaMA-3-8B across compression ratios comparing SVD-LLM and
Ours.
Ratio
Method
WikiText-2 (PPL)
Openb
ARC ct
ARC
WinoG. 1
HellaS. 1
PIQA
MathQA
Mcan
SVD-LLM
0.242
0.579
0.664
0.438
0.2
Ours
11.49
0,252
0.284
0,593
0.658
0.393
0.671
0.268
0.446
0.4
SVD-LLM
80.46
0.134
0.193
0.325
0.523
0.274
0.548
0.208
0.315
Ours
23.30
0.162
0.196
0.340
0.554
0.296
0.552
0,218
0.331
SVD-LLM
729.46
0.104
0.207
0.272
0.521
0.264
0.529
0.211
0.301
Ours
63.09
0.136
0.219
0.285
0.534
0.278
0,535
0.225
0.316
SVD-LLM
6971.65
0.120
0.207
0.259
0.503
0.258
0.531
0.201
0.297
0.8
Ours
181.84
0.140
0.203
0,281
0.520
0.259
0.539
0,208
0.307
We have supplemented detailed experimental results different compression ratios for LLama-2 and LLama-3. As shown in Table and and T our method consistently achieves superior results different compression ratios. Especially under extreme compression ratio conditions, such as Llama3-8B at a compression ratio of 0.8, our method outperforms the SVD-LLM method by nearly two orders of magnitude on the Perplexity metric.
18

---

Published as a conference paper at ICLR 2026
A.5 VISUALIZATION AELAPAATTAAIOO BETA
To clarify the impact of different beta coefficients on the low-rank property of the optimization ob- jective in Section 4.1 visualized the compression of the k-proj Linear layer of the 14th layer of LLama3-8B. As shown in Figure beta affects the low-rank property of the optimization objective. The difference between the most suitable beta and the worst beta within the range of [0.1] signifi- cant. When beta 0.95, as shown in the first subfigure, the proportion of discarded spectral energy 7%, while when beta 0.4, only 4.7% of the spectral energy is lost.
0.856
0.955
0.942
Ratained Energy Ratio
0.950
Batained Energy Rabio
0.940
0.854
0.945
0.892
0.940
0.936
Ratained
min-3.0694
0.914
0.848
0.546
0.930
0.00.10.20
10011022.304055000700000
B
Figure 5: Using different beta coefficients for the k-proj of LLama3-885 8B at layer 14, layer 3 and layer 20 after the SVD in Equation 11. the proportion of retained energy at rank- 1228.
We visualized in Figure 6 the Self-Adaptation beta coefficients of the odd k-proj layers of the LLama3-8B model under the condition of a compression ratio of 0.4. As can be observed from the figure, in the shallow layers of the network, beta tends to be small. As the number of layers deepens, the optimal beta continuously increases. In the last few layers, a downward trend appears.
max bound=0, 0, 7972
0.8
adaptive beta (k aproj_bbetal
Adaptive Beta
0.6
0.4
0.2
min bound=0.125
1
6 7
10
11
12
13 14
15
Layer Index
Figure 6: The visualization of the optimal beta values obtained from k-proj of odd-numbered layers on LLaMA2-7B.
19

---

Published as a conference paper at ICLR 2026
A.6 PERFORMANCE ON MORE ARCHITECTURES AND MORE CHALLENGING TASKS
Table 7: Performance comparison of different compression methods on LLaMA3.1 and Qwen2.5 architectures at 0.2 compression rate across multiple benchmarks. Results are reported for 7 zero- shot tasks (ZeroShot'), long-context understanding (VongEval), code generation (HumanEval), and mathematical reasoning (GSM8K).
Model
Method
ZeroShot'
LongEval
HumanEval
GSM8K
FP
0.61
0.41
0.35
0.15
ASVD
0.49
0.33
0.15
0,06
LLaMA3.1-8B
SVD-LLM
0.57
0.37
0.27
0.10
Ours
0.59
0.39
0.33
0.13
FP
0.67
0.49
0.55
0.37
ASVD
0.54
0.39
0.32
0.26
LLaMA3.1-70B
SVD-LLM
0.63
0.45
0.47
0.32
Ours
0.66
0.48
0.52
0.35
FP
0.60
0.58
0.66
0.71
ASVD
0.50
0.49
0.41
0.42
Qwen2.5-7B
SVD-LLM
0.56
0.55
0.55
0.64
Ours
0.58
0.57
0.63
0.69
FP
0.65
0.79
0.55
0.72
ASVD
0.52
0.61
0.37
0,44
Qwen2.5-32B
SVD-LLM
0.61
0.73
0.46
0.63
Ours
0.64
0.76
0.51
0.68
reports the performance of LLaMA3.1 and Qwen2.5 families under a 0.2 compression ratio, comparing three representative low-rank decomposition approaches (ASVD, SVD-LLM, and proposed SAES-SVD). Across all model scales and tasks, SAES-SVD consistently achieves the best reconstruction fidelity among compressed variants, demonstrating strong robustness in both general-purpose reasoning (ZeroShot"), long-context understanding (LongEval), code generation (HumanEval), and mathematical reasoning (GSM8K).
Notably, SAES-SVE narrows the performance gap to the FP baseline much more effectively than existing SVD-based methods. For example, on LLaMA3.1-8B, SAES-SVD retains 96.7% of FP performance on ZeroShot and preserves near-lossness on LongEval. Similar trends hold for larger models: on I.aMA3.1 1.MMA.11-7BB and Owen2.55-3B, SAES-SVD significantly outperforms ASVD and SVD-LLM across all tasks, highlighting its superior scalability high-capacity architectures. These results collectively demonstrate that SAES-SVD is a more faithful and stable low-rank approxima- tion method, enabling high compression rates while minimizing degradation across diverse bench- marks.
A.7 THE EFFECT OF INTRODUCING THE MIXED- RANK CONFIGURATION
Table 8: Performance of SAES after incorporating the mixed-rank configuration derived from ASVD on the LLaMA-7B model. ZeroShot denotes the averaged accuracy over seven zero-shot tasks consistent with the main evaluation protocol.
Compression Ratio
Method
Wiki PPL
ZeroShot
ASVD
11.14
0.43
0.2
SAES-SVD
7.17
0.50
SAES+mixrank
6.19
0.51
ASVD
1.00E+03
0.30
0.4
SAES-SVD
10.42
0.41
SAES+mixrank 7.48
0.47
Table summarizes the effect introducing the mixed-rank configuration into the SAES frame- work for the I.L.MA-BB model. The mixed-rank strategy is obtained by reusing the adaptive rank
20

---

Published as a conference paper at ICLR 2026
Table 9: Perplexity comparison of SAES-SVD and three representative structured compression methods- Methooss-LLM-Prummm LLMM-runer, SliceGPT, and BlockPruner- LLaMA-7B under various memory bud- gets. Lower perplexity indicates better language modeling quality.
Memory
LLM-Pruner
SliceGPT
BlockPruner
SAES-SVD (Ours)
10GB
9.88
8.78
9.40
7.17
9GB
12.21
12.73
12.76
8.22
8GB
18.94
16.39
19.78
8.96
7GB
21.68
27.41
43.05
10.15
distribution discovered by ASVD, while the reconstruction and optimization procedure remains gov- erned by our SAES architecture. Across both compression ratios(0.2 and 0.4), incorporating mixed rank consistently improves over standard SAES-SVD, At 0.2 compression ratio, SAES+mixrank achieves the lowest Wiki perplexity(6. 19) and the highest zero-shot accuracy (ZeroShot@) 0.55), outperforming both ASVD and SAES-SVI SVD. A similar pattern holds at the more aggressive 0.4 com- pression ratio, where ASVD collapses severely in language modeling, but SAES+mixrant maintains stable performance with a substantial gain in Avg.cc@@ 7(0.47). These results indicate that adaptive rank allocation and SAES's weight reconstruction are complementary: the mixed-rank configura- tion provides better structural flexibility, while SAES preserves semantic fidelity during compres- sion. The combination yields a more robust low-rank approximation that scales effectively to higher compression budgets.
A.8 COMPARISON WITH STRUCTURED PRUNING METHOSS
Table 9 presents the perplexity of LLaMA-7B compressed using three widely adopted structured pruning and slicing LLchniques-LLLMMPP
SliceGPT, and BlockPruner-togetherr with our SAES-SVD method under progressively tighter memory budgets. Across all settings from 10GB down to 7GB, SAES-SVD consistently achieves the lowest perplexity. indicating superior retention of language modeling capability. When memory reduces 10GB, SAES-SVD reaches a perplexity of 7.17, surpassing SliceGPT (8.78) and outperforming LLM-Pruner and BlockPruner by a clear margin. As compression becomes more aggressive, the gap further widens: at 8GB and 7GB bud- gets, structured pruning baselines degrade sharply-BlockPrung even collapsing to perplexity above 40-while SAES-SVD maintains stable performance with perplexity 8.96 and 10.15, respec- tively. This demonstrates that SAES-SVD provides stronger parameter preservation under stringent memory constraints, largely due its ability to reconstruct informative subspaces without relying on brittle structural sparsity assumptions. As result, SAES-SVD serves as more robust alternative traditional structured compression pipelines, especially in low-memory deployment scenarios.
A.9 COMBINATION CAPABILITY OF SAES-SVD AND GPTQ
Comparison between pure GPTQ-3bit quantization and the equivalent 3bit configura- tion obtained by combining SAES-SVD with GPTQ on LLaMA2-7B and Qwen2.5-7B. WikiText2 perplexity and @e represent language modeling quality and the averaged accuracy across seven shoo tasks, respectively.
Table 10:
Model
Method
Wiki PPL
AvgAce@7
GPTQ (3bit)
10.17
0.39
LLaMA2-7B
SAES+GPTQ (equal 3bit)
7.64
0.44
GPTQ (3bit)
12.29
0.51
Qwen2.5-7B
SAES+GPTQ (equal 3bit)
9.87
0.57
Table 10 compares the standard GPTQ 3-bit quantization with our equivalent 3-bit configuration that integrates SAES-SVD into the GPTQ pipeline. The evaluation covers both perplexity WikiText2 and the averaged accuracy across seven zero-shot tasks for two representative models: LLaMA2-7B and Owen2.5-7B. Across both architectures, the combined SAES+GPTQ strategy achieves sub-
21

---

Published as a conference paper at ICLR 2026
stantial improvement over pure GPTQ. For LLaMA2-7B, SAES+GPTQ reduces perplexity from 10.17 to 7.64 and increases AvgAce@7 from 0.39 to 0.44. A similar trend is observed for Qwen2.5- 7B, where perplexity improves from 12.29 to 9.87, accompanied by accuracy gain from 0.51 to 0.57. These results demonstrate that SAES-SVD provides a complementary compression ef- feet GPTQ. By reconstructing more informative low-rank subspace before quantization, SAES effectively smooths weight distributions and reduces quantization sensitivity, enabling GPTQ to op- erate lower bit-widths without incurring substantial performance degradation. Consequently, the SAES+GPTQ pipeline offers stronger alternative to pure 3-bit quantization while maintaining the same effective model size.
A.10 THE THEORETICAL COMPLEXITY ANALYSIS AND COMPARISON OF COMPRESSION TIME
We provide systematic evaluation from two perspectives (1) theoretical complexity analysis and (2) empirical runtime breakdown and compare our method (SAES-SVD) with the SVD- LLM baseline.
1010.1 THEORETICAL COMPLEXITY ANALYSIS
Statisties Collection Phase. For each layer with input dimension d the unfolded calibration set contains N tokens. We maintain only two second-order matrices:
Hr = X E Rdn A. (xf x E dud
For each mini-batch with b tokens, we perform two matrix multiplications with complexity Odd b), and update the statistics via exponential moving averages. The total complexity over L layers is:
O (2TdL,
which matches the order of GPTQ and other second-order methods, and is comparable to the statis- tics collection cost of SVD-LLM.
Closed-form CEALC Solution for a Single Layer. For linear layer We Rddaa X with target rank re, the main computational costs of CEALC are:
Whitening matrix construction: Compute I., (He A// using Cholesky-based proce- dure, with cost (dg,) This cost is shared by all second-order compression methods.
Target matrix construction: Build G = We(He + Le, requiring several matrix multipli- cations with total complexity (dour
Truncated SVD: Perform rank-ry truncated SVD G3, with complexity (ddondiine))
ACES Adaptive Coefficient Selection. During compression, ACES introduces additional com- putations:
Shared whitened matrices: Construct
S Willebe,
Dj W.AeLe,
each requiring (dout
Subspace extraction: Perform rank-r, truncated SVD on S₁ with complexity ddd
Closed-form 31 selection: Using Theorem 4.2, compute small number of Frobenius norms and inner products (a, b, A, B, C) within the fixed subspace and solve univariate quadratic equation in closed form. This step is negligible relative to the matrix operations.
Thus, compared with CEALC alone, ACES introduces only one extra truncated SVD per layer and inexpensive scalar operations.
Fine-tuning Phase. SVD-LLM requires approximately 3.5 hours of fine-tuning for LLaMA-7BB and even more for larger models. contrast, SAES-SVD requires no fine-tuning and operates purely post-hoc compression method.
22

---

Published as a conference paper at ICLR 2026
Summary. SAES-SV SLES-SVD&O and SVD-LLM share the same order of complexity in statistics collection and SVD-based factorization. However, by eliminating the fine-tuning phase, SAES-SVD substan- tially reduces the overall compression time.
A.10.2 EMPIRICAL RUNTIME BREAKDOWN AND COMPARISON WITH THE BASELINE
Table 11:
Comparison of SAES-SVD and SVD-LLM regarding statistics time, compression time, and fine-tuning time on LLaMA-7B.
Method
Data Collection
Compression
Fine-tuning
Total
Reduction
SVD-LLM
0.45h
0.15h
3.5h
4.1h
SAES-SVD
0.30h
0.20h
0
0.5h
-88 %
We further provide in Table R3-1 an empirical breakdown of the compression time for SAES-SVD and SVD-LLM on LLaMA-7B, measured on an NVIDIA A6000 GPU. The results show:
More efficient statistics collection. Although the theoretical complexity is similar, SAES-SVE uses a smaller calibration set (128 vs. 256 samples), reducing statistics collection time to about 67% of that of SVD-LLM.
Controlled compression overhead. The SVD compression phase of SAES-SVDD is slightly more expensive due to ACES, but remains within the same complexity order.
Elimination of fine-tuning. SAES-SVD is a post-training method and does rely any fine- tuning, whereas SVD-LLM requires more than 3.5 hours of fine-tuning.
Overall, on LLaMA-7B, SAES-SVD achieves a total compression wall-clock time less than one- eighth of that of SVD-LLM, while obtaining better accuracy retention across multiple tasks. This combination of higher accuracy and lower end-to-end compression time highlights the practical efficiency of SAES-SVD for real-world deployment.
23

---

Published as a conference paper at ICLR 2026
A.11 ALGORITHM
APPENDIX A. ALGORITHMICC DETAILS
Algorithm SallleffeeonOOOddrrssst--eaniine
This routine gathers the per-layer second-
order statistics required by SAES-SVD without storing raw activations. For each layer f, it maintains a running estimate of the covariance H; = XX and the cross-residual term A (X f X
The update is fully streaming: existing statistics are reweighted by factor y = ne/ (n/ + m) and the current mini-batch is scaled by /2/(nz + m) before accumulation, matching the numerics used in our implementation. Shapes are normalized to (dm: .N) to keep the math consistent across layers. This design avoids caching X₃ or x in memory, reduces I/O, and remains robust under variable batch sizes.
Algorithm 2 (SAES-SVD for single layer). Given layer weights Wz and the statistics (He, Ae), form the whitener L (Hl + X1)-1/2 and the whitened objective matrix G₂(B) = We He + Ae) Le- If alignment strength is provided, 3 is selected by Algorithm 3 otherwise
the given via 3 a1(+))). A single rank-r truncated SVD of (3) yields factors A ú, and B. = 2 V Le, which reconstruct a low-rank approximation of We tailored to the cumulative- error-aware objective. The routine uses only one SVD per layer and a Cholesky-based whitener, making it both stable and efficient.
Algorithm 3 (ACES BetaSelect). This procedure selects the adaptive alignment coefficient without repeated SVDs. We compute single SVD of S W HT. to obtain the rank-r princi- pal subspace, project (S, D) onto the orthogonal complement to get (S1. D1), and then optimize a first-order (fixed-subspace) surrogate. Two objectives are supported: (i) the ratio objective, which a+2b3+o0* and (ii) the energy objective, minimizes the approximate tail/total energy ((33) = A+283+C32
which minimizes the tail energy a+ 26, + cB3 Candidate 3 values are obtained in closed form (sta- tionary roots and interval endpoints), then filtered through guardrails: interval clipping [Smins Smax] cap Reap < 1 to prevent over-alignment, and optional shrink factor (0, for added stability. The final choice 3 is mapped back to a = /1 - 3r) and passed to Algorithm
Complexity and implementation notes. All three algorithms rely only on matrix multiplications. one Cholesky per layer for whitening, and one rank-r truncated SVD per layer (randomized or Lanczos). No iterative backpropagation
fine-tuning is required. In practice, choosing small ridge À ensures numerical stability when He is ill-conditioned; if Cholesky fails, increase A. The ratio objective in Algorithm 3 is recommended when cross-layer robustness prioritized; the energy objective is a conservative alternative that further suppresses absolute tail energy.
24

---

Published as a conference paper at ICLR 2026
Algorithm 1: CollleefeeoondOreerSaaa-----rree (per-layer)
Input:
Calibration dataset D; layer set {é): routines that expose per-layer inputs X and FP references X
Output: Per-layer second-order stats Hş E Rd De Ruud
1 foreach layer do
2
H, - A. + O, ny +0
// running matrices and sample counter
3 foreach mini-batch B D do
// Forward once to cache both compressed path inputs X and
FP references X! 4 run forward; collect (X,, x/)² foreach layer do
// Flatten to 2D and transpose to (dims N)
if (ddm(Xe) then
X else if ndim Xx) = then
X x
8
X < reshape (Xe, -1, dm))
10
else
11
X t flatten.to 2D(;)
12
Do the same for X get X with shape (dm: N)
// Streaming reweighting (match code: scale old stats, add scaled current batch)
13
t m m N, 7 t+m
+
14
H. + HH, 7 +
// Batch scaling by 22/me (as in code)
S+ 2
15
Tr
16
X+s-X. X, X + s. x
11 Update H2 X and a== - x)XT in the metric
17
He - +1x+XX*
18
dX + x1-2 X
19
A - A + dx XT
20
return {(Hr, Ar)}e
Algorithm 2: SAES-SVD for layer
Input: Weights We E Rddoxxxing second-order statistics He, Ax; target rank r; ridge 2 O:
(optional) or 3
Output: Ag E Rd Xr
B, E Rrx s.t. Wt R AeBr
1 Le + (He + XI) XI)-II11 1/2
// Whitening Matrix
2 a is given and is not then
3
+ a a))
4 if 3 is not given then
5 + AWG Ar,
// Adaptive // Whitened objective matrix
6 G2 + (He L.
7 [Ü, Σ,, Vel t- TruncatedsvD(Ga.r)
// Trucated SVD Decomposition
8 A + Ú, 1//2
9 B, + Σ 1//2 V. Lt
return (Ai, Br)
25

---

Published as a conference paper at ICLR 2026
Algorithm 3: ACES BetaSelect (single-SVD, closed-for)
Input: W E Rmxx
H =XXX"
A =
target rank 7, damping > O, interval [8min: Smax], cap Seap < 1, shrink p E (0, 1]. objective
{ratio, energy]
Output: 5 E [Amin Smax] and a* == (110111-1
1 Whitening and decomposition:
+ (H + XT) 1/2 (Cholesky-basedd
3 H.W + D+WAy L
// G(3) 8 3D 11 only one SVD per layer
EU,,
TopRSVD(S,r)
5 Projectors and FOA terms:
6 PL t I-U,U PR --I-VV V,V,T
7 S1 + PLSPRR. D1 t- PL
8 a+ @a++||Si] SS b-($1, ct D1I2
9 A+ |S|3, (5, C+IDD
10 Candidate generation (FOA):
11 object ive ratio then
p(3) - (cB bC) + 3C)
(cA-aC), + (bA-aB)
// stationary points
((3)
C+ roots of p(3)} U Bminns 3max)
+ c//p-(bb// b/c, Pmin Buax)
// minimizes tail energy 2+2bß+c3²
15
(+
Selection with guardrails:
17 foreach 3 E C do
3 + ( Smin), Pmax] 3 + p min ]
19
1/ shrink & cap for stability
20
objective = rat 10 then
(B +- a 2b,3 cg2
// approx. tail/total ratio A3)
2B3 + C,
22
else
23
(co + a + 263 + c32
// tail energy
24 objective = ratio then
25
8* arg mingec score (8)
26 else
3 +- arg minsec (3)
Return: 3 and a' =8* =B1/(1 3*)
26


Reviews:
[{"review_id": "xwq7XDcZdU", "reviewer": "Reviewer_2kex", "summary": "This paper proposes SAES-SVD (Self-Adaptive Error Suppression SVD), a framework for compressing large language models (LLMs) using low-rank decomposition. The key contribution addresses a critical limitation in existing layer-wise compression methods: the accumulation and propagation of reconstruction errors across network layers.", "questions": "1. **Model scaling**: How does SAES-SVD perform on more recent and larger models? Specifically:\n   - Can you provide results on LLaMA 3.1 (8B, 70B) and Qwen 2.5 series?\n2. **Task diversity**: Can you evaluate on more challenging and diverse tasks? Choose 2 of this gourp please.\n   - Long-context reasoning tasks (>8K tokens)\n   - Code generation\n   - Mathematical reasoning\n3. **Practical integration**: \n   - Have you tested integration with popular inference frameworks?\n   - What modifications are needed to existing serving infrastructure?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. **Theoretically grounded approach**: The derivation of closed-form solutions based on second-order activation statistics provides a principled mathematical foundation. The formulation that combines local reconstruction with weighted cumulative error compensation is elegant and well-motivated.\n\n2. **Adaptive mechanism**: The ACES component that automatically adjusts weighting coefficients is a practical contribution that removes the need for manual hyperparameter tuning across different layers and models.\n\n3. **Consistent improvements**: The reported results show consistent improvements over baseline methods on the evaluated benchmarks, with the accuracy drop on LLaMA-7B at 0.2 compression ratio (0.02 vs 0.05) being noteworthy.", "weaknesses": "1. **Outdated evaluation benchmarks**: The datasets used for evaluation appear to be somewhat dated. Modern LLM compression research should include more challenging and recent benchmarks that better reflect current application demands and model capabilities.\n2. **Limited model coverage**: The experiments focus primarily on medium-sized models like LLaMA-7B. To demonstrate the method's generalizability and practical value,\n3. **Insufficient baseline comparisons**: The paper should compare against a broader range of compression techniques including recent quantization methods (e.g., GPTQ, AWQ, SmoothQuant).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W5gyHz0vOA", "reviewer": "Reviewer_5MPy", "summary": "This paper proposes SAES-SVD, motivated by the observation that SVD-based compression for large language models leads to cross-layer error propagation and accumulation. Implementation-wise, the authors gather streaming second-order statistics by running two forward passes on the same mini-batch, thereby avoiding activation caching and keeping overhead manageable. Experiments span LLaMA-7B/13B/30B and LLaMA-3-8B across multiple compression ratios. Under a unified protocol that requires no finetuning and no mixed-rank assignment, the method consistently reduces perplexity, maintains or improves zero-shot accuracy, and delivers ~1.29×–3.79× inference speedups, demonstrating strong practicality and scalability.", "questions": "1. please refer weaknesses.\n2. Please explain the differences and advantages of this article compared to the method in \"AA-SVD: Anchored and Adaptive SVD for Large Model Compression\", which is also submitted to ICLR 2026.\n\nI'm willing to raise my score if my concern is resolved.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Compelling motivation on cumulative error. The paper identifies a real pain point in SVD compression for LLMs, cumulative cross-layer error during inference—and directly targets it. The motivation is well supported by the empirical evidence in Figure 1, which demonstrates the phenomenon clearly.\n\n2. Solid theoretical underpinnings (CEALC & ACES). Both components—CEALC and ACES—come with clear formulations and derivations. The overall approach is coherent: the objective design is principled, and the analysis provides a sound theoretical basis for the proposed procedure.\n\n3. Thorough and convincing experimentation. The method is validated across multiple models and datasets, generally achieving better accuracy and lower perplexity while also reducing end-to-end latency. The breadth of settings and the consistency of gains add credibility to the claims.", "weaknesses": "1. Theoretical limitations and missing robustness analyses.\nThe fixed-subspace approximation used by ACES may break down under small spectral gaps or large perturbations. The current mitigation (β caps and shrinkage) is largely engineering-based. The paper would benefit from robustness curves bucketed by spectral gap, as well as a deeper theoretical justification for using RER and an explicit discussion of how RER improvements translate to final PPL.\n\n\n2. Limited architectural diversity in experiments.\nEvaluations focus primarily on the LLaMA family. Results on other architectures (e.g., Qwen) are missing, which leaves open questions about generality across model designs.\n\n\n3. No combination with mixed-rank strategies.\nAlthough the paper claims to outperform mixed-rank baselines under a uniform-rank setting, it does not explore combining SAES-SVD with mixed-rank schemes (e.g., ASVD, Dobi-SVD). Whether such combinations could further improve performance remains unaddressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QuoxIPiI4P", "reviewer": "Reviewer_khS9", "summary": "This paper proposes SAES-SVD (Self-Adaptive Error Suppression SVD), a framework that jointly optimizes for local reconstruction fidelity and global error compensation. It consists of two core componets: CEALC actively compensates for upstream accumulated errors by aligning each layer’s output with its full-precision counterpart; ACES automatically tunes the error compensation strength for each layer to enhance the low-rank structure of the objective. These components together mitigate the critical issue of error propagation inherent in layer-wise compression. Experiments show that SAES-SVD significantly outperforms existing methods, substantially narrowing the performance gap with the original model without requiring any post-compression fine-tuning.", "questions": "- Detailed time breakdown:\n Could you provide a comprehensive breakdown of the total compression time (including statistics collection and ACES optimization) and compare it with baseline methods?\n- Combination with quantization:\n Have you considered integrating SAES-SVD with quantization techniques such as GPTQ or AWQ? Since these methods address orthogonal types of redundancy (structural vs. numerical), such a combination could potentially achieve even higher compression efficiency and better performance.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Adequate experiments and compelling results:\n  - The experiments are comprehensive, covering multiple models and scales. The results are compelling, consistently outperforming strong SVD baselines (even those requiring fine-tuning), which effectively highlights the method's superiority.\n- Systematic and interpretable methods:\n  - The proposed method is theoretically sound and well-motivated. It systematically addresses the error accumulation problem, and the ACES component provides an elegant, closed-form solution for adaptive tuning, making the framework interpretable and efficient.", "weaknesses": "- Lack of computational complexity and time analysis:\n The paper does not provide a detailed evaluation of the computational overhead during the compression process. The time cost of statistics collection and ACES optimization, relative to baseline methods, is not quantified precisely.\n- Limited comparison beyond SVD-based approaches:\n The evaluation focuses only on SVD-based baselines. It remains unclear whether the proposed method would still outperform non–SVD-based compression methods under the same compression ratio.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BRLnO71OCC", "reviewer": "Reviewer_kM1J", "summary": "This paper introduces the SAES-SVD framework, which leverages two key modules - Cumulative Error-Aware Layer Compression (CEALC) and Adaptive Collaborative Error Suppression (ACES) - to overcome the limitations of prior SVD compression methods that only independently minimize reconstruction error for individual layers while ignoring the layer-wise propagation and accumulation of compression errors throughout the model network. By dynamically optimizing weighting coefficients for each layer, the framework maximizes retained energy under fixed rank budgets, thereby providing a more effective LLM compression strategy.", "questions": "1. Have the authors tested SAES-SVD on other popular LLM architectures beyond the LLaMA family?\n2. How does SAES-SVD perform compared to other major compression paradigms such as structured pruning and quantization methods?\n3. There appears to be an inconsistency between Figure 3 and the \"Comparison on larger-scale models\" section in Section 5: while the text claims Dip-SVD achieves 6.64 perplexity on LLaMA-30B, Figure 3 shows this result missing for LLaMA-30B and only displays 6.64 for LLaMA-13B. Could the authors clarify this discrepancy? On a related note concerning the same figure, could the authors also explain the anomalous trend for the ASVD method, where its performance degrades significantly on the larger LLaMA-30B model, contrary to the behavior of other methods?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed SAES-SVD framework is well-motivated and novel, addressing a fundamental yet previously overlooked limitation in prior SVD-based compression methods.\n- This paper demonstrates exceptional mathematical rigor, providing comprehensive theoretical derivations for both CEALC and ACES components, which elevates the work from empirical observation to a theoretically grounded advancement.\n- The framework achieves strong performance without any additional training, showcasing its straightforward and effective design.", "weaknesses": "- The proposed method is only evaluated on the LLaMA family of models, raising concerns about its generalizability and effectiveness across other popular LLM architectures such as Qwen.\n- The paper only compares with SVD-based methods, lacking comparison with other compression approaches like structured pruning and quantization, which limits understanding of its overall effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)