GlowQ: Group-Shared LOw-Rank Approximation for Quantized LLMs
Authors: Selim An
Il hong Suh
Yeseong Kim
Venue: ICLR 2026 Poster (2026)
Keywords: Quantized large language models
low-rank error correction
group-shared factorization
randomized SVD
selective restoration
low-latency inference
Primary area: optimization

Abstract: Quantization techniques such as BitsAndBytes, AWQ, and GPTQ are widely used as a standard method in deploying large language models but often degrades accuracy when using low-bit representations, e.g., 4 bits. Low-rank correction methods (e.g., LQER, QERA, ASER) has been proposed to mitigate this issue, however, they restore all layers and insert error-correction modules into every decoder block, which increases latency and memory overhead. To address this limitation, we propose GlowQ, a group-shared low-rank approximation for quantized LLMs that caches a single shared right factor per input-sharing group and restores only the groups or layers that yield the highest accuracy benefit. 
GlowQ computes the high-precision projection once per input-sharing group and reuses it across its modules, reducing parameter and memory overhead, and retaining the expressivity of layer-specific corrections. We also propose a selective variant, GlowQ-S, that applies the cached shared module only where it provides the largest benefit. Compared with strong baselines, our approach reduces TTFB by \(5.6\%\) and increases throughput by \(9.6\%\) on average, while reducing perplexity on WikiText-2 by \(0.17\%\) and increasing downstream accuracy by 0.42 percentage points. The selective model GlowQ-S further reduces latency, cutting TTFB by \(23.4\%\) and increasing throughput by \(37.4\%\), while maintaining accuracy within 0.2 percentage points on average.

Published as a conference paper at ICLR 2026
GLOWQ: GROUP-SHARED LOW-RANK APPROXIMA- TION FOR QUANTIZED LLMs
Selim An
Ilhong Suh
Department of Artificial Intelligence
COGA robotics, Korea
DGIST, Korea
mhsuthoogg-zobtticc.com
phantom0 60dgist..... ac.. ar.kr
Yeseong Kim
Department of Electrical Engineering POSTECH, Korea yeseongkinmpoottchh.ccc
ABSTRACT
Quantization techniques such as BitsAndBytes (Dettmers al. 2022). AWQ (Li et al. 2024), and GPTQ (Frantar et al. 2022) are widely used as a standard method deploying large language models but often degrades accuracy when using low- bit representations, e.g., 4 bits. Low-rank correction methods (e.g., LQER (Zhang et al. 2024a). QERA (Zhang et al., 2024b), ASER (Zhao et al. 2025 has been proposed to mitigate this issue, however, they restore all layers and insert enror correction modules into every decoder block, which increases latency and memory overhead. To address this limitation, we propose GlowQ. group-shared shared low-rank approximation for quantized LLMs that caches a single shared right factor per input-sharing group and restores only the groups layers that yield the highest accuracy benefit. GlowQ computes the high-precision projection once per input- sharing group and reuses it across its modules, reducing parameter and memory overhead, and retaining the expressivity of layer-specific corrections. We also propose selective variant, GlowQ-S, that applies the cached shared module only where it provides the largest benefit. Compared with strong baselines, our ap- proach reduces TTFB by 5.6% and increases throughput by 9.6% on average. while reducing perplexity on WikiText-2 by 0.17% and increasing downstream accuracy by 0.42 percentage points. The selective model GlowQ-S further re- duces latency, cutting TTFB by 23.4% and increasing throughput by 37.4% while maintaining accuracy within 0.2 percentage points on average.
INTRODUCTION
As large language models (LLMMs) grow in width and depth, the cost of serving and adapt- ing them becomes primary bottleneck for real-world use. Compression by post-training quan- tization (PTQ) alleviates memory and bandwidth pressure without altering the model architec- ture, and has matured through methods such GPTQ Frantar al. (2022). AWQ Lin al. Dettmers et al. (2022). A complementary thread augments quantized weights with small high-precision, low-rank term so that W W, + AB and the inference output is corrected by adding A(BX X) /Zhang et al. (2024a c); Zhao et al. (2025): ; Zhang et al. (2024b). These lines enable competitive quality at quantized weights across modern transformer stacks.
BITS).BIT
Most low-rank compensation pipelines attach an independent (A, B) module to each layer pro- jection and evaluate the high-precision projection B. repeatedly along the network. This design (i) duplicates the same expensive computation for modules that ingest the same input tensor, (ii) increases memory traffic by materializing multiple B. values, and (iii) selects subspaces with ob- jectives that often ignore the strong anisotropy of real activations Ethayarajh (2019); Godey et al.
(2024), misallocating limited rank to rarely used directions. As a result, the accuracy accuracy-effeiennyy trade-off is weaker than necessary, especially under strict latency budgets.

---

Published as a conference paper at ICLR 2026
Sefect Decader Block
Group the errors and stack
Rondomized SVD
Inference path
Group Stacked
Non-Group path
..-Only once (per Group)
Y
Group path
B
QKV
A
Non-Group
AR
BX
Group Stacked
MEP
W
E
Non-Group
DOWN
A
Figure I: GlowQ Overview
We propose GlowQ, Group-Shared Shared Low-Rank Approximation for Quantized LLMs. As illustrated in Fig. [ GlowQ treats modules that share the same input as a group Vaswani et al. (2017). learns single shared right factor Bshared for that group, and keeps module-speciine inference, it computes It B X once per group and reuses it via A,R, turning many large BX multiplications into several cheap matrix-vector updates. align limited rank with how inputs are actually used, adopt covariance-alinne objective that emphasizes frequently visited directions. Finally, a Selective Restore policy enables only high-payoff groups or layers under a deployment budget. Godget.MMocchaoot Molchanov al. (2019)
left factors {A,]. At
When modules share the input dimension, the joint least-squares problem with a single right factor equivalent to approximating the vertical stack of module-wiss error matrices; its minimizers characterized by the right singular structure of the stacked matrix ("stacked SVD"). We then connect bsage-weightee risk minA.B Ecat A covariance-aligned objective whose global solution is governed by the SVD of the whitened error, where the whitened errors are those rescaled by the input covariance, This provides both rationale for shared B and a principled way to steer it toward data-preferred axes.
right-weighted Frobenius norm, which yields
To avoid forming tall whitened matrices, we introduce QR-reduced randomized SVD routine: thin QR compresses the stacked error into d x d core; randomized SVD with oversampling and power iterations extracts the dominant right subspace; balanced recovery returns (A*,, B*) with improved numerical stability. The solver drops into our grouping and caching runtime with no extra architectural changes.
Group-level shared- B. We formalize input-sharing groups and show that one shared right factor per group suffices for the joint least-squares objective, enabling one-shot B. and multi-module reuse (Sec.
Data-aware alignment. We derive a covariance-aligned objective by bridging usage- weighted risk and right- Frobenius criterion; its global minimizer aligns the shared right subspace with data-prefereed directions (Sec. (II..
Qe-reduced RSVD, We present a practical pipeline that performs QR reduction to a small core and applies randomized SVD with balanced factor recovery, avoiding tall whitened matrices while preserving accuracy (Sec. (.2).
Caching & Selective Restore. We implement a deployment path that caches R = Bahured X once per group and activates only important groups/layers, translating algorithmic savings into latency/throughput gains (Sec. 3.3).
Empirical gains over strong baselines. Across the evaluated model families and bench- marks, GlowQ consistently improves both efficiency and accuracy: it reduces time-to- first-byte (TTFB) by 5.6% and increases throughput by 9.6% on average, while reducing WikiText-2 perplexity by 0.17% and increasing downstream accuracy by 0.42 percentage points. The selective variant, GlowQ-S, further lowers latency, cutting TTFB by 23.4% and increasing throughput by 37.4 37.4%, while maintaining accuracy within 0.2 percentage points of full GlowQ.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Post-training quantization (PTQ). Today's PTQ methods span a variety of designs that re- cover accuracy at the quantization stage without changing the model structure or runtime path. @ et al. (2022) uses second-order information to directly fit quantized weights and pre- serve layer outputs; AWQ et al. (2024) protects important channels based on activation statistics via rescaling. In the BITSANDBYTES family, LLM.INTS Dettmers et al. (2022) uses vector-wise quantization with an outlier- aware mixed precision path where most channels run in INT8. These methods constitute standard baselines for LLM lightweighting.
Quantization error correction via low-rank compensation. Prior work shows that post- quantization errors can be effectively reduced by adding low-rank term to the quantized weights or outputs. LQFR approximates the per-layer quantization error as E zz A B and adds high-precision correction without changing the inference graph (Zhang et al. 2024a). ZEROQUANT- V2 system- atizes low-rank compensation (LoRC) within PTQ pipelines and demonstrates that small-rank correction can recover accuracy at low bit-widths (Yao et al. 2023). QERA derives closed-form, output-error-centric formulation that clarifies when low-rank correction benefits PTQ/PEFT (Zhang et al. 2024b). ASER combines a Chitened-SVD-see
low-rank corrector with activation smoothing to sta tiliziiie low-bit regimes (Zhao al. (025). While these works justify the AB correction prin- ciple, most deploy independent (Ar. BB) at every layer and recompute the high-precision product (B for all layers and tokens, which increases latency and memory traffic; moreover, attaching a low-rank module to every layer inflates GPU memory usage.
Stackee/collective SVD for a shared right subspace. The idea of factorizing multiple matrices with a shared latent factor established in collective/joint matrix factorization: when several ma- trices share the same input dimension, one can vertically concatenate their blocks and fit a single right subspace while allowing matrix-specific left factors (Singh & Gordon 2008). Recent analyses also study the optimal recovery of shared singular subspaces across matrices (Ma & Ma 2024). We adopt this principle for imput-sharing modules in LLMs we stack group-wise error blocks into Eeat and learn one Bahared per group. At inference, we compute the right projection once per group, R Batured X, cache it, and let each module apply only the lightweight left multiplication R, This reduces high-precisinn matmuls and the number of resident correction parameters compared to layer-wise independent AB.
Covariance-aligned selective restoration. Because inputs are anisotropic, plain stacked ob- jective may learn right subspaces misaligned with data-preferred directions. We therefore adopt covariance-aligned (whitened) formulation, measuring residual error in the input-covariance metric so that the shared subspace is guided toward meaningful axes (Golub & Van Loan 2013] Srebro & Jaakkola, (003). Not all layers require restoration; following pruning- inspiieed saliency, we acti- vate only the most beneficial groups under a budget, using (i) an SVD energy enery--capture score A|
per group), (ii) a normalized error ratio |/ ||W (Nagel et al. 2020 Banner et al. 2019). and (iii) layer-order fallback when signals are weak. Coupled with the sign and cached R = X,, this selective restore achieves stronger securacylatency-mmen trade-off's than per-layer low-rank baselines the same cost.
de-
METHOD: GLOWQ
In this section, introduce our method, Group-Shared Low-Rank Approximation For Quantized LLMs (GlowQ). Prior low-rank restoration often (i) restores all layers and (ii) multiplies a per- layer low-rank module with activations, causing heavy overhead. We address both by (a) learning a shared right subspace for modules that share the same input and (b) caching the input projection once per group for reuse. We approximate each error matrix and its vertical concatenation by a rank- factorization: E. RJ A,B and Erat R AB, where A [Ai....; Am) and B is shared within group. At inference, the correction for each module takes the form (BX),, where the projection BX is computed once for the entire group.
3.1 GROUPING QUANTIZATION
CORRECTION MODULES
We aim to find the optimal shared low-rank correction module, in particular a shared right factor B. To this end, first formalize the problem via unweighted baseline (Sec. 3.1.1) and propose
3

---

Published as a conference paper at ICLR 2026
MLP QKV
1.5
QKY MILP
Eizen Valee
y 41704/066
-5
lega
(a) Eigenvalue spectrum
(h) Eigen Value spectrum log scaled
Explaned Vartine Mart
Exphited Variunce Ratin
066
0.4
0.4
0.2
Cor allen
No cmn Cov aligs
100
Roak
(c) Energy capture of Q.K.V
(d) Energy capture MLP
Figure 2: Input spectrum and energy-capture measurements. (a) We stream calibration samples through the model, collect the input activations at the target layer, and plot the eigenvalue spectrum of the empirical input covariance for the QKV and MLP groups, revealing heavy-tailed profile. (b) The same spectra plotted in log Ay-log Y' coordinates; dotted lines show least-squares fits over the approximately linear tail region, indicating power-law decay Ay X r- with exponents OMLP R 0.77 and dookv 1.19. (c-d) For each group, vertically stack the quantization-era plot the cumulative fraction of Frobenius energy recovered by the best rank-r approximation. We show both the unweighted baseline (No cov) and the covariance aligned variant that weights errors by the observed inputs (Cov align). Horizontal dashed lines mark 90% and 95% energy capture.
matrices and
data-aware objective that incorporates covariance alignment to overcome the limitation induced by input anisotropy (Sec. 3.1.2).
3.1.1 UNWEIGHTED BASELINE: STACKED SVD
Let modules i = =1 1,....,W m share the same input dimension d. For error matrices E, E xd define the vertical concatenation
Ecat := [ET
E d [ O.),
A = [
A E (E,0).
(1)
m
We seek shared right factor B E Rexd and blocks A; R that minimize
min A.B
E Beat
ABI
Proposition 1 (Shared-B is optimal). For modules that share the same input,jointll fitting with a single right factor B is equivalent to one low-rank fit of the stacked matrix Eest- By Eckart-Young- Mirsky, an optimal B spans the top-r right-singular subspace of Eeat: allowing mor-mo B, adds no extra expressivity because any differences can be absorbed into invertible reparameterizations of A,. Hence, a single shared B is sufficient and optimal for the group. (Proof and identifiability details are deferred to Appendix[A.1
Real inputs are anisotropic, which can be diagnosed by the eigenvalue spectrum of the covariance Ez. Fig. 2a exhibits a heavy-tailed profile, with an abrupt initial drop followed by long tail, indicating that the usage of the representation space is strongly concentrated in small number of axes. Under such distribution, the relative importance between frequently used directions and the remaining ones diverges markedly. To quantify this behavior, Fig. 2b plots the eigenvalue spectra of the empirical input covariance for the MLP and QKV groups in log A 7 scale: for each group sort the eigenvalues (),) in descending order and plot log X, versus log r. The dotted
4

---

Published as a conference paper at ICLR 2026
lines show least-squares linear fits over the approximately linear tail region, revealing power-law decay X x r with exponents OMLP z 0.77 and aokY 1.19, which quantitatively confirms the heavy-tailed, anisotropic input statistics that motivate covariance-aligna objective.
However, the unweighted cluster SVD selects the shared right subspace purely from the geometry (variance structure) of the error matrices. This can misalign the selected subspace with the axes preferred by the data; at a fixed rank, such a misalignment reduces energy capture and weakens consistency within the group. Alignment arises naturally only in restrictive cases, such as isotropic input or rear-simultaneous diagonalization.
Therefore, treat anisotropy fairly, should evaluate errors in coordinate system where all directions carry equal usage. In such space, frequently used directions are not ander-weighten and rarely used directions are not over-weighted, so the learned shared right subspace aligns better with the data-preferred axes.
3.1.2 D COVARIANCE ALIGNMENT
The evidence in Fig. 2a shows strong input anisotropy; hence reconstruction should account not only for the geometry of error matrices but also for how inputs actually used. For any factors (A, B) and residual M : Eeat
AB, the expected loss under the usage distribution is
E M trMMEt MT) = |M //
which follows from the standard quadratic-form identity together with the Frobenius-trace iden- tity (Petersen & Pedersen 2006). To balance direction-wise usage, we whiten by >/ that the selected shared right subspace is steered toward axes preferred by the data.
Using the definitions from Sec.
3.1.1 we adopt the weighte objective
min || (Eeat AB) 2 2
min A.B
E AB
E = Ecat EV/
A.B
Fd
In the isotropic (Ex T). Eq. reduces to the unweighted baseline in Sec. 3.1.1
Empirically,
Fig. [2c and andodsshows that, at a fixed rank, whitening yields substantially faster growth of the cumu- lative energy capture compared the unweighted variant, indicating better alignment of the learned shared right subspace with data-pre Sata-preferred directions.
Proposition 2 (Usage-weighted risk equals a right-weighted reconstruction error). When in- puts are centered and have covariance Ex. the model's 's expected loss equals the residual energy aver- aged over draws from the input distribution. Equivalently, it the residual measured after weighting columns according to how frequently and how strongly each input direction is used (as determined by Σx). Therefore, minimizing the usage-weighted risk is exactly the same optimization as min- imizing the right-weighted reconstruction error in Eq. 4 full derivation and the nonzero-mean case are deferred to Appendix A.2
Proposition (Covariance-aligned minimizer) The global minimizers (*) of Eq. are given by the rank-r SVD of the whitened error matrix E Ecat ///² In particular, the opti- mal shared right subspace, row(B*), is spanned by the top-r right singular vectors of E; this is the standard Eckart- Young -Mrssky solution specialized to the whitened problem (Eckart & Young. 1936)
Golub & Van Loan 2013
3.2
SCALABLE IMPLEMENTATION VIA QR-REDUCED RANDOMIZED SVD
We present an implementation that solves the covariance-alignee objective
min Ecat AB) 11/2 2
(5)
A.B
without forming the tall whitened matrix. The method follows three steps: (i) QR reduction to compress the tall matrix into d x d core, (ii) Randomized SVD (RSVD) the core to capture the top-r right subspace, and (iii) balanced recovery to obtain BB). This yields practical ad- vantages such avoiding materialization of huge matrices, lower compute/memory cost, improved numerical stability via balanced factors, and direct compatibility with the caching/Selectiv--Restoo pipeline in Sec. 3.3
5

---

Published as a conference paper at ICLR 2026
3.2.1 ALGORITHM & COMPLEXITY
Using thin QR of Ecat, we reduce the covariance-aligned objective to d x d core (Alg. D full SVD on the core costs (dd), whereas randomized sketching recovers the leading right subspace in (r+p) + (x+p)) time. Here, p denotes oversampling (extra sketch columns) and q denotes the number of power iterations used to sharpen the subspace.
Algorithm Covariance-alignel QR reduction and randomized SVD on the core
Require: Stacked error Ecat Rmxd, covariance Σx 0, target rank r, oversampling p, power
iters !
Ensure: Low-rank factors (A+, B*) for the covariance-alignen objective
1: Thin QR of Ecat: compute Q.R. = Eeat with Q Qe = La
2: Core construction: set M +- R. Σx/² E IRdxd
3: Random sketch / range finding: draw 52 ~ NoO, 1) 1)(x( set Y + MΩ; optionally do
power steps + M(M Y)
4: Orthonormalize: Q 4 orth(Y) Y) E [xx((+t)
M: compute Bsmall ÜEVT 6: Lift left factor: U f- QU
5: Compressed SVD: B small < Q
7: Truncate (top-r) & balance: keep (U,, E,, V,) and set A + U, U.E! B'
1/2 use pseudoinverse if Σx is 8: Lift to original variables: A Q. A', B X singular
B
By left-orthogonal invariance of the Frobenius norm, the QR reduction collapses the tall-m problem to d x d core without loss for the covariance- aligned objective (formal proof in Appendix A.3
(Golub & Van Loan 2013)). Randomized sketching on the core provides an efficient and accurate estimate of the leading right subspace with controllable bias via (p, q): we summarize theoretical guarantees in Appendix A.4 and present empirical runtime-accuracy trade-offss (vs. exact SVD) in
Sec. D 1 HHlkoo al. 2011 Balanced recovery yields
A = 1
B - -V
(Eest A ABE'
M U,E.V
(6)
and the resulting B* serves the shared right factor used in Sec. 3.3 for once-per-group caching (R = X) and Selective Restore.
3.3 CACHING AND SELECTIVE RESTORE
The group-shared factorization implies that modules within the same input-sharing group all rely the same right-side projection X B S.hareed Naively evaluating this projection for every module recre- ates the primary inefficiency of layer-wise correction, i.e., multiple bigh-precision matrix-vector multiplications along the critical path. To translate the theoretical shared structure into practical inference gains, GlowQ introduces a caching mechanism that computes the right-sided projection once per group, and complementary selective-restore policy that activates correction only at groups offering the largest accuracy benefit under a deployment budget.
For each layer group G3 that shares the same input dimension, compute single intermediate
RS = X shared E
RR BxTxr
(7)
per group and reuse it across all modules in the group. Each module i Ge then applies only the small correction
y/ w x Re,
(8)
where Ar, goxr and B shared E grxi We adopt anchor policy to materialize R& exactly once and consume it a fixed number of times: in attention, q is the anchor and (k, v) are consumers; in MLP, gate is the anchor and up is the consumer. Solo modules that do not share inputs (e.g., o.proj, down-proj) compute R, == BBB on the fly without reuse.
Given latency memory budget, rank all candidate units (groups solo layers) by im- portance score and activate only the top k. Importance is measured using two metrics: a GSVD- based energy-captur score (Eq. g after covariance alignment (Paige & Saunders 1981
Jolliffe &

---

Published as a conference paper at ICLR 2026
ppl_ gsvd
Ppl. NER
ttfb NER
_EE__NER
ppl. pyvd
ppl NER
tif cosine
8.5
PPI.& TTFD Saatin-nua %60
100
7.23
PPI.& (min-max
80
8.43
80
7.19
60
8.36
ppt
7.15
40
8.29
40
20
8.22
20
7.11
8.15
7.07
10
30 50 60 70 80
90
100
50
100
Restored (roup(%)
Restored (ro)p(%)
(a) LLaMA 3.2-3B
(b) Qwen 2.5-7B
Figure 3: Perplexity (PPL) and time-to-first-byte (TTFB) versus the fraction of restored groups.
Cadima 2016; Halko al., 2011), and a normalized error ratio (Eq. I0 Malinovskii et al. 2025 Dong et al. 2019). At runtime, we apply the cached low-rank correction only to the selected units, skipping inactive ones; because the cache materialized only for active groups, selective restore naturally complements group-shared caching.
Joe (u) Σラ=1 T. (M,)²
E 11..
(9)
Juer (u)
(10)
M
I
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We evaluate LLaMA (3.2-3B, 3.1-8B) Dubey et al. (2024), LLaMA (7B, 13B)
Touvron et al.
2023). Qwen 2.5 (7B, 14B) Qwen al. 2025). Qwen 3 (8B, 14B) Yang et al. (2025). OPT (I.3B,
6.7B Zhang et al. (0022). Mistral 7B Jiang et al. (2023), and Qwenl.5-Mot- M-MMoB-A2 A2.77B
Bai et al.
2023a) with Vieuna reported only in ablations.
All models use W4A16 (int4 weights, fp16 activations) with group size 128; the rank is fixed at 64. Calibration uses 64 sequences of length 2048 shared across methods, with no fine-tuning unless a baseline requires it. We compare Glow and GlowQ-S with various state-of-the-art baselines, including PTQ (Bits/AndBytes Dettmers al. 2022 AwQLin et (2024) GPTQ Frantar et al. (2022) and error-correction methods in literature (L2QER Zhang et al. (202a)). ZeroQuant V2 Yao ct al. 2023 QERA Zhang et al. (202b)). ) All under the same protocol and recommended defaults.
We report perplexity on WikiText-- Merity et al. (206)) and C4 Raffel et al. (2020), and zero-shot accuracy on ARC-E/ARC-C Clark al. (018). PIQA Bisk al. (2020). HellaSwag [Zellers et al. (2019) WinoGrande (Saaauuhi et al. (2021), BoolQ Clark et al. (2019), and LAMBADA Paperno ct al. (2166) via en-eval- waral-hrrnes (defaults). We run the proposed method on A 100 GPUs for covariance/SVD steps while inference is executed on an RTX 4090.
GlowQ-S Configuration GlowQ-S applies the cached correction only to subset of groups, lected according to an importance score. Since different model families exhibit distinct restoration profiles, we adopt model-specific scoring rule for GlowQ-S. We defer the full characterization of these curves and the selection policy to Section 4.6
4.2 MAIN RESULTS: PERPLEXITY AND ZERO-SHOT ACCURACY
Perplexity. Table reports test perplexity (lower is better) for WikiText-2 under a common proto- col: W4A16 with int4 weight groups of 128 and a shared calibration set of 64 sequences at length 2048 for all methods. Overall, GlowQ achieves the best or tied-best perplexity on of model variants, including consistent gains on LLaMA 3 (3.2-3B/3.1 1-B),, Qwen 3 (8B/14B), Qwen 2.5-7B, and Mistral-7B. On Qwen 2.BB GlowQ matches the strongest baselines. Exceptions occur on LLaMA 2-13B (where L2QER slightly leads) and OPT-1.3B (where QERA leads), while OPT 6.7B favors a pure PTQ path. These outcomes indicate that croup-sharee low-rank correction closes much of the int4 gap FP16 across diverse architectures without task-specific tuning. Beyond the W4A16 setting, the lower block of Table evvaluates mixed-precisse weight- activation quantization with W4A4 and W4A8. As expected, W4A4 increases perplexity for all methods, but GlowQ (and
7

---

Published as a conference paper at ICLR 2026
Table WikiText-2 test perplexity (lower is better). GlowQ-S restores 51% of layers for LLaMA 3.2-3B, while all other models use 50% restoration.
Method
Q config
ILaMA 2
I.LMA 3
Qwen 2.5
Qwen
Mistral
OPT
7B
13B
3.2-3B
3.1-8B
7B
14B
8B
14B
7B
1.3B
6.7B
FP16
5,48
7.81
6,24
6.86
5.29
9.73
8.64
5.32
14.62
10.85
BnB
NF4
5.64
4.97
8.29
6.66
7.10
5.64
9.97
8.88
5.51
15.16
10.94
AWQ
INT4, g128
5.61
4.97
8.24
6.64
7.11
6.17
10.19
9.00
5.51
15.22
11.23
GPTQ
INT4, g128
5.65
5.35
9.46
6.63
7.11
5.75
9.98
8,90
5.51
15.00
11.07
ZeroQuant V2
INT4, g128
5.72
4.99
8.44
6.79
8.41
5.75
10.19
9.04
5.53
15.10
11.14
QERA
INT4, g128
5.61
4.98
8.22
6.64
8.09
5.64
10.07
8.85
5.48
14.85
11.00
1,2QRR
INT4, g128
5.68
4.94
8.30
6.75
8.14
5.66
10.07
8.85
5.46
15.30
11,16
GlowQ
INT4, g128
5,58
4.96
8,16
6.59
7.07
5.64
9,90
8.80
5.42
14.84
11.00
GlowQ-S
INT4, g128
5.60
4.96
8.22
6.62
7.09
5.68
9.97
8.89
5.45
15.00
11.00
1.QQRR
W444
5.90
5.18
9.42
7.65
9.11
6.52
10.76
9.36
5.73
27.40
11.32
L2QER
W4A8
5.69
4.95
8.31
6.76
8.15
5.67
10.11
8.86
5.47
14.90
11.00
GlowQ
W444
5,90
5.20
9.21
7.42
8.03
6,55
10.66
9.33
5.74
26.35
11.31
GlowQ-S
W4A4
5,92
5.20
9.25
7.45
8.05
6.61
10.72
9.37
5.79
27.42
11.33
GlowQ
W4A8
5,59
4.97
8,20
6,63
7.12
5.71
10.08
8.85
5.43
14.85
10.97
GlowQ-S
W4A8
5.60
4.97
8.24
6.64
7.13
5.77
10.10
8.92
5.48
14.99
10.99
GlowQ-S) remain competitive with better than L.2QER on most models, and the W4A8 configura- tion nearly recovers the W4A 16 accuracy, indicating that our covariance-aware low-rank correction continues be effective even under joint weight-activation quantization.
Table 2: Average accuracy (f) on seven downstream tasks and C4 perplexity (L).
LLaMA 3.2-3B
LLaMA 3.1-8B
Qwen 3-8B
Qwen
Method
Rank
Acc (t)
C4 (L)
Acc (f)
C4
Acc (t)
C4 (l)
Acc (*)
C4 (L)
67.14
9.00
71.48
14.52
ZeroQuant-Ve V2
65.38
11.45
73.48
9.87
70.19
15.00
72.62
13.79
QERA
65.48
11.04
72.86
9.68
69.86
14.78
73.14
13.29
L2QER
66.19
11.04
72.43
9.63
69.52
14.82
73.24
13.80
GlowQ
66.90
10.98
73.33
9.59
70.71
14.60
73.84
13.26
GlowQ-S
66.33
11.07
72.62
9.78
70.29
14.77
73.24
13.48
Overall quality. Table 2 reports the zero-shot accuracy via Im-eval-harness along with the per- plexity for the C4 dataset. Across four representative models (LLaMA 3 3.2-3B / 3.1-BB, Qwen 14B), GlowQ attains the lowest C4 perplexity among quantized/error-corees livers the strongest average zero-shot accuracy on LLaMA 3.2-3B and Qwen 3-8B/14B (ZeroQuant- V2 leads on LLaMA 3. 1-B). GlowQ improves over the best non-Glow baseline in the zero-shot accuracy by average +0.3%; in C4 perplexity, GlowQ improves by -0.2 ppl on average. Relative to FP16, the remaining C4 gap is +0.4 ppl on average, while average accuracy remains close to FP16 across the board. The selective-restore variant (GlowQ-S) shows the expected efficiency trade-off: -0.55% average accuracy and +0.15 ppl on average in C4 compared to GlowQ.
methods and de-
4.3 LATENCY AND THROUGHPUT BENEFITS FROM CACHING AND SELECTIVE RESTORE
Latency on LLaMA models. Under a common generation protocol (3 prompts, batch=1, max_new_too www.tokens=128 repeats=1, num beamsel and custom CUDA W4A16 kernels, we measure TTFB via a warm-start generate (max.new.tokens) CUDA events (). We establish our baseline using standard Layerwise method, which does not employ caching. This setup ensures a fair comparison, as both the Layerwise baseline and GlowQ utilize the identical custom CUDA W4A1 kernels compiled with the same optimization
and por-token decode latency using
8

---

Published as a conference paper at ICLR 2026
Table 3: Latency comparison LLaMA 2 models for Layerwise vs. GlowQ, GlowQ-S.
Models
Setting
TTFB(uns)
tok/s
Prefilliams) i
Dec(mstok) L
Layerwise
88.45
15.66
95.13
63.17
7B
GlowQ
82.66
17.12
92.23
58.32
GlowQ-S
66.68
21.16
72.35
45.90
LLaMA 2
Layerwise GlowQ
128.70
11.22
141.76
85,91
13B
122.78
136.53
81.15
GlowQ $
100.17
15.68
112.09
62.98
Avg. À BX (%)
+9.61
-3.37
-6.61
Avg. A R50 (%)
-23.39
+37.44
-22.44
-27.01
level, isolating the algorithmic impact of our caching strategy. Compared to this Layerwise base- line, GlowQ consistently reduces end-to-end latency across both sizes: on average TTFB drops by 5.51%, prefill time by 3.37%, and decode latency by 6.61%, yielding 9.61% increase in throughput (tok/s).
Selective restore efficiency. The GlowQ-S, which are restoring about half of the units by an im- portance score, amplifies the gains: average TTFB, prefill, and decode fall by 23.39%. 22.44%, and 27.01 %, respectively, and throughput increases by 37.44% over the Layerwise baseline.
4.4 MEMORY OVERHEAD AND EFFICIENCY ANALYSIS
On memory, GlowQ consistently uses less addi- tional GPU memory than layer-wise restoration at the same rank This follows from maintaining single shared right factor Bstured per input-sharing group and computing R X X once per group for cache-and-res reusee.. Applying GlowQ-S further re- duces overhead, yielding the flattest growth slope even at higher ranks. On accuracy, under an equal- memory budget Fig. (b)) GlowQ attains the low- est PPL, while GlowQ-S preserves PPL close to full GlowQ with substantially lower memory, consis- tently outperforming than layer-wise methods. Con- sequently, GlowQ is the preferred choice when max- imizing performance within fixed memory bud- get, whereas GlowQ-S offers a strong performance- efficiency compromise when memory constraints are tighter or latency minimization is prioritized.
[1 [Layeevse Blloeo) BGGm@0995
5.5
Miltooal Manuee Naage (503
5,4
6%%
5%
14%
5%
2%
1%
Zaceliant 12
QURA
L20ER
Glorg
Gheett
16
32
64
128
Rank (r)
(a) Memory overhead
(b) PPL
Figure 4: Comparison of memory and per- formance trade-off. (a) Memory overhead
of different methods. (b) PPL equal memory budget.
4.5 COMPATIBILITY WITH PTQ METHODS AND GENERALIZATION TO MoE
We further examine the compatibility of GlowQ with diverse LLM configurations, focusing in par- ticular on PTQ baselines and MoE architectures, as summarized in Table Layering GlowQ on top of PTQ baselines reduces perplexity by -0.59 ppl on average for GPTQ and -0.13 ppl on average for BnB. Improvements hold across both evaluated models in each setting, indi- cating consistent add-on gains independent of the underlying quantizer. GlowQ acts as an orthog- onal, plug-and-pin changes a small set of shared parameters for ac- curacy gains while remaining compatible with di- verse PTQ pipelines.
Table 4: Perplexity (L) on Wikitext-2 with and without GlowQ: dense models (top) and Qwwnl.5-MoF....... A.M--A2.27 (bottom).
Method
LLaMA 2-7B LLaMA 3.2-3B
GPTQ
5.64
9.32
low-rank correction: it ex-
+GlowQ
5.60
8.19
BnB
5.64 5.57
8.29
+GlowQ (cn BnB)
8.10
FP16
Quant only
GlowQ
Layerwise
On this MoE benchmark. GlowQ largely recovers the Wikitext-2 perplexity loss from 4-bit weight quantization and ends up only +0.02 PPL. worse
Qwent A.Mo8
7,22
7.70
7.41
7.39

---

Published as a conference paper at ICLR 2026
than the more expensive layer-wise low-rank baseline. The layer- wise variant attaches a separate error-correctoo module to every expert. whereas GlowQ uses a single shared right factor sttard per group across experts and the shared MLP. The whitening- based alignment heatmaps in Fig. 7 and Fig. [ show that expert-specific error subspaces are well aligned with this shared right subspace, explaining why the shared-B3 design can match layer-wise accuracy while reducing the memory footprint of the low-rank correction by about 63%. These results confirm that GlowQ remains ef- fective even on large MoE architectures.
Given the recent trend toward rotation-based saliency-aware PTQ and KV-cache compression, GlowQ can be viewed as a complementary low-rank correction layer that may be attached to strong PTQ baselines such as ROSAQ Yoon et al.] (2025) and GuidedQuant Kim et al. (2025), and fur- ther extended to VV-ccche compression frameworks like Comm Li et al. (2025) exploring such combinations remains an interesting direction for future work.
4.6 BEHAVIOR OF SELECTIVE RESTORATION ACROSS MODEL FAMILIES
Fig. plots PPL and TTFB as function of the restored fraction. On LLaMA 3.2-3B, PPL stays relatively flat and then exhibits an abrupt drop at an elbow point, after which marginal gains saturate quickly. In contrast, Qwen 2.5-7B shows a more gradual, near-monotone PPI. decrease with increas- ing restoration, without a clear knee. Since TTFB generally grows with the restoration fraction, these shapes motivate different selective-restoration budgets. We verify that these family-specific tendencies persist across other sizes within each family in our ablation study (Sec.[G.
Guided by the above curves, GlowQ-S restores for LLaMA, the elbow (steep-drop) operating point to capture most PPL gains with limited overhead, and (ii) for Qwen, a fixed 50% of groups, which offers stable accuracy improvements with moderate TTFB growth. For unit ranking, we follow the importance metries delineated in Sec. 3.3 covariance-aware error capture is adopted as the default criterion. For the model families, when two alternative metrics are available (covariance- aware eπror capture vs. normalized error ratio), we evaluate both on the validation split and, per model, adopt the metric that yields the stronger outcome; all reported results use this per-model best choice.
4.7 IMPACT OF COVARIANCE ALIGNMENT ON ACCURACY
On C4 (Table 5). the E. weeiwhitt Whitened SVD consistently outperforms the unweighted Stacked SVD across both laser-wise and group-shared vari- ants of Qwen 3-8B. Because the unweighted objec- tive ignores the unput-usage distribution embodied in Ex, it tends to select right subspaces misaligned with the axes most exploited by the data, leading to marked degradation in perplexity at fixed rank; whitening, by evaluating errors in a data-aligned coordinate system, improves energy capture and yields lower PPL. Grouped restoration also domi- nates layer-wise under both weightings, and, taken together, these results identify White + Group as the preferred configuration.
Table 5: C4 Evaluation of wei-hhtt (Whitened SVD) vs. unweighted (Stacked SVD) on Qwen 3-8B; lower is better (L).
No-White
White
Layer
Group
Layer
Group
14.97
14.60
13.85
13.40
CONCLUSIONS
We introduced GlowQ, group-shared low-rank approximation for quantized LLMs that replaces per-layer correction with a single right subspace shared among input-sharing modules and cache- and-reuse runtime. By connecting usige-weighted risk to rightt reconstruction objective, our covariance-aligned (whitened) formulation steers the learned subspace toward data-preferred di- rections, and a Qe-reduced randomized SVD provides an efficient, scalable solver. The deployment path computes one right-side projection per group and reuses it across modules, while a selective policy (GlowQ-S) activates only high-importance units under latency or memory budgets. Across modern model families and PTQ baselines, GlowQ consistently lowers perplexity, reduces time- to- byte, increases throughput, and decreases memory overhead relative to layer-wise correc- tion; whitening and grouping combine to yield the strongest results. The approach is architecture- agnostic, drop-in at inference, and complementary existing PTQ pipelines.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (RS-2025-24803164) This research was also supported by the Min- istry of Trade Industry Energy (MOTIE, Korea), under the Technology Innovation Program titled "Development of Navigation Technology Utilizing Visual Information Based on Vision- Language Models for Understanding Dynamic Environments in Non-Learned Spaces" (Project Number: RS- 2024-00445759).
REFERENCES
Eduardo Alvarez, Omri Almog, Eric Chung, Simon Layton, Dusan Stosic, Ronny Introducing NVFP4 for efficient and aceu-
Krashinsky, and Kyle Aubrey. rate low-precision inference.
June 2025. NVIDIA Technical Blog.
T.W. Anderson. An Introduction to Multivariate Statistical Analysis. Wiley Series in Probability and Statistics. Wiley, 1984. ISBN 9780471889878. URL https://bbote goog aeooogle... ee.co. co.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang. Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang. Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023a. URL aittps://arae wrg/ /2399. 16609
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu. Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023b.
Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for rapid-deployment. Proceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2019. Curran Associates Inc.
A. Ben-Israel and T.N.E. Greville. Generalized Inverses: Theory and Applications. CMS Books in Mathematics. Springer New York, 2010. ISBN 9781441918147. URL http:://books..
google. co. kr www...kbbookk..ccoooooooooooo 78HKGAAAA8KTAAAAAA
Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference Artificial Intelligence, 2020.
Åke Björck. Numerical Methods for Least Squares Problems. SIAM, 1996. doi: 10.1137/1. 9781611971484
Jingfei Chang, Yang Lu, Ping Xue, Yiqun Xu, and Zhen Wei. Iterative clustering pruning for convo- lutional neural networks. Know. Based Syst. 265(C), April 2023. ISSN 7051. doi: 10.1016/ j.knosys.2023. 10386. URL. http:/ttts:::oo.rrggggt
100366
101 kno...oss........oooooooooooo... 2023. 110386
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL htt.h:tts/////
.rr.iiooo.ooo.oooooo.ooooo..oo.. org/ abs 803 05457

---

Published as a conference paper at ICLR 2026
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int00: 8-bit matrix multi- plication for transformers at scale. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc.
ISBN
Zhen Dong, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precisiog Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2019.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, and et al. The Ilama 3 herd of models. In arXiv preprint arXlv:2407.21783 2024. doi: 10.48550/arXiv.
2407.21783.
Carl Eckart and Gale Young. The approximation of one matrix by another of rank. Psychome- trika, 1(3)2211-218, September 1936. doi: 1077077770222...... URL httts://idee:
Tepec.org/a/sat httpg/////rr/////////// wttp:ppppppppppppppppps/opcpctpp http:hhoovvv111111311
html
Kawin Ethayarajh. How contextual are contextualized word representations? comparing the ge- ometry of bert, elmo, and gpt-2 embeddings. In Conference on Empirical Methods in Natural Language Processing, 2019. URL http::tt::///.. :/aapi//appp.. sew.ticccooooooooggggoooooooo
202 202120592
Ky Fan. On a theorem of weyl concerning cigenvalues of linear transformations. ii. Proceedings of the National Academy of Sciences of the United States of America, 36(1):31-35, 1950. ISSN 00278424, 10916490. URL http::/tt//wet ott.:///tooo.rrr/ttaee////eee
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training compression for generative pretrained transformers. arXiv preprint 23X11:2210.17323, 2022.
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference, 2021.
Nathan Godey, Éric de Clergerie, and Benoît Sagot. Anisotropy inherent self-attention in transformers, 2024. URL httts:://axxii wwwpxiii.org///aa. abs abtp///1.//1114.....000111111112
Gene H. Golub and Charles F. Van Loan. Marix Computations 4th Edition. Johns Hopkins University Press, Philadelphia, PA, 2013. doi: 1111377/.97814214097 URL https://
epubs. siam. hetg//oo/abbae abs/ 10. 1117/1.1111 0111://.998112111111
N. Halko, P. G. Martinsson, and J. A. Tropp. Finding structure with randomness: Probabilistic
algorithms for constructing approximate matrix decompositions.. SIAM Review, 53(2):217-288, 2011. doi: 10. 10.1133/097771806 URL
Arthur E. Hoerl and Robert W. Kennard. Ridge regression: biased estimation for nonorthogonal problems. Technometrics, 42(1):80-86, February 2000. ISSN 0040-1706. doi: 10.2307/1271436.
URL httlhttps://eeo /tttps://doi.. org/ 10. 21..2000/2221133333
Roger A. Horn and Charles R. Johnson. Matrix Analysis. Cambridge University Press, 1985.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https:
//axxiv.org/abaet h/tp////b///311...//// 06825
lan Jolliffe and Jorge Cadima. Principal component analysis: review and recent developments. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sci- doi: 00.1099/ssta.2015.000
ences, 502020 04 2016.
Jinuk Kim, Marwa El Halabi, Wonpyo Park, Clemens JS Schaefer, Deokjae Lee, Yeonhong Park, Jae W. Lee, and Hyun Oh Song. Guidedquant: Large language model quantization via exploiting end loss guidance. In International Conference on Machine Learning (ICMI., 2025.
12

---

Published as a conference paper at ICLR 2026
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A http://aaaaaa/..... 080.a/888420280222.
whitepaper, 2018. URL
Olivier Ledoit and Michael Wolf. A well-conditioned estimator for large-dimensional covariance
matrices. Journal of Multivariate Analysis, 88(2):3654111 2004. ISSN 0047-259X. doi: https:
URL https: www. scw.neeiieeeiirettt com/
Jungi Lee, Junyong Park, Soohyun Cha, Jachoon Cho, and Jaewoong Sim. MX+: Pushing the Limits
of Microscaling Formats for Efficient Large Language Model Serving, pp. 869-883. Association for Computing Machinery, New York, NY, USA, 2025. ISBN 9798400715736 URL hetps:
//doi. org/ 1.....114537775888 3756118
Junyan Li, Yang Zhang, Muhammad Yusuf Hassan, Talha Chafekar, Tianle Cai, Zhile Ren, Peng- sheng Guo, Binazir Karimzadeh, Colorado Reed, Chong Wang, and Chuang Gan. Commvq: Commutative vector quantization for kv cache compression. Proceedings of the 42nd Interna- tional Conference on Machine Learning (ICML), 2025.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. In MLSys, 2024.
Zhengchi Ma and Rong Ma. Optimal estimation of shared singular subspaces across multiple noisy matrices, 2024. URL https::/aaxii..ooo org..voorrr.eeee... wtt..///aaaa//1.175555555.
Vladimir Malinovskii, Andrei Panferov, Ivan Ilin, Han Guo, Peter Richtárik, and Dan Alistarh. HIGGS: Pushing the limits of large language model quantization via the linearity theorem. Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 10857-10886, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176--88966
doi: 10.18653/v1/ htttps://aclaatthlloosoo wwtp:yy....////00 haw.:/ssaaaaaloll.a....a
Per-Gunnar Martinsson and Joel A. Tropp. Randomized numerical linear algebra: Foundations and algorithms. Acta Numerica, 29:403-572, 2020. doi: 10. 10.1017/809624922200000
Gabryel Miilon-illlams and Fredrik Dahlqvist. What makes good prune? maximal unstructured pruning for maximal cosine similarity. In The Twelfth International Conference on Learning Representations, 2024. URL httt:s:///pperrvie.... net forum?id=jsvvpPiiii
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
L. MIRSKY. Symmetric gauge functions and unitarily invariant norms. The Quarterly Journal of Mathematics, 11(1):50-5, 01 1960. ISSN 0033-5606. doi: 10.1093/qmath/1 1.1.50. URL /ttps://doi..or 1093
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Cameron Musco and Christopher Musco. Randomized block krylov methods for stronger and faster approximate singular value decomposition. In Proceedings of the 29th International Conference Neural Information Processing Systems Volume 1, NIPS'15 15, pp. 1396-1404, Cambridge, MA, USA, 2015. MIT Press.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up down? adaptive rounding for post-training quantization. In Proceedings of the 37th International Conference on Machine Learning, ICML''0. JMI..orrg 2020.
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Baalen, and Tijmen Blankevoort. A white paper on neural network quantization, 2021. URL https://
arxiv. wwwiv.orr/abs.. /2106 08295
13

---

Published as a conference paper at ICLR 2026
Open Compute Project. Ocp microscaling formats (mx) specification, version 1.0. Technical port, Open Compute Project, September 2023. URL BtRL:tttpp://wwwoopeccogg org/ documents/ http://oommmccoccaaaao--------ee
Version
1.0, Sept. 7, 2023.
C. C. Paige and M. A. Saunders. Towards generalized singular value decomposition. SIAM Journal on Numerical Analysis, 18(3):3984405 1981. doi: 10.1137/0718026. URL https:
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context, 2016. URL pttps://arxiy//
abs 606. 06031
R. Penrose. A generalized inverse for matrices. Mathematical Proceedings of the Cambridge Philo- sophical Society, 51 (3):44444,3, 1955. doi: 00.1017/50305004100000000
Kaare Brandt Petersen and Michael Syskind Pedersen. The matrix cookbook. 2006. URL https:
//api. semant ccschola www.:aa....roooooaaaeeee 12217763
Hadi Pouransari, Zhucheng Tu, and Oncel Tuzel. Least squares binary quantization of neural net-
works. The JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Work-
shops, June 2020.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL phttps::/aet
orgg abs / 2112 15115
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21(1), January 2020. ISSN 1532-4435.
Benjamin Recht, Maryam Fazel, and Pablo A. Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Review, 52(3):41-501, 2010. doi: 10.1 10.1137/070697835.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an adver- sarial winograd schema challenge at scale. Commun. ACM, 64(9):99-106, August 2021. ISSN 0001-0782. doi: 10. 10.1145/347438. URL/https::/ www.ddoo.oro/////o wtt.://///............. 1145000.111544443333
Ajit Paul Singh and Geoffrey J. Gordon. Relational learning via collective matrix factorization. In Knowledge Discovery and Data Mining, 2008. URL https:://ppi.semant cacholar.
semantsemmnt
org org/Corpus ID: 9683534
Nathan Srebro and Tommi Jaakkola. Weighted low-rank approximations. In Proceedings of the Twentieth International Conference on International Conference on Machine Learning, ICML'03, pp. 720-727. AAAI Press, 2003. ISBN 1577351894,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshor, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuming Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian. Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen
14

---

Published as a conference paper at ICLR 2026
Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL
org/ aas//a//////// 2307 09288
Lloyd N. Trefethen and David Bau, III. Numerical Linear Algebra. Society for Industrial and Applied Mathematics, Philadelphia, PA, 1997. doi: 10.1137// 978089877977.4 URL https:
//epubs. siam. wwt/doii//oooooooooooo abs/aa....... 10200.1137111 0ttg://....1.11111..112282888110 07..8998889999988
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st Inter- national Conference on Neural Information Processing Systems, NIPS' 17, pp. 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue. Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025, URL https: /ttp://axxiiviiv org/ 2255 09388
Zhewei Yao, Xiaoxia Wu, Cheng Li. Stephen Youn, and Yuxiong He. Zeroquant-v2. Exploring post-training quantization in iims from comprehensive study to low rank compensation. 2023.
URL httph:ttp////c.. /api. semanticschoo/.........m. htt.cholla...rgg//rrusssssss wwtp/oorg./oooo..o.. 258947011
Junho Yoon, Geom Lee, Donghyeon Jeon, Inho Kang, and Seung-Hoon Na. ROSAQ: rotation-based saliency-aware weight quantization for efficiently compressing large language models. CoRR, abs/2506. 13472, 2025. doi: 10448550/ARXXI.2506. 13472. URL pesttp::/doi.coom
48550/arXi 2506.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Cheng Zhang, Jianyi Cheng, George A. Constantinides, and Yiren Zhao. Lqer: low-rank quan- tization error reconstruction for llms. In Proceedings of the 41st International Conference on Machine Learming. ICML'24. JMLR.org, 2024a.
Cheng Zhang, Jeffrey TH Wong, Can Xiao, George Constantinides, and Yiren Zhao. Qera: an analytical framework for quantization eтror reconstruction. arXiv preprint a4Xiv:2410.06040, 2024b.
Rongzhi Zhang, Kuang Wang, Liyuan Liu, Shuohang Wang, Hao Cheng, Chao Zhang, and Yelong Shen. Lore: Low-rank compression for llms kv cache with a progressive compression strategy, 2024c. URL URLlttpss://xxii.orro
03111
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettle- moyer. Opt: Open pre-trained transformer language models, 2022. URL
org/abs/ 2205.2225.0 01068
Weibo Zhao, Yubin Shi, Xinyu Lyu, Wanchen Sui, Shen Li, and Yong Li. Aser: activation smoothing and enror reconstruction for large language model quantization. AAAI25/IAA/25/AAAA 25. doi: 10.1609/umai, v39itiai..i.ii........ v39221.34443.. URL https:
AAAI Press, 2025. ISBN 978- 778-1-57735-897-
radoi.orgg/0.. 160. aaai wet.aaa.vvvv92. w39121.
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.O NOTATION & SHAPES
Refer to 6
Table 6: Unified notation and shapes for stacked errors, low-rank factors, and wovariance-weighted core.
Symbol
Meaning
E, E ROxd
Error matrix of module with output dimension O, and shared input dimension d.
Ecat = Ei:...; Eml E R
Row-stacked errors across modules; total rows m ==
Σ, Oz.
A = A Am] gmxr
Left factor formed by stacking per-module factors A+.
B E Rrxd
Shared right factor; target rank r.
1 min(m.d d}
Admissible rank range.
E = UEVT
Thin SVD with U E RRdd E E Raxd Ve Rdxd orthogonal.
(U,, E,, V,)
Top-r SVD blocks: U, Rmxr, E, E V dx
Σx 2
Input covariance; Ex = [[[xx] "] for centered inputs [[ = 0.
x//
(geudo-)square root of Ex
Ecat = QeRe
Thin QR with Qe e Rmxd Q_ Qe d
La, Re E
M R.E Rdxd
Covariance- SVD core used for randomized SVD on the reduced space.
Å Q A dxx
Variable change (reduced left factor).
B BE
1/2 E gr
Variable change (covariance-weiihhee right factor).
Residual (Eent AB) R mpmx x
Stacked error after factorization (no separate symbol reserved).
A.I STACKED SVD: SHARED RIGITT SUBSPACE AND GLOBAL OPTIMUM (PROOF)
When multiple modules share the same input dimension, we vertically concatenate the module- wise emror matrices E, E R0.x into Eeat. then choose a shared right subspace (the row space B) while allowing module-specific left factors Ag, by solving
min A.B
E AB
16

---

Published as a conference paper at ICLR 2026
This appendix shows that (i) the solution is well-defined, and (ii) the shared B is also optimal in energy/projection sense (Ky Fan; cf. Fan (1550); Golub & Van Loan (2013). Consequently, a single shared B serves strong representative of what one might otherwise try to learn as separate B, per module.
Problem (Unweighted Frobenius Approximation).
min A.B
Eeat
AB F
(A.1.1)
LEMMA A.1.1 EQUIVALENCE OF SEARCH SETS: Mr = Ry.
Let
M (AB: A A BeR R, (X MERR mxd rank(X) S r).
Then M₁ Rr
This is a standard consequence of fankffactorization and the SVD characterization of best rank-r approximants: see, e.g.,, & Young 1936 MIRSKY (1960); Golub & Van Loan (2013): Horn & Johnson (985). We omit the proof.
Proof. By Lemma A.1.1, the problem reduces rank-r approximation of Eeat- By the Eckart- Young-Mirsky theorem Eckart & Young 1936); MIRSKY 1960), the optimizer is the truncated
SVD
= U,E,V
any global minimizer (A, B) must satisfy
AB U
(A.1.2)
o = r1 the optimizer may be non-unique Golub & Van Loan (013)).
L
THBOREM A. 1.2 IDENTIFYING THE SHARED RIGHT SUBSPACE: row (B) sspan (V..)
We determine the optimal shared right subspace for the factorization minA,B EEat AB | Let
Ecat UΣV be thin SVD, and let rank(B). Denote row (B) and the orthogonal projector Ps := B (BBT) pseudoinverse).
D 'B (assume B has full row rank; otherwise use the Moore-Penrose
Fixing B, least-squares normal equations yield (see, e.g., GGolu & Van Loan 2013 $5))
A
E B
CBBBBBT"
(A.1.3a)
A A
B
EcatPs.
(A.1.3b)
Hence, with G E Eeat.
|Eat A B Eeat -I- PI-PP))PP = Ecaa tr (r(P
(A.1.4)
where the last identity is the usual projection-trace formula (cf. Horn & Johnson (1985)).
Therefore, selecting S of dimension is equivalent
max tr(Ps din S =
(A.1.5)
By Ky Fan's maximum principle Fan 1950), the maximizer S is the span of the top-r eigenvectors of G. Since Eeat UΣV implies G VV"V its top-r eigenspace equals span Thus
row B = span V....
(A.1.6)
THBOREM A.1.3 REPRESENTATIVENESS ENERGY OPTIMALITY: SUM OF PROJECTION ENERGIES.
The shared right subspace S row(B) of dimension maximizes the total projection energy Σ. Ps 1? where Ps is the orthogonal projector onto S (c.g., Ps = QQ" for any orthonormal basis Q of S).
17

---

Published as a conference paper at ICLR 2026
Proof.
For each module E,,
E,Ps = FP E, E.).
(A.1.7)
a standard identity using symmetyyidempotence Ps and trace eyclicity (see, e.g., Hom & John-
son 1985 Golub & Van Loan (2013)) Summing over yields
max dian S
Σ
E.Ps
max to Ps Σ E. E.) = max
tr dim
G Σ E
E. E.
Ecat.
dim S
(A.1.8)
By Ky Fan's maximum principle Fan 1950)) (cf. Eq. A.1.5) the maximizer is the span of the top-r eigenvectors of G. Since Eest
UEV implies G V follows that
S span V,)
row [ span (V.).
LEMMA A.1.4 IDENTIFIABILITY AND "BALANCED" FACTORIZATION.
Although the pair (A, B) is non-unique up to invertible reparameteriiatiioss, the right subspace row(B) is identifiable; choosing the SVD half-split E yields numerically stable balanced fac- torization Golub Van Loan (2013).
Non-uniqueness.
For any invertible E Rrrr
(A, (AR, R
AB invariant.
Hence factors not unique, while the projector onto row (B) is unique (right singular subspace; cf. Theorem. A.1.2 and Golub & Van Loan (2013)).
Balanced factorization. half-split
Let Eeat
UEV T and denote by Ur, Er, Vr the top-r blocks. The
A = U,E7/²
(A.1.9a)
B" EL/VT
(A.1.9b)
A'B' V.
(A.1.9c)
satisfies
A = which avoids squaring condition numbers in normal equations and minimizes combined factor norms among reparameterizatioos:
T B'B*T=E, Er
2 (|AI) + BBE |U,
with equality at (A", B*) Recht al. (2010). (Standard facts; see Golub Van Loan 20133
Recht
et al. (2010..
Block Recovery and the Pseudoinverse Given the shared right factor B each module-specie specific left factor A; is obtained by a single least-squares solve. Using the Moore-Penrose pseudoinverse provides the minimum-norm solution and remains valid under rank deficiency Penrose (9555) Ben-
Israel & Greville 2010);; Golub & Van Loan (2013):
A = E, T (
(A.1.10)
suggests that (i) when B" has full row rank, (-)t reduces to the inverse and Eq. A. coin- cides with the normal-equation solution; (ii) in general, (.) yields the unique minimum norm LS solution and is numerically stable under near-singularity (Ben-Isracl & Greville (2010): Golub & Van Loan 2013)
18

---

Published as a conference paper at ICLR 2026
A.2 COVARIANCE-ALIGNE OBJECTIVE: BRIDGE EQUIVALENCE AND GLOBAL MINIMIZER (PROOF)
Sec. 3.1.2 formulates the covariance-alignedd objective
min [| AB) E
A.B
which weights errors by the input usage encoded in the covariance Σx Anderson (984); [Bihhop (2006). This appendix provides a complete mathematical justification: (i) a bridge equivalence
that converts E (Eaat AB)x x into Frobenius form via the trace identity E[x Mx]
tr(MEx) Petersen & Pedersen (206);; (ii) a whitening reduction to standard low-rank approx-
on the right) Golub Van Loan imation by the change of variables B == BE (and Ecar E//² 2x
2013); (iii) a closed-fore form global minimizer given by the truncated SVD of Eeat Ex with balanced factors and the identity of the shared right subspace; and (iv) extensions nonzero-men -ean inputs (centering) and singular Ex via pseudoinverse whitening Benn- & Greville (2010); Penrose (1955).
In our case, the (distributton-weeighed) risk is the expected squared output error under the input law:
R B) E M
Directions used more frequently with larger magnitude (large variance) are weighted more heavily
by Σx. which motivates reigt-weighted objective via / Bishop 2006); empirical counterpart uses samples (xn)
Anderson 1984). An
N
R(A,B)
Σ
M x,
Σx = Σ Xnx
N
THEOREM A.2.1 (BRIDGE EQUIVALENCE).
In this subsection, we prove the bridge identity E M tr(ME,MT) converts the distribution weighted risk into a Frobenius norm amenable to SVD analysis (see the trace/expectation identities in Petersen & Pedersen (2006)).
||ME
1/2 12 1|2, which
For zero-mean inputs with covariance Σx 2 O,
MX 3 = tr(MExM = ||| |ME|/1
(A.2.1)
Proof. (Vector norm trace). Since ||y 1 = tr(yy and trace linear,
E Mx E tr M ([xxx 1 [MM = (M
(Trace > Frobenius). Because |Z|| (ZZ and 2 1/2 Ex, |ME tu( (MEM = tr(ME.M).
Distribution- Distribution-weigheed risk equals the Frobenius norm of the right-whitened residual MD 1/2
LEMMA A.2.2 (NONZERO-MEL MEAN INPUTS).
In this subsection, decompose the risk for E[x] * 0 into covariance term and deterministic mean term, showing M tr(M Cov(x) M + ||M (cf. Anderson (0066).
1984); Bishop
Let fa = [xx] and Cov(x) = B[(x (x) 11) Then
E |MM]] Lr (M + |M
(A.2.2)
19

---

Published as a conference paper at ICLR 2026
Proof.
Write x (x pL) + l and expand:
M 2 MM - 10l12 + 2( - 11), My) + |M
Taking expectations annihilates the cross term since E[x - μM] = 0, yielding the claim.
Risk decomposes into covariance term plus mean-induced term.
THEOREM A.2.3 (VARIABLE CHANGE AND WHITENING).
In this subsection, we show that dard Frobenius low-rank approximation by proving || [Eeat
reduces the covariance-aligned objective stan-
AB)Ex
E ABI with
E Ecat Σx 1/2 B.B BΣ (standard whitening trick; cf. Golub & Van Loan (013)).
Define
E Best EV/2
A A A,
B BEE?
Then
[[ AB) 1/2
1/2
E AB
(A.2.3)
Proof.
Direct substitution:
È Q-A Eeat Σ
- A(BE)/) (Eeat AB) 12
Taking Frobenius norms yields the identity.
Whitening converts risk minimization into plain Frobenius factorization.
LEMMA A.2.4 (WEIGITED LEAST SQUARES FOR A GIVEN B).
In this subsection, we derive the closed-form weighted least-squares minimizer A E E B (BE, for fixed B, and interpret the residual as a Ex-weighted right projection (Golub & Van Loan 2013 Ch. 5), Björck [996); matrix derivatives in Petersen & Pedersen 2006
Consider
f(A) = || [[
AB) /
Let E = Ecat. E. = EΣL/ and B = -BEV/ Then the unique least-squares minimizer is
= (B EΣ B
BB
(A.2.4)
Proof. In whitened variables,
f(A) = E - = = =t. 2tt((AEE + tn ABB AA
Using BA tr( (AACAAT = 2A C for symmetric C and à 2006
8A tr(AM) = M Petersen & Pedersen
Vaí(A) = 2E T + +A = = A E. TT (B -
Substituting E., E2!/ and B = BE gives the second form.
In whitened variables, E. A = E. P3) with Pg projector onto row(ß) in the Euclidean metric. original variables, A'B = EP3 with
BTBBB - the orthogonal
Px E BB
the right projection under the Exewehttted inner product (a standard form of weighted/oblique projection:( cf. Golub & Van Loan (2013). Ben-Israel & Greville (2010): Björck (1996)).
For fixed B, the optimal A is weighted I.S solution; the residual is Σ. -eeeggttedd right projection.
20

---

Eckart & Published as a conference paper at ICLR 2026
THEOREM A.2.5 (GLOBAL MINIMIZER; BALANCED FACTORS; RIGHT SUBSPACE).
In this subsection, we obtain the global solution via the Bokart-Young-Mie theorem Eckart &
Young 1936); MIRSKY (1960), choose balanced factors A U,E,, B* Σ V and
identify the optimal shared right subspace row (3) row V Σx
123 (cf. Ky Fan's principle
and the subspace discussion in Fan [950]; Golub Van Loan (2013
Let E UEVT be an SVD and (Ur, Er. Vv) the top-r blocks. Then
À = U,
B-= ===
achieve the global optimum of min; A B
AB p. with minimum value Eirr o, (E)
Young (9366; MIRSKY (1960); Golub
Van Loan (203). original variables,
A U,> B B Σ
1/2
= -
and
row(B") = row EV Σイ
(A.2.5)
Proof. By Theorem A.2.3,
min A.B
E Eeat
AB)
min A,
E ABIE
Left/right orthogonal invariance of the Frobenius norm reduces the problem to min Y Σ
Y YI? solved by the truncated SVD Y Σ, ① O; hence X = U,E,V V Eckart & Young 1936);
MIRSKY 1960). Choosing A U, E and B Σ V produces X* A' Returning to original variables gives the stated (A B*) and the row-space identity (ef. Fan (1950
Golub &
Van Loan 2013)).
In whitened variables: (A+) A E, and (B+ T
Er In original variables: (A*)T A
E, and B B==x("") = Σ, Golub & Van Loan 2013) For any orthogonal R E R
(AR,R R B) attains the same objective value Golub & Van Loan 2013).
The truncated SVD is globally optimal; the balanced factorization is well-conditioned, and the opti- mal shared right subspace is row(V, xx1/2)
LEMMA A.2.6 (SINGULAR Σx AND PSEUDOINVERSE WHITENING).
In this subsection, extend all results to rank-deficient Σx by showing the objective depends only on Range Σ xx] and that pseudoinverse whitening preserves the conclusions on that subspace Ben- Israel & Greville (2010); Penrose (1955).
Let Σx = QAQ T with A = diag( (A,,..., Ar. O, .......), 0). Define
Σ/ QA 1/1 Q2QTT Σx 1/2 = QAt/2QT
where A 1/2 applies X; to A > 0 and 0 otherwise. Then the objective || (Eeat AB) 1 depends only on Range( Bange(Ex), and Theorems A.2.1 hold unchanged that subspace.
Proof.
Let Q = [Qr Qul with Q, spanning Range Σx) and E1/2 = Q.
1/2 Q,T. Then
[E AB EB 2 [[EE,, A ///
which is the same Frobenius objective restricted to Range [xx) Components along Qo vanish under 1/2 and contribute nothing.
Pseudoinverse whitening discards the nullspace; all conclusions hold on Range Ex).
In our implementation, to estimate and stabilize Ex, we perform ridge/shrinkage regularization Σx - Σx + =I) while using diagonal approximations (cf. Bishop (006); Anderson (1984); Ledoit & Wolf (2004); Hoerl & Kennard (2000) with mini mini-batch and drodunngwidoow since computing full covariances costly.
21

---

Published as a conference paper at ICLR 2026
A.3 QR REDUCTION: SMALL-CORE EQUIVALENCE AND GLOBAL SOLUTION (PROOF)
The covariance-aligned objective
min BERoX
(Eeat AB)
E
(A.3.1)
can solved without ever forming the tall whitened matrix E = Ecat ΣV' E Rmxd A thin
QR Eeat R. (with Qº Q. Ia) collects all the information relevant Eq. A.3.1 into the x core M : R. / because E = Q.M and the Frobenius norm is left-orthogonally invariant [|QZ]] F Z Fr when Q Q I) Golub & Van Loan y: Trefethen & Bau (1997). Thus can reduce the large problem to an equivalent d x d problem, apply standard SVD/EYM analysis
on the core, and lift the solution back (QR reduction core matrix; also Halko et al. (201 1):
Martinsson & Tropp (2020) for randomized variants).
LEMMA A.3.1 (OPTIMAL A LIES IN col(Q..).
For any A, decompose A Q, + A with Q A 1 and set B = BEV? Then
| -E-AB AB Q (M- AB + |A | > (
where E = Q. M and M == R. ///² Hence any global minimizer satisfies A = 0, i.e., A
Q. A It shrinks the search space for A to the d-dimensional column space of Qe: any component orthogonal col(Q.) only increases the loss. (Orthogonal Aecomposition/Pyttagoore the Frobenius inner product; cf. Golub & Van Loan (2013) Trefethen & Bau (1977))
property of
Proof. Use = Q.M and orthogonality: Q (Qe()) = (.) and (A_ B) 0. so the two terms orthogonal in the Frobenius inner product and the squared norm splits. The minimum occurs at A = O.
THEOREM A.3.2 (CORE EQUIVALENCE).
By Lemma A and left-orthogonal invariance of || (i.e., ||QZ|| |Z]| for orthogonal Q: Golub & Van Loan (2013).
min A,B
(Eeat
AB)
min
M
AB
M R. 1.2
(A.3.2)
A.B
Any minimizer (A,, B*) lifts to a minimizer of the original problem via
A' = Q.A
B' = 1'
(A.3.3)
where Σx11 1/2 denotes a (pseudo-)inverse square root when Ex is singular Ben-Israel & Greville 2010
Proof. Restrict A Q. (Eq. A.3.3p. Then [[ AB) 1/2 | QQ
-AB]
M ABIIF. The lifting follows by inverting the change = B
COROLLARY A.3.3 (PRESERVATION OF NONZERO SINGULAR VALUES AND RIGHT SINGULAR VECTORS).
Since (M.M)(QQ.M) M E Q.M and M share the same nonzero singular values and the same right singular vectors. Hence the SVD of M directly yields the optimal shared right sub- space for the covariance-aligned objective (orthogonal invariance of SVD; e.g., Golub & Van Loan 2013): Trefethen & Bau (997)).
Proof.
Immediate from Q Q. Ia-
22

---

Published as a conference paper at ICLR 2026
THEOREM A.3.4 (BALANCED FACTORS, GLOBAL MINIMIZER, AND LIFTING).
Let M UEV be an SVD and (U,, Er, Vv) the top-r blocks. Then
À = U, 2,
B = =
(A.3.4)
achieve the global minimum of M M-AB|| | by the Eckart- BockartYoungge Young Mirsky theorem Eckart & Young (133)]; MIRSKY (1960); Golub Van Loan (203)). Lifting to the original variables gives
A = Q U,
B B" = V Σx
(A.3.5)
The minimum value is |M U, V l and the shared right subspace is row (B*) =
span( V, Σx 1/2) (cf. Ky Fan Fan 1950).
provides a closed-form global minimizer and numerically well-conditioned (balanced) factor- ization.
Truncated SVD is optimal; balancing (X+/2) improves conditioning and scale regularity Golub & Van Loan 2013).
Proof. Apply EYM to the core problem from Eq. A.3.2 choose balanced factors so that U,E, = A.. Use Eq. A.3.3 to obtain (A,,*).
This process makes the thin QR cost O(md)) while forming/using the core costs O(d (or (
/// is prccomputted/struetuueed))) All subsequent optimization on the x d core Golub & Van Loan (203)) Trefethen & Bau 1997). After computing M, we do not materialize M; instead
y), and pass these to RSVD Halko we keep z + Mz R. Σx and y + MTy = [RR
al. 201 Martinsson & Tropp 2020).
From Eq. A.3.5 row(B*) span V Σx 1/2) defines the shared right subspace. GLOWQ, this C subspace is exactly the group-shared projection used to compute and cache R = Bahared once per input-sharing group, thereby enabling efficient A,R. reuse during inference while preserving expressivity via module-specific A; (cf. Sec. 3.3land and the Ky Fan view in Theorem A.1.3).
A.4 RSVD ACCURACY GUARANTEES
Let the core matrix be M == R, E E Rdddd as defined by the QR reduction in Appendix A.3 We target rank S d with oversampling p 2 2 and power iterations q 2 O. By the core equivalence and preservation results, accuracy on M transfers verbatim to the covariance-aligned objective.
ALGORITIIM A.4.1 RSVD ON THE CORE M.
It computes the dominant right subspace (which defines the shared right factor) the small d. x core without ever materializing the tall whitened matrix (standard RSVD; (Halko et al., 2011 Martinsson & Tropp 2020)).
Procedure.
(i (O 1)dx(r+p), Y MMM
(ii) Power iterations: repeat times { (Y + M(M Y)} with o--orhonormmalzztiinn:
(iii) Q + oYth(Y), B Q
U QU; truncate to (U,, Σ,, V,); (v) Balanced core factors: A A'= A,-// B = Σ -
(iv) B Ü.VT,
Find a good range Q via randomized sketching (with optional power iterations), then refine by a small SVD
M. Justification. Within the subspace R(Q). the best rank-r approximation is the truncated SVD of Q M: lifting by Q yields U,E, V as the optimal restricted approxima- tion (Golub & Van Loan 2013 Ch. 2). The randomized sketch ensures (in expectation or with high probability) that R(Q) captures the dominant right subspace of M (Halko et al. 2011 Martinsson & 'rrop 2020).
23

---

Published as a conference paper at ICLR 2026
THEOREM A.4.2 (FROBENIUS ERROR, EXPECTATION).
Let M UEVT with singular values d1 2 ... - od. For p and q = 0,
E M QQ M S [
r
1/2
Σ
oj
1/2
(A.4.1)
p-1
j2r
e.g., (Halko et al.
2011 Thm. 10.5)
quantifies that RSVD matches the optimal tail energy up mild factor depending only (r, p).
U Σ V with Σ1 g"xr and Σg the tail. Let V
[2
Proof. Write M
and
Σg
12
Y = MΩ. Standard analysis of Gaussian sketches gives [[ P S Σg F [|
||
and E| [ P I <r 1) for p 2 2 (Halko al. 2011) Taking square roots and expectations
yields Eq. A.4.I
THEOREM A.4.3 (SPECTRAL ERROR WITH q POWER ITERATIONS).
For 2 0 and modest constant Cr.p (depending gently on r, r)p),
M U,E,V X
122++) 0r+1-
(A.4.2)
r.p
(Cf. (Halko et al. 2011 Martinsson & Tropp 2020
Musco & Musco 2015
Power iterations shrink the subspace-angle gap geometrically toward the optimal or+1 bound.
Each power iteration reduces the gap factor roughly by (-)2/(2++1) exponent toward 0r+1-
Proof. After 9 power steps, (MM MM .t 22
2 Ωt). Block-partitioning
V
and analyzing principal angles between the exact and sketched right subspaces
gives
| < |22l2 (2 24 1/(2++1)
Bounding the Gaussian pseudo-inverse term by Cr.p and using أ 2012 (ar+1/or))
yields Eq. A.4.2
COROLLARY A.4.4 (TRANSFER TO THE COVARIANCE- OBJECTIVE).
By Theorem A.3.2 and Corollary A.3.3.
(Eeat A ALB') - UM-U
(A.4.3)
It links RSVD accuracy the core directly to the original covariance- objective.
Core RSVD error bounds become the error bounds for the original problem, verbatim.
E 1/2 Q.M M and Frobenius norms are left-orthogonally invariant; the optimal truncated approximation M corresponds under lifting to the optimal approximation of (Erat AB) yielding Eq. A.4.3
Proof. We have E = Ecst //2
PROPOSITION (Q-LESS LIFTING: BLOCKWISE RECOVERY OF A₃).
Write E Er;... Em and A = A A conformably. At fixed B', each block admits the closed form
A (B") (B"))"))*',
i= 1,..,.., m.
(A.4.4)
so the tall orthonormal factor Q. need not be stored (least- squares with pseudoinverse; cf. (Björck 1996 Ben-Israel & Greville 2010) Golub & Van Loan 2013
24

---

Published as a conference paper at ICLR 2026
It economizes memory: per-block factors are recovered directly from [E,,B) without retaining Qe-
Per-block least-squares with pseudoinverse yiclds A using only (E,, B*).
Proof. For each block, minimize |,, I The first-order optimality condition is AB (B*) = ET Multiplying on the right by the Moore-Penrose pseudoinverse gives the minimal-norm solution A E, (BB())) which precisely Eq. A.4.4
B EFFECT OF RIGHT WEIGHTTIGHTED SHARED B
In this section, we analyze the effect of the right-weighted shared-B on the GlowQ's correction with the following procedure.
Procedure. (1) Using calibration inputs (xn]n=l 1N
Rdd. estimate the layer input covariance
Σx N Σ xnx,
(optionally: Σx x Σx +EI.
(2) For each module i E {q, k, v. gate, up), form the quantization-erma matrix E; E RRoxd and the row-stack Eeat = [Ej:... Em E Rmxx
(3) Cov-aligned (whitened): compute SVDs of E, == E,S22 and Eest Eeat EV? and take the top-r right bases Vi, and V,-
(4) Unweighted (no-cov): repeat the same without whitening to obtain Vile-cov) and ((no-cow)
(5) For each module, form the absolute cross-basis cosine matrix
C; V Va.r E R"**,
Hungarian-reorder it to maximize the diagonal sum, and visualize as heatmaps.
Impact of optimization with the right weighted objective. As illustrated in Fig. 5 and 6 the whitened condition produces a bright near-diagonal across all groups (Q/K/V and MLP gate/up). indicating one-to-one alignment between the shared right subspace row (Bahared) and each module's right subspace (up to sign/pernutation.. The effect is strongest for Q/K, and slightly more diffuse for the leading axes. In contrast, the unweighted
and for MLP (gate/up), but remains concentrated condition yields moise-like patterns with no diagonal structure.
Right-side covariance weighting is crucial for estimating shared B under anisotropic inputs: exposes a common right subspace across modules that ingest the same input tensor. This validates the shared- assumption and directly motivates our ABx caching strategy, i.e., computing R Bahared X once per group and reusing A; R across modules. Unweighted stacked SVD fails to reveal this alignment, weakening both the shared-B premise and the practical caching benefit.
B. GROUP-CACHED (WEIGHTED STACKED RSVD) VS. LAYER-WISE (WEIGHTED RSVD)
Table 7: Perplexity (lower is better) across model families on WikiText-2. Layer-wise applics layer- wise SVD correction, whereas GLOWQ applies group-wise SVD with shared right factor B; GLOWQ (Selective restore) denotes selective group restoration.
Method
LLaMA 2
LLaMA 3
Qwen 2.5
Qwen
OPT
Vicuna
Mistral
7B
13B
3.2-3B
3.1-8B
7B
14B
8B
14B
1.3B
6.7B
7B
13B
7B
LAYERWISE
5.58
4.96
8.15
6.59
7.06
5.64
9.92
8.80
15.05
11.00
6.89
6.03
5.42
GlowQ
5.58
4.96
8.16
6.59
7.07
5.64
9,90
8.80
15.06
11.00
6.90
6.02
5.42
GlowQ-S
5.60
4.96
8.22
6.62
7.09
5.68
9.97
8.89
15.19
11.00
6.90
6.04
5,45
25

---

Published as a conference paper at ICLR 2026
sherac- basn (torted)
10
0.5
1000 Selwwar badi VecCos
shared- basis index (ortte)
16-
0.2
74-
24
0.0
32
40-
0.
40
0.2
24
40
48
16
40
basis indiox [sorted]
proj
basis indiax (oorted)
(a) Q layer: no whiten
(b) Q layer: whiten
15-
shorch basis Index (or-tel)
16-
(ood beteess DMSS VecTorS
Shored basi tndex toll
16-
0.6
-0.6
<2
32-
0.2
0.2
30-
56-
18
24
32
40
45
56
16
12
40
45
56
E_proj benis inders (icetac))
R_proj banis indlaa (ecetee)
(c) K layer: no whiten
(d) K layer: whiten
10
1.0
8-
Shared 8 bosts Inder (orrosh
(005) becween bad5 Vnctors
sharddl bass indox (aarted)
24
24
0.6
32
0.4
0.4
88
48
0.0
0.0
16
V.peol
proj basis (sorted)
(e) V layer: no whiten
(I) V layer: whiten
Figure 5: Whitening vs. non-whitening alignment matrices. For LLaMA 3.2-3B, we estimate a shared right basis Behare from the stacked error either without covariance weighting (Eeat, left pan- els) or with covariance-aware whitening (Eeat / right panels). Each heatmap shows the absolute basis alignment between row (Bsuured) and the mer-module right subspace for Q. K, V; brighter values denote larger absolute inner products. DiagScore and Affinity summaries are reported in the main text.
Results on Table Across 13 model-size combinations, GlowQ and Layer-wise yield essentially identical perplexity: the mean gap is +0.001 ppl on average, with per-family fluctuations confined to #0.02 ppl. By design, GlowQ-S (Selective restore) trades bit of accuracy for efficiency, trailing Layer-wise by +0.04 ppl on average. In short, the full shared-B configuration matches layer-wist (A,, B,) on WikiText-2 without systematic degradation, while the selective variant incurs a small, consistent increase ppl.
The covariance-aligee aligned cross-basis basis heatmaps exhibit an almost per- fectly diagonal structure after Hungarian matching, indicating near one-to-one correspondence between the shared right subspace and each module's top-r directions. Whitening aligns input us- age that the shared B spans (practically) the same right-singular space that the individual B;
Observation on Fig. 5
6
26

---

Published as a conference paper at ICLR 2026
sharos-e basis indox (ootte)
16-
0.
1000 between DITT VeCOOTS
shardd bods Idexx Gouttel
16
-08
24
80-
40
(005)
24
40
56
1.5
50
basis indix (sorted)
gMte_prdj
(sceted))
(a) MLP gate: no whiten
(b) MLP gate: whiten
Shared- badle Frdlx Cooteed)
16-
[ood] DHS beeWeen DMSSS Veciors
sharec-O beck modex (oorted)
0.6
0.5
32
80-
22
0.2
56
16
52
40
45
56
16
24
40
56
ap_jing bassss intee (wotland)
up, prrg boniis inclox (strted))
(c) ML. whiten
(d) MLP up whiten
Figure 6: Whitening vs. non-whitening alignment matrices. MLP (up/gate).
would select, explaining why GlowQ's perplexity tracks layerwise so closely, and why GlowQ-S, restoring only a subset, shows the small upward shift in ppl.
the covariance-aligner cross-basis heatmaps show the same qualitative behavior as in the dense models once whitening is enabled. Without whitening, all panels (expert gate/up, shared gate/up, and MoE attention) look almost uniformly dark, indicating that the shared right subspace and each expert's 's local top- directions are essentially uncorrelated. After whitening and Hungarian matching, the heatmaps become sharply diagonal for both the representative expert (e.g., expert gate proj / expert59 up proj) and the shared-B3 B MI.P/attention blocks, revealing a near one-to-one align- ment between the shared basis and each expert's own error subspace. This confirms that, once inputs whitened, the grouped MoE FFNs and the shared MLP effectively live in the same right-singular space, so a single shared Bshared can serve all experts with only small residual mismatch. Conse- quently, GlowQ can compress all experts and the shared MLP with one shared right-hand matrix while closely tracking the layerwise baseline perplexity, explaining the tiny +0.02 PPL gap we observe Qwenl 1--.555--22..77
Observation on Fig. 7 9 For the MoE FFN
27

---

Published as a conference paper at ICLR 2026
6.8
6.8
shored bosis Inck Gootecl
shored & bads lnck (oorted
16-
24
(004 berween
C
40
C.0
16
24
40
56
16
32
40
56
basis
index (scrtad)
proj
basis index (orrted)
(a) Q layer: whiten
(b) Q layer: whiten
1.0
1.0
8-
Sharc3 bails indox tootta)
0.8
sharel bast indox toartsh)
C.B
24
0.6
24
0.0
[oo] betwean
0
C.4
48
0.0
6.0
16
32
(sortod)
R. (sortod)
(c) K layer: no whiten
(d) K layor: whiten
shored bodil tnce (ooto:))
26-
0.8
shored bodis thook Gortes)
16-
0.8
C.8
C.8
[004] betwAin
02
80
40
0.2
6.2
30
50
0
16
24
40
56
16
32
40
56
X_Dng besis indles bucrtad))
K_proj banis indles (scrtieds
(e) V laaer: no whiten
(i) V layer: whiten
Figure 7: Whitening vs. non-whitening alignment matrices for Q/K/V in Qwent 1-oo1..--oF-A2..75
As in Fig. 5
error, we estimate a shared right basis Bsturd from the stacked iitention-projecne
either from the raw error ("no whiten", left panels) or after covariance-awee whitening E/ ("whiten", right panels). Each heatmap shows the absolute basis alignment between (oare)))) and the per-module right subspace for Q, K, and V; brighter values denote larger absolute inner products. Whitening again yields sharply diagonally dominant structure, indicating that a single covariance- aligned basis captures the dominant error directions across Q/K/V.
28

---

Published as a conference paper at ICLR 2026
1.0
1.0
sha bosle indox (ooted:
sho cole bast indox boatted
0.6
24
C.6
[coi befowar
32-
6.6
48
6.0
6.0
32
stared__gate, (sontod)
wraard_.cate... (sonted)
(a) ML.P gate: no whiten
(b) ML. gate: whiten
SPorcd bodi tndc (orre))
16-
0.88
sPorch badi Inoe Gorad
16-
0.0
C.W
c.
[004] beEwAnn
40
0.2
62
56
56
10
24
32
40
36
0
16
24
32
40
05
36
shared_up_ p_n basts index (sorted)
shared_upp_ basts index (sorted)
(c) MLP
whiten
(d) MLP up- whiten
Figure 8: Whitening vs. non-whitening alignment matrices for MLP (gate and up) in Qwen1 5-5-
MoE-A2.7B. The construction identical to Fig. 7 but applied to the MLP gate and up projections aggregated over all experts. Whitening produces diagonally dominant alignment, indicating that shared covariance-alignee basis also captures the principal error directions the MLP blocks.
29

---

Published as a conference paper at ICLR 2026
1.0
1.0
sharc0 bosi basis Indox (oored)
shardd basis bas's Index Corted
0.6
24
C.6
LO0 betwaee
32
0.0
0.0
eepprttM gate. prgj banis indle (snrted))
HepertsM gate proj basis (ustted))
(a) MLP gate: no whiten
(b) MLP gate: whiten
Shared bazis indox (cotcc
26-
0.8
shared bast indox (ooted)
0.8
0.6
0.6
[C04] INTOOAn
80
40
0.44
0.2
50
16
24
32
40
56
8
1.
24
32
40
45
56
expert59_
sp_pnoj basts index (sorted)
expert59_jg _up_po cadis index (sorted)
(c) MLP
whiten
(d) MLP up whiten
Figure 9: Whitening non-whitening alignment matrices for the MLP (gate and up) of a single expert (Expert 59) in Qwenl.5-Mool 15-MMo-- We apply the same construction as in Fig 8 but restrict the stacked error and shared basis Bstared to Expert 59 only. The diagonally dominant structure under whitening shows that the covariance-aligned basis remains meaningful even at the per-expert level.
30

---

Published as a conference paper at ICLR 2026
C
TTFB & THROUGHPUT AROUND OTHER MODELS
Table 8: Latency comparison on LLaMA 3 models for Layerwise vs. GlowQ, GlowQ-S.
Models
Setting
TTFB
tok/s f
Prefill
Dec
(ms)
(ms)
(ms/tok)
Layerwise
70.83
17.46
71.07
58.22
3.2-3B
GlowQ
64.92
18.94
66.20
52.04
GlowQ-S
53.17
21.37
60.69
44.35
LLaMA
Layerwise
96.50
14.24
95.72
69.26
3.1-8B
GlowQ
86.44
15.31
90.01
64.47
GlowQ-S
71.70
18.89
73.50
52.34
Avg. A BX (%)
-9.38
+8.00
-6.41
-8.77
Avg. A R50 (%)
-25.32
+27.52
-18.91
-24.13
Results on Table Table the LLaMA evaluation under identical runtime and mea- surement protocol. Two consistent trends emerge: (i) GlowQ reduces all latency components, with the largest relative gains per-token decode; and (ii) GlowQ-S further amplifies these benefits. On LLaMA (3.2-3B, 3.1-8B), GlowQ with BX caching improves serving latency over Layerwise: TTFB 9,.8%, tok/s +8.00%, Prefill -6.41%, and Dec 8.77% average. GlowQ-S (selec- tive restore) amplifies these gains: TTFB 25.32%, tok/s +27.52% Prefill - 18.91%% and Dec -34.13% on average. Improvements are consistent across both model sizes, with the largest reduc- tions appearing in the per-token Dec phase and end-to-end TTFB, reflecting reduced compute on the critical path. In practice, BX caching provides drop-in speedups without modifying weights, while the selective policy (GlowQ-S) offers simple accuracy-lateng knob by reducing the number of A,R applications (Sec. 3.3
Observation on Table BX caching removes redundant right- prote-proeection work by reusing the shared subspace, so each decode step primarily executes lightweight A, updates; this directly lowers Dec and TTFB. The selsttive-restore strategy further trims the executed paths across decoder blocks, yielding additional latency drops with commensurate increase throughput (tok/s). These mechanisms explain the near-linear percentage gains in the RSVD-driven core cost: caching reduces repeated right-side multiplies, while selective restoration shortens the active compute graph along the decoding trajectory.
D HYPERPARAMETER CHANGE
D.1 CALIBRATION DIFFERENCE
29n Jkent -Hg Mard
877
3.5
Naaderef Nf Callaratin streple
100
200
580
Rank-j0
Raak
(c) Right subspace similar-
(a) Energy capture- QKV layer
(b) Energy capture MLP layer
Figure 10: Energy Capture and Cosine similarity of Rightspace number of calibration samples
Fig. 10a
versus rank for different numbers of calibration samples, while Fig. 10c reports the mean pairwise cosine similarity (weighted) between the shared right subspace and the per-layer right subspaces the calibration size varies.
10b
plot energy capture
31

---

Published as a conference paper at ICLR 2026
Varying the number of calibration samples N {32, 64, 128, 256} leaves the energy-captun curves
in Fig. T0a 10b nearly indistinguishable, especially for practical ranks 7 I 128. In Fig. [0cp the weighted cosine similarity between the shared right subspace and layer-wise right subspaces is already high at N = 32 and saturates for N 2 64. These results indicate that a small calibration set suffices recover stable, data-aligned right subspace, consistent with PCA stability under clear spectral gap (Jolliffe & Cadima 2016, Horn & Johnson 1985).
We attribute the observed stability under relatively small calibration set, e.g., N 32 to the following four reasons: (i) Spectral-gap effect: The input covariance Σx is heavy-tailed, so the top directions are separated by a clear eigenvalue gap; the dominant r-dimensional right subspace stabilizes quickly with modest N (Jolliffe & Cadimal 2016; Horn & Johnson, 1985). (ii) Robust weighted objective. We optimize a right-weighted criterion,
min (Eeat AB 1/ A.B
so small perturbations in the estimate Σx have limited effect: large-eigenvalue axes dominate and lead the same top r-subspace (see also weighted low-rank formulations (Srebro & Jaakkola 2003)). Numerical regularization. Shrinkagennormaiizationn of Ex reduces small-sample noise and improves conditioning (Ledoit & Wolf 2004, Hoerl & Kennard 2000; Bishopp 2006). Benefit of group stacking. Building the SVD core from vertically stacked errors increases the effective sample support along rows, which smooths estimation of the shared right subspace (Paige & Saunders! 1981 Golub & Van Loan 2013).
To conclude, calibration sizes small as N x 32-64 already place the system in saturated mode since energy capture at a fixed r and the similarity between the shared and layer-wise right subspaces change only marginally beyond this point. Thus, our covariance-aligned, group-shared B achieves stable performance with low calibration cost.
D.1.1 SHRINK ALPHA DIFFERENCE
Table 9: Perplexity on WikiText-2 while sweeping calibration samples and shrink (lower is better).
LLaMA
Qwen
Calibration Samples
Shrink
3.2-3B
3.1-8B
14B
8.16
6.59
9.89
8.82
32
0.02
8.16
6.59
9.86
8.82
0.05
8.16
6.59
9.88
8.82
0
8.15
6.59
9.90
8.81
64
0.02
8.15
6.59
9.88
8.78
0.05
8.16
6.59
9.87
8.79
0
8.16
6.59
9.92
8.80
128
0.02
8.16
6.58
9.91
8.80
0.05
8.15
6.58
9.90
8.81
0
8.16
6.58
9.93
8.81
256
0.02
8.15
6.59
9.92
8.80
0.05
8.16
6.58
9.92
8.82
We apply a standard covariance shrinkage when forming the input statistic used for covariance- aligned subspace estimation. Let Σx be the sample covariance from N calibration sequences and the input dimension. We construct
tn [x{xx) I, a E [0,
i(()) = =1 (1-a) Ex + a d
32

---

Published as a conference paper at ICLR 2026
Le., convex combination of the sample covariance and an isotropic target (scaled identity); small a reduces small-sample noise and improves conditioning without altering the dominant axes learned from data (Ledoit & Wolf 2004
Bishop 2006; Anderson 1984).
Results on Table 9 Across calibration sizes N E {32, 64, 128, 256} and shrink a e {0, 0.02, 0.05), perplexity remains essentially flat for LLaMA for 3.2-3B and 8B, the sweep changes values by +0.01 ppl average. Qwen shows the same qualitative behavior, with mild benefit from shrinkage: a E [0.02, 0.05] yields -0.02 ppl on average for 3.1-8B and -0.01 ppl on average for 14B (relative a=0 the same N). Aggregating all models, a=0.02 improves by -0.01 ppl average, and increasing N beyond 64 produces only marginal changes (s +0.01-+0.00 ppl on average depending on the family). In short, both the calibration size and a small shrink factor have only second-order effect on WikiTexx-2 perplexity, consistent with the stability suggested by the energy and rosine-similarit panels (Jolliffe & Cadima 2016).
The right subspace stabilizes quickly because (i) the input covariance exhibits pronounced spectral gap, so the dominant r-dimensional space is identified with few samples (Jolliffe & Cadima 2016 HHrnn & Johnson 1985); (ii) the right-weighted objective empha-
Observation Table 9
sizes large-variance directions, making the solution insensitive to small perturbations in Σx: (iii) mild shrinkage damps small-sample noise (Ledoit & Wolf 2004 [Bishop, (006)); and (iv) stacking modules to form the core increases effective sample support along rows (Paige & Saunders) 1981 Golub & Van Loan 2013). Consequently, small calibration sets (N R 32-64) already recover data-aligned shared right subspace, explaining the near-come constant perplexity across the sweep and the slight, consistent gains from E [0.02, 0.05] on Qwen 3.
D.1.2
MEMORY USAGE
epu RAM.
sio:
Boor
350
310
3681
400
310
1999
12009
250
150
OPT-4.00FF4420 OPF--7777 OFT-138 LOPT-EAA OFF--10 OFT-3M OPFT-600 OFT-348
OFT-4-8
OPF1-138
UPI-366
Midel
(a) Runtime breakdown
(b) Memory footprint per model
Figure 11: Calibration runtime and memory footprint as a function of model size and the number of calibration samples N. (a) Stacked bars show the runtime breakdown into calibration and decom- position for each (model, N)) configuration; calibration dominates the total cost and grows nearly linearly with N, while decomposition time remains almost constant. (b) Memory footprint of the error tensor, covariance tensor, and peak GPU/CPU usage for each OPT model; the error and co- variance tensors account for most of the memory and grow steeply with model size.
Results Fig. 11 We profile calibration single A100 80GB GPU for three OPT models (6.7B, 13B, 30B) and calibration sizes N E {32, 64, 128), using SlimPajama-6B as the calibration corpus. Fig. 11a shows that the total wall-clock time is dominated by the calibration pass: for every model, the blue bars (forward passes used to estimate Σx and collect error tensors) account for most of the runtime and grow almost linearly with N, whereas the red bars (randomized GSVD / decomposition) contribute relatively small and nearly constant overhead. Even for OPT-30B, increasing N from 32 to 128 scales the runtime by roughly the same factor, indicating that the cost is predictable and controlled by the choice of calibration size. Fig. 11b breaks down the memory footprint. Peak GPU memory (blue) grows moderately with model size and remains well below the CPU footprint, since we keep the model and activations on GPU but store error and covariance tensors on host memory. The green and purple bars show that these two tensors dominate the CPU
33

---

Published as a conference paper at ICLR 2026
usage and scale with model size: moving from OPT-6.7B OPT-30B increases both err-size and cov_size by several times, and the peak CPU RAM closely tracks their sum.
Observation on Fig. 11 Overall, the results indicate that the main cost of our method comes from a one-time, embarrassingly parallel calibration phase whose runtime scales linearly with N and roughly with model size, while the decomposition step has almost fixed cost. Memory-wise, the GPU footprint is modest and does not require trnger-hannstandaae accelerators; the heavy objects the error and covariance tensors on CPU, which can be streamed, sharded, or discarded immedi- ately after decomposition. Since Sections already yield stable energy capture, right-subspace similarity, and perplexity, practitioners can op- erate in this low-N regime. In practice, this keeps the calibration overhead to a few GPU hours even for 30B models and confines the CPU memory requirement to one-off offline preprocessing step, directly addressing concerns about prohibitive calibration time and memory pressure for large LLMs.
that small calibration sets (N Rs 32-64)
D.2 RANK DIFFERENCE
Table 10: Perplexity WikiText-2 by rank and method, formatted like the calibration-sweep ble.(Lowe) better.)
LLaMA 3
Qwen 3
Rank
Method
3.2 3B
3.1 8B
8B
14B
GlowQ
8.22
6.64
9.95
8.84
8
Layerwise
8.22
6.64
9.96
8.84
GlowQ
8.20
6.63
9.95
8.81
16
Layerwise
8.20
6.62
9.94
8.80
GlowQ
8.18
6.61
9.91
8.81
32
Layerwise
8.18
6.61
9.93
8.80
GlowQ
8.16
6.59
9.87
8.80
64
Layerwise
8.15
6.58
9.88
8.80
GlowQ
8.12
6.56
9.83
8.79
128
Layerwise
8.11
6.55
9.87
8.79
Sweeping the rank r. GlowQ matches layer- wise restoration in perplexity: the gap is +0.02 ppl average across models and ranks (never exceeding +0.04 ppl). Returns dimin- ish beyond moderate ranks: from r=8 to p= 128, the change -0.09 ppl average across families. Most of the gain is realized by r E {32, 64); increases beyond this window yield only marginal improvements (e.g., r=64 128 shifts by just a few hundredths ppl).
Results on Table 10
Observation Table 10 The rank-accuracy curve exhibits family-specific shapes: LLaMA shows a knee around r Rs 32-64 (initially flat, then a brief drop), whereas Qwen decreases more gradually without sharp elbow. In practice, this suggests using r-64 for LLaMA and r=32 for Qwen as strong defaults; GlowQ remains interchangeable with layer-wise restoration in accuracy at fixed r, while retaining the runtime advantages established elsewhere.
D.3 RANDOMIZED SVD PARAMETERS
D.3.1 PROOF OF QR REDUCTION & RANDOMIZED SVD
Discussion. Table II shows that Exact SVD on the d d core M takes 42.86 in total (0.76 per layer on average), whereas Randomized SVD (RSVD) completes in 16-622 (.09' per layer). This 8.2x wall-clock speedup is consistent with the complexity gap between O 3(d3) and
((q+1) (r-pp)) d(r-p))2 when d > r+p (Golub & Van Loan 2013 Halko et al. 2011
34

---

Published as a conference paper at ICLR 2026
Table 11: SVD runtime (s) and perplexity on LLaMA 3.2-3B (WikiText-2). Exact torch. wil.llgll....................... svd on the GSVD core M; Randomized = Halko R-SVD with oversampling p and power iterations q. SVD-only times factorization on M (CUDA-syneed), excluding the core QR used to build M. Total sums over layers; Layer(mean) averages across layers.
Method
SVD time (8) 4
Perplexity +
Total
Layer(mean)
Exact SVD
42.86
0.76
8.16
5.16
0.09
8.22
5.19
0.09
8.21
Randomized SVD
5.20
0.09
8.21
16
5.21
0.09
8.21
24
5.21
0.09
8.21
5.17
0.09
8.17
4
5.19
0.09
8.16
Randomized SVD
5.20
0.09
8.16
16
5.21
0.09
8.16
24
5.21
0.09
8.16
5.17
0.09
8.16
4
5.20
0.09
8.16
Randomized SVD
5.20
0.09
8.15
16
5.21
0.09
8.16
5.22
0.09
8.16
Concretely, with d-3072, r-64, and p fo....,,44 24), we have (r+p)/d S 88/3072 2.9%. so the RSVD term of scales roughly like a few percent of d up constant factors, matching the observed order-of-magnitue reduction in runtime.
Martinsson & Tropp 2020
Effect q and p. Runtime varies only weakly across p E {0, 40,4,8, 16, 24) and q E (0,1.2) (5.16s 5.. This is expected because the dominant RSVD cost is the matrix-block multiplies MΩ, MT (-); increasing p from to 24 changes (r+p) from 64 88 (only 38%), and the extra passes add a small multiple of the same GEMM cost. The lower-order term d(r+p)² is negligible this scale, short, the linear dependence on (r-p) and on (q+1) predicted by
manifests as a near-flat runtime curve because d > r+p and GEMM kernels saturate the de- vice (Halko al. 2011 Martinsson & Tropp 2020
Accuracy. Perplexity stays essentially unchanged: Exact 8. 16: RSVD 8.22 at (q-0, and improves to 8.15-8.16 for q 2 1 (with small p already sufficient). This aligns with randomized SVD theory: even a single power iteration (q=1) sharpens separation between leading and trailing singular directions and yields right subspace that is effectively indistinguishable (for rank-r objective) from Exact SVD in downstream perplexity (Halko et al. 2011 Musco & Musco 2015
Martinsson & Tropp! 2020).
The empirical results agree with the stated complexity: Exact SVD on M incurs (dd) time, while RSVD retrieves the leading right subspace in O ((q+))d(r-p)) time (plus a minor d(r+p)2 term) (Golub & Van Loan 2013 Halko et al.) 2011 Martinsson & Tropp 2020). In practice, q=1 with a modest p (e.g., p E [4, 16) delivers hear-Exat perplexity at 8x lower wall time, and increasing p further yields diminishing returns (Halko et al.
2011 Martinsson & Tropp 2020)
35

---

Published as a conference paper at ICLR 2026
D.3.2 POWER ITERATION & OVERSAMPLING DIFFERENCE
Randomized SVD hyperparameters on WikiText-2, measured on LLaMA-322 -32--BB.. We sweep (a) oversampling p (fixed q 2) and (b) power iterations (fixed p 16) and report perplex- ity (lower is better).
Table 12:
(a) Oversampling p sweep (fixed 2).
(b) Power iterations g sweep (fixed p 16).
Method
PPL +
Method
PPL. +
10
8.16
8.21
12
8.16
Llama 3.2-3B
LLaMA 3.2-3B
1
8.16
16
8.16
24
8.16
2
8.16
10
6.59
0
6.63
12
6.59
Llama 3.1-8B
LLaMA 3.1-8B
1
6.59
16
6.59
6.58 24
2
6.59
10
9.90
9.97
Qwen 3-8B
12
9.89
Qwen 3-8B
9.87
16
9.88
24
9.89
2
9.88
10
8.81
8.79
Qwen 3-14B
12
16
8.81
8.80
Qwen 3-14B
8.81
24
8.81
2
8.81
Table contraats oversampling p q=2 fixed; subtable and power iterations q (with p-16 2-2 fixed; subtable 12b). Empirically, increasing p from 10 to 24 leaves PPL essentially unchanged across models (diffcrences of S 0,01), whereas raising q from to yields small but consistent gains (most visibly QWEN3-8B), after which improvements saturate by q=2.
This pattern aligns with the standard analysis of randomized SVD (RSVD). Oversampling enlarges + p. which reduces the probability of missing near-rank-r directions
the sketch dimension but ultimately does not change the target truncation rank r. Once r already captures the dominant subspace and the spectral gap is reasonable, the marginal benefit of additional p is small; theory predicts only mild reduction of the residual as p grows (e.g.. with expected error bounds that degrade roughly as /r/(p - 1)). so practical guidance typically recommends Re 5-10 Halko al. 2011 Martinsson & Tropp 2020).
By contrast, directly amplifies spectral separation via the power scheme. Forming Y (AA (AAT)9A1 An effectively reweights singular values as g,241, which boosts the ratio between and the tail {a,>r) } and thereby reduces leakage beyond rank r. As a result, the sampled subspace
2q411
aligns better with the true top-r subspace, often yielding noticeable gains from q-0 to g=1, with diminishing returns thereafter; q {1, 2} is commonly recommended in practice (Halko et al. Ma & Ma 2024 Martinsson & Tropp 2020
2011
When r already captures the dominant energy, increasing p beyond a modest buffer offers little accuracy benefit, while single power iteration (q=1) can materially improve approximation for matrices with slowly decaying spectra. In our experiments, this theoretical expectation manifests as flat PPL curves across p and consistent but saturating improvements across q.
E COMPATIBILITY ACROSS QUANTIZATION DATATYPES
We apply weight-only quantization to Mistral-7B and evaluate on the WikiText-2 test set across both integer and booting-point-liic datatypes (Table [3}. For the integer settings (INT2/INT3/IN44) we use uniform weight-only quantization with shared seales within each weight group. For the floating-point-like settings (MXFP4, MXFP6, NVFP4), we adopt microscaling-style formats in
36

---

Published as a conference paper at ICLR 2026
Table 13: WikiText-2 test perplexity (1) for different datatypes.
INT
Floating-point-lie
Method
FP16
INT2
INT3
INT4
MXFP4
MXFP6
NVFP4
Quant only 5.32
1015.39
6.16
5.51
8.05
5.36
6.09
Quant + GlowQ
24.23
5.84
5,41
6.10
5.32
5.63
which weights are first normalized within a small block and then encoded using low-bit floating- point codes. Concretely, MXFP4 and MXFP6 follow the block-wise microscaling design of MX+ and the OCP MX specification, using shared scale per block and 4-bit or 6-bit element codes, respectively Lee et al. (2025): Open Compute Project (0023). NVFP4 follows NVIDIA's reference design with microscaled FP4 representation for weights, described in their low -ew-prcciion in- ference guidelines Alvarez et al. (025). These configurations allow us to test GlowQ not only on conventional integer quantization, but also recent microscaling- floating-point-lik formats.
Layering GlowQ on top of the quant-only baselines reduces perplexity by -991.16 on INT2, -0.32 on INT3, -0.10 10 INT4, -1.95 on MXFP4, -0.04 on MXFP6, and -0.46 NVFP4, relative to the corresponding quant-only settings. Improvements hold across all six evaluated datatypes, indicating that GlowQ behaves as an orthogonal, plug-and-play low-rank correction rather than a mechanism tied to a single integer format or precision; in particular, it remains compatible with recent floating- poiet-like microscaling formats while providing consistent accuracy gains.
F LONGBENCH RESULTS
Table 14: The results of 1--mma-,.--8B-Instteee on LongBench. The model is evaluated on the 15 English subsets using the official LongBench evaluation protocol, with up to 4K input tokens as context.
Method
NarrativeQA
Qasper MultiFielQQA
HotpoiQA
MuSiQue
2WikiMQA
GovReport
QMSum
Baseline
18.26
25.96
7.87
14.95
32.79
W4A4+GlowQ
14.68
10.80
24.95
14.21
8.39
14.20
32.01
22.01
W4A8+GlowQ
15.56
11.77
23.71
14.39
8.41
14.92
32.00
21.19
W4A16+GlowQ
15.46
11.82
23.68
14.39
7.77
14.53
32.32
21.20
MultiNews
LOC
RepoBenchhP
TriviaQA
SAMSum
TRec
PR
Avg
Baseline
26.95
47.00
87.76
44.72
70.00
37.50
34.19
W4A4+GlowQ
26.43
47.50
37.51
85,54
42.05
69.00
36.36
32.38
W4A8+GlowQ
27.03
51.50
35.97
84.10
42.62
68.50
37.08
32.58
W4A16+GlowQ
26.86
50.46
35.59
84.30
42.67
68.50
37.17
32.45
Table 14 15 shows that across both 4K and 8K context settings on the English LongBench bench-
2023b), applying W4 weight quantization with GlowQ (W4A4/8/16+GlowO) leads mark (Bai
to only small differences from the original LLaMA-3.1 tasks. On most tasks, the scores remain within few points of the baseline, and the relative dif- ficulty and ranking among tasks are largely preserved. This indicates that, even under aggressive quantization of both weights and activations, the low-rank correction GlowQ keeps the overall performance stable.
on the 15 English LongBench
When we extend the context length from 4K to 8K, both the baseline and the GlowQ models improve their average scores by similar margin. In other words, in scenarios that benefit from longer context, the GlowQ models track the same performance trends as the full-precision model, without a collapse in reasoning ability in the long-context regime. Overall, GlowQ enables 4-bit quantization while preserving LLaMA-3,0 1-B--BBBBBBBBBB 1-8---Inntrtuttssss performance not only standard contexts but also in long-context settings.
37

---

Published as a conference paper at ICLR 2026
Table 15: The results of ---ma-3.1-8--Insi on LongBench. The model is evaluated on the 15 English subsets using the official LongBench evaluation protocol, with up to 8K input tokens as context.
Method
NarrativeQA
Qusper
MultiFicldQA
HotpotQA
MuSiQue
2WikiMQA
GovReport
QMSum
Bascline
23.50
13.54
27.87
16.83
10.94
16.44
34.27
22.87
W4A4+GlowQ
23.45
12.20
27.41
15.34
9.21
16.15
33.87
22.78
W4A8+GlowQ
25.38
12.61
25,71
15.37
9.93
15.30
34.07
22.67
W4A16+GlowQ
25.36
12.61
25.62
15.13
9.82
15.20
34.00
22.59
MultiNews
L.CC
RepoBench-P
TriviaQA
SAMSum
TRec
PR
Avg
Baseline
26.87
52.81
48.04
90.77
43.94
71.00
73.13
38.19
W4A4+GlowQ
26.39
48.73
38.83
88.78
42.43
70.50
70.52
36.44
W4A8-GGlow
27.14
52.06
38.55
88.49
43.60
71.00
72.73
36.97
W4A16+Gloww
26.96
51.12
38.84
88.67
43.44
71.00
73.50
36.92
G SELECTIVE RESTORATION ACROSS MODEL FAMILY
singelar
aurmaaaedd leyer arder
Tralenita
got degiler valae Jam neeraaleed priee ralo
-frobesiac sero
-1001: sinilarily
8445
1.22
7.
8.35
835
1.12
7.1
190
Reslard
Redbed Gioepp
(a) LLaMA3.2-3B
(b) Qwen2.5-7B
fraction of restored groups for different restoration metrics. For each metric, we sort the groups according to its score (GSVD singular-value sum, normalized error ratio, Frobenius-norm error, cosine similarity, or simple layer order), progressively restore groups back to full precision, and record the resulting perplexity.
Figure 12: Perplexity
Importance metric selection. The performance of GlowQ-S depends on the policy used rank groups for restoration. In Fig. [2] we evaluate five saliency metrics from quantization and pruning literature. These include: (1) gsvd singular value sum, our gee score (Bq. 19). which measures the
captured error "energy" (|] ) in the low-rank factors and follows the standard practice of using
singular-value energy to summarize PCA components (Jolliffe & Cadimal 2016 Halko al. 2011);
(2) normalized error ratio, our Iner score ( Eg. 10). a widely used PTQ-style proxy based on relative weight error Eggl /||WWllr (Nagel al. 2021 Gholami et 2021 Krishnamoorthi 2018 ; (3)
frobenius norm error, the absolute етror Eg |F (Nagel et al. 2021 Pouransari et al. 2020 Zhao
ct al. 2025);; (4) cosine similarity, measuring angular deviation between pre- and post-quantization weights or activations, which has been shown to be strong pruning/quantization proxy (Mason- Williams Dahlqvist, 2024; Chang et al., 2023); and (5) layer order as simple baseline. The results show that gsvd singular value
and normalized error ratio consistently the most ef- fective, yielding the steepest perplexity reduction. However, as noted in Sec. Sec. no single metric is universally optimal. Therefore, final policy (Sec. pragmatically evaluates both fee and gner for a given model and selects the one that performs best, providing a robust, data-driven approach.
Results on Fig. 13 Across the four panels, LLaMA models exhibit a clear knee: perplexity drops steeply once a relatively small fraction of groups is restored, then plateaus. In contrast, Qwen and OPT show gradual, near-linear descent the restored fraction increases. The two evaluation curves in each subplot (ppl gsvd vs. ppl NER) track each other closely, differing mainly in the sharpness of the early descent.
38

---

Published as a conference paper at ICLR 2026
ppl ppl_ped
ppl
NER
156
155
6.78
15.4
6,73
15.3
6.68
15.2
6.63
15.1
6.58
10
20
30
40
50
60
70
300
90
100
0
17
33
50
67
83
100
Restored (roup(7))
Restored (roup(7))
(a) LLaMA3.1-8B
(b) OPT-1.3B
ppl ppl_gadd ppl
gadd -nt NBR
9.1
10.25
10.2
10.1
8.95
8.9
10
8.85
9.9
8.8
9.85
0
20
40
60
80
100
70
Restored Group(%)
Restored Group(7)
(d) Qwen3-14B
(c) Qwen3-8B
A
Figure 13: Perplexity as function of restored group percentage for dif- ferent model families (LLaMA 3.1-8B, Qwen 3-8B, Qwen 3- 14B, OPT-1.3B), We compare GSVD- based restoration (ppl gsvd) against NER-based restoration (ppl NER).
These curves suggest selecting the enror-reoveery metric per model family: outlier/energy rank- ing with small budgets for knee-shaped profiles, and Hessian-llos-weighted ranking with broader budgets for diffuse profiles. This family-aware policy aligns with known outlier, anisotropy, and curvature phenomena in modern LLMs.
39

---

Published as a conference paper at ICLR 2026
Table 16: Zero-shot results on LLaMA 3.2-3B.
Method
Rank
PIQA
ARC-C
ARC-E
HellaS
WinoG
BoolQ
LAMBADA
C4
AVG
Acc 1
Acc
Ace
Acc-norm
Acc
Acc f
Acc 1
word PPL
Acc 1
FP16
72.33
39.33
72.33
63.67
70.33
77.00
71.00
10,30
67.14
ZeroQuant- V2
75.33
39.33
73.33
60.67
68.67
74.67
65.67
11.45
65.38
QERA
76.67
38.67
72.33
61.67
68.67
72.33
64.67
11.04
65.48
L2QER
64
75.33
40.00
71.67
64.00
68.33
73.33
68.33
11.04
66.19
GlowQ
77.67
39.67
72.00
64.00
70.33
74.33
70.33
10.98
66.90
GlowQ-S
77.33
39.67
71.67
64.00
69.67
71.67
70.33
11.07
66.33
Table 17: Zero-shot results on LLaMA 3.1-8B.
Method
Rank
PIQA
ARC-C
ARC-E
HellaS
WinoG
BoolQ
LAMBADA
C4
Acc 1
Acc
Acc
Acc-norm
Acc
Acc t
Acc f
word PPL
Acc f
FP16
78.67
51.67
80.67
67.67
74.67
80.67
79.00
9.00
73.29
ZeroQuant-V2
78.00
51.33
81.67
68.67
76.00
84.33
74.33
9.87
73.48
QERA
77.00
51.33
80.33
69.00
74.33
82.67
75.33
9.68
72.86
L2QER
64
79.67
49.33
80.67
66.67
74.33
80.33
76.00
9.63
72.43
GlowQ
79.67
51.00
81.33
66.00
74.33
82.00
79.00
9.59
73.33
GlowQ-S
79.00
50.33
81.67
66.33
72.00
82.00
77.00
9.78
72.62
Table 18: Zero-shot results on Qwen 3-8B.
Method
Rank
PIQA
ARC-C
ARC-E
HellaS
WinoG
BoolQ
LAMBADA
C4
Acc f
Acc
Ace
Acc-norm
Acc
Ace f
Acc
word PPT. J
Acc f
FP16
77.33
$3.00
83.00
63.67
68.67
87.00
67.67
14.52
71.48
ZeroQunntVV V2
75.67
$2.33
80.33
63.00
71.00
85.33
63.67
15.00
70.19
QERA
76.33
51.33
79.00
62.33
69.67
85.67
64.67
14.78
69.86
L2QER
64
75.67
51.33
79.33
62.67
67.67
85.33
64.67
14.82
69.52
Glow
76.67
52.33
80.33
64.67
71.00
86.33
63.67
14.60
70.71
GlowQ-S
76.33
50.67
80.67
63.33
70.67
85.00
65.33
14.77
70.29
Table 19: Zero-shot results on Qwen 3-14B.
Method
PIQA
ARC-C
ARC-E
HellaS
WinoG
BoolQ
LAMBADA
C4
Acc f
Acc
Ace
Acc-norm
Acc
Acc t
Acc
word PPL
Acc t
FP16
78.33
59.33
80.33
66.67
75.67
92.00
66.33
13.08
74.10
ZeroQuant-Vi V2
78.33
59.33
78.00
65.67
73.00
92.00
62.00
13.79
72.62
QERA
76.98
57.67
79.33
67.00
74.00
92.00
65.00
13.29
73.14
L2QER
64
78.33
56.33
79.67
66.67
75.33
91.67
64.67
13.80
73.24
GlowQ
77.67
56.67
80.00
75.67
91.33
66.67
13.26
73.84
GlowQ-S
77.67
57.00
79.33
67.67
74.33
91.33
65.33
13.48
73.24
40

---

Published as a conference paper at ICLR 2026
Table 20: Zero-shot results Vicuna-7B.
Method
Rank
PIQA
ARC-C
ARC-E
HellaS
WinoG
BoolQ
LAMBADA
C4
AVG
Acc
Acc
Ace
Acc-norm
Acc
Acc t
Acc
word PPL
Acc
FP16
76.00
41.33
70.33
68.00
80.33
72.33
8.70
ZeroQuant-
75.67
43.00
71.00
65.33
68.33
9.07
QERA
76,00
42.67
70.67
67.00
67.33
81.00
69.67
8.91
67.76
L2QER
76.33
42.33
70.00
66.00
67.00
68.00
8.93
67.19
GlowQ
75.67
41,67
70.00
66.67
67.00
80.33
69.67
8.87
67.29
GlowQ-S
76.00
43.67
69.33
66.00
66.67
82.00
70.00
8.99
67.67
Table 21: Zero-shot results Vicuna-13B
Method
Rank
PIQA
ARC-C
ARC-E
HellaS
WinoG
BoulQ
LAMBADA
C4
AVG
Acc
Acc
Acc
Ace-norm
Acc
Acc t
Acc
word PPL.
Acc
FP16
77.33
49.33
73.67
67.33
74.33
86.00
74.33
7.76
ZemoQunnt- V2
77.67
46.67
76.00
66.33
75.00
86.00
74.00
7.86
71.67
QERA
78.00
47.33
76.33
67.33
75.33
86.00
73.00
7.88
71.90
L2QER
77.67
48.67
75.67
67.00
74.00
84.67
74.00
7.79
71.67
GlowQ
78.33
47.67
75.67
67.33
74.67
85.33
74.00
7.85
71.86
GlowQ-S
77.67
57.00
79.33
67.67
74.33
91.33
65.33
7.86
73.24
41

---

Published as a conference paper at ICLR 2026
LLM USAGE DISCLOSURE
WRITING POLISH
After completing the full draft, used a large language model (LLM) purely aid proofreading and light copy-editing. Specifically, the LLM suggested fixes for grammar, spelling, punctuation, typographical errors, and minor wording for clarity and consistency.
RETRIEVAL AND DISCOVERY
We also used LLM literature discovery assistant to broaden search beyond papers had already identified. The LLM helped generate alternative keywords and surface potentially relevant works.
42


Reviews:
[{"review_id": "Lnp68c0nsF", "reviewer": "Reviewer_CZae", "summary": "This work describes an efficient low-rank compensation technique for weight-quantized LLMs. \nThe method finds an optimal shared down-projection factor in a data-driven manner, with whitening and grouping treatment.", "questions": "I have listed major questions in the Weaknesses section above.  Here are a few minor questions in addition.  \n\n* The empirical statistical analysis of the quantization error is essential to this paper.  Do you have any results on any empirical scaling laws of the statistics, in addition to simple qualitative descriptions such as long tail?  It might be helpful to apply logarithmic scale in Figure 2 to expose certain power laws.  \n* Is there any results on how and how well this method could be applied to MoE FFN layers?\n* In addition to full-precision error correction, other orthogonal methods use mixed-precision, how is the choice among these methods subject to practical tradeoff?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The idea is sound, with adequate proof. \n+ The method is relatively well described.  \n+ There is potential practical significance.", "weaknesses": "- The empirical, data-driven, low-rank correction of quantization error is dependent on the data type and its precision.  As most of the results presented are with INT4, there is a lack of demonstration on the effectiveness for different data types and precisions--do the data statistics differ qualitatively under those conditions?  \n- As activation quantization is also important in practice, it is not clear how quantization of the activations, in combination with weight quantization, change the story. \n- Quantization errors tend to accumulation over time across tokens as well, which is particularly relevant in long-context reasoning settings.  Reporting accuracy in language modeling perplexity is insufficient in addressing this issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MKNG52BFMq", "reviewer": "Reviewer_DXnt", "summary": "This paper introduces GlowQ, a group shared low-rank approximation method, that a single rightside projection is used for a group with same input. A BX caching method is proposed to make it deployable. A selective method is proposed to further reduce the memory and latency.", "questions": "- In Fig.3, as PPL is lower the better, what is the definition of the percentage of PPL?\n- In Tab.4, No Caching means GlowQ without caching? Or a layerwise method? If it means GlowQ without caching, the authors should provide performance comparisons with layerwise methods.\n- Modern GPUs already natively support other narrow-bit formats such as MXFP4, NVFP4 and MXFP6, whose peak throughput is significantly higher than that of INT4. The paper should therefore clarify:\na) Is GlowQ directly applicable to these FP4/FP6 formats, or does its low-rank projection rely on integer-only operators?\nb) If applicable, how does GlowQ’s accuracy compare with simply running the network in MXFP4/NVFP4/MXFP6 without any additional compression?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper is technically sound.\n- The experimental results show the effectiveness of the proposed method.", "weaknesses": "- Writing and presentation need refinement. For instance, the symbol E in Eq. 1 is identical to E_cat in Eq. 2—consistent notation should be used. In Fig. 1 (inference-path sub-figure), the chosen colors are too similar to be distinguished; a more discernible palette is required.\n- For GlowQ-S, the fraction of restored groups are not provided, should be provided for each experiment", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OHcQPZf1tk", "reviewer": "Reviewer_HaNH", "summary": "This paper proposes GlowQ, a group-shared low-rank correction method for quantized LLMs that improves both accuracy and efficiency by sharing a single right-factor matrix across input-sharing modules (e.g., Q/K/V), caching the projection to avoid redundant computation, and aligning the correction subspace with data usage via covariance-weighted optimization. Combined with a selective restoration strategy (GlowQ-S), it reduces TTFB by up to 23.4% and increases throughput by 37.4% with minimal accuracy loss, outperforming existing baselines in perplexity and downstream task performance while remaining compatible with standard PTQ pipelines.", "questions": "1. The performance of GlowQ-S is highly dependent on the importance scoring functions (such as gec and gner). I am curious if there are other metrics that can indirectly reflect the effectiveness of these scoring functions. In other words, can these two metrics be proven to be globally optimal across a range of models?\n2. The method relies on the assumption that modules within an input-sharing group can share a common right factor ( B_{\\text{shared}} ), but it does not systematically verify the validity of this assumption across all layers or architectures. For example, when the functions of matrix projections like Q and V are different, it is unclear whether enforcing this sharing introduces bias.\n3. The confinement to W4A16 is a major limitation. Can you provide any results or analysis for a W4A8 or W4A4 setting? How does your group-sharing approximation hold up when the input X to the correction path is also quantized, introducing another layer of error?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. Efficient Group-Shared Correction via Caching: GlowQ reduces redundant computation by sharing a single low-rank right factor across input-sharing modules (e.g., Q/K/V projections), enabling one-time computation and reuse of the projection R=BsharedXR=Bshared​X, which significantly lowers computational overhead during inference.\n2. Data-Aware Subspace Alignment: By incorporating input covariance into the low-rank approximation objective, GlowQ aligns the correction subspace with directions that are most frequently activated in practice, enhancing recovery accuracy without increasing rank or parameters.\n3. Scalable and Deployment-Friendly Design: The method combines a QR-reduced randomized SVD solver for efficient training with a selective restoration strategy (GlowQ-S), achieving substantial latency reduction (up to 23.4% lower TTFB) and throughput improvement (up to 37.4%) while maintaining compatibility with existing post-training quantization pipelines.", "weaknesses": "1. While the group-sharing idea is technically sound and well-executed, the central concept is an adaptation of well-known collective matrix factorization and SVD-sharing across blocks, now framed in the context of quantized LLMs. The extension to covariance alignment is also a known trick, though its synergy with group correction is effective.\n2. While the paper compares GlowQ extensively to prior PTQ and error-corrected LLM quantization baselines (e.g., AWQ, GPTQ, L2QER, QERA, ZeroQuant), it omits direct comparison or numerical discussion of very recent approaches, such as rotation-based saliency-aware quantization methods (e.g., ROSAQ), vector quantization for KV cache (CommVQ, AnTKV), and loss-guided PTQ (GuidedQuant).\n3. This method requires an additional calibration phase to estimate the compensation matrix, but the paper does not seem to list the time and memory costs for the training/calibration phase. Additionally, storing intermediate activations and error matrices for larger models (e.g., 30B+) could lead to significant memory pressure.\n4. The paper claims the use of \"custom CUDA W4A16 kernels,\" but it does not describe the implementation details or whether key techniques such as operator fusion were used in the low-rank correction modules. Therefore, it is unclear whether the comparisons between different methods are made at the same level of optimization.\n5. Critically Limited and Potentially Misleading Experimental Scope (W4A16 only): The paper's most significant flaw is that all experiments are conducted in a W4A16 setting. This is an unrealistic and overly forgiving scenario that ignores the primary challenge in modern PTQ: activation quantization. By using FP16 activations, the main computation path and the input to the correction module remain noise-free, which dramatically simplifies the problem. The paper's claims of improved accuracy and efficiency are therefore unsubstantiated in any practical low-bit setting (e.g., W8A8, W4A4). It is entirely possible that the proposed group-sharing benefits would vanish or even become detrimental once activation quantization noise is introduced.\n6. format error: line339 -> BitsAndBytes ?, line218,268,290,293 -> Missing formula number.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)