CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attention
Authors: Zhongzhu Zhou
Fengxiang Bie
Ziyan Chen
Zhenyu Zhang
Yibo Yang
Junxiong Wang
Ben Athiwaratkun
Xiaoxia Wu
Shuaiwen Leon Song
Venue: ICLR 2026 Poster (2026)
Keywords: Multi Latent Attention
Covariance & Rank aware
Singular value decomposition
Primary area: generative models
TL;DR: CARE converts pretrained GQA/MHA to MLA at KV-parity via covariance-aware SVD and adjusted-rank allocation, reducing perplexity up to 215x and improving accuracy up to 21 points over baselines on Llama-3.1-8B/70B and Qwen3-4B/30B-A3B.

Abstract: Converting pretrained attention modules such as *grouped-query attention* (GQA) into *multi-head latent attention* (MLA) can improve expressivity without increasing KV-cache cost, making it attractive for efficient inference. However, many practical conversion baselines rely on weight-only low-rank approximations (e.g., SVD-style initializations) and uniform rank allocation. They focus on minimizing the difference between weight matrices rather than on how those weights affect input activations, ignore the covariance structure of activations, and enforce uniform rank across layers—causing activation drift and degraded attention fidelity. To address these issues, we propose CARE, a ***C**ovariance-**A**ware, **R**ank-**E**nhanced* MLA conversion pipeline under a fixed KV width. CARE introduces three key steps: (i) *activation-preserving factorization*, which aligns the approximation with the actual input activations rather than just the weights; (ii) *adjusted-rank allocation*, which spreads a fixed KV budget across layers by giving more capacity to layers that need it most; and (iii) *KV-parity mapping*, which reparameterizes the converted K and V to fit the MLA format while keeping the KV-cache size unchanged. Our method outperforms a uniform-rank SVD baseline on Qwen3-4B/30B-A3B-Instruct-2507 and Llama-3.1-8B/70B-Instruct, reducing one-shot perplexity by up to 215× and improving mean accuracy by up to 1.70× at matched KV budgets. With a brief post-SVD "healing" fine-tune, we fully recover the original model's accuracy.

Published as a conference paper at ICLR 2026
CARE: COVARIANCE-AWARD AND RANK-ENHANCER DECOMPOSITION FOR ENABLING MULTI-HEAD LA- TENT ATTENTION
Zhongzhu Zhou Fengxiang Bie', Ziyan Chen', Zhenyu Zhang', Yibo Yang', Junxiong Wang", Ben Athiwaratkun" Xiaoxia Wu Shuaiwen Leon Song!.
1 University of Sydney, 2 King Abdullah University of Science and Technology,
3 Together AI, University of Texas at Austin
ABSTRACT
Converting pretrained attention modules such as grouped-quer attention (GQA) into multi-head latent attention (MLA) can improve expressivity without increasing KV-cache cost, making it attractive for efficient inference. However, many practical conversion baselines rely on weight-only low-rank approximations (e.g., SVD- style initializations) and uniform rank allocation. They focus minimizing the difference between weight matrices rather than on how those weights affect input activations, ignore the covariance structure of activations, and enforce uniform rank across
activation drift and degraded attention fidelity. To address these issues, propose CARE, Covariance-A ware, Rank-Enhanced MIAA conversion pipeline under a fixed KV width. CARE introduces three key steps: (i) intivatioon-reseevvin factorization, which aligns the approximation with the actual input activations rather than just the weights; (ii) adjusted- rank allocation, which spreads a fixed KV budget across layers by giving more capacity to layers that need it most; and (iii) KV-parity mapping, which reparameterizzs the converted K and V fit the MLA format while keeping the KV-cache size unchanged. Our method outperforms uniform-rank SVD baseline on 4Been---//00B--BB---- IB--Instrcct22507 and Llama-3.1 1.1 1-8B/70B--nstruc reducing one-shot perplexity by up to 215x and improving mean accuracy by up to 1.70x at matched KV budgets. With a brief post-SVD "healing" fine-tune, we fully recover the original model's accuracy.
INTRODUCTION
Large Language Models (LLMs)) deliver impressive capabilities but at high inference cost, with the key-value (KV) cache in self-attention emerging as primary memory and bandwidth bot- tleneck (Vaswani et al., 2017] Kwon et al., 2023). In the standard multi-head attention (MHA) formulation, each head materializes and caches its keys and values at every decoding step. causing the KV footprint to grow linearly with sequence length and head count. To alleviate this, ar- chitecture variants such multi-query attention (MQA), which shares single K, V across all heads, and grouped-query attention (GQA), which shares K, V within head groups, have been adopted scale shrink KV cache size (Shazeer 2019; Ainslic al. 2023 Touvron et al. 2023 Jiang let al. 2023 Chowdhery el al., 22022; Shoeybi et al. 2019). While effective, these variants reduce the number of distinct key/value projections, which can limit attention expressivity and introduce quality regressions when compression is pushed aggressively.
more recent line of work reframes the KV-cache problem as one of learned low-rank representa- tion (Wang et al. 2020 Xiong et al. 2021). Multi-Head Latent Attention (MLA) compresses keys and values into dimensinniional latent vectors, caches only these latents, and restores expressivity with lightweight up-down projections at compute time (DeepSeek-A) Team (024). In practice, MLA can dramatically reduce KV size while preserving or even improving task accuracy by trading memory
"qqual advising.
"Correspondigg author: shi rel relyy@tgetterr... .ai, Code is available httpa:///t wtt.:ithbb.ooooooooooooooooooott Fut

---

Published as a conference paper at ICLR 2026
"Wrong" SVD Target
Covariance "Correct" SVD Target
Rank Scheduling
W
Down
Rank
Up
Samples
Layer 0
Upxv
Down
Down
Rank
Up
W
W
Partial Rope
Layer N
Kmpe
Concat...
Kaspe
X Wr*
Cape
W
Kmp
(a) Naive (Joint) SVD
(b)
Figure (a) Naive MIL transfer: jointly factorize W and W/( by SVD and truncate uniform per-layer rank, optimizing | | while ignoring layerwise heterogeneity. (b) CARE: estimate activation covariance factorize vCW, unwhiten via V - to initialize MLA factors, and use the singular spectrum of vCW for global dynamicrank scheduling under KV parity. This preserves activation geometry and yields a stronger one-shot initialization with less healing.
and communication for modest extra Dpatiin--ointtoppratiee (FLOPs) in the projections (DeepSekk
AT Team 2024 Guo et al. 2025 Liu et al. 2024a Geens & Verhelst 2025). Despite these advantages.
the ecosystem is dominated by pretrained MHA/GQA checkpoints (Tourron et al. 2023 Diang et al.
2023 Yang et al. (0024a). Retraining large models from scratch under MLA is expensive, so natural
question arises:
Can we convert strong, pretrained MHA/GQA models into MLA post-hoc, without increasing the KV budget and without incurring large performance loss?
Recent work has explored converting traditional attention (MHA/GQA) into multi-head latent atten- tion (MLA) under fixed KV width. TransMLA (Meng al., 2025) demonstrates that every GQA layer admits an equivalent MLA parameterization and proposes practical post- post-training mapping followed by light finetuning. MHA2MLA (Ji et al.
2025) generalizes to MHA-MIAA by addressing positional encoding mismatches (e-g., partial RoPE adjustments) and initializing W x, Wy with low-rank joint SVD before efficient recovery (Su et al. 20211). X-EcoMLA (Li et al. 2025b) constructs MLA from pretrained MHA via structured SVD decomposition and rross-layer distillation based parameter recy- cling for efficient latent projection initialization. In parallel, general purpose SVD-based compression and approximation methods (e.g., SVD-LLM V2 (Wang et al.) 2024b] (0256) and activation-awan SVD (ASVD) (Yuan et al., 2023)) improve truncation and orientation beyond naïve weight SVD, while cache-centric baselines such Palu (Chang et al. 2024) reduce KV memory via low-rank KV-cache projection. Together, these works establish MLA as promising post-hoe target and highlight low-rank factorization as central to preserving pretrained knowledge under KV-constrained
reparameterizations (Hu et al. 2022 Denil et al. 2013 Denton et al. 2014
Sainath et al. 2013
Eckart & Young 1936).
However, direct SVD initialization has two key shortcomings. First, it minimizes error in weight space (I WII rather than activation space X XW ignoring how the projection actually operates during decoding (Hassibi al., 1993] Wang al. [024bb Yuan et al.) 2023). This mismatch induces attention-logit drift even when the weight approximation is accurate. Second, it enforces a uniform rank across layers, neglecting differences in spectral structure. Layers with fast spectral decay over-compressed, while those with slower decay under-compressed, leading fidelity loss and heavier reliance on post-conversion finetuning.
To address the above two shortcomings, we propose CARE, Covariance- Aware, conversion pipeline, as shown in Fig. First, CARE makes the decomposition activation -wwrre: rather than applying vanilla SVD to W solve whitened approximation problem by applying
2

---

Published as a conference paper at ICLR 2026
SVD to VCW and then unwhitening obtain W where C' summarizes imput activation covariance estimated from a modest calibration set. This ensures that dominant activation directions are preserved and substantially reduces attention-logii error before any finetuning. Second, CARE is rank-adaptive: distributes a fixed KV budget across layers and heads based on their singular spectra, allocating higher rank to spectrally complex matrices and lower rank to intrinsically low-rank ones, akin in spirit to budgeted, importance-aware adapter methods (Zhang et al. 2023 Valipour et al. [2023] | Hu al. 2022, Wang et al. 2025a). This budgeted, importance-awart scheduling maintains fidelity under the KV constraint while reducing reliance on post-conversion finetuning.
Contributions.
Activation-aware Initialization. We propose covariance-awar factorization that min- imizes activation error || --V-XP -V (rather than weight error), implemented via SVD on a whitened operator and subsequent unwhitening. This preserves attention logits more faithfully at equal KV budget to initialize MLA weights.
Rank-adaptive scheduling under fixed KV width. We introduce singular-value-guis allocation that distributes rank unevenly across layers/heads and the {K, VJ matrices, matching spectral difficulty and improving zero-shot fidelity compared with uniform ranks.
KV-parity mapping and practical pipeline. We derive KV-parity reparameterization for MLA conversion and integrate the above techniques into a practical conversion pipeline CARF-converted models exhibit lower activation error and improved task quality over naive (joint) SVD baselines at equal KV cost, while requiring less data to recover residual gaps.
2 NAÎVE (JOINT) SVD IS NOT ENOUGH FOR MLA TRANSFER
Multi-head latent attention (MLA) transfer is often initialized with singular value decomposition (SVD), either per matrix or via a joint factorization across related matrices (e.g., WK. K. Wv). While convenient, this practice implicitly optimizes weight-space eтror || W |||| p and assumes that the spectrum alone reveals task importance, In this section we show both assumptions break down in practice and, consequently, naive (joint) SVD is unreliable recipe for high-fidelity MLA transfer under fixed KV budget.
Layer-wist Sensitivity under 0.5 Ratio Compression
Llama-3 8B Perplexity vs. Rank Group (Monotonic Trend)
Perplexity [WikiText] Text))
Trend
175
150
125
100
75
ABC Challenge (0.5 ratio)
(0.5 ratie)
Basellnc
MMLU Beseline
10
20
25
Rank Group
Layer Index (a)
(b)
Figure 2: (a) Accuracy under 50% rank reduction applied one layer at a time DeepSeek- V2-Lite, measured ARC Challenge and MMLU. Sensitivity is strongly layer-dependent. (b) WikiText perplexity under grouped truncation of GQA attention in Llama-3-8B (layers 30-32). Singular value groups ordered by magnitude; the resulting non-monotone degradation shows that singular values alone are an imperfect proxy for MLA conversion quality.
Observation 1: Accuracy-preserviig rank is not uniform across layers. Fig. (a) exhibits pronounced layer-wise heterogeneity when halve the rank of every layer: layers tolerate aggressive reduction with negligible loss, whereas others incur sharp drops on ARC and MMLU. A
'eere 'whiten' means to use / VCW instead of W our target compress, and 'unwhiten' means the inverse map which recovers the compressed W.
3

---

Published as a conference paper at ICLR 2026
one-size-fits-al policy (uniform pruning or fixed ratio per layer) either over-compresses fragile degrading task performance or under-compresses robust layers wasting KV budget.
Observation 2: Singular values are poor proxies for accuracy importance. A common heuristic treats singular values as importance scores, expecting that truncating smaller values should least affect accuracy. We directly test this with brute-force ablation: given W UEVT we set the i-th singular V treating V as compress of MLA and U as expand of MLA. As shown in 2 the link between singular-value magnitude and downstream accuracy non-monee non-monotonic. We conjecture the root cause is mismatch of objectives and statisties: vanilla SVD minimizes weight error, not activation-span space error XW XW under the true (anisotropic) input distribution.
value zero and reconstruct W66) U diag(o O, .,.... σr)
Thus, naïve (joint) SVD isn't enough: (i) the rank needed to keep accuracy varies by layer, and (ii)
singular values alone don't 't reflect accuracy importance of initialization.
3 CARE: COVARIANCE- AND LANK-ENHANCED MLA CONVERSION
We propose CARE, post-hoe conversion pipeline that maps a pretrained MHA or GQA layer to an MLA layer at the same KV budget, while explicitly minimizing activation error and adapting ranks to spectral difficulty. CARE revisits low-rank factorization through the lens of activation statistics and integrates covariance- wwrihteee--eeighted SVD and non-uniform rank allocation into a single, practical procedure compatible with multi-models' backbones.
Notation. Given a fixed layer with input activations X RTTx with length T and embedding dimension D. The multi-head attention in our setup contains n, heads of size da, where we assume the output space corresponds with imput space, formally n, = D. We denote gn to be the number of GQA groups, where for this layer: Q XWQ. K = xw IRT x (gndu) with gh < nh, with w represents the weight matrix under the GQA setup. By contrast, an MLA layer with latent rank uses K (XW V (XWG) where W7.) Pxr, Wh E (mad)) Only the latent XW RTx is cached, with W E (rx(m)d)) used to recover KV matrix.
K (andd)))) V = xW
3.1 PRELIMINARY: COVARIANCE FOR INPUT ACTIVATIONS
Let x ) RT be the 6th batch of the domain activations with length T₁ some layer (note that the length of tokens remains the same across layers, but can be different across batches). These batches of domain activations are used calculate the extent of preserved rank Sec. 3.2 and initialize the trainable parameters later in Sec. 3.3
We then define ((), the covariance matrix over all the N batches at layer I, as follows:
-²=
= N
b=1
Note that even we denote the above (1) to be covariance matrix, it is slight different from the definition of actual covariance: we do not apply centralize and normalize operation here (see App. for formal definition).
3.2 ADJUSTED-RANE SCHEDULING ACROSS LAYERS
Due to heterogeneous key/value spectra across different layers, the retained rank of each layer using MLA supposed to be different. Let W (Wil) w Miz1 represent the pretrained KV weights from Sec. B, where L represents the number of layers. Note that W contains 2L weight matrices, each with dimension D nydy (recall D n,dd))
4

---

Published as a conference paper at ICLR 2026
Given total key-rank budget (K) (same for value), score the rank by its normalized residual reduction. Let
(g
Pkm
Σ Kit
where K.m (same for V) is the m singular value of VCOW K C is the covariance
(1)
matrix defined in Sec. 3.1 and R& is the rank of VCOW K K we define the priority of allocating one more rank as
For layer eurrently assigned rank
(o Kir
(1) K T
(1)
Krg
18 +1
(1) Pk
#1 (0 (1)
Appendix C proves that the Frobenius residual after rank-r truncation equals the squared tail energy; normalizing by this residual makes layers with different spectral scales comparable. We allocate the budget R) by greedy water-filling: starting from 7 =C for all | where C' is some constant, (1) repeatedly assign one rank to /* = arg maxi 8 K (rk until the budget is exhausted; V is handled identically with its own budget.
3.3 RETHINKING SVD WITH ACTIVATION COVARIANCE
A naive rank lowering attempts to minimize Frobenius error |IW - -WIFF. where W represents the de-ranked matrix for compressing, and the original pretrained weight matrix to compress, W w()) E W, lies in layer I. For inference fidelity, we propose the relevant objective to be minimizing the empirical activation error. Focusing compressing the certain weight matrix W,, we denote to be the target compressed rank calculated in Sec. 3.2 Here fX, == x!() RT 1b=1 represents the small domain activation batches to compute the low-rank decomposition at layer I, where N is the total number of batches, T. is the length of each batch and C == (1) is the covariance matrix defined in Eq. 3.1
Formally, we try to minimize the following:
N -
- -Xa
min rank(W WISt
b=1
This optimization formalizes how well W preserves X,W on relevant inputs Xs. From the definition of C, one can show that 1 MN - For K and V separately, compute
- X,,
= VCC - W) ||:: see App. for the proof.
VCw = UEVT, then W vc U.E.V.
(3)
with Ur, Er, V, the top-r components of U, U, V. In practice, we use shrinkage VC, a) + aAI ensure C is invertible, with (0, 1) and À > 0.
Then we initialize the trainable parameters 4 e Pxr and W e (m)d) s.t. WoW equals the compressed matrix. We map SVD factors to MIA by
Wa
VC E, RTxr in MIAA spans the principal activation
W V.T.
so that W W. The cached latent XW subspace, where X is the actual input at layer l.
3.4 100% MIAA CONVERSION BY HEALING
Based on our initialization of down-and-up matrices W", Wh and number of tokens, we attempt to encode positional information in the MLA-like attention mechanism. Given layer 1, let Q. X, X,Wo
5

---

Published as a conference paper at ICLR 2026
be the usual query at step and let Kc,, (X,W3)WR and Vc, (x, W be the MLA generated keys/values. Following the decoupled RoPE design in (DeepSeck- Team 2024), we bring small RoPE channel of width d, by introducing new trainable matrices
W E DD((ndd)),
W R K E Dxd,
where W maps directly from the activation X to a shared RoPE key of width d, (DeepSeek-like decoupled RoPE). Let R, E (m,,dd x((ad,) denote the standard block-diagonal RoPE rotation at step (one d, x d, rotation per head), and let Rg E Rd,xd, denote the per-head rotation applied to the shared RoPE key. We compute:
QR. (x) Rr.
KR, repeat X, W Re),
where repeat replicates the shared RoPE key across heads, yielding KR, (Ix(()d)), We then concatenate channels and compute attention:
(K()) ddn d, O₂ = Vo,
O7 = [Qc: QRe]: K2 = [KK,a KR.c,, A; = Softmax
where A, denotes attention weights and O denotes the layer output.
Caching and a compact "join" form. To preserve KV-cache efficiency, we cache only the KV-side latents X,W K and W, along with the small shared RoPE key (X,W3, R.WRR.E R, (repeated across heads when forming K n.z). Queries (including Qn,)) are computed on-the-fly. Equivalently, the ML-A-generaedd KV can be written compact.joines form:
Concat(Ke.a, Vc.)) = Concat(X,W2 X.W W Wjoins
(5)
where Concat(:s.) concatenates along the last (feature) dimension and Wjoin = blkdiag(W3,, W E 122xxD jointly represents the two bilinear maps W W and Wyw.
We penalize the low-rank decomposition by its cross-entropy classification error and KL--divergence imitation error, namely the loss functions:
LCe
-t
logp (1141 [zg)).
(6)
t=1
LKD = 1
Σ
KL (oftmaxx
(tttmax(2//7))
(7)
1=1
L= LCF + 3 LKD.
(8)
Here y (x++1 (( the student next-token probability under the converted MLA layer (with RoPE
adapters WR. W and MILA factors W W while 2; and =f are teacher and student logits. We
use (11+1 Ige) = softmax + )am. with temperature r and weight 3 (Hinton et al. 2015
EXPERIMENTAL RESULTS
We evaluate whether CARE improves MLA migration under a fixed KV-cache budget (KV-parit). We report one-shot and post-healing perplexity & accuracy, ablation study, long-context retrieval (NiH), and system efficiency.
Setup. We match cached KV state per token under the same global budget (Tab. CARE estimates
C' = Cov [X] on small calibration set and factors VCW with shrinkage VC + (1 - X/VC + XI; use LM Harness (Gao et al.) 2024) with a matched heal budget; full 100% MLA restoration (TranSLLAMHH2MLA)) is evaluated separately in Sec. 4.3
TransMMLA Meng et al.
2025 and X-FcoMI.A Liet al.
2025b both describe how to add such a RoPE
channel.

---

Published as a conference paper at ICLR 2026
Original, baselines, CARE variants, and datasets. GQA (source). Unmodified grouped-query attention. We compare against K KV-compression baselines: Palu (Chang et al. 2024). a low- rank projection baseline for KV-cache reduction under a fixed cache budget: ASVD (Yuan et al. 2023). activation-aware SVD applied to (Wx, Wv); and SVD-LLM V2 (Wang et al., 2024bb 2025b). truncation-aware SVD-style factorization applied to (WK. Wv). ). We also include conversion baselines TLa,sMLA, MHAZMLA (Meng et al., 2025; Ji et al. 2025). We report their SVD-style initialization in the one-shot setting, while full 100% MLA restoration is evaluated in Sec. 4.3 CARE- U (uniform-rank). Covariance-awerr aware factorization with a uniform rank across layers. CARE-E factorization, but with our adjusted-rank allocation. We evaluate on LM Harness tasks: Wikitext2 (Wiki) (Merity et al., 2016). ARC Challenge (ARC),
(adjusted-rank), The same covariance-awe
ARC Easy (ARE) (Clark et al. 2018), HellaSwag (Zellers et al. 2019). PIQA (Bisk et al.) 2020).
MMLU (Hendrycks et al. 2020), OpenBookQA (OBQA) (Mihaylov et al. 2018). RACE (RA) (Lai
ct al. 2017). and WinoGrande (WG) (Sakaguchi et al. 2021). Hyperparameters are in App.
4.1 ONE-SHOT RESULTS
We report one-shot, direct KV-compression performance under KV-parity before partial-RoPP and healing on Alpaca Taori et al. (2023) calibration dataset; Tab. [ summarizes the main results for LJama3. and In-sttttt-ss App. 1.1 reports additional one-shot results on other models and calibration datasets.
Table I: One-shot comparison Llama3,1-88 and (ree3--B-Instruet-22 against original and baselines on multiple tasks. Calibration samples: 256. Sequence longth: 2048. Calibration dataset:
alpaca. Higher is beller for Accuracy (%) (ACC.) (1) and lower is better for Perplesity (PPL.. (4).
PPL (
ACC (T)
Rank KV Save
Methods
ARC (L)
ARE HellaSwag
PIQA
MMLU
OBQA
RA
WG
AVG
Llama-3.1-88 1amm--3.-8B-Instrucc
GQA (Original)
(1)
79.65
48.05
34,80
40.10
72.69
58.24
MHRZMDA
284863.91(.)
25.94
23.82
26.34
50.92
25.62
27,80
22.11
50.59
31.64
CARE-U (OURS)
983,55
23.89
30.51
26.82
54.73
23.18
26.00)
20.96
49.01
31.89
64
93.75
CARE-E (OURS)
98).03(1)
23.63
31.31
27.49
54.62
22.98
27.40
21.15
50.36
32.37
Palu ISVD)
2260.60(L)
25.77
27.53
26.50
52.18
24.26
27.80
20.86
50.04
31.87
SVD-LLM V2
(4)
26.75
54,90
23.22
26.40
20.86
48.93
31.93
ASVD
2525.33
23.81
26.68
26.68
52.18
22.97
27. 37.80
20.86
50.99
31.50
MARZMLA
15028.91 (1)
25.00
24.12
26.58
52.18
29.20
22.
51.07
31.76
CARE-U (OURS)
(L)
26.71
41.96
33.64
60.99
26.06
27.00
24.21
53,20
36.72
128
87.50
CARE-E (OURS)
(L).74(L)
27.30
42.47
37.96
62.46
29.38
28.20
26.12
54.85
38.59
Palu (SVD)
(0)6.58(L)
23.89
27.02
26.62
50,38
23.14
24,80
22.49
49.25
30.95
SVE-LIM V2
38).23
27.56
42.38
34.06
61.04
26.17
27.80
24.21
$3.99
37.15
ASVD
(L75.54
25.00
27.99
26.61
$2.39
23.04
26.60
31.49
MHAŽKLA
1633.65 (L)
27.47
25.63
26.50
$2.39
23.10
28.20
22.49
50.83
32.08
CARE-U (OURS)
48.35
36.09
60.19
55.75
71.87
45.64
33,20
35.22
62.04
50.00
CARE-E (OURS)
(9.43(()
60.54
72.42
54.29
33.60
35.50
66.06
52.62
256
75.00
Palu (SVD)
(3)
22.61
31.10
27.93
55.22
22.82
27.00
22.68
51.70
32.63
SVE-LIM v2
47.28 (1)
36.26
60.69
56.39
72.20
46.57
33.40
34.83
60.69
50.13
ASUDD
(1).86(L)
21.76
32.03
29.52
55.22
23.05
25.20
24.59
52.64
33.00
MHAZKLA
220.29(4)
25.94
40.95
39.27
61.37
25.54
26. 60
26.79
56.59
37.88
CARE-U (OURS)
9.64())
52.73
76.30
73.98
78.73
62.17
40.64)
41.53
72.61
62.33
512
50.00
CARE-E (OURS)
(9.50
42.41
64.69
68.96
75.46
63.98
37. (00
38.47
71.59
57.82
Palu ISVD)
45.40(L)
28.16
45.
43.37
64.15
24.98
30.80
53.43
39.33
3VD V2
9.63
52.39
76.68
73.57
78.45
62.31
40.40
41.63
72.22
62.21
ASVD
12)02(4)
46.33
69.11
70.75
76.17
41.80
36.60
33.97
66.85
55.20
-
6QA (Original)
10.04())
55.89
83.12
52.65
76.01
73.37
32.04
41.24
68.11
60.30
MHAZKLA
21850.16(.)
25.68
25.46
26.03
51.96
22.92
27.00
22.01
50.20
31.41
CARE-U (OURS)
905.74(L)
29.00
27.61
55.22
23.03
26.60
23.06
50.59
32.32
64
93.75
CARE-E (OURS)
730.93(L)
23.29
30.77
28.33
55.22
22.95
24,80
22.78
51.54
32.46
Pa u (SVD)
(L)
25.34
26.77
25.73
50.44
24.31
28.40
22. 11
50.91
31.75
SVD LIM V2
(9).94(()
23.12
28.58
27.66
$5.11
23.02
27. 27.20
23.44
51.54
32.46
ASVD
6683.95 (L)
27.39
25.00
25.71
50.38
24.08
26.60
22.11
50.43
31.46
MHAŽKLA
526X3.47 (U)
23.81
26.56
26.74
$2.18
24.74
28,20
22.68
49.72
31.83
CARE-U (OURS)
11)
27.47
39.69
37.52
60.50
27.03
.6
26.99
53,59
37.55
CARE-E (OURS)
102.38 ())
63.76
30.52
28.71
54.54
40.57
128
87,50
Pulu (SVD)
22048.79 (L)
26.02
26.18
26.29
51.09
24.49
25.40
21.05 15
49.72
31.28
7

---

Published as a conference paper at ICLR 2026
PPL (
ACC
Rank Save
Methods
ARC(L)
ARE
HellaSwag
PIQA
MMLU
OBQA
RA
WG
AVG
SVD-LIM V/2
(1).76
27.05
40.03
37.02
61.04
26.69
26.80
25. 74
53.28
37.21
ASVD
1682.84 (L)
22.95
30.01
29.29
52.34
23.42
27.00
23.54
50.12
32.33
MIAZKLA
44509.79
22.18
28.49
28.92
52.45
23.00
24.60
24.59
51.30
31.94
CARE-U (OURS)
22.08 (1)
46.42
68,90
59.16
71.55
54.76
36.40
35.02
62.43
54.33
CARE-E (OURS)
28.84 (L)
41.30
59.22
56.53
69.37
53.50
35,20
32.82
61.88
51.23
256
75.00
Pa 1SVM)
2561.97
29.92
$2.88
23.64
51.70
SVD-LIM V2
22.88
44.54
67.17
57.77
70.78
52.81
35,60
34.83
61.48
53.12
ASVD
63.15
32.76
46.84
47.45
63.93
26.38
30.80
30.05
52.01
41.28
100.99 (L)
37.97
CARE-U (OURS)
12.03(J)
54,95
77.23
69.24
76.22
67.46
40.00
39.43
68.43
61.62
CARE-E (OURS)
15.91 (1)
49.23
70.38
64,13
72.80
64.16
36,20
36.56
64.56
57.31
512
50,00
Pa u 159n)
33.97 (U)
35.58
47.64
50.44
65.18
27.85
32,80
30.24
$2.64
42.80
SVD LIM V2
(1)
77.44
68.53
75.68
39.80
67.25
61.14
(1.49
47.61
66.54
67.49
73.01
56.56
35.60
35.22
62.75
$5.60
60
FEI
matalllmmal Satto-Bmmaa-amm---a 1a-.aaamm-------mmeaa All--netractt
mate Btm.a//mmmmmm......mmmmmm wtw.aaammmmallllaaaaatt.eeeeeeee Jiib-lastrcr
Livennee froans Mivatte
Lemvebe
Coalt cred
CAl ros
CAR
Cald mas
Lagur
Lager
Ldger
Liger
Lager
Lagur
Lajur
Layer
Figure 3: Covariance rank profiles across calibration corpora (Alpaca, Wiki'eex22, PTB, C4) for Llama-3. Lama--.11-BB-Insee I-8B-Instruct at target ranks 64, 128, 256, and 512. Across all target budgets, both WK and Wy show depth-dependent increase-small early layers, rising through mid layers-with stronger late-layer growth for Wy. The consistency across corpora suggests model-intrinsic trend.
At very low rank, one-shot MLA remains challenging for all methods, but regarding perplexity, CARE already improves substantially over the naive Palu(SVD) or MHA2MLA initialization. As the rank budget increases to 256/512, the advantage covariance-award initialization becomes much clearer, especially on Llama-3.1-2 trade-off. We provide additional one-shot long-context retrieval results and qualitative examples in App. T.2
Qwen3-4B where CARE-E gives the strongest overall accuracy-PPL
and and App. respectively.
8

---

Published as a conference paper at ICLR 2026
4.2
ABLATION STUDIES
4.2.1 ENERGY DISTRIBUTION VS. COVARIANCE
Using the covariance-aware energy in Sec. 3 we obtain highly consistent rank profiles across C4 (Raffel at. 2020a), Alpaca (Taori et al. (023), WikiText2 (Merity et 2201)) and PTB (Marcus) ct al. 1993). As shown in Fig. 3 rank is front-loaded: across target ranks 64, 128, 256, and 512, both W K and W receive much larger ranks in the first layers and then decrease steadily with depth, approaching the minimum rank in later layers. This indicates that early layers are more accuracy- critical, while deeper layers can be compressed more aggressively. The same trend also appears on other models (From 1.5B 70B) (App. [.4), suggesting that the profile is largely model-intrinsic rather than tied to a particular calibration corpus.
4.2.2 ACCURACY IMPACT OF COVARIANCE
Using -.ama-3.1-88 Instruut we vary (i) calibration samples, (ii) sequence length, and (iii) calibration corpus. Fig. 4 shows that CARE is already strong with small calibration budgets: performance saturates beyond 512 samples, and longer sequences tend to overfit limited calibration data. We therefore use 256 samples with a short sequence length (32) the default trade-off. Across corpora (ALPACA, C4, WIKITEXT2, PTB, ARC, ARE, MMLU), one-shot accuracy changes are modest on average (Tab. L. domain-aligned corpora give mostly local gains. We default to ALPACA for broad coverage and stable generalization.
Additional ablations are deferred the appendix for read- ability. App. [ sweeps the shrinkage coefficient, showing that CARE is not sensitive to the exact regularization mag- nitude and supporting our default a = 0.01. App. L.3 studies cross-domain calibration under distribution shift, App.1 L.7 examines task-weighted covariance mixing, and
Accuracy Ablation on Alguci Llama-3.1-88 -.1-----n-t-
Nasther Samplos
Suqurnce Longth
Cavrriaer
Awerap Artrrey
40
App. 1.8 compares the default vC formulation against using C directly under matched settings. Although the derivation is based on VC, the appendix shows that /C is more consistent than using C' directly, especially beyond the lowest ranks.
30
P00
1000 1566
2068
Figure 4: One-shot accuracy versus cal- ibration samples and sequence length across eight benchmarks. Sequence length is fixed at 256 unless otherwise noted; the red curve denotes fixed 256 samples with varying sequence length. OOM occurs beyond length 512 dur- ing covariance computation.
4.3 RECOVERY
100% MLA WITH SMALL SET BUDGETS
Setup (healing baselines). We compare CARE against Palu(SVD) and TransMLA under matched healing budgets at MLA rank 512. Our hybrid TransMLA + CARE (E) Init keeps the same 100% MLA restoration and healing pipeline as TransMLA, but when TransMLA performs the KV low-rank mapping, we replace its original initial-
ization with the covariance-aware VCW SVD used by CARE(E). All subsequent restoration and healing stages are unchanged. CARE-based 100% MLA Restore use alpaca- -256-255 calibration (256 samples, sequence length 256), TransMLA uses length 256), and Palu(SVD) does not require separate calibration corpus. We report the one-shot initialization (OB) and healed checkpoints after 1B and 3B tokens on the same LM Harness suite.
(256 samples, sequence
Table 2: Healed Llama3.1 1-8B- Instructt comparison at MLA rank 512 across token budgets.
Roak:
Meriab
Callwooe
Takes
ABC
ART It
1924-11
S0ILU/11
(104
RA/11)
MG2001
4512
(eciginas)
18.15
60.33
1945
-8.0
01.30
26.24
Tresetth + case Teis your 1231 MSA Bestine)
1.
$4.00
14:0
64.6
15.05
01.08
77.0
07.41
60.00
11.81
Trewetth x (300.20) Teils yomm 1535 MSA Sentore)
16
11.44
14
62.84
3830
71.8
3150
06.45
12.00
11.7
18
35.40
62
6555
1234
36.5
2180
11.54
61.03
41.0
18
06
55.35
105
18.0
3830
12300
08.65
03.0
50 1002-220--22
201
70
15.7
13.4
5646
0331
7623
1530
850
13.7
119

---

Published as a conference paper at ICLR 2026
We sweep small-scale SFT budgets to quantify post-conversion recovery; results shown in Tab. CARE(E) already starts from the strongest one-shot initialization, and this advantage largely remains after healing. At IB and 3B tokens, TransMLA + CARE (E) Init reaches 62.59 and 63.27 average accuracy, outperforming TransMLAA (61 (61.36661..6 (6/61.86) and Palu(SVD) (47.58/56.30) at matched budgets. While both Palu(SVD) and TransMLA improve substantially with more tokens, CARE attains the best overall recovery with less reliance on long healing runs; optimization objectives. datasets, and hyperparameters in App. H A system-level KV-cache efficiency analysis in App. 1.9
RELATED WORK
Conversion from Traditional Attention MLA. Standard multi-head attention (MHA) underpins modern LLMs but induces KV cache that scales linearly with sequence length and head width, creating a memory and bandwidth bottleneck at inference time (Vaswani et al. 2017: Kwon et al.. 2023). Grouped-Query Attention (GQA) reduces KV heads by sharing keys/values across query groups, lowering KV memory while sacrificing expressiveness (Ainslie et al. 2023 Shazeer 2019). Multi-Head Latent Attention (MLA) addresses KV memory by caching low-dimensional latents with lightweight up/down projections (DeepSeek-A) AI Team 2024). Beyond MLA that trains from scratch, several post-hoc pathways demonstrate conversion feasibility: TransMLA gives a theory- based reduction from GQA to MLA at corresponding KV budget (Meng et al., 2025), MHA2MLA focuses practical alignment via partial RoPE and joint-SVD initialization (i al., (025), ) and X-EcoMLA explores distillation-basedd upcycling of pretrained attention into MLA for extreme KV compression (Li al., 2025b). Also, following MambalnI.ama (Wang et al.] 2024a), Zebra-Llama composes efficient hybrids to improve inference efficiency and can be paired with ML.A-styee KV reductions in deployed systems (Yang et al. 2025).
SVD inspirations. Naive SVD truncation minimizes W - WyllF and ranks by raw singular values, which need not correlate with downstream loss (Eckart Young 1936; LeCun et al.) 1990, Hassibi & Stork 1993 Dong et al.] 2019 Frantar et al. 2022 Krzanowski, (2000. Recent works (weighted by Fisher information) (Hua et al. 2022). SVD-LLM (truncation-awwe whitening sequential low-rank updates) and its V2 variant (improved trunca- tion/rank selection) (Wang et al. 2024b 20256). SoCo learns a diagonal reweighting to optimize the singular spectrum directly for compression rather than trusting singular-value magnitudes (Li et al. 2025a), while Dobi-SVD introduces a differentiable SVD that targets activation-side truncation and ef- ficient reconstruction (Wang et al., 2025a). Together with architecture-aware conversions (Meng et al. 2025 Ji et al. 2025), these SVD-oriented techniques motivate data/curvature-aveee aware orientations and non-umiform rank allocation as core tools for preserving pretrained knowledge. Relatedly, CorDA leverages context/activation statistics to guide decomposition adaptation for parameter-effiiien fine- tuning, further supporting the value of covariance-awes orientations beyond weight-only SVD (Yang et al. 2024b).
refine this in LLMs: F FWSVD
SVD for cache compression.
Orthogonally, some methods compress the cache itself. Palu com- presses KV-cache with low-rank projection, reconstructing full K, V on the fly with efficient rank search and quantization interoperability (Chang et al., 2024). ReALLM combines low-rank compo- nents with vector-quantized latents under unified recipe (Leconte et al.) 2024). These methods can be stacked with MLA-style conversions or used standalone to further lower KV memory.
For more related works, please refer to App. F
CONCLUSIONS
We proposed CARE, a Covariance-Aware, Rank-Enhanced procedure for migrating traditional attention to MLA under a fixed KV-parity. CARE replaces naïve weight-only SVD with a covariance- weighted factorization and assigns per-layer ranks via an energy-driven, water-filling schedule. Em- pirically, CARE preserves MLA's efficiency (identical KV footprint, comparable throughput) while matching or improving perplexitylaccuracy over uniform SVD baselines with less post-conversion tun- ing. It also yields greater robustness to aggressive rank reduction, suggesting stronger initializations for brief SFT.
10

---

Published as a conference paper at ICLR 2026
7 ETHICS STATEMENT
We have read and will comply with the ICLR Code of Ethics. Our study involves no human subjects, personally identifiable information, user-generated content. All datasets are standard, publicly available benchmarks used under their respective licenses; we do not collect or infer demographic attributes. The work focuses model architecturcooptimizzioo and does not introduce capabilities intended for surveillance, profiling, or other harmful use. We identify no foreseeable risks related to privacy, security, fairness, or legal/regulatory compliance, and IRB/ethics approval was required. To support transparency, we will release code, configuration files, and clear instructions to reproduce all results. All findings are reported honestly without fabrication or inappropriate manipulation. The authors declare no confficts of interest and no external sponsorship that could bias the work.
8 REPRODUCIBILITY STATEMENT
We provide an open-source repository /tttps://gitinn com wtw.:///..oo//uuuurr-------- The repo includes exact configuration files for all experiments in Tab. 12 and Fig. 3 scripts to download and verify datasets, deterministic preprocessing, fixed random seeds, and environment specifications (Conda with pinned versions), Algorithmic details appear in Sec. 3 dataset descriptions and evaluation setup are given in Sec. hyperparameters, hardware, optimizer and scheduler configurations are listed in App. H. Detailed zero-shot tables for all models and calibration corpora are in App. L.I additional rank profiles (1.5B-70B, including MoE) in App. L.4. ablations on shrinkage (App- [.6). distribution shift (App. 1.5), covariance mixing (App. 1.7 and VC vs. C' (App. 1.8):
CARE).
long-context NiH evaluation in App. L.2 and KV-cache efficiency analysis in App. L9 Running scripts in the repository recreates reported metrics and regenerates all plots and logs. The repository released under the Apache License 2.0.
REFERENCES
Joshua Ainslie, James Lee- Lhorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical commonsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Chi-Chih Chang, Wei-Cheng Lin, Chien- Yu Lin, Chong-Yan Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S. Abdelfattah, and Kai-Chiang Wu. Palu: Compressing kv-cache with low-rank projection. arXiv preprint arXiv:2407.21118, 2024. URL https
//arxiv.org//ab//200 22111
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595 2023. URL
https://arxxvv org/ 2oos//3666 15595
Aakanksha Chowdhery et al. Palm: Scaling language modeling with pathways. arXiv preprint
arXiv:2204.003311 2022. /tll/httss:///xxvv.orr wwwpg//////g//o//ooe 022///2222202111111
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint
2018.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv 08691
preprint (rXXiv:307.00869) 2023. URL. htt.htttsss//xxii..rrrrg
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with 10-awaremess. Advances Neural Information Processing Systems (Neur 2022. URL org.a//eee....aaa..........oe..r
rence html

---

Published as a conference paper at ICLR 2026
DeepSeek-AI Team. Deepseek-v2: Multi-head latent attention for economical inference. Technical Report, 2024,
com/ deepseek-al ai
Misha Denil, Babak Shakibi, Laurent Dinh, Marc' Aurelio Ranzato, and Nando de Freitas. Predicting parameters deep learning. In Advances in Neural Information Processing Systems 26 (NeurlPS 2013), 2013. URL https: httpppaaprs..//// nips eters eters-in-destt in-ee--n-eee----- learning
Emily L. Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear
structure within convolutional networks for efficient evaluation. In Advances in Neural Information Processing Systems 27 (NewrIPS 2014), 2014. URL httpt::/ttpseeeeeeeeee
nips. cc/paper/5 www.rrurruuuuuuuueeeueee wwwwwwhinncoome convo wutionatuiinaaa networks-f 544-exploit hhw.o:tttttt.....aaeeeeettttrtre or
-
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang. Yuanyuan Xu. Ning Shang, Jiahang Xu, Fan Yang. and Mao Yang. Longrope: Extending IIm context window beyond million tokens. arXiv preprint
203.::24022.3753,, 2024.
Zhen Dong, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), 2019.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- Irika, 1(3):21 211228, 1936.
Xiang Fang, Wanlong Fang, Changshuo Wang, Xiaoye Qu, and Daizong Liu. Rethinking video- language model from the language input perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, 2026a.
Xiang Fang, Wanlong Fang, Changshuo Wang, Keke Tang, Daizong Liu, Siyi Wang, and Wei Ji. Towards unified vision-language models with incomplete multi-modal inputs. In Proceedings of the AAAI Conference on Artificial Intelligence, 2026b.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2022. ICLR 2023.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Alain Le Noac h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. (tLphttss:/ http::://eeooo.ooo www.odd...rg/eeeodds//eeee
Robin Geens and Marian Verhelst. Hardware-centric analysis of deepseek's multi-head latent attention. arXiv preprint (tt.xiv:5000005222, 2025. doi: 01.446550/r2X11225000225555 URL
hittps://axxias org/ abs /2006 02523
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12949 2025.
Babak Hassibi and David G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems (NeurIPS), 1993.
Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In JEEE international conference on neural networks, pp. 293-299. IEEF, 1993.
Pengeheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta:
bert with disentangled attention. In International Conference on Learning Representations (ICLR), 2021. URL htt.l::tss../. //1 r cc/vi rtua /2021 poster/2522222
Dan Hendrycks, Collin Burs, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.033300 03300, 2020.
12

---

Published as a conference paper at ICLR 2026
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NeurIPS Deep Learning and Representation Learning Workshop, 2015.
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length LLM inference with KV cache quantization In Advances in Neural Information Processing Systems (NeurIPS). 2024, URL Btt.lgggg.//ppooedd..ggg....rers patppppppeppeeeeeeelal wtt.apperrreeeeeppppeeeeee
sh/ DBDDBBDcceBBSaaa0000dee2222222e2 -wn.ttraact-offeeeeeeeeeee html
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference Learning Representations (ICLR), 2022.
Ting Hua, Yen-Chang Hsu, Felicity Wang, Qian Lou, Yilin Shen, and Hongxia Jin. Numerical optimizations for weighted low-rank estimation on language model. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1404-1416. Association for Computational Linguistics, 2022. RRRIitssss//claatholooyyy
2. semnlp-main, 91. pdf
Tao Ji, Bin Guo, Yuanbin Wu, Qipeng Guo, Lixing Shen, Zhan Chen, Xipeng Qiu, Qi Zhang, and Tao Gui. Towards economical inference: Enabling multi-head latent attention in transformer-based 1lms. Technical Report, 2025. URL (tttst:: tit.....uuu...... www.ttt............oa..t.tt.t..t JT-UshioU /
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William Sayed. Mistral 7b. arXiv preprint arXiv:23 2023. URL (htpt:::: //arriv...iiv.//aa org abs 231..06825
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy, The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems (NeurIPS), 2023. URL
s. Paper-Conferencee pdf
W. J. Krzanowski. Principles of Multivariate Analysis: User's Perspective. Oxford University Press. 2000.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. arXiv preprint arXiv:2309.06180 2023. URL httpps//axxxx
brg/abs// 2309. 06180
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:/704 04683, 2017.
Louis Leconte, Lisa Bedin, Van Minh Nguyen, and Eric Moulines. Reallm: A general framework for Ilm compression and fine-tuning. arXiv preprint arXiv:2405. 2024. URL https:
7/arxiv. wtwpr///aaaaaaa.e 13155
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems (NeurIPS), 1990.
Dengjie Li, Tiancheng Shen, Yao Zhou, Baisong Yang, Zhongying Liu, Masheng Yang, Bernard Ghanem, Yibo Yang, Yujie Zhong, and Ming-Hsuan Hsuan Yang. Optimizing singular spectrum for large language model compression. CoRR, abs/2502.15092 2025a. URL https: //trxaaavvvv////////a/ org/ ab
1/2502.15092
Guihong Li, Mehdi Rezagholizadeh, Mingyu Yang, Vikram Appia, and Emad Barsoum. X-ecomla: Upcycling pre-trained attention into mla for efficient and extreme kv compression. arXiv preprint arXiv:2503. / 132,, 2025b.
13

---

Published as a conference paper at ICLR 2026
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang. Chuang Gan, and Song Han. Awq: Activation-awart weight quamtization for on-device LLM compression and acceleration. In MLSys, 2024. URL httthtts:///sst httopp/pppoooeeiiiaggg s.ml
Bcclef21-Pae bcclle222-pppe--Cnffeeenncc
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprini 1rXiv:2412.19437, 2024a.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmotric 2bit quantization for KV cache. arXiv preprint 2xXiv2402.022500. 2024b. URL http::ttt////t. http:p::/aaxxivoooo abs/ /200 02750
Qianli Ma. Yaowei Zheng, Zhelun Shi, Zhongkai Zhao, Bin Jia, Ziyue Huang, Zhiqi Lin, Youjie Li, Jiacheng Yang, Yanghua Peng, et al. Veomni: Scaling any modality model training with model-centric distributed recipe zoo. arXiv preprint arXiv:2508.02317 2025.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building large annotated corpus English: The Penn Treebank. Computational Linguistics, 19())313-330, 1993. URL
https: wwwp::///wwwww.wwb....b... org/ httpi///anntooogggooooooooo 2004
Fanxu Meng, Zengwei Yao, and Muhan Zhang. Transmla: Multi-head latent attention is all you need. Technical Report, 2025. URL
hwt.hhh...oomooeeoooeeeeeeeeoeee wwwom//meeg///ocoommmmoeooo Exw.ee//nnnTTaaaaa. TransMLA
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.0278) 0278999999999 2018.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yamn Efficient context window extension of large language models. arXiv preprint arXiv:2309000071, 2023, URL https
7/arxiv. org/ abs / /20../209... 00071
Ofrr Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108, 12409, 2021. URL https
/aaxiv.org/abs/////.
12409
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020a.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020b. URL
https://5mll wwwp:r/pppppp/poe//o
httpppprss/2/22000 vv1/20 074. html
Tara N. Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low- rank matrix factorization for deep neural network training with high high-dimensional output targets. In Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), pp. 6655-6659, Vancouver, Canada, 2013. IEBE. doi: 10. (09/CAASPP.000 3.6638949.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106. 2021.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position represettnprssnnnnaiionnnn. In Proceedings of NAACL-HLT 2018, Volume 2 (Short Papers), pp. 464 468, New Orleans, Louisiana, 2018. Association for Computational Linguistics. doi: 10.18653/v 1/N18-2074, URL https://ach antho ogy.
14

---

Published as a conference paper at ICLR 2026
Noam Shazeer. Fast transformer decoding: One write-head is all you need. In NeurIPS Workshop Efficient Natural Language and Speech Processing, 2019. Multi--uery Attention (MQA).
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-Ima Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:190.0080:: 2019. BRLhttps::/ arxii. orgg abs/ 1909. 08053
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Moudgil, et al. Roformer: Enhanced transformer with
rotary position embedding. arXiv preprint FXiv:2104.09864, 2021a.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with
rotary position embedding. In Findings of the Association for Computational Linguistics (ACL Findings), 2021b.
Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A ettatt--xtrapoolttall transformer. arXiv preprint 41X15:2212.10554, 2022. URL Pattps:/aarae http//aaxvvvr.g.go/// www.orggaaa... /211 2212. 10554
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An -estruction-following Ilama model. 2023.
https://githe: wtw.hh..ooooooooooommmmmmo..o..m coom/ttte tatt saiissssslbbbeee wwwllaaaaaaaeaee hetbb/staffrdd_apppg
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, et al. Llama 2: Open foundation and fine-tur tuned chat models. arXiv preprint arXiv:2307.09228,, 2023.
URL https:://axxiy org/ abs/ 220../337...0 09288
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quipit: Even better LLM quantization with hadamard incoherence and lattice codebooks. In Proceedings of the 41st International Conference on Machine Learning (ICML). PMLR, 2024. URL Btthtts:
//proceedings ml wtwpp//gg..........a..e.eee.e.ee www.ss/v2332 2tt55//tseg22aaa html
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In Proceedings of the 17th Conference of the European Chapter of the ACL (EACL), 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, łukasz. Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Junxiong Wang, Daniele Paliotta, Avner May, Alexander Rush, and Tri Dao. The mamba in the llama: Distilling and accelerating hybrid models. Advances in Neural Information Processing Systems, 37:62432-62457, 2024a.
Qinsi Wang, Jinghan Ke, Masayoshi Tomizuka, Yiran Chen, Kurt Keutzer, and Chenfeng Xu. Dobi-svd: Differentiable svd for IIm compression and some new perspectives. arXiv preprins vrXiv:2502.02223, 2025a.
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04766 2020. URL /tl/atttp///aaxv.oorr
04768
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-Ilm: Truncation-awant singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07377, 2024b.
Xin Wang, Samiul Alam, Zhongwei Wan, Hui Shen, and Mi Zhang. Svd-Ilm v2: Optimizing singular value truncation for large language model compression. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 4287-4294 Albuquerque, New Mexico, 2025b. Association for Computational Linguistics. URL https: aaclanthology .org/2025. naacl long. 217.pdf
15

---

Published as a conference paper at ICLR 2026
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning (ICML). PMLR, 2023. URL htttps:
v/proceedi ngs. ml r. press v202 xiao23c. html
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström- based algorithm for approximating self attention. arXiv preprint arXiv:2102.03900,, 2021. URL https: http://aax...ooa///////0/// 03902
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, et al. Qwen2 technical report. arXiv preprint urXiv:2407.10671, 2024a. URL URLl/ttpps://ziivoorr///aa
0671
Mingyu Yang, Mehdi Rezagholizadeh, Guihong Li, Vikram Appia, and Emad Barsoum. Zebra- llama: Towards extremely efficient hybrid models. arXiv preprint 17X2v 2025. URL
https://arxiv. org/ abs 17272
Yibo Yang, Xiaojie Li, Zhongzhu Zhou, Shuaiwen Leon Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem. Corda: Context-oriented decomposition adaptation of large language models for task-aware parameter-efficien fine-tuning. In Advances in Neural Information Processing Systems 37 (NeurIPS), 2024b.
Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation- aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv::905.07880 2019.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efffcient fine- tuning. In International Conference Learming Representations (ICLR), 2023.
16

---

Published as a conference paper at ICLR 2026
CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head
Latent Attention LLMs
Supplementary Material
A LARGE LANGUAGE MODELS USAGE
We used large language model - ChatGPT (GPT-5 thinking) solely for grammar and spelling edits to author-written text. We used Claude Code to assist code writing. The tool did not generate scientific content, design experiments, analyze data, or select citations, and therefore did not contribute at the level of a contributing author. All edits were reviewed and approved by the authors, who take full responsibility for the final manuscript.
B RECALL: KV-PARITY MAPPING
Grouped-Query Attention (GQA) reduces KV-cache memory by letting multiple heads share the same key-value projection. To convert GQA to Multi-Head Latent Attention (MLA) without increasing the KV budget, we enforce KV parity. Consider GQA layer with n, heads of size da with total multi-head hidden size D = du, we split nıg heads split into yh groups, where each group contains nh heads. The layer uses W
Q E (x(ndd))) and grouped W(O w Dx(addi))
Ih
We conceptually replicate each group's (!( and W.O)) across its 2h members to form the full-size Will WiOh E Dx(ndda)) (no need to materialize in code). This demonstrates that the GQA method K
can be reduced MLA by removing the repeated head blocks (Meng al. 2025). Therefore, we set MLA's latent rank to match GQA's per-token KV width:
Ih dA
(9)
C PROOF OF MAXIMUM ENERGY SVD TRUNCATION
We have the following proposition: Let A E Rmxn have singular value decomposition (SVD) UEVT, where Σ diag(o1,.... :p): p min(m,n) n), and 01 2 ... 2 op For
1 S r < let En = diag(o1...... Or. .,.....................
........)) 0) and A, == UE, V Then
A - A
Σ of and |A ArllF rank(X) Sr
min | - X||F;
with the unique minimizers (when o, > 0r+1) given by X = Ar [Eckart & Young 1936).
Proof. The Frobenius norm is unitarily invariant, so for any X with rank(X) S
| X| || U (A X)V || || YlEE, where Y = U XV and rank(Y) $
Expand the square via the Frobenius inner product (M, = trace(M NN):
[ Y 1 = |ز + | - -2(2, Y).
Let (Y) 2 ... 2 8p (Y) 2 0 the singular values of Y (so 8, (Y) = for r). By von
Neumann's 's trace inequality,
Y2 & sx) = Σ (Y))
1=1
and || " 2Y) El= 2Y)2
Therefore:
p
>
Σ o
8: (Y)2 Σ (Vs Σ (a; s; (Y))
Σ
o
> Σ
o
1=1
1=1
1=+41
i=
17

---

Published as a conference paper at ICLR 2026
This lower bound is attained by taking Y E,, i.e X VT
Ar. for which:
||- A. IF
2y = o
1=++
Thus A, is a best rank-r approximation in Frobenius norm, and the minimum value is the squared 22-tail of the singular values. Uniqueness follows when d, > 0r+1 since then any other minimizer must share the top singular subspaces with A.
Note that the idea and proof of the theorem above follows essentially the same idea as Eckart- Eckart-Young- Mirsky in Eckart & Young (T936).
Relation to Sec. 3.2 The theorem justifies the use of squared singular values in Sec. 3.2 for single layer/matrix, the marginal reduction in Frobenius residual from adding the next rank is governed by the next squared singular value. Sec. 3.2 further divides this local gain by the remaining residual energy of the same layer, which normalizes layers with different spectral scales and makes their priorities comparable under a shared global budget. This normalization changes the cross-layer comparison, but preserves the theorem's 's local interpretation that larger squared singular values remove more residual energy.
D DERIVATION OF SVD TARGET FUNCTION
Let LWW W- WW and
Eb
X Xs. We claim that:
N
1
|X,Ai] ||VCAW||.
Proof.
By the Frobenius-tre trace identity ||A| Tr(ATA) A),
N
N
N
Σ |K
Σ
(X(XAA))TXXAVV N Σ TAAT NA, X X,
N
1=1
Using linearity of trace and the definition of C,
|XAWI = MAWTCAW/
1=1
We can show that C' > since for any vector v, we have:
N
N
v
TC = N
X
XBU
= N
Σ
b=1
b=1
Then we write C' 11/2 to get: T(WTCCWW = TRAWT C C (1 (C -
E DEFINITION OF COVARIANCE
Let E P be the activation (feature) vector of single token. The population covariance is:
Cov[æ] E E(x #=(x 11) TI
/ E(r].
Given samples ( the empirical mean and covariance are:
N
N Σ ( M) - A) 1=1
Cov[ar] = N
Tí. 1=1
rows of X E N D stack the samples and X, = 1 T, then Cov[z) = - X X.. (Unbiased version uses 1/(N -1] instead of 1 1/N.)
18

---

Published as a conference paper at ICLR 2026
F RELATED WORK
Serving throughput is often bounded by how the KV cache is organized and moved across memory. PagedAttention (vLLM) treats KV as pageable blocks to avoid internal/ex- ternal fragmentation and enable sharing across sequences, improving utilization under dynamic batching (Koon al., 2023). Orthogonally, FlashAttention reduces HBM traffic with an IO-aware tiling of exact attention, and FlashArtention-2 2 further improves parallelism and work partitioning for higher FLOPs utilization (Dao al.] 2022] Dao (023). These system/kemeel directions are complementary to architectural changes (e.g., GQA/MLA) and to post-hoe reparameterizations, since better KV layout and IO scheduling directly translate into larger effective batch sizes at a fixed memory budget.
KV Management
Quantization Quantization provides an orthogonal compression path to low-rank methods and can be combined with MLA/GQA conversions. For weigghts/activationn, SmoothQuant migrates activation outliers into weights to enable practical W8A8 PTQ on large models (Xiao et al. (023). AWQ protects small set of salient channels via activation- aware scaling, delivering strong 4-bit weight-only PTQ with hardware-friendl. kernels (Lin et al.] 2024). QulP# pushes extreme regimes (<4-bit) using randomized Hadamard incoherence and lattice codebooks, with state-of-the-an results at low bit-rates (Tseng et al. (2024). For the KV cache, KVQuant NNcurPPS244) introduces pre-RoPE key quantization, sensitivity-awee aware non- con-uniform datatypes, and per-vector dense/sparse schemes to sustain long-com context inference (Hooper et al. (024), while K/V/ shows tuning-free free 2-bit asymmetric KV quantization with favorable throughput/memory trade-offs (Liu et al. 2024b). Together, these methods form a toolbox that is largely complementary to low-rank latent caching.
RoPE and Positional Encodings Positional design strongly affects length generalization and con- version stability. RoPE's complex-valued rotary formulation remains the default in many LLMs (Su al. 2021b). Alternatives include relative positions (Shaw et al. 2018), T5's learned relative bias and DeBERTa's disentangled content/position attention (Raffel et al. 2020b He et al. [22I)) and ALiBi's linear distance bias for train-short/test-lon extrapolation (Press et al. (021). Within the RoPE family, window-extension strategies modify scaling spectra to stabilize extrapolation, such XPOS's multiplicative stabilization, Position Interpolation, YaRN, and very long-window LongRoPE (Sun) al. 2022 Chen et al., 2023 Peng al.) 2023 Ding et al. 2024). Systematic comparisons further show that the chosen positional scheme materially impacts length generalization (Kazemnejad et al. 2023), motivating careful treatment (e.g., partial-RoPP or mixed strategies) during architectural
realignments.
G DISCUSSION
Across two GQA backbones and diverse tasks, CARE- Cora--oorrinnce-Aaare and Rank-Enhanced eecoppoitionneenbbeee MLA migration under KV-parity with accuracy and long-context robustness on par with (or better than) stronger baselines. CARE preserves the throughput/memmry advantages of MIA while mitigating the activation drift observed with weight-only SVD.
Rank allocation matters. Uniform or purely energy-based rank policies overlook the weighted spectral concentration that emerges after covariance/curvature preconditioning. CARE's Adjusted- Rank uses water-filling allocation over weighted singular spectra, honoring a global KV constraint while allocating capacity to the layers and directions that matter most.
Complexity The dominant conversion costs per-layer covariance estimation O(ND²) (with small N) and truncated SVD on VCW e IR Dx(n,di)) at O(D (nA,))) r) using randomized SVD. Layers can be processed sequentially; for each layer, C. = N E X Xx, can all be kept on CPU. At inference, MILA incurs light extra matvecs by We while reducing K KV-cache width from dh (MHA) or gh,da (GQA) down to r = ghdy, (MLA), CARE is orthogonal to quantization and sparsification and compatible with MLA kernels [DeepSeek-Al Team 2024).
Compatibility with MLA migration. CARE complements recent MLA conversions (Ji et al.) 2025 Meng al. 2025) and plays well with partial-RoPR. (Su al. 2021a): removing rota-
19

---

Published as a conference paper at ICLR 2026
tions least-contributivv subspaces further stabilizes long-contexs behavior when combined with activation/curvaturee aware objectives.
Limitations. (i) Statistics freshness: CARE requires small calibration passes; pronounced domain shift may need refreshed covariance/currature. (ii) Diagonal curvature: practicality favors diagonal proxies; structured approximations (e.g.. Kronecker-factored) may yield further gains. (iii) Extreme compression: very low ranks, information bottlenecks dominate and further SFT can be necessary. (iv) Orthogonallty to quantizationneeiccion CARE does not yet co-optimize KV quantization and cache eviction policies. (v) Kernel support: dedicated MLA kernel currently supports per-layer dynamic ranks, making end-to-end lateney benchmarking impractical; we therefore report only theoretical KV-cache savings rather than wall-clock speedups.
Broader impact and future work. CARE suggests general recipe for post-training architectural migrations: align the objective to where errors manifest and distribute capacity by curvature-weighte signal. Promising directions include applying covariance-aware low-rank decomposition to vision- language and multimodal architectures (Fang et al. (2026a]b) where attention modules face analogous KV-cache bottlenecks, as well as data-free calibration, structured curvature, and dynamic rank schedules that adapt latent capacity with context length while maintaining KV-parity.
Apart from that, Covariance Dovariance-weighted SVD initialization minimizes the activation loss each layer, but our true goal is to preserve the output of the model, which is next-token predictions. We may therefore cast low-rank compression as directly minimizing the sequence loss produced by the compressed (student) model under fixed KV budget.
H HYPER-PARAMETER SELECTION
All experiments were conducted equipped with NVIDIA HI00 80 GB GPUs paired with dual Intel Xeon Platinum 8462Y+ processors (2 x 32-core sockets, 64 cores total) and approximately 2 TB of RAM.
All hy hyper-ppee paremmerasssrss are shown below:
Model Configuration:
Base model: Meta; Precision: float16, Sequence length: 8-2048
tokens, Covariance samples: 8-2048.
MLA Rank Settings: Default rank: 256, Min rank: 64, Max rank: 1024, Uniform allocation:
True/False, K/V projection ranks: 256 each.
CARE Parameters: Initialization method: CARE, Damping factor (peredamp): 0.01,
Cholesky decomposition: False, Activation order: False.
Evaluation Datasets: Multi-task benchmarks including WikiText (perplexity). ARC- Challenge/Fasy (reasoning), HellaSwag (commonsense), PIQA (physical reasoning). MMLU (knowledge), OpenBookQA, RACE (reading), WinoGrande (coreference).
Generation Settings: Max new tokens: 512-12800X Temperature: 0.6-0.7, Top-p sampling: 0.9, Sampling strategy: Nucleus sampling with temperature control.
System Configuration: GPU memory free threshold (minimal GPU resources to run experiments) 2048 MB, Parallel GPUs: 1-8 devices, Batch size: Dynamic adjustment, Random seed: [42, 17, 26, 103, 21, 59, 134, 8, 24, 99].
Covaraince Computation: Dataset: CA/RUWWEHiee/AADee//AAAAAAEEEEE instruction-follooingg Sample size: 8-2048, Sequence processing: 8-2048 token windows,
Random Seed and Learning Rate All experimental results are average over 10 random seeds and we choose the best from 3 learning rates.
conducted using the Večmni framework (Ma et al. 2025) for fine-tuning CARE and TransMLA models.
Training Framework All experiments
Learning rate: We choose best learning rate x 10 with linear warmup over the first 0.001 % training steps.
20

---

Published as a conference paper at ICLR 2026
Optimizer and Schedulers: We sweep LR E {1x10-6, 1x10 5x10-5 1x10 5x 10 *). and select the best within the first 0.1B tokens. We use AdamW, weight_ decay= 1.0.
0.01, cosine decay with 1r wwwmup ratio= 0.005 and 1r
Batch size: Global effective batch size of 64 tokens per update step, accumulated across devices.
Precision: 6 mixed precision was enabled reduce memory footprint and improve throughput.
Max sequence length: Input sequences were truncated or padded to a length of 312-128000 tokens.
Training epochs: Each experiment was trained for the number of pre-set tokens.
1 SUPPLEMENTARY RESULTS
1.1 DETAILED ZERO-SHOT TABLES
These tables report the full zero-shot results for each model and calibration dataset combination, using the same metric layout Tab. [ In addition to the main-text variants (CARE-U and CARE-E, which
weighted SVD with uniform and energy-aware rank allocation, respectively), we include CARE-C-based-U and CARE-C-basede
which replace VC with C directly as the covariance weighting; see Sec. L.8 for a detailed comparison of the two formulations.
1.1.1
LLAMA 3.1 3.B INSTRUCT-ALPACE.
Table Detailed zero-shot comparison for Llama-3. 1amma-3.1--3 1-8B- 3B-Instruct with Alpaca calibration. Higher better for Accuracy (%) (ACC.) (f) and lower is better for Perplexity (PPL.) (L).
PPL
ACC (t)
Rank
Methods
ARC-E
ARC-C HellaSwag
PIQA
WG
MMLUU
OBQA
RA
AVG
284863.91
50.59
22.11
CARE-U
(.)
30.51
23.89
26.82
54.73
49.01
23.18
26.00
20.96
31.89
CARE-E
98).03(
31.31
23.63
27.49
54,62
50.36
22.98
27.40
21.15
32.37
64
CARE-C- MARE-C-based--
(03.84(L)
31.40
23.98
26.91
$5.66
50.04
22.91
27.00
21.15
32.38
CARE-C- SARE-C-based2-
837.25(1)
23.46
27.50
56.75
49.96
22.90
27.40
21.34
32.71
(SVD)
2260.60(L)
SVD-SIM V2
967.04
30.72
23.63
26.75
$4,90
48.93
23.22
26.40
20.86
31.93
ASVD
2525.33(L)
26.68
23.81
26.68
$2.18
50.99
22.97
27.80
20.86
31.50
MRA2MTA
(L)
24.12
25.00
26.58
$2.18
51.07
23.82
29.20
22.11
31.76
CARE-U
398.91 (1)
41.96
26.71
33.64
60.99
53.20
26.06
27.00
24.21
36.72
CARE-B
(1).74(1)
42.47
27.30
37.96
62.46
54.85
29.38
28.20
26.12
38.59
CARE-C-based-U
3)5.47(
42.80
27.05
33.64
61.59
53.51
26.79
27.40
24.11
37.11
128
CARE-C-based-E
328.76(1)
54.78
Palu (SVD)
3046.58(L)
27.02
23.89
26.62
50.38
49.25
23.14
24.80
22.49
30.95
SVn---TM V2
(8).23
42.38
27.56
34.06
61.04
53.99
26.17
27.80
24.2.
37.15
ASVD
(675.54(L)
27.99
25.00
26.61
$2,39
48.46
23.04
26.60
21.82
31.49
MIAZYLA
1633.65(L)
25.63
27.47
26.50
$2.39
50.83
23.10
28.20
22.49
32.08
CARE-U
(3)
60.19
36.09
53.75
71.87
62.04
45.64
33.20
35.22
50.00
CARE-E
(9.43(
60.06
38.48
60.54
72.42
66.06
54.29
33.60
35.50
$2.62
CARS-C-based-U
49.35(2)
44.06
33.49
256
-
49.40(2)
61.74
37.54
58.28
72.42
63.22
50.43
33.00
34.64
51.41
Palu (SVD)
537.57
31.10
22.61
27.93
55.22
51.70
22.82
27.00
22.68
32.63
SVD 3IM V2
(7)28(J)
60.69
36.26
56,39
72.20
60.69
46.57
33.40
34.83
50.13
ASVD
312.86(4)
32.03
21.76
29.52
55,22
52.64
23.05
25.20
24.59
33.00
NHA2XLA
21)
40.95
25.94
39.27
61.37
$6.59
25.54
26.60
26.79
37.88
CARE-U
9.64(L)
76.30
52.73
73.98
78.73
72.61
62.17
40.60
41.53
62.33
(9,50(]
71.59
63.98
CARE-C-based-U
10.06(J)
76.68
51.45
72.20
71.35
41.20
41.72
61.69
CARE-C-based-
16.46(J)
68.18
44.03
69.88
76.06
71.90
63.28
38.60
40.10
$9.00
Palu (SVD)
45.40(
45.96
28.16
43.37
64.15
53.43
24.9%
30.80
23.83
39.33
SVD-SLM V2
9.63(L)
76.68
52.39
73.57
78.45
72.22
62.31
40.40
41.63
62.21
12.02(.)
66.85
21

---

Published as a conference paper at ICLR 2026
1.1.2 1.AMA-3.1880 --N.S---CTC--CA
Table 1.2: Detailed zero-shot comparison for 1--ma-3.118B-Instteee with C4 cal- ibration. Higher is better for Accuracy (%) (ACC.) (1) and lower is better for Perplexity (PPL.) (t).
(L)
ACC
Rank
Methods
ARC-E
ARC-C
HellaSwag
PIQA
WG
MMLU
OBQA
RA
AVG
MEAZMTA
284863.91 (L)
23.82
25.94
26.34
50.92
50.59
25.62
27.80
22.11
31.64
CARE-U
(4).17
27.61
24.40
26.98
54.68
49.88
22.95
26.40
20.19
31.64
CARE-E
(1).43(()
28.24
24.74
27.97
55,39
50.12
22.95
26.60
20.86
32.11
CARE-C-based-U
(J8.95(J)
CARE-C-based-4
$82.93())
29.55
24.15
28.15
56.75
52.25
22.95
26.20
20.96
32.62
Palu (SVD)
(260.60(L)
27.53
25.77
26.50
52,18
50.04
24.26
27.80
20.86
31.87
SVD-1TM 92
(69.05
27.74
24.57
27.10
54.62
49. 96
22,95
26.00
20.38
31,66
ASVD
2845.27
23.98
27.01
51.90
48.86
22.98
29.20
22.30
MI1A2LA
(L)
24.12
25.00
26.58
52.18
$1.07
23.82
29.20
22.11
31.76
CARE-U
22)
34.39
24.23
36.25
61.21
54.78
23.46
27.60
25.17
35.89
(0))19())
56.75
CARE-C-based-U
(82.54())
35.23
24.23
36.20
61.92
55.64
23.04
28.00
25.74
36.25
CARE-C-based
(76.68
37.88
26.88
40.07
64.09
55.49
25.02
29.80
26.51
38.22
Palu (SVD)
3046.58(L)
27.02
23.89
26.62
50,38
49.25
23.14
24.80
22.49
30.95
9VD-2IM V2
211,16(4)
34.51
24:40
36.77
60.99
54.78
23.44
27.80
25.65
36.04
1767.53(4)
MIAAMLA
1633.65(L)
25.63
27.47
26.50
52.39
50.83
23.10
28.20
22.49
32.08
CARE-U
35.63
53.07
34.73
60.63
72.14
63.06
40.09
32.60
34.93
48.91
CARE-E
(.)89
53.41
35.75
62.79
71.71
64,88
49.54
32.00
34.26
$0.54
CARE-C-based
34.32(3)
53.70
33.62
58.51
71,76
61.96
37.69
31.80
35.12
48.02
CARE-C-basede
37.79
55.30
35.49
61.34
72.42
64.96
42.85
32.20
35.12
49.99
Palu (SVo)
537.57(4)
31.10
22.61
27.93
55.22
51.70
22.82
27.00
22.68
32.63
SVD-LIM V2
35.49(L)
63.54
313.33(4)
31.57
22.53
29.77
55.55
53.28
23.21
26.00
23.35
33.16
NHA2HLA
22)
40.95
25.94
39.27
61.37
56.59
25.54
26.60
26.79
37.88
9.31(()
71.82
CARE-E
(8.07(.)
60.94
41.21
70.29
75.73
71.59
63.10
37.00
38.56
57.30
CARE-C-based-D
9,58(L)
76.47
50.77
73.14
77.58
71.98
59.61
39.20
42.39
61.39
CARE-C-based
(5.55(.)
66.12
43.09
70.58
76.28
71.27
62.32
37.00
39.90
58.32
Palu (SUM
45.401.)
23.83
V2
9.33(()
76.01
51.11
74,80
77.80
71.51
59.93
40.60
42.01
61.72
ADVD
12.01(5)
68.64
45.65
70.50
76.55
67.88
39.40
37.40
33.88
$4.99
I.1.3 -.I.------SSTTTT--PPPP
Table 1.3: Detailed zero-shot comparison -aama-..1-8--nstiiii with PTB cal-
ibration. Higher is better for Accuracy (%) (ACC.) (t) and lower is better for Perplexity (PPL.) (L).
PPL
ACC (t)
Methods
HellaSwag
PIQA
WG
MMLU
OBQA
2RA.
284863.91 (1)
23.82
25.94 35.94
26.34
50.92
50.59
25.62
27.80
22.11
31.64
CARE-U
(1)
25.00
51.36
49.25
22.95
27.00
20.67
CARE-E
(1),55(1)
25.60
26.51
51.25
49.49
22.94
27.20
20.38
MARE-C-basede
756.56(1)
26.73
25.68
26.28
51.58
50.20
22.95
25.80
21.34
31.32
MARE-C-based-
612.72(1)
27.57
25.43
26.51
$1.74
50.75
22.95
26.00
21.05
31.50
Palu (SVD)
2260.60(L)
27.53
25.77
26.50
$2.18
50.04
24.26
27.50
20.86
31.87
SVD--ITM V2.
801.38(J)
49.41
22.95
20.67
ASVD
2718.72(L)
26.39
25.09
26.91
$1.69
49.01
23.02
29.00
21.72
31.60
MHA2MLA
15028.91 (L)
24.12
25.00
26.58
$2.18
$1.07
23.82
29.20
22.11
31.76
229.71
23.73
CARE-E
19488(()
30.51
24.40
31.96
54.24
56.04
22.97
28.20
22.97
33,91
CARE-C-based-U
200.05 (1)
32.11
23.81
30.01
$4.90
53.67
23.30
26.40
23.54
33.47
CARE-C-based-E
178.91 (1)
33.00
24.74
31.94
$5.88
54.78
23.62
28.60
24.40
34.62
Palu (SWD)
3046.58(L)
27.02
23.89
49.25
23.14
24.80
22.49
30.95
V2
220.92(1)
23.98
53.28
25.07
25.40
23.92
33.39
ASVD
1776.16(4)
27.36
24.57
26.73
53.16
49.41
22.97
27.60
22.20
31.75
MIAZMLA
(633.65(L)
27.47
50.83
23,10
22.49
22
256

---

Published as a conference paper at ICLR 2026
(LL
ACC
Methods
ARC-E
ARC-C
HellaSwag
PIQA
WG
MMLU
OBQA
RA
CARE-U
34,44(J)
43.56
27.73
48.90
61.75
62.35
31.65
30.40
29.09
41.93
CARE-E
33.96 (.)
47.52
34.13
54.84
66.43
63.77
45.54
31.60
32.63
47.06
CARB-C-based-D
33.00(2)
62.51
CAse-C-based-E
30,87(3)
50.55
33.28
52.55
67.08
62.59
42.75
31.20
32.54
46.57
Palu (SVD)
537.57
31.10
22.61
27.93
55,22
51.70
22.82
27.00
22.68
32.63
SVD-1I4 V2
33.36(
47.69
29.52
50.43
64.69
61.88
35.32
30,80
30.43
43.85
ASVD
34).66(()
31.27
22.18
29.34
55.33
53.12
23.14
25,40
23.92
32.96
MIAZMLA
(2)
40,95
25.94
39.27
61.37
56.59
25.54
26.60
26.79
37.88
CARE-U
9,38(()
75.25
51.71
72.93
77.26
72.06
57.86
40.20
40.77
61.00
CARE-E
(J)
56.31
38.99
66.99
73.56
70.56
62.63
34,40
37.89
55.17
512
CARE-C-based-U
9.67(L)
71.19
CARe-C-based-E
13.00(.)
62,96
40.78
68.24
74,10
69.93
61.96
35,40
39.71
56.64
Palu (SVD)
45.40(.)
45.96
28.16
43.37
64.15
53.43
24.9%
30,80
23.83
39,33
SVD-LIM V2
9.42(L)
74.71
49.91
72.18
77.09
71.90
57.83
37,80
40.29
60.21
12.15(J)
1.1.4 1-I---3------ISSRUU--WWIITESS
Table 14: Detailed zero-shot comparison for Llama-3.11 3.1-8B- Instrrcct with WikiTexx2 calibration. Higher is better for Accuracy (%) (ACC.) (1) and lower is better for Perplexity (PPL.) (1).
(PL
ACC
Rank
Methods
ARC-E
ARC-C
HellaSwag
PIQA
WG
MMLU
OBQA
RA
AVG
NHA2HLA
284863.91
23.82
25.94
26.34
50.92
50.59
25.62
27.80
22.11
31.64
CARE-U
237.45
27.31
25.34
26.46
51.63
48.22
22.94
27.60
20.77
31.28
CARE-E
(8).54(
27.86
24.91
26.83
51.96
50.28
22.95
26.80
20.57
31.52
CARE-C-based-U
(1)
50.12
22.95
27.80
20.57
64
CARE-C-based-E
(38.00(L)
28.91
25.43
27.16
$2.39
50.36
22.95
27.60
20.38
Palu (SVD)
2260.60(L)
27.53
25.77
26.50
52.18
50.04
24.26
27.80
20.86
31.87
SVD-LLM v2
219.74
27.15
25.26
26.50
$1.69
48.62
22.94
27.60
20.86
31.33
ASVD
2900.25
26.09
24.06
26.77
51.25
49.25
22.96
29.00
21.91
31.41
MIAZMLA
(L)
24.12
25.00
26.58
52.18
51.07
23.82
29.20
22.11
31.76
CARE-U
(6.44())
32.37
23.98
31.04
$4.08
$2.33
24.10
2%.
23.73
33.78
CARE-E
(5.08
32.41
24.66
34.07
54.30
23.10
29.40
25.36
34.80
CARE-C-based-E
47.26
33.25
25.60
33.59
57.13
54.14
23.37
29.80
24.98
35.23
Palu (SVD)
3046.58(L)
27.02
23.89
26.62
50.38
49.25
23.14
24.80
22.49
30.95
SVD-"TM 92
(2.94
32.11
23.89
31.38
54.68
53.43
24.04
28.60
23.44
33.95
ASVD
1823.04 (L)
50.99
22.92
29
MHA2MLA
(633.65(L)
25.63
27.47
26.50
$2.39
50.83
23.10
28.20
22.49
32.08
CARE-U
16. (6.14(
45.88
29.69
$2.27
63.17
62.
33.26
33,80
32.63
44.
(6.84
34,30
65.59
33.59
CARE-C-based-U
(5.66(
60.85
33.65
31.67
43.59
CARE-C- -aa--C-
(5)53(
53,32
34.64
$5.90
68.01
63.54
41.05
33,80
33.68
47.99
Palu (SVD)
537.57
31.10
22.61
27.93
$5.22
51,70
22.82
27.00
22.68
32.63
SVD 3IM V2
15.88 (L)
48.19
30.03
$2.77
64.80
61.48
34.35
34.00
32.25
44.73
MIAZMLA
21)
40.95
25.94
39.27
61.37
56.59
25.54
26.60
26.79
37.88
CARE-U
(L)
51.54
73.94
77.48
72.53
56.99
39.40
40.19
(1.25())
71.27
CARE-C-based-U
8.29(L)
76.43
50.26
72.00
76.66
72.22
56.83
38.60
41.63
60.58
CARE-C-based-1
9.82(L)
66.96
42.83
69.64
74.76
70.64
61.99
37.20
38.85
57.86
Palu (SWD)
45.40(.)
45.96
28.16
43.37
64.15
53.43
24.98
30.80
23.83
39.33
SVD-LLM V2
8.26(L)
72.30
12.11(()
67.63
44.28
70.07
75.68
67.01
38.78
36.40
33.30
$4.14
I.1.5 SHEENSS-..BB-NNSTTIIITAAAII
23

---

Published as a conference paper at ICLR 2026
Table 15: Detailed zero-shot comparison for Qwen2.5-1.55 1.5.5.1..BBBB.. 3-B-Instruct with Alpaca calibration. Higher is better for Accuracy (9) (ACC.) (1) and lower is better for Perplexity (PPL.) (L).
(LL
(1)
Rank
Methods
ARC-E
ARC-C
HellaSwag
PIQA
WG
MMLU
OBQA
RA
AVG
MRAZMIA
(L)
26.56
25.34
26.57
51.63
50.20
24.60
29.00
22.68
32.07
CARE-1
1098. 80
26.77
23.38
26.95
52.29
51.07
22.95
25.20
23.16
31.47
CARE-E
1022.88 ())
29.00
23.29
26.70
54.73
$1.14
23.00
26.80
22.11
32.10
64
CARE-C-based
909.76(4)
29.00
22.35
26.14
52.50
48.15
23.11
26.60
22.30
31.27
CARE-C-based-E
740,60(4)
Palu (SVD)
114579.38 (U)
25.17
25.26
25.55
50,27
50.20
24.06
27.40
20.67
31.07
BVD-LIM V2
1109.62(4)
28.37
23.63
26.91
53.48
49.80
22.95
23.80
23.92
31.61
ASVD
743
26.81
25.77
26.61
$1.09
49.88
25,46
27.60
22.11
31.92
MIAZMLA
73056.19
26.14
26.71
26.51
52.12
50.28
22,90
26.80
21.82
31,66
CARE-U
(1)
33.96
22.35
29.80
56.26
50.91
24.13
28.20
25.07
33.83
CARE-E
48.58 (J)
49.37
29.01
46.54
66.70
53.28
31.33
30,00
31.29
42.19
CARE-C-based-U
(4),47
52.57
CARE-C-based-E
51,56(.)
52.15
30.46
46.17
67.68
54,70
32,03
31,60
29.47
43.03
Palu (SVD)
(471
25.97
23.98
26.49
50.87
50.43
24.33
30,20
20.57
31,60
SVD LIM V2
24).45
34.89
23.04
30,33
$8.49
50.51
24.81
28.
26.70
34.65
(663.48
NHA2XLA
(L)
26.81
25.68
27.39
$1.03
49.09
26.00
27.00
23.73
32.09
CARE-U
(G)
54.17
31.40
45.62
67.95
54.85
36.38
29.40
34.93
44.34
(_)
54.70
CARE-C-based-U
47)23(J)
55.39
31.66
55.41
35.75
33.68
44.04
CARE-C-based-
30.43(3)
56.61
32.34
51.13
69.53
53,43
40,91
34,00
31.39
46.17
Palu (SVD)
(0284.33(L)
28.45
21.59
27.10
52.29
51.14
25.49
30.60
22.68
32.42
SVU-_LM V2
44)
55.81
31.48
46.29
67.36
54.38
36.65
29.00
35.02
44.50
(482.91
1.1.6 1Ww.2.5-1.5B-INSTRUCCE
Table Detailed zero-shot comparison for Qwen2.55 1...5---1.-B--stttrt with C4 cal-
ibration. Higher is better for Accuracy (%) (ACC.) (1) and lower better for Perplexity (PPL.) (L).
PPL (L)
ACC (t)
Rank
Methods
ARC-E
ARC-C HellaSwag
PIQA
WG
MMLU
OBQA
RA
AVG
MIAZMLA
(L)77.36(L)
25.34
26.57
51.63
50.20
24.60
29.00
22.68
32.07
914.36(1)
22.95
CARE-E
6.4.13(.)
27.15
23.12
27.44
53.43
51.38
22.94
27.00
23.64
32,01
CARE-C-baseds
(79.30((1)
27.74
23.12
26.69
52.77
$1.22
22.95
26.80
23.16
31.81
CARE-C-based-
365.98
28.54
22.70
27.42
$5.01
52.09
22.95
26.40
23.64
32.34
Pa (SUD)
(14579.38 (U)
25.26
50.20
24.06
27
20.67
V2
826.83
26.22
25.26
49.64
22.95
25.40
ASVD
7074.97(L)
26.85
25.43
27.14
50.98
50.91
25.89
30.00
22.49
32.46
MHAZMIA
(L)
26.71
52.12
50.28
22.90
21
CARE-U
176.46())
31.14
20.82
30.39
55.55
$2.49
22.96
25,50
25.26
33.05
CARE-E
41.81
41.71
27.99
47.82
66.65
55.64
27.61
29.50
28.33
40.69
CARE-C- CARS-C-based-U
(51.08
32.15
22.61
31.53
$5.28
51.38
23.52
27.00
27.08
33.82
96
CARE-C-based-
43)55
43.10
27.05
47.85
68.06
55.64
27.45
31,00
31.48
41.45
(SVD)
147110.47
SVD--LM v2
161.86())
31.23
21.50
31.15
56.31
51.22
22.94
26.60
26.51
33.43
ASVD
(864.43(L)
28.75
23.12
27.49
$1.41
$1.30
24.75
27.60
23.16
32.20
(L)
49.09
CARE-U
34)14(3)
51.68
29.10
49.03
66.70
56.91
32.47
28.20
33.97
43.51
CARE-B
24)35(
49.28
29.35
$2.03
69.31
$6.59
33.39
32.00
31.20
44.14
128
CARE-C-based-U
(4)92())
47.98
27.56
48.03
65,51
56.51
32.67
28.80
34.16
42.65
CARE-C-based-E
54.62
Polu (SVD)
40284.33(L)
28.45
21.59
27.10
$2.29
51.14
25.49
30.60
22.68
32.42
SVD-1TM V2.
32.25((J)
51.98
28.58
49.92
67.36
58.25
34.43
29.20
33.78
44.19
ASVD
(244.07
30.77
20.99
27.47
$1.85
50.36
24.63
25.80
24.78
32.08
24

---

Published as a conference paper at ICLR 2026
1.1.7 QWEN2.5-1.5B. WWWN225--1.BB/SSTRUCTPPT.
Table 1.7: Detailed zero-shot comparison for Qwen2.5-1..5B-nnstee with PTB calibration. Higher better Accuracy (%) (ACC.) (1) and lower better for Perplexity (PPL.) (U).
PPL
ACC
Rank
Methods
ARC-E
ARC-C
HellaSwag
PIQA
WG
MMLU
OBQA
RA
AVG
MIAAMLA
(L))
26.56
25.34
51.63
50.20
24.60
29.00
22.68
32.07
2131.72(4)
CARE-E
362.35
29.59
23.55
29.43
52.45
50.83
23,10
25.00
22.39
32,04
CARE-C-based-
(608.42(L)
27.48
23.46
26.79
51.09
50.59
23.01
26.80
21.63
31.36
CARE-C-based-E
701.84(4)
27.95
23.55
27.12
51.52
50.04
22.95
24.80
23.73
31.46
(SVD)
(U)
50.20
20.67
SVD-1TM V2
2075.31
27.36
22.61
27.18
50,82
49.01
23.13
27.60
22.68
31,30
ASVD
8309.98 (L)
26.98
24.83
26.90
50.82
50.67
25.40
28.40
21.91
31.99
(L))
50.28
CARE-U
(67,50
28.79
20.99
29.39
52.29
52.72
23.79
26.00
23.92
32.19
CARE-E
(J)
41.04
26.37
43.62
62.02
53.35
25.74
30.50
29.47
39.05
CARE-C-based-D
(0).18(()
30,26
22.87
28.96
52.50
52.72
23.04
26.00
25.17
32.69
%
CARE-C-based-
57.23
41.12
25.94
43.82
62.95
54.46
27.52
29.20
28.33
39.17
(SVD)
147110.47 (U)
SVD SIM V2
(3)
29.55
20.14
29.52
52.50
52.57
23.02
25,20
24.69
32.15
ASVD
3704.61(4)
28.91
23.38
28.03
52.67
50.36
24.93
28.80
24.50
32.70
MHAZMLA
(L)
26.81
68
$1.03
49.09
26.00
27.00
23.73
32.09
CARE-U
(3.96(.)
45.03
26.96
43,72
61.10
56.12
29.55
28.80
33.49
40.60
CARE-E
22.47(.)
48.74
29.18
49.28
66.87
$5.96
36.80
31.60
31.39
43.73
128
CARE-C-based-U
65.89
44.99
25.60
41.95
60.07
55.64
32.38
27.00
31.20
39.85
CARE-C-based-E
54.93
32.06
(SVD)
(0)84.33
28.45
21.59
27.10
$2.29
51.14
25.49
30.60
22.68
32.42
SVD-SIM V2
(5.47(J)
45.88
28.58
44,00
60.77
55 64
29.19
29.00
32.92
40.75
ASVD
1210:45(L)
30.60
20.73
23.75
52.23
50.59
23.86
27.20
24.69
32.21
1.1.8
Table 18: Detailed zero-shot shot comparison for Qwen2.5-1.55 1-..5-11---nnttuue with WikiText2 calibration. Higher is better for Accuracy (%) (ACC.) (1) and lower is better for Perplexity (PPL.. (1).
(LL
ACC
Rank
Methods
ARC-C
HellaSwag
PIQA
WG
OBQA
MEA2XLA
62077.36
26.56
25.34
26.57
$1.63
50.20
24.60
29.00
22.68
32.07
CARE-U
(1)
26.77
23.12
26.74
52.18
50.20
22.95
26.60
24.50
31.63
(2)
23.12
50.99
22.96
23.54
CARE-C-based-U
209.05
27.86
22.18
$2.56
50.67
22.98
23.64
122.65 (.)
27.86
23.29
27.12
53.48
50.75
22.94
26.20
22.68
31.79
Palu (SVD)
114579.38 (U)
25.17
25.26
25.55
50,27
50.20
24.06
27.40
20.67
31.07
SVD-LIM V2
264.86
26.94
23.21
26.87
$1.96
49.96
22.97
26.40
24.59
31.61
(126.52(L)
50.20
MHAZMLA
73056.
26.14
26.71
26.51
$2.12
50.28
22.90
26,80
21
31.66
CARE-U
(1.60
21.33
29.95
54.24
50.99
23.13
26.20
23.54
32.63
CARE-C-based
(J.36(
30.81
22.10
29.87
53.48
49.72
22.97
26.20
24.50
32.46
CARE-C-basede
19.53
44.40
26.88
46.33
64,74
$4.78
27.99
30,20
30.05
40.67
Pal (SVD)
147110.47 (L)
25.97
23.98
26.49
50.87
50.43
24.33
30.20
20.57
31.60
SVD DIM v2
56.77(()
21.67
22.95
25.60
24.31
32.51
3636.42(L)
28.32
22.95
27.29
50.04
24.07
23.25
32.30
MHA.M
(L)
26.81
25.68
27.39
$1.03
49.09
26.00
27.00
23.73
32.09
CARE-U
(7.88
27.30
27.
28.
33.11
41.21
CARE-B
(5.67(
51.64
30.20
50.50
67.08
55.33
29.83
33.20
32.44
43.78
CARE-C-based
19.29(_)
47.10
25.85
44,14
62.08
55.49
30.69
29.20
32.06
40.83
CARE-C-based
(5.62
51.22
30.72
51,52
68.28
54.62
30.22
33.40
31.39
43.92
Palu (SVD)
(0284.33(L)
28.45
21.59
27.10
$2.29
51.14
25.49
30.60
22.68
32.42
V2
57.06
(241.39(L)
30.56
21.08
27.65
53.81
49.01
23.61
26.20
23.92
31.98
25

---

Published as a conference paper at ICLR 2026
1.1.9 QWEN3-4B QNEN3-B-IINTRRUC--5000
Table Detailed zero-shot comparison for Owen3-4B--nstratt2500 with Alpaca calibration. Higher better for Accuracy (%) (ACC.) (1) and lower better for Perplexity (PPL.) (t).
PPL
ACC (t>
Rank
Methods
ARC-E
ARC-C
HellaSwag
PIQA
WG
MMLU
OBQA
RA
AVG
MIRZKLA
(1850.16(L)
25.46
25.68
26.03
51.96
50.20
22.92
27.00
22.01
31.41
(0),74
29.00
55.22
26.60
32.32
CARE-E
(3),93(L)
30.77
23.29
28.33
55.22
51.54
22.95
24.80
22.78
32.46
CARE CARB-C-based-U
(052.43
29.34
22,70
27.51
54.35
50.67
23.21
26.20
23.64
32.20
64
CARE --based-
941,16(4)
28.75
23.12
27.33
55,33
50.28
23.08
25.60
22.30
31.97
Palu (SVD)
(4)
26.77
31.75
3VD-111
(4)
28.58
23.12
27.66
55,11
51.54
23.02
27.20
23.44
32.46
ASVD
6683.95
25.00
27.39
25.71
50.38
50.43
24.08
26.60
22.11
31.46
(L))
26.56
CARE-U
(4)
39.69
27.47
37.52
60.50
53.59
27.03
26.99
37.55
CARE-E
(L).38
44.82
30.29
42.54
63.76
54.54
30.52
29.40
28.71
40.57
CARE PARB-C-based-D
(48.73((1)
38.93
25.09
32.98
59.09
53.12
23.76
25.60
26.12
35.59
128
CARE B-B--C-
(39.32(4)
39.98
27.05
37.16
61.64
53.20
27.19
27.00
26.99
37.53
SVD LLM V2
116.76())
40.03
27.05
37.02
61.04
53.28
26.69
26.80
25.74
37.21
ASVD
(682.84(4)
30.01
22.95
29.29
52.34
50.12
23:42
27.00
23.54
32.33
MHAZMLA
44509.79
28.49
22.18
28.92
52.45
51.30
23.00
24.60
24.59
31.94
CARE-U
22.08 (.)
68.90
46.42
59.16
71.55
62.43
54.76
36,40
35,02
54.33
CARE-E
28)84(.)
59.22
41.30
56.53
69.37
61.88
$3.50
35.20
32.82
51.23
CARE -ARB-C-based-U
24)63(()
64.06
38.40
52.01
69.04
59.12
45.73
34,20
34.35
49.61
256
CARB-C-based-E
28.67(J)
53.58
1SVn)
2561.97(L)
29.46
26.
29.92
52.88
51,70
24.48
28.40
23.64
33.32
SVD-LIM V2
22.88
67. 17
44.54
57.77
70.78
61.48
52.81
35.60
34.83
53.12
ABVD
63.15(
46.84
32.76
47.45
63.93
52.01
26.38
30.80
30.05
41.28
MHAZKLA
100.99
41.08
27.05
37.97
59.19
$4.06
29.14
27.20
29.47
38.15
CARE-U
(2.03(.)
77.23
54.95
69.24
76.22
68.43
67.46
40.00
39.43
61.62
CARE-E
15.91
70.88
49.23
64.13
72.80
64.56
64.16
36.20
36.56
57.31
CARB-C-based-U
11,90(.)
79.04
64.32
512
(5.52(J)
71.04
47.18
59.86
71.93
63.30
62.19
35.40
36.84
55.97
Palu (SVD)
33.97(()
47.64
35.58
50.44
65.18
52.64
27.85
32.80
30.24
42.80
SVD-LLK V2
(J)
77.44
54.61
68.33
75.68
67.25
67.65
39,80
38.18
61.14
ASVD
15.49
66.54
47.61
67.49
73.01
62.75
56.56
35.60
35.22
55.60
1.1.10 AWEN3--B-INSTRUCT--22
Table 1.10: Detailed zero-shot comparison Bowen3--B-Instrutttee with C4
calibration. Higher is better for Accuracy (%) (ACC.) (t) and lower is better for Perplexity (PPL.) (L).
PPL
ACC (t)
Rank
Methods
ARC-E
HellaSwag
PIQA
OBQA
MARZMLA
21850,16(L)
25.46
25.68
26.03
51.96
50.20
22.92
27.00
22.01
31.41
CARE-U
38).
29.59
22.53
28.13
53.21
22.92
26.20
23.64
32.17
320.78(.))
54.79
32.42
CARB- CARB-C-based-U
494.84(.)
28.91
23.55
27.41
55.11
50.91
22.93
23,60
24.21
32.08
CARE CARB--C-aase-E
447.15(1)
28.28
21.50
27.90
55.28
50.36
22.95
24.40
23.35
31.75
Pall (SV))
(6922.11(L)
26.77
25.34
25.73
50.44
50.91
24.31
28.40
22.11
31.75
LIM V2
39).08
29.34
22.44
28.10
53.97
22.97
26.00
23.64
32.16
ASVD
5126.87(L)
26.98
23.55
26.77
52.23
$2.80
23.79
31.78
MARAMTA
(L)
26.56
23.81
26.74
52.18
49.72
24.74
28.20
22.68
31.83
CARE-U
(5.49
36.57
24.40
41.25
24.33
27.75
37.29
CARE-B
63.64(
40.99
27.99
45.02
62.51
56.20
28.22
29.30
28.80
39.87
CARE CARB-C-based-U 83.75(J)
35.19
22.27
36.49
59.47
50.99
22.97
28.40
27.37
35.39
128
CARE SUR8-C-baaed-E
(7.69
38.13
24.40
40.89
61.04
51.38
23.32
26,80
28.90
36.86
Palu1SVD:
22048.79(L)
26.18
26.02
26.29
$1.09
49.72
24.49
25.40
21.05
31.28
36.20
27.20
ASVD
641.70(1)
31.27
23.46
30.70
52.18
$2.01
23.40
27.20
22.87
32.89
26

---

Published as a conference paper at ICLR 2026
(LL
ACC (
Rank
Methods
ARC-E ARC-C
HellaSwag
PIQA
MMLU
OBQA
AVG
MIRZKLA
44509.79 (L)
28.49
22.18
28.92
52.45
31.30
23.00
24.60
24.59
31.94
CARE-U
18.25 (J)
64.65
41.13
60.92
71.11
64.48
52.19
35.40
35.98
53.23
23)20(J)
CARB-C-based-U
18.31 (2)
56.65
35.92
55,20
69.59
60.46
44.77
33.20
34.64
48.80
256
CARE BARB-C-based-E
22.22(3)
55.43
36. 18
56.03
69.15
61.09
44.64
32,80
34.74
48.76
Palu (SVD)
2561.97
29.46
26.
29.92
52.88
51.70
24.48
28.40
23.64
33.32
LIM V2
18.58
63.80
41.47
60.47
71.60
63.93
51.12
35.60
35.12
52.89
50.93
35.07
MAR2MLA
(0),99
41.08
27.05
37.97
$9.19
54.06
29. 14
27.20
29:47
38.15
CARE-U
(1,57(J)
53.67
69.70
76.50
40.19
CARE-E
14.54(3)
68.01
47.61
65.02
74.37
67.56
62.99
36.00
38.18
57.47
CARE CABR-C-
11.17(J)
79.50
53.84
65.25
74.97
66.46
66.46
38.20
40.86
60.69
CARE -eB--C-
13.08
69.19
45.05
61.41
72.91
67.09
59.17
36,20
39.23
56.28
Palu (SVD)
3..97(.)
47.64
35.58
50.44
65.18
52.64
27.85
32.80
30.24
42.80
SVE-LLX
11.43(J)
77.19
77.20
39.20
40.19
ASVD
14.37(3)
66.84
48.04
68.02
73.61
62.27
57.64
35.80
37.89
56.26
11.11 QWEN3-4B 10TE:------NSTTT-2550-----7
Table LII: Detailed zero-shot comparison for 1an-m3--A--Isstuct--225 with PTB calibration. Higher better for Accuracy (%) (ACC.) (f) and lower is better for Perplexity (PPI..) ()).
(LL
ACC (1)
Rank
Methods
ARC-E ARC-C HellaSwag
PIQA
WG
MMLU
OBQA
RA
AVG
(1))
25.68
51.96
CARE-U
96)
28.79
21.29
27.49
51.31
49.09
22.98
24.60
22.58
31.27
CARE-E
(.1.86(.)
29.50
24.23
27.87
52.39
49.33
22.90
24.80
23.16
31.77
CARE -Cbba---a
1234.23(())
28.49
22.44
26.91
51.74
48.46
22.95
24.20
22.20
30.92
64
CARB-C-based-E
1036.30())
52.67
31.35
Palu (SVD)
56922.11
26.77
25.34
25.73
50.44
50.91
24.31
28.40
22.11
31.75
3VD LLM
1001.12(L)
29.29
23.46
27.37
$1.58
49.96
23,00
24.20
22.58
31.43
ASVD
4441.27(4)
26.56
24.91
25.94
$1.90
51.70
23.69
27.40
21.91
31.75
MBRZMLA
(L)
26.56
23.81
26.74
52.18
49.72
24.74
28.20
22.68
31.83
CARE-U
191.71 (1)
35.14
25.77
35.12
$5.82
52.96
26.43
26.00
27.18
35.55
CARE-E
175.19
39.10
27.30
39.06
57.51
55.17
26.83
29.40
28.04
37,80
CARE CARE-C-based-U
206.34(()
33.16
23.72
32.04
55.22
50.43
20
25.40
26.89
34.01
128
C
194.64(1)
35.82
34.30
$5.66
24.66
35.07
(SVD)
22048.79
26.18
26.02
26.29
51.09
49.72
24.49
25.40
21.05
31.28
SVE-LIX V2
197,57
34.64
25.43
34.97
55.55
51.78
26.49
25.20
27.46
35.19
ASVD
(35.48(()
30.68
23.46
30.53
52,77
49.41
23.12
28.40
23.44
32.73
MHAZKLA
(L)
28.49
22.18
28.92
$2.45
$1.30
23.00
24.60
24.59
31.94
CARE-U
55.61 (3)
61.28
42.58
$5.95
67.63
63.54
47.57
31.40
33.68
50.45
CARE-E
(3)
54.71
38.48
54.11
66.59
63.14
47.64 64.64
31.80
31.39
48.48
256
CARB-C-based-U
44.02(.)
57.53
49.65
64.85
29.6
33.40
49.68
54.92
35.75
50.66
65.29
61.88
36.92
30.20
33.59
46.15
Palu (SVD)
2361.97(L)
29.46
26.11
29.92
$2.88
$1.70
24.48
28.40
23.64
33.32
BVD-113 v2
57.76(
61.24
40.87
54.79
67.19
62.43
44.89
31.60
33.30
49.54
ASVD
42.42(
48.82
34.39
49.72
65.67
28.20
32.60
30.62
43.05
MHR2MTA
100.99(1)
41.08
27.05
37.97
$9.19
54.06
29.14
27.20
29.47
38.15
CARE-U
(3.50
76.89
53.84
69.11
74.86
68.55
66.83
37.60
38.56
60.79
CARE-E
[5.60(
67.76
47.10
63.05
71.55
64
36.00
37.51
56.56
CARB-C-based-U
78.83
63.28
73.88
512
CARE --aeee
13.48(()
66.41
44.88
58.71
71.27
65.51
57.81
34.40
37.61
54.57
Palu (SVD)
33.97(
47.64
35.58
50.44
65.18
$2.64
27.85
32,80
30.24
42,80
SVE-LIM V2
19.51
76.68
51.75
67. .2
75.30
67.56
66.11
36.80
39.33
60.37
ASVD
14.45(.)
68.16
74.05
36.60
56.38
I.1.12 QWEN3-4B- INSTRRUCT-2577-WiiTiHe
27

---

Published as a conference paper at ICLR 2026
Table 1.1: Detailed zero-shot comparison for Qwen3-4B- --------rruttt22500 with Wiki- Text2 calibration. Higher is better for Accuracy (%) (ACC.) (1) and lower is better for Perplexity (PPL.) (4).
(LL
ACC: (1)
Rank
Methods
ARC-E ARC-C HellaSwag
PIQA
WG
MMLU
OBQA
RA
AVG
MRAΣNTA
(68))
25.46
25.68
26.03
51.96
50.20
22.92
27.00
22.01
31.41
CARE-U
(56.42(4)
28. 66
23.29
28.50
52.45
49.96
22.95
26.00
22.11
31.74
CARE-E
(12
30.05
23.29
29.12
52.39
50.75
22.93
26.00
23.35
32.24
CARE C-based-U
(6).94(()
27.99
22.35
27.16
52.18
22.95
25.60
23.35
31.35
64
CARB-C-based-E
(42.06())
Palu (SVD)
56922.11
26.77
25.34
25.73
50.44
50.91
24.31
28.40
22.11
31.75
SVD-ILX V2
(39.64
28.28
22.78
28.47
52.61
50.12
22.95
25.40
22.49
31.64
ASVD
(000.65
26.73
23.89
26.68
52.34
48.46
24.04
29.00
22.01
31.64
MIAZKLA
$2681.01 (U)
26.60
23.89
26.77
52.34
49.96
24,80
28.20
22.68
31.91
CARE-U
35.13(
37.04
25.85
38.24
58.81
53.20
24.36
28.80
27.94
36.78
CARE-E
2..98(.)
41.75
27.65
42.14
59.58
53.91
24.60
30.60
28.13
38.55
CARB-C-based-U
39.73(J)
128
CARE BaRB-C-basad-E
32,49(.)
36.99
25.68
36.85
58.76
52.01
23.62
29.40
28.52
36.48
Palu LSVD)
(L)
26. 18
26.02
26.29
51.09 19
49.72
24.49
25.40
21.05
31.28
BVD LLM V2
34)76(L)
37.33
26.02
37,91
58.11
53.35
24.02
27.60
28.52
36.61
541.20
30.89
53,70
MHAŽKLA
44510.05 (U)
28.54
22.01
28.95
52.67
$1.07
23.00
24,80
24.59
31.95
CARE-U
14.41 (3)
67.85
42.75
59.49
69.97
64.25
50.70
36.60
35.22
53.35
(6,91
U
[3.10(3)
60.98
37.12
53.15
66.38
59.67
38.44
32.40
35.31
47.93
256
BARB-C-based-E
14,51(
59.26
38.74
54.99
67.03
60.14
42.32
32,80
32.44
48.47
Palu:aVD)
2562.82(4)
29.46
25.85
30.02
52.94
51.78
24.40
28.60
23.54
33.32
SVE-LIX V2
14,34(J)
67.76
42.41
58.52
68.99
62.90
48.82
35.60
34.64
52.45
ASVD
39.75(L)
65.67
MRAZMLA
(0).99
41.08
27.05
37.97
59.19
54.06
27.20
29.47
38.15
CARE-U
(1.21())
78.03
54.61
70.03
76.77
67.92
40.40
40.00
62.04
68.35
47.53
72.74
66.54
57.21
CARE-C CARB-C-based-U
10.34(J)
79.92
52.73
64.79
74.92
67.80
66.29
39.20
38.95
60.58
CARE CARB-C-based-E
11.15(J)
71.84
46.93
60.83
71.98
67.17
61.89
37,20
37.89
56.97
Palu (3VD)
33.97(
47.64
35.58
50.44
65.18
57.64
27.85
32,80
30.24
42,80
3VR--TM
10.98(L)
77.99
69.41
76.12
ASVD
(4.29
67.17
48.21
68.16
73.99
62.27
35.80
37.51
56.36
I.1.13
OWEN3-30B-A3B A-N.AAA---NSTTTTT-5-----11II
Table L13: Detailed zero-shot comparison for Owen3-30B-oo 33B--Istructt2500 with Alpaca calibration. Higher better for Accuracy (%) (ACC.) (1) and lower better for Perplexity (PPL.) (L).
(P)
ACC (1)
Rank
Methods
ARC-E ARC-C
HellaSwag
PIQA
WG
MMLU
OBQA
RA
AVG
MERZMLA
1929 (L)
29.46
24.74
26.57
51.58
22.85
27.20
31.55
CARE-U
39.87
53.75
35.75
54.22
67.36
53.91
42.84
30.80
30.43
46.13
CARE-E
59.06
43.56
31.91
50.48
64.96
51.04
27.55
30.60
29.47
41.45
CARE --ubee-
(_.
58.75
38.74
$8.02
71.65
55.33
48.40
36.00
33.78
50.08
128
CARE --Beee--
36.65
50.00
33.36
56.32
69.86
54.46
43.01
33.00
31.48
46.44
Palu)
SVU-LLM
34.91 (3)
55.30
36.60
55.38
67.68
54.22
43.58
31,80
31.20
46.97
ASVD
2023.42(L)
27.48
23.98
28.78
$2.88
49.88
23.54
24.60
23.44
31,82
MHAZKLA
34.09
27.65
33.20
CARE-U
10.02
74.54
54.52
74.44
77.15
66.61
71.64
40.00
39.81
62.34
CARE-2
16.26(
56.82
41.72
69.36
73.01
63.22
69.54
37.60
34.64
$5.74
CARE-C-bae CARE-C-based-U
9.45(L)
78.24
54.10
73.56
78.29
68.82
72.95
39.60
38.18
62.97
256
BaRB-C-based-E
58.63
75.03
Palu (SVD)
6193.83(L)
27.90
24.91
27.54
53,16
49.33
24.01
25.00
22.01
31.73
SVR-TIM V2
9.58
76.47
55.
74.45
77.64
68.19
72.26
41.00
39.81
63.12
ABVD
217.93(1)
36.32
26.71
35,80
54.46
50.43
23.53
28.60
25.36
35.15
28

---

Published as a conference paper at ICLR 2026
Ubma 38-Instruct 20023-256-6
ie-liistruuctttt miarann
0.0
0.0
8.0
13.0
boon
44.0
89
$4.0
070
670
010
28.0
Dapitrll Raade
1000
13000
3076
8330
30/009
5000
Usme
milrann: alpace
Llama 3.1 Instruct miarank
0.-
0.-
Dicenene Dorr
44.0
44.
56.0
$4.0
07.
87.0
100
600
3076
4399
11500
Figure 1.1: Needle-in-a Hayss-ackaaa retrieval heatmaps for budgets. The figure assembles five panel-wise heatmaps comparing CARE variants with different calibration settings against Palu(SVD) across context lengths and needle depths. Greener cells indicate stronger retrieval accuracy.
under matched KV
1.2 LONG-CONTEXT RETRIEVAL: NEEDLE-IN-A-HAASTTAN (NIH)
We evaluate long congectntxt behavior with the Needle-in-z -n-a-aaystack (NiH) task, measuring retrieval accurucy across context lengths L. (1K, 4K, 8K, 16K, 24K, 32K} and multiple document depths (needle positions).
compare the uncompressed teacher with CARE and Palu(SVD) under identical KV budgets and training settings.
CARE stays close to the uncompressed teacher across 1K-32K contexts and document depths, while uniform-rank and SVD-style baselines degrade-most at 18K-24K. Covariance-awarc allocation (especially preserving early-layer rank), longer calibration sequences (e.g., Alpaca-2048 vs. Alpaca- 256) and sequence length further improve stability, and Palu(SVD) shows the largest drop. The panel-wise heatmaps below provide the full retrieval breakdown.
1.3
GENERATION EXAMPLES
Fig.
1.2 and Fig.
1.3
are two generated text examples by 2 different methods.
29

---

Published as a conference paper at ICLR 2026
Question:
What is the capital city of France?
A. Paris B. New York C. London D. Rome
Zero-shot FARE-Rank255
2. What is the capital city of the United States? A. New York B. Washington D. New York C. Washington York What the What the capital city the United States? A. New York B. Washington D. New York C. Washington D. New York 4. What is the What is the capital city of the United States? A. New York B. Washington D. New York C. Washington D. New York...
New
Zero-shot all((S VD)-R ank 256
had a very good (I had just like city, and small, and small, and small, and small, and small, and city of the location, and the 3F (I had very good had small,
just a city, small, a city of location, and the 3F (had very good had just like city, and small, and a small, and a small, and small, and a small, and a city of the location, and the 5 3F..
small,
small,
small,
Figure I.2: Generated Text Example
Question:
What is the capital city of France? Give me some details
Zero-shot CARE-Rank512
We Paris, France.
What famous landmark France? popular?
many
students bus
worned the
suddonly stopped
continue journy.
Iiked the the Eiflel tower. was interesting
whole
that
really
What the activ the French What the cuttuaal
treditions that make the country unique??
euglann
Each
different
dress and different traditions. The
cultural actvilles
would
how President United States
aducaation for the children educated
children
nacassary good
create better
the society going from the children that educatte.
were study degree
you
you
alder?
important good job
your
strong reletionhip employer. my shrong management intellgance...
job
Zero-shot PDD) --DD))RRaaa Rank 512
bbangullyyd CrnhhmmmrmrMraaaa Cesulp mmmoReompsmmme morot erg0moc
qja Pombbolit (px0mmi-0pgteeee Spg.ee wpw. Parti 1i M000 Png.a wwwpphhhhPPPPPhhPhooeeep
1g ofroorrttmr wwtpee/mmmrrnnnniyneeeeeeeee ContymoonPnmme www.eeeenesssmmmmmmoeoooooo www.mmeeeeeiiiiieeeo p000mmmppp
0p000mmm2t hell UolbuuP mmnnanneerrreedpppppmp www.geeeeeeeeerreeeeeerreee
Figure L3: Generated Text Example
30

---

Published as a conference paper at ICLR 2026
1.4 ADDITIONAL RANK PROFILES ACROSS CALIBRATION CORPORA
We present covariance-aware rank profiles for Llama-3.1 1-1-70B-Instruct
2507, Wwen3--B-Insstuut-22 and Qwen2.5-1.555 1-05-115--Insttrue Across all models and some calibration corpora (Alpaca, WikiText2, PTB, C4), the depth-dependent pattern observed the main text for 1imma-3.--B-IInttruu persists: ranks smaller in early layers, grow through the middle blocks, and stay elevated in deeper layers, with stronger growth for Wv than for WK. This consistency across model families, scales (1.5B-70B) and architectures (dense and MоE) suggests that the structure is a general property of pretrained attention. The same qualitative trend also visible on the smaller Qwen2.5-1.. 1..5B 15B-Instruct model, although its feasible target-rank range is narrower.
Quenn-weecch
Figure 1.4: Covariance-aware rank profiles for -0--3--33--A3--nstuuee (MoE, 30B total / 3B active) under Alpaca calibration at target ranks 128, 256. Despite the mixture-of-expper architecture, the same depth-dependent pattern persists, confirming the trend generalizes beyond dense models.
Layer
Lagur
Figure 1.5: Covariance-aware rank profiles for target ranks 128, 256. The same depth-dependent pattern observed in the 8B variant holds at 70B scale, with Wv exhibiting stronger late-layer growth than W k.
under Alpaca calibration at
Qwea Instruct
Qwez/Quea2.5-1555 Instractt
Casl meetio
CABE mou
CAll mas
Layur
Lager
Labur
L.oer
QuessQees.....
Layer
Figure 1.6: Covariance-aware rank profiles for 1.ren..5-1.BBIInsttre across Alpaca, WikiText2, PTB, and C4 calibration corpora at target ranks 64 and 128. Despite the smaller rank budget, the same depth-dependent increase remains visible, again with stronger late-layer growth for Wy.
31

---

Published as a conference paper at ICLR 2026
Phice-Quce--ee Instred 2507
Qnta Ins---ntrudd 1507
CABl
Cold
Cold mali
Lagur
Lager
Laguer
Loger
Layer
Lager
Layer
Layer
QueniQureS-BB (aa--------tmt---22 2507
Lepeenta parc Bock Miaed
Lepeemte Dyoae es
C4B0
CAB4
0.08 (1a0.11
Lance
Layy
Figure 1.7: Covariance-aware rank profiles for Qwen3-44-Inntruct-255 across Alpaca, WikiText2, PTB, and C4 calibration corpora target ranks 64, 128, 256, and 512. Both Wx and Wy show depth-dependent increase, with stronger late-layer growth for Wy.
1.5 DISTRIBUTION SIIFT
We evaluate cross- cross-domain calibration by estimating covariance accuracy changes on out-of-domain tasks. Tab. [1.44 shows that task-related calibration corpora such ARE and ARC give the best average accuracy, but the gains mostly local rather than universal. In contrast, narrower manguage-modeling corpora such as WIKITEXT2 and PTB transfer less well, suggesting that broader or task-relevant calibration data is preferable for robust one-shot performance.
corpus and reporting
Table 1.14: One-shot Llama3.1 1-8B comparison different covariance. Higher is better for Accuracy (ACC.).
Rank
Methods
ARC (?)
ARE (1)
HellnSwag (t)
PIQA (
MMLU (f)
OBQA (f)
RA (f)
WG (?)
AVG (?)
CApa-Alpaca
49.32
35.98
CARE-ARE
38.82
72.90
41.25
73.01
53.13
25.80
31.87
63.69
50.06
CARE-ARC ARC
39.59
71.84
41.33
72.91
52.84
26.60
32.54
62.67
50.04
256
CARE-RERiTeXt2
28.33
57.11
38.07
68.28
37.72
19.60
33.49
62.59
43.15
CARE PTB
27.99
51.91
37.70
67.68
42.15
17.00
31.96
62.59
42.62
CARF-C4
43.00
41.69
CARE MYLU
34.13
64.94
40.65
70.73
55.27
21.20
33.97
64.40
48.16
I.6 SHRINKAGE COFFFICIENT
We sweep the shrinkage coefficient a in C. = (1 -1-aCC a)C a to choose default regularization level: Tab. 1.15 reports the full ablation. Across
1-8---nnstttutt and (wwen-4B-IInstru at target ranks 256 and 512, using the same Alpaca-256-33 calibration setup, one-shot average accuracy highly stable for E {10- 10 10 }}: the variation is below 1 point Llama and below points in all tested settings, with a = 0.01 consistently near the best operating point. This indicates that CARE is sensitive the exact shrinkage magnitude.
32

---

Published as a conference paper at ICLR 2026
Table 15: Shrinkage-cocffiientt ablation for CARE under the Alpaca-256-32 calibration setup. We report
one-shot accuracy across Llama-3.1 1-8B--...-----ttrttere and Qwen3-4B-Instiee at target ranks 256 and 512. Higher is
better for Accuracy (ACC.).
Rank
ARC
ARE (f)
HellaSwag (1)
PIQA (f)
MML.U (f)
OBQA (t)
RA (t)
(t)
(*)
10
34,90
62.42
42.44
72.31
57.15
21.00
34.74
64.48
48.68
256
10 2
34,90
62.46
42.52
72.14
57.20
21.60
35.60
65.35
48.97
Llama-3.1 -B-Instruct
10
32,94
59.34
41.55
70.46
56.88
20.20
35.31
65.75
47.80
10-3
46.16
76.30
53.64
77.09
65.54
29.40
41.44
72.85
57.88
512
10-2
51.79
51,70
74,54
70.13
30.20
39.43
68.27
58.35
10-2
47.27
77.10
54.40
77,80
66.15
28.40
42.58
74.35
58.51
10
43.79
70.72
58.13
58.25
256
10
43.09
72.22
43.71
70.18
$7.10
26.00
35.02
62.51
51.23
Qwen2-IB-Instnen
10-
44.64
10-3
55.31
74,16
47.27
71.55
65.87
28.20
36.65
65.04
55.51
10-2 51.79
74.54
30.20
58.35
10-1
55.90
79,85
50.60
73.07
68.20
27.60
37.13
65.98
57.29
This robustness is expected: shrinkage mainly regularizes the low-energy tail of the covariance spectrum to improve numerical stability, while CARE's rank allocation is governed by the dominant eigendirections that remain largely unchanged. We therefore use a 0.01 as the default setting throughout the paper unless otherwise specified.
1.7 MIx OF COVARIANCE
Our Adjusted-Raal Rank allocator is governed by the spectrum of VOW (Sec. 3.2), so it naturally extends to mixed calibration distributions. We form Caix = Σn H, C2 with the same shrinkage toward I, adaptations
yielding simple multi-objective covariance fusion related in spirit covariance-awe such as CorDA YYngg et al., 2024b). In practice, we use weighted calibration mixing by sampling from task-weighted data mixture. Tab. I.16] keeps all other settings fixed (CARE, Rank 256) and mixes ALPACA with ARC-CHA ARC-CHALLENGE: increasing the ARC-CHALLENT weight improves ARC, while a balanced 0.5/0.5 mix gives the best average accuracy.
Table 1.16: Task weighte covariance mixing for multi-objective rank allocation (Llama3, 1-SB-lnst CARE, Rank 256). A denotes Alpaca and ARC denotes ARC-Cha 1h mixes indicate sampling proportions used to estimate Cmix. Higher better for Accuracy (ACC.).
Runk
Methods
ARC (1)
ARE(1)
HelloSSag (f)
PIQA
MMLU (1)
OBQA
RA (f)
WG (1)
AVG (*)
CARE-alpaca
34.81
65.40
40.76
72.47
49.32
21.20
35.98
62.98
47.87
256
CARE-O. 5.4...5RE. 5ARC
67.72
CARE-0 2..0. BARC
34.73
65.03
42.99
72.25
58.02
23.20
35.41
64.96
49.57
CARE-ARC
39.59
71.84
41.33
72.91
52.84
26.60
32.54
62.67
50.04
Notably, the 0. 2 8ARC mix actually degrades ARC-Challenge accuracy (34.73) relative to pure Alpaca (34.81), despite devoting 80% of calibration mass ARC data. This suggests that over-concentrating the covariance on narrow task distribution can be counterproductive: the heavily skewed Cmix collapses the effective spectrum onto a few ARC-specific directions, starving mid- spectrum components that still contribute to ARC reasoning through shared linguistic features. By contrast, the balanced 5 5ARC mix retains broader spectral coverage from Alpaca while incorporating enough ARC signal to steer rank allocation, yielding the best average accuracy (50.14) 14) and a meaningful ARC gain (+1.28 over Alpaca). This points to a practical guideline: mixing a general-purpos. corpus with moderate task-specific data is preferable to heavy task overweighting when estimating calibration covariance.
1.8 VC VS.
VS. C UNDER LOW-RANK BUDGETS
Throughout the paper, CARE uses VC as the default covariance weighting: the variant that replaces VC with C' denoted CARE-C-basee CARE-E correspond to / We-ghttt uniform and enrgy--wware allocation, while CARE-C- --bsee--UU and C-aRe-C-based-E use C directly.
the detailed tables (Sec. I.1. Concretely, CARE-U and
33

---

Published as a conference paper at ICLR 2026
Table 1.17: Comparison of C' versus vC covariance weighting under Alpaca calibration. Values are zero-shot
AVG accuracy (%); within each CARE variant the better result is shown in bold.
Model
Rank
CARE-U (VO) CARE-C-based-U (C)
CARE-E (vC)
CARE-C-based-E (C)
64
31.89
32.38
32.37
32.71
128
36.72
37.11
38.59
38.56
Ilama-3.1 1.8---------ttuu-
256
50.00
48.79
52.62
51.41
512
62.33
61.69
57.82
59.00
64
31.47
31.27
32.10
32.34
Qwen2.5-1.5B 1...2..5-...B---1 I-Insrrucct
33.83
34.57
42.19
43.03
128
44.34
44.04
45.86
46.17
32.32
32.20
31.97
37.55
35.59
40.57
37.53
Qwen3-4B- Beeen--B-Instruut-2 Instruct-2507
54.33
49.61
51.23
49.64
512
61.62
60.44
57.31
55.97
Tab. 1.17 summarises the zero-shot AVG accuracy under Alpaca calibration. The full per-benchmark breakdowns appear in the detailed tables of Sec. [.] (Tables 1.1-1.12).
When does C help? Using C instead of VC squares the eigenvalues of the covariance, amplifying the contribution of dominant eigendirections and suppressing the spectral tail more aggressively. At very low rank budgets (e-g., rank 64-96), there few latent dimensions to allocate, and this sharper concentration can be beneficial: forces the decomposition to focus on the handful of directions that carry most activation energy, reducing the most impactful reconstruction errors first. This effect is most visible on the smaller 1...c..5-1.5BIInstiie where the CARE-E with C' weighting outperforms vC at ranks 64, 96, and 128, and on Llama-3. 3-B- IB--nust at rank 64.
Why vC is the default. As rank grows, the allocator can afford to preserve mid-spectrum directions that C' discounts too heavily. On Llama-3.1 -SS--nstruut at rank 256 and Qwen3-4B-Instruct across all ranks, VC consistently outperforms C-often by substantial margin (e.g., +4.7 points for CARE-U on Qwen3-4B at rank 256). The gentler spectral re-weighting of VC retains enough emphasis on dominant directions while preserving informative mid-range singular values, yielding more robust default across model sizes and rank budgets.
1.9 SYSTEM EFFICIENCY ANALYSIS
We evaluate inference efficiency in terms of KV-cache footprint. Since KV memory scales linearly with sequence length, a 32K-context calculation reflects the practical regime. For L = 32768, B = 1, FP16, and 32 layers, the original GQA model caches 1024-dimensional keys and values, requiring 4294.97 MB. After full 100% MLA conversion, TransMLA + CARE(E) Init stores 148-dimensional key NoPE latent and 512-dimensional value latent, reducing the footprint to 2013.24 (53. 13% reduction vs. GQA), as shown in Tab. [.18
Table 1.18: Theoretical KV KV-cache footpriat at 32K context length (L 32768, B 1, FP16, 32 layers). Higher reduction indicates lower KV-cache memory.
Method
Cached State
Memory (MB)
Reduction GQA
GQA (Original)
1024, V: 1024
4294.97
TransMLA CARB(E) Init (Ours, 100% MLA Restore)
KNoPg 448, Viston 512
2013.24
53.13%
34


Reviews:
[{"review_id": "j61H4TXnCu", "reviewer": "Reviewer_iFpw", "summary": "This paper presents a practical pipeline, called CARE (Covariance-Aware and Rank-Enhanced), to convert already-trained MHA/GQA attention in LLMs into MLA, while keeping the same KV-cache budget. The motivation is that many current methods only minimize weight-space error by a joint SVD on (W_K, W_V), and they use a uniform rank for all layers, which does not consider the real activation distribution at inference and makes some sensitive layers over-compressed. Experiments on Llama-3-8B and several benchmarks show that, under the same KV budget, CARE can keep lower PPL and better task scores than uniform-SVD style baselines.", "questions": "1. How robust is the covariance-aware step if the calibration set is distributionally far from the deployment set (e.g., code LMs but text-only calibration)? Do you recommend streaming/online covariance updates, or is healing enough?\n2. Different tasks seemed to benefit differently at low KV ratios — could you expose task weights to the rank allocator to get multi-objective water-filling?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem is realistic and currently unsolved: we have many deployed MHA/GQA models, but we want the MLA advantages (smaller KV, more efficient decoding) without re-training from scratch.\n- The paper does not only say “data-aware compression is better”, but it derives the form by considering the activation-space objective, which is more solid.\n- The global and non-uniform rank allocation is motivated by data (different layers have different sensitivity) and the solution is standard and optimal under the budget.\n- Experiments cover multiple tasks and multiple KV budgets, and they include the correct baselines, so the message is quite consistent.\n- The design of KV-parity shows the authors think about real deployment and real KV bandwidth limitation.\n- Reproducibility seems good, with anonymized repo and scripts mentioned.", "weaknesses": "1. The robustness of the covariance estimation is not fully shown. All calibration corpora are relatively similar; it is not clear whether for code or instruction-heavy data the same rank allocation is still good.\n2. The relation to other data-/curvature-aware compression methods could be compared more directly on at least one setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vjn4RWwfI5", "reviewer": "Reviewer_77dF", "summary": "CARE proposes a post-hoc conversion method to transform pretrained GQA/MHA attention modules into Multi-Head Latent Attention (MLA) while maintaining fixed KV-cache budgets. CARE replaces the common joint‑SVD initialization with an activation‑aware factorization, enabling post-hoc conversion of pretrained attention models to memory-efficient MLA.", "questions": "1. Can the authors provide ablation results for: CARE decomposition (activation-aware SVD) with uniform rank allocation and standard SVD with CARE's adaptive rank allocation? This would clarify whether improvements come primarily from covariance-weighting or adaptive scheduling.\n\n2. Model scale and architecture diversity: Can the paper provide results on larger models (Llama-3-70B, Qwen-72B) where KV-cache is more critical? How does CARE perform on MHA to MLA conversion? It would also be helpful to see Table1-style results for other model families (Qwen, Mistral, Gemma) at comparable scales? For instance, for MoE models (e.g., Mixtral), do different experts need different rank allocations? Without broader architectural validation, it would be unclear if CARE is a general solution or specifically tuned to Llama-style architectures. \n\n3. Can the authors also evaluate on long-context benchmarks (LongBench, InfiniteBench, Needle-in-Haystack)? This would help see CARE's advantage scale with sequence length (1K, 4K, 8K, 16K, 32K tokens) and confirm if the covariance estimated on 2048-token windows (Appendix D) generalizes to longer contexts?\n\n4.  Shrinkage parameter α: C_λ = (1-α)C + αλI was chosen for invertibility, it would help to have more details around how were α and λ chosen and include ensitivity analysis over α ∈ {0.001, 0.01, 0.1}.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. While covariance-weighted factorization exists in prior work (FWSVD, SVD-LLM), the specific formulation of SVD(CW) followed by C^(-1) unwhitening for MLA conversion is fairly novel.\n\n2. Adaptive rank allocation under fixed budget: The water-filling algorithm for distributing rank across layers based on weighted singular spectra is a good solution, with the empirical observation (Figure 2) that layers have heterogeneous sensitivity to rank reduction motivating this approach. The paper also does a good job of combining activation-aware factorization, layer-wise rank scheduling, and KV-parity constraints into a cohesive pipeline.\n\n3. Evals: Experiments across 8+ benchmarks (WikiText, ARC, HellaSwag, MMLU, etc.) with both zero-shot and fine-tuned evaluations are reported. The paper studies calibration dataset sensitivity (Table 2), rank profiles across corpora (Figure 3), and layer-wise heterogeneity (Figure 2a-b), providing empirical support for design choices. Some ablations are well-designed: Section 4.2.1 shows consistency of rank profiles across four calibration corpora, supporting the claim that rank distribution is model-intrinsic. Section 4.2.2 demonstrates robustness to calibration corpus choice and Figure 2's layer-wise sensitivity analysis supports the heterogeneous rank allocation.", "weaknesses": "1. The paper's central claim uses ||C(W - Wc)||²F as a proxy for ||√C(W - Wc)||²F (page 5, Section 3.4), justified by a brief eigenspace argument that both are left-multiplied by the same eigenspaces of C with different weightings (λ²ᵢ vs λᵢ).  This essentially squares the importance weights, which could over-emphasize dominant directions and under-represent moderate-variance directions that still matter for downstream tasks. The paper's claim that this \"tends to preserve ordering of dominant components\" is unsubstantiated.\n\n2. Results focus almost entirely on Llama-3-8B, with Qwen models mentioned only in Appendix D hyperparameters but absent from main results tables. Scalability to larger models (13B, 34B, 70B) where KV-cache can be more critical remains unvalidated. The paper is also missing evaluation on models with different attention patterns (e.g., Mixtral MoE, Qwen with different GQA ratios) and would benefit from analysis of whether the depth-dependent rank profile (Figure 3) generalizes across architectures.\n\n3. MHA2MLA is dismissed for zero-shot evaluation due to partial-RoPE, but no fair comparison is provided after fine-tuning (Table 3 only compares TransMLA). Other activation-aware SVD methods (SVD-LLM V2, FWSVD) are cited in related work are not empirically compared.\n\n4. Missing long-context evaluation: Despite KV-cache being important for long sequences, no experiments on long-context benchmarks (e.g., LongBench, ZeroSCROLLS) or needle-in-haystack tasks are provided to validate usecases where MLA/CARE matter most.\n\n5. Fine-tuning details: Table 3 shows \"healed\" results but doesn't specify exact token budgets, learning rates, or whether hyperparameters were tuned per method. The claim that CARE \"requires less data\" isn't quantitatively supported with ablations over data budgets.\n\n6. Computing covariance (O(ND²)) and SVD on CW requires non-trivial cost, but there's no comparison of conversion time vs. naive SVD is provided. The paper also states CARE preserves MLA's \"comparable throughput\" but provides no actual latency/memory measurements or throughput benchmarks on realistic workloads. The paper claims CARE is practical, but doesn't provides runtime/memory analysis of the conversion process itself to validate this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MdQPF0Zt2S", "reviewer": "Reviewer_cgmu", "summary": "The authors propose CARE, a post-training adaptation method to transform trained MHA/GQA module to MLA for improved efficiency and performance. The CARE method essentially performs SVD as best low-rank approximation not over weight matrices but on the product of weights and inputs. They also propose to distribute rank budgets to different layers based on the observation of heterogeneous KV rank across different layers of a Transformer model. The model after CARE conversion is trained with a teacher-student knowledge distillation loss along with additional SFT to heal / recover lost performance due to the conversion. The resulting method achieves better performances compared to other baseline in this domain on various language modeling task.", "questions": "- It’s good to see that the CARE method is better compared to lots of baseline in terms of perplexities / accuracies etc, but how expensive is it compared to other baseline? It just seems like optimizing equation 2 is very costly.\n- I’m happy to raise my score if authors address my concerns & questions.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed method is a clear improvement over prior works, and the insight on performing SVD over activations is indeed a good one.\n- The paper is well-written and easy to understand.\n- I really like observation 1 & 2 and it’s an interesting read.", "weaknesses": "- Line 068-069: I disagree that this is a whitening operation. Whitening is a very well-defined operation. If X is a matrix of shape (B, D), where B is batch size and D is dimension. Then whitening is X * (1/(B-1) * X^T X)^{-1/2}. Also we assume that X is centered. So I suggest avoiding using the very specific term of whitening with a mathematically precise definition.\n- In general, all the text in the figures are too small and impossible to read if you print out the paper.\n- Line 205-211: I don’t think this is precisely the covariance matrix. You are missing a 1/(T-1) multiplier and a centering operation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RLRLkDiJgm", "reviewer": "Reviewer_w58u", "summary": "This paper proposes a Covariance-Aware, Rank-Enhanced MLA conversion pipeline under a fixed KV width for converting Grouped Query Attention (GQA) architectures into Multi-Linear Attention (MLA). The goal is to improve the efficiency of large language model (LLM) inference while maintaining strong performance.\n\nThe method introduces three main innovations:\n\t1.\tActivation-preserving factorization — a decomposition scheme that minimizes error in activation space (∥XW − X̂W∥) rather than in weight space, aligning the approximation with the actual activations encountered during decoding.\n\t2.\tAdjusted-rank allocation — a rank-adaptive scheduling strategy that redistributes a fixed KV cache budget across layers, allocating more capacity to layers with higher representational importance.\n\t3.\tKV-parity mapping — a reparameterization technique that reformulates the converted K and V projections to match the MLA format while preserving covariance structure.\n\nEmpirically, the paper reports that this approach achieves superior zero-shot performance compared to TransMLA, suggesting better preservation of representational fidelity under constrained KV width.\n\nHowever, the fairness of the comparisons and the novelty relative to TransMLA warrant further clarification—particularly the relationship between the proposed activation-preserving factorization and KV-parity mapping versus TransMLA’s RoRoPE and BKV-PCA components.", "questions": "See Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well-motivated, addressing a meaningful and practical problem in efficient LLM inference under KV cache compression. The proposed Covariance-Aware, Rank-Enhanced approach presents a sound and intuitive idea for improving the quality of MLA conversion under a fixed KV width.\n\nMoreover, the ablation studies are well-designed and effectively demonstrate the effectiveness and contribution of each proposed component, providing convincing empirical support for the overall method.", "weaknesses": "1. Comments on Methodology and Comparative Analysis. \nIn Lines 077–092, the manuscript states that TransMLA employs a direct SVD initialization that minimizes the error in weight space (∥W − Ŵ∥) rather than in activation space (∥XW − X̂W∥), thereby overlooking how the projection actually operates during decoding. However, it should be noted that the original TransMLA paper introduces three key innovations—RoRoPE, FreqFold, and BKV-PCA—all of which are explicitly designed to perform activation-aware decomposition. The manuscript would benefit from a clearer discussion of this distinction and its implications.\n\n2. Concerns on Rank-Adaptive Scheduling and Efficiency.\nThe proposed rank-adaptive scheduling under a fixed KV width raises concerns regarding the variability of KV cache sizes across layers and attention heads. Such heterogeneity may lead to inconsistent memory allocation and suboptimal utilization of hardware resources, potentially diminishing the actual inference speedup. It is therefore recommended that the authors conduct empirical speed comparison experiments based on established efficient attention frameworks such as FlashMLA and FlashAttention to validate the claimed efficiency gains.\n\n3. Comparative Evaluation and Missing Baselines.\nTable 1 currently reports results only in comparison with TransMLA. In their experiments, compressing the Llama-3-8B KV cache to 1−576/2048 = 71.875% yields a perplexity of 25.8047 [1], which is substantially lower than that reported in this manuscript. This discrepancy warrants further investigation. In addition, the evaluation should be expanded to include other relevant GQA-to-MLA conversion methods, notably Palu [2] and MHA2MLA [3], to ensure a fair and comprehensive comparison.\n\n4. Clarification on RoRoPE and Absorb Operations.\nThe operation described in Line 291 appears conceptually related to the RoRoPE mechanism proposed in TransMLA. The manuscript should more explicitly delineate the differences and connections between these two approaches. Furthermore, it remains unclear how the paper achieves the conversion from GQA’s RoPE to NoPE. The authors should also clarify whether their method supports the MLA Absorb operation, which is a crucial design feature in TransMLA enabling the transition between efficient training and efficient inference.\n\nReferences\n\n[1] TransMLA: https://github.com/MuLabPKU/TransMLA\n\n[2] Palu, https://arxiv.org/pdf/2502.14837\n\n[3] MHA2MLA, https://arxiv.org/abs/2502.14837", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)