Multi-Head Low-Rank Attention
Authors: Songtao Liu
Hongwu Peng
Zhiwei Zhang
Zhengyu Chen
Yue Guo
Venue: ICLR 2026 Poster (2026)
Keywords: ML System
Efficient Decoding
Primary area: foundation or frontier models, including LLMs

Abstract: Long-context inference in large language models is bottlenecked by Key--Value (KV) cache loading during the decoding stage, where the sequential nature of generation requires repeatedly transferring the KV cache from off-chip High-Bandwidth Memory (HBM) to on-chip Static Random-Access Memory (SRAM) at each step. While Multi-Head Latent Attention (MLA) significantly reduces the total KV cache size, it suffers from a sharding bottleneck during distributed decoding via Tensor Parallelism (TP). Since its single latent head cannot be partitioned, each device is forced to redundantly load the complete KV cache for every token, consuming excessive memory traffic and diminishing TP benefits like weight sharding. In this work, we propose Multi-Head Low-Rank Attention (MLRA), which enables partitionable latent states for efficient 4-way TP decoding. Extensive experiments show that MLRA achieves state-of-the-art perplexity and downstream task performance, while also delivering a 2.8$\times$ decoding speedup over MLA. Code is available at https://github.com/SongtaoLiu0823/MLRA. Pretrained weights, along with the training and evaluation data, are available at https://huggingface.co/Soughing/MLRA.

Published as a conference paper at ICLR 2026
MULTI-HEAD LOW-RANK ATTENTION
Songtao Liu Hongwu Peng*
Zhiwei Zhang'
Zhengyu Chen' Yue Guo
'The Pennsylvania State University
University of Connecticut *University of California, Los Angeles
CCanegie Mellon University
ABSTRACT
Long-context inference in large language models is bottlenecked by Key-Value (KV) cache loading during the decoding stage, where the sequential nature of generation requires repeatedly transferring the KV cache from off-chip High- Bandwidth Memory (HBM) to on-chip Static Random-Access Memory (SRAM) at each step. While Multi- Head Latent Attention (MLA) significantly reduces the total KV cache size, it suffers from sharding bottleneck during distributed decod- ing via Tensor Parallelism (TP). Since its single latent head cannot be partitioned, each device is forced redundantly load the complete KV cache for every to- ken, consuming excessive memory traffic and diminishing TP benefits like weight sharding. In this work, we propose Multi-Head Low-Rank Attention (MLRA), which enables partitionable latent states for efficient 4-way TP decoding. Exten- sive experiments show that MLRA achieves state-of-thh the-trr perplexity and down- stream task performance, while also delivering 2.8x x decoding speedup over MLA. Code is available at https: : h/tbttuu.bb../////////////.. com/ SongtacLiuuSe MLRA. Pretrained weights, along with the training and evaluation data, are avail- able at https: htggs///gggggiggac..e... co / h/t///uuuggn////ee MMRA.
INTRODUCTION
Inference-time scaling (OpenAl et al., 2024) is critical for large language models (LLMs) to pro- duce high-quality responses. Both long chain-of-thought (CoT) reasoning (Wei et al., 2022) rely on maintaining long context before generating the final answer, substantially increasing the number of tokens that must be processed at each decoding step. Sequential token generation under Multi- Head Attention (MHA) (Vaswani et al., 2017) requires reloading the Key-Value (KV) cache from bighhbandwidth memory every step, so data movement (Ivanov et al., 2021; Gholami et al., 2024), not computation, dominates latency for long-context inference (Sadhukhan et al., 2025). The small amount of compute per step relative to this data movement leads to poor GPU utilization (He & Zhai, 2024; Zadouri et al., 2025).
generation (RAG) (Lewis et al., 2020) and
A series of recent studies (Shazeer, 2019; Hu al., 2024; Zadouri al., 2025; Zhang et al., 2025; Zheng et al., 2025) have developed alternative attention mechanisms aimed at improving decoding efficiency and overall model quality. Multi-Head Latent Attention (MLA) (DeepSeek al., 2024c) compresses the KV cache into latent head (4.5d), per token). By absorbing the up projection ma- trices into the queries during decoding, it delivers better efficiency compared with MHA. However, MIA is unfriendly to tensor parallelism (TP) because its single latent head cannot be sharded. In this work, we address the limitation that MLA does not support TP.
We first show that partitioning the MLA latent head and the NoPE (Yang et al., 2025b) KV up- projection matrices into four blocks makes the NoPE key and value equivalent to the sum of four block-wise projections. Motivated by this insight, we propose Multi-Head Low-Rank Attention (MLRA), which explicitly decomposes the latent head into four latent heads, independently up- projects each latent head to form NoPE KV, and sums the resulting attention outputs. This design naturally supports 4-way TP and reduces the per-device KV cache loading. Based on our 2.9B scale experiments, MLRA-4 achieves the lowest perplexity (13.672 13.727 for MLA and 14. 133 for GQA) and highest zero-shot common-sense reasoning accuracy (58.84% vs. 58.75% for MLA and 57.89% for GQA). Our kernel delivers 105-1.26x speedup over GQA in long-context decoding.
*Correspondence to: Songtao Liu <sk15761 @psu.edu>..

---

Published as a conference paper at ICLR 2026
2 BACKGROUND
2.1 MULD LATENT ATTENTION
All notation used in this paper is summarized in Appendix A. Given a sequence of tokens with hidden states H E 1n MLA derives the query and KV states as follows:
C RMSNorm (HWD)), WDe E Beddd
OKK RMSNNrm (HWDKY), W-D E gdxd,
K RoPE = RoPE HW.R), W E Rdxd
where de: d' < hd, denote the dimensions for KV and query latent states, respectively. The learn- able down-projection matrices WDQ and DDKV produce the latent states Cº and CKV, while WKR generates the partial RoPE (Su et al., 2024) key, denoted K K Following DeepSeek- V3 (DeepSeek et al., 2024b), we set de = 4d, and di 0.5d, without loss of generality. Both CKV and RROEE are cached during inference to optimize efficiency. Finally, MLA derives the attention heads for queries, NoPE keys, and values through the following up-projections:
Q NoPE
RoPE
RoPE (CRWOR)
W (R(d() W E I==
K NoPE
WKK
V= C-WWW WLK WUV Rd.x(hda),
where Wit W and WiV denote the learnable up- projection matrices. To facilitate multi-head computation, the resulting queries, NoPE keys, and values are reshaped into head-wise tensors:
Q NaPF Reshape
Q
NoPE
(m, h, dnl).
Q RoPE:
Reshape
(Q RoPF
[m, h, dk]
K NoPE Reshape
K
NoPE
[n, h, dnl)
V Reshape (V, [n, h, dal).
where Q NoPE KNPE V E Ruxhxdd and Q QoPEE xx For head E {O...., h- 1}. we define
the head-specific 2D slices by indexing into the respective 3D tensors:
Q NoPE = NoPE [,, i, =], RoPE := QROPE [:, i, :], KNoPE. .. == NoPE.
[s, i, :], V.,.: := V|
To incorporate positional information, MLA shares common RoPE key K RoPE across all atten- tion heads. The final position aware query and key for head i are formed by concatenating their respective NoPE and RoPE components:
Q f.: Concat
((
NoPE
RoPE
dim=l), K...: Concat ([KNOPE
K RoPE
dim=l).
Efficient Decoding. Recall that MLA utilizes up- up-rrojection matrices WUK and WUV We extract the d, do-column slices for head i define its head-specific projections:
WUK WUK [., id, (i + 1)da). WUv
(() == Wvv [:, id, : (i 1)da).
:.)
We refer to the DeepSeek official inference implementation (DeepSeck et al., 2024b) to illustrate how to "absorb" up-projection matrices into the queries to avoid explicit KV materialization in MLA decoding. For the prefix sequence (O...., 1} with cached components CKV and K RoPE define the head-wise up-projections by partitioning WUK and WUV into h heads, (WUK 4-1
h-1 For the last prefix token at position and {W:UV -=0
1, let Qn,i, denote the query vector for the i-th attention head. To maintain variance during the dot- not-product operation, we apply the scaling and compute the attention output for the token at position - as follows:
factor
ddu+ t4R
O On-1,4 Softmax
TQ NoPE 1.2.
(CVV
) :.())
+ Q
On-ii (K (o-oPE))
RoPE
(c (uKW ((0)
Softmax
Q NoPE
WUUK ((())
(CKV) T
+ + 1 (KROPE))
RoPE
KV CUV 2(0))
6 I. ED
where O,-1,4: is the attention output for head i position 1. We next present three-step algorithm that leverages the associativity of matrix multiplication to avoid matcrializing the heads of NoPE keys and values, thereby optimizing the decoding efficiency.
2

---

Published as a conference paper at ICLR 2026
Step 1 (Query-Side Weight Absorption). We first reorganize the NoPE key and value up-
projection matrices into head- wise tensors:
UK
(WUk ((0))
E Rd
W
W :(())
E Rdxxd,
where W UK
E hxdd xde and W Rexde For the NoPE query at position 1, Q 1
NoPE
we absorb the up up-projection weight tensor W UK
directly into the query via Einstein summation:
NoPE n 1,p = einsum
"hp, hpc->he", h--le 1,0
NoPE
wh). p= dr, C c= das
UK)
n -- E R
NoPE
Step (MQA-Style Decoding on Latent KV Cache). Given the KV cache CKV and K KROPE define the shared key and value tensors by concatenating and reshaping the latent represen- tations as K = Reshape (Concat ( ([CKY, K KRoPE), dim=1), [n, 1, de + d E [xx and V Reshape (C [n, 1, del) Rnxixde Under this formulation, decoding reduces to an MtAestyle attention mechanism which the attention logits (i.c., query-key inner products before softmax) computed in (d. di ddimennional space using these shared KV states. Incorporat- NoPE
ing the concatenated query Q, - Concat (
1-1 dim=1)
RoPE
TR (x the
attention output is calculated as follows:
- Attention Qn-i, Repeatinterleave (K h, dim = 1). RepeatInterleave (V. dim
1)
(1)
where -.. E Redd FlastAttention-3 (Shah et al., 2024) and FlashMLA (Jiashi Li, 2025) provide highly optimized kernels designed to implement the Step-2 decoding computation directly.
Step 3 (Output Up-Projection). Finally, the up-projection tensor maps the intermediate attention output to the final attention output:
n- 1a: cinsum ('he, hep->hp", Zn-1. W ) c= de, p = dn. On-1a: Rhxds
Block Multiplications. For each head i, we define the constituent sub-blocks W (1))(0) E Rd.xd. by partitioning the up-projection matrices into dy -sized row blocks for b E {00 1, 3;: (b) == UK [bdr (b + 1)da, id,, (i+ 1)da], Uv [bd, (b + 1)d,, id, (i+ 1)d,].
UK
WUUV (0),0)
Consequently, each head-specific up- propoojection matrix can expressed as a vertical stack of these four row-blocks:
WUK WUk
Wivv W w
())
(
WUUK
wuk
WUV 500
(2)(0)
WUV (2)(0)
wuk
(3).())
(3)(6))
Similarly, we partition the KV latent matrix KV CKV [:, bdl, (6 + 1)da], such that CV
Roxde into horizontal channel blocks CV
CKY :(0)) :.00)....
CKV ((3))
This block decomposition allows
the key and value projections for head i to be reformulated as a sum of four sub-block products:
K NoPE (K).:
C.V WUk (()) (b),
V V.,6
Σ 6=0
KV Wuv (()) (b)(2))
2.2 GROUPED LATENT ATTENTION
Grouped Latent Attention (GLA-2) (Zadouri et al., 2025) bisects MLA's single latent head into two latent heads, using the first latent head (00) CKV .(11)) for the first half of attention heads and the CKY for the second half. We define the group-mapping function as:
second latent head (CkY
:(2)
(.(8)
[0, h/2,
7()) h i=i-
(1)
{0, .0.....h/ h/2 h/2-1) 1).
(3)
1,
2 h/2,
2
3

---

Published as a conference paper at ICLR 2026
Let U( W (2/2) denote the
up up-projection matrices for latent group
(()) E [0,1]. We extract the head-specific slices for head by indexing into these matrices:
Winl))
[. idz 1)da, (( w [&. id, (i + 1)da]
3(0)
:00)
where W (K))
W (00)) :(())
E Rdd To further facilitate block-wise computation, parti-
.(())
tion these slices into d, row blocks wla(a); for b E {0, 1]. defined as: (b),60)
W (1(0)u (6)())
W W(((), [bdy (b + 1)da, ida (2 1)da],
ll (M)(2)
w((0),U [bd, (6 1)da, id, (i + 1)da],
where each block W (U)))KKK
w (x()).v Rd. xda, This partitioning allows to decompose the
(t)
(b).(i)
head-specific up-projection matrices into two row-wise blocks:
W (x00)
(0) W wr(tt)).uv LVV
W (TO)
(0)
w 5,0
:00
(x (1).0)
W ( (1)(())
Consequently, the NoPE key and value computations for head i can be expressed as the summation of two block products:
NoPE (i).
CKK
W (K)) (0)()
+ +CV
W (nO))KK (0).0)
((2x(())
((22(0++1)
V. .0. = OKY
W (60)..V
(v()).VV
CKY
((2((0++1) W (1(0),V (1)(0)
(4)
((2))())
(0)(6)
3 MULTI-HEAD LOW-RANK ATTENTION
Building on the block decompositions in Sections 2.1 and 2.2, we propose MLRA. By shifting summation from KV computation to attention output, MLRA treats each block projection independent low-rank branch and sums their outputs. MLRA is illustrated in Figures 8 and 9.
3.1 MLRA-4
By substituting the block-paree paee-ppatitionedd identities from Eq. (2) into the attention mechanism, the output for head can be expressed as:
A, Softmax
TQ.
NoPE C.
Σ h=
CKY W ((b))
+ +
RoPE
(K
RoPE
CKY KY :.bb) (0)
WUU
(6)(6))
Motivated by Eq. (2), we propose MIRA-4, which computes attention independently on each block- wise branch and sums the resulting outputs:
3
Softmax
NoPE
(CKY,
Wirkt) (0).0)
+
RoPE
(KROPE)
7)
KV W...) (5)
b=u
3.2
MLRA-2
Following the grouping logic of GLA-2 from Eq. (3) and substituting the block-wise identities from Eq. (4), the attention output for GLA-2 can be expressed as:
O l.: Softmax
NoPE
KY (22)())
(r (0),0)
KV
((2((0+1 W (x(0)),
7 FA.
RoPE (K<RoPE) !)
(ONy W (Y (())))))
KY ())00)) (0)(0)
(V),()
Analogously, we derive MLRA-2 by moving the block summation outside the attention operator, yielding sum of two branchwise attention outputs:
... Softmax
Q
NoPE
(x00)C)) (0),6)
+
RoPE
K ROPE)
')
(-(), (0),00
+ Softmax
KV
((2((0+))
(x()). (1).6)
RoPE K RoPE)
")
(O (x0)CC
((22(()44)
(0,69
(6)
4

---

Published as a conference paper at ICLR 2026
MLRA-2 and MLRA-4 differ primarily in their latent-to-head mapping and branching factor. In MLRA-2, each latent block is up prpprooeeted to serve h/2 heads, with the final output resulting from a two-branch summation. Conversely, MLRA-4 utilizes four latent blocks that each up-projected to serve all h heads, resulting in a four-branch summation. Despite these structural differences, both variants decompose the computation into independent branches that require only final reduction. This architecture naturally facilitates 4-way TP decoding, reducing the per-head attention logit space 1.5dg after absorption-a significant reduction compared 4.5da, in MLA and 2.5da, in GLA-2.
3.3 SCALING QUERY/KEY-VALLE LATENT STATES AND ATTENTION OUTPUT
Recent work (LongCat et al., 2025) observes that the RoPE key (KRoPE) can exhibit significant
variance mismatch relative to other attention components (Q, KNOPE, v). This discrepancy arises
in MLA because RMSNorm (Zhang & Sennrich, 2019) is applied the latent states HWDD and HWV prior to the up-projections that generate the final query, NoPE key, and value tensors. To formally investigate this variance instability, we introduce the following assumption:
We assume that all elements of the weight matrices De WDKY, W W W WKR, and Wtv are independent and identically distributed (i.id.) random variables with zero mean and variance o²- Furthermore, these weight matrices are assumed to be mutually in- dependent of the input signals at each layer. Finally, for the multi-branch of MLRA, we assume the attention oulputs O (O), originating from different latent blocks b are mutually uncorrelated, implying that Cov (O (a): (0). Re 0 for all + b.
Assumption 1.
RoPE Key Variance. Recall that MLA computes the RoPE key as KROOPE RoPE (HWKR). Let RoPE = (HWKR),.. )t,a
Since the hidden states H are RMS- normalized, (H(,,, Re 1. With E WKR = and E (WKR) m.u =
Ed Htm WKR m.u
de we have
m,a.
d
Var
K-RoPte f
Σ
[E[(H,,m)) ] E (WWKR (WWKR)) - (E E
1.02 do?
m.u
m==1
m==1
Since RoPE is an orthogonal transformation, it does not change the variance:
Var K RoPE)
do
NoPE Key Variance. Next, we derive the variance of the NoPE keys, K NoPE By definition, the latent KV state CKV RMSNNrm H HDDK) is constrained such that each
CKV UUK
element has approximately unit mean square, i.e., E (CKY)²] 1. Considering arbitrary entry KN
Eie - CKY wit and assuming the weights UKK are zero-mean with variance o the variance of the product is:
de = Σ 1=1
de 1.0 = doo m
Var
K
NoPE f.u
(B[(CKY)T E -
E CE E(W.k1)))
(8)
Because reshaping does not alter the underlying variance, Var K NoPE this derivation the remaining attention components, obtain the variances for the value and query tensors as follows:
remains d.o Extending
Var d Var
Q NoPE
d'o2
Var Q RoPF
d=o0²
Variance Mismatch and Calibration. Comparing our variance derivations shows that Var( (RROEE))
RJ d which explains the mismatch noted in LongCat et al. (2025) when the latent dimension de is much smaller than d. This is corrected by applying scaling factors to the latent states before the up-projection. Specifically, using ag = /d/d' and ogr = Vd/d, ensures that the Q, K KNOPF) achieve parity with the variance of the RoPE key.
Var( KNoPE)
query and NoPE key components
5

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of parameters and KV cache loading among attention mechanisms. Some results are taken from Zhang et al. (2025) and Zadouri et al. (2025). For attention mechanism details, refer to Appendix C.
Method
Purameters
Loading Per Token Dovice
Loading Por Duvics GP
Por Device GPC)s)
Loading Per
Loodingg Per Por Devier GPU))
MIHA Veeeomi et 2017)
Adhol.
Rld.
1986.
6/6.
16vl.
MQA (Stuzeer. 2019)
2dd, th
24, 2pds
26,
2d.
2/t
25.
GQA (Aistle et al. 2023)
2dha (h gt
164s
5ds
MIA (DoopSock at. 3014g)
d. (d 2hdts) didds
(dt hol + idf) + dol
4 4
4.5d
1.54
4.5%%
1.54%
MFA (ffu al., 2024)
463 28. (In du) 261
fel4 lo 2dh,)
2d,
(Zhang ctall (005)
Aldss +. + huff) dal shull
18,, (f++ +-6)
54,
4.5c
CLA-2 (Zalouri et at., 2025)
4.5ds
2.4
2.5d
2.5d
CTA (Zaloui et 2025)
didd, dlads daff dird,
sel +:4g
8.5d,
4.54.
2.5d
1.54
(uf)) dol shat, 4 haf)) del 4.4, dids
4.3d
2.54
1.5
1.5%
MIRA-4
4. 4.440
4.5d.
2.54.
1.5%
1.56
Adopting the variance-calibration strategy from LongCat et al. (2025), we apply analogous rescaling to our MLRA variants. For MLRA-2 and MLRA-4, we rescale the query and KV latent states to ensure that the variance of NoPE queries and keys aligns with that of the partial RoPE components across all branches.
d de
4d Vo
aO
CO
CKV
OKY.
Since summing the attention outputs from multiple branches alters the variance, apply rescaling factor to the attention outputs of MLRA-2 and MLRA-4 as follows:
1
(MLRA-4) O...: 1 O .AA.:
(MLRA-2) O..,,
O
(10)
V2
Remark 1. Although these weight matrices are typically initialized with zero mean and a common variance, these conditions are not guaranteed during training. Consequently, Assumption may not strictly hold in practice. However, the effectiveness of this scaling is best assessed through ablation studies, the results of which are detailed in Section 4.2.2.
3.4 ANALYSIS
KV Cache. We evaluate the per-device KV cache loading under various TP configurations using Qwen3-32B (Yang et al., 2025a) and Kimi-K2 (Team et al., 2025) as base architectures. Qwen3- 32B utilizes GQA with 64 query heads and 8 KV heads (d,, = 128), setting g = 8 KV heads. Kimi-K2 adopts MLA with 64 heads and a partial RoPE dimension (d 64). For TPA, we maintain the original configuration with Bko 2. Table summarizes the per-device KV cache loading under TP as the number of devices increases. To support TP, the official MIA decoding implementation, FlashMLA (Jiashi Li, 2025), distributes up-projection matrices across devices by head. However, this approach leads to redundant KV cache loading; as a result, the per-device loading remains constant at 4.5d, regardless of the TP degree. TPA constructs its /t key-value heads as linear combinations of 8&u shared heads. It supports TP only for the combination coefficients, while the shared heads must be redundantly loaded by each device. Consequently, the per-device KV cache loading is 4d, + 2d., where &o denotes the number of TP devices. GLA-2 partially addresses this by partitioning the latent head into two smaller latent heads, reducing the per-device loading to 2.5d, under 2-way TP. Notably, for MLA with TP > 1 and GLA-2 with TP > 2, the KV cache loading becomes invariant to the number of devices due to sharding constraints, causing the per-device loading to plateau at 4.5d,, and 2.5d,, respectively. While GQA and GTA require 8-way TP to reduce the per-device loading to 2d, and 1.5d,, MLRA achieves 1.5dh with only 4-way TP.
Attention Decoding Arithmetic Intensity. Arithmetic intensity (AD) (Williams et al., 2009), de- fined as the ratio of floating-poin operations to memory access (FLOPs/byte), serves as a criti- cal metric for identifying whether a workload is memory-bound or compute-boun (Zadouri et al., 2025). Given that the context length n is the dominant factor in long-context decoding, we eval- uate the arithmetic intensity (Al) of various attention mechanisms, with the results summarized in

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of attention decoding arithmetic intensity among attention mechanisms.
Method
MHA MQA GQA
MLA
MFA
TPA
GLA-2
GTA
MLRA-2
MLRA-4
Arithmettc Inteusity
Amddi amnd.
Ambui
dmbds. angdh
dnk 2d
Sud di, Lendfdd
2oth
inbal (+4f))
Ancids in(po
2nd 4
what
ands.
2n(4.-0]
4n 246
4n3 Tel.)
Znl (+4)))
R2 2h
R h
2
(149 Aholeedss
R h
Ri 2h
Table 2. MLRA-2 and MLRA-4 achieve AI values of h and 2h, respectively, maintaining the high arithmetic intensity characteristic of MLA and GLA-2. By significantly increasing the compute-to- to- memory ratio, MLRA shifts the decoding process away from the HBM bandwidth ceiling toward compute-limited regime.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Model Configuration. We adopt the Llama-3 (Llama et al., 2024) architecture (Appendix D) and compare MLRA against the following attention mechanism baselines: MHA (Vaswani al., 2017), MQA (Shazeer, 2019), GQA (Ainslie et al., 2023), MLA (DeepSeek et al., 2024c), MFA (Hu et al., 2024), TPA (Zhang et al., 2025), GLA-2 (Zadouri al., 2025), GLA-4, and GTA (Zadouri al., 2025). GLA-4 compresses the KV cache into four latent heads. We initialize the MHA baseline with the Llama3.2-33 (Llama et al., 2024) configuration and use it as our parameter-coum refer- ence. Following Zadouri et al. (2025), for each other attention variant, we adjust the Network (FFN) intermediate dimension to match the total number of parameters of this MHA base- line. Full details of the architectural hyperparameters are provided in Appendix F.1. All models are implemented on top of the nanoGPT (Karpathy, 2022) codebase.
Pretraining Configuration. We pretrain all models at the --B-paraeettr scale on FineWeb-Edie 100B (Penedo et al., 2024). Each model is pretrained from scratch on 98.3B tokens, with an ad- ditional 0. tokens for validation. We use the GPT-2 tokenizer with a vocabulary size of 50,304 and follow the standard GPT-3 pretraining setup. We use Adam (Loshchilov & Hutter, 2019) as the optimizer with (3,, (22 = (0.9, 0.95), c 10-8, weight decay 0.1, and gradient clipping at for the first 2,000 steps. then annealed with cosine decay (Loshchilov Hutter, 2017) to 10% of the peak. Peak learning rate is 1.6 x 10-4, We train with context length of 2,048 tokens and global batch size of 480 sequences 1983,040 tokens per step, 1.0M) for 100,000 steps. All models are pretrained on 8 NVIDIA H100 80GB GPUs.
1.0. The learning rate is linearly warmed
Evaluation Benchmark. In addition to evaluating the perplexity from the Fine vali- dation dataset, we evaluate our models on six additional datasets: Wikipedia, C4 (Raffel et al., 2020), the Pile (Gao et al., 2020), RefinedWeb (Penedo et al., 2023), Cosmopedia (Ben Allal al., 2024), and FineWeb (Penedo et al., 2024) using O.IB tokens per dataset. We evaluate zero-shot performance on common-sense reasoming benchmarks, including ARC-Easy (ARC-E) (Yadav et al., 2019), ARC-Challenge (ARC-C), OpenBookQA (Mihaylov et al., 2018), BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019), Winogrande (Sakaguchi et al., 2021), and PIQA (Bisk et al., 2020), using the an--vvaluation-harreee racy for ARC-E/C, Open HellaSwag, and PIQA, with standard accuracy for all other tasks.
(Gao al., 2024) package. We report normalized accu-
4.2 PRELIMINARY ABLATION RESULTS
4.2.1 INITIALIZATION
We follow the GPT (Radford et al., 2018) initialization method, where all model weights are ini- tialized using an (V a = 0.02) distribution. However, TPA employs zero initialization for the output projection parameters of the attention and FFN modules, which is an approach also explored in muP (Yang et al., 2021) and LoRA (Hu et al., 2022). To evaluate these different approaches, conduct an ablation study comparing zero initialization against (000 projection parameters across all models. It important to note that for MLA, GLA-2, and GLA-4, apply scaling as discussed in Section 3.3. As illustrated in Figure 1 and Table 38, the results for
0.02) for the output
7

---

Published as a conference paper at ICLR 2026
Initialization Ablation Study
Scale Ablation Study
0.09
0.01
MIDA
MI.
6QAA
0.008
Loss difference
0.07
MLA
Loss difference
MFA
0.006
0.05
CTA
0.004
0.03
0.002
0.01
0.0
20
40
60
80
100
0
20
40
60
80
100
Training tokens (B)
Training tokens (B)
Figure Loss difference between (O 0.02) and zero initialization, calculated by sub- tracting the loss of the latter from the former.
Figure Loss difference between models with- out and with scaling, calculated by subtracting the loss of the latter from the former.
Double Heads Ablation Study
Gating Ablation Study
0.01
0.018
Loss difference
0.008
Loss difference
0.015
MLKA-2
MLRA-
0.006
0.012
0.004
0.009
mmma
0.002
0.006
0.0
0.003
20
40
60
80
100
0
20
40
60
80
100
Training tokens (B)
Training tokens (B)
Figure 3: Loss difference between models with and without double heads, calculated by sub- tracting the loss of the latter from the former.
Figure Loss difference between models with- out and with gating, calculated by subtracting the loss of the latter from the former.
loss and perplexity demonstrate that zero initialization outperforms the \o0 0.02) distribution. Unless otherwise specified, all models in the following experiments utilize this zero initialization.
4.2.2 SCALING
We evaluate the effectiveness of the scaling on MLA, GLA-2, and MIRA-2. As illustrated in Figure 2, all three models exhibit improved convergence when scaling is applied. As shown in Table 39, all three models achieve lower average perplexity after scaling. Notably, MLA and GLA- 2 show more substantial improvements, while MLRA-2 yields a marginal gain. Unless otherwise specified, MLA, GLA, and MLRA in the following experiments utilize this scaling.
4.2.3 DOUBLE HEADS
While MLRA-2 and MLRA-4 do not increase the number of query heads, their multi-brance de- sign increases the number of attention heads involved in computation. Consequently, we double the number of attention heads for GQA, MLA, and GLA-2 while keeping the KV-cache size fixed to evaluate whether this increase contributes to performance gains. To maintain constant parameter budget during this adjustment, we reduce the FFN intermediate sizes; the corresponding architec- tural hyperparameters are detailed in Appendix F.4. As illustrated by the loss curves in Figure and the results in Table 40, doubling the number of attention heads leads to higher loss and fails to decrease perplexity across all three models. These findings suggest that doubling the number of at- tention heads does not yield any measurable performance improvement. Unless otherwise specified, GQA, MLA, and GLA use the default head count (no head doubling).
8

---

Published as a conference paper at ICLR 2026
Table Validation perplexity (lower is better) across seven datasets: Wikipedia, C4, Pile, Refined- Web, Cosmopedia, FineWeb, and Fine Web-Edu. The best results are indicated in bold, while the second best are underlined.
Method
Wikipedia
C4
Pile
RefinedWeb
Cosmopedia
FineWeb
Fib--eb-Edu
Avg
MHA
14,624
16.575
12.929
18.698
9.102
15.656
9,434
13,860
MQA
15.134
16.837
14,008
19.202
9.484
15.942
9.533
14.306
GQA
15.057
16.628
13.758
18.885
9.504
15.713
9,427
14.139
MLA
14.567
16.345
12.965
18.523
8,966
15,440
9.284
13.727
MFA
15.693
16.738
13,903
19.125
9,423
15.815
9.506
14.315
TPA
14.789
16.622
13.333
18.971
9.130
15.717
9.333
13.985
GLA-2
14.605
16,323
13.225
18,509
9.118
15.424
9.249
13.779
GLA-4
14.547
16.436
13.229
18.578
9.076
15.535
9.307
13.815
GTA
14.733
16.599
13.402
18.924
9.129
15.672
9.346
13.972
MLRA-2
14.615
16.342
13.236
18. 68,602
9.153
15,439
9.242
13.804
MLRA-4
14.407
16.286
13.124
18.398
8.937
15.361
9.193
13.672
Table 4: Downstream evaluation on seven common-senss reasoning benchmarks: ARC-E, ARC-C, OpenBookQA, BoolQ, HellaSwag, Winogrande, and PIQA. ARC-E/C, OpenBookQA, HellaSwag, and PIQA use normalized accuracy (%); others use standard accuracy (%). Best is bold; second best underlined.
Method
ARC-E
ARC-C
OpenBookQA
BoolQ
HellaSwag
Winogrande
PIQA
Avg
MHA
69.11
40.80
62.26
74.86
MQA
66.16
38.31
41.80
62.05
60.24
59.83
74.48
57.55
GQA
67.13
39.42
42.00
63.39
61.29
56.91
75.08
57.89
MLA
68.22
39.16
42,60
64.10
61.39
60.06
75.68
58.75
MFA
69.02
39,93
42,40
63.49
60.72
58.96
75.19
58.53
TPA
69.44
40.61
41.60
60.03
61.02
57.85
74.54
57.87
GLA-2
68.01
40.19
40.60
63.94
61.54
58.41
75.41
58.30
GLA-4
68.77
41.04
41.20
61.96
61.61
58.09
74.65
58.19
GTA
67.97
39,68
42.60
59.72
61.03
75.14
MIRA-2
67.89
42,24
42.00
61.65
61.49
59.98
75.52
58.68
MLRA-4
67.63
41.38
43.00
61.74
62.16
61.48
74.48
58.84
4.3
MAIN RESULTS
As shown in Table 3, MLRA-4 achieves the best average perplexity (13.672), outperforming all other models, including MLA (13.727). Notably, MLRÅ-4 also delivers the lowest perplexity FineWeb-BBB -du (9.193). Furthermore, Table demonstrates that MLRA-4 attains the highest average zero-shot accuracy across all common-sense reasoning tasks. This consistent superiority of MI.RA over MI M-- across both evaluations highlights the benefits of increasing the number of branches.
4.4 GATED ATTENTION
Following Qiu al. (2025), we introduce gating mechanism prior to the attention output projec- tion (Appendix E). To maintain a constant parameter budget, reduce the FFN intermediate size accordingly; detailed architectural hyperparameters are provided in Appendix F.5. As illustrated in Figure 4, all five models exhibit improved convergence with gating applicd. As shown in Table 5, gating consistently improves perplexity across all models, with MLRA-4 achieving the best overall average perplexity and MLRA-2 attaining performance comparable MLA (13.651 vs. 13.642).
4.5 DECODING EFFICIENCY
Decoding Speed. We benchmark long-conted attention decoding speed for GQA, MLA, GLA-2, and MLRA-4 on an NVIDIA H100 80GB GPU. For MLA, GLA-2, 2, and MLRA-4, we follow the at- tention decoding formulation in Eq. (1). All models use 64 heads with a head dimension of 128; for MLA, GLA-2, and MLRA-4, the partial RoPE dimension 64. MLA is evaluated using DeepSeek's official implementation FlashMLA (Jiashi Li, 2025). GQA and GLA-2 nels (Dao et al., 2022; Dao. 2024; Shah et al., 2024). implement our MLRA-4 kernel based on 3 We evaluate decoding speed across sequence lengths from 131,07 2,097,155
FlashAttention-3 ker-

---

Published as a conference paper at ICLR 2026
Table 5: Validation perplexity (lower is better) w/ gating across seven datasets. The best results are indicated in bold, while the second best are underlined.
Method
Wikipedia
C4
Pile
Wef
Cosmopedia
FineWeb
Fib--eb-Eduu
Avg
GQA w/ gating
14.362
16.484
13.113
18.696
9.098
15.581
9,311
13.806
MLA w/ gating
14.346
16.297
12.866
18.456
8,936
15,383
9.212
13.642
GLA-2 w/ gating
14.597
16.286
12.997
18.473
8,986
15,369
9.198
13.701
MLRA-2 w/ gating
14.424
16.252
13.017
18.407
8.924
15.351
9,180
13.651
-1 gating
14.431
16.170
13.073
18.386
8.874
15.266
9.148
13.621
Decoding Latency
Decoding Throughput
900
GQA (TP-8)
Throughput (tokens /ms)
GQA (TB-5)
750
MLA (DP)
(IP-2
GLA-22 Desa)
Latency (Uu)
600
(TP=4)
600
300
300
150
256
1024
2048
4
10
12
16
Sequence Length (K)
Sequence Length (K)
Figure 5: Decoding latency (lower is better) ver- sequence length (batch=1) for GQA, MLA, GLA-2, and MIRA-4.
Figure 6: Decoding throughput versus sequence length (batch=128) for GQA, MLA, GLA-2, and MIRA-4.
tokens (128K-2M), As shown Figure 5, MLRA-4 consistently outperforms all baselines at every length, yielding 1.05x-1.26x speedups over GQA. The gap grows with context length against GQA and GLA-2, while the speedup over MLA remains steady at about 2.8x, indicating that MLRA-4 with TP=4 substantially reduces long- long-context decoding latency.
Decoding Throughput. We evaluate decoding throughput for GQA, MLA, GLA-2, and MLRA-4 on eight NVIDIA H100 80GB GPUs, fixing the number of attention heads to 128 and the hidden size to 7168, following DeepSeekVs V3 (DeepSeek et al., 2024a). We set = 16 for GQA. For MLA decoding deployment, there trade-off between data parallelism (DP) and tensor parallelism. With DP, we assign different requests to different devices, so attention parameters are replicated across devices and the load can become imbalanced due to varying sequence lengths. With TP, the up-projection parameters are sharded by head, but the KV cache loading is repeated across de- vices. Following SGLang (Zheng et al., 2024), we aim to climinate redundant KV cache loading. Therefore, we use DP=8 for MLA, TP=2/DP-4 for GLA-2, TP-4/DP-2 for MLRA-4, and TP-8 for GQA. Throughput reported for sequence lengths ranging from 1,024 to 16,384 tokens, and end-to-end measurements include both the pre-attention stage that prepares inputs for the at- tention kernel and the attention computation itself. We accelerate pre-attention computation with torch.compile (Paszke al., 2019) for MLA, GLA-2, and MLRA-4, and with custom Triton ker- nels for GQA. As shown in Figure 6, MLRA-4 achieves the highest decoding throughput across both short and long sequence lengths. This suggests that MLRA-4 with TP=4/DP=2 reduces pa- rameter redundancy relative to MLA's DP=8, while introducing only modest partial RoPE duplica- tion, thereby yielding higher throughput than MIA. For short sequences, GQA outperforms MILA and GLA-2 because pre-attention dominates latency. However, MLRA-4 remains competitive with GQA due to having even fewer query, key, and value parameters, as shown in Appendix F.I.
CONCLUSION
We propose Multi-Head Low-Rank Attention (MLRA), novel attention mechanism with native 4-way tensor parallelism support. At the 2.9B scale, MLRA-4 achieves state-of-the-arr performance on perplexity and zero-shot common-sense reasoning benchmarks. Furthermore, MIRA achieves the lowest decoding latency for long-context sequences (up 2M tokens) and the highest throughput across sequence lengths from IK to 16K tokens with 4-way tensor parallelism.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
We thank Songlin Yang for helpful discussion. We thank all the anonymous reviewers for their helpful comments and suggestions.
REFERENCES
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head check- points. In Empirical Methods in Natural Language Processing. 2023.
Sotiris Anagnostidis, Dario Pavllo, Luca Biggio, Lorenzo Noci, Aurelien Lucchi, and Thomas Hof- mann. Dynamic context pruning for efficient and interpretable autoregressive transformers. In Advances in Neural Information Processing Systems, 2023.
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. In Advances in Neural Information Processing Systems, 2024.
Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Cos- mopedia. https: htt.ghhgggggggcecececc/oo co datattttssssssseee////aeaas H/wgg/gggggggggggee aceeTceeT TB/ cosmopedia, 2024.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense natural language. In AAAI Conference on Artificial Intelligence, 2020.
Ruisi Cai, Yuandong Tian, Zhangyang Wang, and Beidi Chen. Lococo: Dropping in convolutions for long context compression. In International Conference Machine Learning, 2024.
Chi-Chih Chang, Wei-Cheng Lin, Yh Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S. Abdelfattah, and Kai-Chiang Wu. Palu: KV-cache compression with low-rank projection. In International Conference on Learning Representations, 2025.
Renze Chen, Zhuofeng Wang, Beiquan Cao, Tong Wu, Size Zheng, Xiuhong Li, Xuechao Wei, Shengen Yan, Meng Li, and Yun Liang. Arkvale: Efficient generative LLM inference with re- callable key-value eviction. In Advances in Neural Information Processing Systems, 2024a.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Lon- gloRA: Efficient fine-tuning of long-context large language models. In International Conference on Learning Representations, 2024b.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In North American Association for Computational Linguistics, 2019.
Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In Inter- national Conference Learning Representations, 2024.
Tri Dao and Albert Gu. Transformers SSMs: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning, 2024.
Tri Dao, Daniel Fu, Stefano Ermon, Atri Rudra, and Christopher Re. Flashattention: Fast and memory-efficient exact attention with 1O-awareness. In Advances in Neural Information Process- ing Systems, 2022.
DeepSeek et al. Deepseek-v3 technical report. arXiv preprint
2024a.
DeepSeck et al. Decpscek-v3 technical report. DeepSeek- 2024b.
DeepSeek et al. Deepscek-v2: strong, economical, and efficient mixture-of-experts language model. arXiv preprint 4rXiv:2405.0444 2024c.

---

Published as a conference paper at ICLR 2026
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lmms. In Advances in Neural Information Processing Systems, 2023.
Rahul Dey and Fathi M Salem. Gate-variants of gated recurrent unit (gru) neural networks. In International Midwest Symposium on Circuits and Systems, 2017.
Harry Dong, Xinyu Yang, Zhenyu Zhang, Zhangyang Wang, Yuejie Chi, and Beidi Chen. Get more with less: Synthesizing recurrence with kv cache compression for efficient Ilm inference. In International Conference Machine Learning, 2024.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se trantrfffsssssss: 3d roto- translation equivariant attention networks. In Advances in Neural Information Processing Sys- tems, 2020.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint 1eXiv:2101000000 2020.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter. Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 2024.
Suyu Ge. Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for LLMs. In International Conference on Learning Representations, 2024.
Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W Mahoney, and Kurt Keutzer. Ai and memory wall. IEEE Micro, 2024.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In Conference on Language Modeling, 2024.
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured
state spaces. In International Conference on Learning Representations, 2022.
Jiaao He and Jidong Zhai. Fastdecode: High-throughput gpu-efficient Ilm serving using heteroge- neous pipelines. arXiv preprint arXiv:2403. 2024.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.
Coleman Richard Charles Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In Advances in Neural Information Processing Systems, 2024.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022.
Jingeheng Hu, Houyi Li, Yinmin Zhang, Zili Wang, Shuigeng Zhou, Xiangyu Zhang, Shum, and Daxin Jiang. Multi-matrix factorization attention. arXiv preprint arXiv:2412.1925: 2024.
Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun, Shigang Li, and Torsten Hoefler. Data movement is all you need: A case study optimizing transformers. In Machine Learning and Systems. 2021.
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin- Yew Lin, Yuqing Yang, and Lili Qiu. MInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse attention. In Advances in Neural Information Processing Systems, 2024.
12

---

Published as a conference paper at ICLR 2026
Shengyu Liu Jiashi Li. Flashmla: Efficient mla decoding kemels.. deepseek-ai/Flasttee 2025.
https://githii com/
Andrej Karpathy. Nanogpt.
2022.
Angelos Katharopoulos, Apoory Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In International Conference on Ma- chine Learning, 2020.
Jang-Hyun Kim, Junyoung Yeom, Sangdoo Yun, and Hyun Oh Song, Compressed context memory for online language model interaction. In International Conference on Learning Representations, 2024.
Woosuk Kwon, Zhuohan Li. Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Symposium on Operating Systems Principles, 2023.
Patrick Lewis. Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. aeuteetal- generation for knowledge-intensive nlp tasks. In Advances Neural Information Processing Systems, 2020.
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. In Advances in Neural Information Processing Systems, 2024.
Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. ReloRA: High-rank training through low-rank updates. In International Conference on Learning Representations, 2024.
Chi-Heng Lin, Shangqian Gao, James Seale Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. ModeGPT: Modular decomposition for large language model compression. In International Conference on Learning Representations, 2025.
Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang. Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan. Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang, Cachegen: Kv cache compression and streaming for fast large language model serving. In ACM Special Interest Group Data Communication, 2024a.
Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Vietor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression test time. In Advances Neural Information Processing Systems, 2023.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-fres asymmetric 2bit quantization for KV cache. Interna- tional Conference on Machine Learning, 2024b.
Llama et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.,
Meituan LongCat et al. Longcat-flase flash technical report. arXiv preprint arXlv:2509.01322, 2025.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In Inter- national Conference on Learning Representations, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. kernel-based view of language model fine-tuning. In International Conference on Machine Learning, 2023,
Fanxu Meng, Pingzhi Tang, Xiaojuan Tang, Zengwei Yao, Xing Sun, and Muhan Zhang Transmla: Multi-heas latent attention is all you need. Advances Neural Information Processing Sys- fems, 2025.
13

---

Published as a conference paper at ICLR 2026
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. In Empirical Methods in Natural Language Processing, 2018.
Piotr Nawrot, Adrian Lañcucki, Marcin Chochowski, David Tarjan, and Edoardo Ponti. Dynamic memory compression: Retrofitting LLMs for accelerated inference. In International Conference on Machine Learning, 2024.
OpenAl et al. Openai ol system card. arXiv preprint arXiv:2412.16720 2024.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zachary Detito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, 2019.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refined- web dataset for falcon LLM: Outperforming curated corpora with web data only. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
Bo Peng, Daniel Goldstein, Quentin Gregory Anthony, Alon Albalak, Eric Alcaide, Stella Bider- man, Eugene Cheah, Teddy Ferdinan, Kranthi Kiran GV, Haowen Hou, Satyapriya Krishna, Ronald McClelland Jr., Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song, Hao- qin Tu, Ruichong Zhang, Bingchen Zhao, Qihang Zhao, Jian Zhu, and Rui-Jie Zhu. Fagle and finch: RWKV with matrix-valued states and dynamic recurrence. In Conference on Language Modeling, 2024.
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. Random feature attention. In International Conference on Learning Representations, 2021.
Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for se- quence modeling. In Advances in Neural Information Processing Systems, 2023.
Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-frre. In Advances in Neural Information Processing Systems, 2025.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. OpenAl Technical Report, 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2020.
Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian En- Hsu Yen, Avner May. Tianqi Chen, and Beidi Chen. Magicdec: Breaking the latency- throuuuuu tradeoff for long context generation with speculative decoding. In International Conference on Learning Representations, 2025.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 2021.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural net- works. In International Conference on Machine Learning, 2021.
Imanol Schlag, Kazuki Irie, and Jürgen Schnidhuber. Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning, 2021.
14

---

Published as a conference paper at ICLR 2026
Jay Shah, Ganesh Bikshandi. Ying Zhang. Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-:: Fast and accurate attention with asynchrony and low-precision In Advances in Neural Information Processing Systems, 2024.
Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint
arXiv:1911.02156 2019.
Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. In International Conference on Learning Representations, 2023.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv:1505.003887 2015.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 2024.
Hanshi Sun, Li-Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. ShadowKV: KV cache in shadows for high-throughput long-context LLM inference. In International Conference on Machine Learming. 2025.
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang. and Furu Wei. Retentive network: A successor transformer for large language models. arXiv preprint (.Xiv:237.006622 2023.
Yutao Sun, Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures for language models. In Advances in Neural Information Processing Systems, 2024.
Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. QUEST:
Query-aware sparsity for efficient long-context LLM inference. In International Conference Machine Learning, 2024.
Xiaojuan Tang, Fanxu Meng, Pingzhi Tang, Yuxuan Wang, Di Yin, Xing Sun, and Muhan Zhang. Tpla: Tensor parallel latent attention for efficient disaggregated prefill & decode inference. In International Conference on Architectural Support for Programming Languages and Operating Systems, 2025.
Kimi Team et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534 2025.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Information Processing Systems, 2017.
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. SVD-LLM: Truncation- aware singular value decomposition for large language model compression. In International Conference on Learning Representations, 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thor of-in-of-thouugt prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. Advances in Neural Infor- mation Processing Systems, 2018
Samuel Williams, Andrew Waterman, and David Patterson. Roofline: an insightful visual perfor- mance model for multicore architectures. Communications of the ACM, 2009.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations, 2024.
Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, junxian guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Ducattention: Efficient long-contex. LLM inference with retrieval and streaming heads. In International Conference Learning Representations, 2025.
15

---

Published as a conference paper at ICLR 2026
Vikas Yadav, Steven Bethard, and Mihai Surdeanu. Quick and (not so) dirty: Unsupervised selection of justification sentences for multi-hop question answering. In Empirical Methods in Natural Language Processing, 2019.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang. Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.08 09388, 2025a.
Bowen Yang, Bharat Venkitesh, Dwaraknath Gnaneshwar, Hangyu Lin, David Cairuz, Phil Blun- som, and Acyr Locatelli. Rope to nope and back again: A new hybrid attention strategy. In Advances in Neural Information Processing Systems, 2025b.
Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. In Advances in Neural Information Processing Systems, 2021.
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficien training. In International Conference on Machine Learning, 2024a.
Songlin Yang. Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear trans- formers with the delta rule over sequence length. Advances in Neural Information Processing Systems, 2024b.
Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. In International Conference on Learning Representations, 2025c.
Ted Zadouri, Hubert Strauss, and Tri Dao. Hardware-efficiet attention for fast decoding. Con- ference Language Modeling, 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? Association for Computational Linguistics, 2019.
Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation. In International Conference on Learning Representations, 2024.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances in Neural Information Processing Systems, 2019.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficien fine-tuning. In International Con- ference on Learning Representations, 2023a.
Yifan Zhang, Yifeng Liu, Huizhuo Yuan, Zhen Qin, Yang Yuan, Quanquan Gu, and Andrew Chi- Chih Yao. Tensor product attention is all you need. Advances in Neural Information Processing Systems, 2025.
Yu Zhang, Songlin Yang, Rui-Jie Zhu, Yue Zhang, Leyang Cui, Yiqiao Wang, Bolun Wang, Freda Shi, Bailin Wang, Wei Bi, Peng Zhou, and Guohong Fu. Gated slot attention for efficient linear- time sequence modeling. In Advances in Neural Information Processing Systems, 2024a.
Yuxin Zhang. Yuxuan Du, Gen Luo, Yunshan Zhong, Zhenyu Zhang, Shiwei Liu, and Rongrong Ji. Cam Cache merging for memory-efficient LLMs inference. In International Conference on Machine Learning, 2024b.
16

---

Published as a conference paper at ICLR 2026
Zhenyu Zhang, Ying Sheng. Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Re, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2o: Heavy- hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems, 2023b.
Chuanyang Zheng, Jiankai Sun, Yihang Gao, Yuehao Wang, Peihao Wang, Jing Xiong, Liliang Ren, Hao Cheng. Janardhan Kulkarni. Yelong Shen, Atlas Wang, Mac Schwager, Anderson Schnei- der, Xiaodong Liu, and Jianfeng Gao. Sas: Simulated attention score. In Advances in Neural Information Processing Systems, 2025.
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao. Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, 2024.
Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez De Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi. Mikhail Yurochkin, and Justin Solomon. Asymmetry in low-rank adapters of foundation models. In International Conference on Machine Learning, 2024.
17

---

Published as a conference paper at ICLR 2026
Appendix
Notation
19
Theorem
19
B.1 Translation Equivariance
19
B.2 Rotary Position Embedding
20
C Attention Mechanism
20
C.1 Multi-Head Attention (MHA)..
20
C.2 Multi-Query Attention (MQA) and Grouped-Quest Attention (GQA)
21
C,3 Multi-Head Latent Attention (MLA)
C.4 Multi-matrix Factorization Attention (MFA)
22
C.5 Tensor Product Attention (TPA)
23
C.6 Grouped Latent Attention (GLA)
24
C.7 Grouped- -ied Attention (GTA)
24
D Llama-3 Architecture
25
Gated Attention
25
Architectural Hyperparameters
26
F.1
Architectural Hyperparameters for Main Results
26
F.2
Architectural Hyperparameters for Initialization Ablation Study
27
F3
Architectural Hyperparameters for Scaling Ablation Study
28
F.4
Architectural Hyperparameters for Double Heads Ablation Study Study............
29
F55
Architectural Hyperparameters for Gated Attention Study
29
Additional Experimental Results
30
H
Illustration
31
I
Related Work
31
18

---

Published as a conference paper at ICLR 2026
A
NOTATION
Table 6: Notation used throughout this paper.
Symbol
Shape
Type
Meaning
scalar
Sequence length (number of tokens).
d
scalar
Model/hidden dimension.
h
scalar
Number of attention heads.
du
scalar
Per-head dimension.
di
scalar
Partial RoPE dimension.
dr
scalar
RoPE rotation dimension in Theorem 1 (even); typically d. = dy, or d, di.
d
scalar
MLP intermediate (FFN) dimension.
g
scalar
Number of groups (or KV heads in MQA/GQA).
scalar
Repeat factor h/g when KV heads broadcast ht query heads.
Ogyy Olky, Olaten
scalar
Variance-calibrating rescaling factors for query/KV latents and at- tention outputs.
Bgy Bkv
scalar
Number of low-rank components in TPA for queries keys-values.
de
scalar
Latent KV dimension MLA/GLA.
d'
scalar
Latent Q dimension MLA/GLA/MAA.
integer
Translation offset.
by. iu
integer
Query/key token positions.
b
integer
Block index.
i
integer
Head index (i E ..., 1H).
j
integer
Group index (j E {0, ...,, 9 - 1}).
(1)
integer
Mapping from head index group index.
integer
Head index within group in GLA-2, = (i144/2.
scalar
Softmax scaling factor, + = 1/ /d, + d.
integer
Number of censor-paralle devices.
RoPE (-)
function
Rotary Position Emhedding applied to vectors (implemented via ro- tation matrices).
RMSNorm (.)
function
Root-mean-squee square normalization.
Reshape (.)
operator
Tensor reshaping (no data change). Replication along the head dimension (e.g., broadcasting g KV heads to heads).
RepeatInterleave (-)
operator
Concat (+)
operator
Concatenation along the last dimension unless stated otherwise.
B THEOREM
B. TRANSLATION EQUIVARIANCE
Equivariance is a fundamental property geometric systems such as molecules, where vector fea- tures such as atomic forces or dipole moments must transform consistently with the coordinate system (Weiler et al., 2018; Fuchs et al., 2020; Satorras et al., 2021). In the context of sequence models, a common transformation is sequence translation. Let X (0), (.))..... (n-1)) e X be sequence of tokens, and define translation operator T, X + X that translates the entire sequence by s positions. Let Φ X W be function that maps a sequence to a matrix of attention scores (X) R where each element & )))
A (t) (e (x(+)) denotes the attention score between tokens (fx) and (ta), We say that Φ is translation equivariant if there exists corresponding output-space transformation S. Y > Y such that:
(11) This property ensures that the attention score between two tokens depends only on their relative positions, not their absolute positions. That is crucial for batch inference using left padding, where
Φ (T, (X)) S. ( (X)), vs.
19

---

Published as a conference paper at ICLR 2026
sequences of different lengths offset to align ends. The first non-padding token of a sequence is no longer at position 0, yet attention scores remain equivariant to this translation.
B.2 ROTARY POSITION EMBEDDING
Rotary Position Embedding (RoPE) (Su et al., 2024) is positional encoding method designed to incorporate relative position information directly into the attention mechanism. We show in this section that RoPE is translation equivariant.
Theorem 1. Given two tokens with query and key k at positions to and to respectively, let RoPE qg, bn) and RoPE (ke (x) denote the RoPE-encoded vectors. We show that translating both positions by an offset leaves the inner product unchanged:
(RoPE (q, t4 + s) RoPE (k, to + s)} = (RoPE (q)tt)) RoPE (k. tx)).
Equivalently, for the allention-score matrix (xX) ) E IRux induced by RoPE-based dot products, ó translation equivariant the sense of Eq. (11) with Sg being the simultaneous row/column shift operator.
Proof. Let the RoPE dimension d, (assumed even). RoPE applies a block-diagonal rotation matrix R2 E xd, at position t. Writing RoPE as right-multiplicttion row vectors, we compute the inner product under RoPE as:
d-122-1
RR, (kR,,, RR,
R = Re
Σ
(2222222 kipn22422 (ll (0)
t=
where 0g is the angular frequency for the f-th 2D block and (-)* denotes complex conjugation under the standard R²2 C identification.
Now consider translating both tokens by an offset The relative displacement is unchanged:
(ty +) (fx + s) = ty tk. so the factor el(ty -t) remains unchanged for every Therefore,
(R, + s) RoPE (k, ta + s)) (RoPE (y) RoPE (k: fx))
This proves RoPE-induced dot-product scores satisfy translation equivariance in Eq. (11).
Remark 2. While RoPE preserves dot- dot-product translation equivariance, applying arbitrary lin- ear map after RoPE generally breaks this property. Specifically, consider:
(RoPE (q. tg) W RoPE (k, t&) W*) = qR. (W*)* R.
The term w (W*)² breaks translation equivariance by disrupting the expression's 'ss dependence on the relative position to - tg. The property would only be preserved in the specific case where we (W*) = I, which would reduce the expression its original form. However, since this constraint is difficult to enforce during training, translation equivariance is generally lost when applying linear projection after RoPE.
C ATTENTION MECHANISM
C.I MULTI-HEAD ATTENTION (MHA)
Consider sequence of n tokens with hidden states H E Raxd, We first project these hidden states into queries, keys, and values using projection matrices Wº, W WV E (bdd))
Q = RoPE (HW*), K = RoPE (HW)), V
where Q V E (hda), is the number of attention heads, and d, the dimensionality of each head. Next, reshape these matrices separate the head dimension:
Reshape (Q, [n, h, dxl), Kc Reshape (K, [rt, h, dal). vc such that Q, S yc
Reshape (V. [n,, h. dal).
Raxhxds We cache KC and vc accelerate inference.
20

---

Published as a conference paper at ICLR 2026
Remark 3. Let Q, k Ruxhxda denote the RoPE-encoded queries and keys after projection and reshaping. For head i, define:
Quaik Q ltg. i, =], Kiz,di == KC [txx i, :]
Then, for any translation offset s, it follows from Theorem that:
(1a+add Kesto,s)) (Qtarkis Key.ke)
C.2 MULTI-QUERY ATTENTION (MQA) AND GROUPED- QUERY ATTENTION (GQA)
Both MQA and GQA reduce the number of key and value heads compared to MHA, while main- taining the full number query heads. MQA takes this to the extreme by using a single key-value head for all query heads, whereas GQA partitions the query heads into groups that each share key-value head. Given a sequence n tokens with hidden states H gaxd, the queries are computed using the same projection in MHA. To reduce KV cache, WK WV E Radddy where < (e.g.,
projection matrices
for MQA), to compute:
RoPE HW*), V HWV.
These are then reshaped into per-head form:
KC = Reehape
K [n, 9, drl): vc Reshape (V., [n, 9, dul).
We cache K' and vc during inference. We repeat them by a factor of r = h/g g along the head axis to match the h query heads:
(KC r, dim = 1) E Ruxhxdd
=
= RepeatInterleave (ve,
dim = E Raxhxdn
Remark 4. Let Q Raxhxds and Ke Raxaxdn denote the RoPE-encoded queries and cached keys, respectively. For head i, define:
Qt Q Ulg: i, i), Kr,ii: Kc
Lk:
Since both vectors are RoPE-encoded, for any offset s (with valid indices),
(Qittaiie Kectoss) (Qtyskis Key.a))
C.3 MULTI-HEAD LATENT ATTENTION (MLA)
Given a sequence of n tokens with hidden states H E Iundd MLA first computes queries as:
º 0ig RMSNorm (HWDD), NoPE -c
RoPE
-RoPE (C
DD Iddd WUQ E x((dd))))
QOR (
where 0ig v d is the rescaling factor for query states CO2 We then reshape queries to separate
heads:
a NoPE = Reshape
NoPE [n, h, dul)
Q RoPE = Reshape
Q @RoPPE [n. h, di
where Q NoPE Ranhhdd and Q RoPE Raxhxdf" These concatenated along the last dimension
to form the final query:
Concal
[ NoPE ROPE
dim=2).
To reduce the KV cache, MLA obtains shared compressed KV states via down-projection:
KV
ake RMSNorm
(HWDKV), DKV e Bdxd,
K RoPE = RoPE (HWKR), KKKR E Rdxdf
21

---

Published as a conference paper at ICLR 2026
where Orkv = VV a. Both CKV and K are cached during inference. MIA computes keys and values using learnable up-projection matrices:
K C C. WUk = ck puy, Wuk wuV E (
These are reshaped into per-head form:
K NoPE
Reshape
K NoPE [n, h, dul). KROPE Reshape (K RoPEE, [n, 1, df.))
V Reshape (V, [n, h, d,.).
where K NoPE E Rnxxhxd and V E Raxxxds,
To obtain per-head position-aware keys, MLA repeats the partial RoPE key across heads:
K = Concat ([KNOPE, RepeatInterlent (KRIPE, h, dim=1)] dim=2):
Remark 5. We analyze the translation equivariance property of MIA. les Ouydet
Concat NoPE (bg i, :]; Q RoPE ('g, i, :1) and Kec,i. = Concat NoPE [ta: i, :), KR It,, denote the query and key vectors for head i positions bg and lu respectively. The attention score for this head is given by the inner product:
(QRuPE (Q Kinde.) = -a NoPF (bg, i, ;], K NoPE [1x, i, .. + ( RoPE (1g i, ;] RRRP (Lk: :1)
Between the two terms, the second term with RoPE due to Theorem 1. The first term is mosition-independerr and thus unchanged under joint translations of by and 14- Therefore, the attention score ( (Q.,,.:3 ki,,:;) equivariant to translation imput positions. Although MLA introduces positional inductive bias via partial RoPE and is translation equivariant, we refer to this property as semi-translation equivariance to distinguish it from full RoPE translation equivariance.
yet translation equivariant,
C.4 MULTI-MATRIX FACTORIZATION ATTENTION (MFA)
Given a sequence of n tokens with hidden states E Raxd MFA uses h query heads but only a single shared key-value head (i.e., g 1). MFA first projects H low-rank space and applies RMSNorm:
C RMSNorm (HW*9), W Raxd
It then up-projects to all query heads and applies RoPE:
Q RoPF (CoWU)), w E (1-2d)))))))
Finally, we reshape into per-head form:
Q = Reshape @2, [n, h, 2dz)) E Raxhx2d.
To reduce KV cache, MFA computes only one key head and one value head:
K RoPE (HW*), vc HWV, W w. E Idx2d.
where K. vc E Enx2dd, We reshape them as
kc = Reshape (KC, [n, 1, 2dhl). Vc Reshape (v', [n, 1, 2dhl)
and cache KC and cc during inference. We repeat them along the head axis to match the h query heads:
RepeatInterleave (KC ht dim = =1) E Raaxx2d..
Repeatinterleave (v h, dim = 1 nuxhx2d
The analysis of translation equivariance similar that of MQA.
22

---

Published as a conference paper at ICLR 2026
C.5 TENSOR PRODUCT ATTENTION (TPA)
TPA achieves KV cache compression through low-rank factorization. represents each head's key/value token as low-rank mixture of Skg components: component vectors in R and head- specific scalar coefficients. During inference, TPA caches the component tensors and coefficient tensors, and computes keys/values on the fly via linear combination.
Given sequence of tokens with hidden states H E Raad TPA first computes the query/key/value factors:
Q HW
w E (d
E (Rxx((
Q HW
W E (dx(
E R (Pydan
K HWAR
WAR E (dx))
K E (ax((
K HW
W (deod))))))
K E Rax((Roda))
V HWAY
W E (dx(Sa)h)
V E (
V HW
W (Be(dd)))))
V' (ux((e)ds))
We reshape the projections into 3D tensors:
Reshape @Q", (n, Bgg In):
Q RoPE Reesaae (Q
[n, Sy dil)):
K Reshape K [n, Bkos hil)
KC RoPE Reshape (K_, [n,, BRos drl))
V Reshape (VA, [mt, Bkus nl).
Vc Reshape (V_, [ra, Blus drl)
so that Q x
E paxA xds, K* E RuxxBeu Kc E RuxBeexds v e Rax
yc Xhe
For each token position as:
E {o...., n - 1}. the final query, key, and value matrices are computed
1
(a* it, i, :1) T a [t. :, Raxds
it, 1,
Bg
K [1, ;,
(K* [1, is Kc [1, ; Rhxdd
BEu
[t, ;, :]
(VA (1
vc [t, i e Raxds
BRe
During inference, TPA caches K., KC VA yc.
Remark 6. Fix a head index i and token positions typ th- Let Qugiit K = K [4x, i, :) Rdd, From the computation above, we have
[ty: i, :] Rdd and
8
Akw
Qrgihi 3y by=0
Σ a² [tys by.
Q [tgg by: :I
Kig
Σ buo=0
Kd [he, been K [ti, buer i]
Buu
Therefore, the inner product expands as
Bire
(Qughhi Keyiie)
a¹ [Mgs bys i) KK (tx, bevn 2] ( [tg. by :), KC [tx, beer :1).
23

---

Published as a conference paper at ICLR 2026
Since a and ke are RoP RoPE-encoded, Theorem implies that for any offset
@ [ty s. by: :1, KC [tx + bluu :1 (
(o(((t (tgo bg. =] KC [t&, bkus :1),
a bbg
bku-
Because the scalar coefficients Q' [tg. bg. i) and Ktt [tk, bko i) shift with the tokens under transla- tion, the full double-sum is equivariant under jointly translating by and ly by the same offset 8:
+sita Kictass) (Qtyihit Kaycha)
Thus, TPA preserves translation equivariance of attention scores with RoPE.
C.6 GROUPED LATENT ATTENTION (GLA)
Given a sequence of n tokens with hidden states H E Raxd GLA divides the h attention heads into groups (e.g. = 2), where each group has r h/g heads. GLA adopts the same query computation mechanism as MLA.
Instead of single compressed KV state, GLA computes g independent compressed states: C ake RMSNorm (HW)DKY), WADDK E Rdx(d//o),
where j E ........ - 1}. de is the total latent dimension, each group uses de/g dimension, and
gd "
The RoPE keys remain shared across all groups:
K RaPE
RoPE (HWK)), WRR dd
During inference, we cache (CiKY, C9-1.V) and RoPE with total KV cache size of de + di token.
Each group independently computes its keys and values:
K J.NoEE C5.KY V C..Y W..K Wo.uv E ((de/))x(rd)))
where h/g is the number of heads per group.
Reshape into per-head form for each group:
Ko.NoPE Reshape
K i.ooEE [n, r, dil). V Reshape (V", [n, r, dal).
RoPE Reshape (K- RoPE
[n, 1, d
Construct position-awar aware keys for each group by repeating the shared RoPE keys:
K' Concat ([
RepeatInterleave
ROPE r, dim=1)] dim=2) E T xx
We finally concatenate all K2, V to obtain:
= Concat [K_ K Km-1] '] dim=1)
([vo..... v--1] I] dim=1)
= Concat
where [u and V gaxhxd, The analysis of translation equivariance is similar to that of MI.A.
C.7 GROUPED- TIED ATTENTION (GTA)
Given sequence of tokens with hidden states H B GTA uses h query heads and y value heads, and computes queries as
Q = -WW Wº E Rdx(hds)
Q Reshape (Q, [n, h, dal) nxxxds.
24

---

Published as a conference paper at ICLR 2026
We split Q into NoPE and RoPE parts and apply RoPE the latter:
NoPE
Q
[ i, dn d E IR " (d)
RoPE
@ i dA - di :1) E Buxxxxa
Q Concat
Q NoPE
Q RoPE dim=2) E IR Rnxxhxd
GTA computes single RoPE key shared across all heads:
K
RoPE
HWWK
W
R K-ROPE RoPE
K RoPE
KRoPE Reshape RoPE, [n, 1, df
To reduce KV cache, GTA computes grouped value states with only g heads:
V HWWY WKV E (dd(ad)) vc -Reshape (V, [n, 9: dal)
We cache vc and K RRPE during inference. We repeat vc along the head axis with r = h/g to form
the final values:
Repeatlnterleave (ve, r, dim -1) E puxxxxd
We then form the final keys by tying the NoPE part of keys to the values and concatenating with the shared RoPE key:
K = Concat
([V [ : d, di)
Repeat (KROPE, h, dim=1)] dim=2) RRxxhxd
]
The analysis of translation equivariance similar that of MLA.
D LLAMA-3 ARCHITECTURE
Given hidden states H E Raxdd for sequence of tokens, first compute the attention output
H' RMSNorm (H)
Oalin
Attention (H') (adx()d))
then project back to the model dimension and add a residual:
H - H + O Ollt 0,at
Oattt E R (hdx)xd
Next, an MLP block (gated form) applied:
H' RMSNorm (H)
mlp (H'"") @ (H'W2), W1 2 Raxd,
followed by the output projection and residual:
H H + mil O.mlp
O.mipp E Iddy
where (-) is an elementwise nonlinearity function such as SiLU and @ denotes elementwise multi- plication.
E GATED ATTENTION
Given hidden states H E BBuxd for a sequence of n tokens, we first compute the attention output with gated score
G < (HW*)
H' RMSNorm (H)
I Attention (H') O G E (hdd))
25

---

Published as a conference paper at ICLR 2026
then project back to the model dimension and add residual:
H H Oatm W.O.atn
W.O.tn E IR (g(hd))dd
Next, an MLP block (gated form) is applied:
H RMSNorm (H)
Omlp = = ( R (H'W)),
W W E Raxd;
followed by the output projection and residual:
H + H + Omip O.mppp
W W.ipp E IR xd
where s(-) an elementwise nonlinearity function such sigmoid and @ denotes elementwise multiplication.
F ARCHITECTURAL HYPERPARAMETERS
ARCHITECTURAL HYPERPARAMETERS FOR MAIN RESULTS
Our model is based on the Llama-3 architecture, adopting configuration largely consistent with Llama-3.2-3B but modified to 24 layers (down from 28). The architecture utilizes 24 attention heads, model hidden dimension (d) of 3072, head dimension (d₂) of 128, and an intermediate Feedforward Network (PFN) dimension (dj) of 8192. The architectural hyperparameters for our baselines are aligned with their original implementations. Specifically, MLA configured with latent dimensions d' = 12d,, de 4d and di = 0.5d,; GLA, along with our proposed MLRA, adopts d' 8dh, de = 4d,, and dff = 0.5dg; and TPA uses ranks Sg 6 and ßke = 2. For GQA and GTA, we set the number KV heads to g h/4. We report the detailed architectural hyperparameters for our main experiments in Tables 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, and 17.
Table 7: Model configuration of MHA for main results.
Model Size
Parameters
# Layers
h
dn
dj
2.9B
2872.59M
24
24
3072
128
8192
Table 8: Model configuration of MQA for main results.
Model Size
Parameters
# Layers
h
d
dr
df
2.9B
2872.00M
24
24
3072
128
10152
Table 9: Model configuration of GQA for main results.
Model Size
Parameters
# Layers
d
d₁
dj
2.9B
2872.59M
24
3072
128
9728
Table 10: Model configuration of MLA for main results.
Model Size
# Parameters
# Layers
h
d'
d.
d
Oag
Olkov
dn
d dj
2.9B
2872.05M
24
24
1536
512
3072
v2
128
64
9448
Table 11: Model configuration of MFA for main results.
Model Size
# Parameters
# Layers
h
g
d'
d
d,
2.9B
2873.23M
24
24
2048
3072
256
8024
Table 12: Model configuration of TPA for main results.
Model Size
# Parameters
# Layers
dp
d
2.9B
2873.18M
2
3072
128
10760
26

---

Published as a conference paper at ICLR 2026
Table 13: Model configuration of GLA-2 for main results.
Model Size # Parameters # Layers h
g
d'
c de
d
Qgq
Okkv
dn
d
dj
2.9B
2872.63M
24
24
2 1024
512 3072 v3
V12
128
64 10048
Table 14: Model configuration of GLA-4 4 for main results.
Model Size
# Parameters
# Layers
h
d'
de
Ocko
dn
d
dj
2.9B
2873.22M
24
24
1024
512 3072
v3
V21
128
64
10136
Table 15: Model configuration of GTA for main results.
Model Size
Parameters
Layers
h
d
dn
R
dg
2.9B
2872.00M
24
24
3072
128
64
9960
Table 16: Model configuration of MLRA-2 for main results.
Model Size
# Parameters Layers
h
d'
de
d
Qgg
Okkv
Cltttn
dp
dh dy
2.9B
2872.63M
24
24
1024 512 3072 v3 /24
128
10048
Table 17: Model configuration of MIRA-4 for main results.
Model Size # Parameters Layers
d'
de
d
0gg Olke
Caatnn
d,,
d
d
2.9B
2873.22M
24
24 1024
512 3072 v3 224
128
64 9880
F.2 ARCHITECTURAL HYPERPARAMETER FOR INITIALIZATION ABLATION STUDY
In our initialization ablation study, we focus on the initialization of the attention and FFN output projections parameters aWn mlp). We evaluate two distinct initialization strategies: zero initialization versus a Gaussian distribution (0, a 0.02), to identify which yields better per- formance. To isolate the impact of the initialization strategy, the model architecture and all other hyperparameters are kept identical to those used for our main results. We report the detailed ar- chitectural hyperparameters for our initialization ablation Tables 18, 19, 20, 21, 22, 23, 24, 25, and 26.
Table 18: Model configuration of MHA for initialization ablation.
Model Size
Parameters
# Layers
dn
d,
2.9B
2872.59M
24
24
3072
128
8192
Table 19: Model configuration of MQA for initialization ablation.
Model Size
Parameters
# Layers
d
dn.
dy
2.9B
2872.00M
24
24
3072
128
10152
Table 20: Model configuration of GQA for initialization ablation.
Model Size
# Parameters
# Laaeers
h
d
dn
d,
2.9B
2872.59M
24
24
3072
128
9728
Table 21: Model configuration of MLA for initialization ablation.
Model Size
# Parameters
# Layers
h
d'
d,
d
Okg Oliev dn
dR dg
2.9B
2872.05M
24 1536
512
3072
/6
128
9448
27

---

Published as a conference paper at ICLR 2026
Table 22:
Model configuration of MFA for initialization ablation.
Model Size
Parameters
# Layers
h
d'
dn
dj
2.9B
2873.23M
24
24
2048
3072
256
8024
Table 23: Model configuration of TPA for initialization ablation.
Model Size
# Parameters
# Layers
3q
Skev
d2
dj
2.9B
2873.18M
6
2
3072
128
10760
Table 24: Model configuration of GLA-2 for initialization ablation.
Model Size
# Parameters # Layers
h
d'
de
d
Cxq
dn
d
dj
2.9B
2872.63M
24
24
1024
512
3072
12
128
64
10048
Table 25: Model configuration of GLA-4 for initialization ablation.
Model Size
# PParameters
# Layers
h
d'
de
Orke
dn
%
dj
2.9B
2873.22M
24
24
1024
512
3072
24
128
64
10136
Table 26: Model configuration of GTA for initialization ablation.
Model Size
# Parameters
Layers
h
d
dp
d
d
2.9B
2872.00M
24
24
3072
128
64
9960
F.3 ARCHITECTURA, HYPERPARAMETERE FOR SCALING ABLATION STUDY
In scaling ablation study, investigate the impact of the scaling factors Og. Okke and Oatttn applied to the query latent states (C9), the KV latent states (CKV), and the final attention output (O), respectively. To determine the optimal configuration, we compare the model's 's performance with and without these scaling factors, where the 'without' setting corresponds to fixing Og. Oke and Oaattn 1. To isolate the impact this scaling strategy, the model architecture and all other hyperparameters remain identical to those used in our main results. Detailed architectural specifications for these ablation experiments are provided in Tables 27, 28, and 29.
Table 27: Model configuration of MLA in the absence of scaling factors.
Model Size
# Parameters
# Layers
h
d'
de
d
Org
Ol
dp
d
d, 9448
2.9B
2872.05M
24
24
1536
512
3072
1
128
Table 28: Model configuration of GLA-2 in the absence of scaling factors.
Model Size # Parameters # Layers
h 9 d' de
d
Oiq Olkv dn
dR dg
2.9B
2872.63M
24
24
2 1024 512 3072
128 64 10048
Table 29: Model configuration of MLRA-2 in the absence of scaling factors.
Model Size # Parameters # Layers
h
d'
de
d
0gg Odjev Oattnn
dn
d dy
2.9B
2872.63M
24
24
1024
512
3072
1 1 1
128
64 10048
28

---

Published as a conference paper at ICLR 2026
F.4 ARCHITECTURAL HYPERPARAMETER FOR DOUBLE HEADS ABLATION STUDY
In our head-count ablation study, we investigate whether doubling the number of attention heads for GQA, MLA, and GLA-2 improves performance. Specifically, we increase the number of heads to 48 while maintaining the original KV cache size. To maintain parameter parity with our main results, we decrease the Feed- Feed-Forward Network (FFN) intermediate dimension. By keeping all other hyperparameters identical, we isolate the specific impact of the doubled head count. The detailed architectural specifications for these experiments are provided in Tables 30, 31, and 32.
Table 30: Model configuration of GQA parameterized with 2x attention heads.
Model Size
# Parameters
# Layers
d
dn
dy
2.9B
2872.59M
24
48
3072
128
7680
Table 31: Model configuration of MLA parameterized with 2x attention heads.
Model Size
# Parameters
# Layerr
h
d'
de
d
Ogg
Ofee
dx
di
dj
2.9B
2873.23M
24
48
1536
512
3072
v
v6
128
64
7320
Table 32: Model configuration of GLA-2 parameterized with 2x attention heads.
Model Size
# Parameters
Layers
g
d'
de
d
0gg
Ol
dn
h
dj
2.9B
2873.22M
24
48
1024
512
3072 v3 12 128
64 8344
F.5 ARCHITECTURAL HYPERPARAMETERS FOR GATED ATTENTION STUDY
In our gated attention study, investigate whether incorporating gating mechanism (Hochreiter & Schmidhuber, 1997; Srivastava et al., 2015; Dey Salem, 2017: Qiu et al., 2025) into GQA, MLA, GLA-2, MLRA-2, and MLRA-4 improves performance. Specifically, we integrate gated attention into these architectures as shown in Appendix E. To maintain parameter parity with our main results, proportionally decrease the Feed-Forward Network (FFN) intermediate dimension to offset the additional gate parameters. By keeping all other hyperparameters identical, we isolate the specific impact of the gating strategy. The detailed architectural specifications for these experiments are provided in Tables 33, 34, 35, 36, and 37.
Table 33: Model configuration of GQA incorporating gated attention.
Model Size
# Parameters
# Layers
g
d
dn
d,
2.9B
2872.59M
24
24
3072
128
8704
Table 34: Model configuration of MLA incorporating gated attention.
Model Size
# Parameters
# Layers
h
d'
de
d
0ag
Olke
dR dj dp.
2.9B
2872.05M
24
24
1536
512
3072
/2
V6
128
64 8424
Table 35: Model configuration of GLA-2 incorporating gated attention.
Model Size # Parameters # Layers
h
d'
de
d
Gg Okkvv
dn
d dj
2.9B
2872.63M
24
24
1024 512 3072
v3 / 12 128 64 9024
Table 36: Model configuration of MI.RA-2 incorporating gated attention.
Model Size # Parameters # Layers
h
d'
de
d
Olg Olker Olattn.
dn
di dj
2.9B
2872.63M
24
24
1024
512 3072 /3 /24
/2
128 64 9024
29

---

Published as a conference paper at ICLR 2026
Table 37: Model configuration of MLRA-4 incorporating gated attention.
Model Size # Parameters
# Layers
h
d'
de
d
04q
akke
Qattn
dn
d dj
2.9B
2873.22M
24
24
1024
512
3072 v3 /4
2 128
64 8856
G ADDITIONAL EXPERIMENTAL RESULTS
Table 38: Validation perplexity (lower better) across seven datasets: Wikipedia, C4, Pile, Re- fibe Cosmopedia, Fine Web, and FineWeb- Edu. We compare two initialization strategies, zero
versus Gaussian (N 0.02)). applied to the output projection weights W atin and W mlp
Method
Initialization
Wikipedia
C4
Pile
RefinedWeb
Cusmopedia
FineWeb
FineWeh-Edu
MHA
Ni0,a 0.02)
14.759
16.800
13.282
18.988
9.356
15,904
9.571
14,094
MHA
zero
14.624
16.575
12,929
18.698
9.102
15.656
9.434
13,860
MQA MQA
a00, = 0.02)
14.708
17.075
13,500
19.301
9.510
16.190
9.697
14.283
zero
15.134
16.837
14,008
19.202
9.484
15.942
9.533
14,306
GQA
N0,, 0.02)
14.687
16.882
13.528
19.084
9.422
15.974
9,571
14,164
GQA
zero
15.057
16.628
13.758
18.885
9.504
15.713
9.427
14.139
MLA
N0,0 0.02)
14.571
13.113
18,837
9.110
15.740
9.490
13,927
MLA
ze1o
14.567
16.345
12:965
18.523
8.966
15.440
9.284
13.727
MFA
a00, = 0.02)
15.123
17.032
13.752
19.133
9,550
16.138
9.707
14.374
MFA
zero
15.693
16.738
13.903
19.125
9.423
15.815
9.506
14.315
TPA
a0 0.02)
15.205
16.227
TPA
zero
14.789
16.622
13.333
18,971
9.130
15.717
9.333
13.985
GLA-2
(00, = 0.02)
14.717
16.675
13.216
18.886
9.259
15.799
9.510
14,009
GLA-2
zero
16.323
13.225
9.118
15.424
9.249
GLA-
a0, 0.02)
14.858
16.791
13.522
18.953
9.374
15.914
9.571
14.140
GLA-4
zero
14.547
16.436
13.229
18,578
9.076
15.535
9.307
13.815
GTA
a(0, 0.02)
19.896
16.959
13.621
19.277
9.536
16.061
9.647
14.285
GTA
zero
18.924
9.129
13.972
Table 39: Validation perplexity (lower is better) across seven datasets: Wikipedia, C4, Pile, Re- finedWeb, Cosmopedia, FineWeb, and FineWeb-Edu. This analysis specifically compares models without and with scaling.
Method
Scaling
Wikipedia
C4
Pile
RefinedWeb
Cosmopedia
FineWeb
FineWeb-Edu
Avg
MIAA
w/o 14.461
16.386
13.218
18.636
8.961
15.485
9.307
13.779
MLA
w/
14,567
16.345
12.965
18.523
8.966
15.440
9.284
13.727
GL-A-2
w/o 14.518
16.467
13.179
18.612
9.138
15.565
9.305
13.827
GLA-2
w/
14.605
16.323
13.225
18,509
9.118
15.424
9,249
13.779
MLRA-2
w/o 14.326
16.485
13.145
18.657
9.168
15,570
9,304
13.808
MLRA-2
w/
14.615
16.342
13.236
18.602
9.153
15.439
9.242
13.804
Table 40: Validation perplexity (lower is better) across seven datasets: Wikipedia, C4, Pile, Refined- Web, Cosmopedia, Fibe and Fine Web-EEd This analysis specifically compares models with and without 2x attention heads.
Method
2x Attention Heads
Wikipedia
C4
Pile
RefinedWeb
Cosmopedia
FineWeb
FineWeb-Edu
Avg
GQA
w/ 15.280
16,702
13.789
18.961
9.486
15.785
9.490
14.213
15.057
13.758
9.427
MLA
w/
14.771
16.432
13.108
18.615
9.029
15.529
9.371
13.836
MLA
w/o
14.567
16.345
12.965
18.523
8.966
15.440
9.284
13.727
GLA-2
14.669
16.313
13.428
18.569
15.410
9.281
13.851
GLA-2
w/o
14.605
16.323
13.225
18.509
9.118
15:424
9.249
13.779
30

---

Published as a conference paper at ICLR 2026
H ILLUSTRATION
Training Loss Curves
2.7
MON
2.6
GQA
Training loss
2.5
GILAZ
GLA4
RA-1
2.2
100
Training tokens (B)
Figure 7: Training loss curves for all models.
000 ...
000
Cachod During Inference
DDC ... O03
C0O ... OOC
Hull-Hind Altentun
HolleHeee Attestam
Ridii-Hedd Allrttion
Kuli- Headl--adlll Altenison
T s
1000
Ooo
Concatonate
M
Apply Rage
C0 v. O aten
@a) 227
@ok AM
Input Hidden & 200 ... 000
Figure 8: Illustration of MLRA-2
300
OOO
Cached During Inforence
Uutput lllden
XOO
OOO
CO0 OOC
Atsention
Atlention
1oo0
la I)
ooo
4
ooo Pha
1o0 MBB
o00 [k]]
Gancatstore
(
Appdy Rope
20 O+) ef
@ LaV
@ 2.v
c @ 2AY
leput Hiddes b 1000 ... 000
MIRA-4
Figure 9: Illustration of MLRA-4.
1 RELATED WORK
KV Cache Compression. Recent works (Liu et al., 2023; Anagnostidis et al., 2023; Zhang et al., 2023b; Ge et al., 2024; Xiao al., 2024; Kim
al., 2024; Zhang al., 2024b; Nawrot et al.,
31

---

Published as a conference paper at ICLR 2026
2024: Tang al., 2024; Liu et al., 2024b; Dong et al., 2024; Cai et al., 2024; Liu al., 2024a; Hooper et al., 2024; Sun et al., 2024; Chen et al., 2024a; Jiang et al., 2024; Li et al., 2024; Xiao et al., 2025; Sun al., 2025; Meng et al., 2025; Tang et al., 2025) don't introduce new attention mechanisms; instead, they compress the KV cache for pretrained models. Some of these works (Liu al., 2024b; Hooper et al., 2024) use quantization to store the KV cache in low-bit formats. Some other approaches (Zhang et al., 2023b; Xiao et al., 2024; Li et al., 2024; Xiao et al., 2025) retain important tokens and discard others compress the KV cache.
Low-Rank Approximation. Low-rank approximation (Hu et al., 2022; Malladi et al., 2023; Zhang et al., 2023a; Dettmers et al., 2023; Lialin et al., 2024; Zhu al., 2024; Zeng Lee, 2024; Chen et al., 2024b; Lin et al., 2025; Wang et al., 2025; Chang et al., 2025) are widely used to com- press representations to a low-dimensional space, then up-project to recover full representations. These methods greatly reduce trainable parameters (Hu et al., 2022; Dettmers al., 2023) during fine-tuning and decrease the number of parameters (Lin et al., 2025; Wang et al., 2025) for pretrained models.
System for Attention. FlashAttention (Dao et al., 2022; Dao, 2024; Shah et al., 2024) uses tiling and online softmax minimize reads and writes between high-bandwidth memory and on-chip SRAM, shifting attention from a memory bottleneck to compute bottleneck. FlashMLA (Ji- ashi Li, 2025) avoids explicit KV materialization during attention decoding by absorbing the key up-projection matrices into the queries. The following attention computation is similar to MQA with shared KV states. Inspired by classical virtual memory and paging in operating systems, Page- dAttention (Kwon al.. 2023) and vLLM use block-level memory management and preemptive request scheduling to reduce fragmentation and redundant duplication.
Linear Attention. Linear attention (Katharopoulos et al., 2020; Peng et al., 2021; Schlag et al., 2021: Gu et al., 2022; Smith et al., 2023; Sun et al., 2023; Qin et al., 2023; Yang et al., 2024a; Dao & Gu, 2024; Peng al., 2024; Gu & Dao, 2024; Beck et al., 2024; Zhang et al., 2024a: Yang et al., 2024b; 2025c) reformulates the attention mechanism by substituting the exponential kernel in softmax with a dot product between the query and key vectors. It reduces the memory complexity per decoding step from O (n) for full attention to O (1).
32


Reviews:
[{"review_id": "G7U1NGnTE9", "reviewer": "Reviewer_xaCs", "summary": "The paper tries to address the long‑context decoding bottleneck in LLMs, repeatedly loading large key–value caches from off‑chip memory by proposing Multi‑Head Low‑Rank Attention (MLRA) a dual‑path attention mechanism that works with tensor parallelism. MLRA keeps a base path that compresses the KV cache into a single latent head (maintaining model quality) and adds a low‑rank path that compresses the KV cache into h tiny latent heads that can be sharded across devices to enable TP.", "questions": "1. Are there any preliminary results at larger scales (7B, 13B, or beyond)? Even partial training runs or fine-tuning experiments would help strengthen the paper's claims.\n\n2. Is there any theoretical or empirical argument for why the dual-path design should scale favorably? For instance, does the rank r need to grow with model size, or can it remain constant?\n\n3. The improvements over MLA are consistent but modest (0.9% perplexity, 1.6% accuracy). Are the results in Tables 3-4 from single training runs or averaged over multiple seeds? If single runs, can error bars be added? Did the authors conduct significance tests? Given the small differences, are the improvements statistically significant at p<0.05? If the authors ran multiple seeds at smaller scale (354M), what was the variance? This would help assess whether 2.9B differences are meaningful.\n\n4. Section 3.3 proposes running the base path on H100 and sharding the low-rank path across RTX 4090s, but provides no experimental validation, have the authors actually implemented and tested this setup?\n\n5. Table 13 shows the paper used zero initialization for W_O in TPA and MLRA, but N(0, 0.02) for other methods, is this cherry-picked or is there a principled reason? How does MLRA perform with N(0, 0.02) initialization? The ablation shows large effects for other methods (0.3-0.5 perplexity points).\n\n6. Figure 2 shows naive sharding hurts quality, motivating the dual-path design. Did the authors explore intermediate designs? What about 2, 4, or 8 latent heads instead of h?\n\n7. The paper claims thatMLRA is TP-friendly but doesn't measure communication costs. Can the authors profile the all-gather/reduce-scatter operations? What fraction of total latency is communication vs. computation? In Figure 4, GQA uses TP=8 while MLRA uses TP=4. For fair comparison, can the paper also show GQA with TP=4 (which should reduce its communication overhead)?\n\n8. The paper motivates MLRA for long-context inference but evaluate only on standard benchmarks. Have the authors tested on long-context tasks like SCROLLS, LongBench, or needle-in-haystack up to 100K+ tokens? Figure 4 shows decoding latency up to 2M tokens. Can you provide quality metrics at these lengths? Does MLRA maintain advantages?\n\n9. All efficiency results focus on decoding (Figures 4-5). What is MLRA's prefill latency and throughput compared to baselines? Since prefill is also memory-bound for long contexts, can MLRA help there too. Are there any results to validate this?\nc) For end-to-end inference (mixed prefill + decode), what are the overall speedups in realistic scenarios (e.g., RAG with 50K context + 2K generation)?\n\n10. Throughput evals: Figure 5 shows one batch size (128) with mixed DP/TP strategies. Can the paper also provide a comprehensive evaluation varying both batch size (1, 16, 32, 64, 128, 256) and sequence length? The paper uses DP=8 for MLA vs. TP=4/DP=2 for MLRA. Is this comparison reasonable? What if MLA also used TP=4/DP=2?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper does a good job of outlining how different mechanisms behave when sharded (Table 1), which shows MLRA achieving 1.5 dₕ per device with 4‑way TP, below GQA’s 2 dₕ target and lower than MLA/GLA under the same conditions.\n\n2. The authors formalize translation equivariance for RoPE (§2.2–§2.3; Theorem 1), explaining why post‑RoPE projections typically break it and hence using partial RoPE so MLRA maintains a (semi) equivariance property (§3.1–§3.3).\n\n3. Efficiency: Long‑context latency and throughput are reported with realistic parameters: MLRA is 1.05–1.26× faster than GQA and ≈ 2.8× faster than MLA for 128K–2M tokens (Figure 4), and achieves the highest decoding throughput from 1K–16K tokens. This is supported by Arithmetic‑intensity analysis in Table 2 explaining why MLRA remains competitive under memory‑bound decoding.\n\n4. The paper goes over architecture/training hyperparameters (Tables 5–11), provides an initialization ablation (Table 13), and derives the decoding procedure (Eqs. 10–13), making it easier to reimplement these experiments.", "weaknesses": "1. Scale:  The paper's primary motivation is to enable efficient inference for production-scale models but the  experiments only extend to 2.9B parameters. This creates a significant gap since it's uncertain how architectural complexity introduced by the the dual-path design will scale. At 70B scale with 80 layers, it's unclear whether the base path still maintain quality advantages, or will the low-rank path dominate. The paper shows that naively sharding MLA hurts quality (Figure 2, 354M model), but this is at a very small scale, which  again, might not translate to bigger models, where the per-layer capacity is much higher. Table 3 shows MLRA's advantage over MLA shrinks from 1.3B (14.414 vs 14.355) to 2.9B (12.998 vs 13.115). This suggests diminishing returns, and more experiments are needed to confirm the same.\n\n2. Marginal gains: Perplexity (2.9B): 13.115 to 12.998 (0.9% relative improvement, within noise for many tasks). Downstream accuracy improves from 55.46% to 57.06%. On individual datasets (Table 3), MLRA wins 3/6 but differences are often <1%.\nDecoding speedup over MLA is strong (2.8×), but over GQA is modest (1.05-1.26×). The magnitude of these improvements may not justify the proposed increase in architectural complexity.\n\n3. Section 3.3 proposes running the base path on H100 and sharding the low-rank path across cheaper RTX 4090s, but does not provide any experimental validation.  The heterogeneous deployment is positioned as a key advantage, but needs more validation. The arithmetic intensity analysis (Table 2) shows the low-rank path has much lower AI (~h/p+6 vs. ~2h for base path), which suggests potential bottlenecks.\n\n4. Efficiency analysis is missing some important calculations: TP overhead is not completely accounted for. There's no measurement of all-gather/reduce-scatter communication costs. All experiments focus on decoding (Figures 4-5), but prefill is also memory-bound for long contexts and it would be helpful to compare prefill costs as well.\n\n5. Experimental methodology: MHA/MQA/GQA/MLA/GLA use N(0, 0.02) for W_O while TPA and MLRA use zero initialization. The reasoning for this difference has not been called out. The ablation (Table 13) shows large effects: MLA with zero initialization degrades 0.363 perplexity points on average, which makes makes it unclear how much of MLRA's gains come from architecture v/s initialization. There's also no indication of multiple training runs or error bars. Figure 3's training curves could be noisy single runs and it's unclear how reliable these trends are. Downstream results (Table 4) show differences of 1-2% which could be within noise\n\n6. Only 6 downstream tasks are evaluated on (Table 4), all knowledge/reasoning tasks. It would have been very helpful to also include long-context tasks (SCROLLS, ZeroSCROLLS, needle in haystack), instruction following (MT-Bench), coding (HumanEval). Since MLRA targets long-context inference, evaluation on long-context benchmarks should be essential.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0PFYIfvAmW", "reviewer": "Reviewer_K7UD", "summary": "This paper proposes Multi Head Low-Rank Attention (MLRA), a dual-path attention mechanism designed to optimize LLM inference under Tensor Parallelism (TP).\nMLRA introduces a dual-path architecture, including a base path similar to MLA, and a low-rank path that splits KV cache into smaller vectors for partition.\nExperiments on models up to 2.9B size show that MLRA achieves lowest perplexity with faster decoding speed than GQA and MLA.", "questions": "- Ablation of the dual-path: I'm curious about the specific contribution of each path (e.g., disabling one path at inference time), providing deeper insights into the mechanism.\n- For MLA, existing implementations often employ simple data parallelism. The authors should discuss the comparison between MLRA and MLA in this scenario.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem is well-motivated since MLA's latent vector cannot be sharded across TP devices, leading to sub-optimal speed when multiple GPUs are available.\n\n- MLRA is designed with the constraints of tensor parallelism paradigms. The integration with system-level optimizations is a major strength.\n\n- Extensive experiments demonstrate MLRA‘s superior performance, achieving higher efficiency compared with SOTA attention implementations like GQA and MLA.", "weaknesses": "- Long context tasks: The optimization of KV cache becomes more important under long context scenarios. Adding some long-text tasks (like tasks from LongBench/RULER) can better demonstrate the effectiveness of MLRA.\n- Model architecture: The experiments are based on LLaMA-3 architecture. MLRA should be validated on other architectures (e.g., Qwen) to prove generality. Considering the pre-training setting, it's understandable if the author can not complete this experiment during the rebuttal period.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KYfgWgeIKF", "reviewer": "Reviewer_iTQG", "summary": "This paper proposes a new attention mechanism, MLRA, which is TP-friendly latent attention. They especially introduce low-rank attention paths that can be sharded across GPUs (distributing them across lower-cost GPUs is also possible for this). They evaluate the proposed methods in sufficiently large scales and showed good performance and quality.", "questions": "See above Weakness parts.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Paper is well-written and has good structure to explain their methods (detailed specification of dimension is quite helpful).\n- Motivation is highly clear, and proposed algorithm is quite well-aligned.\n- They compared various attention mechanisms in large scaled settings.", "weaknesses": "- What and why do you think MLRA improves the model quality fundamentally? \n- Could you elaborate more about throughput results of GQA? Why does it have lower and lower throughput with much longer sequences (due to custom Triton kernel or only using TP?)\n- Some studies regarding the effects of different DP and TP degree would be valuable to compare these attention mechanisms. \n- In Figure 2 experiment, why does Shard approach have higher training loss?\n- It'd be good to add explanation for GLA to Appendix as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6pCMcbDUXN", "reviewer": "Reviewer_8pjf", "summary": "This paper proposes Multi-Head Low-Rank Attention (MLRA), a novel attention mechanism that addresses the inefficiency of Multi-Head Latent Attention (MLA) under tensor parallelism (TP). MLRA decouples the KV cache into a base path (shared across devices) and a low-rank path (sharded across devices), reducing per-device KV cache memory to d_h + 3d_h/h with TP4, compared to 2d_h for MLA and 2d_h/g for GQA. The method compresses attention projections into low-rank shared subspaces while preserving model accuracy. \nThe authors use small-scale pretraining experiments to achieve state-of-the-art performance with 2.8 speedup compared with MLA.", "questions": "What is the overall advantage of head-wise TP in MLA over different parallelisms like data parallelism or sequence parallelism?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper improves the performance of LLM inference of KV cache transfer and memory bandwidth using tensor parallelism. The MLRA architecture decomposes latent representation into base and low-rank parts, enabling effective sharding and parallelization. I like the derivations in Section 3.\n\nThe authors compare MLRA with (MHA, MQA, GQA, MLA, GLA, and TPA) with LLAMA-3 models. The results show good performance. The advantage across different scale 354M -> 1.3B -> 2.9B show robustness.\n\nThe paper is well organized with clear figures. The equivariance analysis is a nice.", "weaknesses": "Limited large-scale validation: Although results up to 2.9B parameters are convincing, it remains unclear whether the same gains will persist at tens-of-billions scales (e.g., LLaMA-3-70B). The authors mention this as future work but do not provide any scaling projections or memory–throughput modeling beyond the small regime.\n\nComplexity of math: Sections 3.1–3.3 are mathematically dense and may hinder accessibility. A higher-level pseudocode could improve clarity and readability.\n\nAblation on α and rank: We wish to know α and rank would affect the final performance; ablations would help a lot.\n\nReproducibility: The paper references public implementations (FlashMLA, SGLang), but MLRA’s own kernel is not yet open-sourced, opensource them all would help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)