Householder-Diagonalized Linear Attention (HDLA): Utilizing Enhanced Decay Mechanism for Efficient Sequence Modeling
Authors: Jiefu Zhang
Zhen Qin
Jiabo Tong
Shijie Mei
JiaKui Hu
Yuqi Pan
Anjie Hu
Man Yao
Bo XU
Guoqi Li
Venue: ICLR 2026 Poster (2026)
Keywords: Linear Attention
Language Model
Foundation Model
Primary area: foundation or frontier models, including LLMs

Abstract: Linear attention mechanisms have emerged as efficient alternatives to Softmax attention, exhibiting steady improvements in language modeling capabilities driven by increasingly sophisticated designs for decay matrices—though their structural complexity has typically been limited to the Diagonal-Plus-Rank-1 level. To further advance the understanding and capabilities of linear attention via more complex decay structures, this work makes two primary contributions: (1)  We propose the HDLA linear attention mechanism, which utilizes efficient matrix decomposition to achieve a Diagonal-Plus-Rank-2 structure, thereby extending the decay matrix to a broader, more expressive, rank-enhanced and structured class. (2) We propose a more general chunk-wise parallel algorithm that accommodates both diagonal-plus-rank-$r_{ab}$ decay structure and key-value outer products of rank $r_{kv}$, thus providing a versatile foundation for future research. Comprehensive experiments demonstrate that, compared to linear attention baselines, HDLA sets new SOTA results on language modeling and retrieval tasks at 2.8B parameter scale, delivers at most 80\% and 58.2\% performance gains over baselines on retrieval-based MQAR and RULER tasks, and achieves an average score improvement of 4.39–7.66 on the synthetic MAD benchmark, respectively. Our proposed HDLA model, as well as the rank-generalized chunk-wise parallel algorithm, together provide a versatile algorithmic foundation and promising research prospects for the design of rank-enhanced, structured linear attention mechanisms.

Published as a conference paper at ICLR 2026
HOUSHHOLDER-DIAGGNAAIZZDD LINEAR ATTEN- TION (HDLA): UTILIZING RANK-ENHANCED DECAY MECHANISM FOR EFFICIENT SEQUENCE MODELING
Jiefu Zhang" 15+ Zhen Qin Jiabo Tong 1.3, Shijie Mei', Jiakui Hu', Yuqi Pan', Anjie Hul.3,
Man Yao?, Bo XU", Guoqi Li
Institute of Automation, Chinese Academy of Sciences
2 Taptap
3 Beijing Zhongguancun Academy
Peking University
5 School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences
ABSTRACT
Linear attention mechanisms have emerged as efficient alternatives to Softmax at- tention, exhibiting steady improvements in language modeling capabilities driven by increasingly sophisticated designs for decay matrices -thricc their structural complexity has typically been limited to the Piagonal-Plus-Ranin ther advance the understanding and capabilities of linear attention via more plex decay structures, this work makes two primary contributions: (1) We propose the HDLA linear attention mechanism, which utilizes efficient matrix decompo- sition to achieve a Shagonal-Plus-Rankee matrix broader, more expressive, structure, thereby extending the decay and structured class. (2) We propose a more general chunk-wise parallel algorithm that accommodates both diagonal-plus-rank-aa decay structure and key-value outer products of rank rke- thus providing versatile foundation for future research. Comprehensive exper- iments demonstrate that, compared linear attention baselines, HDLA sets new SOTA results on language modeling and retrieval tasks at 2.8B parameter scale, delivers at most 80% and 58.2% performance gains over baselines on retrieval- based MQAR and RULER tasks, and achieves an average score improvement of 4.39-7.66 on the synthetic MAD benchmark, respectively. Our proposed HDLA model, well the annkgeneralized chunk-wise parallel algorithm, together provide a versatile algorithmic foundation and promising research prospects for the design of rank-enhanced, structured linear attention mechanisms.
level. fur-
INTRODUCTION
Softmax attention, the core component of the Transformer [Vaswani et al. 2017), exhibits superior token mixing capabilities (Tolstikhin al. 2021 Yu et al. 2022) and supports highly efficient parallel training (Dao et al.) 2022). However, is severely limited in long context scenarios, by quadratic time complexity and key-value (KV) cache that grows linearly with the sequence length.
Linear attention presents an efficient alternative to softmax attention by reducing the time complexity to (()) and compressing the infinite key-value sequences into fixed-size hidden state (Katharopou- los et al. 2020). Not only does it demonstrate great research potential, but the hybrid architecture combining linear and softmax attention in 7:1 ratio has been successfully deployed as the founda- tional framework for large language models (LLMs) in practical applications, achieving exceptional throughput and advanced long-context reasoning capabilities (MiniMax al. 2025). Through pro- gressively more sophisticated hidden state decay mechanisms, linear attention has steadily improved
"Equal contribution for Jiefu Zhang and Zhen Qin. This work was completed while Zhen Qin was re- searcher at Taptap. Code is available at ahtttps: htt/://ittuu com/ Zhang Zhang jieu7777 EtTT//// HDLL- -mmpl 'Coresponding author, guoqi. ia.....n

---

Published as a conference paper at ICLR 2026
its language modeling performance. Nevertheless, series of recent works including DeltaNet Yang et al. 2024b), Gated DeltaNet (Yang et al. 2025), TTT-Linear (Sun et al., 2024 restrict
the structural complexity of their decay matrices to at most Maagonal-Plus-Rante
This insight naturally gives rise to compelling question regarding the future of linear attention: Does the Diagonal-Plus-Ree Rank-1 decay structure truly represent the celling of hidden state manage- ment and utilization? Or to say, can we extend the decay matrices to broader, structured, and more expressive classes, thus further elevating the performance ceiling of linear attention mechanisms?
Our work addresses the aforementioned questions through the following two primary contributions.
Firstly, propose the Hbussholddrr-Diggnnil ments language modeling capacity via a more sophisticated decay matrix structure while maintain- ing reasonable computational costs. We refine three necessary restrictions when designing complex and efficient decay structures: parameter efficiency, memory efficiency, and computational effi- ciency. Based on these restrictions and inspired by the congruence diagonalization theory of real symmetric matrices, we employ generalized Householder matrices to diagonalize the decay matrix, and show that HDLA's structured decay a specific instance of the Measonal-Plus-Rakkee class.
Linear Attention (HDLA) method, which aug-
Secondly, we introduce a rank-generalized chunk-wise parallel algorithmic framework, which si- multaneously accommodates the arbitrary diagonal term in linear attention mechanisms" decay. When formulating the chunk-wise parallel algorithm for HDLA, we achieve broad generalization that accommodates both Diagonal-Phun Planaal----------aas decay structures and rank-Tko key-value updates. This advance not only subsumes HDLA special case, but also provides robust foundation for future research on linear attention mechanisms.
Comprchensive experiments fully demonstrate the superior performance of our proposed HDLA model: (1) Achieves state-of-the-arr (SOTA) results in terms of language modeling perplexity (up to 2.8B parameter scale), with retrieval capability at 2.8B scale surpassing all linear attention baselines. (2) On the retrieval- -based RULER (Hsieh et al. (2024) experiment, achieves up to 58.2% accuracy improvement compared to Gated DeltaNet (Yang et 2025). (3) In synthetic MAD (Poli al.) ex- periment, the average score exceeds linear attention baselines by 4.39-7.66 7.66, significantly narrowing the performance gap with Softmax Attention (Vaswani et al. 2017). (4) In synthetic MQAR (Arora et al. (223a) experiments, at sequence length 2048, the accuracy is about 80% higher than the more computationally intensive Gated DeltaProduct with n, = 2 (Siems al., 2025).
While achieving superior performance, HDLA maintains a relatively limited and reasonable compu- tation amount. Even when compared to Gated DeltaProduct with nı (Siems et al., 2025), whose computation amount is about 2x that of HDLA, HDLA still shows clear performance superiority.
Our HDLA model and the generalized chunk- parallel algorithm together provide a foundation for future research rank-enhanced structured linear attention, showcasing promising prospects.
2 BACKGROUNDS AND RELATED WORKS
For notational conventions in this work, we use bold lowercase letters denote column vectors (e.g., qr), bold uppercase letters for matrices (c.g.. Q, O), and italic uppercase letters for Iearnable param- eters (c.g.. 6,). Note that any matrix without a subscript is constructed by concatenating its corre- sponding lowercase column vectors, e.g., Q denotes the column- concatenation of qı, 92. We also use lowercase letters to represent tensors of a single timestep with more than 1 columns. For instance, K. E XTau, and K is assembled by concatenating k1, k2, column-wise.
Unified Recurrent Form of Linear Attention. In linear attention, an input x, E Rdx1 is trans- formed into a group of query q E [dde11, key k, E Rdd Xrkv and value E Rd X at first:
qt = [x4, 0,), k. = 0x) =
(1)
The above transformation fęr fu, fo are typically linear functions, possibly with activation or nor- malization, and 0,, Oz, O, are their projection parameters. Then, hidden state S, E Rdaxd, decay matrix P, TRdx and the attention output o, E IRdux1 are computed as follows:
2

---

Published as a conference paper at ICLR 2026
P. fp(x (xx, 0p) R
(2)
S: = P,S + ki E gdxdd,
(3)
o: = S q E gdxx1
(4)
The hidden state S, seeks to compress information from arbitrarily long key-value pairs into fixed- size memory. The decay matrix P, balances the relative importance between historical information S,-1 and the incoming new information v Different structures of P, lead to different levels of model performance, as well as varying parallel forms and strategics of sequential parallelism.
The original purpose of linear attention. Linear attention (Katharopoulos al. 2020) is originally targeted at addressing the time and space complexity issue of Softmax attention. uses linear kernel functions to approximate the high-cost non-linear Softmax operation. Leveraging the associative property of matrix multiplication, it enables each key-value pair to be processed only once, and achieves (n) time complexity while compressing infinite key-value sequences into fixed size S..
The developmental trajectory of decay matrices. The evolution of linear attention methods has moved from the original variant (Katharopoulos al. (2200) lacking any decay mechanisms- cannot forget unimportant historical information to approaches with learnable constant decay such as RetNet (Sun al. 2023) and TransNormer (Qin al. 2024a). While these mitigate forgetting to certain extent, they remain insensitive to the relative importance between historical information S,-1 and newly arriving information k More recently, diagonal input-dependent decay mech- anisms, introduced by models such as GLA (Yang et al., (024a), Mamba (Gu & Dao 2024), and HGRN2 (Qin et af., 2024d), enable adaptive weighting of historical context but are constrained by their diagonal structure, leading to lack of cross-row interaction during hidden state updates. As result, these mechanisms permit only partial forgetting of old information without negative erasure. To address this, recent works have adopted input-dependent non-diagonal decay structures (typi- cally Diagonal-Plen Plus-Rank-1) and have demonstrated superior performance over earlier approaches. DeltaNet (Yang al. 2024b) and TTT-Linear (Sun et al. 2024) were the first to employ generalized Householder matrices as non-diagonal decay matrices. Gated DeltaNet further improves language modeling capabilities by incorporating a scalar forget gate into DeltaNet. RWKV-7 (Peng et al.] 2025) adopts a more general diagonal-plus-rank- decay structure, in which the diagonal terms are analogous the input-dependee repeats the recurrent step of Gated DeltaNet for 7ı, times at cach timestep, which is equivalent to ap- plying a Diagonal- -ainaal-Puu--aaeee Rank-n,, single-step rank-enhanced decay. The resulting decay matrix lacks strong structural properties, and its performance improvement is limited even as the computational amount grows or 2 times. Therefore, we aim to explore structured enhanced decay method that achieves greater performance gains, while incurring only limited additional computational cost compared to Planaaa-lllu--aaak----
decay used in GLA. Gated DeltaProduct (Siems et al. 2025)
Table 1: The structures of decay P. in different linear attention mechanisms (a, Qt 3r E IR; ke, At, a,, Wt, Rs. Rdxx1).
Chunk-wise parallel acceleration algorithm of linear attention. The core idea of chunk-wise parallel al- gorithms for linear attention is to di- vide the computation along the time dimension into chunks, sequentially compute the checkpoints of hidden states before entering each sequen- tial chunk, and then process the linear attention outputs of different time intervals in parallel. Lightning- Attention (Qin et al. (024bb) and Lightning-Attentionn QQnn et al. 2024c) address the parallelization
Model
P I al Diag(Ae). I Bekrk,
Original Linear Attention RetNet, Trame GLA, Mamba, HGRN2
DeltaNet, TTT-Linear
Gated DeltaNet
ar 3:kiki
Gated DeltaProduct (n,, iterations)
a. 3.k.k,
RWKV-7
Diag(w:) (a R.)
problem in the case of diagonal scalar decay, while Yang et al. 2002a) tackles the parallelization
for diagonal vector decay. ZeCO (Chou et al.) 2025) further addresses the communication bottle- neck in multi-GPU scaling based on previous algonthmms. (Gated) DeltaNet (Yang et al. 2024b] 2025) solves the parallelization for the case of diagonal plus rank- I decay. ParallelFlow (Cirone & Salvi 2025) provides a certain degree of parallelism for identity plus rank-n decay, but it does not accommodate the arbitrary diagonal terms that common in the decay matrices of linear attention.
3

---

Published as a conference paper at ICLR 2026
Test-Time training. If the hidden state S, is regarded the projection parameter of linear layer, then the autoregressive update formula for the hidden state most linear attention mechanisms can be interpreted as stochastic gradient descent (SGD) S,, usually aiming at next value predic- tion (using k, Sy to predict vt). This update process is referred to as Test-Time Training. TTT- Linear (Sun al. 2024) and DeltaNet (Yang al.] 2024b) were the first to interpret and design linear attention mechanisms from this perspective. Titans (Behrouz et al. 2025c) introduces mo- mentum to the stochastic gradient descent. Miras (Behrouz al., 2025b) proposes a broad unifying framework that integrates linear attention and Softmax attention under the view of Test-Time Train- ing, utilizing components such as memory architectures, memory learning methods, attention bias, and retention gates. ATLAS (Behrouz et al., 2025a) and MesaNet (von Oswald et al.) 2025) make improvements on the stochastic gradient descent (attention bias) objective by optimizing the average loss of all tokens within a sliding window or a global window, thereby achieving better performance.
METHOD
3.1 LINEAR ATTENTION WITH HOUSEHOLDER
DIAGONALIZED DOLLSHOLLDER-DAAGNVVAZZZZD DECAY
Our goal is to achieve better language modeling capabilities through extending the decay matrices to a broader, structured, and more expressive class, while simultaneously meeting efficiency constraints in parameters, memory, and computation. Specifically, our idea is to parameterize the Diagonal- Plus-Rank-2 decay structure by utilizing a certain kind of efficient matrix decomposition method.
3.1.1 EFFICIENCY CONSTRAINTS OF COMPLEX DECAY MATRIX DESIGN
Parameter, memory, and computational efficiency common challenges dur- ing the design of linear attention mech- anisms. When designing complex de- cay structures, we'd d like to revisit and refine these constraints, so
SWIGLU
Desay
de
P
limit the extremely broad design space and to preliminarily validate the practical- ity of our approach. (1) Parameter effi- ciency. The (d2) decay matrix should be obtained through O(dx) parameters, to maintain a balance with the parame- ter counts of 0o 0K and Ov, avoiding excessive parameters and learning over- head. (2) Memory efficiency. Each of the (dd)) decay matrices or their cu- mulative products should be compactly stored in (dg)) memory on average, matching the memory footprint of qr, k, and vi. (3) Computational efficiency. The cumulative product of decay matrices must maintain reasonable computational costs. Moreover, hidden state updates across sequential blocks should be enabled through concise one-pass matrix multiplications.
HDLA
Liver
Anow
Conry
Lieer
Owter
Masii Mafijoicationn
Activation Fanction
The architecture of HDLA, well as its inte- gration within a Transformer layer. Details like output gates and activation keys and values are omitted. See appendix C. for the exact architecture of HDLA.
Figure I:
3.1.2 HOUSEHOLDER DIAGONALIZED LINEAR ATTENTION (HDLA)
To efficiently parameterize complex decay Pt. is advantageous to decompose into simpler components through matrix decomposition theory. Note that any real symmetric matrix P, can undergo congruence diagonalization via some invertible matrix H. E RRde i.e., P. = H,A,H, Utilizing this inspiration, the parameterization of Pt can be reduced to two sub- nob-problems: (P1) Learning the diagonal eigenvalue matrix A,. (P2) Selecion of the invertible transformation H;-
For (P1), make the parameterization of A, analogous GLA's input-dependent diagonalized de- cay, equipping the model with fundamental capability to dynamically forget historical information. For (P2), adopt generalized Householder matrices our transformation operator, inspired by recent research of Diagonal- Pacoonal-PPuu-PPPe Plus-Rank- decay structure (Yang et al., 2224b) /Snn et al.] 2024)) Yang
al. 2025
Siems al. 2025). The corresponding hidden state update formulae follows:
4

---

Published as a conference paper at ICLR 2026
)A ) A, Diag(Ar) E R A (WA))) R
P. (I -I- B,k,k,
B,k,k,)) IR Rdxxds,
(5)
(6)
We make 3g (0.2) enhance the model's state tracking capability, following the conclusion of Grazzi et al. (2025). a-) is an activation function ranging in (0, 1), and we adopt sigmoid(-) here.
Compared with GLA, the only excessive parameter is the projection matrix (of ((dz) scale) map- ping the input X, into 3t, confirming the parameter efficiency of HDLA. Deduction of chunk-wise parallel algorithm the following section will verify its computational and memory efficiency.
3.2
GENERALIZED CHUNK-WISE PARALLEL ALGORITHM
3.2.1
DERIVATION AND RANK EXTENSION OF GENERALIZED HIDDEN STATE UPDATE RULE
During the derivation of HDLAA' chunk-wise parallel algorithm, we first reformulate its decay matrix special case of the Diagonal-Phe Plus--l-P--nk--22222 structure (see Appendix appendix C.2 for details):
P, D, A, E Rdaxdd, A,, E Rdax2
(7)
Based on the above reformulation, and to provide a foundational support for future research both theoretically and practically, we aim develop broader chunk-wise parallel algorithm for the following hidden state recurrent update rule, which generalizes the ranks of A,B, and V, to arbitrary values simultaneously (i.e., setting A,, B, Rdea K, E RddxxKK V (Rdo))
LD,D,,,,B. ),B,))s - E Bddadd.
(8)
3.2.2 RANK GENERALIZED CHUNK- WISE PARALLEL ALGORITHM
Notational conventions. Define two kinds of matrices' cumulative products as follows:
MP Pt, <<j D - D,, i i<j d = D e Rdxx1 i2j
P
(9)
L.
i2j
All timesteps in this work start at 1. The input tensors are partitioned along the sequential dimension into chunks of size C. We abuse the subscript [n] to refer tensors relevant to the n-th sequential
chunk. An E Id aTanC Bínj E gd xras KKn,) IR Xra. Vin) E &r.. are concatenated column-wise from the corresponding input tensors of each timestep inside the chunk, while S/m] =
S (n-1)C 1yc denotes the hidden state right before processing the first timestep of the n-th chunk.
Computation Flow. Since the linear attention of qnC+t over the first nC tokens can be coalesced into the interaction between qrC+1. Sjn]. and paC+t our method adopts a two-phase computa- tion scheme similar Lightning Attention (Qin et al. 2024b) and Gated Linear Attention (Yang et al., 20244): (1) Sequentially computing the hidden state checkpoints S(o): Sii]..... -IN-1j: and (2)
Computing the linear attention outputs O [o] .... [IN- across different time ränges in parallel.
These two computation phases correspond to the following eq. (10) and eq. respectively:
nC Σ
Sim) = PrC (n- l)c Sn
P
K.V. E Rd xd,
(10)
110+1
1-1+
1)++
$ -T (in1)//nnj-- 1|++
(n-1)C
1
Σ
V.K. -. p(
gr
(11)
(x-1)
inter- chunk attention
Intra-chunkl attention
5

---

Published as a conference paper at ICLR 2026
Rank Generalized WY Representation. Let P(n] PRC (n-1)C
I (n== D P. R xde,
(m)) = -t= 1) K, RR4x For efficient computation of eq. 10 and eq. 11
RC
becomes imperative to identify optimized representations for both P(n) and Hg
[n] that eliminate their
original dependence cumulative summation (Σ) and cumulative product ( operators.
Employing mathematical induction, we optimize the representations P [(x) and H[n] as follows:
P [m] = D
RC (n- Ijc B
m] Di- (n-1)e (K (M) MM BBny (m)
(12)
Un| Rddx and W(n) Rrax are core components of arbitrary rank WY representation:
U (mittri Xres (K (Knn) Am] [n]: 1) (I triu, Xrab (B A Amj
(13)
W [n] A [n] (T triur Xrab (B|n) A (n): 1))
-1 E RRdex
(14)
The above custom operator triur, xra xx.xnxx i) serves analogous to standard triu(Raxxa i) linear attention, except for treating each 71 72 sub-block of R,, Xn.raxn as a single element.
Anj [n| is obtained from [nj using the following element wiise multiplication (3) on each of its col- umn vectors (e.g., A Inla,draatt where is the time index inside the sequential chunk, and r the rank index. Bin) and K (x) are obtained similar to A Ang. but with element-wise division (2) instead:
A (nladrratt A An C d (n- (n--))+(t-1)) 1jC 1)++
(15)
[n]::treut B B Biladdaatt Bilaaraatr 1)-
K
K [n)attr @ d (n- a(n-1)0++ 1)C+
(16)
[n-1)C
Resulting Formulae. Leveraging the WY representation defined in eq. and eq. (14). eq. (TO)
and eq. (TT) can be reformulated into the following form, enabling efficient parallel computation of attention output in different time range, after sequential computation of hidden state checkpoints:
S/²] D RC
(n- 1je (I B(n) W S m)) n-1 + (n- Ijo (J []]
(K
VT)
[n] B UT))
(17)
0 [n] = S i- Qnj + Vmnjtriu, x1 (K T (m)) 0) (S (--1) W [m] + Um) (B [n] 0).
(8/--1)
(18)
Here only present some key conclusions. For detailed derivations, please refer appendix C.3
3.3 DISCUSSIONS
Understanding HDLA from the Perspective of Test-Time Training (TTT). If S, is regarded as the projection parameter of linear layer, then single step of hidden state update in HDLA is equivalent to the following three-step optimization process (see appendix C.5 for details):
Br V (min |lk] S.- 1!²),
S,1 S
(19)
2
S:
S
S.
(TraceeST,, diag )))
(20)
3.
S, S
(S [S, S.2 v. [1?).
(21)
HDLA is characterized by preemptively penalizing high inner product similarity between k, and each column of S,-1 (as demonstrated in eq. (19), ) which partially removes redundant information stored in S before performing gradient descent using delta rule YYang et al. 2024b) in eq. 21

---

Published as a conference paper at ICLR 2026
Comparisons between HDLA v.s. Gated DeltaProduct. Gated DeltaProduct performs n4 value predictions and optimizations at each timestep. According to Yang et al. (2024b), all its iterations within single timestep be merged into rank-enhanced iteration with Tab Tke na. How- ever, its coalesced Diagonal-Pluss Hiagonal-Plus-Rank-raa Rank-rab decay does not exhibit a highly structured pattern. We will demonstrate in the experiments that when n4 3, the computation amount is about 2x of HDLA, the performance of Gated DeltaProduct still falls considerably short of our method.
Table 2: Comparison on the computation amounts of HDLA, GDP2 (Gated DeltaProduct,
74 = 2), and GDP3 (Gated DeltaProduct,
3) of single recurrent timestep. We uniformly
calculate the computational cost of recurrent hidden state updates according eq. (17) (setting
C = 1), and omit the estimation of computation required by the cumbersome WY Representation.
Method
Fab
Tee
Input Projection
Hidden State Update
Output Generation
HDLA
2
d(3da do +
(8d, + 5)
d.de dedg
GDP2
d(3d; + 2d, + +3)
d (12d, + 6)
GDP3
3
3
d(Ad; + 3d, + +4)
d&(18d, (18d, + 9)
dedg
4 EXPERIMENTS
We've conducted series of experiments, ranging from synthetic tasks (MAD and Zoology), lan- guage modeling experiments, retrieval-based tasks (NIAH), image classification and ablation stud- ies, to comprehensively validate the effectiveness of our model. In the following, we use GDP2 and GDP3 abbreviations for Gated DeltaProduct when n, and n, 3, respectively. Both models incur significantly higher computational and memory overhead compared to HDLA, yet their overall performance still remains inferior to our proposed method. (See appendix for detailed settings)
Mechanistic Architectural Design (MAD). The MAD benchmark (Poli et al.) is composed of 6 kinds of scall synthetic tasks, and de- signed to evaluate model'ss core lan- guage modeling capabilities includ- ing in-context recall, memorization, information compression, selective copying and noise suppression, etc. The scores across all synthetic tasks are averaged to predict the model's performance at large scales, accord- ing to scaling law (Kaplan et al.
Figure 2: Accuracy on the synthetic MQAR task.
100%
Accuracy
75%
50%
25%
0%
256
512
1024
2048
Sequence Length
- HDLA
Gated DeltaProduct
Galed DellaNet
2020 Shen al. 2024).
Table 3: Performance Comparison on MAD benchmark aligned with MAD protocol Mem: Memorization. ICR: In-Context Recall
Method
Compression
Fuzzy ICR
ICR
Mem.
Noisy ICR
Selective Copy
AVG.
Softmax Attention
48.85
39.74
95.98
84.41
88.12
99.03
76.02
GDP2
39,40
10.59
99.29
49.84
95.06
97.68
65.31
DeltaProduct
40.77
14.16
99.85
46.08
99.66
99.95
66.74
Gated DeltaNet
41.41
12,90
99.73
55.64
99.40
99,91
68.17
DeltaNet
42.27
16.42
99.88
42.46
99.85
99.93
Mamba
48.20
10.24
86,90
89.48
94.50
82,14
68.58
HDLA
51.01
14.56
99.73
89.34
93.42
89.73
72.97
The results table demontttaee that: (1) The non-diagonal decay baselines suffer from severely impaired memorization capability, with scores not exceeding 60, whereas our HDLA performs well. (2) Compared to the five linear attention baselines, HDLA demonstrates balanced and comprehen- sive advantages across all tasks, and significantly narrows the performance gap with softmax atten- tion. (3) HDLA underperforms softmax attention on Fuzzy In-Context Recall task requiring accurate value prediction from keys interleaved wth arbitrary noisy tokens. This kind of limitation

---

Published as a conference paper at ICLR 2026
can be attributed to the strong recency bias (Pan et al. 2025) in linear attention mechanisms where the diagonal decay coefficients decrease cumulatively over time. As a result, HDLA cannot achieve high upper bound for attention scores on distant tokens, thereby failing to effectively aggregate information from important tokens distributed sparsely and discretely, and ultimately impairing its retrieval capability.
Multi-Query Associative Recall (Zoology). We conduct Multi Multi-Query Associative Recall (Zoology, Arora et al. (2023a) experiment against Gated DeltaProduct (ni = 2) (Siems et al. (0255) and Gated DeltaNet YYnn et al. 2025), with parameter scale aligned to 1.65M. The evaluated lengths include 256, 512, 1024 and 2048. See fig. forr the results. When the maximum evaluation length is extended to 2048, HDLA still maintains an accuracy higher than 81% 81% while the two baselines nearly fail to produce correct answers, demonstrating HDLA's advantage in recall ability.
Table 4: Perplexity comparison on language modeling, The parameter scales of the three columns from left right are: 0.4B, 1.45B and 2.8B, respectively. Wiki: Wikitext. (Merity et al. 2016) LMB: Lambada (Paperno et al. 2016).
Model
Wiki
LMB.
Avg.
Wiki ppl 4
LMB. ppl 4
Avg. ppl J
Wiki ppl
LMB.
Avg.
ppl
ppl 4
ppl
ppl 4.
ppl J
Linear Attention
HDLA
29.04
43.09
36.06
22.49
22.16
22.32
20.16
16,99
18.58
GDP2 Siems et al.
2025
30.98
51.59
41.28
23.51
25.79
24.65
20.94
19.82
20.38
GDP3
Stems ct al
2025
31.52
60.92
46.22
24.63
28.97
26.80
Gated Della Yang el
2025
30.06
56.07
43.06
23.09
26.56
24.83
20.47
18.74
19.60
DeltaNet Yang ct a.
30.75
58.34
44.54
23.74
31.14
27.44
21.66
23.72
22.69
HGRN2 Onn
30.87
47.81
39.34
23.26
24.70
23.98
20.93
19.69
20.31
Mamba2 [Dao & Go 2024
30.26
51.00
40.63
23.93
27.53
25.73
21.95
23.61
22.78
GLA Yang et al. 2024a
30.95
56.55
43.75
23.44
29.41
26.42
21.08
21.82
21.45
TM (Qin et al. 2024a
31.33
51.17
41.25
24.15
28.41
26.28
21.47
21.97
21.72
Soflmax-Attention
[lama Touvron et al. 2023
28.46
46.73
37.60
22,29
25.07
23.68
20.32
21.10
20.71
Language Modeling. We train parameter scales of all the models: 0.4B, 1.45B and 2.8B on 10B/50B token datasets sampled from FineWebFEdu. Perplexity results in table 4 demonstrate that HDLA surpasses all the selected linear attention baselines by notable margins, and even outperforms the Transformer-based architecture Llama (Touvron al. 2023). 7 that our method con- sistently surpasses both Llama and linear attention baselines in zero-shot commonsense reasoning. For retrieval tasks, our method is competitive all parameter scales, and achieves the best perfor- mance among all linear attention mechanisms when scaled up to 2.8B parameters. However, there still a considerable gap between our model and Llama retrieval performance. The reason is that the limited hidden state size of linear attention mechanisms fundamentally restricts their ability to perform in-context cross-step retrieval, both explicitly and implicitly (Wen et al.) 2025).
Retrieval-Based Tasks. We further trained Gated DeltaNet and HDLA models with 1.45B param- eters until the total number of tokens reached 50B (see table 7 for language modeling and retrieval results), and then evaluated the models on the retrieval-basee task RULER (Hsieh et al. 2024). As demonstrated by 6 compared to Gated DeltaNet, HDLA has a significant advantage in retrieval capability. Especially the S-NIAH-3 task, its accuracy leads by 3 31.4% and 58.2%.
Table 5: Comparison zero-shot commonsense reasoning and retrieval augmented genera- tion with 50B training tokens. We evaluate the models on BQ Clark et al. (2019), PIQA: Bisk et al. 20200, HS Zellers al. (2019). WG Sakaguchi al. (2021), Arc-e and Arc-c Clark et al. (2018), OBQ Mihaylov et al. (2018), SIQA Sap et al. (2019), Lockard et al.) (2019), SC Ra-
jpurkar et al. (2018) and FDA Arora al. (2023b). AVG-CSR: Average CommonSense Reasoning
accuracy. AVG-RET: Average RETrieval accuracy.
Model
BQQ
PIQA
WG.
Arc-
OBQ
SIQA
SWDE
SC
AVG-CSR AVG-RUT
ncc
ace f
acc-n
acc
1
acc-n 7
800
acc f
acc 1
ace
BC0 ?
80c 1
acc-n
Parameter Scale:
1458,
Number
50B
HILLA
1.45
73.50
57.33
57.62
73.44
38.14
41.60
42.02
41.40
36.76
16.61
54.81 53.92
31,59
Gaied DelaNoo
1.45
73.23
56.23
56.51
72.43
38.14
41.20
39.71
37.89
35.86
16.88
30.21
8

---

Published as a conference paper at ICLR 2026
Table 7: Comparison on zero-shot commonsense reasoning and retrieval augmented genera-
tion with 10B training tokens. We evaluate the models on BQ Clark et al. (2019). PIQA: Bisk
al. ) HS Zellers al. (2019), WG Sakaguchi al. (2021), I,, Arc-e and Arc-c Clark et al.
(2018), OBQ Mihaylov et al. (2018). SIQA Sap et al. (2019), SWDE Lockard Lockad et al. (2019), SC Ra-
jpurkar et al. (2018) and FDA Arora al. (2023b). AVG-CSR: Average CommonSense Reasoning
accuracy. AVG-RET: Average RETrieval accuracy.
Model
BQ.
PIQA
HS.
WG
Are-c
Arc-c
OBQ
SIQA
SWDE aec
SC
FDA acc
AVG-SRR
AVG-RET
acc
SCc
acc-n
ace
scc
acc-n
acc
uec 1
aec
acc
acc-n
Parameter Scale:
HDLA
61.50
66.41
40.48
51.14
00.65
28.58
31.80
58.13
10.26
21.78
3.197
47.46
11.21
GDP2
61.07
66.87
38.49
51.78
57.70
27.73
34,00
38.08
9.36
22.39
3.63
46,97
11.79
GDPS
57.53
3.36
Gared NetiaNer
58.41
667.63
39.41
31.85
58,
27.13
33.60
36.75
8.01
20.78
2.63
46.63
10.47
DellaNet
59.69
66.59
37.74
50.67
58,00
27.99
32,60
37.41
11.79
22.62
5.54
46.33
13.32
HGRN2
10.73
Mamha2
60.00
63.94
38.24
50.99
56.90
27.99
31 .44
38.38
13.23
27.92
4.99
46.23
15.38
GLA
58.53
67.41
39.50
50.91
59.97
27.
34,60
38.38
7.29
17.46
2.18
47.08
8,98
LM
59.51
35.
2,00
Liana
60.73
66.65
38.8%
$1.62
58.63
28.24
33.40
38.95
47.07
30.86
17.15
47.14
31.69
Parameler Scale:
1451, Number tokens:
T0B
HDLA
60.52
11.00
47.77
$2.38
67.17
32.68
35.00
40.84
21.69
28.23
8.17
51.06
19.36
GDP2
57.83
69.75
46.22
52.53
64.35
31.91
35. 60
38.89
17.82
27.98
6.99
49.61
17.60
GDP3
60,80
68.50
18
51.70
63.43
31.45
35.60
38.89
14.04
26.34
5.26
15.21
Gased DeliaNet
61.47
69.97
47.11
53.12
65.36
33.11
35. .40
40.84
20.43
27.61
7.35
50.80
18.46
DellaNet
61.31
69.31
44.32
53.04
65.32
31,23
34,80
39.61
21.87
26.91
10:25
49.87
19.68
HGRN2
60.70
Mamba2
60.46
69.70
45.00
51.78
63.43
31.23
34,60
39.87
22.23
29.42
9.35
49.51
20.33
GLA
57.31
69.31
47.25
54.06
66.46
33.79
36.60
39,82
16.29
23,83
4.81
50.58
14.98
TransNormeLLLMM
64,
4.26
16.59
Llama
61.68
69.42
46.89
53.20
65.82
30.89
35.40
39.82
62.29
38.47
39.38
50.39
46.71
Parameler Scale:
288, Number of tokenss:
T0B
61.13
$1.93
34.90
16.61
24.
GDP2
58.75
71.16
50.31
55.41
67.59
34.73
38.40
40.17
27.00
30.56
8.71
52.07
22.09
Gated DeltaNet
60.80
71.76
51.17
54.54
69.49
35.67
38.20
39.71
29.07
31,13
13.79
52.62
24.66
DeltaNet
71.16
47.79
67.13
29.12
HGRN2
61.56
70.57
50.49
53.04
68.90
34,81
39.00
40.43
28.44
29.19
14.61
52.35
24.08
Mumbu2
60.73
71.06
48.55
53.43
64.77
32.17
38.20
39.15
23.94
34,55
8.98
51.01
22.49
TransNrmeLLLM
58.59
70.29
50.04
54.54
68.35
33.96
35.60
41.76
24.21
29.42
7.62
51.64
20-42
Llama
61.10
70.89
50.36
56.20
67.38
32,51
36.20
40.07
61.57
36.23
41.02
51.84
46.27
Table 6: Accuracy different S-NIAH tasks for 1.45B HDLA and Gated DeltaNet.
Model
S-NIAH-1
S-NIAH-2
S-NIAH-3
Sequence Length
1024
2048
1024
2048
1024
2048
HDLA
100.0% %
98.8% 97.2%
96.4% %
52.2% 45.8%
82.0%
65.2%
Gated DeltaNet
99.6%
96.4% %
50.6%
7.0%
Image Classification. We conduct bidirectional image classification experiments on ImageNet- 1K (Deng et al. 2009). Baselines include Deit (Touvron et al. (2021) which is Transformer-bant architecture, and some other linear attention architectures. Results of baselines are directly borrowed from Chou et al. (2024). As show in table & HDLA performs better than most of the baselines.
Supplementary Experiments. appendix we provide the following supplementary experi- ments: (1) State expansion experiments on HDLA and baselines. (2) Fine-tuning on some hyperpa- rameters of HDLA (e.g. learning rate, the range of Be. the type of activation functions on k,, v.).
Table 8: Results of image classification on ImageNet-1k.
Model
Accuracy Param(M)
Model
Accuracy Param(M) Model
Accuracy Param(M)
HDLA 74.84%
6.1
MetaLA
75.33%
6.1
GDP2
73.81
6.1
Gated DeltaNet
72.33%
6.1
HGRN
74.40%
6.1
GLA
72.47%
6.1
Mamba 73.39%
6.1
Deit
72.20%
5.7
Throughput comparison. The training and inference throughput results for each model are pre- sented in table 9 where the inference throughput is compared only during the decode stage. We acknowledge that the throughput of HDLA is lower than all baselines we' ve adopted. The reason is, compared to Gated DeltaNet, the vector decay, the number of key-value pairs, well as the larger computational scale of WY Representation, all introduce significant additional overhead. In addition, found that the mismatch between Tab and Tke also hinders kernel speed optimization.

---

Published as a conference paper at ICLR 2026
Table 9: Training throughput and memory cost comparison between HDLLA and baselines. All
models have parameter scales aligned to 1.3B and are tested on single A A100-SXM-80GB GPU. Throughput measured in Tokens Per Second (TPS), and Memory Cost measured in GB.
Training
Inference
Model
Throughput (TPS)
Memory Cost (GB)
Throughput (TPS)
Memory
Cost (GB)
GDN
19277.18
39.33
15.3
5.3
DP2
16335.64
46.58
14.4
5.3
DP3
13492.85
54.99
14.9
5.3
DeltaNet
22060.12
39.74
14.7
5.3
GDP2
14546.89
46.61
13.9
5.3
GDP3
11769.86 866
55.06
13.7
5,3
HGRN2
19229.97
46.22
14.3
5.2
GLA
20998.87
36.30
14.7
5.2
HDLA
7340.01
52.56
13.1
5.3
Nevertheless, compared to softmax attention, HDLA still maintains a fundamental linear advantage in terms of linear complexity, and its decode throughput reach about 89. % of GLA.
CONCLUSION
In this work, we propose HDLA, linear attention mechanism with enhanced structured decay while maintaining reasonable computational and I/O cost, verify its effectiveness across various types of experiments, and obtained its theoretical justification from Test-Time Training perspective. Its robustness demonstrates that more sophisticated, structured and rank cank-cnhanced decay structures improve the effectiveness of linear attention mechanisms. We've also derived more general algorithmic framework of linear attention, enabling both Baggonal-plus-ranke key-value outer product updates, laying solid foundation for future research.
decay and rank-rke
Discussion and Limitation. Despite its superior experimental performance and sound theoretical explanations, this work has least the following limitations: (1) In terms of state expansion, this work only explores naive approach by altering the number of attention heads. Yet, to further bridge necessary to introduce more efficient multi-level
the performance gap with Softmax attention, and functionally differenciated state expansion methods. (2) Purely linearized hidden state update operations limit the model's expressive power. As suggested in Behrouz et al.) (2025c). it is impor- tant to appropriately introduce non-linear operations on the hidden state to enhance expressiveness. Nevertheless, HDLA has defined a more efficient utilization mechanism for single hidden state, and holds significant potential to inspire subsequent research in the rank-enhancement design trends. (3) Due to constrains on computational resource, most pre-trainn vee-trainnng experiments with parameter sizes of 1.45B and 2.8B (except for the 1.45B HDLA and Gated DeltaNet) did not reach the Chinchilla optimal number of tokens (at least 20x the parameter count), resulting in insufficient convergence.
6 ACKNOWLEDGEMENT
This work was partially supported by CAS Project for Young Scientists in Basic Research (YSBR- 116), National Natural Science Foundation of China (62325603, 62236009, U22A20103), Beijing Science and Technology Plan (Z241 77011100004224011 and shanghai NeuHelium Neuromorphic Tech- nology Co., Ltd.
REFERENCES
Simran Arora, Sabri Eyuboglu, Aman Timalsina. Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language mod- els. arXiv preprint arXiv:23/2. 049977 2023a.
10

---

Published as a conference paper at ICLR 2026
Simran Arora, Brandon Yang, Avanika Narayan Sabri Eyuboglu, Andrew Hojel, Immanuel Trum- mer, and Christopher Ré. Language models enable simple systems for generating structured views of heterogeneous data lakes. arXiv preprint arXiv:2304.09433, 2023b. URL https:
09433
Ali Behrouz, Zeman Li, Praneeth Kacham, Majid Daliri, Yuan Deng, Peilin Zhong, Meisam Raza- viyayn, and Vahab Mirrokni. Atlas: Learning to optimally memorize the context at test time, 23735
Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, and Vahab Mirrokni. It's all connected: A journey through test-time memorization, attentional bias, retention, and online optimization, 2025b. URL
13173
Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time. arXiv preprint arXiv:2501.0066) 2025c.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense natural language. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7432-7439, 2020.
Yuhong Chou, Man Yao, Kexin Wang. Yuqi Pan, Rui-Jie Zhu, Jibin Wu, Yiran Zhong, Yu Qiao, Bo Xu, and Guoqi Li. Metala: Unified optimal linear approximation to softmax attention map. In Advances in Neural Information Processing Systems, 2024. URL httts:://aaxx..ootttt
abs/ 241
10741
Yuhong Chou, Zehao Liu, Ruijie Zhu, Xinyi Wan, Tianjian Li. Congying Chu, Qian Liu, Jibin Wu, and Zejun Ma. Zeco: Zero communication overhead sequence parallelism for linear attention, 2025.
http::::////xx...rr///o org. 2200/5500000 010044
Nicola Muca Cirone and Cristopher Salvi. Parallelflow: Parallelizing linear transformers via flow discretization, 2025. URL httptttss///a.....rrttttt wwtp:///.rggb/////bb/o//o 20492004440
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pp. 2924 2936, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint TrXXv:1803.00455 2018.
Tri Dao and Albert Gu. Transformers ssms: Generalized models and efficient algorithms through structured state space duality. In Forty-first International Conference on Machine Learning, 2024.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16359, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255, 2009.
Riccardo Grazzi, Julien Siems, Arber Zela, Jörg K. H. Franke, Frank Hutter, and Massimiliano Pontil. Unlocking state-tracking in linear mns through negative eigenvalues. In Proceedings of the 13th International Conference on Learning Representations (ICLR), 2025. URL https:
//pperreview.ro wwwp://wwwww...tttmmemeememmm XxW
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling. 2024. URL htttps://openevviiiiee net/ forum?id tEYskw VY2

---

Published as a conference paper at ICLR 2026
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long- long-context lan- guage models? In The 1st Conference Language Modeling (COLM), 2024. URL https: //arxiv.org/aataa 2404. 06654
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0830/ 2020. URL httph:ttp//// arxiv. org/ abs//
08361
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In International Conference on Ma- chine Learning, pp. 5156-5165. PMLR, 2020.
Colin Lockard, Prashant Shiralkar, and Xin Luna Dong. Openceres: When open information ex- traction meets the semi-structured web. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 3047-3056, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-111 URL [ttttss::://ar wawaacaaanoooooooo wrtccolooy..rg////
1309
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. URL httts://axxiii org/ aasp//bb//1699
07843
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct elec- tricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381-2391, 2018.
MiniMax, Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang. Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang. Jiayuan Song, Jin Zhu, Jingtao Han, Jingyang Li, Junbin Xie, Junhao Xu, Junjie Yan, Kaishun Zhang, Kecheng Xiao, Kexi Kang, Le Han, Leyang Wang, Lian- fei Yu, Liheng Feng, Lin Zheng, Linbo Chai, Long Xing. Meizhi Ju, Mingyuan Chi, Mozhi Zhang, Peikai Huang, Pengcheng Niu, Pengfei Li, Pengyu Zhao, Qi Yang. Qidi Xu, Qiexiang Wang. Qin Wang, Qiuhui Li, Ruitao Leng. Shengmin Shi, Shuqi Yu, Sichen Li, Songquan Zhu, Tao Huang, Tianrun Liang, Weigao Sun, Weixuan Sun, Weiyu Cheng, Wenkai Li, Xiangjun Song, Xiao Su, Xiaodong Han, Xinjie Zhang, Xinzhu Hou, Xu Min, Xun Zou, Xuyang Shen, Yan Gong, Yingjie Zhu, Yipeng Zhou, Yiran Zhong, Yongyi Hu, Yuanxiang Fan, Yue Yu, Yufeng Yang, Yuhao Li, Yunan Huang. Yunji Li, Yunpeng Huang, Yunzhi Xu, Yuxin Mao, Zehan Li, Zekang Li, Zewei Tao, Zewen Ying, Zhaoyang Cong, Zhen Qin, Zhenhua Fan, Zhihang Yu, Zhuo Jiang, and Zijia Wu. Minimax-01: Scaling foundation models with lightning attention, 2025.
Yuqi Pan, Yongqi An, Zheng Li, Yuhong Chou, Ruijie Zhu, Xiaohui Wang, Mingxuan Wang, Jinqiao Wang, and Guoqi Li. Scaling linear attention with sparse state expansion, 2025. URL https:
7/ /aaxxiv.or//aaaaavv.. attp/////a/////0/0 16577
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint 1606.06600 2016.
Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Haowen Hou, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, Nathan Wilce, Johan S. Wind, Tianyi Wu, Daniel Wuttke, and Christian Zhou-Zheng. Rwkv-7 "goose" with expressive dynamic state evolution. arXiv preprint arXiv:2503. 14456, 2025.
Michael Poli, Armin W Thomas, Eric Nguyen, Pragaash Ponnusamy, Bjöm Deiseroth, Kristian Kersting, Taiji Suzuki, Brian Hie, Stefano Ermon, Christopher Re, et al. Mechanistic design and scaling of hybrid architectures. In Forty-first International Conference on Machine Learning.
Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for quence modeling. In Advances in Neural Information Processing Systems, volume 36, pp. 33202- 33221, 2023.
12

---

Published as a conference paper at ICLR 2026
Zhen Qin, Dong Li, Weigao Sun, Weixuan Sun, Xuyang Shen, Xiaodong Han, Yunshen Wei, Bao- hong Lv, Xiao Luo, Yu Qiao, and Yiran Zhong. Transnormerllm:: A faster and better large lan- guage model with improved transnormer, 2024a.
Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Various lengths, constant speed: Efficient language modeling with lightning attention. In Forty-first International Conference on Machine Learning, 2024b. URL
net forum? ?id=
Lwm6TiUP44
Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Lightnnng attention-22 2: A free lunch for handling unlimited sequence lengths in large language models, 2024c. BtLhttts:// /ttp///axvvvv//aa///i//.../.aaaa orgoorgg//sss.. abs/ 2201241111.1 04658
Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. HGRN2: Gated linear RNNs with state expansion. In First Conference on Language Modeling, 2024d. URL BRLL/ttss///ppeeeeeae
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In Iryna Curevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, Mel- bourne, Australia, 2018. Association for Computational Linguistics. doi: 10. 1/1188665111111-11111
URL
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Common- sense reasoning about social interactions. In Proceedings of the 2019 Conference on Empiri- cal Methods Natural Language Processing (EMNLP), 2019. URL https::/axxxv....
abs /104/ 09728
Xuyang Shen, Dong Li, Ruitao Leng, Zhen Qin, Weigao Sun, and Yiran Zhong. Scaling laws for linear complexity language models. arXiv preprint ar Xiv:224006 166900 2024.
Julien Siems, Timur Carstensen, Arber Zela, Frank Hutter, Massimiliano Pontil, and Riccardo Grazzi. Deltaproduct: Increasing the expressivity of deltanet through products of householders. In ICLR 2025 Workshop on DeLfa, 2025. / wbs./a/////a/o...eaaa..../eaaaea /502.
/wtp/aaxivvv/vva///vv///a.e org
10297
Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin. Learning to (learn at test time): Rons with expressive hidden states. 2024. URL httpts///axxi... org/ abs/2407 02620
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor transformer for large language models. arXiv preprint arXiv:2307 0,0.0261,, 2023.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mlp-mixer: An all-mip architecture for vision. In Advances in Neural Information Processing Systems, 2021.
Hugo Touvron, Matthieu Cord, Mathhij Douze, Francisco Massa, Alexandre Sablayrolles, and
Herve Jegou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, volume 139, pp. 10347-10357, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Ilama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 5998-6008, 2017.
13

---

Published as a conference paper at ICLR 2026
Johannes von Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari, Songlin Yang, Maximil- ian Schlegel, Kaitlin Maile, Yanick Schimpf. Oliver Sieberling, Alexander Meulemans, Rif A. Saurous, Guillaume Lajoie, Charlotte Frenkel, Razvan Pascanu, Blaise Agüera y Arcas, and João Sacramento. Mesanet: Sequence modeling by locally optimal test-time training, 2025. URL
05233
Kaiyue Wen, Xingyu Dang, and Kaifeng Lyu. RNNs not transformers (yet): The key bottleneck on in-context retrieval. In The Thirteenth International Conference on Learning Representations, 2025. httptttst/pppeeeeeeeeee htt.://wwww..ttttooooooooooaaaaa
Songlin Yang and Yu Zhang. Fla: A triton-based library for hardware-efficient imple- mentations of linear attention mechanism.
Flasb-linear-attentt
2024.
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with harlware-effficien training. In Forty-first International Conference on Machine Learning. 2024a. URL
net forum? id=1a5XvxPUJJ
Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transform- ers with the delta rule over sequence length. In Proceedings of NeurlPS, 2024b.
Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. In Proceedings of the 13th International Conference on Learning Representations (ICLR), 2025. hal.httss: / openrevi ew net. fo
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer actually what you need for vision. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10819-10829, 2022.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties. Association for Computational Linguistics, 2019.
A DECLARATION OF LARGE LANGUAGE MODEL (LLM) USAGE
To make the language more fluent and smooth, we've used large language models (LLMs) for pol- ishing during the writing process. We assure that all methods and experiments have been conducted manually and are authentic and valid.
B SUPPLEMENTARY EXPERIMENTS
B. STATE EXPANSION EXPERIMENTS
The hidden state size S of linear attention mechanisms can be computed by the following formula, where п is the number of attention heads, and dę and de are the total dimensions of the keys and values, respectively:
dk
de
dedg
S Thh
(22)
Tig
Therefore, without changing dy and d.,
adjust the hidden state size by altering 1i4-
MAD experiment after state expansion. table 3 we follow the MAD protocol by setting all linear attention baselines n4, 8, d₁ = 128, and d, 128, resulting in aligned per-layer hidden state size of S = 2048. Here, we naively achieve state expansion by setting n, = 4, so that the pay-layer hidden state size of each linear attention model is aligned to = 4096.
14

---

Published as a conference paper at ICLR 2026
Table 10: Performance comparison on MAD benchmark after expanding the hidden state size from 2048 to 4096. Mem: Memorization. ICR: In-Context Recall.
Method
Compression
Fuzzy ICR
ICR
Mem.
Noisy ICR
Selective Copy
AVG.
GDP2
41.69
19.96
99.86
64.46
99.80
99.93
70.95
DeltaProduct
42.74
21.35
99.93
52.74
99.79
99.96
69.42
Gated DeltaNet
44.03
18.34
99.89
66.85
99.87
95.61
70.77
DeltaNet
43.76
24.08
99.94
42.32
99.96
99.92
68.33
Mamba
44.82
12.21
87.24
89.25
88.74
83.08
67.56
HDLA 48.47
18.34
99.99
89.24
94.42
94.55
74.17
The results in table 10 shows that after state expansion, the average score of HDLA improves by 1.20, still significantly outperforming other linear attention baselines, and is only 1.85% behind Softmax Attention in table demonstrating the effectiveness of HDLA under state expansion.
Language modeling results of HDLA after state expansion. The results in table 11 demonstrate that after state expansion, the commonsense reasoning ability of HDLA remains nearly unchanged, while its retrieval performance shows a clear improvement of 2.42% and 3.37%.
PS (B)
8A
BQ
PIQA
WG
Arc-2
Arc-c
08Q
SIQA
SWDE
SC
FDA
AVG-CSK
AVG-RET
0.17
3
58.01
63.93
33.46
50.51
53.87
25.00
30.80
37.56
17.76
2.09
44,14
9.62
0.17
12
59.11
64.74
33.32
49 80
52.78
26.37
30.40
36.90
4.95
15.65
1.00
44.18
7.20
0.4
61.38
39.43
13.32
23.32
13.91
0.4
12
58.81
67.57
39.42
51.22
60.35
28.33
33.60
38.79
8.10
21.08
2.45
47.26
10.54
Table 11: Commonsense reasoning and retrieval results HDLA before and after state expan-
sion. d₃ dy 768 at 17B parameter scale, while dę d, 1024 at 0.4B parameter scale.
B.2 HYPERPARAMETER FINE-TUNING EXPERIMENTS
Fine-tuning experiments learning rate. In addition to the 3e-4 learning used in the main text, we also compare a range of learning rates (2.0e-4, 2.5e-4, 3.0e-4, and 6.0e-4) following the setup in Dao & Gu 2024). The results show that the models generally perform better with the relatively large learning rate of 6e-4. Due to computational resource constraints, we have not applied this setting to the language modeling experiments 1.45B and 2.8B scales.
Model
PS(R)
BQ
PIQA
HS
WG
Arc-e
Arc-c
QBQ
SIQA
SWDE
SC
FDA
AVG-CSR
AVG-RET
GDN
0.2
2.0e-4
59.4
63.1
31.9
50.0
52.0
25.3
29.6
37.2
4.8
13.2
2.0
43.6
6.7
GDN
0.2
2.5e-4
53.8
64.0
32.3
51.0
51.9
26.0
31.2
36.4
6.1
14.3
1.4
43,3
7.3
GDN
0.2
3.0e-
60.1
64.1
32.7
50.0
53.6
25.3
30.8
36.7
5.9
14.3
1.0
44,2
GDN
0.2
6.0e44
56.9
64.4
33.9
52.8
54.6
25.1
31.2
37.1
6.6
19.7
2.5
44.5
HDLA
0.2
2.0e-4
61.6
63.8
32.5
50.0
54.5
25.0
31.2
37.5
6.5
15.4
1.9
44.5
HDLA
0.2
2.5c-4
61.4
64.9
33.4
49.5
55.6
26.1
30.8
37.6
6.8
17.1
1.4
44.9
HDLA
0.2
3.0e-4
53.8
64.3
33.5
49.9
53.6
24.9
29.8
38.6
7.7
16.9
1.1
43.5
HDLA
0.2
6.0e44
50.6
65.2
34.4
50.9
56.0
25.6
33.8
38.3
8.6
20.2
1.8
44.4
10.2
GDN
0.4
2.00-4
58.8
67.0
38.1
50.2
59.1
26.5
32.8
39.0
9.7
20.5
2.8
11.0
GDN
0.4
2.5e-4
58.0
66.1
38.4
51.9
59.3
27.3
33.2
38.8
9.7
21.3
2.7
46.6
11.3
GDN
0.4
3.0c-4
58.4
67.6
39.4
51.9
58.4
27.1
33.6
38.2
10.9
23.8
2.5
47.4
12.4
GDN
0.4
6.0c-4
59.4
67.9
40.6
50.8
61.7
28.
32.6
38.2
10.9
23.8
2.5
47.4
12.4
HDLA
04
2.0c-4
66.6
39.0
50.2
59.8
28.0
37.7
9.5
22.0
2.5
11.4
HDLA
0.4
2.5e-4
59.4
67.5
39.3
51.1
59.7
27.7
33.4
37.8
10.0
21.8
3.7
47.0
11.8
HDLA
0.
3.0c-4
61.5
67.4
40.5
51.1
60.7
28.6
31.8
21.8
3.1
HDLA
0.4
6.0e-4
57.6
67.1
41.6
50.8
60.4
27.9
33.6
39.0
12.1
23.7
4.3
47.3
13.3
Table 12: Commonsense reasoning and retrieval results when fine-tuning learning rate.
Fine-tuning experiments 8 range in HDLA.
15

---

Published as a conference paper at ICLR 2026
PS(B)
BQ
PIQA
HS
WG
Arc-c
Arc-c
OBQ
SIQA
SWDE
SC
FDA
AVG-CSR
AVG-RET
0.22
53.76
64.25
33.52
49.83
53.58
24.91
38.39
7.65
1.09
43.34
8,53
0.17
0.1
55.11
63.66
33.33
50.83
54.84
25.77
32.2
37.46
7.29
16.99
1,09
44.15
8.46
0.2
67.41
40.48
51.14
00.65
28.58
31.8
38.
10.26
21.78
3.09
47.46
11.71
00.1
60.52
68.01
39.88
50.99
60.44
28.33
33.8
38.02
11.25
23.83
2.45
47.50
12.51
Table 13: Commonsense reasoning and retrieval results of HDLA under different Br intervals.
Fine-tuning experiments on the key/value activation function in HDLA.
PS(B)
Act
BQ
PIQA
HS
WG
Are-e
Are-c
OBQ
SIQA
SWDE
SC
FDA
AVG-CSR
AVG-RET
0.17
SI.U
53.76
64.25
33.52
49.88
53.58
24.91
23.8
38.59
7.63
16.85
1.09
43.54
8.33
0.17
ReL.
59.30
64.36
32.91
50.51
52.74
25.26
30.8
36.80
6.39
16.62
1.00
44.09
8.00
0.17
1+elu
59.66
64.25
32.97
50.59
53.54
24.66
30.4
37.15
6.30
17.33
1.54
44.15
8.39
SiLU
61.50
67.41
40.48
51.14
60.65
28.58
31.8
38.13
10.26
21.78
3,09
47.46
11.71
ReLU
61.71
67.63
39.65
51.78
59.68
27.56
31.0
38.49
10.53
20.44
2.54
47.19
11.17
1selu
61.28
65.94
39.00
50.67
59.51
27.82
35.8
37.72
10.62
24.80
3.36
47.22
12.93
Table 14: Commonsense reasonnn and retrieval results of HDLA with different activation functions.
C DETAILED ALGORITHMIC RESULTS
C.1 THE EXACT ARCHITECTURE OF HDLA AS A TOKEN MIXER
Here we present some details omitted in fig. using mathematical formulae:
q. 0qx, E Rdex1, k. = xx E Rd = 0yx, E Rd.x1
(23)
B. 03X; R, Ae Oxx, E RRdxx1
(24)
q = (iLU(q)), -iUUUK)),,, =-SiiL((v))
(25)
k, norm/, (k:)
(26)
S. (I -B,k,k, Diag(A:))( B.kkk, S ) k.
(27)
Yt = S.S. E Rd.x1
(28)
g: = x,0, E Rd.x1
(29)
0, y: ① gr Rdxx1
(30)
C.2 COMPUTATION OF A, AND B,
Consider factorizing each element of A, the product of its square roots, then coupling them with the left and right Householder transformations H,. This yields the reformulation of P as follows:
P. (Diag( Vxx)= - B,k,(k, @ vxi")) (Diag(
(Diag(VA))> (Ar (r, O vo(k..
(31)
This form of P, is a special case of two diagonal-plus-ranke matrices' product:
P = (Dr,) k()) (P) a2,
(32)
Utilizing the compact WY representation of diagonal-plus-rank matrices' cumulative prod- uets (Yang & Zhang 2024. P. can be rewritten as the following form:
P; = -.A,
ddd
xd, A,, B, X2
(33)
We demonstrate the detailed formulation of A,, B, follows:
16

---

Published as a conference paper at ICLR 2026
P. (D=(D,(1)))) a1 (a)) ai,())b(((2))
= DD,(1) - ax, b De.(2) - -Da,(1) - (a) b(,e0)
Let:
D. Da, Dr((2) E Rdx
(34)
A, a1(11)
(D2,)) (a)a)aa)))a.)))) E Rddxx
(35)
B. = Dr.22)ba br,22)] E Rdxx
(36)
Then P, can be rewritten as eq. 33 's form.
C.3 SUPPLEMENTARY DEDUCTION THE FORWARD CHUNK- WISE PARALLEL ALGORITHM
For deduction of cumulative products of the decay matrices P1, ... observe that:
P1 D1 A
P2 (D) A, (D, A²B,))
DD,D2 A D2 (D1 A,)
P3 (D,D, A, (D, A, A,BJ)
D,D,Ds A,D,D,
-(D, A, A D3 (D,D2 A, -D, A,B.)A,B])) AsB]
Suppose:
P. D
W.B.DD
Then:
P P41= =
D: - A411
i=1
= DD
W,B. D 1-1
(D's Σ W.B.D.)).. B
i=1
1=1
H+1
D
D
i=1
(I
W.B. D +1
Where:
17

---

Published as a conference paper at ICLR 2026
W, = (D) 1-
W,B,
[-1
B
= -(I-
W
(A,
D
(I-
WB
i=1
For compact form of S₁, S
observe that:
S = V
S SD A, V2K.
= =V,K, - V,K A, + V2K3
S3 =S2 A, V3Kg =V,K, D V2K, D + V3K, V,K, A
- -(V,K V A&B + As) B
Suppose:
S, (VK] UBB Di
Then:
- + VoIK
E K D + V ( U.B.).AAL B
H1 Σ KV - D
H1
= Σ T - UBB D
Where:
U, Σ V. - UBB UUB A
Σ -
C.4 BACKWARD CHUNK-WISE PARALLEL ALGORITHM FOR LINEAR ATTENTION WITH MAGGONL-PPLUS-RANE DECAY STRUCTURE AND RANK-TKe KEY- VALUE UPDATES
For the sake of simplicity, make the following definitions:
18

---

Published as a conference paper at ICLR 2026
A [X] (n-1)C
a (1=
(n-1)C-2
doC
E Rdx
(37)
(n-1jC
[n-1)c]
A [n]
d 1)0+1
dic (n-1)0+2
d
E R2
(38)
A [g]
1
(n-1)C-1
d=C-1 (n-1)C]
E R
(39)
C)C
[n-1JC
For [nj. [n) ' @ operation with K A In] or [n), (nj. A are repeated across the sequential
dimension in interleaving manner (analogous torch.repeat forch.repeatimmerleavy interleave) before taking element- wise multiplications.
Let's review the forward formulae of linear attention with diagonal-plus--rnk-raa decay structure and rank-rkw key-value updates:
U [m] V [n] triu, No Xras (K A (n]- 1) (I triu xrab (B (m)
E Redex
(19)
I triu xna (B A (n| 1)) -1
W [|]
A [n]
E RRdexrac
(20)
S I
W [x]] + [n]]
(40)
[m]
[n] + Vinitri x1 (K (n) 0) tmittriu a (B [m] Q (n) 0), S In-1]
(41)
S|]] DaC (1-1) + [M AmJ) V [n] (Bnn) C Ama C [n]
(42)
In the following subsections, we deduct the gradient of C(n] and - first, which needs to be
computed serially. Then, the gradient of (n), K m). V [m). An]: [mm] of each chunk can be computed
in parallel. Finally, we will derive concise form of the decay matrices' diagonal term.
C.4.1 DEDUCTION OF (C) AND aS [-11]
Since Cjn] participates in the computation of [n] (eq. 41
and the update of Sjm] (eq. (42 ). its
gradient is composed of two parts:
8C = -0 trill raa (Q TT
Bin) (x). 0)
+ as T [ O (nm)
(43)
8C jntaraa
8C [n],
Since -i- participates in the computation of C(n), [n] as well as the update of (inj. the gradient of S [n-1] is:
0S [- = OC + 00 In
D nC
(n- 1)c OS
(44)
as n. 1jiiate
as n ]|jitter
C.4.2 DEDUCTION OF 0Q[n], OK jn], 0V [n]: AAi AND 3B()
Since Qnn] participates in the computation O both intra-chunk and inter-chunk, its gradidient composed of:
aQ intrapppar (K [n] Am) triu x1 [n] 00 (n).0)
(45)
0Q mn].ntra.part2 = B nj o A pn) Ariitru
80 [n]. (n
(46)
eQ in]iinter -i [n)
(47)
iQimi = aQ in].intra.part. + aQ (n) intr,
8Q (m] inter
(48)
19

---

Published as a conference paper at ICLR 2026
Similarly, the gradients of [n),, B [n] [x)) relevant to rank-enhanced gated linear attention can be derived as follows:
o [n),gla =
(Q(m))
@ xrae (00 T V [mj.0) 0)) Ø [m] + (ÖS V (m) @
[m)
(49)
(m]
2K pulaaaiiiit
0K (nl.ala.
gi = - =0((0m)) 2 A Anj]trillxx (00 CinI, 0)) 2 Ann + (-(8S CIn)) O O
(50)
[n) iila.inter
V = 80 In trih xrao (Q K [n]: 0) + STo Duc
DjC m
(51)
@V (nniintra
@V (n))inasr
@V (ni.ga
80 [n] trill xra (Q Bi ml 0) + + Dac (n-1)cB 70
(52)
80 (m))itnr
Now we consider the gradients of each input corresponding to the arbitrary-rank WY compact rep- resentation (eq. (13) and eq. (14)) The matrix inversion operation can be avoided in the backward
pass, utilizing the following observation of u, 's recurrent definition.
(v, T -u,,,
U
0at.part1
[Kyv,- byu, Jou.
i=1
+ 0A
K [n initu xrao (V [n 0U mn. 1) - Bin triu, triu xna (UT, OU [x)
AA partt
Anx
(K(m)
(m) Amj xrai (V UU (m),
An O (B [n] 2 Amjuuriu xres (U OU
Imi
(53)
W. w = (I
Wb; (i) o D
a(a @ D6-1) = (1
(b, 2 D )ow,
= ((Ap)) 4 A (mn) aW [m]
B [n] @ A Anrtriu W (xj ow (m). 1)
Am 3 (o [B Ø Anjutri (W [nj aW [n). 1))
- 0A par12
(54)
0A [x]
A pmi, 0A ml]part2
(55)
u, Σ (vi D3) u, o D.) @ D
a(b, @ D u. du, D
Tb., part1 = -(1 Dj)) Σ
(a 2 Dj du u:
#=
a (). wyrpp (10A(m) (AAm) Am (aU) [n]. 1))
(56)
20

---

Published as a conference paper at ICLR 2026
Wi I Σ wb Dj)
(as c D
(b; 2 D ww; (w, 3 DT
@bi = -11 Dz) Σ (aa 3 jow] w.)
j=it1
a nn wyy.part -(10 (IA(m) o At xras (oW W [n). [n]]
11)
(57)
äB 8B [nl = - 2 in)) ((Am)) @ Anj (OU U U[n): -1)
- (1 2 (m)) (A @
(oW W [n]: -1)
(58)
Similar observation and deduction yields the following results:
Oki,y
D "auuu, Vi
3 g=i+1
=(1DD))
Σ (Di-la;)0uf vs
j=it1
-10D:) (aa))uu j=i
[Kin wyy.wy [1ØD] D0D 10D o (ai Vin: -1))
+ 0K |n),wy 1110App o (Amm) O triil (OU V [n] -1)
(59)
(v.(k, D u,(b; D)) o D
U1
Ovi,y
Σ
Ou,(a, 2 DT- (k: @ D3)
j=1t1
0V (m].wy = (U tritrran XTr ((Am) @ A pa) (K @ Am)):-1)
(60)
If tensor participates the computation of WY representation and rank-enhanced gated linear attention at the same time, then its total gradient is the summation of gradients relevant to the former and the latter computations. For example:
@V
OV (n].gla
OV
[n]wy
(61)
C.4.3 DEDUCTION OF ( AND σÂ [n]
Notice that the diagonal terms of Diagonal-Plus- Planal-Plus----rraaa decay matrices applied on other input tensors, using their chunk-wise cumulative products, and they are applied on Q and Aqri (Kj)) and B(n) in element-wise multiplication (division) manner:
21

---

Published as a conference paper at ICLR 2026
Pmnaat (mlec O 1)-4)
(n-1)C
= A (m)aitaat (n-1]]+((-1)
brastr
[n-1]C
B ioli,broott
B Bol
(n-1)C
(-+1
K inlesteruut
(n-1)C+1 (n-1)C
Now, we'd like to derive the gradient for the chunk cumulative sum of the logarithms of diag- onal decay terms, i.e. d==1) dda=1)c+2
(n-1)0++1
For simplicity, consider the special case when Tab = 1, and the result can be easily generalized to arbitrarily chosen rab-
Let's derive the gradient of d ((=11)++1 and d first, for each 1 S C:
(n-1)C
(n-1)C
0d (1-)C+++ (0q 3 (n-1)C++)
(n-1)C
(0k -Ok 1)0++ ((a-1) @ (d (ida-11c- o aa-ntctt
(n-1)0
(n-1jC
(62)
Od (t-t-)))+(- = (0A 3 ann
(n-1)0
-(0bin-1JC++ ((n-))C+1) @ (d (d (n--))tt(-1))
dd (
(n-
(n-1je
(63)
Given vector y, the relationship between dy and Ölogy is:
alogy=y(0g = =y(0y)
Thus:
(a (0qin) @ 9(n 1) O (1-1jC+r (x-1)C
(ök (ok 1((-1) (d (1=1))(+)
(64)
Ölogd( (1-1)0+(1-1) (Da 1JC++ 1 (n-1)c+(1-1)
(n-1)C
(n-1)C
(ōbj.) @ bin) Ø (d (4 (1-)C++(-11)
(65)
Notice that:
0q 0 0qp)) 2 d (n-1)C+t
(66)
In-1)C
0a m Oa
(n- (n-1)++(--1)
(67)
1)C
0k 0k = 0k(n-11C4+ ①
(n-1)C+
(68)
(n-1)C
0b 0b (b = Ob(n-1)C++ (n-1)C
(n-1)C++
(69)
(70)
Substitude them into (eq. (64 and eq. 65 ) the result is:
alog (n-1)C
(1)
(dq(n-1)C++ 2 (in=1))++)
(ōk @ k(n-1)0++) - (Ob @ b(-+))+
(71)
Ölogd (n-1))+(1-1) (Da(n-1jC++ C a(n a(n+))
(72)
(n-1)C
22

---

Published as a conference paper at ICLR 2026
Review that Anj (A))) is the column-wise concatenation of d [<=-1] n-th sequential chunk:
(d (a(=-1)c+(t1))
(n-
inside the
[n]
d [n- 1JC
(n-1)0+2 [n-1]C
d d
[n-1)0]
(73)
d (2-1)C I)C
[1+1
gnC-1 (n-1)C
(74)
(n-1)C 1JC
(n-1)C
Therefore, the corresponding chunk-wise parallel forms for [n] and ØlogAin) are as follows:
(logA(n) (n) = aQ [n] C Qn] [n] oK (n) AlogA(n) AA (m] [n]
2 Kjn]
[B(r) 2 B
(75)
(76)
For arbitrarily chosen and Tke. define the following rankgather,,c operation. Suppose the operator matrix corresponds to "rank"
Eaxc = rankgather, , (E (E(rc))
(77)
Then Edxc is defined as follows:
E.
E
100044
(78)
==
Simply utilizing the above operator, we can extend eq. (75) and eq. (76) into rank-ras low-rank term of decay with rank-rku key-value updates:
OlogA[n] [m] 8Q mm) Q
rankgather, C. (0K()) Km.)) rankgather,...c (nn)
(79)
ŌlogAp1 rankgather,.. res, C (0A O Am))
(80)
C.5 DEDUCTION OF HDLA's ' EQUIVALENT TEST-TIME TRAINING FORMULAE First, let revisit the optimization problem:
3r
S min
|k, S
(81)
2 8t-1
(S )1)S
S,2 min
(82)
3 min [k, St. v, /
S, 2 S,
(83)
For the first optimization subproblem, online SGD yields:
B min | 11
= V.
(84)
2 St-1
(85)
-S - -Bikkk S
(86)
(87)
(I - B.kk, S
(88)
23

---

Published as a conference paper at ICLR 2026
For the second optimization subproblem, online SGD yields:
St,2 = St Vs., ("Traco(ST diag( - )s)S
(89)
(90)
St,1 (I- diag())))S.,,
(91)
(92)
diag(Ar)S.,1
(93)
(94)
= diag(Ar) B,k,k, )
(95)
For the third optimization subproblem, online SGD yields:
Bt
S, = S1,2 Vss
min | S1.2 -
v /
(96)
2 812
(97)
=S,,2 - -Bik St.2 (K,T
3e vi))
(98)
(99)
(I- Bekck, )8², k.v:
(100)
(101)
= -I- 3,kkk, )diag(A B,k,k, ) k.v
(102)
Thus, this shree-step optimization problem yields the HDLA recursive formulation.
D EXPERIMENTAL DETAILS
Mechanistic Architecture Design. In strict compliance with the MAD protocol, we employ a two layer token mixer-channel mixer architecture, where each layer's linear attention is aligned with hidden state of dimension of 2048 (8 attention heads with dş = dg = 16), and run all experiments on NVIDIA H200 GPUs using bfloat16 6 precision. For state expansion experiments, the number of attention heads has been changed to 4.
Zoology (Multi-Query Associative Recall). The learning rates swept by: np.logspace(-4, -2, 4) for sequence length 256, np.logspace(-5, -3, 4) for sequence length 512, [le-5, 5e-5, le-4, 5e-4, le-3, 5e-3, 1e-2] for sequence length 1024 and 2048, and take the best result from all learning rates. The parameter scales are aligned to 1.65M.
Language Modeling, We trained models on fineweb-edu, including small-scale and large-scale ver- sions. For the small-scale version, we trained for 10B tokens with a learning rate of 3e-4, sequence length (seqlen) of 2048, and total batch size of 256 (num gpu x batch per gpu x grad acc). For the large-scale version, we trained for 100B tokens with a learning rate of 3e-4, seqlen=8k, and a total batch size of 128. All experiments were conducted on 8/32 A100 GPUs. We used FLA to imple- ment the model, the Flame framework for training, and Im-eval-harness for evaluation. We report results on wikitext, lambada_opent openbookqa, social.iqa. swde, compleeion and fda. For wikitext (word-pprpless) bada_openai (perplexity), as well as for swde, gquad_completion and fda, we report the Exact-Match (EM) score; for hellaswag, are challenge, and openbookqa, report acc norm; for the rest, port accuracy (acc).
boolq, piqa, hellaswag, winogrande, arc_easy, arc_challenge,
and lam-
Image Classification. Each model is trained and evaluated on 4 NVIDIA A800 GPUs using Pytorch DDP. The input size of ImageNet is 224 x 224. Following Deit, the batch size is set to 2048 during 300 training epochs with cosine decay learning rate whose peak value is 2.4 10-3 33 The warmup
24

---

Published as a conference paper at ICLR 2026
epochs is set to 20. We choose AdamW (31 0.9, B2 0.98) with 0.05 weight decay as the optimizer. Note that we do not use cutmix or mixup during the training. Results of MetaLA Chou et al. (2024), HGRN Qin et al. 2022), GLA Yang al. (2024a), Mamba Gu & Dao (2024) and Deit Touvron et al. 2021 are directly borrowed from CCho et al. (2024).
25


Reviews:
[{"review_id": "ClB7PLKz41", "reviewer": "Reviewer_pRZx", "summary": "The authors propose Householder-Diagonalized Linear Attention (HDLA), a new linear attention formulation that generalizes prior diagonal-plus-rank-1 decay structures to a more expressive diagonal-plus-rank-2 form. Their approach uses efficient matrix decomposition via generalized Householder transformations to achieve increased expressivity without significant computational or memory overhead. The authors also propose a rank-generalized chunk-wise parallel algorithm that extends parallel computation to arbitrary diagonal-plus-rank-r decays and rank-r key-value updates. Their empirical results show that HDLA outperforms different baselines across synthetic, retrieval, and language modeling tasks. It also performs on-par or outperforms in some case, the default softmax attention approach.", "questions": "- You showed that HDLA underperforms softmax attention on Fuzzy In-Context Recall because it requires accurate value prediction from keys interleaved with arbitrary noisy tokens. Can you expand on the consequences of this recency bias limitation? Does this mean that for instance for time series data that might have some higher level structure (e.g., seasonality or periodicity), your approach would not be suitable?\n- What is the wall-time comparison between the different approaches? Is HDLA faster for a fixed token budget? \n\nI am open to raising my score if the questions/weaknesses are properly addressed.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clear conceptual derivation of HDLA; the two problems it solved are well motivated and the parameterization choices are clearly explained. \n- Thorough comparison to baselines: the comparison to different baselines is captured first through Table 2 in terms of computation costs then through experiments that span different capabilities, datasets, and model sizes. \n- The proposed approach achieves increased expressivity and significantly better results in some settings without significant overhead, making it a good alternative to the default softmax attention. \n- The connection to Test-Time Training formulations in section 3.3 add theoretical depth and understanding of the proposed approach.", "weaknesses": "- The number of tokens used for training seems to follow what's chinchilla optimal for the smallest model (0.4B parameters, about a factor of 20), but it is quite low for larger models: 1.45B and 2.8B (table 7). Under-training these models severely could have a big impact on the results, i.e., results with the correct token budget may look different. \n- The limitations are not discussed thoroughly, e.g., impact of the recency bias (see question below). \n- Intuitive/qualitative analysis for how this approach differs from existing baselines is limited. \n\nVery minor notes: \n- Need a second pass for typos: eg, in lines 363-364: \"interleaved wth arbitrary noisy tokens\" -> \"interleaved with arbitrary noisy tokens\"\n- Need to double check the references, eg the first reference is: \"Sanjeev Arora, Bowen Yang, Selim Eyuboglu, Aditya Narayan, Alexis Hojel, Imanol Trummer, and Christopher Re. Language models enable simple systems for generating structured views of ´ heterogeneous data lakes. arXiv preprint arXiv:2304.09433, 2023a. URL https://arxiv.org/abs/2304.09433.\" -> you got the wrong first author name.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LtAZKQnGVH", "reviewer": "Reviewer_ADzH", "summary": "The authors present an extension of DeltaNet by proposing a more sophisticated decay mechanism. DeltaNet's state update is S_t = (I − βkk^T)S_{t−1} + βkv^T (some dependencies on t suppressed for clarity), which involves a Householder-like transformation. The net effect is to update the state by partially 'erasing' the old target value from the memory and partially 'adding' the new target to the memory. \n\nIn this paper, a new sort of update mechanism is proposed: S_t = (I − βkk^T)Λ(I − βkk^T)S_{t−1}+ kv^T, which the authors show is equivalent to a diagonal-plus-rank-2 decay mechanism. This is more expressive than the decay proposed in DeltaNet, which is a diagonal-plus-rank-1 mechanism. They derive the matrix equations necessary to implement chunkwise parallel training with the more complex update. Their experiments show improvements on an array of synthetic and real-world benchmarks, including Mechanistic\nArchitecture Design, QA tasks, retrieval, perplexity on Wikitext, etc.", "questions": "I hope that the authors will consider including empirical results regarding scaling (as the sequence length increases) and throughput for HDLA versus other methods, and find the tweaks needed to strengthen the results for retrieval tasks in Table 7.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 3, "strengths": "Investigating different decay mechanisms for linear attention is an interesting research direction, and the extension presented in this paper is a very natural one. The proposed method does appear to be novel, contemporaneous work notwithstanding.", "weaknesses": "The results on the non-synthetic benchmarks (Table 7) are very modest. For example, the average common-sense reasoning accuracy improvement with HDLA versus Gated DeltaNet at the 1.45B and 2.8B model sizes are 0.26% and 0.42% respectively. Of course, the rank-2 HDLA should generally outperform the rank-1 methods because it's more expressive and computationally expensive, but the gains are very marginal given the increase in complexity. While there is some discussion of the number of operations per recurrent step in Table 2 of HDLA vs GDP2 and GDP3, it doesn't address the increase in cost versus rank-1 methods. \n\nFor papers that are concerned with efficient methods (as all linear attention papers are), there needs to be some discussion of the tradeoff between model throughput and model performance. The paper is incomplete without some sense of what we are paying in HDLA's added computational cost in exchange for the accuracy/perplexity improvements that it provides. This is not an unusual analysis; I would note that other linear attention papers have addressed this directly (e.g., Yang et al, 2024 compares the training tokens/sec for their DeltaNet implementation against Gated Linear Attention, Transformer++, and Mamba.)\n\nAlso, the number of tokens used for LLM training seems to have been limited to 10B tokens (with the exception of the 1.45B models for the comparison between HDLA and Gated DeltaNet, which used 50B), which is quite small in comparison with published work. For example, Yang et al (2024) used 100B tokens from FineWebEdu for its 1.3B models, and Siems et al (2025) used 35B tokens in its experiments. The small training corpus makes it difficult to accept the surprising claim that HDLA outperforms the standard softmax attention mechanism in the Llama model (L407-408) in both perplexity and real-world tasks. Indeed, that finding might not hold up in more realistic training scenarios.\n\nThe results are not significantly better than the simpler alternatives on real-world benchmarks, the implementation appears to be significantly more complicated than the rank-1 approaches, and it isn't clear what the throughput penalty is in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9emNRYkVsn", "reviewer": "Reviewer_aNs6", "summary": "This paper proposes HDLA, a linear attention mechanism that parameterizes the decay matrix with a structured Diagonal-plus-Rank-2 form. This design aims to improve expressivity over the common Diagonal-plus-Rank-1 family while maintaining parameter, memory, and computational efficiency. The authors also derive a rank-generalized chunk-wise parallel algorithm that supports arbitrary Diagonal-plus-Rank-$r_{ab}$ decay and rank-$r_{kv}$ key–value updates, using a WY-style representation and custom block-triangular operators to enable efficient intra-/inter-chunk computations. Empirically, HDLA outperforms strong linear-attention baselines on language modeling (0.4B/1.45B/2.8B), synthetic benchmarks (MAD, MQAR), and retrieval tasks (RULER), with particularly large gains on MQAR and RULER. The paper also provides a Test-Time Training interpretation of the update rule and reports competitive ImageNet classification performance. Limitations are discussed (recency bias, need for state expansion and potential non-linear updates).", "questions": "Please see Weaknesses above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- **Principled architectural design**: The authors motivate and provide a principled, structured Diagonal-plus-Rank-2 decay that extends rank-1 methods while preserving parameter efficiency (O(dk)).\n- **Generalized chunk-wise parallel algorithm**: The authors also present a unifying \"WY-\"style framework accommodating Diagonal-plus-Rank-$r_{ab}$ decay and rank-$r_{kv}$ KV updates, which subsumes several prior works as special cases.\n- **Strong and diverse empirical results**: The architecture shows consistent improvements over strong linear-attention baselines (GLA, Mamba2/HGRN2, DeltaNet/Gated DeltaNet, GDP2/3) across LM perplexity at multiple scales, synthetics such as MQAR and MAD, and longer context RULER tasks.", "weaknesses": "- **Presentation**: Some instances could be improved with a bit more detail. \n  - For example, the \"WY Representation\" should be defined or at least alluded to, e.g., with a reference to [1]. \n  - Separately, while I appreciate the authors' acknowledged comparison to softmax attention, it would be good to see the delta between HDLA and softmax attention on the retrieval tasks such as NIAH or MQAR (i missed these in Table 6 and Figure 2). \n  - To present the advantage of the more expressive Rank-2 structure, could the authors also present a results table (or expand Table 1) that puts side-by-side the model, it's expressivity, and the average empirical performance binned by evaluation (ppl, retrieval acc, LM-eval) ? \n  - Please also fix the first citation: \"Sanjeev Arora, Bowen Yang, Selim Eyuboglu, Aditya Narayan, Alexis Hojel, Imanol Trummer, and Christopher Re.\" fwiw in a random sample of a couple other citations they all seemed to correspond to real author lists. \n\n- **Wall-clock / \"Real World\" efficiency**: While I appreciated the complexity analysis and inclusion of efficient algorithms (e.g., chunk-wise parallel), and the claims mostly center on presenting a new decay structure + evaluating it's modeling quality, in being a work on linear attention it would be good to validate how the efficiency translates to real world efficiency boosts, e.g., by measuring wall-clock inference time or generation throughput. \n\n[1]  Christian Bischof and Charles Van Loan. The WY Representation for Products of Householder Matrices. 1985.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xD8nigawYD", "reviewer": "Reviewer_BfPn", "summary": "The paper proposes HDLA which is a linear attention variant that parameterizes a decay matrix as a diagonal matrix with two structured factors computed through generalized Householder matrices. This HDLA attention mechanism performs relative well at a small scale when compared to softmax attention and better than appropriate baselines.", "questions": "* Why does criteria P2 presents $H_t$ as an invertible transformation instead of a orthonormal basis? Indeed, $H_t$ is\ninvertible when it is an orthonormal basis but it is a stronger result from the spectral theorem that you can find such\nan $H_t$ for a real symmetric matrix. Moreover, the construction in equation 5 requires orthonormality. Please help me\nsee if I'm missing a detail.\n\n* I didn't follow the three-step optimization process in equations 19-21 (lines 309-318). I only see an optimization\nproblem in Eq. 19, the next two lines appear to be just updates. Am I missing something?\nIs there any reason as to why you didn't shared code through an anonymized link? I'm wondering how you implemented HDLA\nand its chunk-wise parallel version. Especially as you emphasize the I/O cost which to optimize it would require custom\nkernels.\n\n* In Table 3, why is Mamba considered a linear attention baseline? Does Mamba not scale as $\\mathcal{O}(N \\log N)$?, where\n$N$ is the sequence length.\n\n* Could you provide runtime comparisons of HDLA for training and inference? Some examples suffice to get an estimate.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* I find sound and comprehensive the efficiency constraints laid out in section 3.1.1.\n* The empirical results compared to different variants of the Gated DeltaProduct are strong and comprehensive.\n* I appreciate the connections of HDLA from the TTT perspective.", "weaknesses": "* The community's experience with subquadratic alternatives to softmax attention is that you can find efficiency benefits\nfor small models (around 7B params) but these benefits fade away as we increase the scale. \nDoes your paper advocate to try HDLA for LLMs? Or are you content with the results at the < 3B scale?\nI understand that as a researchers we might not posses the access to compute to train such large models,\nbut is there any indication that HDLA would not suffer diminishing returns at larger scales?\n\n* It is unclear to me how can I incorporate HDLA into a transformer right now. The equations are presented for each\nelement in the sequence, which is great for the analysis, but then it is unclear how to incorporate this \nmethod in practice and what implementation details need to be considered.\n\n* There are no runtime comparisons which makes it harder to understand the overhead of HDLA against baselines like DeltaProduct.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2PfqjfEhQC", "reviewer": "Reviewer_Pma4", "summary": "The authors propose a diagonal-plus-rank-2 decay / state-transition matrix for linear attention/SSM by using products of householder matrices and diagonal matrices. They also provide a chunk-wise parallel algorithm for this sequential recurrence as a generalization of previous versions with rank 1 constraint. In addition, the authors show that the proposed HDLA method can be viewed as performing online update to the recurrence hidden state, which is a form of “test-time training”. The proposed method achieves good performance on MAD synthetics, MQAR recall tasks, language modeling perplexity and downstream reasoning tasks, retrieval-based S-NIAH tasks, and image classifications.", "questions": "- It’s good to impose new structured assumption in the decay matrix P_t for balancing between efficiency and expressivity, and the chunking algorithm also alleviate some of the computational overhead. However, the proposed HDLA includes a lot more matrix multiplications. In fact, any structured matrix which is formed by a composition of permutation, block-diagonal assumption, summations etc can be viewed as a deep linear transformations which causes more I/O due to sequential matrix loading, computation, and writing between HBM and SRAM. How do you optimize the I/O efficiency here?\n- By using a diagonal-plus-2-rank update, you allocate more compute for the decay matrix. But the underlying hidden state matrix S_t is still finite and will still suffer from retrieval failures under long sequence input. What if you allocate that compute not to the decay matrix P_t but to have basically a bit more S_t, i.e. perhaps allocating a few smaller S_t matrix across the sequence dimension. What do you think would be superior here?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed method combines lots of prior efforts (DeltaProduct, Gated Delta Net, etc) to continuously improve the linear attention family of models. This is a good contribution to this line of work.\n- The derivation of the chunking algorithm is quite involved but definitely useful for the community.\n- Empirical experiments are extensive", "weaknesses": "It’s not immediately obvious how the newly proposed decay matrix is a form of diagonal-plus-rank-2 update. It’s better to lay this out more clearly instead of leaving it to the reader. For general machine learning audience, this is not that obvious so it should be reflected more in the writing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)