Gated Delta Networks: Improving Mamba2 with Delta Rule
Authors: Songlin Yang
Jan Kautz
Ali Hatamizadeh
Venue: ICLR 2025 Poster (2025)
Keywords: linear RNN
state-space model
linear transformer
subquadractic model
linear attention
delta rule
mamba
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce Gated DeltaNet, which combines the gating mechanism from Mamba2 with the delta rule from DeltaNet, achieving superior performance compared to both models individually.

Abstract: Linear Transformers have gained attention as efficient alternatives to standard Transformers, but their performance in retrieval and long-context tasks has been limited.  To address these limitations, recent work has explored two distinct mechanisms: gating for adaptive memory control and the delta update rule for precise memory modifications. We observe that these mechanisms are complementary—gating enables rapid memory erasure while the delta rule facilitates targeted updates. Building on this insight, we introduce the gated delta rule and develop a parallel training algorithm optimized for modern hardware. Our proposed architecture, Gated DeltaNet, consistently surpasses existing models like Mamba2 and DeltaNet across multiple benchmarks, including language modeling, common-sense reasoning, in-context retrieval, length extrapolation, and long-context understanding. We further enhance performance by developing hybrid architectures that combine Gated DeltaNet layers with sliding window attention or Mamba2 layers, achieving both improved training efficiency and superior task performance.

Published as a conference paper at ICLR 2025
GATED DELTA NETWORKS:
IMPROVING MAMBA2 WITH DELTA RULE
Songlin Yang" MIT CSAIL yangs166emit.. t. edu
Jan Kautz
Ali Hatamizadeh
NVIDIA
NVIDIA
jkautzenvidia.com
ahattmizzadeheeniddaa.com
ABSTRACT
Linear Transformers have gained attention as efficient alternatives to standard Trans- formers, but their performance in retrieval and long- tasks has been limited. To address these limitations, recent work has explored two distinct mechanisms: gating for adaptive memory control and the delta update rule for precise memory modifications. We observe that these mechanisms are complementary-gaee enables rapid memory erasure while the delta rule facilitates targeted updates. Building on this insight, we introduce the gated delta rule and develop parallel training algorithm optimized for modern hardware. Our proposed architecture, Gated DeltaNet, consistently surpasses existing models like Mamba2 and DeltaNet across multiple benchmarks, including language modeling, common-sense rea- soning, in-context retrieval, length extrapolation, and contect understanding. We further enhance performance by developing hybrid architectures that combine Gated DeltaNet layers with sliding window attention or Mamba2 layers, achieving both improved training efficiency and superior task performance. Code: betps://git thub. wwt.eoolwso///eeeecocceowooo tedDel taNet
INTRODUCTION
The Transformer architecture has significantly advanced the capabilities of Large Language Models (LLMs), showcasing exceptional performance across wide range of tasks due to its effective attention mechanism. This mechanism excels in precise sequence modeling and leverages the parallel processing capabilities of modern GPUs during training. However, the self-attention component scales quadratically with sequence length, leading to substantial computational demands that pose challenges for both training and inference.
To mitigate these issues, researchers have explored alternatives such as linear Transformers (Katharopoulos et al., 2020a), which replace traditional softmax-base attention with kernliizdd dot- dot-product-basee based linear attention, substantially reducing memory requirements during inference by reframing linear RNN with matrix-valued states. While early versions of linear Trans- formers underperformed in language modeling tasks compared to standard Transformers, recent enhancements such incorporating data- fetandependent gating mechanisms akin to those in LSTMs, exemplified by models like GLA (Yang et al., 2024a) and Mamba2 (Dao & Gu, 2024a)- have shown promising improvements. However, challenges persist in managing information over long sequences, particularly for in-context retrieval tasks where traditional Transformers maintain their advantage (Arora et al., 2023a; 2024a; Jelassi et al., 2024; Wen et al., 2024; Akyürek et al., 2024).
This phenomenon is not surprising: linear Transformers can be interpreted as implementing an pruer-product-bbaeed key-value association memory, reminiscent of tensor product representation (Smolensky, 1990). However, the number of orthogonal key-value pairs they can store is bounded by the model's dimensionality. When the sequence length exceeds this dimension, "memory collisions" become inevitable, hindering exact retrieval (Schlag et al., 202
Mamba2 addresses this limitation by introducing simple gated update rule, S. a u,k[ which uniformly decays all key-value associations at each time step by a dynamic ratio, o (0, 1). However, this approach does not account for the varying importance of different key-value
"Equation contribution. Work done during SY's internship at NVIDIA.

---

Published as a conference paper at ICLR 2025
associations, potentially leading to inefficient memory utilization. If the model needs to forget specific key-value association, all key-value associations are equally forgotten, making the process less targeted and efficient.
In contrast, the linear Transformer with the delta rule (Widrow et al., 1960), known as DeltaNet (Schlag et al., 2021a; Yang al., 2024b), selectively updates memory by (softly) replacing old key-value pair with the incoming one in a sequential manner. This method has demonstrated impressive performance in synthetic benchmarks for in-context retrieval. However, since this process only modifies a single key-value pair at time, the model lacks the ability to rapidly clear outdated or irrelevant information, especially during context switches where previous data needs be erased. Consequently, DeltaNet has been found to perform moderately on real-world tasks (Yang et al., 2024b), likely due to the absence of a robust memory-clerring mechanism.
Recognizing the complementary advantages of the gated update rule and the delta rule in memory management, we propose the gated delta rule, a simple and intuitive mechanism that combines both approaches. This unified rule enables flexible memory control: it can promptly clear memory by setting a, - O, while selectively updating specific content without affecting other information by setting 0r + 1 (effectively switching to the pure delta rule).
The remaining challenge lies implementing the gated delta rule in a hardware-efficient manner. Building upon Yang et al. (2024b)'s efficient algorithm that parallelizes the delta rule computation using the WY representation (Bischof & Loan, 1985), we carefully extend their approach to incor- porate the gating terms. Our extension preserves the benefits of chunkwise parallelism (Hua et al., 2022b; Sun et al., 2023a; Yang et al., 2024a;b), enabling hardware-efficiee training.
Our resulting architecture, Gated DeltaNet, consistently outperforms both Mamba2 and DeltaNet across a comprehensive suite of benchmarks, including language modeling, commonsense reason- ing, in-context retrieval, length extrapolation, and long-context understanding. Building on these results, we also develop hybrid architectures that strategically combine Gated DeltaNet layers with sliding window attention or Mamba2 layers, further enhancing both training efficiency and model performance.
2 PRELIMINARY
2.1
MAMBA2: LINEAR ATTENTION WITH DECAY
It is known that the linear transformer (Katharopoulos al., 2020b) be formulated as the following linear recurrence when excluding normalization and query/key activations:
S. S.-1 v.kk] IRd..xd
o, S.q. E Rd.
where dy and dy represent the (head) dimensions for query/key and value, respectively. By expanding the recurrence, we can express it in both vector form (left) and matrix form (right) as follows:
Σ (u,k])
u. (kTTg) Rdd,
O (QKT O M)V R
where L is the sequence length, and M e 14xL is the causal mask defined by M., = 0 when i < j. and otherwise.
However, this vanilla linear attention underperforms Transformers in language modeling by large margin. To address this, it is common to add decay term to forget historical information. Here we take Mamba2 (Dao & Gu, 2024a) as an example, which can be represented by the following linear recurrence (up to specific parameterization):
S, =a,, + TykT
O, S.q.
where a, E (0, 1) is a data deta-deeendent scalar-valued decay term that varies with t. Define the cumulative decay product 7/1 = I a;, and by expanding the recurrence, we can express the result in both a vector form (left) and a matrix parallel form (right):
0 = Σ
7tt u.kk
qi = Σ vi
k.
O =[QQTT T))
%
i=1
2

---

Published as a conference paper at ICLR 2025
Here, E R T is decay- aware causal mask where T3 24. if j and Fiy 0 otherwise. The equivalence between these parallel and recurrent forms is also referred to as the state space duality (SSD) described in Dao & Gu (2024a). This recurrence structure appears in several other architectures including Gated RFA (Peng et al.. 2021), xLSTM (Beck et al., 2024), and Gated RetNet (Sun et al., 2024b). When "/r is data-independent the formulation reduces to RetNet (Sun et al., 2023a) and Lightning-Ateen Attention (Qin et al., 2024a). Furthermore, 7t is extended to be scalar-valued, efficient training algorithms remain possible when parameterized with an outer-product structure, as demonstrated by Yang et al. (2024a) and used by Yang et al. (2024a); Peng et al. (2024); Qin et al. (2024b); Zhang et al. (2024); Chou et al. (2024); He et al. (2025); Lu et al. (2025).
rather than
Chunkwise training However, both the recurrent and parallel forms are not ideal for efficient training (Hua et al., 2022b; Yang al., 2024a), which motivates the use of the chunkwise parallel form (Hua et al., 2022b; Sun et al., 2023a) for hardware-efficient, linear-time training, as introduced below. To summarize, the chunkwise parallel form splits inputs and outputs into several chunks of size C, and computes outputs for each chunk based on the final state of the previous chunk and the query/key/value blocks of the current chunk. Following the notation of Sun et al. (2023b); Yang et al. (2024a;b), we take the query block, as an example. We denote Q == qcc+ (+++)+++ as the query block for chunk t, and Ti = quc++r as the r-th query within chunk t. The initial state of chunk t is defined as S/4 := Shg
1 11. Se
By partially expanding the recurrence, have
S/g Stg + Σ 2in TT E Xde,
xde
og 9/ Sin
(kTT% Rd.
Equivalently, in matrix form:
S|++1| = SS + [i4 K] E Rde
0 = S/: + (Ql KT) R M
Vig
Rxd.
where M RCC is the causal mask. The above equations are rich in matrix multiplications (matmuls), allowing for tensor-core-base hardware optimization. This chunkwise algorithm could be easily extended to linear attention with decay:
S|-1] S10
[e]
[c]
E
TR.. xd. Rdd.
Otg P1=QQus + (9 K7 o Tii) Vi E Cxd,
(1)
[] ay. Here we use the left arrow ( or the right arrow to denote a variable decaying to the first position and the last position of each chunk, respectively,
where (4))))s
Mi 7
T Ma 9
decaying each vector to the first position of chunk decaying each vector to the last position of chunk l
71
T
1/ 7/8
C Ste)
decaying the state matrix over the entire chunk
(2)
and likewise for other variables (e.g. こ). The SSD decomposition algorithm introduced in Mamba2 largely equivalent to this chunkwise algorithm. For a more generalized approach, Yang al. (2024a) proposed an extended chunkwise algorithm for linear attention that incorporates fine-grained decay mechanisms.
2.2 DELTA NETWORKS: LINEAR ATTENTION WITH DELTA RULE
The delta update rule (Widrow et al., 1960; Schlag et al., 2021b) dynamically erases the value (ogld) associated with the current input key (k₂) and writes new value (Dfe"), ) which is a linear combination of the current input value and the old value based on the "writing strength" B, (0, 1).2
S, S - (S KT + (B,u, + (1 - S, (,))k;)) K. = S - (-B,k,k?)) + B
'eere we slightly abuse the notation of y denote the cumulative product for each chunk (starting with the first position of each chunk separately) instead the entire sequence. PI is possible to set 8. E (0, 2) to allow negative eigenvalue to unlock the state tracking abilities of DeltaNet (Grazzi al., 2024; Siems et al., 2025).
3

---

Published as a conference paper at ICLR 2025
As shown above, DeltaNet implements a first-order linear recurrence with generalized Householder transition matrices (I - ), Despite demonstrating superior associative recall and language modeling performance (Schlag et al., 2021a), DeltaNet received received limited attention due computational inefficiency until Yang et al. (2024b) introduced a hardware-efficient chunkwise training algorithm, detailed below.
Chunkwise parallel form.
By partially expanding the recurrence, we have
S
STq]
II
i i ki
1 kij
(1 M j=++1
II
where P// involves cumulative products generalized Householder matrices, which could be optimized by the classical WY representation (Bischof & Loan, 1985):
PlO = I Σ
w AiT E Rd (i
W - Big
k/6
w (1(((((TATT))
E Bd
(4)
Likewise, Hi could be represented as:
H Σ
hig E Rd xd.
u1q = Big 0/0 Σ
(u (115
E RRd
(5)
and in matrix form: Ptg =I W
Kg
ln Kg e R4 H(y [t)
Kg Rd.xdd, By using the UT
transform (Joffrain al., 2006), can further write W and U in matrix form:
) diag (9ta) E R
Tp I strictLower (diag( ( KT (8)
(6)
W = TT Ky E RCdds
U [ = Tp) Vig E Cxd.
(7)
Substituting these back into Eq. yields hardware-efficient chunkwise algorithm for DeltaNet that leverages matmuls, enabling tensor core based GPU optimization:
++| S P + H = S + (UH) - WHiST) K( 0 /
R pddxxd
(8)
2 S
+
(Qg K o M) (UN) WHIST))
E C'xd
(9)
GATED DELTA NETWORKS
3.1 FORMULATION: GATED DELTA RULE
The proposed gated delta rule simple yet effective:
S, === S (ax (I- B,k,k()) + S.ock?
(10)
where the data-dependent gating term a; (0, 1) controls state decay. This formulation unifies the advantages of both gating mechanisms and the delta rule: the gating term enables adaptive memory management, while the delta update structure facilitates effective key-value association learning.
We present a formal analysis of the gated delta rule through the lens of the online learning framework introduced by Liu et al. (2024). In this framework. recurrent state updates emerge as closed-form solutions to an online learning problem, typically incorporate a regularization term in their online learning objective to prevent state divergence from previous values, thereby enabling memory retention. However, this retention mechanism becomes problematic when the state becomes saturated with information. In such cases, each state would encode a superposition of multiple information pieces, making precise retrieval challenging. To address this limitation, Mamba2 and Gated DeltaNet introduce an adaptive scaling factor a that relaxes the regularization term, allowing controlled deviations between S, and S,-1. This modification enables dynamic memory management through selective forgetting, which could be useful in filtering irrelevant information (see $3.2).
shown in Table 1. Recent linear RNN architectures
4

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of different linear RNN models and their corresponding online learning objectives using the framework from (2024), For convenience, we simplify Longhorn's setor-valued scalar B.
Method
Online Learning Objective
Online Update
LA
[[ S F 2(S, We)
S, St-1 voli
Mamba2
[| a 1 2(S, k., vz)
S: a,S 11 k,
Longhom
|S: S.- i S. wzll"
S (1 cke,k + exve to? ,0==
1x Baki ke
|S: S:-1 1| ks, Be (v k.))
S (I Bekeke Me
Gated DeltaNet
|| 0. 2,, (ve S.-.
S. S (cr Deke D.Ock
Table Zero-shot performance comparison on S-NIAH benchmark suite for 1.3B models (see $4 setups)
S-NIAH-1 (pass-key retrieval)
S-NTAH-2
S-NTAH-3 (uuid haystack)
(oumber in haystack)
Model
1K
2K
4K
8K
1K
4K
8K
1K
2K
4K
DeltaNet
97.4
96.8
99.0
98.8
98.4
45.6
14.4
85.2
47.0
22.4
Mamha2
99.2
98.8
65.4
30.4
99.4
98.8
56.2
17.0
64.4
47.6
4.6
Gated DeltaNet
98.4
88.4
91.4
91.8
100.0
92.2
29.6
86.6
27.6
the other hand, Linear Attention (LA) and Mamba2 use a simple negative inner-product loss As, vt), while Longhorn (Liu et al., 2024) uses more expressive online regression objective |S - vell2 for better modeling of key-value associations. The resulting Longhorn's 's update rule closely resembles the delta update rule, suggesting the superiority of the (gated) delta rule Mamba2 in in-context associative recall.
From the perspective of fast weight programming (Irie et al., 2022a) and test-time training (Sun et al., 2024a) and regression (Wang et al., 2025), the hidden state S can be interpreted as a (fast) weight matrix, with the delta rule optimizing the online regression objective L(S.) = - o.ll via test-time stochastic gradient descent (SGD):
S S. C.S) = S, - - - v.)A. = S, (I 3, + B,0Kk
where 3e represents the (adaptive) learning rate. From this perspective, the gated delta rule can be viewed as incorporating an adaptive weight decay term into the SGD update, a technique widely used in deep learning (Krogh & Hertz, 1991; Andriushchenko al., 2023). Concurrently, Titans (Behrouz et al., 2024) demonstrated the effectiveness of incorporating weight decay mechanisms in RNN test-time SGD updates.
3.2 CASE STUDY: SINGLE NEEDLE A HAYSTACK (S-NIAH)
To better understand the complementary strength between the delta rule and the gated rule, present a case study on the Single Needle- An-A-HHa Haystack (S-NIAH) benchmark suite from RULER (Hsich et al., 2024), where key-value pair acts needle in the haystack (context) and the model must recall the value when given the key. Table 2 presents the results and we draw three main observations:
Decay hurts memory retention. In the simplest S-NIAH-1 setting with repeated synthetic context, models memorize minimal information, testing long-term retention. DeltaNet achieves pear-perfect performance across all sequence lengths. Mamba2 degrades significantly beyond 2K sequences since it decays historical information too quickly, while Gated DeltaNet's degradation is less severe thanks to the use of delta rule.
Gating facilitates filtering. In S-NIAH-2/3 with real-world-essay context, models store all po- tentially relevant information, testing efficient memory management. With fixed state size, lack becomes superimposed and indistinguishable.
of clearance causes memory DeltaNet's performance drops significantly longer sequences due to poor memory clearance. Mamba2 and Gated DeltaNet maintain better performance through gating mechanisms that filter irrelevant information.
'The theoretical distinction lies in the optimization approach: Longhorn uses implicit online learning (Kulis & Bartlett, 2010) to derive closed-form globally optimal updates, while DeltaNet optimizes the same objective through one-step explicit gradient descent, noted by Liu al. (2024).
5

---

Published as a conference paper at ICLR 2025
Delta rule helps memorization. In S-NIAH-3, values change from numbers to UUIDs, testing com- plex pattern memorization, Mamba2's 's performance drops quickly, while Gated DeltaNet performs better, verifying that the delta rule indeed has better memorization ability.
3.3
ALGORIIHM:: HARDWARE-EFFICIENE CHUNKWISE TRAINING
In this subsection, we derive a hardware-efficiens chunkwise algorithm for training Gated DeltaNet. By partially expanding the recurrence in Eq. 10, we have
S = St
II 1=1
aial (I (I- Si0 k
+
II oin (I ( -=
Gi
It is easy to see that F =7/0 Plu = (ol. As for Gla-- we adapt Eq. 5 as follows,
)) 1R
G
(
Me]
E gddx i[g = O1
u 70
i=1 T(
(see $A for a proof). By UT transform, we have the matrix form:
U [1] I+ strictLower (diag (3(1) (T @ K ((j [[)] diag (514 V(g
RRCx,,
Similar to how Mamba2 extends linear attention (Eq. 1), we can adapt DeltaNet's chunkwise algorithm (Eq. 8-9) for Gated DeltaNet to enable hardware-efficier training as follows:
S(++1] S [4)
W 1r
STa)
K [
Rdd xd.
114
I + (Q
K M)
(Ug)
W ST))
Rg
where [1]
Ha Tap wH = Va Tap
k Tu
and S
like we defined in Eq. 2.
3.4 GATED DELTA NETWORKS AND HYBRID MODELS
Token mixer block. The basic Gated DeltaNet follows Llama's 's macro architecture, stacking token mixer layers with SwiGLU MLP layers, but replaces self-attention with gated delta rule token mixing. Fig. (right) shows its block design. For the gated delta rule (Eq. 10), queries, keys and values {q. k. e} are generated through linear projection, short convolution and SiLU, with L2 normalization applied to q, & for training stability. a. 3 use linear projection only.* Following Sun et al. (2023a), the output is processed through normalization and gating before applying output projection.
Hybrid models. Linear transformers have limitations in modeling local shifts and comparisons, and their fixed state size makes it hard for retrieval tasks (Arora al., 2024a). Following recent hybrid architectures like Griffin (De et al., 2024) and Samba (Ren et al., 2024), we combine linear recurrent layers with sliding window attention (SWA), resulting in GeteeDeltaNNt-HH.. We also stack Mamba2, GatedDeltaNet and SWA, resulting in GatedDeltaNet-HH..
EXPERIMENTS
Setup Our experiments encompass comprehensive comparison of recent state-of-the-ae archi- tectures, including pure Transformer models, RNN-based approaches, and hybrid architectures. We evaluate against the following baselines: RetNet (Sun al.. 2023a), HGRN2 (Qin al., 2024b), Mamba (Gu & Dao, 2023), Mamba2 (Dao & Gu, 2024b), Samba (Ren et al., 2024), and DeltaNet (Yang et al., 2024b). For fair comparison, all models are trained under identical conditions with 1.3B parameters on 100B tokens sampled from the Fine dataset (Penedo et al., 2024). We use the Adam W optimizer with a peak learning rate of 4e-4, weight decay of O. and gradient clipping of 1.0. The learning rate follows a cosine annealing schedule with IB token warm-up period and batch size of 0.5M tokens. All models employ the Llama2 tokenizer with a vocabulary size of 32,000. For sequence modeling, a sliding window size of 2K. See B. for evaluation settings and & B.2 for ablation studies.
the training length to 4K tokens, with Samba and our hybrid models using
WWe use Mamha2's parameterization for a hut omit it for brevity.

---

Published as a conference paper at ICLR 2025
MLP
Linear
SWA
ML.P
Norm
MLP
Gated Delta Rule
A:
Gated DeltaiNt
MLP
Gated DeltaNet
Conv
MLP
Linear
Lincar
Lin.
Linear
Mamba2
Gated DeliaNet-H1
Gated DeltaNetHH2
Block Design
Figure 1: Visualization of the (hybrid) architecture and block design of Gated DeltaNet models. Gated DeltaNet- HI and H2 use Gated DeltaNet SWA and Mamba2 Gated DeltaNet SWA patterns, respectively. In the block design, query/key paths consist of linear proj.. shortconv., SiLU and L2 norm; value path includes linear proj.. shortconv. and SiLU; alpha/beta use linear proj.; and output gate applies linear proj. with SiLU.
Model
Wiki. ppl
LMB. ppl 4
LMB.
PIQA
Hella. acc_n
Wino. acc
ARC-e 1
ARC-c acc
SIQA
BoolQ acc 1
Avg.
asc
Recurrent models
RetNet
19.08
17.27
40.52
70.07
49.16
54.14
67.34
33.78
40.78
60.39
52.02
HGRN2
19.10
17.69
39.54
70.45
49.53
52.80
69.40
35.32
40.63
56.66
51.79
Mamha
17.92
15.06
43,98
71.32
52.91
$2.95
69.52
35.40
37.76
61.13
53.12
Mamba2
16.56
12:56
45.66
71.87
$5.67
55.24
72.47
37.88
40.20
60.13
54.89
DeltaNet
50.93
40.22
55.29
Gated DeltaNet
16.42
12.17
46.65
72.25
$5.76
57.45
71.21
38.39
40.63
60.24
55.32
Attention hybrid models
Transormer-+
18.53
18.32
42.60
70.02
50.23
53.51
68.83
35.10
40.66
$7.09
52.25
Samba
16.13
13.29
44.94
70.94
53.42
55.56
68.81
36.17
39.96
62.11
54.00
Gated DeltaNet-Hl
16.07
12.12
47.73
72.57
56.53
58.40
21.75
40.10
41 40
63.21
56.40
57.77
41.91
61.55
56.18
Table 3: Performance comparison language modeling and zero-shot common-sense reasoning.
Common-sense reasoning In Table 3, we present the language modeling perplex- ity and zero-shot accuracy on common- sense reasoning benchmarks for models with 400M and 1.3B parameters. Gated DeltaNet consistently outperforms other linear models, including RetNet, HGRN2, Mamba, Mamba2, and DeltaNet, at both scales. As expected, the hybrid variant fur- ther enhances performance.
Models
SSWDE SQD
TQA NQ Drop Avg
Recurrent models
RetNet
14.0
28.5
7.0
54.4
16.2
17.3
22.9
HGRN2
Mamha
9.8
25.8
3.7
54.3
14.9
17.4
21.0
Mamba2
19.1
33.6
61.0
20.8
19.2
29.8
DeltaNet
17.9
30.9
18.4
53.9
17.3
18.6
26.2
Gated DeltaNet
25.4
34.8
23.7
60.0
20.0
19.8
30.6
Attention hybrid models)
Transfomer=+
29.5
38.0
58.3
22.5
21.6
37.0
Samba
33.0
39.2
50.5
57.7
23.5
20,2
37.3
Gated DeltaNet-I11
35.6
39.7
52.0
60.1
24.6
22.2 39.0
Gated DeltaNet-H2
38.2
40.4
50.7
63.3
2488
23,3 40.1
In-context retrieval on real-world data Table 4 presents results real-world recall-intensive tasks used by Arora et al. (2024b). As expected, linear recurrent models show significant performance gap compared to Trans- formers, while hybrid models combining linear recurrence and attention outperform pure attention models in retrieval tasks.
Table 4: Accuracy recall- world retrieval tasks with input truncated 2K tokens. SQD: SQUADE. TQA: Trivial QA.
7

---

Published as a conference paper at ICLR 2025
For pure recurrent models, despite DeltaNet's superior performance synthetic in-context retrieval tasks (Yang et al., 2024b), its real-world retrieval performance lags behind Mamba2, consistent with observations in S-NIAH-2 and S-NIAH-3 (Table 2). Gated DeltaNet outperforms both DeltaNet and Mamba2 thanks to its gated delta rule, though the improvement margin is smaller than in Table 2. We attribute this reduced performance gap instruction-unalle unntlignd small language models being prone to repetition emror,, which are the primary source of errors in these tasks (cf. Arora et al. (2024b, Appendix E)). Since this issue is largely independent of the update rule choice, the performance differences between models are less pronounced compared to Table 2.
GovReport
QMSum
NarrativeQA
18
20
Perplexity
16
18
14
16
8k
20k
14k
20k
20k
Qusper
CodeParrot
PG19
20
20
15
Perplexity
18
15
16
10
14
13
4k
8k
14k
20k
4k
8k
14k
20k
14k
20k
Length
Length
Length
Mamhal
DeltaNet
Mamha2
Samba
GutedDeltaNet
GatedDeltaNer-HI GatedDelaaNet-H2
Figure 2: Length extrapolation six long benchmarks.
Length extrapolation on long sequences. As shown in Fig.2, we evaluate the models' capacity to extrapolate to sequences up to 20K tokens across six long-context benchmarks. Gated DeltaNet achieves the lowest overall perplexity across tasks among RNN models. While we observe mixed re- sults in length extrapolation, Gated DeltaNet exhibits relatively more robust performance, suggesting better memory management. The hybrid models further improve upon this by leveraging atten- tion for local context modeling, which reduces the memory management burden on their recurrent components. Future work will explore these models' capabilities even longer sequences.
Long context understanding As demonstrated in Table 5, we evaluated the models' performance on LongBench (Bai et al., 2023). In recurrent models, Gated DeltaNet shows consistent advantages, especially in single-doc QA, few-shot in-context learning, and Code tasks, demonstrating its superior capabilities in retrieval, in-context learning, and state tracking, respectively.
Throughput Comparison. The training throughput comparison across different models is presented Fig. 3. As our analysis shows, the proposed gated delta rule introduces only marginal overhead compared to the original delta rule, with Gated DeltaNet achieving essentially the same throughput as DeltaNet. Both are slightly slower than Mamba2 (2-3K tokens/sec) due their more expressive transition matrices.
The Transformer++ achieves the best performance the 2K context window domain, thanks to the highly optimized
kernel (Dao, 2023). Consequently, hybrid approaches combining 2K window-size SWA attention with other token mixers demonstrate higher throughput than stan- dalone mixers: Samba outperforms Mamba, while Gated DeltaNet-HI and -H2 outperform Gated DeltaNet. Notably, Gated DeltaNet-Hl maintains compelling training throughput across all sequence lengths, even short sequences.
8

---

Published as a conference paper at ICLR 2025
Single-Doc QA
Multi-Doc QA
Summarization QMS
Few-shot TQA
Code
Avg
Model
NQA
QQA
MFQ
HQA
2WM
Mus
GvR
MNS
TRC
SSM
LCC
REP
Recurrent models
10.7
18.0
HGRN2
10.7
12.1
19.1
11.3
15.7
6.0
5.2
15.1
9.2
16.0
15.8
10.3
18.6
20.8
13.5
Mamba
13.0
10.1
20.4
10.1
16.7
6.0
7.2
15.9
8.4
23.1
21.9
11.2
17.9
19.0
14.6
DeltaNet
12.9
10.8
21.5
10.9
13.2
5.1
6.5
13.5
7.2
15.5
23.3
11.6
17.6
20.3
13.6
Mamba2
11.1
11.3
18.6
11.8
15.1
6.7
6.7
14.5
7.4
13.0
23.6
8.4
17.9
20.6
13.5
Gated DeltaNet
14.1
14.0
23.3
13.7
14.4
5.8
7.5
16.4
30.0
22.4
23.0
18.7
22.1
16.6
Attention kybid models
Transformer++
11.8
9.3
10.0
10.9
4.2
6.1
7.4
15.8
16.9
13.5
3.9
17.2
18.7
11.0
Samba
12.5
12.9
25.4
11.2
19.7
6.8
9.1
15.7
11.0
20.0
22.7
22.8
18.1
21.1
15.9
Gated DeltaNettHHI
14.5
12.3
26.6
12.6
23.6
6.1
16.1
12.8
33.5
23.9
26.8
15.5
19.2
17.8
Gated DeltaNet-HR H2
12.7
13.0
27.1
12.7
20.6
7.5
10.4
16.2
13.0
40.5
22.7
27.9
19.9
22.1
18.4
Table S: Accuracy on 14 tasks from LongBench (Bai et al., 2023): Narrative QA, QasperQA, MultiField QA, HoipotQA, 2WikiMulti QA, Musique, GovReport, QMSum, MultiNews, TRec, Trivia QA, SamSum, LCC, and RepoBench-P by order.
60
Thousunds Token Per Second (KUs)
55
50
45
40
35
Transformer++
DeltaNet
30
Gated DeltaNet
Mambal
Mamba2
Gated DeltaNe--HI
25
Samba
Gated DeltaNet-H2
2K 16
4K
8Kx4
16K
Sequence Length x Batch Size
Figure Training throughput comparison of 1.3B models on single H100 GPU.
5 RELATED WORK
Gated linear RNN. Large linear recurrent language models have attracted significant attention due their training and inference efficiency. The field of linear RNNs has rapidly evolved from using data-inden data-independent decay mechanisms, as exemplified by models like S4 (Gu et al., 2022). S5 (Smith al., 2023), LRU (Orvieto al., 2023), RWKY4/5 (Peng et al., 2023), and RetNet (Sun et al., 2023a), to incorporating data-dependent decay mechanisms in more recent architectures such as HGRN1/2 (Qin et al., 2024b; 2023b), Mambal/ 1/2 (Gu Dao, 2023; Dao & Gu, 2024a), RWK V6 (Peng et al., 2024), GSA (Zhang et al., 2024). This transition stems from the proven advantages of gating/forgetting mechanisms (termed selective mechanisms in Mamba)-a classical concept originating in the gated RNN literature (Gers al.. 2000) whose significance has been consistently reaffirmed (Greff et al., 2015; van der Westhuizen & Lasenby, 2018; Qin al., 2024b; 2023b; Gu & Dao, 2023).
Modern forget gates differ from traditional designs like those in LSTT by removing the dependency the previous hidden state, relying solely input data. This modification enables efficient parallelism across sequence lengths (Martin & Cundy, 2018; Qin et al., 2023b). The absence of a forget gate has been a notable limitation DeltaNet, and our gated extension addresses this gap in a natural, effective, and hardware-efficient way. We also note recent concurrent work RWKV-7 using a similar idea, but with more relaxable formalism using diagonal cisoonal-plus-lonn plus-low-ran transitions:

---

Published as a conference paper at ICLR 2025
S, S (dia a(,;) + u where d,. a,, b, R The chunkwise algorithm could be similarly adapted to this case, as implemented in Flash Linear Attention (Yang & Zhang, 2024).
Delta rule. The delta learning rule demonstrates superior memory capacity compared to Heb- bian learning (Gardner, 1988; Prados & Kak, 1989), an advantage DeltaNet leverages while linear transformers rely on -ibbiian-like rules. This memory capacity advantage is evident in synthetic in-context learning tasks and extends to language modeling (Iric et al., 2021; Yang et al., 2024b), reinforcement learning (Irie et al., 2022b), and image generation (Irie & Schmidhuber, 2023). Yang et al. (2024b) parallelized delta rule computation and demonstrated how DeltaNet's data-dependent identity-plusloww
k, k.)) offers greater flexibility than Mamba2's data-dependent diagonal matrices (ay I). This structural advantage could enable complex reasoning, including regular language recognition (Fan al., 2024; Grazzi et al., 2024) and state-tracking beyond TC complexity (Merrill et al., 2024) crucial for coding and reasoning applications.
structure
Despite these significant advantages, the delta rule faces theoretical limitations (Irie et al., 2023) and shows only moderate performance on real-world datasets (Yang et al., 2024b), suggesting room for improvement. Previous attempts to enhance expressiveness through nonlinear recurrence (Irie et al., 2021; 2022b) addressed some limitations but sacrificed training parallelism, creating a performance- efficiency tradeoff. Recent work proposes some enhancements without compromising parallelism for better state tracking performance, including using negative eigenvalues (Grazzi et al., 2024) and multiple products of houscholder transition matrices (Siems et al., 2025) which enable high-rank transformations. These methods could be applied to Gated DeltaNet scamlessly.
From a (online) learning objective perspective, alternative formulations could further extend ex- pressiveness: nonlinear regression (C(S,) = fl..s, (kt) 1/
in TTT (Sun et al., 2024a) and
Titans (Behrouz et al., 2024), where fs is a nonlinear function parameterized by S; or regression considering the entire history (C(S,) = / Ei= |,,A,; val (2) as in Mesa layer (von Oswald al.,
the difference between Least Mean Square and Recursive Least Square algorithms. However, these more expressive variants introduce nonlinear recurrence and require workarounds, such as performing nonlinear updates only after processing entire chunks (as in TTT and Titans); or approximating nonlinear recurrence methods like Lim et al. (2024); Gonzalez et al. (2024); Schöne et al. (2025).
Hybrid models. In this work, we explore interleaving hybrid attention layers across layers, which commonly used such in MiniMax-01 (MiniMax et al., 2025) and Hybrid Mamba2-Attention (Waleffe et al., 2024). It is also interesting to investigate hybrid linear/softmax attention within single layer (Hua al., 2022a; Zancato et al., 2024; Munkhdalai et al., 2024; Nunez et al., 2024; Dong et al., 2025; Zhang et al., 2025).
CONCLUSION
In this work, introduced Gated DeltaNet, which enables better key-value association learning compared to Mamba2 and more adaptive memory elearance than DeltaNet, leading to consistently better empirical results across various tasks. We extended the parallel algorithm from Yang et al. (2024b) to enable hardware-efficient training of Gated DeltaNet. Our hybrid Gated DeltaNet model achieves even higher training throughput and overall performance, making well-suited for practical deployment.
ACKNOWLEDGMENT
We thank Yu Zhang for assistance with figure creation and model evaluation; Kazuki Irie for providing valuable feedback on the draft; Simeng Sun and Zhixuan Lin for insightful discussions on long-sequence task evaluation settings; and Eric Alcaide and Volodymyr Kyrylov for their helpful discussions on the online learning perspective DeltaNet.
http:////ttioccoooieesssaaaaatte delta_rule
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Ekin Akyürek, Bailin Wang, Yoon Kim, and Jacob Andreas. In-Context Language Learning; Archi- tectures and Algorithms, 2024. URL https://arxii http::///arrxiv.or/aa////// 12973..
Maksym Andriushchenko, Francesco D' Angelo, Aditya Vardhan Varre, and Nicolas Flammarion. Why do we need weight decay in modern deep learning? ArXiv, abs/2310.04415, 2023. URL https:/ /app. semante cismantiscchlaar..... www.ccccccch.oooooeoeeeeeeeeeeee
org/CorpusID: 263829417.
Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models. ArXiv orgg/bs//31111 04927.
preprint, abs/2312.04927 2023a. URL https:
Simran Arora, Brandon Yang, Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trummer, and Christopher Ré. Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes, 2023b. URL https://arxx..oog//b////// wtt.v//.///abb/2344444 23t.///34344..33....
Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recoll-throuhhput tradeoff. ArXiv preprint, 0bs/2202.18666 2024a. URL pittps:///axiie orgg/ abs/2402. 18668.
Simran Arora, Aman Timalsina, Aaryan Singhal, Benjamin Spector, Sabri Eyuboglu, Xinyi Zhao, Ashish Rao, Atri Rudra, and Christopher Ré. Just read twice: closing the recall gap for recurrent language models, 2024b. URL https://arxis: wtw.:oogg/aaaao h/gg///b///// 2487.05483.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. ArXiv preprint, abs/2308. 2023. URL https://arxiv.. or org/abs/ 2308. 14508.
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. ArXiv preprint, abs/2405.04517, 2024. URL 2405. 04517.
Ali Behrouz, Peilin Zhong, and Vahab Mirronni. Titans: Learning to memorize at test time, 2024.
URL https://arxix www.:oorg/abb///..ooooo 00663.
Christian H. Bischof and Charles Van Loan. The WY representation for products of householder matrices. In SIAM Conference on Parallel Processing for Scientific Computing, 1985. URL
httpss//apt semanti wwchontisschola...mmm .owaaarorrggooo orgg/CorpussDD 36094006.
Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirly-Second Innovative Applications of Artificial Intelligence Con- ference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY. USA, February 7-12, 2020, pp. 7432-7439. AAAI Press, 2020. URL 6239.
https://aaal: heapiiorr//o///iiieeeee http://aaarrrrooo....////////aaa hwtp::/p//////aaatle//eeeee
Yuhong Chou, Man Yao, Kexin Wang, Yuqi Pan, Rui-Jie Zhu, Jibin Wu, Yiran Zhong, Yu Qiao, Bo XU, and Guoqi Li. MetaLA: Unified optimal linear approximation to softmax attention map. In The eiihth- Annual Conference Neural Information Processing Systems, 2024. URL https://openrest www.:ewww.e.eeeeeeeeeeeeeee net//forumRid-yyYrrCEEEE
Christopher Clark, Kenton Lee, Ming-Wai Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies. Volume 1 (Long and Short Papers), pp. 2924-2936, Minneapolis, Min- nesota, 2019. Association for Computational Linguistics. doi: 10. 18653// 1///3//11N11///1 1300. URL
11

---

Published as a conference paper at ICLR 2025
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv preprint. abs/ tbs/1803.05457, 2018. URL
1803.05457.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. ArXiv 2301206777........
preprint, abs/2307.08691, 2023, URL
Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, arXiv preprint arXiv: 2405.2 /060, 2024a.
Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Proceedings of the 41st International Conference on Machine Learning. volume 235 of Proceedings of Machine Learning Research, pp. 10041-10071, PMLR, html.
2024b. URL https: / http//ppreeens../////// mlr. wwtppeprrssss/deeeeeeesss
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A dataset of information- questions and answers anchored in research papers. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer. Dilek Takkani-Tur Iz Beltagy, Steven Bethard, Ryan Cot- terell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 4599-4610, Online, 2021. Association for Computational Linguistics. doi: 10. 1/203/111/22111 mainacl-main.66.... URL ht /wtpaaaoaaaaatoo....ooooo .rg/
(rrg/202 2021. 2021.naacc..... nwa.clmmainnnn.m 365.
Soham De. Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Marsian,Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, and Caglar Gulcehre. Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models, 2024. URL https: http///xxxvvv///////e//e// org/abs/2402. 19427.
Xin Dong. Yonggan Fu, Shizhe Diao, Wonmin Byeon, ZIJIA CHEN, Ameya Sunil Mahabaleshwarkat Shih-Yang Liu, Matthijs Van keirsbilek, Min-Hung Chen, Yoshi Suhara, Yingyan Celine Lin, Jan Kautz, and Pavlo Molchanov. Hymba: A hybrid-head architecture for small language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview.o wwt. htt://orrmmiiittzzyy///
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gard- ner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Confer ence of the North American Chapter of the Association for Computational Linguistics: Human Language lechnologies, Volume (Long and Short Papers). pp. 2368 23788, Minneapolis, Min- nesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19. 1246. URL https: https://accanthhlooge org/N19- 1246.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.). Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1074-1084, Florence, Italy, 2019. Association for Computa- tional Linguistics. doi: 18.18653/v 1/P19-11/2 URL ht https: http:://aaaaaahhlaacc
an tnthooogy.... org/219-1102.
Ting-Han Han Fan, Ta-Chung Chi, and Alexander Rudnicky. Advancing regular language reason- ing in linear recurrent neural networks. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Associ- ation for Computational Linguisties: Human Language Technologies (Volume 2: Short Pa- pers), pp. 45-53, Mexico City, Mexico, 2024, Association for Computational Linguistics. URL https://aclanthon wwwpenthhooooooooo///eeeeo 202/ naacl-short...
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 2021.
E. Gardner. The space of interactions in neural network models. Journal of Physics A, 21:257-270. 1988. URL https:: / //api. semanticscholar .aallaarrrrrgggoee :t
12

---

Published as a conference paper at ICLR 2025
Felix A. Gers, Jürgen Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with LSTM. Neural Comput. 12(10):2451-277) 2000.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: A human- annotated dialogue dataset for abstractive summarization. In Lu Wang, Jackie Chi Kit Cheung. Giuseppe Carenini, and Fei Liu (eds.), Proceedings of the 2nd Workshop New Frontiers in Summarization, pp. 70-79, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10.18653/v 1/L16653/1/DD19544400 1/D19-5409. URL https::/aclanttolllee
org/D19-5409.
Xavier Gonzalez, Andrew Warrington, Jimmy T.H. Smith, and Scott Linderman. Towards scalable and stable parallelization of nonlinear RNNs. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL
wwweereeewwwwweeeeeeeeeeeeeeeeee net/forumiid-hBCxx
Riccardo Grazzi, Julien N. Siems, Jorg K. H. Franke, Arber Zela, Frank Hutter, and Massimiliano Pontil. Unlocking state-tracking in linear rans through negative eigenvalues. 2024. URL https: //api.
org/CorpusID:274144 274144455
Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, and Jürgen Schmidhuber. Lstm: A search space odyssey. IEEE Transactions on Neural Networks and Learning Systems, 28: 2222-2232, 2015, URL pertss:/aapt semanticscholar wotg/orrpusteeeooooo (CCopprI:::::: 3 3356463.
Albert Gu and Tri Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. 2023.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https: /ttp:///eoeeeeeeee/eeeeeeeooeeee wwwpreeiiiiieeeeoouuiiiiiii uYLFoZ1vIAC.
Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian McAuley. Longcoder: A long-range pre-trained language model for code completion. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). International Conference
Machine Learning, ICML. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 12098-12107. PMLR, 2023. html.
URL https:
//proceedings. mlr prress/v202/guuee
Zhihao He, Hang Yu, Zi Gong, Shizhan Liu, Jianguo Li, and Weiyao Lin. Rodimus*: Breaking the accuracy-efficiency trade-off with efficient attentions. In The Thirteenth International Conference on Learning Representations, 2025. URL /
net httt/forumiidddIIii
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Com- putational Linguistics, pp. 6609-6625, Barcelona, Spain (Online), 2020. International Com- mittee on Computational Linguistics. doi: 10. 10.18653/11/2020.colingg wtw.::oooiiinnmiinni.i... m80n.580. URL https:
// //aclanthology.orgg/// http://ooor...rr/2222oo.oooggg main. 580.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? ArXiv preprint, abs/2404.0665. 2024. URL httpp://arxivv http:/awwxv.....reeeeeee 2404 06654.
Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc V. Le. Transformer quality in linear time. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.). International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. PMLR, 2022a. URL https: pttttp:://rocceedings mlr. pwtpss/vv2/huu2222.2. html.
Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc V. Le. Transformer quality in linear time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 9099-9117 PMLR, 2022b. URL https: htttp:::/prrceedingg mlr wttpprre///v//2h222222 html.
13

---

Published as a conference paper at ICLR 2025
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp- 1419-1436, Online, 2021. Association for Computational Linguistics. doi: 10.18653/ 1/203/1112021.1 naacl-main.,n.. 112. URL https: //aclanthology. . org 2221 naacl-main. 112.
Kazuki Irie and Jürgen Schmidhuber. Images weight matrices: Sequential image generation through synaptic learning rules. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openrevieer net/pdf?id did=ddadPPNUWW..
Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. Going beyond linear transform- ers with recurrent fast weight programmers. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.). Advances in Neural Information Processing Systems 34: Annual Conference Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual. pp. 7703-7717, 2021. URL https://prooeediings neurips.o cc cctpappr//021/////cc http:/hhhhh.tt9977/ffff3f33 httppp///88/////.////////oaaaaa0
Kazuki Irie, Róbert Csordás, and Jürgen Schmidhuber. The dual form of neural networks revisited: Connecting test time predictions to training patterns via spotlights of attention. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 9639-96559 PMLR, 2022a. URL htwps:tp::////tt http::///roceeeeegeeeeeee mlr................ www.m..res//.... /1/::///iiiii 12//rie22a.... html.
Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. A modern seff-referential weight matrix that learns to modify itself. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.). International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 9660-966669999 9677. PMLR, 2022b. URL https: / /tpp//eeocennnss/////////////.sp mlr. press/v162/iriee2b. html..
Kazuki Irie, Róbert Csordás, and Jürgen Schmidhuber. Practical computational power of linear transformers and their recurrent and self-referential extensions. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9465-9465, Singapore, 2023. Association for Computational Linguistics. doi: 10.18663/ m88n URL httptt:://ccaaathhhht //alaattolaalagoyyooooooo oog.orgg////// 2023 www..mmm.......m.mmemmmm amnlp-mainn 588.
Samy Jelassi, David Brandfonbrener, Sham M. Kakade, and Eran Malach. Repeat After Me: Transformers are Better than State Space Models at Copying. ArXiv preprint, abs/2402.01032, 2024. URL https://arxiv.. hwtprriwwwwrgrbs/////e//
01032.
Thierry Joffrain, Tze Meng Low, Enrique S. Quintana-Orts Robert A. van de Geijn, and Field G. Van Zee. Accumulating householder transformations, revisited. ACM Trans. Math. Softw. 12:169-179, 2006. URL https::/appl semanticscholar wwtpettiiiiccoooooooorrrrreeeeee prrg/CorpusID: 15723171.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly su- pervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp. 1601-1611, Vancouver, Canada, 7a. Association for Computational Linguistics. doi: 10.18653/w1/P17-11444
URL https://aclanthont 1ww.thhologoooo//111-111111 org/P17-1147
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly su- pervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, Vancouver, Canada, 2017b. Association for Computational Linguisties. doi: 10.18663/11/P17-1110 URL https: httpss//acllanthllle. orgg//7711//111/
14

---

Published as a conference paper at ICLR 2025
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 5156-5165. PMIR, 2020a. URL. http: html.
h/proceedings.mie mlr. //
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 5156-5165. PMLR, 2020b. URL http: html.
//proceedings. mlr. 1
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The Narrative reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018. doi: 10.1162/tacl a a_00023, URL
https://aclanthologgy - 1023.
Anders Krogh and John A. Hertz. A simple weight decay can improve generalization. In Neural Information Processing Systems, 1991. URL https://api semanticscholar, www.laarr.rr/CopussDDD... 10137788.
Brian Kulis and Peter L. Bartlett. Implicit online learning. In Johannes Fürnkranz and Thorsten Joachims (eds.), Proceedings of the 27th International Conference on Machine Learning (ICML.- 10), June 21-24, 2010, Haifa, Israel, pp. 575-582. Omnipress, 2010. URL https::/icml.... cc/ Conferences/201 http:///00//pppp//eeeee// .pf.f
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wes Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slay Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019. doi: 10.1162/tacla a -0276.. URL
Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Conference Computational Linguistics, 2002. URL
Yi Heng Lim, Qi Zhu, Joshua Selfridge, and Muhammad Firmansyah Kasim. Parallelizing non-lineas sequential models over the sequence length, 2024. URL
12252.
Bo Liu, Rui Wang. Lemeng Wu, Yihao Feng, Peter Stone, and Qiang Liu. PrcullDBLLiouunnb//orrabbbb-- 2407--4bb2, author = Bo Liu and Rui Wang and Lemeng Wu and Yihao Feng and Peter Stone and Qiang Liu, title = Longhorn: State Space Models are Amortized Online Learners, journal = CoRR, volume = abs/2407.142201 year = 2024, url http:///oo..og////.555550ii.... 242717 doi (0.445550/ARXI.22407.1 142077 eprinttype = arXiv, eprint = 14207, timestamp = Fri, 23 Aug 2024 08:12:16 +0200, biburl = bibsource dbip computer science bibliography, https://dblp.org State space models are amortized online learners. ArXiv preprint, abs/2407.14207, 2024. URL
http:///xxivvoorrab/////// 14207.
Tianyang Liu, Canwen Xu, and Julian McAuley. RepoBench: Benchmarking repository-level code auto-completion systems. ArXiv preprint, abs/2306.03091, 2023. URL abs/2306. 03091.
Colin Lockard, Prashant Shiralkar, and Xin Luna Dong. OpenCeres: When open information extraction meets the semi-structured web. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computationnl Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 3047-3056, Minneapolis, Minnesota, 2019, Association for Computational Linguistics. doi: 10.18653/v1/N19-11 1309. URL
org/N19 1309.
Peng Lu, Ivan Kobyzev, Mehdi Rezagholizadeh, Boxing Chen, and Philippe Langlais. Regla: Refining gated linear attention, 2025. URL https: / /ttp///axvvv//e///ee/eee/eeae.v/ wawiii.orgai ab////as//222 578.
15

---

Published as a conference paper at ICLR 2025
Eric Martin and Chris Cundy. Parallelizing linear recurrent neural nets sequence length. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC. Canada, April 30 May 3. 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https: //openreview. net/ het//orrumid-HHyUNWllC
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th Interationall Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: http::/ee///rrr// for.:ennetttoooommidddyyii/
William Merrill Jackson Petty, and Ashish Sabharwal. The Illusion of State in State-Space Models, 2024. URL https::/aaxxi..orrg//////////// http://www.rg/bb/////e// 08819.
MiniMax, Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang. Jiayuan Song, Jin Zhu, Jingtao Han, Jingyang Li, Junbin Xie, Junhao Xu, Junjie Yan, Kaishun Zhang, Kecheng Xiao, Kexi Kang, Le Han, Leyang Wang. Lianfei Yu, Liheng Feng. Lin Zheng, Linbo Chai, Long Xing, Meizhi Ju, Mingyuan Chi, Mozhi Zhang, Peikai Huang, Pengcheng Niu, Pengfei Li, Pengyu Zhao, Qi Yang, Qidi Xu, Qiexiang Wang. Qin Wang, Qiuhui Li, Ruitao Leng, Shengmin Shi, Shuqi Yu, Sichen Li, Songquan Zhu, Tao Huang, Tianrun Liang, Weigao Sun, Weixuan Sun, Weiyu Cheng, Wenkai Li, Xiangjun Song, Xiao Su. Xiaodong Han, Xinjie Zhang, Xinzhu Hou, Xu Min, Xun Zou, Xuyang Shen, Yan Gong, Yingjie Zhu, Yipeng Zhou, Yiran Zhong, Yongyi Hu, Yuanxiang Fan, Yue Yu, Yufeng Yang, Yuhao Li, Yunan Huang. Yunji Li, Yumpeng Huang, Yunzhi Xu, Yuxin Mao, Zehan Li, Zekang Li, Zewei Tao, Zewen Ying. Zhaoyang Cong, Zhen Qin, Zhenhua Fan, Zhihang Yu, Zhuo Jiang, and Zijia Wu. Minimax-01: Scaling foundation models with lightning attention, 2025. URL http:://arxiv.org/aa///550 08313.
Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with infini-attention. ArXiv preprint, abs/2404.07143, 2024. URL http:://arrii..rr//bb//2.... wwt.oorg///////////////// 071433.
Elvis Nunez, Luca Zancato, Benjamin Bowman, Aditya Golatkar, Wei Xia, and Stefano Soatto. Expansion span: Combining fading memory and retrieval in hybrid state space models, 2024. URL
https://arxiv. wrgg wrg./bs///////////o/// abs/aa//.... 1...13322 13328.
Antonio Orvieto, Samuel L. Smith, Albert Gu, Anushan Fernando, Çaglar Gülçchre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 26670-26698. PMLR, 2023. html.
URL httts::/proceediiie mml. pwwps/ess//eeeeeeeee https///2222/vvvveeee
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Katrin Erk and Noah A. Smith (eds.), Pro- ceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1525-1534, Berlin, Germany, 2016. Association for Computational Linguistics. doi: N0.18653/v1/PB
144. URL https:
org/P16 114444411144
Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. ArXiv preprint, abs/2406. 17557, 2024. URL htttts:///aaaatte wwtp:///.or/////////////oo/ss/// 2406. 17577.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao. Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito. Guangyu Song, Xiangru Tang. Johan Wind, Stanistaw Wožniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: Reinventing RNNs for the transformer era. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023,
16

---

Published as a conference paper at ICLR 2025
pp. 14048-1407) Singapore, 2023, Association for Computational Linguistics. doi: 10.18653/v1/ URL https: hattato//alannhhoooogy.. org/2023.... findings-emnnnn 936.
2023 wer..fiinnggemmnnn
Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, Przemyslaw Kazienko, Kranthi Kiran GV, Jan Kocoñ, Bartlomiej Koptyra, Satyapriya Krishna, Ronald McClelland Jr., Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song. Haoqin Tu, Stanislaw Wožniak, Ruichong Zhang, Bingchen Zhao, Qihang Zhao, Peng Zhou, Jian Zhu, and Rui-Jie Zhu. Eagle and Finch: RWKV with Matrix- Valued States and Dynamic Recurrence, 2024. URL httpp:://axxi...... wwwg/bb///aaa 05892.
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. Random feature attention. 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://opennreview net/forrum2ddeQtTTTTVvFFF
DI. Prados and SC Kak. Neural network capacity using delta rule. Electronics Letters, 3(25):197-199, 1989.
Zhen Qin, Dong Li, Weigao Sun, Weixuan Sun, Xuyang Shen, Xiaodong Han, Yunshen Wei, Baohong Lv, Fei Yuan, Xiao Luo, Y. Qiao, and Yiran Zhong. Transnormerllm: A faster and better large language model with improved transnormer. 2023a. URL https://app wen.emiimmmtssooiiie org/ CorpusID: 260203124.
Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for sequence modeling. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023b. URL http: http://aapppssssse cctppprrrcppppp...pppppppp httpp/papeeeeee//aeeeaaaaaaeeeee Files/paper/ 2023 1023/hashe (twee//69bbb34497779993333333999
6fele -bttsaattttttett-ttattttt----oo
html.
Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Lightnnng attention- 2: A free lunch for handling unlimited sequence lengths in large language models. 2024a.
Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. Hgrn2: Gated linear rnns with state expansion. ArXiv preprint, abs/2404.07904, 2024b. URL 07904.
https://arxil: http:://axii..oo/abb///////
Pranav Rajpurkar. Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784- 789, Melbourne, Australia, 2018. Association for Computational Linguistics. doi: 10. 18.5333//v1 P18-2124. URL https: / /aals//tollaaalloooooooo org/PP1-2124,
Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling. ArXiv preprint, abs/2406.07522 2024. URL nttps://arxinn https/////xiiooooaab////////
org/abs/24000 07522.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Con- ference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 8732-8740. AAAI Press, 2020. URL https:: /ttp://aaaaaaaaa http:://aaa...oo.......eee.ep/aa
6699.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Common- sense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP. IJCNLP), pp. 4473, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-141 URL htt.htttss//ccaanhoollooo org 1454.
17

---

Published as a conference paper at ICLR 2025
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers secretly fast weight programmers. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 9355-9366. PMLR, 2021a. URL. http: www.eednggg......ress/////////ee press//139/sschlgg111 html..
//proceedings. mlr.
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 9355-9366. PMLR, 2021b. URL http: html.
//proceedings.co mlr. press/v139/schlagg2) /
Mark Schöne, Babak Rahmani, Heiner Kremer, Fabian Falck, Hitesh Ballani, and Jannes Gladrow. Implicit language models are mns: Balancing parallelization and expressivity, 2025. URL https:
//arxiv. http://bb//2022078.0
Julien Siems, Timur Carstensen, Arber Zela, Frank Hutter, Massimiliano Pontil, and Riccardo Grazzi. Deltaproduct: Increasing the expressivity of deltanet through products of householders, 2025. URL https:/ /ttp:::///xvvv.oror
10297.
Jimmy T. H. Smith, Andrew Warrington, and Scott W. Linderman. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: htctpp://opemervieew.rr SHw3-kkKsS
Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artif. Intell., 46 46(1-2):159-216, 1990. doi: 10. 1011066000443770/999900077- 1016/0004-1000000000000000011000 (700333322(99)900))
URL https:://ol org/e/...rr1...
Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin. Learning to (learn at test time): Runs with expressive hidden states. ArXiv preprint, abss22407004620, 2024a. URL https: //axivv.//iiiv////////i www.ooggg//........... aa////0s///// 00.4407.4222
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuging Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. ArXiv 08621.
preprint, abs/2307.086211 2023a. URL
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. ArXiv http:/aaxvvvooorraaaa www.ooggg//b////oooo/ooo.o..os// 2307 08621.
preprint, abs/2307 00.621,,, 2023b. URL
Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures for language models. ArXiv preprint, 242220055055544 2024b. URL htttp:///axxivv wwtp:/vv..w.//b///////////// 2005240555.......
Harsh Trivedi, Niranjan Balasubramaniaan, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 1 5554, 2022. doi: 10. 10.1162/ 1 00475. URL https::/aclanteee http://aaaannnhooooorrrr org/ 2022. ta11-1.31.
Jos van der Westhuizen and Joan Lasenby. The unreasonable effectiveness of the forget gate. ArXiv preprint, abs/ 1t///884..040 804.04849 2018. URL https: /ttpp/////aaivoo///bbb////// 000p//111444449... 04849.
Johannes von Oswald, Maximilian Schlegel, Alexander Meulemans, Seijin Kobayashi, Eyvind Niklas- son, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Blaise Agliera Arcas, Max Vladymyrov, Razvan Pascanu, and João Sacramento. Uncovering mesa-optimizationn algorithms in transformers, 2024. URL htttps://aaaate. http://////axxw.o////b////// www.:orgg/bb///eee/e 2388285888...
Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, Garvit Kulshreshtha, Vartika Singh, Jared Casper, Jan Kautz, Mohammad Shoeybi, and Bryan Catanzaro. An empirical study of 07887.
mamba-based language models, 2024. URL https: httss:://axxvvooorle
18

---

Published as a conference paper at ICLR 2025
Ke Alexander Wang, Jiaxin Shi, and Emily B. Fox. Test-time regression: unifying framework for designing sequence models with associative memory, 2025. URL https://arxiv.. org/abs/ / 2501. 12352.
Kaiyue Wen, Xingyu Dang, and Kaifeng Lyu. RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval. ArXiv preprini, 195 2024. URL https://arxiv, .or.lv.orrgggooo.oo org/abs/ 2402. 18510.
Bernard Widrow, Marcian Hoff, et al. Adaptive switching circuits. In IRE WESCON convention record, volume 4, pp. 96-104. New York, 1960.
Songlin Yang and Yu Zhang. FLA: A Triton-Based Library for Hardware-Efficient Implemen- tations of Linear Attention Mechanism, 2024. URL https:/ /tts://bbb.b......////. thub. wwt. susttssosssssss Flash-lineat attention. original-date: 2023- 0023-12-20T0::5011822
Songlin Yang. Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efffiiiet training. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 56501 56523. PMLR, 2024a. URL https: / http://oooeeed.n...c..eo mlr. httparrrree//////agg222224 html.
Songlin Yang. Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. NeurIPS, 2024b.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answer- ing. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun't ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-2388, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10. 11865//1/DD8812555 URL https: / wwwpatoooancto..o..cco.o org/D18- 1259..
Luca Zancato, Arjun Seshadri, Yonatan Dukler, Aditya Golatkar, Yantao Shen, Benjamin Bowman, Matthew Trager, Alessandro Achille, and Stefano Soatto. B' MOJO: Hybrid state space realizations of foundation models with eidetic and fading memory. In The Thirty- Annual Conference Neural Information Processing Systems, 2024. URL htttp:://pperreeee RnQdRY1h5v.
net/forum?id=
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791 4800, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/ P19- 1472. URL https: /ac//aclaam// htt///aaaannnooolyooo/oo /
Michael Zhang. Simran Arora, Rahul Chalamala, Benjamin Frederick Spector, Alan Wu, Krithik Ramesh, Aaryan Singhal, and Christopher Re. LoLCATs: On low-rank linearizing of large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL
net/forummid-BBtteyyyy
Yu Zhang, Songlin Yang, Ruijie Zhu, Yue Zhang, Leyang Cui, Yiqiao Wang, Bolun Wang, Freda Shi, Bailin Wang, Wei Bi, Peng Zhou, and Guohong Fu. Gated slot attention for efficient linear-time sequence modeling. 2024. URL https: / /apii.
org /rrg/CCrpusIDD../ 272593079.
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. QMSum: A new benchmark for query-based multi multi-domain meeting summarization. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5905-5921, Online, 2021. Association for Computational Linguistics. doi: 10.1 00.18653/v1/2211 naacl- mone---ain47222 URL https: / ac aclanthology...t.. org/2021. mpacl-main.. 472.
19

---

Published as a conference paper at ICLR 2025
A EXTENDED WY REPRESENTATION FOR GATED DELTA RULE
To reduce notation clutter, we only consider the first chunk here. For S, the extended WY representation is
Mt u, u kit
S
u, = Br U1
We proof this by mathmetical induction.
Proof.
S (ac+1(I B Be-1ke41k(+a)) + Be+10++11144
= 0++11
22t u a44184411
u1 1x 1) + Bet1
f=1
1=1
1++1
k +
Bct1
1i+1
Yo+1 k,k ki+1
kipt
7
1=1
ui+1
7++1
u
+
7
U
7
Ye+1
2++1
u;k!
B EXPERIMENT CONTUNUED
B.1 EVALUATION
Commonsense reasoning Following Gu & Dao (2023), we evaluate our model on multiple com-
monsense reasoning benchmarks: PIQA (Bisk et al., 2020), HellaSwag (Hella.; Zellers et al., 2019), WinoGrandd (Wino.; Sakaguchi et al., 2020), ARC-easy (ARC-e) and ARC-challenge (ARC-c) (Clark et al., 2018), SIQA (Sap et al., 2019), BoolQ (Clark et al., 2019), Wikitext (Wiki.; Merity et al., 2017), and LAMBADA (LMB.; Paperno et al., 2016).
In-context retrieval Our evaluation comprises both synthetic and real-world tasks. For synthetic tasks, utilize the Needle-In-A Saree--------tttte Haystack Single (NIAH-S) benchmark suite from RULER (Hsich et al., 2024), which includes three increasingly complex tasks: S-NIAH-1 (passkey retrieval), S-NIAH- 2 (numerical needle in haystack), and S-NIAH-3 (word-based needle in haystack). For real-world tasks, following Arora et al. (2024b), we evaluate on diverse datasets: SWDE (Lockard et al., 2019) for structured HTML relation extraction, FDA (Arora et al., 2023b) for PDF key-value retrieval, and several question-answering datasets including SQuAD (Rajpurkar et al., 2018), TriviaQA (Joshi et al., 2017a), Drop (Dua al., 2019), and NQ (Kwiatkowski et al., 2019). Since our pretrained models lack instruction tuning, we employ the Cloze Completion Formatting prompts provided by Arora al. (2024b), which better align with our models' next-word-prediction training objective.
Long context understanding We evaluate on 14 tasks from Longbench (Bai et al., 2023), encom- passing: narrative comprehension (Narrative QA (Kočiský et al., 2018)), scientific understanding (QasperQA (Dasigi et al., 2021)), multi-hop reasoning (MultiField QA, HotpotQA (Yang al., 2018), 2WikiMulti QA (Ho et al., 2020). Musique (Trivedi et al., 2022)), document summarization (GovReport (Huang al., 2021), QMSum (Zhong et al., 2021), MultiNews (Fabbri et al., 2019)). and various specialized tasks (TRee (Li Roth, 2002), Trivia QA (Joshi et al., 2017b), SamSum (Gliwa al., 2019), LCC (Guo et al., 2023), and RepoBench-P (Liu al., 2023)).
20

---

Published as a conference paper at ICLR 2025
Table S.I: Ablation study the Gated DeltaNet block. Avg-PPL and Avg-Acc denote average perplexity and zero-shot commonsense reasoning aceuracy (as Table 3), respectively. All models have 400M parameters and are trained for 15B tokens on the same subset of Fine dataset (Penedo et al., 2024).
Gated DeboNer Ablations (400M)
Avg: PPL (1)
Avg-Acc (1)
Gated DeltaNet Head Dim 128,
27.35
47.26
Macro Design
maive Delta Rule
w/o. Short Conv
28.95
46.16
w/o. Output Gate
29.12
45.46
w/o. Output Norm
27.55
47.07
Normalization Feature Map
Lu-norm ReLU
30.79
45.92
Li-norm 1+ELU
30.34
Li-norm SiLU
30.18
46.09
La-norm & ReL.U
27.67
46.94
L2-norm & 1+ELU
27.58
47.17
Model Dimensions
Head Dim 64
28.31
46.35
Head Dim 256
27.13
47.38
B.2 ABLATION STUDY
Model
PIQA
Hella.
Wino.
ARC-e
SIQA
BoolQ
Avg.
ppl
acc
Hybrid Ablations (500M/15B)
Gated SWA Mamha2
40.84
Gated DeitaNet Mamba2 SWA
23.69
26.83
36.17
67.51
41.51
51.85
61.19
29.77
38.58
53.73
47.54
Mamba2 SWA Gated DeliaNet
24.14
25.21
36.79
64.96
41.18
52.01
60.90
30.03
38.07
59.44
47.92
Mamhu2 Gated DeltaNet SWA
23.54
24.11
36.92
66.48
41.70
52.72
61.06
30.54
39.91
60.51
48.73
Table S.2: Ablation studies of Gated DeltaNet models. All evaluations performed by using In-evaluation-han (Gao et al., 2021). All models the Llama tokenizer and trained the same subset the FineWeb-Edu dataset (Penedo et al., 2024).
Table S.I presents ablation studies on the Gated DeltaNet block's components. Our experiments demonstrate that both the short convolution and output gate crucial for model performance, while output normalization yields marginal improvements. Consistent with Yang et al. (2024b), we found L2 normalization be essential for optimal performance, though the choice of feature map was less influential. Nevertheless, SiLU consistently outperformed other activation functions, aligning with observations from Qin al. (2023a). Through empirical analysis, we determined that head dimension of 128 provides an optimal trade-off between performance and computational efficiency. Additionally, Table S.2 demonstrates that among various hybrid architectures, the combination of Mamba2, Gated DeltaNet, and SWA in this specific order produces superior results.
21


Reviews:
[{"review_id": "3MgIS5q0hx", "reviewer": "Reviewer_eAAH", "summary": "To mitigate quadratic scaling with sequence length of standard Transformers, Linear Transformers with kernelized dot-product linear attention are proposed. In this paper, the authors propose Gated DeltaNet, which leverages the gating mechanism from Mamba2 and the delta update rule from DeltaNet. The hybrid architecture of Gated DeltaNet, SWA, Mamba2 layers outperforms baselines in language modeling, reasoning, and recall-intensive tasks.", "questions": "- There are some typos in Section 3.1 (e.g., L.239 or L.256).", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper is well-organized and clearly written.\n- Their presentation, especially highlighting contribution points, was good to understand and follow.\n- The proposed method is simple, but seems effective in various tasks.", "weaknesses": "- How are the baselines trained? Those few-shot performance look a little lower than other papers (like Mamba). And also, could you compare with some other well-pretrained Transformer models to show the effectiveness of the proposed architecture?\n- State sizes of Gated DeltaNet H1 and H2 seem much larger than all the other baselines---models that use SWA have the larger sizes. In case of Gated DeltaNet (with 256 state size) performance looks similar to Mamba2 and DeltaNet, where the gated delta rule can be seen as having minimal impact on performance. Also, as Samba already shows a very good performance, I guess the performance gain would come from hybrid structure, not from gated delta rule itself.\n- I suggested the authors to compare efficiency of models, which is an important factor for a new architecture.\n- In length extrapolation experiments, why does Gated DeltaNet show robust performance while Mamba2 and DeltaNet show higher perplexity as the length grows. Gated DeltaNet is the combination of both methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ma0UHD1STW", "reviewer": "Reviewer_JW3m", "summary": "This paper addresses the limitations of linear Transformers in handling long sequences and recall-intensive tasks. While linear Transformers reduce computational complexity compared to traditional Transformers, they often struggle with tasks that require remembering and recalling information over extended contexts. Authors focus on (1) Gating Mechanism: Allows the model to adaptively forget irrelevant information by applying a dynamic decay to the hidden state. This enables fast and efficient memory erasure but may not precisely target specific memories. (2) Delta Update Rule: Enables precise and targeted updates to memory by selectively replacing old key-value pairs with new ones, to propose a new approach called Gated Delta Rule, which combines the strengths of both mechanisms.", "questions": "1. Could you provide quantitative metrics on the computational efficiency of Gated DeltaNet compared to other models, such as training/inference speed and memory consumption?\n\nI will read the rebuttal in the discussion period and adjust my evaluation.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "## Strengths\n\n1. This paper propose a new linear Transformer that implements the gated delta rule, allowing for more effective memory management over long sequences.\n2. The authors extend the delta rule's parallel algorithm to incorporate gating, ensuring hardware-efficient training using chunkwise parallelism and tensor core acceleration.\n3. Experiments demonstrate that Gated DeltaNet can outperforms existing models like Mamba2 (which uses gating) and DeltaNet (which uses the delta rule) on various tasks.", "weaknesses": "## Weakness\n\n1. The paper lacks an in-depth theoretical analysis of why the combination of gating and the delta rule improves memory capacity and how it affects the model's ability to handle long sequences, providing theoretical insights regarding the memory retention and forgetting properties of GatedDeltaNet would strengthen the paper's contributions.\n2. Although the authors claims that Gated DeltaNet generalizes better to longer sequences than DeltaNet / Mamba2, the experimental results on length extrapolation are limited (only one analysis on PG19). The evaluation focuses on sequences up to a certain length without exploring the model's performance on substantially longer sequences. Expanding the extrapolation experiments to include much longer sequences would provide stronger evidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g7Fq3F49yx", "reviewer": "Reviewer_jNuR", "summary": "The paper introduces Gated DeltaNet as an enhancement to linear transformers, combining two memory management mechanisms: the gating mechanism from Mamba2 and the delta update rule from DeltaNet. Linear transformers, known for efficiency, often struggle with recall-intensive tasks. The gating mechanism enables flexible, rapid memory clearance, while the delta rule allows for targeted updates. By integrating these, Gated DeltaNet achieves both efficient memory erasure and selective memory retention, addressing limitations faced by linear transformers in handling extended sequences and complex memory recall tasks.\n\nThe proposed model demonstrates superior performance compared to Mamba2 and DeltaNet across various tasks, including language modeling, commonsense reasoning, and associative-recall. Additionally, the study explores hybrid models that combine Gated DeltaNet with sliding window attention or Mamba2 layers to further improve retrieval capacity. Extensive experiments validate the effectiveness of the gated delta rule, highlighting its advantage in managing memory over long sequences and maintaining hardware efficiency through chunkwise parallelism for training.", "questions": "1. Is there any visualized results or case studies about why the introduction of scalar value decay term resulted in following improvements in downstream task like AR?\n\n2. How about the training / inference speed e.g. throughput comparing with DeltaNet, Mamba2, and hybrid model e.g. Samba?\n\n3. Does the UT transform really helps the acceleration of training? How about some ablation studies on w/ and w/o the UT transform?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper gives detailed and structured introduction to a series of d^2 linear attention model e.g. GLA, DeltaNet and Mamba2, following a newly proposed Gated DeltaRule as its main contribution.  The improvements of delta rule can be considered as one strength in this paper, with a scalar-values decay term for easy training. Also, the UT transform for using Tensor Core with high-efficiency serves as a mitigation to the extended WY representation is unique, yet could be an individual interest to the community.\n\n The motivation for combining gating with the delta rule is compelling, showing how these two mechanisms address specific weaknesses in memory management. The writing in this paper is generally clear, structured, and detailed.", "weaknesses": "One weakness of the paper lies in the use of a scalar-value decay term, which, while intended to enhance memory management, does not consistently outperform Mamba2 in associative recall tasks. As seen in Table 2, although Gated DeltaNet shows a slight improvement in average accuracy across six associative recall tasks, Mamba2 still performs better in three of them. This raises questions about the efficacy of the decay term in improving recall performance. Intuitively, adding a decay term implies a rigid truncation of information over time, which may not necessarily benefit associative recall tasks that rely on longer memory retention. Providing case studies that track changes in hidden states during tasks could clarify the impact of the decay term on associative recall. Such insights would help to understand how the decay term influences memory retention and retrieval processes within the model, making the improvement more transparent and justifiable.\n\nAnother issue concerns the length extrapolation experiments involving DeltaNet. The paper would benefit from clearly labeling the four model types and specifying the experimental setup in detail. For example, it appears that DeltaNet performs even worse than Mamba in extrapolation, which contradicts findings from other experiments within the paper. This inconsistency suggests that additional clarification on the model configurations, including whether any of the models employed a hybrid architecture, is necessary. Given that attention mechanisms offer significant advantages in associative recall capabilities, it is essential to specify if the models were using sliding window attention or other hybrid elements, as this could substantially influence the results and better contextualize the model comparisons.\n\nTypo: \n- line 256, \"As we can see in .\" need a reference?\n- line 313, 319, 334, \"Training\" -> \"Training.\"\n- line 374, \"State size is strongly correlated with final performance.\" need a \\paragraph{}.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ja6vgbX0IK", "reviewer": "Reviewer_3Thw", "summary": "This paper proposes Gated DeltaNet, a novel model architecture combing gated recurrent mechanism and Delta update rule. Gated DeltaNet support chunk-parallel computation as efficiency optimization. Besides simple Gated DeltaNet, the paper also discuses the hybrid design with Sliding-Window Attention or Mamba layers.\n\nIn experiments, the paper shows strong modeling capability of Gated DeltaNet. First, Gated DeltaNet outperforms Transformer and other linear-complexity models in language modeling and zero-shot downstream tasks. Second, Gated DeltaNet shows a strong length-extrapolation ability in language modeling. In ablation studies, the paper makes comparisons with other design choices.", "questions": "How do you think the application scenario of linear-complexity models? Since it is going to be common sense that linear complexity makes a sacrifice on long-context capability.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written, showing the connection and improvement with previous works.\n\n2. Gated DeltaNet makes kernel optimization to achieve better training and inference throughput, which is essential for downstream application and community reproduction.\n\n3. Gated DeltaNet shows strong performance than other linear-complexity models, which is a good academic contribution in model architecture area.", "weaknesses": "1. This paper does not discuss the performance on long-context tasks. Since the quadratic complexity only matters when the sequence is long, long-context performance is the most important metric for linear-complexity models.\n\n2. This paper does not compare Gated DeltaNet with other linear-complexity models on training and inference efficiency. Since Delta-rule is harder for parallel computation, there may be a trade-off between performance and computation cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)