Parallelizing Linear Transformers with the Delta Rule over Sequence Length
Authors: Songlin Yang
Bailin Wang
Yu Zhang
Yikang Shen
Yoon Kim
Venue: NeurIPS 2024 poster (2024)
Keywords: linear transformer
recurrent neural network
hardware-aware algorithm
state space model
sequence modeling
efficient training
Primary area: deep_learning_architectures
TL;DR: DeltaNet, i.e., Linear Transformer with delta rule, suffers from slow recurrent training. This work develops a novel algorithm that enables the parallelization of training over sequence length, allowing for large-scale experiments

Abstract: Transformers with linear attention (i.e., linear transformers) and state-space models have recently been suggested as a viable linear-time alternative to transformers with softmax attention. However, these models still underperform transformers especially on tasks that require in-context retrieval. While more expressive variants of linear transformers which replace the additive update in linear transformers with the delta rule (DeltaNet) have been found to be more effective at associative recall, existing algorithms for training such models do not parallelize over sequence length and are thus inefficient to train on modern hardware. This work describes a hardware-efficient algorithm for training linear transformers with the delta rule, which exploits a memory-efficient representation for computing products of Householder matrices. This algorithm allows us to scale up DeltaNet to standard language modeling settings. We train a 1.3B model for 100B tokens and find that it outperforms recent linear-time baselines such as Mamba and GLA in terms of perplexity and zero-shot performance on downstream tasks. We also experiment with two hybrid models which combine DeltaNet layers with (1) sliding-window attention layers every other layer or (2) two global attention layers, and find that these hybrids outperform strong transformer baselines.

Parallelizing Linear Transformers with the Delta Rule over Sequence Length
Songlin Yang
Bailin Wang
Yu Zhang
Yikang Shen'
Yoon Kim
Massachusetts Institute of Technology
Soochow University MIITIIM Watson AI Lab
yangs1660mit.edi
Abstract
Transformers with linear attention (i.e., linear transformers) and state-space mod- els have recently been suggested viable linear-time alternative transformers with softmax attention. However, these models still underperform transformers especially on tasks that require in-context retrieval. While more expressive vari- ants of linear transformers which replace the additive update in linear transformers with the delta rule [DeltaNet; 101] have been found be more effective asso- ciative recall, existing algorithms for training such models do not parallelize over sequence length and are thus inefficient train on modern hardware. This work describes hardware-efficient algorithm for training linear transformers with the delta rule, which exploits a memory-efficien representation for computing prod- ucts of Householder matrices [11]. This algorithm allows us to scale up DeltaNet to standard language modeling settings. We train a 1.3B model for 100B tokens and find that it outperforms recent linear-time baselines such as Mamba [31] and GLA [124] in terms of perplexity and zero-shot performance on downstream tasks. We also experiment with two hybrid models which combine DeltaNet layers with (1) sliding-window window attention layers every other layer or (2) two global attention layers, and find that these hybrids outperform strong transformer baselines.
1 Introduction
The attention mechanism [8, 116] has been shown to be an important primitive for accurate sequence modeling. Attention is moreover efficient during training as it is rich in matrix multiplications and thus take advantage of highly parallel processing capabilities and specialized accelerators modern GPUs. However, the complexity of attention is quadratic in sequence length, and hence is fundamentally expensive primitive. And while recent techniques have made it possible scale at- tention to longer sequences through hardware-awan restructuring of the intermediate computations [20, 18, 59, 14], these methods still require storing the key/value vectors of previous elements, and this "KV cache" (whose size grows linearly) can be unwieldy to manage for long sequences.
Linear attention transformers [48] replace the exponential kernel in softmax attention with dot- product over (possibly transformed) key and query vectors. This makes possible to formulate linear attention as a linear RNN with matrix-valuce hidden states, thus obviating the need for KV cache and enabling constant-memen memory inference. While initial variants of linear attention generally underperformed softmax attention language modeling, gated variants of linear attention which incorporate data-dependent gating factor have recently been shown to be competitive against strong transformer baselines [124, 92, 9, 79]. These gated linear transformers, along with time-narying state space models such as Mamba [31, 19] (which can be reparameterized as a gated linear transformer [124]), have been suggested
potential alternative ordinary transformers. However, despite
The parallel DeltaNet layer is made available as part of the FLASHLINEARAATEETIOO library [124, 123]: https://githhb. com/fla..//...o...ooo.....o....e ht/.f/laallrraarreeeeee org/flash- linear attention
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the competitive language modeling performance, these models have been shown to underperform transformers on recall-intensive tasks [6. 7], which is important for many practical downstream tasks of interest (e.g., in retrieval-augmented generation [53]).
To enhance associative recall over long contexts, Schlag al. [101] propose DeltaNet, variant of a linear transformer which uses delta rule-like update [121] to retrieve and update value vector that is associated with the current key. DeltaNet was found to be effective on synthetic tasks and small scale language modeling/machinn a sequential algorithm that did not parallelize across sequence length, thus resulting in hardware- inefficient training, and it has not been clear how to scale DeltaNet to larger models and datasets.
translation. However, the original work used
This work describes cerdwwre--fficienn training algorithm for DeltaNets which parallelizes the for- ward/backward passes across sequence length. We reparameterize the DeltaNet as a matrix-valued RNN whose recurrence is given by a generalized Householder transformation. This reparameteriza- tion enables the use of the compact WY representation eliminating the need materialize the hidden states of matrix size at each time step during paral- lel training, which would otherwise result in high I/O costs. The memory-efficient representation makes possible to straightforwardly extend the chunkwise parallel strategy for training linear at- tention models [34, 108, 124] to the DeltaNet case. We scale DeltaNets to moderate-scale language modeling benchmarks (1.3B models trained on 100B tokens), where DeltaNet is found to obtain better language modeling and zero-shot downstream task performance than strong linear recurrent models such as Mamba [31] and GLA [124]. For in-context retrieval and learning evaluation, we evaluate DeltaNet on synthetic and real benchmarks [4, 2, 84, 6], where it is again found to perform well against linear recurrent baselines. Finally, we experiment with hybrid approach where we combine DeltaNet layers with sliding attention layers or global attention layers, and find that these hybrid models can improve upon ordinary transformers, as well as the pure DeltaNet transformer.
[11] for products of Householder matrices,
2 Background
2.1 Linear Transformer: Transformers with Linear Attention
Given sequence of d-dimensional input vectors I I,...., Lr transformers use the softmax attention mechanism to attend over the entire past,
exp(ki q:)
qe: kr, Va = WQ WXY,, WVx,,
O EE= exp(k'g.)) Ui.
where Wo. WK. Wy E [dd k, Uc. Or E Rdd (Here we assume a single attention head for simplicity). Linear attention [48] replaces the exponential kernel exp(k; g:) with the dot-product (kk,)" ((q.) where á _R + R is feature map. This makes it possible to rearrange computations to represent linear attention as a linear RNN with matrix-valued hidden states,
EE (k;))) d(q.)
(k:) p(q))
S,²(q:)
o
v. (5=10(ky)) Φ(q()
==1 (k;,)) ((qi))
2, (())))
where S, = = "ko(k; Rd and z: = Ei= okk)) E R". we allow n to go to infinity, linear attention can feature maps associated with polynomial kernels to compute a polynomial approximation to the exponential kernel as a dot product, and can thus approximate softmax attention arbitrarily well [6]. The denominator z, ((q)) E R can result in numerical instabilities [86] and is removed in recent works [101, 63]. It is also common to use the identity mapping for Φ [63, 108), which results in the following simplified linear transformer: S. S + v,k,, o: S.q.
Efficient training. Let Q.K. V e R be the stacked query, key, value vectors, e.g., Q = qi We can then compute the output O E 1xd in parallel via O = (QK" @ M,) V, where Mr E I I. is the causal mask. This fully "parallel form" and the above "recurrent form" have different FLOPs and parallelization tradeoffs. The parallel form takes O + Ldf) and thus requires more FLOPS than the recurrent form, which takes O(Ld). However, the parallel form is often much faster in practice for moderate-length sequences as can be done in O(1) steps. This sequence-level parallellism also enables high GPU occupancy. The recurrent form requires fewer

---

FLOPs but cannot be parallelized across sequence length' and the elementwise operations involved in recurrence moreover cannot make use of specialized matmul accelerators (e.g., tensor cores).
Chunkwise parallel form. The chunkwise parallel form [34, 108, 124] strikes balance between the parallel and recurrent forms, allowing for fewer FLOPs than the parallel form and more sequence- level parallelism than the recurrent form. Concretely, suppose the query/key/value vectors are split T chunk t, and let qig = 9iC++ be the i-th query vector within the t'th chunk; the key/value chunks are defined similarly. Note that I E [0, TL/C), i [1, C). The state matrices are also re-indexed such that (/ Sic+i. and we additionally define S = S
into & chunks where each chunk is of length C. Let Qu4 E IRC'dd be all the query vectors for
[t-1)> i.e., the initial state of a chunk is the last state of the previous chunk. We can then obtain the following identity for the hidden state and output vector for the r-th element within the t-th chunk,
6 S
Σ
+ Σ ofel
(8)
By further rewriting the intra-chunk computation based on the parallel form, we obtain following,
S S|++1| = [S + Vin K(t)
E Rddd
(1)
0 [1] Q/1 (q Mc) V(d
R Cxd
where we let S/n = Stg to reduce notational clutter. With this "chunkwise parallel form", informa- tion is propagated chunk-to-chunk. through Stg,, and the intra-chunk states Sig for i E [1, C] need not be materialized, thus saving memory.
The complexity of the chunkwise parallel form is O(LCd+ Ldf), and the number of steps (without chunk-level parallel scan) is O))). Hence, C = T. recovers the fully parallel form and C = 1 recov- ers the recurrent form. The chunkwise parallel form allows us to interpolate between the two forms, in essence trading off the number of sequential computations against sequence-level parallelism. In practice C is set to a small constant (usually 64 or 128), allowing for subquadratic training. This chunkwise form enables practical speed-ups against parallel-form-ont softmax attention even on moderate-length sequences, as demonstrated by
[124, 123]
2.2 DeltaNet: Linear Transformers with the Delta Update Rule
The above linear transformer employs simple linear recurrence: S, S + v,kk This can be seen as additively updating the memory S with new key-value associations at each time step. However, purely additive update rule makes difficult deallocate past key-value associations, eventually leading to key "collisions" when L > d, as pointed out by Schlag et al. [101]. A model should ideally learn to remove less important key-value associations to make room for new ones, and this removal should depend on the interaction between the new key and the memory content.
From a fast weight programming [102] perspective, the recurrent hidden state of linear attention is the fast weight mapping the input qe to output o, with a Hessian-lin like update rule, which has limited memory capacity [68]. DeltaNet, on the other hand, uses the delta update rule or the Widrow-Hof learning rule [121] for fast weight update: S, = S (S KG Uy kk where 3r is the learning rate, k, represents the current prediction, e, the target value. The method derives its name from the core principle of updating weights based on the "delta" (difference) between the prediction S.- k; and the target v., which has been shown better memory capacity [28, 85, 56, 101]. This process can also be regarded as optimizing an online regression loss using single step of SGD,2
(S) ]
1 Sk, vy 11112, S. S V.. (S,,)))) S ( vy) v.)kk
which has been discussed in several recent works [110, 58, 125, 10]. In contrast, linear transformers
'It is possihle in theory to use parallel scan [13] to parallelize the recurrent form, which would enable the computations to be performed in O(log L) steps and O(Ld)) FLOPs. However, this approach requires materializing the 2D hidden state for each time step. which would incur significant memory I/O cost unless the state size small enough such that materialization can happen in faster memory (i.e., as Mamba [31]). "This perspective was discussed as early as Widrow et al. [121], which later became known the Least Mean Squares (IMSS algorithm and has found widespread applications in signal processing [97].
3

---

can be regarded as optimizing an online linear (negative inner (nnor-product loss Le = (Sk; Uz).
An alternative interpretation for DeltaNet is from the perspective of key-value retrieval. It first retrieves the old value using the current key, og = S It then obtains a new value ot by interpolating between the old value and the current value vy, which replaces will in the memory:
ot 3,0, + (1 3e) 0
S. S of K
H. K
remove
wrile
Here o B E (0, 1) is soft "writing strength": when 3g = 1, the old value is completely removed and di new 0y; when 3, = 0, the memory remains unmodified and we have S, - The output computation is the same as vanilla linear attention, i.e., Og S.qr. The complexity of this recurrent form the same as that of vanilla linear attention, i.e., O(Ld))
Schlag et al. [101] demonstrate that DeltaNet outperforms ordinary linear transformers on small- scale language modeling and synthetic in-context retrieval tasks. However, their training algorithm. an extension of the memory-efficient recurrent implementation for linear Transformers [48, $3.3.1], strictly sequential and thus hardware inefficient, as discussed in [124, $3.2], motivating us to derive an equivalent chunkwise algorithm to train DeltaNet at scale, as introduced below.
3 Parallelizing DeltaNet Across the Sequence Dimension
3.1 A Memory-efficient Reparameterization
We first observe that S, admits a purely additive representation of the form S, = u for Ui, k; E Id,, since we can simply set u; D.aw woah Bi(u; o Recall from $2.1 that simple linear attention has the form S, = Ei= i= u,k: Thus, DeltaNet simply replaces the value vector 0; in linear attention with the "pseudo" value vector u, Once the u, 's have been constructed, the rest of computation can proceed as in ordinary linear attention, i.e., (QK' M) U where U E Lxd is the row-wise concatenation of the u, vectors.
However, computing naïvely requires explicitly materializing S to compute uo which would require (d2) memory. We now show that we can obtain the u, 's without explicitly materializing S,-1 in (d) memory. Our simple proof (by induction) relies on application of the WY represen- tation for products of Householder matrices [11]. The base case is clear since we have S = B Bi k so u1 = 31 For the inductive step, we first observe that the DeltaNet update is given by.
new k S 3, (S,- k B. S B,k,k²) + B S, S ex k
which can be seen as applying a generalized Householder transformation (i.e., matmul with an iden- tity plus rank-one matrix) the previous state. The inductive step is then given by,
1-1
S. = S. 3,kaki + B,ock
Σ u;kk + Br Uz Σ
ui
k
ke
ki
Σ u (3)
Y=1
defined u4,
Note that u, does not require materializing any of the hidden states and requires O(d) memory to compute, thus completing the proof. While we have avoided materializing S, 'ss computing u, 's for all L (that is, U) takes O (L) and moreover cannot be fully parallelized, unlike in linear attention where we can calculate all the value vectors V in parallel O(1) steps. We now show that the above trick still enables an efficient chunkwise parallel form for DeltaNet.
3.2 Chunkwise Parallel Form for DeltaNet
To derive the chunkwise parallel form, we first unroll the recurrence,
( (I- (1-3, (- j=i+1
S. S -1-kkk))kk Boock = Emlerh
'While quadratic loss (used DeltaNet) provides gradients that scale with error magnitude, offering stronger corrections when predictions are far from less of prediction error. This gradient behavior provides an intuitive explanation for why linear transformers underperform DeltaNet in in-context retrieval tasks.
the target, the gradient of linear loss remains constant regard-

---

We then define the following variables: P = II (I - 3eksk)) E Raxd Hi Ex= (vrkk)
pdxd, where we let P whenever i j. Intuitively, P// is the "de-
cay factor" to be applied to S; for obtaining S, and H2 represents the contributions to Sj starting from token (Hence S, H The chunkwise recurrence can then be written as.
S/ S Pi + Hi
(5)
where we define the chunkwise variables S/g/ Sic+i, P PiCtr Hi =H
IC41 Here
have & chunks of size C. The trick is to now efficiently represent the [e]. Hio using a similar approach described in $3.1, so that these matrices can be stored in (dd) memory.
[c] E Raxd matrices
P = I
[0]
[
10 [0)
E Rdxd
(6)
i=1
Σ ==1
0 Σ
" = 1/1
4/1
Big
E IR
The derivations for the above can be found in the appendix. Subsequently, based on Eq. 5, we can obtain the chunk-level recurrence for hidden states and outputs as,
1/ +S/
w
ET [1
+
= S/ + Σ (ute: syywi))) ki0]
iml
x=1
% = T = S/ C
+ Σ
i=1
Letting S/y = (c). the above can be simplified to matrix notations similarly to Eq. 1-2,
S S = Sig + (Ung WwiS(a) Kjg-
(8)
O [1] Q l + (Q K 3 M) (Um W W (1
(9)
where mg =0
[e) E Cxd for O E (Q,K, V, U, W) defines the chunkwise matrices that are formed from stacking the q:, k2, Ut, ot, u4 we vectors.
Practical considerations. In the above, Eq. 7 is fully recurrent and thus cannot use tensor cores written is. To solve this, further leverage the UT transform [44, 23] (see $B.2 for derivations):
T = I tril(diag(()) (31))) K K()- (1)
1))
-1 diag (3t1)
(10)
W1O [g = K
U [ = T. V
(11)
to rewrite most operations in matmuls. The inverse of lower triangular matrices could be solved efficiently using forward substitution. Once computed, the hidden state updates (Eq. 8) and the output computations (Eq. 9) are largely the same as MAASHLLNNARATEENTII
hest diazz64 dim-- head dim-256
vanilla linear attention. We adapt
Spect
[124] to implement Eq. and with hidden states recomputed during the backward pass for saving GPU memory. The Py- Torch pseudocode for the forward pass is shown Listing 1.
10
0.5
Speed comparison. We implement both the pure recurrent form' and the chunkwise parallel form Triton [112] and show the speed-ups for various sequence lengths (I.) and head dimen- sions (dhead) in the right figure, where the model dimension d' is 2048 and we vary batch size and sequence length so that they multily to 16384.5 Our chunkwise al- gorithm achieves greater speed-ups as sequence length L and head dimension dhead increase, where the use of sequence-level parallelism (for high GPU occupancy) and tensor (for fast matmuls) become more important [124, $3).
Soquence Longth
Figure 1: Speed-up of the chunkwise parallel form vs. the recurrent form.
Note that our recurrent kernel already 2x faster than the original CUDA kernel from Schlag et al. [101]. *oo far we have been assuming single head (dueat = d) for easier exposition. In practice we use multiple heads where the head dimension dhead is smaller than the model dimension d. We thus have S, E Rd
5

---

Fully Parallel Form for DeltaNet. For completeness, we also discuss the fully parallel form of DeltaNet. While we use the concept of "pseudo" value, it possible avoid modifying values. From Eq. 4, it is straightforward compute the attention matrix A: A., = kj 1+1 j S i and O otherwise. Notably, A has the matrix form A = (QK" @ M) T, obtained by combining Eq. 3 and 11. However, computing 'T requires a matrix inverse (Eq. 10), which scales cubically with sequence length without further algorithmic changes. Due to the above we avoid using the fully parallel form for training DeltaNet; however the "attention" matrix derived from this form could be of interest to the interpretability study for RNNs [3, 134].
Octpats
3.3 DeltaNet Transformer
We describe how the DeltaNet layer primitive is used to build up a transformer-like model using standard modules. We largely follow the -arhitecccctecturre (Transformer++,, 114] and simply replace the self-attention layer with the DeltaNet layer. We also apply normaliza- tion before output projection for stable training [86, 69]. As the additional parameters for com- puting scalar 3, terms are negligible, parame- ter allocation is roughly the same as in Trans- former++, i.e., 4d for the DeltaNet layer and 8dl for the SwiGLU FFN layer [103].
KMNNNerm
Liteer
SwGGLEU
RMSNiem
Deli
+
A,3
DeluNee
Narm
RMSNoett
Lioy
Linear
Linex
Linear
Inpurs
Figure 2: An illustration of DeltaNet neural architec- ture.
SiLU W 2x) Feature map and normalization. Our key/query vectors are given by k2 ||SLU nz llla q:
[SILUU q=)))l= Schlag et al. [101] originally follow Katharopoulos et al. [48] and apply a "ELU+1"
[17] to nonlineary transform the key/query vectors. We instead use the SiLU activation [24), which was found to perform better [88, 19]. For stability, it is crucial to ensure that the norm of each eigenvalue of the transition matrices does not exceed one. The eigenvalues of I - Bekak, are I with multiplicity d - 1 and Be ||/2 with multiplicity 1. Schlag et al. [101] used the l, norm to normalize query/key vectors, ensuring that 0 S 1 1-3e k; ||2 S 1. We instead apply I.2 normalization, which we found to perform better and offers a more intuitive interpretation: when Br = 1, - k,k; becomes projection matrix, erasing information in one subspace while preserving the other d - 1 subspaces. This is beneficial for retaining information while enabling more targeted forgetting.
3.4 Hybrid Models
Following recent work combining subquadratic token-mixing layers with existing neural network primitives [6, 21, 55], we also experiment with hybridizing DeltaNet models.
Convolutional layers. Recent linear recurrent models typically incorporate a lightweight depthwise-sepaaable convolution layer after the query/key/value projections [31, 9, 19]. This "short convolution" layer [83] generalizes the shift SSM [26], and is efficient in both number of parameters and computational cost. We also add a short convolution layer after the query/key/value projections.
Local sliding window and global attention. Linear attention largely uses a content-based ad- dressing mechanism [29] and lacks positional information [129]. Arora et al. [6] also argue that linear attention lacks the ability to perform precise local token shifts and comparisons, thus facing difficulties on retrieval-intensicc tasks. Motivated by this, we experiment with two different hybrid architectures that incorporate softmax attention. We first explore sliding window attention (SWA) which has been shown to significantly improve linear attention [86, 6. 57, 75]: we follow Griffin [21] and Samba [95] to interleave DeltaNet layers and SWA layers. We also experiment with global attention, which has been found to be helpful [51, 35] even if only few of the recurrent layers replaced with global attention [55]. We follow Fu et al. [26] to replace only two layers with global attention: the second layer and the ( 1 + 1)-th layer, where N is total number of layers.
4 Empirical Study
We compare the DeltaNet against strong baselines in both synthetic and real-world language model- ing settings. Our main baselines include: A Transformer++ [114]; RetNet [108). linear attention Transformer with non-data-dependen exponential decay and large head dimension;

---

GLA [124], a linear attention Transformer with data cuta-dependent decay; and Mamba [31], a selective state-space model with data-dependent decay.
4.1
Synthetic Benchmarks
We evaluate on three synthetic benchmarks: Muery associative recall [MQAR; 4), Mechanistic Architecture Design IMAD; 84), and in-context language learning [RegBench; 2].
Modd
Compress
Fuzzy Recall
In-Context Recall
Memorize
Noisy Recall
Seleetive Copy
Sequence Longth: 512, Key Value Piiss 64
100
Transformee
Hyena
51.6
45.2
29.8
7.9
81.7
94.1
85.2
89.5
78.8
86.8
99.6
93.1
66.0
74.5
Accomoy (90
Manto
DebaNee
Multihead Hyena
41.8
14.4
99.0
89.4
98.6
93.0
73.2
Mamba
$2.7
6.7
90.4
89.5
90.1
86.3
69.3
GLA
38.8
6.9
63.3
81.6
88.6
DeltaNet
42.2
35.7
100
52.8
100
100
71.8
135
255
512
Figure Results on the synthetic MAD benchmark. Results other than DeltaNet are directly borrowed from Poli al. [84]. (Multi-head) Hyena, Figure 4: Accuracy (%) on MQAR. DeltaNet and Mamha make use of convolutions, whereas GLA does not.
Moddet dimeeesoo
MQAR evaluates language models' ability to (in-context) recall information within context when faced with multiple recall queries. We use Arora al. [4]'s training setting and for DeltaNet we use 2 heads. We do not use convolutions for these experiments. Figure 4 shows that DeltaNet performs perfectly (even without convolution) in the hardest setting and outperforms Mamba (which uses convolutions) in the low-dimension setting. Next, we consider the MAD benchmark [84], suite of synthetic token manipulation tasks designed to probe capabilities of model architectures. The results shown in Table 3. Compared with other architectures, including MHA, DeltaNet is better at recalling tasks, especially on Fuzzy Recall as expected, although it somehow struggles on the "Memorize" task. We defer the RegBench results to the A.2 due to space constraints.
4.2 Language Modeling
Experimental setup. Following prior work [31, 124], we evaluate Wikitext perplexity and zero-shot common sense reasoning tasks, including LAMBADA [LMB.: 77], PiQA [12), Hel- laSwag [Hella.; 127], WinoGrande [Wino.; 99], ARC-easy (ARC-e) and ARC-challenge (Are-c) [16]. Following Arora et al. [6], we also evaluate the models real-world recall-intensive tasks, in- cluding FDA [5], SWDE [60], and SQUAD [93]. Both SWDE and FDA focus on extracting struc- tured information: SWDE from raw HTML to identify semi-structured relationships, and FDA from PDFs retrieve key-value pairs. SQUAD evaluates language models on reading comprehension by providing text passage and related question. See $A.1 for hyperparameter settings.
Our main language modeling results are shown in Table 1. Since Mamba uses convolu- tions by default while GLA does not, we retrain the GLA with convolution, and also train DeltaNet without convolution. For the 1.3B setting we only train the DeltaNet with convolution due to limited compute resources. In general we find that DeltaNet outperforms the strong Mamba/GLA baselines in terms of both perplexity and downstream task performance. For recall-intensive tasks (i.e., SWDE, SQuAD, FDA), we find that under the same state size at the 340M scale, DeltaNet outperforms GLA, confirming the effectiveness of the delta rule. However, at the 1.3B scale, DeltaNet underperforms GLA due to its poorer state size scability (see $5.3), since state size plays an important role in recall- intensive tasks. Finally, confirm the benefits of hybrid architectures [21, 55]: both the sliding window and global attention hybrids work well, outperforming the strong Transformer++ basclines. We also scale DeltaNet to the 3B parameter scale trained with IT tokens using the same settings as Shen et al. [104]. The results are shown in Table 5, where 3B DeltaNet slightly underperforms a Transformer architecture trained with the same setting (PowerLLM-3B), but outperforms other RNN baselines in the 2B-3B range (though these are trained for a different number of tokens so are not exactly comparable).
Results.
Ablations. In Table 1 (bottom) we ablate the choice of feature map and normalization. We find that simply replacing the Ln-morm with the L2-norm greatly increases performance. For the feature map, we experiment with (ReLU, 1 + ELU, SiLU} and find that SiLU performs the best, consistent with prior work [88].
Training throughput. Figure compares the training throughputs of different 1.3B models in different training lengths and batch size settings. The training speed of DeltaNet is close to GLA
7

---

Model
Wiki.
LMB.
LMB.
PIQA
Hella.
Wino. acc
ARC-
Avg. acc_n
ARC-
SWDE
SQuAD
FDA
Stale
ppl
ppl 4
acc
acc
asc_nn
acc
acc f
acc
acc
exp.
340M params / 158 tokens
Transformer+
28.39
42,69
31.0
63.3
34.0
50.4
44.5
24.2
41.2
42.2
22.1
21,4
N/A
RetNet (wfo, com)
32.33
49.19
28.6
63.5
33.5
52.5
44.5
23.4
41.0
13.3
27.6
2.9
512x
comr)
39.66
65.0
GLA (wlo, conu)
28.65
43.35
30.3
64.8
34.5
51.4
45. 1
22.7
41.5
18.6
27.2
8.1
128x
comv)
29.47
45.53
31.3
65.1
33.8
51.6
44.4
24.6
41.8
24.0
24.7
7.3
128x
DeltaNet (wle, comv)
29.08
50.87
30.0
63.6
33.6
51.7
46.0
23.0
41.3
24.6
26.9
4.5
128A
DeliaNet (w. comv)
28.24
37.37
32.1
64.8
34.3
52.2
45.8
23.5
42.1
26.4
28.9
12,8
128A
Sliding
27.06
38.17
33.4
64.0
35.3
50.9
45.9
23.2
42.1
39.3
32.5
18.8
N/A
Global Attn (2 layers)
27.51
35.04
33,5
64.0
34.5
51.7
46.0
23.3
42.1
42.9
32.1
23.1
N/A
1.3R params/ / 100B tokous
Transformert+
16.85
13.44
48.9
70.8
49.6
53.6
56.0
26.5
50.9
66.6
31.5
27.4
N/A
RetNet (wle. comu)
18.64
17.27
43.3
70.0
47.3
52.5
54.8
25.6
48.9
42.8
34.7
14.3
512x
Mamia (w: comv)
17.06
13.89
46.2
72.2
40.
54.1
59.0
28.2
50.0
41.4
35.2
622
64X
GLA (wlo, comv
17.22
14.47
51.0
50.6
42.6
19.9
(w. conv)
17.25
14.92
46.2
70.6
49.9
53.0
55.3
27.0
50.4
52.4
37.4
22.3
256A
DeltaNee com)
12.21
57.2
49.5
37.4
17.2
Sliding Attn
16.56
11.74
49.2
71.8
51.1
52.8
58.9
28.8
52.1
53.3
43.3
22.3
N/A
Global Alnn layers)
16.55
12.40
48.8
70.8
50.7
54.2
58.4
28.1
51.8
71.0
43.0
29.8
N/A
DeliaNer Ablations (340M)
norm 1+ELU
55.96
26.3
23.9
To-nnrmm & L+ELU
28.03
37.62
32.2
65.7
34.7
51.8
45.4
22.5
42.1
23.8
28.6
13.1
128x
Ly-torm & ReLU
28.75
43.53
30.2
64.0
33.9
48.9
45.6
22.8
40.9
27.2
26.7
9.0
128x
Table 1:
Main language modeling results against Transformert+, RetNet [108], Mamba [31], and GLA [124].
All models are trained the same subset of the SlimPajama dataset with the Mistral tokenizer. The Trans- former++. RetNet, Mamba, GLA (w/o. conv) results are taking from Yang et al. [124]. For hybrid models, "Sliding Attn" interleaves a sliding window attention every other layer, and "Glohal Attn" uses full global at- tention on two layers. The 340M/1.3B models are trained for 15B/100B tokens respectively. All results are obtained through an-evaluatioon ioniioonnhaaa harness [27]. The last column denotes the expansion ratio of the recurrent state size relative to the product of the number of layers and model dimension (see Zhang al. [131. App. C)).
Tuadsow 00 Made
Model
ARC
HellaSwag OBQA PIQA
WinoGrande MMLE
Average
L.ama-3.2.38 (111)
PowerLM-3B [104]
60.5
14.6
43.6
79.9
70.0
45.0
62
59.8
RecurrenGeemmnaee (30)
57.0
71.1
42.0
78.2
67.6
31.8
57.9
RWKV 6-38 [79]
49.5
68.6
40.6
76.8
65.4
28.4
54.9
Mambu-2.78 [31]
Figure Zero-shot model performance for models. Llama-3.2-3B and PowerLM-3B are Transformer models, while the others are recurrent models. ARC results are averaged over normalized accuracy across ARC-Easy and ARC- Challenge.
selected benchmarks
Thikiing legghss
Figure 6: Training throughput of 1.3B models on a single H100.
and significantly faster than Mamba. All linear-time models outperform Transformers for longer- sequence training.
Discussion and Related Work
5.1 DeltaNet vs. State Space Models Linear RNNs
To discuss DeltaNet against existing linear RNNs (including state-space models) we first introduce a general class of associative RNNs with matrix-valued hidden states. Given matrix-valued hidden state S, [ and current input æ, E RR, these models have the following form:
S.11 M, + v,k
(recurrence)
S.qr.
(memory read-out)
where o an associative operator (e.g.. Hadamard product, matrix multiplication, etc.). The matrix M, and vectors vr, kt, qi (potentially non-linee non-linear) functions of the current input æ;.
As is the case in vector-valued linear RNNs [64, 106), the use an associative operator enables the use of parallel scan [13] to calculate .i,..., Sr. in O(log L) steps and ((L) work (ignoring the terms associated with the associative operation) if the inputs æ1,..., 2. are given (though see our discussion in footnote 1). Hence, long the associative operator is not too expensive, training

---

Model
Recurrence
Memory read-out
Linear Attention [48, 47]
S, = St-1 + veki
or S.q.
+ Kernel
Sr = S.11 + (ro(ks)
Sab(q:)
Normalization
S. S, 1 ),((kk) 2; 2 ((k))
S,6(9.) (6(q))(==2((0))
DeltaNet [101]
S. = S: 1(1 - -.ck + B.okk;
0: Siq.
Gated REA [81]
S. = 9tS: + (1 9.)ez 2: = 9. 1 (1 g:)
S Siq./a; q:)
S4 [32, 106]
S. S, 19 e
(A)( + B (v,1)
(S, @ C)1 d
ABC [82]
S/" S." 1 k.O., S = v
softmax (S g.)
DFW [63]
S. = St-1 2 (3:a)) m
0, S.q.
RetNet [108]
S. = 78,
n,k,
S,q.
Mamba [31]
S = S, 1 1 exp( (a.1 ex)( + (ai @ viji
Siq. @
GLA [124]
S. = S, 1 (1a;) + vikk = St (ai + viki
O. S:q.
RWKV-6 [79]
S = S, S ( + v.k,
(S. + (d @ 8z )4)4
HGRN-2 [92]
S, = S:-, Diag(a:) + v.(1 a.)
S.q.
mL.TTM [9]
S. = f.8 ienki: 2 = fiz--1 icke
S.q./ max (1, |z| 4:l
Mambu-2 [19]
S. = -.S.-1 + vckk
O. S:q.
GSA [131]
S S Diag(a.) + k.di S Sy Diag(as) + t
S softmax (s) q.)
Gated DeltaNet [125]
S. = S (a.(I - (. + B.okk!
O, S.q.
Table 2: Overview of recent linear recurrent models that have been proposed and applied autoregressive language modeling (ordered in rough chronological order). These works make use a matrix-valued hidden state S, E Rdx= (or two matrix-vale valued hidden states S ST e.g., [82, 131J) updated through an associative recurrence followed by an outer prier-produc--ee based addition. Here @ is the Hadamard product. Some models make of an additional linear RNN with hidden state vector 2t. which used to normalized the query vector qt- Variahles with the subscript t (e-g., Be; Ote fe: (t)) are (potentially non-linear) functions of the current input parameters (c.g., A, (d,7) are denoted without subscripts; these parameters are either learned or set to fixed values. Matrices are denoted with bold upper case letters, vectors with hold lower case, and scalars with italic letters. Many models make use of a kernel é (e.g., [101, 81J) but we subsume them into the key/value vectors reduce notational clutter.
ay. Non-time-varynee
can be efficient. However, parallel scan by itself is not sufficient for training language models at practical scale due to some associative operator's ' being too expensive. Recent models such as such as Mamba [31] and gated línear attention Transformers [108, 124, 92, 79, 9] thus make use of cheap element-wise recurrence updates, in particular the Hadamard product, i.e., = C. See Table 2 for how recent models can be cast into this form.
Standard matrix multiplications (i.e., S M, S M,) on the other hand can model richer interactions that go beyond elementwise recurrence. Without any structural assumptions on M, how- ever, these operations would take O(dn²) for each update (as opposed to (dd) for elementwise products), which would be prohibitively expensive. Hence, DeltaNet's use of M, = - k can be seen exploiting structured matrices to efficiently model interactions beyond clementwise re- currences. Our chunkwise algorithm could generalize to a broader class of matrices in the Diagonal- Plus-Low-Rank (DPLR) form M, = D a,,;, which has been explored in S4 [32], although their DPLR transition matrices are data data-independent. We adopt DeltaNet's 's parameteriizatin in this work (i.c.. D = I, a = Byks, b, = k2) as we are primarily interested in improving recall (through DeltaNet's key-value update rule) while maintaining parameter efficiency. We leave the exploration of more generalized parameterizations for future work.
5.2 Towards a Unifying Framework for Efficient Autoregressive Sequence Transformations
While the above class of models makes it possible to unify recent models, we do not claim that it is the "right" level at which view (autoregressive) sequence transformations of the form {x + {or) where o, cannot depend on any 2. if j > I. For example, this framing makes difficult to (neatly) capture other subquadratic models that have been shown to be effec- tive [126. 49, 98, 83]. An alternative unifying framework might be to view the above sequence transformations as a discretization of continuous state space model [32, 106, 31], or as matrix multiplication with masked structured matrix [76, 87, 46, 19]. What does seem important, however, that a framework should ideally expose efficient algorithms for training, and the algorithm should be cendware--fficiem which, in the case of modern GPUs, means that it should be rich in matrix multiplications. From this perspective, the state-space duality (SSD) framework recently proposed by Dao and Gu [19], which provides a connection between SSM-based sequence transformations and structured matrix multiplications with a semiseparable matrix, seems a promising candidate. However, this framework may not capture important class of models, e.g., models where the as-

---

sociative recurrence involves matrix multiplication with an unstructured matrix, or models that make use of more exotic associative operators (c.g., in Peng et al. [80].
Finally, we observe that there have been many recent linear-time models that have been proposed which purportedly match outperform classic transformers. As can be seen in Table 2, the "se- quence mixing" component of these works are closely related to one another. However, the way in which the token-mixing primitive is used to build up a transformer-like model varies widely. For ex- ample, while most recent works make use of depthwise-separable convolution layers (not shown in Table 2) [31, 9, 19, 15, 125], earlier works generally do not [48, 101, 81]. There are also differences in the parameterizations of the feedforward layers used for the "channel mixing" component. Such variations should be taken into account before declaring a particular model layer superior another.
5.3 Limitations and Future Work
Our work has several limitations. First, in terms of computation, although we propose new hardware-efficient algorithm, the training speed still lags behind that of GLA. This is due to the over- head caused by modeling state-to-state dependencies described above, which requires "marginal- izing" over the head dimension inside the kernel, similar to the case of softmax attention. However, for GLA since there are no intra-state dependencies (everything is elementwise), and thus it is easy to use tiling to support arbitrary size of head dimension, as implemented in Yang and Zhang [123]. This limitation would potentially limit DeltaNet's memory size, consequently lowering the recall- intensive task performance as we observed in $4.2. However, it may be feasible to adopt block di- agonal generalized Houscholder transition matrices with block sizes fitting GPU SRAM (e.g. 128) while maintaining a overall large head dimension (and thus a large recurrent state size).
We also found that the length generalization of DeltaNet Mamba to an extent) have been found to be able to extrapolate beyond the training length [124]. We speculate that this is because DeltaNet lacks explicit decay factors. This could be improved through incorporating a gating term in the recurrence, as demonstrated in a recent work by Yang et al. [125].
limited, while GLA and RetNet (and
6 Related Work
We briefly discuss related work here and give an extended discussion in Appendix C.
Linear transformers can be can provide perspectives on the limitations and improvements of linear attention transformers. For example, vanilla linear transformers use Hebbian-like update rule, which has been shown have limited memory capacity [68]. Later works in Hopfield networks use higher-order polynomials [22] and exponential kernels [94, 50] enhance the memory capacity, which is also related to linear attention with polynomial kernels [45, 6, 1j. On the other hand, the delta rule has been shown to have better memory capacity [28, 85, 56, 101]. In this sense, given the fixed size recurrent state, using the delta rule is able to achieve a better frontier of the recall-memory tradeoff curve [6], and has recently been applied to enhance real-world retrieval tasks [74, 96). Moreover, it outperforms the additive rule used in vanilla linear transformers across multiple domains [101, 38, 40, 36, 42].
a type of iterated Hopfield networks [72], and this connection
Despite these advantages, Irie al. [42] revealed theoretical limitations of the delta update rule in terms of expressiveness. Recurrent enhancements of DeltaNet, such as Recurrent DeltaNet [37] and the Modern Referneelllll Weight Matrix [41], and the mesa-layer [117] were proposed and found superior. However, these models extend beyond linear RNNs and cannot be parallelized across sequence length. This suggests fundamental trade-off between parallelism and expressiveness [70]. How to further enhance DeltaNet without sacrificing parallelism remains an open question, and the hybrid cross-chunk nonlinear and intra-chunk linear strategy used in TTT 10] might provide a suitable middle ground. Finally, we remark that delta rule is closely related to meta or online learning via gradient descent [73, 39], which has been revisited in recent works like Longhorn [58] and TTT [110]. Recently, Titans [10] improves TTT by introducing a momentum and weight decay term.
7 Conclusion
We describe an algorithm that parallelizes DeltaNet training across the sequence length dimension, achieving significant speed-ups against existing implementations on modern hardware. This makes possible to scale up DeltaNet to moderate-scale language modeling settings, where we find that performs well compared to recent linear-recurrent baselines.
10

---

Acknowledgements
This study was supported by funds from MIT-IBM Watson AT Lab and the MIT-Google Program for Computing Innovation. We are grateful Mayank Mishra for assistance with training and evaluating the 3B models, to Kazuki Irie for valuable feedback on the draft, and to Simran Arora, Liliang Ren and Eric Alcaide for their insightful discussions. We also thank Michael Poli and Armin Thomas for sharing the raw results from the MAD benchmark experiment, as well as to Fan Zhou for identifying eπror in Eq. 10.
References
[1] Y. Aksenov, N. Balagansky, S. M. L. C. Vaina, Shaposhnikov, A. Gorbatovski, and D. Gavrilov. Linear Transformers with Learnable Kernel Functions are Better In-Context Models, 2024. URL hitts:///axivy orggabsgaa..o.oooooeeo.oooeeeo /2t.///22.... 10644.
[2] E. Akyürek, B. Wang, Y. Kim, and J. Andreas. In-context language learning: Arhitectures and algorithms. ArXiv preprint, abs/2401.11973,, 2024. URL https://arxis org/ abs/ 2401 12973.
[3] Ali, Zimerman, and L. Wolf. The hidden attention of mamba models, 2024.
[4] S. Arora, S. Eyuboglu, A. Timalsina. I. Johnson, M. Poli, J. Zou, A. Rudra, and C. Ré. Zoology: Measuring and improving recall in efficient language models. ArXiv preprint. abs/2312.04922 2023. URL https://arxiv org/ 2212 04927..
[5] S. Arora, Yang. S. Eyuboglu, Narayan, A. Hojel, Trummer, and C. Ré. Language Mod- els Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes, 2023. URL htttpp:://axxv.orrr org/ 2b3/ 09433.
[6] S. Arora, S. Eyuboglu, M. Zhang, A. Timalsina, S. Alberti, D. Zinsley, J. Zou, A. Rudra, and C. Ré. Simple linear attention language models balance the thcoll-throughput tradeoff. ArXiv preprint, 0os/2402.18668 2024. URL https: /ttttp:://axiiii /tt///axxvv/v//////////e////aav/ wwwpiii.orr///bae 2b42 2
[7] S. Arora, A. Timalsina, A. Singhal, Spector, S. Eyuboglu, X. Zhao, A. Rao, A. Rudra, and
C. Ré. Just read twice: closing the recall gap for recurrent language models, 2024. URL https://arxiv. org/ aa/aab//24400 05483.
[8] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In Bengio and LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: /tt///axivv/////////////////aava www.ii.og/abbil...aoa wwg..bs.////ooooooooto..oo....o/ 1409 0473.
[9] M. Beck, K. Pöppel. M. Spanring. A. Auer, O. Prudnikova, M. Kopp. G. Klambauer, J. Brand- stetter, and S. Hochreiter. xlstm: Extended long short-term memory. ArXiv preprint, 044524 2024. hww.h:tt////rrre www.w.r......... 240aaa//...... 04517.
[10] A. Behrouz, P. Zhong, and V. Mirrokni. Titans: Learning memorize at test time, 2024. URL https://arxiss org/abs 2ot.//0000.........
[11] C. H. Bischof and C. V. Loan. The WY representation for products householder matrices. In SIAM Conference on Parallel Processing for Scientific Computing, 1985. URL https: // api. sew.nticschhol.rr..........c org/ ID:
[12] Y. Bisk, R. Zellers, R. LeBras, J. Gao, and Y. Choi. PIQA: reasoning about physical common- sense in natural language. In The Thirty-Fourdey AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances Artificial Intelligence, EAAI 2020, New York, NY. USA, February 7-12, 2020, pages 7432-7439, AAAI Press, 2020. URL https:://aatt /aa/i./aaa.../// org/ j index.dxxxii pttp/AAttaarriclle/reee///33
[13] G. E. Blelloch. Prefix sums and their applications. 1990.

---

[14] W. Brandon, A. Nrusimha, K. Qian, Z. Ankner, T. Jin, Z. Song, and J. Ragan-Kelley. Striped Attention: Faster Ring Attention for Causal Transformers. ArXiv preprint, abs/2311.09431 2023. URL. htw.::ttp:/// //triv////xiveee//a / 2211 094431.
[15] Y. Chou, M. Yao, K. Wang. Y. Pan, R.-J Zhu, J. Wu, Y. Zhong, Y. Qiao, B. XU, and G. Li. MetaLA: Unified optimal linear approximation to softmax attention map. In The Thirry- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://
[16] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, Schoenick, and O. Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. ArXiv preprint, abs/1803.0544. 2018. URL https://axxii.... www.riiii......oe.ooooooo aas//bs////// 1803. .0557..
[17] D. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by ex- ponential linear units (clus). In Y. Bengio and Y. LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http: / /arxivaivvv////////////a///aaaa/ org/ abs/ 1t////111111111991 07289.
[18] T. Dao. FlashAttention--2: Faster Attention with Better Parallelism and Work Partitioning. ArXiv preprint, abs/2307 2023. URL httpt://aaxiiiii org. ab3/ 08691.
[19] T. Dao and A. Gu. Transformers ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv: 2405.2 2024.
[20] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Infor- mation Processing Systems 35: Annual Conference on Neural Information Process- ing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 Decem- ber 9, 2022, 2022. URL http: /tapp:/ wwapperss.iip..o http:cccpppppeipeeeeeeee fileffilaapppeeee www.e//paper////e/eeee ppppp//022//////////2p ha/h//hhhh///// 67457 77765773ee0000a0772220211116666
Abstract- Conference. html.
[21] S. De, S. L. Smith, A. Fernando, A. Botev, G. Cristian- Muraru, A. Gu, R. Haroun, L. Berrada, Y. Chen, S. Srinivasan, G. Desjardins, A. Doucet, D. Budden, Y. W. Teh, R. Pascanu, N. De Freitas, and C. Gulcehre. Griffin: Mixing Gated Linear Recurrences with Local Atten- tion for Efficient Language Models, 2024. URL httph:tt//// ptttpp:///rrivv .rgg httgg/bb/224000 19427.
[22] M. Demircigil, Heusel, M. Löwe, S. Upgang, and F. Vermet. On model of associative memory with huge storage capacity. ArXiv preprint, abs/1702.01929, 2017. URL https:// arxiv. wwwg/ba////.....oooogo wtabs//00...m 01929.
[23] A. T. Dominguez and F. Q. Orti. Fast blocking of householder reflectors graphics pro- cessors. 2018 26th Euromicro International Conference on Parallel, Distributed and Network- based Processing (PDP), pages 385-393, 2018. URL https: //api. semanticscholat htt//oopusID:446eee 46960439.
[24] S. Elfwing, E. Uchibe, and K. Doya. Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning, 2017. URL https://axxiv org/ abs/ 0///1702.0111 03118.
[25] M. Fathi, J. Pilault, P.-.. Bacon, C. Pal, O. Firat, and R. Goroshin. Block-state transformer. ArXiv preprint, abs/2306.09539, 2023. URL https: :/axxivv//////i////aaeaaa org/ httpa/ab//2300005/5/
[26] D. Y. Fu, T. Dao, K. K. Saab, A. W. Thomas, A. Rudra, and C. Ré. Hungry hungry hippos: To- wards language modeling with state space models. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL /ttp:r/////ppeeveeeee wtt//eet/ffftett pptppdddcodyyoo////..
[27] L. Gao, J. Tow, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, Hsu, K. McDonell, N. Muennighoff, J. Phang, L. Reynolds. E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou. A framework for few-shot language model evaluation, 2021.
[28] E. Gardner. The space of interactions in neural network models. Journal of Physics A, 21: 257-270, 1988.
12

---

[29] A. Graves, G. Wayne, and I. Danihelka. Neural Turing Machines, 2014. URL http:// arxiv.org/ www./b/g///oooo...ot..oo.../..e. 1410.11155000 5401. arXiv:1410.5401 [es].
[30] R. Griffin and G. Teams. Recurrentgemma: Moving past transformers for efficient open language models. ArXiv preprint, abs/2404.07839, 2024. URL https://arxiv. org/abg/ 2404. 07839.
[31] A. Gu and T. Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. In Proceedings of COLM, 2023.
[32] A. Gu, K. Goel, and C. Ré. Efficiently modeling long sequences with structured state spaces. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https: https: /ttp://///peeeeeeeeeeeeee wew.errrviieeeeetorunn net/forum? id=uYLFozIv1AC.
[33] K. Helffiih, D. Willmott, and Q. Ye. Orthogonal recurrent neural networks with scaled cayley transform. In J. G. Dy and A. Krause, editors, Proceedings of the 35th International Confer- ence on Machine Learning. ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10- 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1974-1983. PMLR, 2018. URL http: http://pooceennnneee mlr, www..peess/veee prres//800hellrichhh httph/eeffrrhaaa.... html....
[34] W. Hua, Z. Dai. H. Liu, and Q. V. Le. Transformer quality in linear time. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and Sabato, editors, International Conference Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Muchine Learning Research, pages 9099-9117. PMLR, 2022. URL https://procest
mlr, press///////eeee cesss//vv2/// 62/6/222aaaa.....//a html.
[35] F. Huang, K. Lui, Y. Cai, Z. Qin, Y. Fang, G. Tian, and G. Li. Encoding recurrence into transformers. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URI.httpss//openeeee id
[36] K. Irie and J. Schmidhuber. Images weight matrices: Sequential image generation through synaptic learning rules. In The Eleventh International Conference on Learning Representa- tions, ICLR 2023, Kigali. Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https:// openreview net/ pet/ppff/t http:/dfddddd Pid-daddPPUUU..
[37] K. Irie, I. Schlag, R. Csordás, and J. Schmidhuber. Going beyond linear transformers with recurrent fast weight programmers. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 14, 202 virtual, pages 7703-7717, 2021. URL https: htt //poceedd/s//sosee/g//// iewriidnnnnnnnunneeeeeeeee..u wwwrips.cc/pape..... cc/paper/ 2021
Abstract html.
[38] K. Irie, I. Schlag, R. Csordás, and J. Schmidhuber. Going beyond linear transformers with recurrent fast weight programmers. In M. Ranzato, A. Beygelzimer, N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14 1414 2021, virtual, pages 7703-7717, 2021. URL httpt:::pp/pocceeedineee neurips cc/paper/ 1o91/hash/3f9e331 0tt.5/66777710100000000604666660 wtttr///ttttaaatsrrtttt.cc Abstract. html.
[39] K. Irie, R. Csordás, and J. Schmidhuber. The dual form of neural networks revisited: Con- necting test time predictions to training patterns via spotlights of attention. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference Machine Learning. ICML 2022, 17-23 July 2022. Baltimore, Maryland, USA. volume 162 of Proceedings of Machine Learning Research, pages 9639-9659. PMLR, 2022. URL attps://pat /ppooe///poceeinggs mlr. trress/v1e2/rricc
html.
[40] Irie, F. Faccio, and J. Schmidhuber. Neural differential equations for learning to program neural nets through continuous learning rules. In S. Koyejo, S. Mo- hamed, Agarwal, Belgrave, K. Cho, and Oh, editors, Advances in Neural In- formation Processing Systems 35: Annual Conference on Neural Information Process-
13

---

ing Systems 2022, NeurIPS 2022, New Orleans, LA USA, November 28 Decem- httpa//appppee///a/a///// html.
ber 9. 2022, 2022. URL http:: cwtp://rriippcccc/e.eeeeooooette wtwp:cccppppppeeeee f
[41] K. Irie, I. Schlag, R. Csordás, and J. Schmidhuber. A modern self-referential weight matrix that learns to modify itself. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Re- search, pages 9660 96700-9677..00 PMLR, 2022. URL htttpp://proceedingss mlr. h.r..pesss/v12// irie22b. html.
[42] K. Irie, R. Csordás, and J. Schmidhuber. Practical computational power of linear transform- ers and their recurrent and self-referential extensions. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9455-9465, Singapore, 2023. Association for Computational Linguis- main.588. URL https: /acas///oo/lattooogo.yo org/2023.
tics. doi: 10. emnlp-man maammmin... 588.
[43] I. Jing, C. Gulcehre, J. Peurifoy, Y. Shen, M. Tegmark, M. Soljacic, and Y. Bengio. Gated Orthogonal Recurrent Units: On Learning to Forget. Neural Computation, 31(4):765-783,
2019. ISSN 0899-7667, 1530-888X. doi: (1l6/////o_a_011111 mit. htab://ncoo/rticclle//eeooee
URL https://directt:
783//484...
[44] T. Joffrain. T. M. Low, E. S. Quintana- R. A. van de Geijn, and F. G. V. Zee. Accumu- lating householder transformations, revisited. ACM Trans. Math. Softw., 32:169-179, 2006.
URL. wtttts:///apppt httpapppsssmmaiccccooolooooooool http://aaccc.cooooooooroopopppao org/CorpusID::5725 15723317.
[45] P. Kacham, V. Mirrokni, and P. Zhong. Polysketchformer: Fast transformers via sketches for polynomial kernels. ArXiv preprint, abs/2310.01655, 2023. URL https://aaxxi org/ abs/2310 01655.
[46] Kang, Tran, and H. Sterck. Fast multipole attention: divide-and-conquee attention mechanism for long sequences. ArXiv preprint, abs/2310. 11960, 2023. URL. https:// arxiv.or arggabss/3100 11960.
[47] J. Kasai, H. Peng, Y. Zhang, D. Yogatama, G. Ilharco, N. Pappas, Y. Mao, W. Chen, and N. A. Smith. Finetuning pretrained transformers into RNNs. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, editors, Proceedings the 2021 Conference
Empirical Methods in Natu- ral Language Processing, pages 10630-10643, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10. 00.188633/v1/022 12..6653/112222..mmm-mmaiii 880.. URI. /Rl.:https://allattooogggg/ aclaaccoaaaatooooooooooooooooala wttph//2.ooo.og2111...... commp-mmiin mm000.0................
org/2021.
[48] A. Katharopoulos, A. Vyas, N. Pappas, and Fleuret. Transformers are mns: Fast autore- gressive transformers with linear attention. In Proceedings of the 37th International Con- ference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5156-5165. PMLR, 2020. URL http:// / proceedings.o mlr. wttessss/111//ttaaoooolllsseee htmml.
[49] N. Kitaev, L. Kaiser, and A. Levskaya. Reformer: The efficient transformer. In 8th In- ternational Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://oppereevew net//forum id id=rkgNKHHtvB.
[50] D. Krotov and J. Hopfield. Large associative memory problem in neurobiology and machine learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:// net// frrmmiddx4y_100xx hX.
[51] T. Lei. When attention meets fast recurrence: Training language models with reduced com- pute. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Prucessing, pages 7633-7648, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguis- ties. doi: 10. 18653/v1/2021 1et.222l.mmmppmmiiie 602.. URL https: //ae::///ccc htt/a/aaaaatoooyyy.ooo org//021. emnlp-man mappmaanncom m02n.602.
14

---

[52] T. Lei, R. Tian, J. Bastings, and A. P. Parikh. Simple recurrence improves masked lan- guage models. ArXiv preprint, 0bs/2205.11588, 2022. URL https://arxiv. org/abs/ 2205. 11588.
[53] P. S. H. Lewis. E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, S. Riedel, and D. Kiela. Retrieval-augmenter genera- tion for inowledge-intensiin can, and H. Lin, editors, Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020. URL
NLP tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M. Bal-
nerssnnurpppp.. wwtrrisscccppppeeee wwtpppppe////////////// Aww.ract.btttt...bo.ommoooo html.
[54] B. Liao, X. Wang, L. Zhu, Q. Zhang, and C. Huang. Vig: Linear- complexty visual sequence learning with gated linear attention. ArXiv, abs/2405. 18425, 2024. URL https /api. semanticscholar
wng/CorpusID 270068351.
[55] O. Lieber, B. Lenz, H. Bata, G. Cohen, J. Osin, I. Dalmedigos, E. Safahi, S. Mcirom, Y. Be- linkov, Shalev-Shwartz, al. Jamba: A hybrid transformer-mamaa language model. ArXiv preprint, abs/2403.19887, 2024. https:// /ttttps://ariiii orgg/vvorr//iie abs/2403. 19887.7777......
[56] K. C. Lingashetty. Delta learning rule for the active sites model. arXiv preprint
wrXiv:100.00417, 2010. 2010.
[57] L. D. Lingle. Transformer-vq: Linear-time transformers via vector quantization. ArXiv preprint, 10222339.133344 2023. URL https: //wrx//xvvv/////////////////aav/ org/ abs aa/aabba/2200/ hbbb/2099.65444 16354.
[58] B. Liu, R. Wang, L. Wu, Y. Feng, Stone, and Q. Liu. Longhorn: State space models are amortized online learners. ArXiv preprint, hos/2407.142001 2024. URL fttps:///arxit org/ abbb/247. 14207.
[59] H. Liu, M. Zaharia, and P. Abbeel. Ring Attention with Blockwise Transformers for Near- Infinite Context. ArXiv preprint, abs/2310.01889 2023. URL https://arxiv. org/ abs/ 2310. 01889.
[60] C. Lockard, P. Shiralkar, and X. L. Dong- OpenCeres: When open information extraction meets the semi-structured web. J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3047- 3056, Minncapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10. 18653/v1/N19- 1309. URL https: // acl anthoolog org/N19-- 1309.
[61] X. Ma. C. Zhou, X. Kong, J. He, L. Gui, G. Neubig, J. May, and L. Zettlemoyer. Mega: Mov- ing average equipped gated attention. The Eleventh International Conference on Learning Representattons, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL bttps://openreeee htt.r//.......opp..ppooaoilll.g. QNLeiqgE1.
net met/pdffid=qllee..
[62] Ma, Yang, W. Xiong, B. Chen, L. Yu, H. Zhang, May, L. Zettlemoyer, O. Levy, and C. Zhou. Megalodon: Efficient IIm pretraining and inference with unlimited context length. ArXiv preprint, abs/2404.08801, 2024. URL https: //axxivv//xxxv.../// www.aaawwivrr.aaaaaaaaaaae 18810
[63] Mao. Fine-tuning pre-trained transformers into decaying fast weights. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Meth- ods Natural Language Processing, pages 10236-10242. Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics. doi: 10. 18653//1/002.emmm/// 6977 URL https::t////...t. http//aaaaannttoggy..o org/2022.
[64] E. Martin and C. Cundy. Parallelizing linear recurrent neural nets over sequence length. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL attps://openrevier net/ aorumidd-yUUWwwll..
15

---

[65] E. Martin and C. Cundy, Parallelizing linear recurrent neural nets over sequence length. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openreviee wtt.foeuntddddlllyyeee... Hrrmmid-
[66] A. Mathiasen, F. Hvilshoj, Jørgensen, A. Nasery, and D. Mottin. What if neural networks had svds? ArXiv, 0os/2009.13977, 2020. URL httpp::/aap...emmatiiccchh www.steeaasts..eeeeee cscholaar org/
CorpusID: 221995651.
[67] A. Mathiasen, F. Hvilshoj, J. R. Jørgensen, A. Nasery, and D. Mottin. Faster orthogonal parameterization with householder matrices. In ICML, Workshop Proceedings, 2020.
[68] J. McEliece, E. C. Posner, E. R. Rodemich, and S. Venkatesh. The capacity of the hopfield associative memory. IEEE Trans. Inf. 33:461,33:611 33:4611482, 1987.
[69] J. Mercat, I. Vasiljevic, S. Keh, K. Arora, A. Dave, A. Gaidon, and T. Kollar. Linearizing large language models. ArXiv preprint, abs/2405.0664d 2024. URL https://arxiv org/ abs/2405 0tt//2005.06640 06640.
[70] W. Merrill, J. Petry, and A. Sabharwal. The Illusion of State in State-Space Models, 2024.
URL https:/ arxiv. org/absa h////bbs/440 /404. 08819.
[71] Z. Mhammedi, A. D. Hellicar, A. Rahman, and J. Bailey. Efficient orthogonal parametrisation recurrent neural networks using houscholder reflections. In D. Precup and Y. W. Teh, editors, Proceedings the 34th International Conference
Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 20/7, volume 70 of Proceedings of Machine Learning Research, pages 2401-2409. PMLR, 2017. URL http: httpp://proceedingss mlr. www.mml....rs///e press/v70/ mhammedi 17a. html.
[72] Millidge. Linear Attention as Iterated Hopfield Networks. URL http::/wtw//t http://wwwwbbbeeeeeeeeee beren. io/ 2024 03 -0- Attention- Iterated -tt----reete-----iee------------
[73] T. Munkhdalai, A. Sordoni, T. Wang, and A. Trischler. Metalearned neural memory. In H. M. Wallach, Larochelle, A. Beygelzimer, F. d' d'Alcéé-Buc, B. Fox, and Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural In- formation Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13310-13321, 2019. URL https:: http:::::////oeeeeeeseeee .ew.iidinnnnnnnr............. wwwrriis.ccpppppeee www.aaarrtttttttttttt..... html.
2019/hash/ httph/hahhhhhhh /ttp///2222222222772222222222222
[74] T. Munkhdalai, M. Faruqui, and S. Gopal. Leave no context behind: Efficient infinite context transformers with infini-attention. ArXiv preprint, abs/2404.07143, 2024. URL https://
arxiv. www.oo////baaee aa////bb//////// 0714.0714..
[75] Y. Nahshan, J. Kampeas, and E. Haleva. Linear Log-Normal Attention with Unbiased Con- centration, 2023. URL https: http///xxxvvvo/// wrw.
13541.
[76] T. M. Nguyen, V. Suliafu, S. J. Osher, L. Chen, and B. Wang. Fmmformer: Ef- ficient and flexible transformer via decomposed near-field and far-field attention. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Ad- vances in Neural Information Processing Systems 34: Annual Conference
Neural In- formation Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 29449-29463. 2021. URL https: https::/ptoceeeeeeee :/tp://eecceeeeeesee/eeee neurips.ced cwurrpss.cc/pppr/22222 paapr//22t/haaal/e f621585df 011556622445551177773777779----3 Abstsstateeebbt--tttattt....c.. html.
[77] D. Paperno, G. Kruszewski, A. Lazaridou,
Q. Pham, R. Bernardi, S. Pezzelle, M. Ba- roni, G. Boleda, and R. Fernández. The LAMBADA dataset: Word prediction requiring K. Erk and N. A. Smith, editors, Proceedings of the 54th An-
broad discourse context. nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1525-1534 1534, Berlin, Germany, 2016. Association for Computational Linguistics. doi: 10. 10.18653/v1/P1C 1444 URL
[78] J. Park, J. Park, Z. Xiong, N. Lee, J. Cho, S. Oymak, K. Lee, and D. Papailiopoulos. Can mamba learn how to learn? a comparative study on in-context learning tasks. ArXiv preprint, Abs/2402.04424,, 2024. https: : /ttp///xvvv////////i//e/////aav/ wttps///.rrabb////////./ 04248.
16

---

[79] B. Peng, D. Goldstein, Q. Anthony, A. Albalak, E. Alcaide, S. Biderman, E. Cheah, X. Du, Т. Ferdinan, H. Hou. Kazienko, K. K. GV, J. Kocoñ, B. Koptyra, S. Krishna, R. McClel- land Jr. N. Muennighoff, F. Oheid, A. Saito, G. Song, H. Tu, S. Wožniak, R. Zhang, B. Zhao, Q. Zhao, P. Zhou, Zhu, and R.-J. Zhu. Fagle and Finch: RWKV with Matrix- Valued States and Dynamic Recurrence, 2024. URL https: /wwxiv.vvvvv////i///e/////.aa/aa wwrxiv.org//ooee. 2404. .58992.
[80] H. Peng, R. Schwartz, S. Thomson, and N. A. Smith. Rational recurrences. In E. Riloff, Chiang, Hockenmaier, and Tsujii, editors, Proceedings of the 2018 Conference on Em- pirical Methods in Natural anguage Processing, pages 2003-1214, 14, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.1 11885v11/118-1111 URL https:// aclanthology org/D18-113
[81] H. Peng, N. Pappas, D. Yogatama, R. Schwartz, N. A. Smith, and L. Kong. Random feature attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL. https:// /eetteeerreeeeeeeeeee forum?id=QtTTTTaVFBB.
[82] H. Peng. J. Kasai, N. Pappas, D. Yogatama, Z. Wu, L. Kong, R. Schwartz, and N. A. Smith. ABC: Attention with bounded-memory control. In S. Muresan, P. Nakov, and A. Villavicen- cio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7469-7483, Dublin, Ireland, 2022. Associa- tion for Computational Linguistics. doi: 10. 12.16666////2222cccclooo
URL https://
aclanthology orgg/rg//// 2022. acl-long. 515.
[83] M. Poli, S. Massaroli, E. Nguyen, D. Y. Fu, T. Dao, S. Baccus, Y. Bengio, S. Ermon, and Ré. Hyena hierarchy: Towards larger convolutional language models. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Confer- ence on Machine Learning, ICML 2023. 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 28043-28078, PMLR, 2023, URL https: attps://proceeding.
mlr. wwwss/vv20 20002//ooi23a..... html...m........
[84] M. Poli, A. W. Thomas, E. Nguyen, P. Ponnusamy, B. Deiseroth, Kersting, Suzuki, B. Hie, S. Ermon, C. Ré. C. Zhang, and S. Massaroli. Mechanistic Design and Scaling of Hybrid Architectures, 2024. URL https: http://wwwwww...aaaaaa///ass
17844.
[85] D. Prados and S. Kak. Neural network capacity using delta rule. Electronics Letters, 3(25): 197-199, 1989.
[86] Z. Qin, X. Han, W. Sun, D. Li, L. Kong, N. Barnes, and Y. Zhong. The devil in linear transformer. In Y. Goldberg, Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 7025-7041, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics. doi: 10. URL https::/ haapps////laaahhllgr/o/o org/ 2022/ emnlp-main.
473.
[87] Z. Qin, X. Han, W. Sun, He, D. Li, D. Li, Dai, L. Kong, and Zhong. Toeplitz neu- ral network for sequence modeling. In The Eleventh International Conference on Learning Representationss ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL attps://openrevice
net/pdf pett/dffid=IIaee Iid=IImmsmsxxxuu
[88] Z. Qin, D. Li, W. Sun, W. Sun, X. Shen, X. Han, Y. Wei, B. L.,, F. Yuan, X. Luo, et al. Scaling transnormer to 175 billion parameters. ArXiv preprint, abs/2307. 2023, URL attxs://arxiv. org/ abs/ /2307 14995.
arg/aba///
[89] Z. Qin, W. Sun, K. Lu, H. Deng, D. Li, X. Han, Y. Dai, L. Kong, and Y. Zhong. Linearized Relative Positional Encoding, 2023. https:/:::/ //arxi/v orgg htt///bs/2300 22002/277770.00.
[90] Z. Qin, S. Yang, and Y. Zhong. Hierarchically gated recurrent neural network for sequence modeling. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, edi- tors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http::/ppper iwtpapers..ipps... wwcpppapr_fiileeeee www.flllappppeee aates//ppprr/2222 naper/2023/hash/ 6921o33-89977ecc899944eeeaeddd http://ddssssseeeeeeee-teeeeeeee Abstract- CeretrtCoofefenneeeem htmml. html.
17

---

[91] Z. Qin, W. Sun. D. Li, X. Shen, W. Sun, and Y. Zhong. Lightning attention-2: A free lunch for handling unlimited sequence lengths in large language models, 2024.
[92] Z. Qin, S. Yang. W. Sun, X. Shen, D. Li. W. Sun. and Zhong. HGRN2: Gated Lin- ear RNNS with State Expansion. 2024. URL https: //api. semanti cannaltiscchlarrr org/ CorpusID: 269043328.
[93] P. Rajpurkar, R. Jia, and P. Liang. Know what you don't know: Unanswerable questions for SQuAD. In I. Gurevych and Y. Miyao, editors, Proceedings of the 56th Anmual Meeting of the Association for Computational Linguisties (Volume 2: Short Papers), pages 784-789, Melbourne, Australia, 2018. Association for Computational Linguistics. doi: 10.18653/v1/ 2124. URL https: http://aaaanthoooggoooo org/P18-222 2124.
[94] H. Ramsauer, B. Schäfl, J. Lehner, Seidl, M. Widrich, L. Gruber, M. Holzleitner, Adler, D. P. Kreil, M. K. Kopp, G. Klambauer. J. Brandstetter, and S. Hochreiter, Hopfield networks is all you need. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL (ot.orummmnd--1112IIIII
[95] L. Ren, Y. Liu, Y. Lu, Shen, C. Liang, and W. Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling. ArXiv preprint, abs/2406.07522, 2024. URL https: petps://arxtte //tp//aaxvvv//////////ee//aaaav/ wwwpxii..rg//bbe.. wtgp/bs////.......////// 3ttp//abb/2000.052222 07522.
[96] I. Rodkin, Y. Kuratov, A. Bulatov, and M. Burtsev. Associative recurrent memory transformer. ArXiv preprint, abs/2407.06 0044444 2024. URL https:::/taai...oooto
04841.
[97] S. Rogers. Adaptive filter theory. Control Engineering Practice, 4:1629-1630, 1996. URL www.ntcascooaaar.... www.aaatttccccooo....oooo/orrppe orrg/CorpusDD:10552355 httpps///pp...mmatttccchooli..
[98] A. Roy, M. Saffar, A. Vaswani, and D. Grangier. Efficient content- based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53- 68, 2021. doi: 10.1162/tacl 00353. URL https / http//alaaan/ /acp//aaaantooyyyy.ooo org/2021 arr/2021.tacl-11 4.
[99] K. Sakaguchi, R. L. Bras, C. Bhhaavvatula,, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8732-8740. AAAI Press, 2020. URL h /aaaaaaaaaa/////a///aaa//aaaaa.a org/ wetg/oj//indde/ php/ AAAI/AAAAA/aa// htw.a//arriiceeeie///SSS
[100] Y. Schiff, C.-H. Kao, A. Gokaslan, T. Dao, A. Gu, and V. Kuleshov. Caduceus: Bi- directional equivariant long-range dna sequence modeling. ArXiv, abs/2403.03234, 2024. URL https://agy wag...ssaatisscooll..a... wwggCorpgssliI 268253280.
[101] I. Schlag, K. Irie, and J. Schmidhuber. Linear transformers secretly fast weight program- mers. In M. Mcila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. (CML2021) 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 9355 PMLR, 2021. URL http: h http:://proceeedngg. mlr. wtt.press/v39///hhiie hwtps//cchlllllee html.
[102] J. Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recur- rent networks. Neural Computation, 4(1):131-139) 1992.
[103] Shazeer. Glu variants improve transformer. ArXiv preprint, abs220000.000 2020. URL 05202.
https://arxiv.on
[104] Y. Shen, M. Stallone, M. Mishra, G. Zhang. S. Tan, A. Prasad, A. M. Soria, D. D. Cox, and Panda. Power scheduler: batch size and token number agnostic learning rate scheduler. ArXiv preprint, abs/2408. 13359, 2024. URL https : /ttp/axxvvv../ org/ A.339990..13359 13359.
18

---

[105] J. T. H. Smith, A. Warrington, and S. W. Linderman. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representa- lions, ICI. 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023, URL https:// openreview.. net/ Pdd.Addddd-A--AHAAAAA-
[106] J. T. H. Smith, A. Warrington, and S. W. Linderman. Simplified state space layers for sequence modeling. In The Eleventh International Conference Learning Representa- tions, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023, URL https:// openreview. net sett/pfffiddAiiBBaa..
[107] W. Sun, Z. Qin, D. Li, X. Shen, Y. Qiao, and Y. Zhong. Linear attention sequence parallelism. ArXiv preprint, abs/2404.02882 2024. URL https::///// /ttp///axxvvvv//a .rgg/ abs/ 22802424442......
[108] Y. Sun, L. Dong, S. Huang. S. Ma, Y. Xia, J. Xue, J. Wang, and F. Wei. Retentive network:
A successor to transformer for large language models. ArXiv preprint, abs/2307.08621, 2023. URL attps://Ba arxiv. org/ abs/ 2307. 08621.
[109] Y. Sun, L. Dong, Y. Zhu, S. Huang, W. Wang, S. Ma, Q. Zhang, J. Wang, and F. Wei. You only cache once: Decoder-decoder architectures for language models. ArXiv preprint, abs/2405.0522 055454. 2024. URL atttps://arxits wrwgv.rgg/bbbe wtgg///////0// 05254.
[110] Y. Sun, X. Li, K. Dalal, J. Xu, A. Vikram, G. Zhang, Y. Dubois, X. Chen, X. Wang, O. Koyejo, T. Hashimoto, and C. Guestrin. Learning to (learn at test time): Rnns with expressive hid- den states. ArXiv preprint, abs/2407 0os/2407.00460, 2024. URL https:/ /arxiv www.ii.rg//..oeeooooooooee abs 2207. 04620.
[111] L. Team. The llama 3 herd of models. ArXiv preprint, abs/2407.21783, 2024. URL https:// arxiv. arxiv.org/a wwwoorgabss.. abaaabbs/200 21783.
[112] P. Tillet, Kung, and D. D. Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPIAN International Work- shop on Machine Learning and Programming Languages, MAPI, PLDI 2019, pages 10-19. ACM, 2019, doi:
[113] J. M. Tomezak and M. Welling. Improving Variational Flow, 2016. URL https: //tp///ax/vvvveee/////eeeeeaaa// orooooog/aa..oooooooooooooo. boggabbs/6111. 1191/16611996......
using Houscholder
[114] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, al. Llama: Open and efficient foundation language models. ArXiv preprint, abs/2302. 13971, 2023. URL htttts://aaaeett /at/ivxxvvvv//.i org/ abs/ abs/2302. 13971.
/
[115] R. van den Berg, L. Hasenclever, J. M. Tomezak, and M. Welling. Sylvester normalizing flows for variational inference. In A. Globerson and R. Silva, editors, Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, Cal- ifornia, USA, August 6-10, 2018, pages 393-402, AUAI Press, 2018 URL http://auai. 2tgg/aaao0//prrreeeingeg pepocceedngg/pppppeeee (hapers/555 /166 .df.
[116] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Guyon, U. von Luxburg. S. Ben- gio, H. M. Wallach, R. Fergus. S. V. N. Vishwanathan, and R. Gameet, editors, Ad- vances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages
2017. URL https: //ttoeee//proceddnggg www.nnnuun... cttpscer/pppp/211/77/ Awwtaa-bbbttttttteee.aaaae htmll. html.
[117] J. von Oswald, Niklasson, M. Schlegel, S. Kobayashi, N. Zucchet, N. Scherrer, N. Miller, M. Sandler, B. A. y Arcas, M. Vladymyrov. R. Pascanu, and J. Sacramento. Uncovering mesa- optimization algorithms in transformers. CoRR, hbb//2309005888 2023. URL https://doi. 10//10..22 org/10.48550/
48550/ arXiv. 2309. 05858.
[118] E. Vorontsov, C. Trabelsi, S. Kadoury, and C. Pal. On orthogonality and learning recurrent networks with long term dependencies. D. Precup and W. Teh, editors, Proceedings of the 34th International Conference Machine Learning, ICML 2017, Sydney, NSW, Australia,
19

---

6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 3570- 3578. PMLR, 2017. URL httphtt://ppooeeee /ttp:::////cceiings.... mlr ottpp//eoiiiaii..ppppp/ee///rooa
html.
[119] C. X. Wang, O. Tsepa, Ma, and B. Wang. Graph-mamba: Towards long-range graph sequence modeling with selective state spaces. ArXiv, abs/2402.00088, 2024. URL https:// api www.ssematiicchhcc
org/ Ses//CrpuusDDD 267364853.
[120] J. Wang, J. N. Yan, A. Gu, and A. Rush. Pretraining without attention. In H. Bouamor, J. Pino, and K. Bali, editors, Findings of the Association for Computational Linguisties: EMNLP 2023, pages 58-69, Singapore, 2023. Association for Computational Linguis-
tics. doi: 10. 10.18653/v1/2022 10./65//1/02..nnnnnnmmmm/// findings-emmll.5. URL https: htttps//acllanthooogg..ooo org.//223.. findings-emnlp..
[121] B. Widrow, M. Hoff, et al. Adaptive switching circuits. IRE WESCON convention record, volume 4, pages 96-104. New York, 1960.
[122] N. Yan, J. Gu, and A. M. Rush. Diffusion models without attention. 2023.
[123] S.
Yang and Y. Zhang. FLA: Friton-Based Library for Hardware- Implementa- tions of Linear Attention Mechanism, 2024. URL attpps//gtt http//ggtuubbbbb thub. com/ sustcsong11n/ flash- linear-- attention. original-date: 2023-12-20T06:501188
[124] S. Yang, B. Wang, Y. Shen, R. Panda, and Y. Kim. Gated linear attention transformers with hardware-efficient training. ArXiv preprint, abs/2312.06635, 2023. URL htttps///axiii org/ 06635.
[125] S. Yang. J. Kautz, and A. Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule, 2024. URL https:tt///// http////xxvvvvv/////i//////...ia org/ aa///////////// 06464.
[126] M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontañón, P. Pham, A. Ravula, Q. Wang, L. Yang, and A. Ahmed. Big bird: Transformers for longer sequences. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances ence on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings. peaeeeediings..uuiipi..... nwwrinsurnecmm cc/ wwt.l/cccppppp/eeee
Neural Information Processing Systems 33: Annual Confer-
wweee--bbbrtttteeeeee html.
[127] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. Traum, and L. Mårquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791- 4800, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/ P19-1472. URL https: pesttss//aclanttiie htt////aaaaaoooo.oorrrrrose org//19-14777
[128] J. Zhang, Q. Lei, and S. Dhillon. Stabilizing gradients for deep neural networks via efficient SVD parameterization. J. G. Dy and A. Krause, editors, Proceedings of the 35th Interna- tional Conference on Machine Learning, ICML 2018, Stockholmsmüüssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 5801- 5809. PMLR, 2018. URL http: http://proceeddngg mlr wwwpp//w0seeeee/eepeeee 2a.80//2hag188gg html.
[129] J. Zhang, S. Jiang, J. Feng. L. Zheng, and L. Kong. Linear Attention via Orthogonal Memory, 2023. URL https: /ttp///axvvvvvve//////eee/eaaav. org/
11135.
[130] Q. Zhang, D. Ram, C. Hawkins, S. Zha, and T. Zhao. Efficient long-range transformers: You need to attend more, but not necessarily at every layer. In H. Bouamor, J. Pino, and K. Bali, ed- itors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2775- 2786. Singapore, 2023. Association for Computational Linguistics. doi: 10. 20.18653/v1/202. URL https: //tp///aaaaanntggoc.e//mo org/ 2023/222...g findings- ceaddings-mnllpp 183.
[131] Y. Zhang, S. Yang, R. Zhu, Y. Zhang, L. Cui, Y. Wang, B. Wang. F. Shi, B. Wang, W. Bi, P. Zhou, and Fu. Gated slot attention for efficient linear-time sequence modeling. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
20

---

[132] L. Zhu, Z. Huang, B. Liao, J. H. Liew, H. Yan, J. Feng. and X. Wang. Dig: Scalable and efficient diffusion models with gated linear attention. ArXiv, abs/2405.18428, 2024. URL attpss//apt semanticscholaa
Tng/CorpusID: 270068366.
[133] L. Zhu, B. Liao, Q Zhang, X. Wang. W. Liu, and X. Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. ArXiv, 0bb//2411.09417, 2024. URL https://app: semanticscholl
org/CorpusID 267028142.
[134] I. Zimerman, A. Ali, and L. Wolf. A unified implicit attention formulation for gated-linear recurrent sequence models. ArXiv preprint, abs/2405. 165044 2024. URL https: /ttee///arxiie arrxiv. org/ mrg/abs/4405.. 16504.
21

---

A Experiments Continued
A.1 Hyperparameters
We used H100 GPUs for 340M and 1.3B language modeling experiments. Each model uses AdamW for optimization, with peak learning rate of 3 10-1 The 340M models trained using 15 billion tokens and a batch size of 0.5M tokens, while the 1.3B models are trained with 100 billion tokens and batch size of 2M tokens. We use cosine learning rate schedule, starting with warm-up phase of 0.5 billion tokens for the 340M models and billion tokens for the 1.3B models. Both configurations have initial and final learning rates at 3 10 We apply weight decay of 0.01 and use gradient clipping at a maximum of 1.0. The head dimension of DeltaNet is set to 128, and the kernel size for convolution layers is set at 4.
A.2 Synthetic tasks
We additional conduct experiments on RegBench [2], synthetic data set designed to assess the in-context lan- guage learning capability of different model architectures. Each input sequence in this benchmark consists of 10 to 20 strings drawn from a distinct language defined by probabilistic finite automaton (PFA), so that a model needs to infer the underlying language from the context on the fly. During testing, model is evaluated dicting the next token of testing sequences generated from held-out PFAs. Here again we find that DeltaNet performs strongly compared to baselines, shown in Figure 7.
100
Acconsy (50
cou)
75
GLA canr)
coms)
Manba (a. cons)
Manba (oel. cour)
Trining examples (K)
Figure 7: Accuracy (%) on RegBench.
pre-
B Method Continued
B.1 WY representation derivation
To reduce notational clutter, we discuss only the first chunk here.
We first show P., = I- En= wikk by induction,
P.
(.k,k'))
[=1
P Pn-1(I- )
n-1
wik, (II Baknkn))
n-1
Σ
wiki - Br kin +
we wik, Bakki,
R-1
m-1
=I
wikk
Bykn - Bn
Σ
(k k.
1=1
1
w,ki
tm
22

---

Similarly, we show S, = Ei=1 u,k, by induction,
S, -n Bnknk + B Un
uphig
(I Bakinkin) + By Wn
n-1
n-1
uek:
Σ
ucki
Baknk + Bn Un
1=1
n-1
n-
upki +
Un - Bn
Σ
Ut
kig
kin
t=1
1=1
B.2 UT transform derivation
Here we provide detailed derivation of the UT transform used in $3.2. Specifically, we show that the matrix formulation using the UT transform (Equations 10 and 11 in the main text) is equivalent the recursive update equations (Equation 7).
We begin with the recursive formulation for computing wi/) and uis as given in Equation 7:
kit
w/ (
(a))
"(q = Sh
Our goal is to show that these recursive updates can be rewritten in matrix form using the UT transform:
TØ [I (r))(diag((1)) Kn) -1)) diag (914)
W []] K
Un Tn Vi
First, consider the computation of wh. For any row r. we can write:
Wir = 1 K Kpjr
Pig
W a (K (r.
This system of equations can be written matrix form. Let us define:
[ diag((11) Lig tril( K K -1)
Then the system becomes:
W i Li W = B
Thus, we can solve for W :
W [0] = (I + Lnj) BB K1 = Tiy K
where
T = (I [(j) B[e))
The same derivation applies for Un by replacing K [s] with Vy in the final step, yielding:
U/ T4 V (
(12)
23

---

C Related Work Continued
Chunkwise linear attention. Hua et [34] first proposed chunkwise form for linear attention; however, they used a hybrid linear and nonlinear attention model similar to Munkhdalai et al. [74]. possible adapt their algorithm to compute the exact output of the pure linear attention, as shown in Sun et al. [108] and Yang et al. [124]. The chunkwise linear attention algorithm has also been independently discovered several works [108, 45, 19). Yang et al. [124] and Qin et al. [91] discuss 1/O-aware hardware optimization for chunkwise linear attention and Sun et al. [107] make generalization to multi-node distributed training. Inspired by the chunkwise form, we propose a new algorithm for hardware-efficient DeltaNet training, significantly improving the training efficiency and allowing for large-scale experiments.
Hybrid models. Linear recurrent models including state-space models [32, 105, 31, 120], gated linear RNNs [63, 90], and linear attention mechanisms [48, 124] have demonstrated remark- able potential alternatives to traditional softmax attention across diverse domains [122. 132, 54, 133, 119, 100]. Given their complementary strengths, recent research has increasingly fo- cused on developing hybrid architectures that combine linear recurrent layers with local chunk attention [61, 130, 25, 62, 74] or sliding window attention [130, 6, 21, 95] or global attention [51, 52, 35, 26, 55, 78, 109]. Poli et al. [84] systematically study the scaling law of hybrid mod- els. We similarly show that combining DeltaNet with classic attention is an effective strategy.
type of orthog- onal matrix extensively used in machine learning [67, 71, 128, 113, 89, 115]. These matrices allow for efficient computation of inverses and their Jacobian determinant of one, making them partieu- larly suitable for applications in normalizing flows [67, 115]. Notably, Mathiasen et al. [67] and Mathiasen et al. [66] developed chunkwise fast algorithm for computing the cumulative product of Householder matrices for normalizing flows, leveraging the WY representation. Our approach, while sharing the same high-level concept, tackles different problem and arguably more general.
Householder matrices. Houscholder matrices, known for preserving norms,
There has also been significant interest in using orthogonal matrices to parameterize the transition matrices of RNNs [71, 43, 118, 33] for mitigating vanishing gradients. Mhammedi et al. [71] use the WY representation to reduce the memory footprint when training nonlinear RNNs with Householder transition matrices.
24

---

D Pseudo code
def chunk_delle laaaaaddddaa_ueefooorrreee K, v, beta, C)
Q/K/V: query, key, value of shape [L, d] beta: beta of shape [L]
C: chunk size
...
# L: sequence length, d: head dimension
L, d s.shape
10
chunking
Q, K, V map (lambda (-1,,,) [Q. K, v])
12
beta reshape C)
K_beta = K beta. unsquezzee (-1)
V_beta
beta. unsquezzee (-1)
16
compute eq. 10 with vectorized forword substitution for fast inverse T - (K_beta @ tril(())))
18
for i in range C):
T[i, :i] T[i, :i] (Tíi, ;, None] T[:, :1]) s-m( T corcheeee(C))
20
compute Eq. 11
22
W T @ K_beta
U T @ V_beta
24
chunkwise parallel. Eq. 8-9
S torch. (dr d)
0 torch. parch.empt_llkke for in rL/
q-i, k_i, w_i Q[i], K[i], W[i]
uii U[i] 0 o_inter q-i e A_i (q_i (..i@ki.tt))...... k.i.t()).tri) o_intra A_i A u_i S k_i.t() u_i 0[i] o_intra + o_inter
35
return 0 (Is d)
Listing 1: Pytorch-like code snippet the forward pass our chunkwise algorithm for training DeltaNet. We omit the dimensions of batch size and number of heads for clarity.
25

---

NeurIPS Paper Checklist
The checklist designed to encourage best practices for responsible machine learning research, ad- dressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to the checklist ation. While "[Yes] generally preferable to "[No] is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] " not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification elaborate, All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point the section(s) where related material for the question can be found.
of the factors in their evalu-
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist".
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: This paper's contributions and scope are reflected abstract and introduction part clearly.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
26

---

Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work in $5.3.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
This paper does not include theoretical results that require a full proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided the core of the paper should be comple- mented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
27

---

The paper provides sufficient details on hyperparameters and training proce- dures in $A.1 reproduce the results supporting its main conclusions.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our code is publicly available at https://github com/ hom/sustesongll flash- lite-llinear--e attention. Our primary training corpus is Slimpajama, an open-source dataset.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines huttps://nigs http:::://ipsscccpuul//////////
for more details. The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
28

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes)
Justification: We have detailed all the training and evaluation settings before the main results in the experimental part.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not have enough resources to obtain error bars as running the experi- ments multiple times is computationally expensive due to the large model size.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
29

---

Justification:
We provide information of GPU type and number of GPUs used for running
our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / http//enurrpp.ppe htcpppbllc//thccccuddellllllllll
Answer: [Yes]
Justification: This work follows the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We foresee no potential societal impact of this work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
30

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We foresee no such risks posed by this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
allow for controlled use of the model, for example by re-
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All of the datasets we use are publicly available at huggingface site, and we have properly cited all the training and evaluation datasets we used.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, asperswithcodee com/ dataseta has curated licenses for some datasets. Their licensing guide can help determine the li- cense of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer:
[NA]
Justification:
This work does not release assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
31

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "hQNYHP50sF", "reviewer": "Reviewer_eUyw", "summary": "This paper proposes a hardware-efficient algorithm for training linear transformers with a delta update (DeltaNet; SMS21). This architecture has an attention formulation that prevents the direct application of chunk-wise parallel algorithms for computing its output. To address this issue, the authors introduce a re-parameterization of DeltaNet as a matrix-valued RNN whose recurrence is given by a generalized Householder transformation. This enables the use of WY representation which is memory efficient and eliminates the need to materialize the hidden state matrices. Experiments on synthetic benchmarks and language modeling tasks shows competitive performance compared to strong baselines (Mamba, GLA) and faster speed than the original Deltanet implementation.", "questions": "- Q1: What is the effect of chunk size in the chunk-wise parallel algorithm for DeltaNet? Varying the chunk size $C$ and showing its effect in efficiency would be interesting to explore. \n- Q2: The chunk-level hidden states $S_{[t]}$'s are discarded to save memory. From Eq. 7, it seems that their computation depends on the previous hidden states $S_{[t-1]}$'s. Are these kept in memory for the re-computation in the backward pass?  \n- Q3: GLA with convolution performs worse than w/o convolution with the larger model size. Do you expect this to be the case for DeltaNet as well? It would be good to add this result if possible. \n\nMinor:\n- In Table 2, is the L1/L2 norm referring to the normalization of queries and keys? Please specify. \n- In Eq.1, why is this equation showing the state $S_{[t+1]}$ instead of $S_{[t]}$? The latter is used in Eq. 2. Same for Eq. 7.  \n- l172: stable -> stabilize\n- l213-214: we -> we follow\n- l321: vallina -> vanilla", "limitations": "Yes, they have.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well motivated and situated with respect to prior work. It provides sufficient background for linear transformers, demonstrates great scholarship in crediting prior work, and has a clear exposition of the proposed idea. In addition, it presents an informative overview that compares the formulations of recent linear transformers that highlights their differences.  \n- Proposes an efficient algorithm for training linear transformers with the delta update which is a competitive variant. The re-parameterization is non-obvious and leverages WY representation for Householder matrices in a novel way. Previously, this architecture could not be easily scaled to larger models and datasets with a recurrent formulation. In addition, it introduces two competitive hybrid methods based on DeltaNet that leverage local and global full attention. \n - Demonstrates the effectiveness of the proposed approach on two synthetic benchmarks and eleven language modeling and understanding tasks compared to strong baselines such as Mamba and GLA. The results are consistent, have a good coverage, and are important for the researchers working on efficient transformers. \n- The experiments are thorough and have convincing settings, namely all the variants are trained from scratch with the same configurations, there are ablations to justify the design choices, and the experimental reporting is very detailed.", "weaknesses": "- W1. In terms of scale, the model explores two different architectures of increasing size up to 1.3B parameters. Even though this size is considerable, it is still relatively small compared to the LLMs that are widely used such as Llama, Mistral (7B+ size). There is always the question of whether the quality is maintained with further model increase.\n- W2. The improved results compared to Mamba and GLA make use of additional architectural components: convolution and local/global attention, without them the results are comparable to the other models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review.\n## W1: larger scale experiments\nAs noted by the reviewer, it is difficult to conduct experiments at 1B+ scale. Nonetheless, we are currently running some larger-scale experiments at the 3B parameter scale. Here are some preliminary results:\n\n\n| Model      | # Tokens  | wikitext PPL | arc-c | arc-e | boolq | hellaswag | openbookqa | piqa  | sciq  | winogrande | average   | \n|------------|--------|-------|-------|-------|-----------|------------|-------|-------|------------|-----------|--------------|\n| 3B-Transformer++   | 167B   | 15.39 | 29.78 | 62.96 | 63.06 | 46.15     | 27.8       | 72.96 | 91.1  | 60.62      | 56.80  | \n| 3B-DeltaNet| 167B   | 15.34 | 29.78 | 63.8  | 65.08 | 46.57     | 26.2       | 74.32 | 88.5  | 58.09      | 56.54   |\n\nAs we can see, the DeltaNet results are comparable to the Transformer++ results at this scale. We will make sure to include these results once they have finished training (we are aiming for 300B-1T tokens, depending on the availability of compute). We also plan to open-source the pretrained models so that researchers can study these models in more detail.\n\nWe have also run experiments on the recently-released [MAD benchmark](https://arxiv.org/abs/2403.17844), a suite of synthetic tasks designed to test the capabilities of language models beyond perplexity. Here are the results:\n\n| Model         | Compress | Fuzzy Recall | In-Context Recall | Memorize | Noisy Recall | Selective Copy | Average |\n|---------------|----------|--------------|-------------------|----------|--------------|----------------|---------|\n| Transformer   | 51.6     | 29.8         | 94.1              | 85.2     | 86.8         | 99.6           | 74.5    |\n| Hyena         | 45.2     | 7.9          | 81.7              | 89.5     | 78.8         | 93.1           | 66.0    |\n| Multihead Hyena | 44.8   | 14.4         | 99.0              | 89.4     | 98.6         | 93.0           | 73.2    |\n| Mamba         | 52.7     | 6.7          | 90.4              | 89.5     | 90.1         | 86.3           | 69.3    |\n| GLA           | 38.8     | 6.9          | 80.8              | 63.3     | 81.6         | 88.6           | 60.0    |\n| DeltaNet      | 42.2     | 35.7         | 100               | 52.8     | 100          | 100            | 71.8    |\n\n\n\n\n## W2: The improved results compared to Mamba and GLA make use of additional architectural components: convolution and local/global attention, without them the results are comparable to the other models.\n\nThanks for the comment! Note that Mamba uses conv layers by default, while GLA does not. This is why we trained our own GLA+conv baselines. Our main experiments compare DeltaNet+conv against Mamba+conv and GLA+conv, i.e., we give all models the chance to use convolution layers for a fair and meaningful comparison. In this setting, DeltaNet outperforms both Mamba and GLA. Morevoer, these depthwise separable \"short convolution\" layers are  cheap both in terms of the number of parameters and compute, and hence we think that this convolution primitive is a practical approach to modeling local interactions.\n\n\n\n\n\n\n\n\n## Q1 Effect of Chunk size C\nThe computational complexity of the WY representation is O((N/C) * C^3) = O(NC^2). If C is too large, WY representation computation will be very expensive (recall that WY computation is fully recurrent).\n\nIf the chunk size is too small, and we adapt the materialization version of FlashLinearAttention, we need to write more hidden states to HBMs, resulting in a high I/O burden and thereby slowing down the actual running speed. If we adapt the non-materialization version of FlashLinearAttention, it will lack sequence-level parallelism, which is not desirable in large-scale training. Please refer to Section 3.3 of the GLA paper for more discussions.\n\nTo provide some intuition, we measured the running time (forward and backward pass) on a single A100 GPU while varying the chunk size \\( C \\):\n\n| Chunk Size \\( C \\) | Forward + Backward Time (ms) |\n|--------------------|------------------------------|\n| 16                 | 4.8738                       |\n| 32                 | 3.6616                       |\n| 64                 | 5.2382                       |\n| 128                | 16.0602                      |\n\nThe experimental settings were as follows: sequence length \\( \\text{seq\\_len} = 4096 \\), batch size \\( B = 2 \\), head dimension \\( \\text{d\\_head} = 128 \\), model dimension \\( \\text{d\\_model} = 2048 \\), and number of heads \\( \\text{num\\_head} = 16 \\). As we can see, the moderate chunk size of 32 performs the best. When the chunk size is less than 32, the I/O cost surpasses the WY representation cost, and vice versa. \n\n\n## Q2: Chunk state\nYes this is correct! Concretely, we adapt the \"non-materialization\" version of FlashLinearAttention for chunkwise DeltaNet implementation. In this version, the hidden states of each chunk are materialized on HBMs in the forward pass, then discarded to save memory. During the backward pass, the hidden states of each chunk are recomputed.\n\n\n## Q3: DeltaNet w/o shortconv in large scale\nThank you for the suggestion. We agree that this will be an interesting experiment to run. We are currently using all our resources for the 3B experiments, but we will run this ablation once the 3B experiments are done.\n\n## Minor \nThanks for identifying the typos/errors! We will fix these.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o2xPg2Q6RX", "reviewer": "Reviewer_pqA2", "summary": "This paper introduces a novel algorithm for the efficient training of DeltaNet Linear Transformers. DeltaNet enhances contextual associative recall using a delta rule-like update but was previously limited by inefficient parallelization in its training algorithm. The work described in this paper presents a hardware-efficient algorithm that leverages the memory-efficient WY representation for computing products of Householder matrices, enabling the scaling of DeltaNet similar to other linear Transformer models. The authors trained a 1.3B parameter model on 100B tokens and found that it outperforms strong linear-time baselines such as Mamba and GLA in terms of perplexity and zero-shot performance on downstream tasks.", "questions": "I have no questions for this paper.", "limitations": "Yes", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a novel hardware-efficient algorithm for training DeltaNet Linear Transformers, leveraging the WY representation of Householder matrices, which effectively addresses the parallelization limitations of previous algorithms.\n- Through large-scale experiments, the authors demonstrate that DeltaNet significantly outperforms existing models like Mamba and GLA in terms of perplexity and zero-shot performance on downstream tasks.\n- The new algorithm enables the scaling of DeltaNet to larger datasets and parameter sizes, which is crucial for large language models.", "weaknesses": "The algorithms presented in this paper are satisfactory in terms of efficiency and performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review! \n\nWe are adding some additional results in case they are of interest.\n\nFirst, we have preliminary results with 3B models trained for 167B tokens:\n\n\n| Model      | # Tokens  | wikitext PPL | arc-c | arc-e | boolq | hellaswag | openbookqa | piqa  | sciq  | winogrande | average   | \n|------------|--------|-------|-------|-------|-----------|------------|-------|-------|------------|-----------|--------------|\n| 3B-Transformer++   | 167B   | 15.39 | 29.78 | 62.96 | 63.06 | 46.15     | 27.8       | 72.96 | 91.1  | 60.62      | 56.80  | \n| 3B-DeltaNet| 167B   | 15.34 | 29.78 | 63.8  | 65.08 | 46.57     | 26.2       | 74.32 | 88.5  | 58.09      | 56.54   |\n\nWe are aiming to train these models for longer (300B-1T tokens) and will report these results in the next iteration of the paper.\n\nWe have also run experiments on the recently-released [MAD benchmark](https://arxiv.org/abs/2403.17844), a suite of synthetic tasks designed to test the capabilities of language models beyond perplexity. Here are the results:\n\n\n\n| Model         | Compress | Fuzzy Recall | In-Context Recall | Memorize | Noisy Recall | Selective Copy | Average |\n|---------------|----------|--------------|-------------------|----------|--------------|----------------|---------|\n| Transformer   | 51.6     | 29.8         | 94.1              | 85.2     | 86.8         | 99.6           | 74.5    |\n| Hyena         | 45.2     | 7.9          | 81.7              | 89.5     | 78.8         | 93.1           | 66.0    |\n| Multihead Hyena | 44.8   | 14.4         | 99.0              | 89.4     | 98.6         | 93.0           | 73.2    |\n| Mamba         | 52.7     | 6.7          | 90.4              | 89.5     | 90.1         | 86.3           | 69.3    |\n| GLA           | 38.8     | 6.9          | 80.8              | 63.3     | 81.6         | 88.6           | 60.0    |\n| DeltaNet      | 42.2     | 35.7         | 100               | 52.8     | 100          | 100            | 71.8    |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jq3aXcAGvg", "reviewer": "Reviewer_dem4", "summary": "This paper proposes the Delta Rule method to construct the state updates for Linear Attention. Furthermore, the paper introduces a chunk-wise training approach, allowing the computational cost of training to grow subquadratically with the text length. Experimentally, the paper validates the effectiveness of the model architecture using three synthetic benchmarks: MQAR, MAD, and RegBench. Additionally, the paper uses Common Sense Reasoning and Retrieval tasks in LLM pre-training to verify the model's performance in real-world tasks. The model has been validated at scales ranging from 340M to 1.3B parameters. Furthermore, this paper explores the possibility of combining the Delta Rule with Sliding Window Attention and Global Attention, demonstrating the positive impact of the hybrid architecture on model performance.", "questions": "I would like to discuss the following questions with you:\n\nDo you think linear models can fundamentally bridge the gap with transformers in memory-based tasks?\n\nIs there an inherent conflict between the ability to handle long context and the performance of memory-based tasks?", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Solid work. The paper provides a good derivation, offering a more general method for state updates in Linear Models.\n2. The experiments are comprehensive and effectively demonstrate the validity of the model architecture.", "weaknesses": "1. Have you conducted experiments on long context? For example, measuring extrapolation and scenarios akin to \"looking for a needle in a haystack\"? As a linear model, I would like you to further discuss its capability to generalize to long context.\n2. The algorithmic speed of Delta Net increases linearly, but it seems to be slower than GLA. Can you analyze the factors contributing to this?\n3. Could you further explain the insights of the Delta Net updates? I understand there are algorithmic differences compared to GLA operators, but what unique benefits do they bring? Is there any theoretical analysis?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review!\n\n## W1 Long context experiments\nThanks for your suggestion. Models at 1B scale are currently not powerful enough to provide meaningful results on needle-in-the-haystack style long-range benchmarks. We are currently training models are larger scale (3B parameters), and will run the needle-in-the-haystack tests with these models once they have finished training. (We give the 3B model results, which have been trained on 167B tokens so far, on the other benchmarks. We find that DeltaNet performs comparably to Transformers at this scale.)\n\n\n| Model      | # Tokens  | wikitext PPL | arc-c | arc-e | boolq | hellaswag | openbookqa | piqa  | sciq  | winogrande | average   | \n|------------|--------|-------|-------|-------|-----------|------------|-------|-------|------------|-----------|--------------|\n| 3B-Transformer++   | 167B   | 15.39 | 29.78 | 62.96 | 63.06 | 46.15     | 27.8       | 72.96 | 91.1  | 60.62      | 56.80  | \n| 3B-DeltaNet| 167B   | 15.34 | 29.78 | 63.8  | 65.08 | 46.57     | 26.2       | 74.32 | 88.5  | 58.09      | 56.54   |\n\n## W2: reason of deltanet being slower than GLA.\nBoth DeltaNet and GLA scale linearly with respect to sequence length; however, the recurrent transition matrices in GLA are diagonal matrices, making them amenable to tiling techniques (commonly used to accelerate matrix multiplications on GPUs) because the hidden states are independent of each other. The recurrent transition matrices in DeltaNet are more expressive, modeling state-to-state dependencies and thus requiring \"marginalizing\" over the entire head dimension, making tiling less amenable, as discussed in the limitations section (lines 301-306).\n\nAdditionally, the computation of the WY representation in DeltaNet is more expensive than computing the cumulative product of decays in GLA. From Eq. 7-8, we can see that the DeltaNet chunkwise form involves more matrix multiplications than vanilla linear attention (and also GLA). \n\n\n\n\n## W3 Intuitive explanation and theoretical justification of delta update rule \n\n  \nWe found [1] motivating the delta rule intuitively from the perspective of key-value associative memory: the key intuition of the delta rule is to subtract the value associated with the current input key, namely the old value, from the memory, and write a new value that is a linear combination of the old value and the current input value into the memory. This encourages the formation of key-value associations, making retrieval easier.\n\nRegarding theoretical justification, in lines 322-323, we reference several theoretical papers that reveal the superiority of the delta rule in memory capacity in a mathematical way.\n\n[1] Linear Transformers Are Secretly Fast Weight Programmers https://arxiv.org/pdf/2102.11174\n \n\n \n## Q1 Do you think linear models can fundamentally bridge the gap with transformers in memory-based tasks?\n\n\nWe don't think linear models cannot fundamentally bridge the gap with Transformers in memory-based (or recall-intensive) tasks. Several theoretical papers reveal such limitations [1, 2].\n\nStill, we believe the use of the delta rule will push the Pareto frontier of the recall-memory tradeoff curve, a concept that was advocated in [2].\n\nIt is possible and promising to replace the subquadratic encoder in YOCO [3] with DeltaNet to fully bridge the gap with Transformers in memory tasks.\n\n\n\n\n[1] RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval. https://arxiv.org/abs/2402.18510\n\n[2] Simple linear attention language models balance the recall-throughput tradeoff https://arxiv.org/abs/2402.18668  \n\n[3] You Only Cache Once: Decoder-Decoder Architectures for Language Models https://arxiv.org/abs/2405.05254\n\n## Q2: Is there an inherent conflict between the ability to handle long context and the performance of memory-based tasks?\n\n \nThis is a good question, and one for which the community doesn't have a good answer for yet (in our opinion). Our sense is that memory-based tasks will always require some attention-like mechanism which \"retrieves\" portions of thecontxt. It is possible that subquadratic-but-superlinear attention mechanisms (e.g., those based on clustering) may enable efficient long-context modeling while still enabling good performance on memory-based tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: It is well-known that standard softmax dense attention has quadratic complexity with respect to sequence length.  In recent years, there has been a great deal of interest in linear transformers, which have O(N) complexity, a fixed-size memory, and a theoretically unlimited context.  In practice, of course, it is not possible to fit an unlimited context into a fixed-size memory, so the "delta rule" introduces a mechanism that allows linear transformers to forget.

Unfortunately, the delta rule is not necessarily easy to implement efficiently in practice.  This paper develops an efficient parallel implementation of the delta rule, using the WY representation for computing products of Householder matrices.  The authors validate the effectiveness of the implementation with a variety of experiments, and also introduced new and compelling results on the MAD benchmark during the discussion period.

The reviewers were unanimous in recommending acceptance, and I concur.

Despite the overall high scores, my personal opinion is that the space of "alternatives to softmax attention" is at this point very crowded.  IMO, the primary reason for considering alternatives is to enable long-context.  However, the "forget" mechanism is not necessarily well-suited to long-context, and as the authors explain in the discussion, the models they are using (1-3B params) are too small for good long-context experiments.  As a result, although I believe this paper is a solid contribution, I don't recommend it for a spotlight or oral.
Decision: Accept (poster)