Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformers
Authors: Kazuki Irie
Morris Yau
Samuel J. Gershman
Venue: NeurIPS 2025 poster ()
Keywords: sequence processing
transformers
linear transformers
fast weight programmers
key-value memory
memory modeling
hybrid transformers
Primary area: deep_learning
TL;DR: We combine two types of memory systems from quadratic and linear transformers into a single hybrid memory system to leverage their complementary strengths in context coverage, precise retrieval, and expressivity.

Abstract: We develop hybrid memory architectures for general-purpose sequence processing neural networks, that combine key-value memory using softmax attention (KV-memory) with fast weight memory through dynamic synaptic modulation (FW-memory)---the core principles of quadratic and linear transformers, respectively. These two memory systems have complementary but individually limited properties: KV-memory offers precise retrieval but is constrained by quadratic complexity in sequence length, while FW-memory supports arbitrarily long sequences and enables more expressive computation but sacrifices precise recall. We propose and compare three methods to blend these two systems into a single memory system, differing in how and when input information is delivered to each system, to leverage the strengths of both. We conduct experiments on general language modeling and retrieval tasks by training 340M- and 1.3B-parameter models from scratch, as well as on synthetic algorithmic tasks designed to precisely illustrate the benefits of certain hybrid methods over others. We also evaluate our hybrid memory systems on reinforcement learning in partially observable environments. Overall, we demonstrate how a well-designed hybrid can overcome the limitations of its individual components, offering new insights into the design principle of neural memory systems.

Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformers
Kazuki Irie' Morris Yau? Samuel J. Gershman' 1.3
Department of Psychology and Center for Brain Science,
Harvard University, Cambridge, MA, USA
MIT CSAIL, Cambridge, MA, USA Kempner Institute for the Study of Natural and Artificial Intelligence,
Harvard University, Cambridge, MA, USA
(kirie, bershman]&ffal harvard edu, morrisy@mit.ccmmmm edu
Abstract
We develop hybrid memory architectures for general-purpose sequence process- ing neural networks, that combine key-value memory using softmax attention (KV-memory) with fast weight memory through dynamic synaptic modulation (FW- memory)-the core principles of quadratic and linear transformers, respectively. These two memory systems have complementary but individually limited proper- ties: KV-memory offers precise retrieval but is constrained by quadratic complexity in sequence length, while FW-memory supports arbitrarily long sequences and enables more expressive computation but sacrifices precise recall. We propose and compare three methods blend these systems into single memory system, differing in how and when input information is delivered to each system, to leverage the strengths of both. We conduct experiments on general language modeling and trieval tasks by training 340M- and 1.3B-pan --BB-parameter models from scratch, as well as on synthetic algorithmic tasks designed precisely illustrate the benefits of certain hybrid methods over others. We also evaluate our hybrid memory systems on rein- forcement learning in partially observable environments. Overall, we demonstrate how a well-designed hybrid can overcome the limitations of its individual compo- nents, offering new insights into the design principle of neural memory systems."
Introduction
Modern transformers come two flavors. The standard "quadratic transformer" (QT) [1] leverages softmax attention over explicit key-value memory storage which enables precise memory retrieval [2], while the length of sequences they can handle is limited in practice due to its quadratic computational complexities (as reflected in our model terminology). On the other hand, "linear transformers" (LT), also known as fast weight programmers (FWPs) [3, 4] sacrifice the precision of the softmax function (the complexity bottleneck of QT), but in return they support the processing of arbitrarily long sequences. Furthermore. some of their variants (called DeltaNet models 8]) enable more expressive computations (e.g., certain types of state- tate-tracking), which QTs struggle to perform [7-16].
While the unique advantages of quadratic and linear transformers may seem inherently incompatible to achieve simultaneously. they are all desirable in an ideal general-purpose memory system. How could achieve multiple computational properties that are incompatible within single system? The brain's 's apparent solution is to integrate multiple types of memory mechanisms; each with distinct properties that meet specific demands under various constraints, thereby enabling flexible problem
'Our code is public: https://githut: wtw. wwwp///ssukktirrr////rrtt.......
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Table 1: Complementarity of the memory systems in two transformer types.
Property
Key-valuc memory
Fast weight memory
Complexity
quadratic
linear
Context length
bounded
unbounded
Retrieval precision
high low
low
Expressivity
high
solving. One example of such a view is the classic Complementary Learning Systems architecture [17. 18], which hypothesizes that the brain has division of labor into separate (but interacting) episodic and semantic memory systems whose functions are incompatible (remembering specifics vs. extracting generalities). Here we propose different division of labor into complementary linear and quadratic memory systems each with complementary strengths (Table 1).
We propose and compare three blending methods that conceptually well-motivated-dher differ in how and when input information is delivered to each memory system, and empirically evaluate them ultimately converge
single architecture. We conduct experiments test general language modeling and in-context retrieval abilities (using the standard eeeeevaaaiionaaaraeeee by training 340M- and 1.3B ----paraeeter language models from scratch using 15B tokens of the HuggingFace Fine Web-Edu dataset [20]. We also conduct experiments on synthetic algorithmic tasks (parity and modular arithmetics) [7] determine the type of hybrid methods that can preserve the expressivity advantages of DeltaNet (which we use as the LTT/WWP component of our hybrid)- - critical discussion and evaluation that are completely absent in prior discussions on quadratic and linear transformer hybrids [21, 22]. Overall, we demonstrate the benefits of hybrid model designs that can take the full advantage the latest LT/FWP models, and show how a well-designed hybrid can mitigate the limitations of its individual components; this offers novel insights into the design principle of neural memory systems.
[19]),
Finally, all our models are scalable as they are compatible with the algorithms and implementations for highly efficient quadratic and linear transformers that have been proposed in prior work on flash -tsshattention [23] and flash -iiharr---rr--ee -ateentio [24] (see our code link on page I).
2 Background
Here we provide background necessary describe the proposed hybrid models Sec. 3, namely: key- value memory (Sec. 2.1) and fast weight programming/memory (Sec. 2.2), which are the comeestoones of quadratic and linear transformers, respectively.
Common Settings. Let din, douts dhey, denote positive integers. All the models studied here are causal sequence processing neural networks, which, every time step i, receive an input E Rd and produce an output y, E dRdu while maintaining an internal memory of all inputs received so far.
2.1 Key-value memory & Softmax attention
The main sequence processing computation in the standard/quadratic transformer [I] is the self- attention operation, which, at every time step receives an input æ; Rd and produces an output y: Rdoa, while maintaining the key-value memory, represented as two matrices K; E 14 xt and
V, E Rddaxt as follows:
[qt, Ki, Uo] slloow
(1) (2)
K, K, + ke: V. = Vi-1 + v
Visoftmax K, q:)
where qt, Kr RRo E RRuu (with denoting vector concatenation). Wallow R (2d + xd. is the trainable weight matrix (meaning of the superscript "slow" becomes clear in Sec. 2.2), @ as in
K₁-1 + k; denotes concatenation of vector k; E d to matrix K,-1 E Bh x (t-1) along the time dimension, yielding K, E Rd xt (K, and Vo are initially empty). We omit the 1/ 1/ scaling inside softmax, as well as the output projection, which are irrelevant to our discussion.

---

Eq. 3 is the so-called softmax attention operation; the query vector from the current step t is compared the keys from all the time steps through dot product; these similarity scores are sharpened through the softmax function (crucial for precise retrieval), and the resulting scores are used as coefficients to compute the weighted average of all the value vectors to produce the output. As can be seen in Eqs. 2, the sizes of key and value memory storage linearly grow with the time step (i.e., sequence length), resulting in quadratic complexity w.r.t. sequence length in the attention computation in Eq. 3. Consequently, practical self-attention requires predetermining maximum context length, i.e., size of the sliding window, and any input that falls outside the window is discarded. On the other hand, training of such model can be highly efficient modern hardware as the computation above is parallelizable over time steps/sequence elements.
2.2 Fast weight memory & Linear attention and delta rule
By replacing the softmax normalization in Eq. 3 by a non-linea activation function applied each key and query vectors individually (which we denote 6), the attention computation of Eqs. 2-3 can be reorganized [3, 25, 5, 26] to derive the so-called "recurrent form" of linear attention (LA) or fast weight programming (FWP) operation. Let @ denote outer product. The "recurrent form" of a linear attention [3, 4] a sequence operation that, at each time step 4, transforms an input æ, E IRda to an output y, Rd while maintaining the so-called fast weight matrix W, E Rd as memory
state, as follows:
[qc, kr, 2a} = Walow It
(1)
W. W tu @ (kks)
(4)
y/ = Wiolq:)
(5)
where Eq. I remains the same as in Sec. 2.1, and the "fast- "fast-changing" weight matrix W, E Ru is initially set to O, i.e., Wo 0. This can be viewed as system of two networks where one net (the slow net; Eq. 1) learns "program" the fast net (Eq. 5) by generating its weight changes (Eq, 4h-the origin of their names. Given that the computation per step is constant w.r.t. the time step/sequence length, the time complexity of this model linear w.r.t. sequence length.
Note that the recurrent form of LA derived by Katharopoulos et al. [3] introduces an extra temporal variable (z, -t-1 + of kt); with 2g = 0) which is used compute the normalizing term (z, o(q:) ) applied to the output (Eq.5) 5). However, more recent work has shown that such extra computation is unnecessary in practice [5, 27-29] with appropriate choice of ø (discussed below).
We refer to the constant-size, context-dependent, time-varying fast weight matrix W, as fast weight memory (a terminology which has also been used in past work augmenting an LSTM with other fast weight programming operations [30]), to contrast with the key-value memory of Sec. 2.1. We denote them as FW-memory and K KV-memory, respectively, for short.
Chunk- wise Parallel Training. In practice, using the linear-complexity recurrent form of LA above for training is inefficient as it purely sequential computation. Instead, a more practical method for training is a "hybrid" approach, called chunk-wise parallel training. The main idea is to divide a training sequence, as well as the corresponding computation, into small chunks; and causally process one chunk after another. The intra-chunk computation leverages the parallel computation, while the inter-chunk contribution uses the recurrent form. More formally, let and n denote positive integers; all the outputs in the n-th chunk of size S, denoted as Y, E IR XS
XS can be computed as:
Y, W.Q.. Vn(K ④ M): W W. V,K
(6)
where Qn Kn E R and Vn vectors within chunk n, and Wn E Iad dhey is the fast weight memory up to chunk n (exclusive), with W. 0, M E 5x5 is the causal mask, and @ denotes dementtwisc multiplication. The first and second terms of Eq. 6 (left) correspond to the inter- and intra-chunk computations, respectively (while Eq. 6 (right) is the fast weight memory update). This results in efficient sub-quadratic complexity algorithm for training LA (and other FWPs) in practice [31, 27, 28).
denote the matrices containing the query, key, value
Delta rule extension. Motivated by prior practices and successes of replacing purely additive Hebbian weight modification rule by the delta rule [32, 33], DeltaNet [5] replaces the purely sum
3

---

update rule of LA (Eq. 4) by an update rule with functional form of the classic delta rule [34]:
[9ts ki, Ut. Pr] slow Dt
(7)
W Wi-1 + (B)) - (k-))(k()) @ (kks)
(8)
Y Wlo(q:)
(5)
where the dimension of W/show is increased to be + sion (+11) is introduced to produce an addition variable 3 E R. a is an activation function applied to Be, which is typically sigmoid or 2 times sigmoid (the factor 2 introduces negative eigenvalues in the state transition matrix enabling strong state- -tae--racking abilities [7]). The choice of Φ is particularly important for stability when the delta rule is used [5]; here we define & as element-wise sigmoid linear unit (SiLU; æ times sigmoid) followed by L2 normalization as in Yang et al. [29]. Eq. 8 corresponds rank-one update of the fast weight matrix, from W i to Wr, through the delta learning rule
i.c., extra output dimen-
[34], where the slow net-generated variables, Vt. k(k,), and a (Br), play the role of target, input, and learning rate of the delta rule, respectively.
DeltaNet has seen a recent revival as Yang et al. [29] developed a scalable chunk parallel training algorithm for it. While prior extensions of DeltaNet have solely focused improving its expressivity [35, 36, 10], recent extensions remarkably preserve the efficient parallel training property [6, 8]. DeltaNet distinguishes itself from other linear transformers due its capabilities for state tracking [7, 8]: therefore, we'll adopt DeltaNet as the FW-memory component in our hybrid models. Nevertheless, our hybrid approach can naturally extend other LT models or DeltaNet extensions.
3 Method: Hybrid Quadratic-Linear Transformers
We study three strategies for hybridizing FW-memory (Sec. 2.2) and K KV-memory (Sec. 2.1) in a single system: Delayed-Streaming (Sec. 3.1), Delayed- Chunkk (Sec. 3.2), and Synchronous (Sec. 3.3) approaches. As we'll show, each of these three approaches is motivated by different arguments and has connections prior work. We conceptually discuss their advantages and validate them empirically in Sec. 4. We refer to these Hybrids of Quadratic and Linear Transformers as HQLTs.
3.1
HQLT
The first variation of HQLT we discuss here is derived by the main goal of addressing the context window size limitation of KV-memory (Sec. I). A typical KV-memory operates on a sliding window with a limited size covering only recent past, while discarding key-value pairs which fall outside of such KV-memory span. In the proposed -ttrand--t HQLT", instead of completely discarding old key-value pairs, they integrated into separate FW FW-memory. This yields the following sequence model.
Let é denote the 'remove' operation taking matrix and one of its column vectors to be removed as arguments. S denotes the window size, i.e., the number of key-value pairs K KV-memory can store.
At every time step t, an HQLT takes an input 2, E Rd and produces an output y: E Rd while maintaining two types of memory: (1) KV-memory, denoted two matrices K₁-1 E gd and Vz-1 E S and (2) FW-memory, denoted as a matrix W,- E 1gdanxde, As in DeltaNet (Sec. 2.2), the trainable parameter matrix is Wsstow E t22 and variables are first generated from the input follows:
(g:, ki, Vt. 8.] = Wrslow
(9)
KV-memory is updated by removing the oldest key/value pair (ki-s, U1-s) from the memory, and adding the newly generated one (kt, v:):
K, = K, e k. + k.
(10)
V, = V, U.ss $ 12
(11)
In Delayed-Streen Streaming HQLT, the key/value pair removed from K KV-memory is fed to FW-memory (Sychronous HQLT differs in this aspect; as we'll see in Sec. 3.3):
W.- = TipdateRuleWW.-.. ki-s, 5t-5, Bt)
(12)
= W.- + (B,)(( W,-10(k, @ (k;,s))
(13)

---

A
B
Softmax Attention
Attention
value memory
Key value memory
weight memory
Fast weight memory
Kg
Kt
An illustration for Hybrid Quadratic-Linear Transformers (HQLTs). Two variations are shown. In the "Delayed-Stream" variant (A), the newly generated key/value pair only fed to the key-value memory (K (KV-memory) system, and the old key/value pair that falls outside the context window of KV-memory is fed to the fast weight memory (r--memor) system. In the "Synchronous" variant (B), the key/value pair generated at time step é is fed to both KV-memory and FW FW-memory systems. The dynamic learning rate variable Bt and memory mixing variable " omitted.
Figure I:
This last equation corresponds to the delta rule as Eq. 8. Note, however, that the choice of UpdateRule is arbitrary in principle; e.g., Gated Delta Rule [6] or Delta Product Rule [8] could be used. Here we focus on the most basic DeltaNet.
Finally, the output is computed by combining outputs from the two memory systems:
y: = Fast Weight W,, q:) + Key VelueMeorry K2, V,, 4:)
(14)
Wio(q:) + Visoftmax(Kis K? q:)
(15)
In this Pelayed-Streaminn HQLT, the division of labors between the two memory systems is as follows: FW-memory We is responsible for storing all the relevant information for time steps t (anything that older than S time steps), while KV-memory (K,, Vi) enables precise retrieval on the most recent context t S < <<
HQLT achieves this untruncated-conter sequence modeling with fixed memory size corresponding to the total KV and FW memory matrices, i.e., S(dnn + daa)) + domt din- The model illustration can be found in Figure 1A.
Memory mixing/gating. We further enhance Eq. 15 by exploring weighted mixing strategies between the FW-memory and KV-memory outputs, by generating additional context-dependent weights Eq, 9 by increasing the output dimension Waslow accordingly. We refer to Eq. 15 as sum mixing. In addition, we propose dynamic scalar mixing which generates two scalar variables at a E [0, 1] (we apply sigmoid to achieve this bound) as:
Yt = a
FW Fast Weight Wt, q:) + +a, KV
KoyValueMemory(KK V,, q:)
(16)
and dynamic vector mixing by generating a vector variable ? dh which is used as:
y: = % @ Fast Weight W,, q:) + m) @ KeyValucMenory(KK
v:)
(17)
The dynamic scalar mixing variation is akin to the one used in prior work [22], except that generate and apply a separate scalar on each term.
3.2 Delayed-Chunk HQLT
While Delayed-Streaming HQLT above seems natural from the perspective of streaming causal sequence processing, another "delayed" variation can be motivated by the chunk-wise training algorithm of Eq. 6 (left). In fact, HQLT can be naturally obtained by introducing the softmax
5

---

activation in the intra-chunk attention operation of Eq. 6 (left; the second term), i.e., (K, Qu @ M)); and by using this chunk-wise processing for inference too. We call this approach "Delayed-Chum HQLT". We provide further comments about this model in Appendix B.1.
This model also directly connects to a closely related prior model proposed in Munkhdalai et al. [22] (see also [37]). However, we'll see how these "delayed" architectures are sub-optimal compared to the synchronous approach below, in light of recent advances in DeltaNet models, and to design hybrid models capable of leveraging the expressivity advantage of their FW-memory component.
3.3 Synchronous HQLT
The delayed versions are conceptually elegant in their approach to introducing a division of labor based on the age of key-value pairs in memory (K V--mmmoryy is responsible for recent ones, while FW-memory takes care of old ones). However, this is not compatible with the idea of potentially leveraging the expressivity complementarity; recent advances in FWPs demonstrate an expressivity advantage of DeltaNet over softmax attention [7, 8, 10]. Thus, in a hybrid system, FW-memory could play a crucial role when the system has to deal with types of computations that KV-memory cannot perform, and this may require FW-memory to also process the most recent inputs.
This motivates "Synchronous HQLT" in which both KV-memory and FW-memory process the same input simultaneouuly. The corresponding equations are identical to those of Belayed-Streaming HQLT (Sec. 3.1) except that in the Synchronous one, the key-value pair produced at time step is fed to both KV-memory and FW-memory (i.e., Eq. 12 takes (k,, v2) as inputs). Figure 1B illustrates the model.
This approach also directly connects the hybrid model explored in Arora et al. [21], which uses the vanilla LA, whose performance is known to largely lag behind those of more advanced FWPs, such as DeltaNet and GLA [5, 35, 28]. As we demonstrate experimentally, taking into account the latest advances in FWPs/DeltaNet is crucial to study the optimal design choice of HQLT a critical aspect that has been missing in prior work [21, 22].
4 Experiments
We conduct main experiments using three types of tasks: general language modeling tasks (for general evaluation and sanity checks of HQLTs; Sec. 4.1), synthetic algorithm tasks (to evaluate whether HQLT's can make use of the advantageous computational expressivity of FW-memory to remediate K KV-memory's 's deficiency; Sec. 4.2), and retrieval intensive tasks (to test whether HQLTs can leverage the precise recall ability of KV-memory which is lacking in FW-memory; Sec. 4.3). Further experimental details, including training hyper-parameter and detailed task descriptions, can be found in Appendix A.
4.1 Evaluating general language modeling capabilities
We begin with evaluating general language modeling capabilities of the proposed HQLT models as a general sanity check. We train language models with either 340M or 1.3B trainable parameters (using sequence lengths of 2048 and 2240, respectively, unless otherwise noted) from scratch on 15B tokens of the HuggingFace Fine
of the recent work on L.Ts/DeltaNe [28, 29, 7] and their most recent recommendations available on the flame repository [24], including the choice of tokenizer (fla-hub/ howph//tttrsfome--11111 ormer -.3B-- 3---100B) 100B). The subword-um -nnt vocabulary size is 32K for all models. All models have 24 layers. The baseline Transformer architecture is from [38] (denoted as Transformer++, following prior convention) and the DeltaNet configuration is from [29]; both models/configurations are used in prior work [29, 7J. Note, however, that for fair comparison with the baseline quadratic transformer, we do not use windtw convolution in the DeltaNet, unlike in recent work [29].
dataset [20]. These choices mostly follow the
We evaluate the trained models through two perplexity evaluation settings on WikiText-2 [39] (Wiki.) and LAMBADA (LMB.) [40], and six zero-shot common sense reasoning tasks: PiQA [41], HellaSwag (Hella.) [42], WinoGrande [43] (Wino.), ARC-easy (ARC-e) and ARC-challenge (Arc-c) [44]. The choice of these tasks also follow the common settings in prior work [28, 29, 7]. We use the standard 1m- evaluation con--aa harness [19] for these evaluation runs.

---

Table 2: General language modeling experiments for Hybrid Quadratic-Linent Transformers (HQLTs). Both the 340M/1.3B models are trained for 15B tokens. Unless otherwise indicated ablation studies, the window size for KV KV-memory is 64 tokens, and the dynamic vector mixing is used to combine is used to produce the results.
K KV-memory and FW-memory in HQLTs. Am-evalluttion-haraaee
Model
Wiki. LMB.
LMB. acc
PIQA acc
ARC-e ARC-c Avg. acc n ace acc f acc n
Hella.
Wino.
ppl 4 ppl
340M params
Transfomer++
26.5
34,9
33.9
67.6
41.0
53.7
60.2
29.0
47.6
DeltaNet
27.6
35.0
32.8
67.1
40.8
52.6
58.5
28.8
46.8
HQLT
Delayed-Strent
26.4
33.1
33.6
67.9
42.1
51.8
29.5
Delayed-Chunk
26.7
29.9
33.5
66.8
42.3
50.9
61.1
30.6
47.5
Synchronous
26.3
29.4
33.3
66.2
42.7
53.8
61.5
29.4
47.8
1.3B params
Transformer++
19.8
17.9
42.6
71.0
50.3
55.8
65.2
33.2
53.0
DeltaNet
20.6
19.9
39.3
70.1
49.5
52.5
68.5
34.2
52.3
HQLT
Delayed-Stream
20.0
16.5
43.5
70.7
51.6
56.0
69.3
36.0
54.5
Delayed-Chunk
20.2
16.3
41.3
71.8
50.9
55.0
67.9
35,2
53.7
Synchronous
19.8
15.9
42.8
72.0
51.5
56.1
68.1
33.1
53.9
Ablations with 340M params HQLT Synchronous
sum mixing
27.7
34.8
32.5
67.0
41.2
52.4
60.9
28.9
47.2
dynamic vector mixing (25M)
26.3
29.4
33.3
66.2
42.7
53.8
61.5
29.4
47.8
w. Linear Atin. (no DeltaNet)
33.3
114.2
21.0
63.2
36.4
51.7
26.8
window x2 = 128
26.6
27.8
35.6
68.1
41.7
51.2
61.4
30.1
48.0
window x4 = 256
26.7
27.7
35.4
67.0
42.2
52.7
59.8
29,0
47.7
window x8 512
27.0
28.0
35.9
66.3
41.3
53.2
60.1
29.0
47.6
Results. Table 2 shows the results. We first observe that the two baselines, the quadratic transformer (Transformer++) and DeltaNet perform comparably at both 340M and 1.3B scales on this general evaluation setting; DeltaNet only lags behind slightly. Consequently, models to also perform similarly on average over the six common sense reasoning tasks, with a slight improvement DeltaNet (up 1% absolute average) which roughly matches the transformer performance. One task where we observe significant improvements by an HQLT is the LAMBADA perplexity evaluation (second column; LMB. ppl): the best HQLT model, the Synchronous variant, achieves about 15% relative improvements over both transformer and DeltaNet baselines, in both 340M and L.3B-parameter cases. Interestingly in the 1.3B setting, HQLT Delayed-Streant produces the best results on ARC tasks, achieving the best average performace over tasks. Neverthless, we find all HQLT's perform favorably compared to the transformer and DeltaNet baselines in the 1.3B case.
also observe all the hybrid
Ablation studies. Table (bottom part) also provides several ablation studies on HQLT Synchronous. First of all, we confirm that the choice of FW-memory type matters: replacing DeltaNet with the vanilla linear attention [3] (Linear Attn.) yields large performance drop (large average performance drop from 47.8 42.2; and large increase in perplexities,
LMB. in particular). Second, KV- memory's 's window size (increased from 64, up to 512) has minimal impacts on the general language modeling performance of HQLT. Finally, we also ablate the type of memory mixing strategies: with the exception of the LMB. perplexity evaluation, where dynamic vector mixing outperforms naive sum mixing, the overall performance is very similar these general evaluation tasks.
4.2 Evaluating Expressivity
Here we compare HQLT variants' abilities to leverage expressivity advantages of the WW-memory component, DeltaNet. We evaluate models on two popular regular language recognition tasks, parity and modular arithmetic without brackets, which are well-known tasks that quadratic transformers and other linear recurrent models fail (due to lack of expressivity), while more expressive, DeltaNet performs well [7]. We mainly follow Grazzi al. [7]'s experimental settings: we use 2-layer and 3-layer models for parity and modular arithmetic, respectively; training and test sequences have

---

lengths from 3 to 40, and from 40 to 256, respectively. The window size for all HQLTs is set to 8 (except for Delayed-Chiee Chunk which uses 16). We refer Appendix A.2 for further experimental details.
Table 3 shows the results. As expected, the two "delayed" variants, Delayed-Stream and Delayed- Chunk, fail on these tasks just like the quadratic transformer. In these "delayed" variations, FW-memory has a delay of a few time steps to access an input, corresponding to the quadratic attention window size; this could be too late to leverage FW-memory's capabilities for these tasks requiring State-tracking in "real time". In contrast, Synchronous HQLT has no such limitation and successfully solves these tasks.
Table 3:
Evaluating Expressivity of Hybrid Quadratic-Lineat Transformers (HQLTs) using reg- ular language recognition tasks: Parity and Modu- lar Arithmetic without brackets (Mod Arith). The top-block results are taken from [7]. We show normalized accuracies [%] (0% is chance level).
Model
Parity acc
Mod Arith
acc
Transformer [7]
2.2
3.1
Mamba [7]
100.0
24.1
DeltaNet [7]
100.0
97.1
As an extra ablation, confirm that the choice of FW-memory type is important: as expected, even the Synchronous version fails these tasks, if vanilla LA (Linear Attn.), which has no known expressivity advantage over softmax attention. is used as the FW-memory module instead of DeltaNet.
HQLT
Gelayed-Streant
3.3
27.8
Delayed-Chuni
2.8
1.4
Synchronous
100.0
97.0
w. Linear Attn.
2.5
44.5
Given that the delayed versions have no other significant advantage over the synchronous version (apart from the conceptual elegance in introducing explicit division of labors over time), we consider this result as a strong argument to prefer the Synchronous variant over other blending strategies for HQLT. This also allows us to conclude that the design of "Infini-attention" from prior work [22], which makes use of the Delayed-Chunk strategy, is sub-optimal in light of expressivity.
4.3 Evaluating in-context retrieval abilities
Here evaluate HQLT's perfor- Table 4: Evaluating Hybrid Quadratic-Linear Transformers mance on recall-intensive tasks. Fol-
(HQLTs) retrieval-intensive tasks. Both the 340M/1.3B models are trained for 15B tokens. The default window size
lowing prior work [29, 45], we
focus on three tasks: FDA [45],
for KV-memory is 64 in HQLTs. Dynamic mixers (scalar
SWDE [46], and SQuAD [47] tasks
and vector) add 0.4M and 25M parameters, respectively.
on which we observe large perfor- mance gaps between the two baselines, quadratic transformer and DeltaNet. Results on these datasets are notori- ously known to be sensitive to the evaluation protocol; following the recommendations from prior work [6]. we use the original evaluation script from [45] for FDA (instead of An--vvaluttion-areee Taking into account the conclusions from the experiments above (Sec. 4.1
Window SWDE SQuAD FDA Avg. acc
size
acc
acc 1
340M params Transformer++ Transformer++ DeltaNet
2048
44.9
36.9
52.3
44.7
1024
30.4
25.5
31.2
29.0
18.5
25.2
8.6
17.4
HQLT Synchronous
sum mixer
64
13.3
26.2
12.6
17.4
[19]).
dynamic sealar
64
21.1
27.7
11.4
20.1
dynamic vector
64
20.0
28.4
10.9
19.8
window x2
128
16.9
30.5
17.9
21.8
and 4.2), focus studying HQLT Synchronous. We additionally focus
window x4
256
18.6
35.6
15.1
23.1
window
512
22.9
35.7
17.3
25.3
the compute-buddee friendly 340M- parameter setting for this study (we provide single run for the 1.3B- parameter HQLT with a window size of 64 only).
window x x16
1024
34.0
37.1
50.1
40.4
1.3B params
Transformer++
2048
53.7
41.5
64.7
53.3
DeltaNet
32.9
29.9
23.6
28.8
HQLT Synchronous
64
31.9
31.2
30.2
31.1
Table shows results. We first observe that the type of memory output mixer plays significant role here (second block from top). The mixer tends to underperform other mixing strategies on SWDE, while both the simple dynamic scalar mixing and dynamic vector mixing strategies perform similarly; ablation on the window size (below) is conducted with the dynamic vector mixing strategy.

---

B
C
Phase
380
1.0
Transformer
DetiaNet
300
HOLT
Sucenss Rate
0.8
Phase
200
150
Transforer
0.2
50
HQLT
Phase
0.0
0.00 0.25 0.50
0.75
1.00 1.25 1.50 1.75 2:00 Env Stops
Stops
Figure 2: Evaluation of the Synchronous Hybrid Quadratic-Linent Transformer (HQLT) on reinforce- ment learning in partially observable environments, using "passive visual match" task [48, 49]. A: In this task, an agent (the beige pixel) navigates in 2D grid world (of size 7 11) delimited by imper- 5-grid centered on the agent;
meable walls (black). The agent can only observe the nearby pixels illustrated by the light-blue boxes). An episode in this task has three phases. During Phase (whose duration 15 time steps), the agent observes color, randomly drawn from three choices, red, green or blue (here blue). In Phase 2 (750 steps), the agent is in a room with apples (green); collecting an apple yields reward of 1. There are initially 10 apples, and they reappear every 20 steps; their positions are random. In Phase 3 (max. 15 steps), if the agent reaches the pixel with the color that matches the one provided in Phase the episode ends successfully; it yields a reward 100. Alternatively, Phase terminates if the agent reaches pixel with the wrong color or when the limit of 15 steps is reached (no reward is given these cases). B and show the average return and success rate over 20 test episodes as function of training environment steps, respectively. We show the average and 95% confidence intervals computed using three training seeds. The variation in success rate is high for Transformer, as one of the seeds consistently achieved the 100% success rate after certain training steps, while other seeds did not. Similarly for HQLT, one of the seeds consistently achieved above 70%.
Next, expected, the window size is important parameter for the performance of HQLT these tasks overall. However, breaking down the performance on the task level, there are mixed results. For example, increasing the window size from 64 to 128 yields large degradation on SWDE (from 20.0 down to 16.9), while increasing it up to 512 yields an effective improvement (SWDE accuracy going up 22.9). Similarly, increasing the window size from 64 to 128 yields large improvement FDA (from 10.9 to 17.9) but increasing it further to 256 yields degradation (down to 15.1). These obser- vations confirm previously reported sensibility of these tasks [6]. Nevertheless, overall, observe gradual performance increase on average by increasing the window size from the default value of 64 to 512, and finally up 1024. Another interesting comparison is between HQLT with window size of 1024 and the baseline transformer trained with the same window size (second row); we obtain a large improvement (40.4 vs. 29.0) showing the benefit of extra FW-memory that covers a larger context, even though retrieval is not the main strength of FW-memory. Finally, by increasing the parameter count 1.3B, HQLT with tiny window size of 64 still yields large improvement over the standalone DeltaNet on FDA. The compute requirements for HQLTs are discussed in Appendix A. 1 and B.2.
4.4 Evaluation beyond the language domain: Reinforcement learning in POMDPs
To complete the set of experiments above focused on the language domains, here we evaluate HQLT on reinforcement learning (RL.) tasks in partially observable environments (POMDPs) [50]. We use the "passive visual match" task [48, 49]. whose illustration and description can be found in Figure 2A and its caption. This task is representative example of memory retrieval with distractions-the task of remembering the initial color after the sub-task of collecting apples in partially observable room (both tasks require some short-term memory), which is reminiscent of real-life situations, such as remembering where the car is parked after grocery shopping or engaging in a conversation with friend: these activities act distractions that interfere with recalling the parking location.
The sequence length of an episode is 780 steps (15, 750, and steps for each phase, respectively). We compare Transformer (with the full context size of 780), DeltaNet, and Sychronous HQLT (with a short quadratic attention window size of 64). Following et al. [49], we use the hidden size of 100 for all models (we use attention heads). We tested different numbers of layers from to

---

for all models; we found 2 to be the most stable for Transformer, while 3 layers worked best for DeltaNet and HQLT. We train all models using the soft-actor critic method for discrete action space [51, 52] (we have four actions, up/down//eft/right, for navigation). Further experimental details can be found in Appendix A.3. Figures B and C show the results. We observe that the Transformer with softmax attention outperforms DeltaNet on this retrieval-focused task, while HQLT largely closes this performance gap using a short quadratic attention window of only 64.
5 Related Work and Discussion
Related work. The idea of hybridizing heterogeneous models into a single network is not uncommon [6, 53-61J. A popular approach is the layer-wise combination strategy, which use different model types in different layers in deep neural network. For example, xLSTM [55] makes use of two types of memory layers, mLSTM and SLSTM; mLSTM is a gated variant of LT/FWP [3, 4], enabling parallel training over sequence elements, while sLSTM enables expressive recurrent computation (as it retains the structure of the classic LSTM [62], with the only distinction being the block-diagone recurrent structure arising from the multi-head architecture). Other notable examples are Griffin [53] and Samba [57] which interleave gated linear recurrent layers [63-66] and local attention layers.
While such layer-wise hybrid methods may be natural solution combine memory systems that are structurally very different, e.g.. recurrent neural networks and transformer variants, quadratic and linear transformers offer unique opportunity for blending them a lower leve]-within single layer as an unified memory system, as they share the main variables (namely, key, value, and query vectors) used in their respective computation. Such an opportunity has been previously explored [45, 22] (both of which we related to our models in Sec. 3). In particular, the focus of Arora et al. [45] is on the stal--throughhut trade-off- which alone does not cover the full aspects of complementarity between QT and LT, as we' ve seen; and also crucially, their choice of FW- memoryy is the vanilla linear attention, whose perform is well-known to lag largely behind the softmax attention (as we also see in our ablation studies replacing DeltaNet with vanilla LA); in contrast, we consider recent advances in LT/FWPs which have developed more competitive LT models, including DeltaNet. Munkhdalai et [22] use a form of delta-rule in their FW WW-memory component; however, recent findings that enhance the delta rule in FWPs [5, 29] seem overlooked (e.g., careful choice of é, omission of the extra normalizer; use of dynamic learning rate Be), which makes it difficult to conclude on the true potential of HQLT. Finally, Munkhdalai et al. [22] use the "Delayed-Chume Chunkk" hybrid scheme, which incompatible with the idea of leveraging FW-memory's 's expressivity (Sec. 4.2). In this sense, our work offer updated and unique discussions on designing a hybrid attention model which cannot be found in any prior work.
Limitations. While this work has shown generally promising results in combining quadratic and linear transformers, the results on the retrieval tasks are still unsatisfactory (Sec. 4.3): seems that, a large K KV-memory window is unavoidable for precise retrieval (even though there was non- zero chance that with enough layers/depth, even short window could yield precise long-range retrieval). One potential solution to remediate this may be to introduce a mechanism for more complex communication between KV-memory and FW-memory, which FW-memory selectively revives certain memory events and reinserts them into the limited K KV-memory-an as opposed to the current design of sliding window attention in which KV-memory can only operate on the recent past. Investigating such a mechanism is not straightforward in the current era of model development, which sets hardware-efficient as a hard requirement. Nevertheless, we believe that developing such alternative memory architectures is an exciting direction for future work.
6 Conclusion
The quadratic transformer (K (KV-memory system) and linear transformer (FW-memory system) are two complementary memory systems with unique advantages. We develop method to blend these two systems into a single hybrid memory system to leverage the strengths of both. By taking into account various types of complementarity between these two systems complexity, context length, retrieval precision and expressivity, we empirically evaluate our models on general language modeling, retrieval, and synthetic algorithmic tasks and reinforcement learning settings, to finally conclude that the best overall hybrid method is the one that allows K V-mmeoory and FW-memory to operate synchronously, as opposed to alternatives that introduce division of labor over time. This offers a novel perspective on the development of hybrid memory systems for sequence processing.
10

---

Acknowledgments
The authors are grateful for support from the Kempner Institute for the Study Natural and Artificial Intelligence, and from the Department of Defense MURI program under ARO grant W91INF--3-J- 0277. Kazuki Irie thanks Songlin Yang for a helpful discussion and recommendations regarding the fla [24] and flame [67] frameworks.
References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. Advances in Neural Information Processing Systems (NIPS), pages 5998-6008, Long Beach, CA, USA, December 2017.
[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Int. Conf. on Learning Representations (ICIR), San Diego, CA, USA, May 2015.
[3] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. Proc. Int. Conf. on Machine Learning (ICML), Virtual only, July 2020.
[4] Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139, 1992.
[5] Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear Transformers are secretly fast weight programmers. In Proc. Int. Conf. on Machine Learning (ICML), Virtual only, July 2021.
[6] Songlin Yang. Jan Kautz, and Ali Hatamizadch. Gated delta networks: Improving Mamba2 with delta rule. In Int. Conf. Learning Representations (ICLR), Vancouver, Canada, April 2025.
[7] Riccardo Grazzi, Julien Siems, Jörg KH Franke, Arber Zela, Frank Hutter, and Massimiliano Pontil. Unlocking state-tracking in linear RNNs through negative eigenvalues. In Int. Conf. on Learning Representations (ICLR), Vancouver, Canada, April 2025.
[8] Julien Siems, Timur Carstensen, Arber Zela, Frank Hutter, Massimiliano Pontil, and Riceardo Grazzi. Deltaproduct: Improving state- tracking in linear RNNs via householder products. Preprint arXiv:2502.11223 2025.
[9] Yash Raj Sarrof, Yana Veitsman, and Michael Hahn. The expressive capacity of state space models: A formal language perspective. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Vancouver, Canada, December 2024.
[10) Kazuki Irie, Róbert Csordás, and Jürgen Schmidhuber. Practical computational power of linear transformers and their recurrent and self-referential extensions. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), Sentosa, Singapore, 2023.
[11] Michael Hahn. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics, 8:156-171 101,,1111000. 2020.
[12] Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal. On the ability and limitations of transform- ers to recognize formal languages. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 7096-7116. Virtual only, November 2020.
[13] William Merrill and Ashish Sabharwal. The parallelism tradeoff: Limitations of log-precision transformers. Transactions of the Association for Computational Linguistics (TACL., 11: 531-545, 2023.
[14] William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state state-space models. In Proc. Int. Conf. Machine Learning (ICML), Vienna, Austria, July 2024.

---

[15] Lena Strobl, William Merrill, Gail Weiss, David Chiang, and Dana Angluin. What formal lan- guages can transformers express? a survey. Transactions of the Association for Computational Linguistics (TACL., 12:543-561, 2024.
[16] Juno Kim and Taiji Suzuki. Transformers provably solve parity efficiently with chain of thought. In Int. Conf. Learning Representations (ICLR), Vancouver, Canada, April 2025.
[17] James L McClelland, Bruce L McNaughton, and Randall C O''Relllly Why there are comple- mentary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
[18] Randall C O'Reellly and Kenneth A Norman. Hippocampal and neocortical contributions to memory: Advances in the complementary learning systems framework. Trends in cognitive sciences, 6112):005-510, 2002.
[19] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf. Aviya Skowron, Lintang Sutawika, Eric Tang. Anish Thite, Ben Wang, Kevin Wang. and Andy Zou. The language model evaluation harness, 07 2024. URL
wtg/records/ htte//rccords/226660000 12608602.
[20] Guilherme Penedo, Hynek Kydlícek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin A. Raffel, Leandro von Werra, and Thomas Wolf. The fineWeb datasets: Decanting the web for the finest text data at scale. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Vancouver, Canada, December 2024.
[21] Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall- throughput tradeoff. In Proc. Int. Conf. on Machine Learning (ICML), Vienna, Austria, July 2024. OpenReview.net.
[22] Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave context behind: Efficient infinite context transformers with infini-attention. Preprint 20X1:22404.07143, 2024.
[23] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. Preprint arXiv:2307.08661 2023.
[24] Songlin Yang and Yu Zhang. Fla: triton-based library for harddwrre-ffficien implementa- tions of linear attention mechanism, January 2024. URL https: /tttt::://gthiii htt///gtttuub..../ com/flam.. org/
wiinerr-atentt..
[25] Jimmy Ba, Geoffrey E Hinton, Volodymyr Mnih, Joel Z Leibo, and Catalin Ionescu. Using fast weights to attend to the recent past. In Proc. Advances in Neural Information Processing Systems (NIPS), pages 4331-4339, Barcelona, Spain, December 2016.
[26] Mark A. Aizerman, Emmanuil M. Braverman, and Lev I. Rozonoer. Thcoretical foundations of potential function method in pattern recognition. Automation and Remote Control, 25(6): 917-936, 1964.
[27] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. Preprint arXiv:2307.086211 2023.
[28] Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. In Proc. Int. Conf. on Machine Learning (ICML), Vienna, Austria, July 2024.
[29] Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Vancouver, Canada, December 2024.
12

---

[30] Imanol Schlag, Tsendsuren Munkhdalai, and Jürgen Schmidhuber. Learning associative infer- ence using fast weight memory. In Int. Conf. on Learning Representations (ICLR), Virtual only, May 2021.
[31] Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoe V. Le. Transformer quality in linear time, In Proc. Int. Conf. on Machine Learning (ICML), Baltimore, MA, USA, July 2022.
[32] Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence, 46(1-2):159-22 1990.
[33] Tsendsuren Munkhdalai, Alessandro Sordoni, Tong Wang, and Adam Trischler. Metalearned neural memory. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pages 13310-13321, Vancouver, Canada, December 2019.
[34] Bernard Widrow and Marcian E Hoff. Adaptive switching circuits. In Proc. IRE WESCON Convention Record, pages 104, Los Angeles, CA, USA, August 1960.
[35] Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. Going beyond linear transformers with recurrent fast weight programmers. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Virtual only, December 2021.
[36] Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. A modern self-referential weight matrix that learns to modify itself. In Proc. Int. Conf. on Machine Learning (ICML), pages 9660-9677 Baltimore, MA, USA, July 2022.
[37] Mude Hui, Rui-Jie Zhu, Songlin Yang. Yu Zhang, Zirui Wang. Yuyin Zhou, Jason Eshraghian, and Cihang Xie. ARFlow: Autogressive flow with hybrid linear attention. Preprint arXiv:2501.1608) 2025.
[38] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Preprint arXiv:2307.099: 099889888 2023.
[39] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In Int. Conf. on Learning Representations (ICLR), Toulon, France, April 2017.
[40] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proc. Association for Compu- lational Linguistics (ACL), Berlin, Germany, August 2016.
[41] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning
about physical commonsense in natural language. In Proc. AAAI Conf. on Artificial Intelligence, New York, NY, USA, February 2020.
[42] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proc. Association for Computational Linguistics (ACL), Florence, Italy, August 2019.
[43] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proc. AAAI Conf. on Artificial Intelligence, New York, NY, USA, February 2020.
[44] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the A12 reasoning challenge. Preprini arXiv: 1803005437, 2018.
[45] Simran Arora, Brandon Yang. Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trummer, and Christopher Ré. Language models enable simple systems for generating structured views of heterogeneous data lakes. Preprint MrXiv:2304.09433 2023.
[46] Colin Lockard, Prashant Shiralkar, and Xin Luna Dong. Openceres: When open information extraction meets the stri-strue web. In Proc. North American Chapter of the Association for Computational Linguistics on Human Language Technologies (HITCI-HILT), Minneapolis,
MN, USA, June 2019.
13

---

[47] Pranav Rajpurkar, Robin Jia, and Perey Liang. Know what you don't know: Unanswerable questions for SQuAD. In Proc. Association for Computational Linguisties (ACL), Melboume, Australia, July 2018.
[48]
Chia-Chun Hung, Timothy Lillicrap. Josh Abramson, Yan Wu, Mehdi Mirza, Federico Carnevale, Arun Ahuja, and Greg Wayne. Optimizing agent behavior over long time scales by transporting value. Nature communications, 10(1), 2019.
[49] Tianwei Ni, Michel Ma, Benjamin Eysenbach, and Pierre-Luc Bacon. When do transformers shine in RL? Decoupling memory from credit assignment. In Proc. Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA, December 2023.
[50] Leslie Pack Kaelbling, Michael I. Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence. 101(1-2):99-13, 1998.
[51] Petros Christodoulou. Soft actor-critic for discrete action settings. Preprint arXiv: 1XXxx:910000207. 2019.
[52] Tuomas Haamooj,, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. In Proc. Int. Conf. Machine Learning (ICML., Stockholm, Sweden, July 2018.
[53] Soham De, Samuel I. Smith, Anushan Fernando, Aleksandar Botev, George Pristian-Muuaru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models. Preprint arXiv:2402. 19427, 2024.
[54] Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Proc. Int. Conf. on Machine Learning (ICML), Vienna, Austria, July 2024.
[55] Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xLSTM: Extended long short-term memory. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Vancouver, Canada, December 2024.
[56] Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh. Deepak Narayanan, et al. An empirical study of mamba-based based language models. Preprint arXiv:2406.0753 2024.
[57] Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling. Preprint arXiv:2406.07522. 2024.
[58] Luca Zancato, Arjun Seshadri, Yonatan Dukler, Aditya Golatkar, Yantao Shen, Benjamin Bowman, Matthew Trager, Alessandro Achille, and Stefano Soatto. B'MOJO: Hybrid state space realizations of foundation models with eidetic and fading memory. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Vancouver, Canada, December 2024.
[59] Elvis Nunez, Luca Zancato, Benjamin Bowman, Aditya Golatkar, Wei Xia, and Stefano Soatto. Expansion span: Combining fading memory and retrieval in hybrid state space models. Preprint arXiv:2412.13328. 2024.
[60] Xin Dong. Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sunil Mahabalesh- warkar, Shih-Yang Liu, Matthijs Van Keirsbilck, Min-Hung Chen, Yoshi Suhara, et al. Hymba: A hebrid-head architecture for small language models. Preprint 2rXiv:2411.13677 2024.
[61] Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, al. Hybrid computing using neural network with dynamic external memory. Nature, 538
(7626):471-476 2016.
14

---

[62] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()): 1735-1780, 1997.
[63] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, In Conference Language Modeling (COLM), 2024.
[64] Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for sequence modeling. In Proc. Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA, December 2023.
[65] James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. Quasi-recurrent neural networks. In Int. Conf. on Learning Representations (ICLR), Toulon, France, April 2017.
[66] Tao Lei, Yu Zhang, Sida I. Wang, Hui Dai, and Yoav Artzi. Simple recurrent units for highly parallelizable recurrence. Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLPP, pages 1470-44811 Brussels, Belgium, November 2018.
[67] Yu Zhang and Songlin Yang. Flame: Flash language modeling made easy, January 2025, URL https://gitte git.g.th........ com fla- ooaoorggggllool flame.
[68] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Nearocomputing, 568, 2024.
[69] Hector J. Levesque. The winograd schema challenge. In Logical Formalizations of Common- sense Reasoning, AAAI Spring Symposium, Technical Report SS-11-06, Stanford, CA, USA, March 2011.
[70] Qiang Hao, Rui Cai,, Yanwei Pang, and Lei Zhang. From one tree to a forest: a unified solution for structured web data extraction. In Proc. of International Conference on Research and Development in Information Retrieval (SIGIR), Beijing, China, July 2011.
[71] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proc. Conf. Empirical Methods in Natural Language Processing (EMNLP), Austin, TX, USA, November 2016.
[72] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Int. Conf. on Learning Representations (ICLR), San Diego, CA, USA, May 2015.
[73] Kazuki Irie, Albert Zeyer. Ralf Schlüter, and Hermann Ney. Language modeling with deep Transformers. In Proc. Interspeech, pages
Graz, Austria, September 2019.
[74] Kazuki Irie. Why are positional encodings nonessential for deep autoregressive transformers? Revisiting a petroglyph. In Proc. Findings of Association for Computational Linguistics (ACL- Findings), Vienna, Austria, July 2025.
[75] Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information. In Findings of Conf. on Empirical Methods in Natural Language Processing DEMNLP-Findinge. pages 1390-1390, Abu Dhabi, United Arab Emirates, December 2022.
[76] Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. In Proc. Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA,
December 2023.
15

---

A Experimental details
A.1 Language modeling experiments
Here we provide details of the language modeling experiments presented in Sec. 4.1 and 4.3.
Training details. Table 5 shows the training and model hyper-parametest used to train the 340M and 1.3B parameter models. The Ke-memory component of HQLTs use the RoPE positional encodings [68]; Appendix B.3 provides the corresponding ablation study. We train with an effective batch size of 64 per GPU with a sequence length of 2048, using 4 GPUs, for 28,672 steps; this yields 55,032,385,55 386,536 tokens. The 1.3B models were trained with a slightly increased sequence length of 2240; this increases the training token count to 16,441,67 66800 (for simplicity, in the main text, we refer to both of them as trained for 15B tokens) Note that the actual parameter counts of 340M models are about 370M; we follow this convention from prior work [29, 6, 28, 7] (likely related to the fact that if the models shared the input and output embedding matrices of about 30M parameters, they would have 340M parameters; but this is not the case, either in work, nor in the prior work).
Training time. Training of 340M models using 4 H100-80GB GPUs take about hours for the baseline transformer and 10 hours for DeltaNet and all the HQLT models with the window size from 1024 tokens. For the 1.3B models, these numbers become 26 hours for the baseline transformer and DeltaNet, and 30 hours for all the HQLT's variants. Given that the training for the 1.3B models uses 16B tokens, training throughputs are 170K tokens/second and 148K tokens/second for the baseline Transformer/DeitaN and HQLT, respectively.
We the fla [24] toolkit implement the models, and flame [67] to train them.
Table 5: Hyper-parameters of language models.
Model
340M
1.3B
Number of layers
24
Feedforward block multiplier
4
Total hidden size
1024
2048
Number of heads
8
16
Sequence length
2048
2240
Effective Batch size
64
Learning rate
le
Warmup steps
1024
Minimum learning rate
0.1
Max norm clipping Std. of weight initializers
1.0
0.02
Implementation details. Both the Synchronous and Delayed-Streest Stream variants of HQLT studied in this work can directly make use of flash-attention [23] and mash-linear-attentio [24] implementations (without modifying the corresponding Triton kernels) for the quadratic and linear components the models, respectively. For Delayed-Bloe Block HQLT, we wrote custom Triton kernel to replace intra- attention in the DeltaNet implementation of [29] by an efficient softmax attention implementation [23], and modified the backward function accordingly. Note, however, that there is still room for optimization as the intra and inter-chunk operations are currently implemented in two separate kernels (they may potentially be fused into single kernel for further speed optimization).
Evaluation Details. Here we provide further descriptions of the evaluation datasets used in our general language modeling and retrieval tasks.
The six zero-shot common sense reasoning tasks we used in Table 2 are as follows. LAMBADA (LMB.) [40] is task of predicting the last word in sentence following some context sentences.
PiQA [41] and HellaSwag (Hella.) [42] evaluate models' common sense knowledge (learned in weights) through question answering with multiple choices. WinoGrande [43] (Wino.), inspired by the Winograd Schema Challenge [69], is a set of pronoun-resolution problems. The ARC dataset
[44] is set of grade school-level questions about natural science, which is split into two subsets,
16

---

eac- (ARC-e) and ARC-challenge (Are-c), according to their difficulties (determined based on whether certain bascline models can solve it). We use the standard 1m- im-evaluation -aallaation--aresss [19] for evaluation on these datasets.
In Table 4, use three additional datasets evaluate models' in-context retrieval abilities. SWDE is an information retrieval task based on the Structured Web Data Extraction dataset [70] (e.g., extracting some priect--redicat-objjee information from a raw HTML webpage about movie). Similarly, FDA is an information retrieval task, extracting some key-value pairs from set of PDFs from the FDA website. SQuAD [47] is the latest version of the Stanford Question Answering Dataset [71] which a set of reading comprehension problems, which evaluate models' ability to answer question based on provided text passage, For SQuAD and SWDE, we use Im-evaluation wwwallationnaareess [19] for evaluation but by removing leading and trailing spacing in the document (see a related note at https: //gi thub. com//EleutherrA/Ime
For FDA, we use the com htt.:///aaaarrrppeaieeeeeeeeeere
original script from https: main/ 1mm eval httaa/eee//rmppppccccppse//e/eee http/mm-evalllee dation by Yang et al. [6] provided at http:///tt......oo/////aaaaaaaaa readme- fvillee).
hf sh (following the recommen-
The choice of all these tasks follow prior work [28, 29, 7].
A.2 Synthetic algorithmic tasks
Here we provide details about the experiments with two regular languages presented in Sec. 4.2. All the basic settings follow those of Grazzi et al. [7].
Tasks. In "Parity", an input is a sequence of zeros and ones, and the task is to determine whether the number of ones in the sequence is odd even. This essentially corresponds to modulo 2 addition, with the chance-level accuracy of 50%.
In "Modular Arithmetic (Mod Arith)", , the task is modulo 5 addition and multiplication task (without brackets). Each symbol in an input sequence is either a number (from 0 to 4, in the module case, which is our setting), or a mathematical symbol (there are five of those: {+. -, *, =, eoos where the last symbol is the extra "end-of-sequence" token; which is not necessary for the modulo 5 case but included by convention). Here not only the output, but also the numbers in the sequences, are drawn between 0 to 4. This makes the total vocabulary size of 10 with a chance level accuracy of 20%.
Model configuration. The model hidden size is set to 128 and the number of heads is 4. For HQLTs, we use the dynamic vector mixing strategy, and the chunk size is set to 8 (except for the Delayed-Chunk variant, set it to 16 for an implementation reason). We use 2 layers for Parity and 3 layers for Modular Arithmetic. Naturally, the crucial factor 2 is applied after the sigmoid activation on the dynamic learning rate Bg in DeltaNet (Eq. 8) enhance its stteettacking ability [7].
Training settings. We train with a batch size of 1024 for 20,000 steps. We search for the best learning rate among (5e -3, le -3- 3, 5e le=1) (the only difference compared to Grazzi et al. [7] is that this list includes 5e- instead of le -2), each with three seeds. We directly measure the validation accuracy to determine the best configuration. In the main result table, Table 3, we report the best/max result among the seeds for the best configuration, while Table 6 shows variability among seeds. This is a standard practice in formal language recognition tasks [12, 10]. We train with sequence lengths from 40, and validate sequences of lengths from 40 to 256. Each training run single H100 takes about 70 min.
Evaluation. We report "normalized accuracies", that is, by denoting the raw accuracy as Anaw and the chance level accuracy as Aphonce, we report (Arrm - Achhnce) 50% for parity and 20% for modular arithmetic (modulo 5), such that all the accuracies are scaled to be between 0 and 100, where 0 is chance-level and 100 is perfect accuracy.
- Aehance), ) where Achance is
A.3 Reinforcement learning experiments
Here we provide experimental details of the RI experiments presented in See. 4.4. Our main settings follow Ni et al. [49]'s (our implementation is based on their code base), which implement the soft-
17

---

Table Median accuracy and median absolute deviation over 3 seeds using the best learning rate for each case. This is to complete Table which shows the best/max result among the seeds. The top-block results are taken from [7]. We show normalized accuracies [%] (0% is chance level).
Model
Parity acc f
Mod Arith
acc
Transformer [7]
0.3 = 1.3
1.8 + 0.9
Mamba [7]
100.0 0.0
21.4
2.7
DeltaNet [7]
99.9 = 0.6
82.6 + 14.6
HQLT
Delayed-Stream
3.0 - 0.3
22.2
5.6
Delayed-Chunk
2.3 - 0.1
1.4 +
0.1
Synchronous
99.7 0.1
93.2 = 3,2
w. Linear Attn.
2.1 = 0.3
32.9
11.6
actor critic method for discrete action space [51, 52). Regarding the environment, we set a reward of 100 for successful episode, i.e., when the agent successfully reaches the correct color in Phase 3, instead of 10 in Ni al. [49].
An observation (a 3 x 5 x 5 image) is first processed by a 2-layer convolutional neural network to produce 144-dimensional vector, which is then fed to the sequence model (described in the main text). The output of the sequence model is 100-dimension vector, which is fed to all the policy and value networks; each of which parameterized by a 2-layer feedforward network with a hidden dimension of 256, and their final output dimension is 4, corresponding to the number of actions.
We batch size of 64 and learning rate of 34-1 optimizer [72]. We apply a scale 0.1 on the entropy term in the loss. Importantly, we use dropout with dropping rate of 0.1 inside the sequence models, including
1 for all the system components, using the Adam
the observation embeddings. We keep the dropout active during rollouts to enhance exploration as is done Ni et al. [49] (we found this be important in practice). We refer to the code for any further details.
B Further discussion
B.1 Further clarifications for the Delayed-Chunk variant
Delayed-Streaming HQLT and Delayed-Chunk HQLT are not equivalent (i.e., their outputs are different for the same input) because the classic equivalence between recurrent vs. chunk mode computation in linear attention does not hold when softmax is applied within the intra-chunk attention. To be more specific, the Delayed-Strenn i.e. at every time step, the oldest token from the K KV-memory window is fed to the FW-memory, and a new token enters the KV-memory window; as a consequence, the memory content of FW-memory is updated at every time step, whereas in the Delayed-Chunk variant, the sequence is processed chunk-by bounk-by-chunk, i.c., the content of FW-memory is only updated at the chunk-boundarics and remains constant while the system is processing the current chunk. Within chunk, the content of KV-memory is updated at every time step by adding the new element the chunk; its content is reset (to empty) at the chunk boundaries. Figure 3 provides an illustration. Delayed- -hhnkk HQLT naturally compatible with efficient training: we can apply flash-linear-attee linear-attention for inter-chunk attention computation, and flash-attention to the intra-chunk attention with softmax.
variant has sliding window attention with stride of
HQLT may argued to be a better approach as the KV-memory always makes use of the full window size (except at the very beginning of the sequence), but we also included Delayed-Chuni which is not only a natural extension of chunk-wise parallel form of linear attention, but was used in prior work by Munkhdalai et al. [22].
Intuitively,
18

---

Softmax Attention
Fast weight memory
Key value memory
*
O
Chunk
Chunk
Figure 3: An illustration for the Chunk-Delayed variant of Hybrid Mondratic-Linent Transformers (HQLTs). Here the chunk size 4. KV-memory only has access the tokens within the current chunk (i.e., the last three tokens here), whereas FW-memory does not contain any information about the current chunk.
B.2 Practical computational costs of increasing the window size
In Sec. 4.3/Table 4, we demonstrated how increasing the quadratic attention window size in HQLTs improves their retrieval abilities. Here we discuss the computational costs for such improvements. In practice, within the range of window sizes discussed here, the extra computational cost is negligible, since during inference with a batch size of I, short-window attention typically memory-bound rather than compute-bound and can be efficiently computed in few parallelizable matrix multiplication steps on a GPU, regardless of whether the window size is 64 or 1024. Training time also only minimally affected by the increased window size training is parallelized over the sequence elements. However, the total state size (which has direct impact on the memory requirement) increases as function of the window size S S(din dow)) dout dip per layer and per head.
B.3 Positional Encodings in KV-memory
All the HQLT models presented in the main text use the RoPE positional encodings [68] in KV- memory. However, it is also known that multi-layer self-attention can process sequences without explicit positional encodings [73-75], and such "No Pos" approach has certain generalization benefits [12, 76]. Here we provide an ablation study on the positional encoding in KV-memory.
Table 7: General language modeling experiments for Synchronous HQLT. Both the 340M/1.33 .B models are trained for 15B tokens. "No Pos" indicates no positional encoding in KV-memory. "windoww" specifies the window size for KV-memory, and the dynamic vector mixing is used to combine KV-memory and FW-memory.
Model
Wiki. LMB. LMB. PIQA Hella. ppl
Wino. ARC-e ARC-e Avg. acc
ppl
acc
acc
acc_n"
ace 1 acc_nt 1
340M params
window
RoPE
26.3
29.4
33.3
66.2
42.7
53.8
61.5
29.4
47.8
No Pos
27.3
31.8
32.4
66.2
40.7
51.9
60.3
29.8
46.9
window 1024
RoPE
26.8
34.6
66.7
50.0
28.1
No Pos
27.2
34.5
32.3
66.3
40.2
54.0
57.3
29.4
46.6
1.3B params
window 64 RoPE
19.8
15.9
42.8
72.0
51.5
56.1
68.1
33.1
53.9
No Pos
20.7
17.7
40.4
70.7
50.6
54.5
67.1
33.9
52.9
19

---

Table 8: Evaluating Synchronous HQLT serrivallinnensive tasks. Both the 340M/1.3B models are trained for 15B tokens. "No Pos" indicates no positional encoding in K K--memory. "window" specifies the window size for KV-memory, and the dynamic vector mixing is used to combine K KV-memory and FW-memory.
SWDE SQuAD FDA Avg. acc 1
ace
acc f
340M params
window 64
RoPE
20.0
28.4
10.9
19.8
Pos
17.1
28.4
13.4
window 1024
RoPE
34.0
37.1
50.1
40.4
No Pos
36.5
33.3
55.8
41.9
1.3B paramy
window 64
RoPE
31.9
31,2
30.2
31.1
No Pos
32.7
32.9
26.3
30.6
We compare the Synchronous HQLT models with and without RoPE for the 340M models with window size of or 1024, as well as the 1.3B model with a window size of 64.
Tables 7 and show the results for general language modeling and retrieval tasks, respectively. Regarding the general language modeling tasks, find RoPE be generally beneficial. We observe that RoPE consistently improves the perplexity on Wiki. and LMB. (the two first columns in Table 7); and also on the zero-shot tasks Table 7, with the exception of Wino. and ARC-c in the 1024 window size case, RoPE yields notable improvements. Another interesting observation here is that the model with window size of 1024 slightly worse than the 64-size model these general language tasks; this may be acceptable given the substantial improvements observed in the eetrieval-intensie tasks.
The results to the sensibility of these tasks (as mentioned above). For example, we observe a large improvement FDA by removing the positional encoding in the 340-parameter models (in both the 64- and 1024-window size cases), while this is not the case in the 1.3B model. In fact, in the 340-parameter 1024-window model, "No Pos" outperforms RoPE on both SWDE and FDA, while the trend is reversed on SQuAD. Nevertheless, given the general language modeling results above, using RoPE seems be more robust option overall. Further investigation into the length generalization benefits of "No Pos" HQLT's is beyond the scope of this work.
mixed in the cerrevval-nntesive tasks (Table 8), which is also likely related
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Both in the abstract and the introduction, we accurately describes the scope and objective of the work, as well as the nature of our experiments, and obtained result overview.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have a dedicated paragraph for limitations under Sec. 5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer:
[NA]
Justification: Our work is empirical.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
We will release code and configuration files on public GitHub repository upon acceptance (unpolished codebase is already provided in the supplemental material). Our codebase is based on a popular framework for linear transformers, called flame [67] and fla [24]. which further facilitate exchanges of our experimental praticalities.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
22

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: The training data we used can be easily downloaded from Hugging- http:://huggiggg:: //whggigggaaceee.//o//////.
Face: https:
aaatttastts/ugginggaeettoo ation was also done using the publicly accessible 1m- an-evaluation saraaluuaion-hhheesss [19].
edu. Evalu-
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https: Setpe//ooeSSummmeee
/ //iips.co cats.cccppuu wwtppblcc//aaeeccccccccccccc
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The most important details provided in the main text, and all other experi- mental details can be found in Appendix A in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We have dedicated (sub)sections in Appendix A in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have a dedicated section in Appendix the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://///aaapppp..cpc/b//////// http:///tl..tttiiiiiiicieeeeeee.
Answer:
[Yes]
Justification:
We do foresee any such issue in our work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We do not foresee any such issue to be discussed in our work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not foresee any such issue with our models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we make sure to fully credit them.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. paperswithcode. com/ datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
25

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
We do not produce any new data asset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This is not applicable to our research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This is not applicable to our research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
26

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: This is not applicable to our research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: https///nnuuuueeeeeee httprrpps.cc//oofeeeneee//////// for what should or should not be described.
27


Reviews:
[{"review_id": "id7W1exJHU", "reviewer": "Reviewer_Mb9d", "summary": "This paper proposes a hybrid quadratic-linear architecture that combines two types of memory systems: key-value memory (used in transformers for precise recall but with quadratic complexity) and synaptic memory (used in linear transformers for efficient computation). The authors design and compare three hybrid models—Delayed-Streaming, Delayed-Chunk, and Synchronous—and show that the Synchronous hybrid best leverages the strengths of both memory types. Through experiments on language modeling, expressivity tasks, and in-context retrieval tasks, the paper demonstrates the hybrid model's enhanced performance and scalability.", "questions": "see weakness.", "limitations": "see strength 3.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "strengths\n\n1. In-layer hybrid architecture series validated on linear attention with delta rule. Although incremental, it is still worthwhile to explore the combination of two very different memory mechanisms.\n\n2. Experiments on different hybrid methods on multiple tasks and their detailed ablations.\n\n3. The author's discussion of this work is valuable. The fixed-size memory of linear attention is difficult to accurately handle retrieval tasks, but the infinitely expanded kv memory is not necessarily optimal. Although the author did not provide a sufficiently novel solutions in the article, how to better combine the two types of memory is an important research topic.\n\nweakness\n\n1. The abstract and introduction lack a brief introduction to the method. The authors should have elaborated on the \"Three blending methods\"(i.e. various types of HQLT).\n\n2. I thought that Delayed-Streaming HQLT and Delayed-Chunk HQLT just correspond to the serial and chunk parallel forms in linear attention, and the calculation results are the same. But from the authors' experiment, Delayed-Chunk HQLT is different from Delayed-Streaming HQLT in both calculation process and results, which confuses me. I hope the author can explain why Delayed-Chunk HQLT is a hybrid method independent of Delayed-Streaming HQLT.\n\n3. How does the training efficiency of hybrid architectures compare to linear models and transformers? A supplementary experiment may answer my concerns.\n\n4. The authors mentioned in the limitation that \"Investigating such a mechanism is not straightforward in the current era of model development, which sets hardware-efficiency as a hard requirement.\" I value this point of view of the authors. Recently, Log-linear attention [1] has achieved a balance in exploring the utilization of intermediate-scale memory and computational efficiency. I wonder what the authors think of this work. (This is just an open question)\n\nminors:\n5. The word \"synaptic memory\" used in the abstract is confusing, as the authors does not provide any explanation in the paper. Either explain it in the text, or use FW memory instead, which may be easier to understand.\n6. Some HQLTs are misspelled as HQTLs. Use ctrl+F to find them. \n\n[1] Guo, et al. Log-Linear Attention.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback and overall positive comments. \nWe sincerely appreciate your thoughtful and high-quality review. We believe that we have good explanations to resolve the reviewer’s remaining concerns. Please find our responses below.\n\n> 1. The abstract and introduction lack a brief introduction to the method. The authors should have elaborated on the \"Three blending methods\" (i.e. various types of HQLT).\n\nThank you for this suggestion. We will amend the corresponding sentence in the abstract as follows:\n“We propose and compare three methods to blend these two systems into a single memory system, **differing in how and when input information is delivered to each system**, to leverage the strengths of both.”\nWe will similarly improve the introduction (Line 35).\n\n> 2. I thought that Delayed-Streaming HQLT and Delayed-Chunk HQLT just correspond to the serial and chunk parallel forms in linear attention, and the calculation results are the same. But from the authors' experiment, Delayed-Chunk HQLT is different from Delayed-Streaming HQLT in both calculation process and results, which confuses me. I hope the author can explain why Delayed-Chunk HQLT is a hybrid method independent of Delayed-Streaming HQLT.\n\nThe calculation results are different in Delayed-Streaming HQLT and Delayed-Chunk HQLT because the classic equivalence between recurrent-vs-chunk mode computation in linear attention does not hold when softmax is applied within the intra-chunk attention. To be more specific, the Delayed-Streaming variant has a sliding window attention with a stride of 1, i.e., **at every time step**, the oldest token from the KV-memory window is fed to the FW-memory, and a new token enters the KV-memory window; as a consequence, the memory content of FW-memory is updated at every time step, whereas in the Delayed-Chunk variant, the sequence is processed chunk-by-chunk, i.e., the content of FW-memory is only updated at the chunk-boundaries and remains constant while the system is processing the current chunk. Within a chunk, the content of KV-memory is updated at every time step by adding the new element in the chunk; its content is reset (to empty) at the chunk boundaries. Intuitively, Delayed-Streaming HQLT may be argued to be a better approach as the KV-memory always makes use of the full window size (except at the very beginning of the sequence), but we also included Delayed-Chunk which is not only a natural extension of chunk-wise parallel form of linear attention, but was used in prior work by Munkhdalai et al. [15]. \n\nThis is an important point we should have better clarified in the paper.  We will add a clarification sentence in Sec. 3.2 in the revision. Thank you very much for pointing this out. \n\n> 3. How does the training efficiency of hybrid architectures compare to linear models and transformers? A supplementary experiment may answer my concerns.\n\nThe information in the supplementary material provides an answer to this question (Appendix A.1 Line 386): “Training of 340M models using 4 H100-80GB GPUs takes about 8 hours for the baseline transformer and 10 hours for DeltaNet and all the HQLT models with the window size of 64 tokens. For the 1.3B models, these numbers become 26 hours for the baseline transformer and DeltaNet, and 30 hours for all the HQLTs variants.”\n\nGiven that the training for the 1.3B models uses 16B tokens, training throughputs are 170K tokens/second and 148K tokens/second for the baseline Transformer/DeltaNet and HQLT, respectively.\n\n> 4. The authors mentioned in the limitation that \"Investigating such a mechanism is not straightforward in the current era of model development, which sets hardware-efficiency as a hard requirement.\" I value this point of view of the authors. Recently, Log-linear attention [1] has achieved a balance in exploring the utilization of intermediate-scale memory and computational efficiency. I wonder what the authors think of this work. (This is just an open question)\n\nThank you for this open question for discussion. We believe it is a very interesting approach that nicely extends linear attention, while preserving the efficiency through an elegant parallel-form computation. That said, the fundamental memory mechanism (reading/writing primitives) of the proposed log-linear attention models still relies on those of the existing FW-memory models. In this sense, it is orthogonal to the comment we made in the Discussion, as our discussion was about future development to improve upon these memory primitives.\n\n> minors: 5. The word \"synaptic memory\" used in the abstract is confusing, as the authors does not provide any explanation in the paper. Either explain it in the text, or use FW memory instead, which may be easier to understand.  6. Some HQLTs are misspelled as HQTLs. Use ctrl+F to find them.\n\nThank you very much for pointing these out. We will address these points in the final revision.\n\nWe hope our responses above have successfully resolved the remaining concerns of the reviewer. If so, we would be grateful if the reviewer could consider raising the score. Thank you again for all the excellent suggestions that help us improve the final version of the paper.\n\n---\n**ADDITIONAL NOTE**: We will add one more set of experiments in the final revision. We conducted extra experiments to evaluate the Synchronous HQLT on **reinforcement learning in partially observable environments** involving retrieval, using the “passive visual match” task [Rebuttal Ref. 1]. This allows us to further validate our method on a domain outside of the language. The task is illustrated in Figure 2 of the corresponding paper [Rebuttal Ref. 1] (we are referring to this paper, because this year, the NeurIPS review system does not allow us to provide our own figures and plots during the rebuttal): In this task, an agent navigates in a 2D grid world (of size 7×11) delimited by impermeable walls. The agent can only observe the nearby pixels (5 × 5-grid centered on the agent). An episode in this task has three phases. During Phase 1 (whose\nduration is 15 time steps), the agent observes a color, randomly drawn from three choices, red, green or blue. In Phase 2 (750 steps), the agent is in a room with apples; collecting an apple yields a reward of 1. There are initially 10 apples, and they reappear every 20 steps; their positions are random. In Phase 3 (max. 15 steps), the agent is placed in a room with three colors; if the agent reaches the pixel with the color that matches the one provided in Phase 1, the episode ends successfully; it yields a reward of 100. Alternatively, Phase 3 terminates if the agent reaches a pixel with the wrong color or when the limit of 15 steps is reached (no reward is given in these cases). \n\nThe episode/sequence length for this task is maximum 780. The attention window size for the baseline quadratic transformer is set to 780 such that it covers the full episode sequence. We set the KV-memory window size of HQLT to 64 (i.e., less than 10 times the transformer window size). We obtain the following result in terms of average return (over 3 seeds):\n\n|  Model                |  Average Return |\n|-------------|----------------|\n| DeltaNet    |   253.7 |\n| HQLT         |   311.4 |\n| Transformer  |  335.1 |\n\nIn the final version, we will include the corresponding learning curves with 95% confidence interval, as is typically done to report RL experiments. We believe this will broaden the scope of the work and make it more appealing to the wider NeurIPS community.\n\n[Rebuttal Ref. 1] Ni et al. When do transformers shine in RL? Decoupling memory from credit assignment. NeurIPS 2023."}, {"review_id": "8wf1vLdxsN", "reviewer": "Reviewer_Ww5m", "summary": "This paper addresses the well-known limitations of quadratic and linear attention mechanisms in sequence modeling. The quadratic attention (i.e., standard softmax-based key-value attention) provides precise token retrieval but suffers from quadratic time and memory complexity with respect to sequence length. In contrast, linear attention (i.e., fast-weight programmer) scales linearly with sequence length and supports long-context processing, but struggles to precisely recall specific tokens, especially when the number of tokens to remember exceeds the hidden state dimension. To tackle these limitations, this paper proposes **Hybrid Quadratic-Linear Transformers (HQLTs)**, which combine the precise retrieval capabilities of quadratic attention and the expressive state-tracking of linear attention (specifically, DeltaNet). In particular, this paper introduces and compares three blending strategies (Delayed-Streaming, Delayed-Chunk, and Synchronous) and memory mixing and gating mechanisms to effectively integrate KV-memory and FW-memory outputs. Comprehensive experiments on language modeling, retrieval, and algorithmic tasks demonstrate the potential of HQLTs to mitigate the limitations of each memory system and offer new insights into the design of hybrid neural memory architectures.", "questions": "- **Q1**. Regarding the W1, could the authors clarify what distinguishes HQLT from existing hybrid models that also combine quadratic and linear attention mechanisms?\n- **Q2**. This paper blends quadratic and linear attention within a single layer. Could the authors elaborate on why this within-layer hybridization is preferable to a layer-wise separation strategy (e.g., using quadratic attention in earlier and linear attention in later layers, or vice versa)?\n- **Q3**. Is there potential for an intermediate approach with partial overlap between the Delayed-Streaming (no overlap) and Synchronous (full overlap) strategies? I would like to hear the authors’ perspective on this idea, including any expected benefits and drawbacks.", "limitations": "The authors adequately addressed the limitations of their work.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- **S1**. The paper clearly motivates the complementarity of quadratic and linear transformers, drawing an interesting connection to the theory of complementary learning systems.\n- **S2**. It systematically explores multiple blending strategies that are possible architectural designs for hybridization and empirically evaluates them on diverse tasks, including language modeling, retrieval, and algorithmic benchmarks.\n- **S3**. The proposed HQLT framework is compatible with any linear attention variant, making it extensible to future advances in fast-weight or linear attention methods.\n- **S4**. The broad range of experiments demonstrates the effectiveness of the hybrid model compared to using either quadratic attention alone or linear attention alone.\n\n### Weaknesses\n- **W1**. The concept of hybrid approaches that combine quadratic and linear attention is not entirely novel; prior works such as [1] have explored similar ideas.\n- **W2**. As shown in Table 3, on specific tasks (e.g., parity and modular arithmetic), all HQLT variants underperform compared to standalone DeltaNet, suggesting that hybridization can sometimes weaken the strengths of the fast-weight memory.\n- **W3**. While the Synchronous HQLT generally outperforms other variants, it is notable that in the 1.3B setting, the Delayed-Stream variant achieves better results on the ARC tasks. This indicates that we could face additional difficulty finding the most appropriate blending strategy for a given use case.\n\n[1] Transformer quality in linear time, ICML 2022", "was_revised": true, "final_justification": "The response adequately addresses the main concerns raised in the initial review. Accordingly, I have decided to retain my original score.", "rebuttal": "We thank the reviewer for many positive comments and their valuable feedback. We sincerely appreciate your thoughtful and high-quality review. We believe that we have good explanations to resolve the reviewer’s remaining concerns. Please find our responses below.\n\n> W1. The concept of hybrid approaches that combine quadratic and linear attention is not entirely novel; prior works such as [1] have explored similar ideas.\n\n> Q1. Regarding the W1, could the authors clarify what distinguishes HQLT from existing hybrid models that also combine quadratic and linear attention mechanisms?\n\nYes, we fully acknowledge that several prior works have discussed such hybrid modeling (including the reviewer’s reference [1] which is Ref. [25] in our work). Here, we contribute unique perspectives to hybrid strategies by introducing \"expressivity\" (see Table 1) as a new dimension for building hybrid models using the latest FW-models (no prior work on hybrid models have such a discussion), and how that can affect the precise choice of the hybridizing strategies (the choice made in prior work is arbitrary; e.g., Arora et al. [14] used the Sychronous approach, while Munkhdalai et al. [15] used Delayed-Chunk; but no evidence or explanation was provided regarding this choice). Similarly, we also discussed various mixing strategies, including “sum” (Arora et al. [14]) vs. “scalar mixing” (Munkhdalai et al. [15]). Overall, we provide more comprehensive and updated discussions on building hybrid models (Reviewer Z43k also highlighted this). We believe this is important given the increasing interest in developing hybrid models.\n\n> W2. As shown in Table 3, on specific tasks (e.g., parity and modular arithmetic), all HQLT variants underperform compared to standalone DeltaNet, suggesting that hybridization can sometimes weaken the strengths of the fast-weight memory.\n\nThank you for pointing this out. There was some sub-optimality with our hyper-parameter search space at the time of submission when reporting results in Table 3. We detected and fixed this problem while working on the supplemental materials. As we reported there (Table 7 in Appendix), after fixing this sub-optimal hyper-parameter search, the performance of HQLT matched the DeltaNet performance reported by prior work by Grazzi et al. [8] on these tasks: Synchronous HQLT achieves 100.0% and 97.0% accuracies on the parity and modular arithmetic tasks, respectively. We believe this update directly resolves the reviewer’s concern.\n\n\n> W3. While the Synchronous HQLT generally outperforms other variants, it is notable that in the 1.3B setting, the Delayed-Stream variant achieves better results on the ARC tasks. This indicates that we could face additional difficulty finding the most appropriate blending strategy for a given use case.\n\nThis is a valid point and we agree with the reviewer. However, the Delayed variants are fundamentally limited in their ability to leverage the expressivity advantages of FW memory (Table 3). More generally, we expect that performance differences between these model variants on the general language modeling results may vary depending on the specific choice of KV-memory (e.g., regular softmax attention vs. forgetting attention [Rebuttal Ref 2]) and FW-memory (e.g., Gated DeltaNet [6]) that will be used in the future. In contrast, we expect the importance of being Synchronous to leverage the expressivity strength of FW-memory will broadly remain a valid conclusion for any model types. Therefore, our work demonstrates a strong reason to prefer the Synchronous variant overall (which can be an important result for future work on hybrid models).\n\n[Rebuttal Ref 2] Lin et al. Forgetting Transformer: Softmax Attention with a Forget Gate. ICLR 2025\n\n> Q2. This paper blends quadratic and linear attention within a single layer. Could the authors elaborate on why this within-layer hybridization is preferable to a layer-wise separation strategy (e.g., using quadratic attention in earlier and linear attention in later layers, or vice versa)?\n\nThere are two conceptual motivations for within-layer blending. One is that the alternative, layer-wise separation strategy requires human ingenuity to decide where to put which type of memory models, whereas here in the within-layer strategy, the model can flexibly learn to use both types of memory systems in any layer in a context dependent fashion. We believe this yields a more general-purpose layer.\n\nAdditionally, for the quadratic-linear transformer hybrids, the two types of transformers are both based on query/key/value variables which can be naturally shared by the two memory systems within the same layer; this is conceptually elegant, and allows us to create hybrid models with a parameter count similar to the non-hybrid models. These explanations can also be found in our Discussion section 5 (paragraph Line 327).\n\n> Q3. Is there potential for an intermediate approach with partial overlap between the Delayed-Streaming (no overlap) and Synchronous (full overlap) strategies? I would like to hear the authors’ perspective on this idea, including any expected benefits and drawbacks.\n\nThis is a very interesting question. One conceptually sound approach (which we briefly mentioned in Sec 5. Line 353 Limitations) would be to introduce a “memory router” mechanism that dynamically decides, at every time step, which key/value memory pair should be sent to FW-memory (with or without delay), and which should remain in the KV-memory short window. We expect such a more sophisticated communication between KV-memory and FW-memory may give rise to a more powerful memory system. However (as we also mentioned in the same paragraph), given that hardware-efficiency and scalability is a hard requirement for machine learning models currently, such a design may not be straightforward for efficient training (but remains an open research question).\n\n\nWe hope our responses above have successfully addressed the remaining concerns of the reviewer. Thank you again for all the excellent suggestions that help us improve the final version of the paper.\n\n----\n**ADDITIONAL NOTE**: We will add one more set of experiments in the final revision. We conducted extra experiments to evaluate the Synchronous HQLT on **reinforcement learning in partially observable environments** involving retrieval, using the “passive visual match” task [Rebuttal Ref. 1]. This allows us to further validate our method on a domain outside of the language. The task is illustrated in Figure 2 of the corresponding paper [Rebuttal Ref. 1] (we are referring to this paper, because this year, the NeurIPS review system does not allow us to provide our own figures and plots during the rebuttal): In this task, an agent navigates in a 2D grid world (of size 7×11) delimited by impermeable walls. The agent can only observe the nearby pixels (5 × 5-grid centered on the agent). An episode in this task has three phases. During Phase 1 (whose\nduration is 15 time steps), the agent observes a color, randomly drawn from three choices, red, green or blue. In Phase 2 (750 steps), the agent is in a room with apples; collecting an apple yields a reward of 1. There are initially 10 apples, and they reappear every 20 steps; their positions are random. In Phase 3 (max. 15 steps), the agent is placed in a room with three colors; if the agent reaches the pixel with the color that matches the one provided in Phase 1, the episode ends successfully; it yields a reward of 100. Alternatively, Phase 3 terminates if the agent reaches a pixel with the wrong color or when the limit of 15 steps is reached (no reward is given in these cases). \n\nThe episode/sequence length for this task is maximum 780. The attention window size for the baseline quadratic transformer is set to 780 such that it covers the full episode sequence. We set the KV-memory window size of HQLT to 64 (i.e., less than 10 times the transformer window size). We obtain the following result in terms of average return (over 3 seeds):\n\n|  Model                |  Average Return |\n|-------------|----------------|\n| DeltaNet    |   253.7 |\n| HQLT         |   311.4 |\n| Transformer  |  335.1 |\n\nIn the final version, we will include the corresponding learning curves with 95% confidence interval, as is typically done to report RL experiments. We believe this will broaden the scope of the work and make it more appealing to the wider NeurIPS community.\n\n[Rebuttal Ref. 1] Ni et al. When do transformers shine in RL? Decoupling memory from credit assignment. NeurIPS 2023."}, {"review_id": "qm6rDcu2mE", "reviewer": "Reviewer_Z43k", "summary": "This paper introduces Hybrid Quadratic-Linear Transformers (HQLTs), a new class of sequence models that blend two complementary neural memory systems: quadratic key-value (KV) memory, known for precise retrieval but limited context due to quadratic complexity, and linear fast-weight (FW) memory, known for unbounded context and expressivity but imprecise recall. The authors explore three methods to integrate these systems - Delayed-Streaming, Delayed-Chunk, and Synchronous - and conduct extensive experiments to evaluate them across language modeling, retrieval, and synthetic algorithmic tasks. Results show that the synchronous variant, which allows both memory systems to operate concurrently on the same input, outperforms others by effectively combining the strengths of each memory system. The work establishes that such hybridization not only overcomes individual limitations but also enables new capabilities in memory system design, especially when advanced FW memory models like DeltaNet are used.", "questions": "1. Can you show a clear case where hybridization gives a meaningful win over both standalone Transformer and DeltaNet, even with tuning or scaling? This would help justify the added complexity.\n\n2. The choice between sum, scalar, and vector mixing seems arbitrary. What guides this design? Is there a principled way to adapt mixing across tasks or contexts?\n\n3. Retrieval tasks still rely heavily on the KV-memory window size. Could you explore a mechanism where FW-memory actively contributes to retrieval, not just as a backup store?\n\n4. Comparison to other recent hybrids like Infini-attention or Mamba-based models is missing. Can you provide head-to-head results to clarify what your method uniquely improves?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Quality: The paper is technically solid and presents a complete piece of work. The proposed hybrid models are carefully evaluated across a wide range of tasks, and the use of advanced fast-weight methods like DeltaNet adds credibility. The authors are honest about limitations and support their claims with solid empirical evidence. However, the performance improvements are sometimes small, and the architectural blending techniques feel more incremental than transformative. Some choices, like gating mechanisms, would benefit from further theoretical justification.\n\nClarity: The writing is mostly clear, and the paper is well-organized. The motivation and high-level ideas are easy to follow, but certain sections - especially those describing the hybrid mechanisms - are dense and could be made more accessible. The notation can be overwhelming at times, and clearer visualizations or example-driven explanations would help. Overall, the reader can understand the core contributions, but digesting the technical details takes effort.\n\nSignificance: The paper tackles a meaningful problem in transformer architecture design - merging long-context capability with precise memory. The proposed models are relevant to practical scenarios and could influence future work on efficient attention. That said, the real-world impact is still limited by modest gains and unresolved issues with retrieval. The contribution is useful, but not groundbreaking, and the adoption by the broader community will depend on future extensions.\n\nOriginality: The idea of combining quadratic and linear attention is not new, but this paper brings a more thorough and updated evaluation, especially by using stronger fast-weight models. The synchronous design and detailed empirical analysis are clear advances over prior work. Still, the methodological novelty is limited - the core ideas are largely recombinations of known mechanisms. The paper is more valuable for its careful integration and insight than for introducing radically new techniques.", "was_revised": true, "final_justification": "I appreciate the author's clarifications and have adjusted my evaluation upward accordingly.", "rebuttal": "We thank the reviewer for their valuable feedback and overall positive comments. We sincerely appreciate your thoughtful and high-quality review. We believe that we have good explanations to resolve the reviewer’s remaining concerns. Please find our responses below.\n\n> (Quality) … Some choices, like gating mechanisms, would benefit from further theoretical justification.\n\n> (Question 2) The choice between sum, scalar, and vector mixing seems arbitrary. What guides this design? Is there a principled way to adapt mixing across tasks or contexts?\n\nWhile we have no \"theoretical justifications”, the choices of gating/mixing mechanism studied here are not arbitrary: the “sum” and “scalar mixing” strategies are those used by the closest prior works on hybrid models by Arora et al. [14] and Munkhdalai et al. [15], respectively. (However, based on the prior work, it is unclear what’s the impact of these choices). The last variant, “dynamic vector mixing” is another classic way to combine two pieces of information in deep learning in the style of LSTM-gating. We hope this explanation will resolve the reviewer’s main concern regarding the “arbitrariness” of the gating mechanisms chosen to be studied here.\n\n> (Clarity) The writing is mostly clear, and the paper is well-organized. The motivation and high-level ideas are easy to follow, but certain sections - especially those describing the hybrid mechanisms - are dense and could be made more accessible. The notation can be overwhelming at times, and clearer visualizations or example-driven explanations would help.\n\nThank you very much for sharing this valuable opinion. One challenge we are facing here is that further simplifying the notations would result in ambiguous mathematical model descriptions. We also tried our best to intuitively illustrate the model in Figure 1. Given that there are other reviewers who seem to like our thorough descriptions/equations (e.g., Reviewer GAJr), we kindly ask the reviewer to confirm (and potentially tell us) if there are specific parts of our description or figure that have room for improvement in terms of accessibility/clarify. We will be more than happy to address the corresponding writing following the reviewer’s suggestions.\n\n> (Significance) … That said, the real-world impact is still limited by modest gains and unresolved issues with retrieval. The contribution is useful, but not groundbreaking, and the adoption by the broader community will depend on future extensions.\n\nWe fully acknowledge this limitation. As the reviewer also highlighted positively in the evaluation  of Originality as “The paper is more valuable for its careful integration and insight than for introducing radically new techniques.”, we believe our conceptual contribution has significance for future work as we highlighted how a specific choice of hybrid strategy may have significant impact on the hybrid’s abilities to leverage certain properties of individual systems, here by focusing on the expressivity of FW-memory (Table 3). Also, to the best of our knowledge, no prior work discusses the complete picture of complementarity between quadratic/linear attention that includes expressivity, as is done in our Table 1.\n\nFurthermore, to introduce one additional contribution as an empirical work, we conducted an additional experiment to evaluate the Synchronous HQLT on **reinforcement learning (RL) in partially observable environments**, using the “passive visual match” task [Rebuttal Ref. 1]. \n\n[Rebuttal 1] Ni et al. When do transformers shine in RL? Decoupling memory from credit assignment. NeurIPS 2023.\n\nThis allows us to validate our method on a domain outside of the language, and also generally highlight the relevance of hybrid models outside of the language modeling world. The task is illustrated in Figure 2 of the corresponding paper [Rebuttal Ref. 1] (we are referring to this paper, because this year, the NeurIPS review system does not allow us to provide our own figures and plots during the rebuttal): In this task, an agent navigates in a 2D grid world (of size 7×11) delimited by impermeable walls. The agent can only observe the nearby pixels (5 × 5-grid centered on the agent). An episode in this task has three phases. During Phase 1 (whose duration is 15 time steps), the agent observes a color, randomly drawn from three choices, red, green or blue. In Phase 2 (750 steps), the agent is in a room with apples; collecting an apple yields a reward of 1. There are initially 10 apples, and they reappear every 20 steps; their positions are random.  In Phase 3 (max. 15 steps), the agent is placed in a room with three colors; if the agent reaches the pixel with the color that matches the one provided in Phase 1, the episode ends successfully; it yields a reward of 100. Alternatively, Phase 3 terminates if the agent reaches a pixel with the wrong color or when the limit of 15 steps is reached (no reward is given in these cases). \n\nThe episode/sequence length for this task is maximum 780. The attention window size for the baseline quadratic transformer is set to 780 such that it covers the full episode sequence. We set the KV-memory window size of HQLT to 64 (i.e., less than 10 times the transformer window size). We obtain the following result in terms of average return (over 3 seeds):\n\n|  Model                |  Average Return |\n|-------------|----------------|\n| DeltaNet    |   253.7 |\n| HQLT         |   311.4 |\n| Transformer  |  335.1 |\n\nIn the final version, we will include the corresponding learning curves with 95% confidence interval, as is typically done to report RL experiments. We believe that including this RL experiment will broaden the scope of the work and make it more appealing to the wider NeurIPS community.\n\n> (Question 1) Can you show a clear case where hybridization gives a meaningful win over both standalone Transformer and DeltaNet, even with tuning or scaling? This would help justify the added complexity.\n\nThank you for this question. First of all, we’d like to justify the added complexity by achieving a single model that can simultaneously perform expressive computation that Transformers cannot, and do better retrieval than DeltaNet. Nevertheless, our response to Question 3 below provides evidence for the case where HQTL outperforms both individual Transformer and DeltaNet.\n\n> (Question 3) Retrieval tasks still rely heavily on the KV-memory window size. Could you explore a mechanism where FW-memory actively contributes to retrieval, not just as a backup store?\n\nThis is an excellent question. We conducted extra experiments to examine if FW-memory could also potentially complement KV-memory in retrieval tasks. We trained one more 340M-param baseline transformer with a context window size 1024 (as opposed to 2048 in the main baseline). We compared this KV-memory-only baseline to the Synchronous HQLT model with sliding window size 1024. We obtained the following results on the retrieval-intensive tasks of Table 4:\n\n| Model | KV window size |  SWDE | SQuAD | FDA | Avg |\n|-------------|----------------|-------------|----------------|-------------|----------------|\n| Transformer ++ | **1024** | 30.4 | 25.5 | 31.2 | 29.0 |\n| Synchronous-HQLT | **1024** | 34.0 | 37.1 | 50.1 | **40.4** |\n\nHere we observe that HQLT with FW-memory outperforms Transformer with only KV-memory, when both models have the same KV-memory window size. While this remains a small-scale experiment, we believe this is a highly encouraging empirical result demonstrating that FW-memory can potentially behave more than just as a “backup storage” for retrieval, when its span covers longer context than the standalone KV-memory. We will add this discussion in the final revision. We really appreciate the reviewers’ comment prompting us to conduct these useful experiments.\n\n> (Question 4) Comparison to other recent hybrids like Infini-attention or Mamba-based models is missing. Can you provide head-to-head results to clarify what your method uniquely improves?\n\nWe thoroughly discussed this in our discussion section (from Line 337 and forth; note that Munkhdalai et al. [15] is the Infini-attention paper). Infini-attention can be obtained in our modeling framework by replacing DeltaNet with either a vanilla linear transformer or a sub-optimal version of DeltaNet (sub-optimal in light of recent advances in DeltaNet [5, 23] as we explained in detail in Line 337) and by using the Delayed-Chunk strategy. As we have shown, such a setting yields a strictly weaker model than our best Synchronous model (see Table 3). Similarly, Mamba is strictly more limited than DeltaNet in terms of expressivity (we did report Mamba performance from Grazzi et al [8] in Table 3), and Mamba also has been shown to underperform other advanced models such as (Gated) DeltaNet in prior work (see [6] and [23]) etc. as a standalone sequence model. Therefore, to save compute-resource, we eliminated these weak baselines that are known to be conceptually and/or empirically less powerful than the models discussed here based on results from well-known prior papers.\n\nWe hope our responses above have successfully resolved the remaining concerns of the reviewer. If so, we would be grateful if the reviewer would consider raising the score. Thank you again for all the excellent suggestions that help us improve the final version of the paper."}, {"review_id": "sjtPATQcq5", "reviewer": "Reviewer_GAJr", "summary": "This paper proposes Hybrid Quadratic-Linear Transformers (HQLTs) that combine key-value (KV) memory from standard transformers with fast-weight (FW) memory from linear transformers, specifically using DeltaNet. The authors argue these systems have complementary properties: KV-memory provides precise retrieval but suffers from quadratic complexity, while FW-memory supports unbounded sequences with higher expressivity but lower retrieval precision. The authors proposed 3 strategies to blend these two systems together: Delayed-Streaming (old KV pairs feed into FW-memory), Delayed-Chunk (chunk-wise processing), and Synchronous (both memories process inputs simultaneously). The authors evaluate on language modelling (340M and 1.3B parameters trained on 15B tokens), synthetic algorithmic tasks (parity, modular arithmetic), and retrieval tasks (FDA, SWDE, SQuAD), concluding that the Synchronous variant performs best.", "questions": "1. **On theoretical grounding**: Why invoke \"Complementary Learning Systems\" when your dichotomy (efficiency vs. precision) fundamentally differs from the established neuroscientific concept (episodic vs. semantic memory with different learning rates)?\n2. **On efficiency claims**: Table 4 shows HQLT needs 1024-token windows for competitive retrieval. The appendix claims costs are \"negligible\" but provides no analysis. Can you provide concrete FLOPs/throughput comparisons? Specifically: HQLT-1024 vs Transformer-2048?\n3. **On LAMBADA**: Why does HQLT dramatically outperform on LAMBADA with just 64 tokens when other retrieval tasks fail? This contradicts your other results and deserves investigation.\n4. **On synergy**: How do the algorithmic task results demonstrate synergy rather than just \"DeltaNet works when included\"? What can the hybrid do that neither component could achieve alone?\n5. **On comparisons**: Why no comparison to other long-context solutions (Mamba, Transformer-XL, EM-LLM, etc.)? How does HQLT compare to simply using Flash Attention with longer contexts?\n\n_Clear criteria for score improvement_:\n\n- Provide thorough computational analysis showing efficiency gains\n- Demonstrate retrieval working with small windows (<256)\n- Show true synergistic capabilities beyond individual components\n- Correct theoretical framing, better justify your choice or remove CLS references\n- Compare against modern long-context baselines", "limitations": "The authors acknowledge some limitations (e.g., large windows still needed for retrieval) but don't adequately address:\n- Computational overhead of dual memory systems\n- Questionable practical advantages over simpler alternatives\n- Sensitivity of results to hyperparameters\n- Missing comparisons to other long-context solutions", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n**Quality:**\n- The paper includes a comprehensive experimental evaluation across diverse tasks (language modelling, algorithmic reasoning, retrieval).\n- The authors tried different model scales (340M, 1.3B) with consistent training protocols. Of course, more scales could be useful, but I do recognise that this requires very large computation capabilities that are not always available.\n- The paper also contains thoughtful ablations on mixing strategies, window sizes, and FW-memory choices.\n- The authors provided their source code, which aids reproducibility and is commendable.\n- Finally, there is a clear mathematical formulation of the three proposed hybrid variants.\n\n**Clarity:**\n- The paper is well-structured with clear motivation and background.\n- It contains helpful visual illustrations (Figure 1).\n- It also contains explicit equations for each variant\n\n**Originality:**\n- The paper updates prior hybrid attempts by using the (relatively) state-of-the-art and very interesting DeltaNet model instead of vanilla linear attention.\n- It also includes the systematic exploration of three conceptually distinct blending strategies mentioned above.\n- Finally, the evaluation on expressivity tasks to test memory synergy is (to my knowledge) novel.\n\n### Weaknesses\n**Quality:** I believe there are a few very important issues here which the authors need to clearly address to have their work accepted in a conference like NeurIPS.\n1. **Misappropriation of established terminology**: The first big issue is that the paper invokes \"Complementary Learning Systems\" (CLS) from neuroscience but proposes an entirely different dichotomy, as far as I understand. Classical CLS (McClelland et al., 1995) describes hippocampal-neocortical differences in learning rates and memory types (episodic vs. semantic), not computational trade-offs between precision and efficiency. The authors do not justify this properly which leads me to believe that it is a misuse of the term, thus raising concerns about theoretical grounding. However, there might be a rationale behind this that is not clear from the current version of the manuscript, in which case I would kindly ask the authors to elaborate.\n2. **Core claims not supported by results:**\n\t* On standard language tasks, improvements are typically at least <1% over baselines. The only notable gain is ~15% on LAMBADA perplexity, which appears task-specific. Given the added architectural complexity, these minimal gains don't justify the approach.\n\t* Table 4 shows HQLT requires large windows (1024 tokens) to approach Transformer performance. With small windows (64), performance is dramatically worse (19.8 vs 44.7 avg accuracy). This suggests the hybrid isn't truly leveraging complementary strengths but rather defaulting to brute-force quadratic attention.\n3. **Missing critical computational analysis**: Despite computational efficiency being a primary motivation, the paper provides no FLOPs analysis or throughput comparisons. The appendix dismisses this with \"the actual computational cost is negligible\" (line 459), which is inadequate. How does HQLT with 1024-token window compare to Transformer++ with 2048 tokens?\n4. **The expressivity results don't demonstrate synergy**: Synchronous HQLT solving parity/modular arithmetic merely shows DeltaNet works when included (see Table 3 and Appendix Table 7), not that the hybrid achieves something neither component could alone.\n\n**Clarity:** (Minor issues)\n- Inconsistent terminology (HQLT vs HQTL)\n- The LAMBADA anomaly (strong performance with small windows) is unexplained\n- Defers crucial efficiency discussion to appendix\n\n**Significance:**\n1. **Limited practical impact**: If large windows are still required for retrieval, the hybrid approach fails to solve the fundamental limitation it targets\n2. **Algorithmic task results are unsurprising**: As mentioned above, showing that Synchronous HQLT (which includes DeltaNet) solves tasks that DeltaNet already solves doesn't demonstrate synergy.\n3. **No comparison to other long-context solutions** (Transformer-XL, EM-LLM, Compressive Transformer, etc.)\n\n**Originality:**\n- Core hybridization concept exists in prior work (Arora et al., Munkhdalai et al.)\n- Main distinction is using DeltaNet, which feels incremental\n- The successful \"Synchronous\" approach is conceptually straightforward", "was_revised": true, "final_justification": "After reviewing the authors' rebuttal and additional experiments, I am raising my score from 3 to 4 and the quality rating. The new HQLT-1024 vs Transformer-1024 results (40.4 vs 29.0) demonstrate genuine complementarity between the memory systems, and the systematic finding that synchronous blending preserves DeltaNet's expressivity while delayed variants don't provides valuable architectural insights for future hybrid models. While practical improvements remain modest and computational trade-offs exist (~20% throughput reduction), the work makes meaningful contributions to understanding how to effectively combine complementary memory systems, which is particularly relevant given the community's current interest in hybrid architectures.", "rebuttal": "We sincerely thank the reviewer for their valuable and thorough feedback. We believe there are a few factual misunderstandings that have resulted in the given score, which we’d like to resolve. Overall, we believe that we have good explanations to resolve all the concerns.\n\n> 1. (Misappropriation of established terminology & Question 1) Why invoke \"Complementary Learning Systems\" ?\n\nThe decision of referencing CLS was carefully discussed among the authors; we justify it as follows. First of all, we very explicitly state that the complementarity we discuss is **different** from those of the classic CLS; please see Line 33: “Here we propose a **different division of labor** into complementary linear and quadratic memory systems each with complementary strengths (Table 1).” \n\nWe still concluded that “complementary” is an appropriate/compelling term to use when contrasting the two transformer variants.  The fundamental question we ask here is (Line 26): “How could we achieve multiple computational properties that are incompatible within a single system?”  Under this context, we invoke CLS, because, beyond the specific nature of complementarity represented in CLS, it also represents an important example of a high-level principle of achieving multiple incompatible goals by combining complementary systems. We find it inappropriate to dismiss such an important prior discussion on complementarity. \n\nIf the reviewer still believes that there are misleading sentences that may lead readers to believe we conflate the specific complementarity we discuss with that of CLS, we kindly ask the reviewer specify which sentences are problematic. \n\n> 2. (Core claims not supported by results:) On standard language tasks, improvements are typically at least <1% over baselines. The only notable gain is ~15% on LAMBADA perplexity, which appears task-specific. Given the added architectural complexity, these minimal gains don't justify the approach.\n\nThere seems to be some misunderstanding. We have no such “core claims” based on the experiments of Table 2. On the contrary, we very explicitly stated that these general language modeling experiments are sanity checks: see in Line 202: “We conduct our main experiments using three types of tasks: general language modeling tasks (for general evaluation and **sanity checks** of HQLTs; Sec. 4.1)” as well as in Line 210 “We begin with evaluating general language modeling capabilities of the proposed HQTL models as a **general sanity check.**”. \n\n> (2. continued) Table 4 shows HQLT requires large windows (1024 tokens) to approach Transformer performance. With small windows (64), performance is dramatically worse (19.8 vs 44.7 avg accuracy). This suggests the hybrid isn't truly leveraging complementary strengths but rather defaulting to brute-force quadratic attention.\n\nWe would like to respond to this in two steps. First, we conducted extra experiments by training one more 340M-param baseline transformer with a context window size of 1024 (as opposed to 2048 in the main baseline). We compared this KV-memory-only baseline to the Synchronous HQLT model with sliding window size 1024 (reported in the submission). We obtained the following results on the retrieval-intense tasks of Table 4:\n\n| Model | KV window size |  SWDE | SQuAD | FDA | Avg |\n|-------------|----------------|-------------|----------------|-------------|----------------|\n| Transformer ++ | **1024** | 30.4 | 25.5 | 31.2 | 29.0 |\n| Synchronous-HQLT | **1024** | 34.0 | 37.1 | 50.1 | **40.4** |\n\nHere we see that the combination of KV- and FW-memory effectively improves over the KV-only system, showing that FW-memory can potentially help KV-memory in retrieval even though it is not its main strength.\n\nSecond, we clarify that we made no claim regarding such a “synergy”: we clarified upfront that the strength of KV-memory is retrieval, and FW-memory’s strength is expressivity. In this regard, we clearly commented that (Line 354): “the results on the retrieval tasks are still unsatisfactory (Sec. 4.3): it seems that a large KV-memory window is unavoidable for precise retrieval”). Therefore, we respectfully disagree with the reviewer’s claims that our “Core claims are not supported by results”. All our claims are based on our experimental results, and we honestly discussed limitations where appropriate (Reviewer Z43k highlighted this among our strengths).\n\n> 3. (Missing critical computational analysis & Question 3): Despite computational efficiency being a primary motivation, the paper provides no FLOPs analysis or throughput comparisons. The appendix dismisses this with \"the actual computational cost is negligible\" (line 459), which is inadequate.\n\nFirst of all, computational efficiency is not our “primary motivation”. This is an important misunderstanding to be resolved. The core essence of this study is that **we have multiple goals**: one is indeed to minimize long context quadratic attention, another one is to obtain strong expressivity, and yet another one is to enable precise retrieval. Our main claim is that these goals cannot be all achieved by an individual system. Building and studying hybrid systems to simultaneously achieve these normally incompatible goals is our primary motivation.\n\nSecondly, we kindly ask the reviewer to consider the full sentence (not just a partial segment which is misleading). The full sentence was (Line 459 in the appendix): “In practice, within the range of window sizes discussed here, the actual computational cost is negligible, since during inference with a batch size of 1, short-window attention can be efficiently computed with a few/two steps of parallelizable matrix multiplication on a GPU, regardless of whether the window size is 64 or 1024. Training time is also only minimally affected by the increased window size as training is parallelized over the sequence elements.” etc… We did not “dismiss” it.\n\nThe run-time comparison was reported in Appendix A.1 (Line 386): “Training of 340M models using 4 H100-80GB GPUs take about 8 hours for the baseline transformer and 10 hours for DeltaNet and all the HQLT models” etc… Given that the 340M models are trained using 15B tokens, all the information was available to compute the throughputs. Regarding Transformer-2048 vs. HQLT-1024, their throughputs are 520K tokens/second vs 417K tokens/second, respectively. We agree it is unfortunate that there is no gain in efficiency overall (we made no such a claim), but in exchange, the resulting hybrid can perform expressive computation that transformers cannot (Table 3), and do better in retrieval than DeltaNet (Table 4).\n\n>4. (The expressivity results don't demonstrate synergy & Question 4): Synchronous HQLT solving parity/modular arithmetic merely shows DeltaNet works when included (see Table 3 and Appendix Table 7), not that the hybrid achieves something neither component could alone.\n\nThat is our conclusion. We never claimed that we expect “synergy” between the two systems in terms of expressivity. We are currently not aware of any formal language tasks on which transformers outperform DeltaNet (even though such an investigation is an interesting future research direction). We clarified upfront that expressivity is FW-memory’s strength that is lacking in KV-memory.\n\n> (Question 4, part 2) What can the hybrid do that neither component could achieve alone?\n\nThis is an excellent open question. We are currently not aware of tasks on which both FW-memory and KV-memory fail systematically. We believe this is a very important question for future work.\n\n> (Question 3) On LAMBADA: Why does HQLT dramatically outperform on LAMBADA with just 64 tokens when other retrieval tasks fail? This contradicts your other results and deserves investigation.\n\nThere is another misunderstanding: LAMBADA is not a retrieval task. None of the tasks in Table 2 is. It is a general text completion task relying much more on the recent contexts compared to the retrieval-intensive tasks studied in Table 4. There is nothing contradictory about this result.\n\n> (Question 5) On comparisons: Why no comparison to other long-context solutions (Mamba, Transformer-XL, EM-LLM, etc.)? How does HQLT compare to simply using Flash Attention with longer contexts?\n\nOur work contains neither claims nor experiments regarding long-context processing, which is well-studied in prior work on linear attention/DeltaNet, including comparison to Mamba and Transformer-XL; see e.g., [5, 23, 15]. All our language experiments are evaluated with the length of 2048 (and as we specified, all the quadratic attention implementations use FlashAttention2 [16]).\n\n> (Clear criteria for score improvement): Demonstrate retrieval working with small windows (<256)\n\nWe already reported that the short window size of 256 is not enough for the retrieval tasks studied in Table 4. Here we conducted an extra experiment to evaluate the HQLT on reinforcement learning in partially observable environments involving retrieval (due to the rebuttal length limitation, we kindly ask the reviewer to check the ADDITIONAL NOTE in our response to Reviewer **Mb9d** for further details). There, we demonstrate strong retrieval performance of HQLT using a small window size of **64** compared to a transformer with the full context access of length 780.\n\n> (Limitations:) Sensitivity to hyperparameters\n\nWe’d like to remind the reviewer that unfortunately, this is not specific to our work. We are not aware of any (recent) work on language modeling at this scale which shows hyper-parameter sensitivity w.r.t., e.g., learning rate.\n\nWe hope our responses above will, at very least, resolve the core misunderstandings. We provided our thorough response to all the critiques/criteria for score improvement raised by the reviewer.  If the reviewer found our response useful, we would be grateful if the reviewer would consider raising the score. In any scenario, we thank the reviewer again for their thorough and thoughtful review."}]

Author rebuttal:

Meta-review: Accept.

This paper provides a valuable and systematic study of how to best combine quadratic and linear attention mechanisms. Its key finding is that a "Synchronous" blending strategy preserves the expressivity of the linear component (DeltaNet), a crucial insight for designing future hybrid models.





While the practical performance gains on some standard benchmarks are modest, the authors' rebuttal significantly strengthened the paper. They introduced new experiments providing clear evidence that the hybrid approach offers genuine complementarity, outperforming a standard Transformer on retrieval tasks when controlling for window size. All reviewers converged on acceptance after the rebuttal , acknowledging the work's importance to the research community's growing interest in hybrid architectures.
Decision: Accept (poster)